Best Practice Guide Cfd 2

ALESSIA: EP 28189 Application of Large Eddy Simulation to the Solution of Industrial problems: BEST PRACTICE GUIDE: LES AND ACOUSTICS

Issue 1

ALESSIA/AEA/Best Practice Guide Issue 1

ALESSIA: EP 28189 Application of Large Eddy Simulation to the Solution of Industrial problems: BEST PRACTICE GUIDE: LES AND ACOUSTICS

Authors Christiane Montavon*, Ian Jones*, Dick ten Bosch+, Stefan Szepessy †, Roland Henriksson†, Hans Moberg†, Roberto Tregnago ‡, Zoubida El Hachemi #, Massimiliano Piccirillo#, Michel Tournour#, Sylvie Dequand#, Frederic Tremblay !, Rainer Friedrich!

* AEA Technology Engineering Software + Shell International Oil Products ‡ Centro Ricerche Fiat # LMS Numerical Technologies † Alfa Laval Separation ! Technical University, München Issue 1 Internal Version 1.4 Date 18/6/2002

AEA Technology Engineering Software ii

ALESSIA/AEA/Best Practice Guide Issue 1

SUMMARY Time dependent flow phenomena play an important role in many industrial processes, including aerodynamic noise generation. In many cases experimental and theoretical analyses of the problems are not feasible, either due to their high expense or to extreme operating conditions. The primary aim of the Alessia Project has been to develop validated and supported software tools for the prediction of industrially important fluctuating flow problems. The tools are based around the technique of Large Eddy Simulation (LES), which solves for the large-scale fluctuating flows, the Large Eddies, and uses ‘sub-grid’ scale turbulence models for the small-scale motion. The leading European software packages, CFX for Computational Fluid Dynamics (CFD), and SYSNOISE for the acoustics, have been coupled together in this project to perform the prediction of industrially relevant flows, and the consequent noise field. The main limitation to the solution of these problem has been the computational cost of computer predictions. HPCN has been an enabling technique that has provided the solutions for these multi-physics and multi-scale problems. This report is a subset of one of the deliverables of the project and summarises some of the experiences learnt from the project, in the form of a Best Practice Guide. It also contains some considerations for the choice of computing system for LES computations.

AEA Technology Engineering Software iii

ALESSIA/AEA/Best Practice Guide Issue 1

CONTENTS

1

INTRODUCTION

1

2

THE OVERALL STRATEGY

1

3

WHEN TO USE LES

2

4

RECOMMENDATIONS FOR LES PROBLEM SET UP

3

5

4.1

Boundary Layers

3

4.2

Inlets

3

4.3

Outlets

3

4.4

Geometrical Domain

4

4.5

Meshes

4

4.6

Numerical Considerations

4

4.7

Advection scheme

4

4.8

Algorithmic Considerations, Time stepping

4

4.9

Accelerating Convergence

6

4.10

Choice of LES model

6

4.11

Storage Of Results And Back Up

6

AEROACOUSTIC CALCULATIONS

7

5.1

SYSNOISE modules

7

5.2

Harmonic Acoustic FEM

7

5.3

Coupled Harmonic FEM

8

5.4

Acoustic I-FEM

8

5.5

Harmonic Acoustic BEM

8

5.6

Coupled Harmonic BEM

9

5.7 Numerical Consideration 5.7.1 Mesh coarsening

9 9

AEA Technology Engineering Software iv

ALESSIA/AEA/Best Practice Guide Issue 1

5.7.2 5.7.3 6

7

Free edges Junctions

9 9

SYSTEM REQUIREMENTS & CHOICE OF PLATFORMS

10

6.1

Operating environment

11

6.2

STORAGE OF RESULTS AND BACK UP

11

REFERENCES

12

AEA Technology Engineering Software v

ALESSIA/AEA/Best Practice Guide Issue 1

1 Introduction Time dependent flow phenomena play an important role in many industrial processes, including aerodynamic noise generation. In many cases experimental and theoretical analyses of the problems are not feasible, either due to their high expense or to extreme operating conditions. The primary aim of the Alessia Project [ 1] has been to develop validated and supported software tools for the prediction of industrially important fluctuating flow problems. The tools are based around the technique of Large Eddy Simulation (LES), which solves for the large-scale fluctuating flows, the Large Eddies, and uses ‘sub-grid’ scale turbulence models for the small-scale motion. The leading European software packages, CFX for Computational Fluid Dynamics (CFD), and SYSNOISE for the acoustics, have been coupled together in this project to perform the prediction of industrially relevant flows, and the consequent noise field. The main limitation to the solution of these problems has been the computational cost of computer predictions. High Performance Computing and Networking (HPCN) has been an enabling technique that has provided the solutions for these multiphysics and multi-scale problems. A summary of Validation results for this project are to be found in [ 2]. The partners in the project have combined their experiences on these problems to produce a Best Practice Guide and Recommendations. This focuses on: • • • •

When to consider the use of LES, instead of conventional Reynolds Averaged modelling. How to use CFD software for large-scale transient simulations in an HPC environment. This will include the recommendations on solver parameters, for efficiency. Recommendations for size of cluster, and cluster configurations, speed of interconnection, memory requirements for different size problems and applications. Description of the parallel algorithms, and the specific issues on the running of large dynamic transient cases

2 The Overall Strategy Turbulent flows are notoriously difficult to predict, and they are one of the Grand Challenge problems of science and engineering. The usual approach is to use the ‘Reynolds Averaged Navier –Stokes’ equations, (RANS), which solve for time averaged quantities. However, there are some situations where the approach is not adequate, and the alternative approaches of Large Eddy Simulation (LES) or Direct Numerical Simulation (DNS) can be adopted. With these methods, time dependent equations are solved for the chaotic motion with either no approximations and all relevant scales resolved (DNS) or the equations are filtered in some way to remove very fine time and length scales (LES). These approaches require fine grids and can give very good results. In addition, they can give details on the structure of the turbulent flows, such as pressure fluctuations and Lighthill stresses, which cannot be obtained in other ways. This has always required the use of tuned special purpose software, using the largest supercomputers available. This type of software is also usually very restrictive in the kind of geometries that can be handled, with simple regular grids used to resolve the geometries. This has restricted the problems that can be typically solved to cylinders and boxes, with ‘immersed boundary methods’ (blocking off cells) to deal with complex

AEA Technology Engineering Software 1

ALESSIA/AEA/Best Practice Guide Issue 1

geometries. This has wasted significant memory because of the blocking off. This project has attempted to use the latest generation of commercial CFD software, with powerful methods for dealing with complex geometries in conjunction with LES. Because of the difficulty of validating the methods for solving problems which are inherently chaotic, the strategy has been to use very accurate solutions from the MGLET software, from the Technical University of Munich, to provide benchmark quality data for testing and evaluating the industrial software. This has inevitably required the use of the largest supercomputers available in Europe, for the benchmarking, along with lower cost ‘industrial strength’ systems, based on commodity workstations, for the industrial computations.

3 When to use LES This section outlines the situations where the underlying LES methodology can be used. For Low Reynolds numbers, < 5000, consider DNS if you have a large computer available, and it is important to be able to resolve the scales, for example for transition to turbulent flow. For Higher Reynolds numbers, LES might be a suitable option for cases where: • the flow is likely to be unstable, with large scale flapping of a shear layer or vortex shedding. • the flow is likely to be unsteady with coherent structures (cyclone, flasher) • the flow is buoyant, with large unstable regions created by heating from below, or by lighter fluid below heavier fluid (multi-phase flows in inclined pipelines). • conventional RANS approach are known to fail (for example due to highly anisotropic turbulence) • a good representation of the turbulent structure is required for small-scale processes such as micro-mixing or chemical reaction. • the noise from the flow is to be calculated, and especially when the broadband contribution is significant. • other fluctuating information is required (e.g. fluctuating forces, gusts of winds) • the user can afford to wait for up to a week for results, using an 8-16 processor system. Note that if the user has a simple geometry, requires good quality results, and has access to large computers, he should consider the use of a special purpose code such as MGLET, rather than a less accurate commercial package. For more complex geometries however, a commercial package with LES should be more suitable.

AEA Technology Engineering Software 2

ALESSIA/AEA/Best Practice Guide Issue 1

4 Recommendations for LES problem set up 4.1 BOUNDARY LAYERS Is the boundary layer structure important: If so, resolve it, with at least 10 points across the boundary layer, and with the first grid point at a position of y+ around 1. The Technical University of Munich’s experience is that for accurate LES calculations, they cannot gain much over a Direct Numerical Simulation (DNS) and would recommend at least 30 points as a minimum. In planes parallel to a wall, it is possible to use coarser grids, provided the flow remains attached.

4.2 INLETS The representation of the turbulent structures at inlets can be a difficult part of the setup. As could be seen from the NTH case, the detailed properties of the incoming flow had a strong effect on the development of the jet in the enclosure. For the case of the cyclone however, the turbulent conditions at the inlet seemed to have relatively little impact on the flow in the device, which was essentially determined by very strong anisotropy effects in the cyclone body. Cases for which the inlet turbulence plays a significant role are developing boundary layers and turbulent jets. If the inlet turbulence is felt to play an important role, the user should consider a) Going upstream, with an appreciable region of the inlet resolved to use LES for the flow upstream. b) Use of pre-computed LES flows, storing only the values on a plane c) Use of SNGR model and RANS models to generate inlet flows with the right turbulent structures. d) New methods such as Detached Eddy Simulation (DES) show promise, but they are not fully proven yet. In particular, there could be upstream effects from the LES portion, or the turbulent fluctuations in the upstream boundary layer could be necessary to trigger off the unstable shedding of vortex sheets. This is a new area, and more experience is required. Beware the claims of the originators of new methods, since they may not report the bad experiences, and the difficulties encountered.

4.3 OUTLETS With the transport of turbulent eddies outside of the computational domain, some recirculation can occur at outlets. Our experience shows that if the code is allowed to bring some fluid back into the domain at outlets, it can destabilise the solution. We therefore recommend in CFX-5 to use OUTLET boundary conditions rather than OPENING when performing a LES. OPENING type of boundary conditions allow the flow to come back in, whereas with OUTLET boundary conditions, the code builds artificial barriers at the boundary, when the flow tries to come back in. These barriers are later taken away when the flow goes out again.

AEA Technology Engineering Software 3

ALESSIA/AEA/Best Practice Guide Issue 1

The use of artificial walls at outlets might introduce some not very physical behaviour locally, but it increases the robustness of the calculation.

4.4 GEOMETRICAL DOMAIN Even though there might be symmetries in the geometry and flow, the geometrical model should include the full region, since while the time averaged flows may be symmetric, the instantaneous flows are not, and restricting the domain could constrain the turbulence. TwoDimensional Approximations are particularly bad.

4.5 MESHES Remember that the mesh and time steps are an inherent part of the model. The LES models make use of the grid scale for filtering out the turbulence. If the mesh is anisotropic to resolve a jet, for example, the longer length scale in the flow direction may also have an undue effect on the cross-stream turbulence. For this reason, consider the use of isotropic grids, maybe using tetrahedral elements, rather than hexahedral meshes.

4.6 NUMERICAL CONSIDERATIONS VLES Models, (Very Large Eddy Simulations) can be considered when small-scale turbulence is not important, and large-scale structures are naturally unstable. An example of this is for the vortex shedding from a cylinder.

4.7 ADVECTION SCHEME • •

Where possible, methods should be at least second order accurate in space. Second order CENTRAL DIFFERENCE methods have certainly given good answers within the Alessia project, both in CFX-5 and MGLET. Second order ‘Higher Upwind’ Differencing Methods and the QUICK scheme, based on backward difference methods appear to be a little too diffusing, and can damp out turbulence. Similarly, almost second order accurate methods, with a flux limiter to prevent overshoots also seem to be too diffusive. Some Japanese experience, however, indicates that QUICK, in conjunction with no turbulence model, appears to give good results in some cases. This is not recommended by the results from the Alessia project.

4.8 ALGORITHMIC CONSIDERATIONS, TIME STEPPING •

Tuned LES solvers conventionally use explicit methods, with central differencing in space, with second or fourth order methods both in space and time, with regular rectangular grids. Very simple, Smagorinsky based models are typically used. These are best for highly accurate solutions, which challenge computing limits. With these methods there is a reluctance to use methods that transport turbulence quantities, for issues such as memory usage. This limits the method to length scales such that the transport of sub-grid scale turbulent kinetic energy is not important.

AEA Technology Engineering Software 4

ALESSIA/AEA/Best Practice Guide Issue 1

• •

• •

• •

•

•

•

• •

For tuned LES explicit time-stepping programs, the dominant cost is that of solving a Poisson equation at each time step, for the pressure field. This can still be very time consuming. CFX-5 has the advantages for complex geometries, where the use of a body-fitted or unstructured grid is necessary, either to resolve the geometry, or to resolve the flow (at steps, near separation points, in boundary layers). It will also be able to make use of new models being developed, eg for combustion or multi-phase flows, and provide commercial support and infrastructure. Spectral and pseudo-spectral methods can give the most accurate results, but they are very limited in scope, and when there is insufficient grid resolution, the results can have large errors. Use second order time stepping. 2nd order backward differencing seems to be relatively robust, and gives the right behaviour, despite the fact it is not an ‘A-stable’ method. That is, it can have different stability properties to that of the underlying differential equation. Second order centred time stepping, like Crank-Nicolson, leapfrog, are more difficult to run, and are prone to having different solutions on alternate time steps. First order fully implicit methods in time usually are too diffusive, and the turbulence is damped out. For highly unstable problems, such as the cyclone, lower order methods may work, but the results will be very damped, unless very small time steps are used. For Explicit methods, keep CFL condition below 0.25. For a non-uniform grid, this can restrict the time step to be constrained by the smallest grid size in the problem, even if it is not in an important region of the flow. Refining the mesh also means that much smaller time steps have to be taken, to maintain the CFL condition. Implicit methods can have a higher CFL condition, especially in regions that are not particularly important, and the grid size is small for geometric considerations. However, for accuracy, CFL numbers of Order 0.5 – 1 should be used. Larger values can give stable results, but the results may be damped. For smaller time steps, with an implicit coupled solver as in CFX-5, one iteration (Coefficient loop) can be carried out at each time step, to give a semi-implicit method. This method may not be as stable as using two or three iterations. When the flow properties have settled down, then the method may be better1. A large proportion of computing time is spent getting to the stage when it is possible to start taking statistics. If this time can be shortened, faster results can be obtained. This is discussed in more detail later. 1,000 – 10,000 time steps are typically required for getting converged statistics. More steps are required for second order quantities, (variances) than for means. Check the convergence of the statistics. For a vortex-shedding problem, several cycles of the vortex shedding are required. This effect has also been seen in experimental results, where the evidence of the results of TUM indicate that measurements should have been taken for a longer time scale.

1

When more complex cases are considered with e.g. mixing of multicomponent flows, it is advised to perform more coefficient loops per time step (typically 6-8). For these cases, unless the mass fraction equations are solved in a fully coupled way together with the momentum and continuity equation, the mass conservation at the component level will not be guaranteed with only one coefficient loop per time step.

AEA Technology Engineering Software 5

ALESSIA/AEA/Best Practice Guide Issue 1

4.9 ACCELERATING CONVERGENCE As indicated above, it can take a lot of time steps to get to the stage where it is possible to accurately compute statistical quantities. It is necessary for the flow to have all the right length scales for the turbulent flows, and this involves the cascade of turbulence from large scales to small scales, and maybe associated back-scatter, from small scales to larger scales. Consider the following ways of accelerating the development of statistically steady flows: • Use the results from a similar case to start the flow • Use less accurate methods with larger time steps to allow the large scale structures to develop • Use results from a RANS calculation to give the right mean fields. • Use the SNGR model to give the flow the correct turbulent length scales. • Get convergence on a coarse mesh, and interpolate the results onto a fine mesh. This should retain the larger turbulent scales, and the smaller scales will develop more quickly.

4.10 CHOICE OF LES MODEL •

•

•

The Smagorinsky model contains one free parameter, c0 that has a default value of 0.1. The standard value has been giving good results in most applications, although it appears that for boundary layer flows and for jets, a lower value can improve agreement with measurements (e.g. CRF test case, NTH test case). It has however been observed that it is easy to blame the model constant, when some other shortcomings like a bad representation of the upstream conditions can affect the flow just as much as a change in the constant (e.g. NTH case). To use dynamic models such as Germano, which compute the model coefficient incorporating explicitly filtered resolved fields, the resolution needs to be sufficiently good to give better results than Smagorinsky. Any zero order turbulence model will have the same considerations. It is sometimes argued that it is important for the models to allow for backscatter (energy cascading from the smaller to the larger scales close to walls) to improve the simulation of boundary layers. Models such as INC & DINC developed by TUM have shown encouraging results during a priori testing. These models however when used in a simulation still require arbitrary clipping of the viscosity in order to be stabilised. These models still lack the required robustness to be implemented in a commercial package.

4.11 STORAGE OF RESULTS AND BACK UP During the course of a calculation, we typically recommend to do a full backup of the results every 50 time steps. At the end of the computations, there will then be two files, a full results file at the end of the run, and a separate one created earlier in the run, at the last backup.

AEA Technology Engineering Software 6

ALESSIA/AEA/Best Practice Guide Issue 1

The system should definitely have a read-write CD on one of the systems, and possibly a large dat tape system. DVD writers are just becoming available commercially, (600 Euros) and may also be a viable alternative. Once it is becomes necessary to create an animation, or results for acoustic calculations, it is necessary to dump these results out at more frequent intervals, possible every time step. To minimise file space, the user ought to exercise care as to what and when data is required. Ideally the system should have a very large scratch disk available for the storage of these results. Note that because the acoustic calculations are typically carried out in the frequency domain, the whole time information is required, in order to perform the FFT for the frequency domain. If the grid for the acoustic calculations is known a-priori, it is possible to create a CFX definition file for this grid, and interpolate fine grids onto the coarser mesh, for acoustics postprocessing.

5 AEROACOUSTIC CALCULATIONS The Alessia project has used an approach of decoupling the flow and acoustics calculations, using the Lighthill analogy. This is good for situations where the acoustics is required in the far field, and the flow domain can be much smaller. For near field calculations, the fluctuating pressure field the ‘pseudo-pressure’ can give a good indication of the local noise field, but not if reflections and diffraction effects are important. Some authors advocate the use of highly accurate methods, to couple the flow and acoustics in a single framework, using very high order methods. This may be appropriate for some simple flow cases, but does not make full advantage of the facilities and models available in commercial software packages. If the flow is strongly compressible, it may be necessary to couple the flow and acoustics more closely, for example by carrying out a full compressible flow calculation, within the CFD solver. This is particularly good when there are acoustic resonances, and the acoustics affects the flow. In the approach chosen in the project (relevant for incompressible, low Mach number flows), the aeroacoustic calculations are performed using SYSNOISE. The next two sections give guidelines on which module to use in SYSNOISE together with some numerical consideration.

5.1 SYSNOISE MODULES 5.2 HARMONIC ACOUSTIC FEM Acoustic FEM can be used to model closed (‘interior’) fluid regions. 1-d (line/pipe) 2-d (plane or axisymmetric) and 3-d (volume) models are possible. The benefit of FEM is that different elements can have different sound speeds and/or densities, to model element-by-element variations in acoustic properties (eg, due to temperature change, or – using special volume

AEA Technology Engineering Software 7

ALESSIA/AEA/Best Practice Guide Issue 1

absorbent elements – to model thick absorbent regions, such as foam or wool fill). Transfer across gaps between element free faces can be modelled using special transfer impedance relationships, for example to model a perforated membrane. Boundary conditions can include surface vibrations (assumed, imported from a structural model, or from tests) surface impedance (absorption layer) and acoustic sources. Solutions can be direct acoustic response to the excitation, or modal analysis (resonance of the enclosed region) followed by acoustic response by modal superposition.

5.3 COUPLED HARMONIC FEM Coupled Acoustic FEM enables the full two-way fluid-structure interaction to be taken into account in an acoustic FE model. The module includes a structural element library (mainly shells and plates) and also enables the import of structural models from standard FE codes, as modal models. Structural modal damping can then be applied. Boundary conditions can then include structural constraints (fixations, hinges) and forces and moments (mechanical excitation).

5.4 ACOUSTIC I-FEM Infinite elements extend finite element acoustic models into infinite or partly-infinite domains. The acoustic results (pressure, intensity, radiated power) can be determined in the infinite elements as well as in the near field (modelled with acoustic finite elements). The advantage of FEM plus I-FEM over BEM, is that the near field region can have any of the standard FEM features (inhomogeneous fluid, volume absorber, …). Two-way fluid-structure coupling can also be applied, as in the Coupled FEM module. The disadvantage of FEM plus I-FEM is the need to create a significant near-field volume mesh for the FEM part. As well as a geometry-modelling/meshing task, this can also result in a larger numerical problem size in some cases (hence, longer calculation times). The balance between FEM/I-FEM and BEM is often very application-dependent.

5.5 HARMONIC ACOUSTIC BEM Acoustic BEM can be used to model both closed (interior) and infinite (exterior) domains, and mixed situations where the interior is open to the exterior through certain holes etc. SYSNOISE has both Direct BEM (which requires a single, closed mesh) and Indirect BEM (which allows quite arbitrary topologies, including free edges and t- and x-junctions). The Indirect method in particular has the benefit of enabling efficient models of thin structures having (the same) fluid on both sides, for example flexible casings or loudspeaker membranes. The mesh generation for a BE model is much simpler than for FEM, since only a ‘shell-like’ mesh on the surfaces is required. However, solution times for BE models are typically longer, especially if many frequencies are to be analysed, due to the frequencydependence of the solution parameters and because only direct solutions (no modal approaches) are possible. 2-d (plane and axisymmetric) and 3-d (volume) domains can be modelled.

AEA Technology Engineering Software 8

ALESSIA/AEA/Best Practice Guide Issue 1

Boundary conditions include surface vibration (as in acoustic FEM) surface impedance (absorption layer) and acoustic sources. In the Indirect BE method, surface-related boundary conditions can be applied on one side of the element only, if required. Transfer through selected elements can be modelled using special transfer impedance relationships, for example to model a perforated membrane. Fluid properties have to be homogeneous throughout the domain. Acoustic results at any location can be found by using appropriate field point grids.

5.6 COUPLED HARMONIC BEM Coupled Acoustic BEM enables the full two-way fluid-structure interaction to be taken into account in an acoustic BE model. The module includes a structural element library (mainly shells and plates) and also enables the import of structural models from standard FE codes, as modal models. Structural modal damping can then be applied. Boundary conditions can then include structural constraints (fixations, hinges) and forces and moments (mechanical excitation).

5.7 NUMERICAL CONSIDERATION 5.7.1 Mesh coarsening An important step in the preparation of a boundary element or finite element model is to coarsen as much as possible the original CFD grid. This step is important because it reduces the solution time of the acoustic problem that is proportional to N3 where N is the number of unknowns. The six-element-per-wavelength rule for predicting the maximum frequency of an acoustic mesh is recommended. Nevertheless, this rule ensures the convergence of the matrix operators but does not consider the distribution of the sources. Therefore, this discretisation of the acoustic model must be such that at least six elements per wavelength are used and that the sources distribution is correctly represented. 5.7.2 Free edges When working with the indirect approach of the boundary element method, the primary unknowns are the single and double layer potentials. The layer potentials correspond to the velocity jump and pressure jump through the boundary surface. Along free edges, zero jump of pressure must be applied. This condition represents the fact that there can’t be any pressure discontinuity through the surface along free edges because the two sides are in contact with each other. 5.7.3 Junctions In indirect boundary element approach, it is possible to model a structure composed of subparts that are joined together at a given edge. For example, a T- or X-junction can be modelled. The line along which the different surfaces join is called a junction and requires some special treatment.

AEA Technology Engineering Software 9

ALESSIA/AEA/Best Practice Guide Issue 1

A node located along a junction needs to be duplicated (eventually several times) to correctly represent the different pressure discontinuity through the surface. A compatibility equation needs to be defined to account for the fact that the total pressure discontinuity is equal to zero. It is recommended to use the default automatic junction algorithm that has been tested on a wide range of problem and proved to accurately detect junctions.

6 System requirements & Choice of platforms Details on parallel performance of CFX-5 and SYSNOISE for the various supported platforms are presented in [ 1]. The conclusions about system requirements and choice of platforms are repeated here. LES computations are very expensive in term of CPU time, and really require HPCN systems for such simulation to be performed on industrial applications. From a price performance viewpoint, the testing done in the project on various platforms shows that there is no point to using dedicated supercomputers unless: • Your organisation has one, or it is very cheap. • The software you are using can extract high efficiency from the vector and parallel system. This usually means special purpose programs. However, they can gain an order of magnitude in performance. • You need ultimate performance,(eg real time) as price performance is much better from the commodity workstation systems. For larger problems, the Unix workstations have 64 bits operating systems, and efficient execution of double precision calculations. They also can address more memory. Consider Unix systems with shared memory paradigm. (SGI, Compaq Ultra Server, SUN, IBM, HP) for these problems. IBM SP systems seem to be more difficult to administer. If you regularly run large problems in a multi-user Unix environment, then the large memory will enable mesh generation and visualisation on large problems, whilst still running LES calculations. For low cost systems based on Intel / AMD systems, then we found that: • On similar systems, Linux shows a similar efficiency to Windows NT or 2000. • Efficiency of low-level parallel libraries, (PVM / MPI) for NT varies considerably. • Dual processor systems provide good performance but not as good as single processor system. Both on Linux and Windows there can be a performance hit using dual processors. • On Windows NT/2000, for more than two processors, CFX-5 does not work as efficiently in a split shared-memory and networked manner. Reasons for this are not understood, but the conclusions could change with newer versions of operating systems or software. • Linux and distributed memory using PVM most cost effective, for problems up to 2 M cells. • PVM is more robust than MPI, for released versions of CFX-5, especially on clustered systems.

AEA Technology Engineering Software 10

ALESSIA/AEA/Best Practice Guide Issue 1

• • • • • •

Ethernet using 100 Base T connections seems to be adequate. Consider partitioning the network to minimize network traffic, or using a dedicated network for ultimate performance. It is perfectly feasible to use under-used systems for providing additional parallel systems. CFX use its training network for this purpose, for example. 1 Gbytes per processor is recommended, (previously 500 Mbytes recommended). For these systems, it is often not possible to run the larger problems on a small number of processors, because of the memory limitations of these systems. As large disks as possible, at least 120 G Bytes. Compilers are free for Linux. Buy for now. Whatever you buy will be outdated in a year.

Results obtained on Unix / Linux can still be processed on Windows NT/2000.

6.1 OPERATING ENVIRONMENT One of the major issues to consider when using HPC is the overall system management, especially the allocation of resources to users, and the scheduling of jobs. This can impose a significant cost to the management of these systems, as a departmental system can often be run in a more informal manner. However, consideration should be given to • For more than one user, consider using a system such as LSF to dynamically manage systems and queues. There are other systems available, some public domain. We have no experience of these. • There are also other system utilities, such as Perfmon.exe , which can be used to check on whether a system is being utilised.

6.2 STORAGE OF RESULTS AND BACK UP The system should definitely have a read-write CD on one of the systems, and possibly a large dat tape system. DVD writers are just becoming available commercially, (600 Euros) and may also be a viable alternative. Once it is becomes necessary to create an animation, or results for acoustic calculations, it is necessary to dump these results out at more frequent intervals, possible every time step. Careful though about what data is required, and when that data is required is very important. Ideally the system should have a very large scratch disk available for the storage of these results. Note that because the acoustic calculations are typically carried out in the frequency domain, the whole time information is required, in order to perform the FFT for the frequency domain. If the grid for the acoustic calculations is know a-priori, it is possible to create a CFX definition file for this grid, and interpolate fine grids onto the coarser mesh, for acoustics postprocessing.

AEA Technology Engineering Software 11

ALESSIA/AEA/Best Practice Guide Issue 1

7 References [ 1] Jones, I P and Montavon, C A, The Alessia Project, available from http://www.software.aeat.com/cfx/european_projects/alessia/alessia.htm [ 2] Jones, I P and Montavon, C A, The Alessia Project, Validation Report, available from http://www.software.aeat.com/cfx/european_projects/alessia/alessia.htm

AEA Technology Engineering Software 12

ALESSIA/AEA/Best Practice Guide Issue 1