Investigation of the influence of cpu and memory configuration - Epyc rome

[EagerToLearn]

Hello everyone,

I am going to write a paper about the influence of the CPU and memory configuration on things like FEM calculations. Full disclosure: I intend to publish my results in a peer-reviewed scientific journal, which means it is not going to be open access. However, if any of you are interested in proof reading it, I won't turn you down

Why: I've read numerous times, that the memory bandwidth is the limiting factor in FEM calculations, but couldn't find real investigations using modern hardware. Also, how much does the performance change, if different parameters are varied.

What have I planed: Compare the performance (e.g. calculation time for FEM) for different setups: 4 and 8 memory modules, 3200 MHz, 2933 MHz and 2666 MHz as well as different CPU speeds (2,6 GHz, 3,45 GHz and 4 GHz (if the 7502 will do that)) and core counts ( 8 - 16 -32 - if the software supports it)

This week I plan on finishing the preparations (find suitable FEM models etc.). If you have anything you would like tested, please let me know.

List of things I will test (will be updated based on your suggestions):
- FEM implicit (using ABAQUS)
- FEM explicit (using ABAQUS)
- Cinebench
- Windows Performance Monitor (for keeping track of memory/cpu utilization)

If anyone is interested in co-authoring, feel free to message me. Especially if you have FEM experience, deeper knowledge of computers or a modern Intel system

Kind regards
Mike

 

[ari2asem]

i am willing to provide some data for you.

i have 2 systems:
- dual epyc 7551 with 16* 8gb 2666mhz (totall of 128gb at 2666mhz). no overclock

- dual epyc 7742 with 16* 32gb 3200 mhz (totall of 512gb at 3200mhz). no overclock.

no experience with FAM.

let me know if you are interesseted

 

[EagerToLearn]

Thank you very much, that is a very kind offer. The main problem will be the software licence, since I am going to use the one from my university and I am not sure, if I could "lend" that to you. But I will check.

Do you have any ideas/suggestions for other programs/benchmarks that might be interesting?

 

[ari2asem]

please, give me an exact name of FEM-software you use for Windows

List of finite element software packages - Wikipedia

en.wikipedia.org

another suggestion....using photogrammetry software like Metashape, or

Comparison of photogrammetry software - Wikipedia

en.wikipedia.org

very cpu-intensive softwares when you don't use GPU-cards

 

[EagerToLearn]

I am using Abaqus (and if ANSYS licences are free, maybe that as well for comparison). I'll take a look at your list, thx.

 

[ari2asem]

i red this as HW or software requirements for Abaqus

Hardware configurations, operating system and operating system patches recommended for the use with this level of Abaqus products.

Windows Operating System
Windows 10 Professional and Enterprise
Windows Server 2012 R2, 2016 (with or without HPC Pack)
Processor x86-64 Intel or AMD
C++ Compiler Visual Studio 2017 (15.9.4)
FORTRAN Compiler Intel Visual Fortran 16.0 Update 1
MPI and Interconnects
Microsoft MPI
Microsoft MPI and supported interconnects
Workload Manager Windows HPC (only Windows Server)
GPGPU Select NVIDIA and AMD

is this correct? do i need all these things to run Abaqus?

 

[EagerToLearn]

That is a good question, I have never tried running it without the listed components.

Although, as far as I know, Fortran/C++ compiler is only required when using certain features like user defined materials

 

[alex_stief]

You don't necessarily need everything from that list.
I'm like 99% certain it will run on any flavor of Windows 10. The software is just not validated for the cheaper options because ya'know, professional application...
No need for a compiler if you don't want to add your own code.
The installation should come with its own MPI library. Depending on how you set up the simulation, Abaqus/standard solver runs without MPI anyway.
Node interconnects are only needed if you want to run distributed across multiple machines.
Not sure why a workload manager would be needed in Windows.
And you don't need a fancy GPU if you don't want to run GPU accelerated.
But yeah, you will need licenses, which don't come cheap. Especially for running on many cores.

Be careful with Ansys. Their student versions have all the features of the regular versions, just cell/node count limitations. But Ansys explicitly states that running performance benchmarks is not allowed with the student licenses.

I think you can scrap the idea of running cinebench. It is on the exact opposite end of computational intensity, i.e. doesn't scale well with memory bandwidth.
Something you might not get around with the chosen topic is stream benchmark. It is the de-facto standard for evaluating memory bandwidth in the HPC community.

I am not sure if you are doing yourself a favor by using Abaqus as a centerpiece for this work. First of all, it is closed source, so it will be nearly impossible to estimate code balance and derive a performance model (like e.g. roofline). Two steps present in the vast majority of HPC publications I read. Maybe you can dig deeper with tools like likwid or some other profiling software. Warning: very steep learning curve ahead.
And while Abaqus/explicit might behave as expected (depending on the cases you run), Abaqus/standard might be difficult making any sense of the results. Last time I checked, scaling was influenced by a surprisingly large serial portion left in the code. But that might have been a result of the the case I had to work with. And the ability to run a hybrid parallelization (like OpenMP+MPI) really complicates things. Before drawing any conclusions from such runs, you will have to experiment with different core binding strategies. Or find a way to make core binding work at all, something I gave up on with that software. Using a CPU like Epyc Rome with its chiplet design will add another layer of complexity here.
But of course, it depends where the focus of your publication is, and to an extent, where it will be published. The hardcore HPC community might not be too fond of running some tests with a closed source code. A more application oriented community might be more welcoming towards that idea.

 

[EagerToLearn]

Thank you very much for the detailed response Alex.

That is a lot of information, I am going to process on the weekend. But a quick look at your suggested programs made me think, that those are exactly the suggestions and critical thinking I am looking for

 

[Spartus]

We should talk. I might be able to co-author with you. This is basically my specialty, although it comes from an ANSYS-centric viewpoint.

YGPM