Requirements for out-of core FEM solvers

[alex_stief]

I was wondering which metric would be the most important to boost performance for out-of-core solvers. Sequential read/write, IOPS or latency? I.e. which spec should I focus on to get the best performance.
Commodity SSDs like Samsung 970Evo/Pro could be striped to deliver maximum sequential read/write. Wit 4 of them running out of SLC cache should not be an issue. More expensive enterprise SSDs can deliver more IOPS. And then there are some rather specialized pieces of equipment that boast very low latency, like Samsung Z-NAND. Or would Optane wipe the floor with the competition?

My take on this: most of these solvers were developed in the dark ages when fast-spinning drives were everything we had. So I would imagine that the codes were developed with sequential I/O in mind, avoiding high latency and low IOPS of these drives. This would mean that maximum sequential read/write would be the way to go. Is there a point of diminishing returns?
Anyone who has experience in this field?

 

[alex_stief]

Nothing? Is the question too vague or is the topic too far from the area of expertise of the STH forum?

 

[Evan]

I am sure some people here have experience but probably a rather limited number. Try a HPC type forum to get more potential input ?

Also try running it on a RAM disk, that will tell you something about how it will really respond to better latency and bandwidth.
As you say it was probably written long time ago and you may find the code itself is more the limit that the IO platform it sits on.

 

[alex_stief]

So I dug a little deeper, and as I suspected, sequential read/write is where it is at.

What I need now is a setup that can provide a fairly high sequential read and write speed, but there is a catch: the budget is surprisingly tight. Somewhere in the 1000€ range.
My initial thought was 4 consumer-grade m.2 NVMe SSDs on a card like this: ASRock > ULTRA QUAD M.2 CARD
Then either do a software RAID0 or let the FEM package handle its out-of-core data on multiple drives simultaneously, which apparently is a thing.

But I can't decide on which drives to use. The cheaper m.2 drives all have very limited PBW ratings, so I fear that we would be killing them in less than a year. And on top of that, they can run out of SLC-cache rather quickly, and info about the actual size of the SLC cache plus write speed without it is rather scarce.
My other option would be 2 Intel 900P 480GB. The advantage would be consistent performance, but much lower peak performance. And compared to e.g. 4x1TB commodity SSDs, the storage space is much lower.

So I guess I am looking for a recommendation for relatively cheap m.2 NVMe SSDs in the 500GB-1TB range, that can handle sustained loads without loosing too much performance, and rather high write endurance.

Edit: the host system is a HP Z8 G4, so PCIe 3.0 is the limit.

 

[alex_stief]

Had a very informative phone call with a developer today.
Apart from lots of tips on file system and RAID settings, the most important takeaway was this: sequential read matters even more than sequential write.
So I settled on two Samsung PM1725b 1.6TB. With 5400MB/s sequential read, 8.76PB write endurance and a price of approximately 500€, they are the perfect match for this requirement.

 

[amalurk]

Raid-0 them and post benchmarks here before you get started so we can drool.

 

[Deslok]

Did you consider used enterprise drives on ebay? The PM1725b on ebay is in the 250USD range for the capacity you're looking at letting you go for four of them Samsung MZWLL1T6HAJQ-00005 PM1725B 1.6TB PCIe Gen3 X4/ 2-PORT Solid State Drive | eBay

 

[alex_stief]

If it was for me, I would choose used enterprise gear whenever the opportunity presents itself.
But since this is for my day job at a larger company, there is just no way to buy used gear. Getting a green light to upgrade anything with parts I picked is hard enough. I can't even influence where these will be sourced, and have to rely on the company's preferred hardware vendor to be in the same price range.

 

[Deslok]

If it was for me, I would choose used enterprise gear whenever the opportunity presents itself.
But since this is for my day job at a larger company, there is just no way to buy used gear. Getting a green light to upgrade anything with parts I picked is hard enough. I can't even influence where these will be sourced, and have to rely on the company's preferred hardware vendor to be in the same price range.

Fair enough, just thought I'd toss that out there(corporate IT sucks some days I'm there myself)

 

[alex_stief]

Can someone help me calm my nerves, I am having second thoughts

I just realised the PM1735 became available, which is the successor to the PM1725b.
But I also realised that these drives are OEM only. I don't care too much for the warranty, but I am not sure how much I need to care for missing driver and firmware support for NVMe SSDs.
Based on stories I read on the internet, NVMe performance can be abysmal without proper drivers installed. At least on Windows.
These drives will be used in a Linux workstation, how is the situation there?

 

[vanfawx]

My testing of NVMe drives is limited, but all the Intel NVMe drives I've used on Linux have always performed as expected on Linux using the built-in nvme kernel module. I would expect the same from your Samsung drives, but never hurts to test with FIO.

 

[alex_stief]

I will surely run a few tests, but at the end of the day, testing will only reveal the symptoms.

 

[alex_stief]

The hardware (2x PM1735 1.6GB) finally arrived, and getting it to work was surprisingly straightforward.

I opted against a RAID0 of both drives, for two reasons.
The license for this solver only allows us to use 4 threads for a single simulation. Although I have not run conclusive tests on this matter, I highly doubt that 4 threads will fully saturate sequential read speeds beyond 10GiB/s.
Instead, the second drive will go into a similar workstation. This way, we can at least run 2 simulations at the same time.
Along with the drives, I ordered 24x32GB of DDR4-2666 reg ECC. These go into the first workstation (HP Z8 G4) for a total of 768GB, and the leftover RAM will go into the second workstation (also HP Z8 G4), making it 384GB there.
EDIT: I was wrong, the software can fully saturate the sequential reads/writes of both drives in RAID0. So RAID0 it is! Turned out we need the higher storage capacity anyway.

For the EXT4 file system on the drives, I made sure to have proper alignment (Partition Alignment - Thomas-Krenn-Wiki) and disabled journaling (just another blog: ext4 disable journal). The fstab entry looks like this:
/dev/nvme0n1p1 /scratch ext4 defaults,data=writeback,noatime,nodiratime,nodiscard,barrier=0,sync 0 0

An initial test with a custom program confirms that the drive does operate very close to its spec sheet performance values of 2400MB/s sequential write, 7000MB/s sequential read. So far, so good. All that is left is a test with the actual program.

 

[alex_stief]

And lastly, the results:
For the compute part of the benchmark I used, I got a speedup factor of around 12.
A horribly unoptimized library for some special output format (Medina) drops the overall speedup to around 5. But that can be worked around by reducing the output quantities, or switching to a different format altogether.