Recommended FS for Shared storage to back NVME-oF/RoCE?

[IamSpartacus]

I'm putting together parts (and repurposing a lot of parts) for home shared storage server that will serve up fast storage (10-16 PCIe 3.0 x4 NVMe U.2 drives) to a 2 hosts + QDevice Proxmox cluster.

Main services running will be 4K media streaming/serving to lots of clients simultaneously. So this data will house the media itself, along with all the databases and metadata that organizes and servers that data. This storage will also house a handful of Windows (daily driver workstations) and Linux VMs and about 30-40 microservices running in LXCs.

In the past I've just used ZFS RAID10 mirrors for my U.2 drives but (admittedly I haven't tried to do a TON of tuning) I've never gotten read/write performance anywhere close to what my disks are rated for.

Open to suggestions for those who have done fast storage over Ethernet.

 

[T_Minus]

You don't have a random read\write workload no need to tune for that performance.

Your workload sounds 90% write 10% read and 99% sequential.


Keep the mirrored setup, make sure your recordsize is tuned (higher) for your streaming work-load, and you don't need sync so hopefully you weren't trying to force that on all.

I don't see how your system is constrained for streaming with that # of mirrored NVME, unless you're using crappy NVME drives that severely throttle or "lots of clients" means 100s and something else is going on.

 

[IamSpartacus]

You don't have a random read\write workload no need to tune for that performance.

Your workload sounds 90% write 10% read and 99% sequential.


Keep the mirrored setup, make sure your recordsize is tuned (higher) for your streaming work-load, and you don't need sync so hopefully you weren't trying to force that on all.

I don't see how your system is constrained for streaming with that # of mirrored NVME, unless you're using crappy NVME drives that severely throttle or "lots of clients" means 100s and something else is going on.

To clarify, as I can see from re-reading my OP (I've edited it) it's not as clear (and missing information) as it was in my head...I'd say the workload is more the opposite of what you describe. Probably more like 70% read and 23% write. The performance issues I speak of are when I do benchmarking or sequential read/write testing of large 4K files on my ZFS RAID10 pool of 6 x Intel P4510 4TB drives. I can't even get much higher than 3.5GB/s. Sure that is fine if I'm only serving media off this storage and nothing else. But if I'm moving this storage to a "shared storage" model, then it will also be used for some VMs (a few daily driver workstations) and lots of micro services running in LXCs.

Currently I'm using 2 pools. 6 x Intel P4510's in RAID10 mostly for storing media. 4 x Intel 905p's for all my appdata, databases, and metadata.

 

[i386]

For Hdds I would go with 256KByte strip/chunk size per hdd
64 KByte for nand ssds
4 KByte for optane ssds

I can't even get much higher than 3.5GB/s.

With software raid I always have to think twice or more times about the math and what the numbers mean
If the 3.5GByte/s is the "net" speed than the gross speed is 7GByte/s (mirror = 1 "orginal" data + 1 "copy" data), if it's gross speed than the net speed would be ~1.75GByte/s (or about 600 MByte/s per leg of the raid 10) and that would definitely be slow (too slow for the intel mentioned ssds!)

 

[Rand__]

Keep in mind that each nvme drive might use a cpu core and that higher frequency means more performance, so depending on CPU choice this might be an issue

This will be less an issue on regular operations (where the average process might not utilize a core completely ) but could be a problem while benchmarking

 

[IamSpartacus]

Keep in mind that each nvme drive might use a cpu core and that higher frequency means more performance, so depending on CPU choice this might be an issue

This will be less an issue on regular operations (where the average process might not utilize a core completely ) but could be a problem while benchmarking

I've been doing all testing and benchmarking on an EPYC 7443 in performance mode. It can all core/thread boost up to about 3.4Ghz but single core boost 4.0Ghz. With about half the cores maxed I've seen them all hit 3.8GHz.

 

[Rand__]

Ok, that should be plenty

It sounds as if you're mixing different requirements (iops/latency relevant databases), throughput oriented (streaming clients) so you might need to cater for that (dataset level).

How have you tested (locally vs remote, which tool and parameters, throughput or latency) and why do you think it won't scale up with more turning wheels?

You said you ran 6 P4510's (which are not the fastest), so if you have 3.5GB/s write over network thats not bad actually (6 drives, 3x2 drives mirror = 3 write devices = 1.1GB/s per drive) ...

The problem with RoCe is that its not enough if the cards speak it, but the whole chain needs to comply, client side to server software (smb/nfs/iscsi) needs to be rdma enabled... I have not found anything that combines that with a nice gui :so that means linux cli (which is not a problem per se but not done in 5m)

 

[IamSpartacus]

Ok, that should be plenty

It sounds as if you're mixing different requirements (iops/latency relevant databases), throughput oriented (streaming clients) so you might need to cater for that (dataset level).

How have you tested (locally vs remote, which tool and parameters, throughput or latency) and why do you think it won't scale up with more turning wheels?

You said you ran 6 P4510's (which are not the fastest), so if you have 3.5GB/s write over network thats not bad actually (6 drives, 3x2 drives mirror = 3 write devices = 1.1GB/s per drive) ...

The problem with RoCe is that its not enough if the cards speak it, but the whole chain needs to comply, client side to server software (smb/nfs/iscsi) needs to be rdma enabled... I have not found anything that combines that with a nice gui :so that means linux cli (which is not a problem per se but not done in 5m)

To be clear I have not done benchmark testing over the network yet as I have not set this configuration up yet. My benchmark testing has been local only and that 3.5GB/s is write speed. Reads I can achieve 7.6GB/s.

But yes, I do have mixed usage of data that will need to be accessed over this RoCE setup. Currently I just use different pools for those different types of data. But I need to account for those different datasets obviously when adding this networking layer on top of it. Which is why I'm opening the discussion about it since I don't have any experience in the RDMA/RoCE area.

 

[IamSpartacus]

If I want to run a mix of ZFS pools and Mergerfs pools (for spinners) on the same storage server, is there something I should be looking at over just a plain vanilla Debian Linux server?

 

[IamSpartacus]

Record size does indeed have a huge impact. Bumping up my cache zpool from default (128K) to 1M boosts my sequential speeds to 5.6 GB/s writes and 13.7 GB/s reads.

 

[T_Minus]

Yeah, as expected.

Now just use a different pool with slog device for an optimal data store for VM workload, and then you're good to go.

 

[IamSpartacus]

Yeah, as expected.

Now just use a different pool with slog device for an optimal data store for VM workload, and then you're good to go.

I have the 4 x 960GB Optane 905p's I could use as SLOG/special devices but seems like the capacity would probably be wasted.

These are the NVMe's in my system ATM.

Node                  Generic               Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            INTEL SSDPE2KX040T8                      0x1          4.00  TB /   4.00  TB    512   B +  0 B   VDV10131
/dev/nvme1n1          /dev/ng1n1            UCSC-NVMEHW-H3200                        0x1          3.20  TB /   3.20  TB    512   B +  0 B   KNCCD101
/dev/nvme2n1          /dev/ng2n1            UCSC-NVMEHW-H3200                        0x1          3.20  TB /   3.20  TB    512   B +  0 B   KNCCD101
/dev/nvme3n1          /dev/ng3n1            INTEL SSDPE21D960GA                      0x1        960.20  GB / 960.20  GB    512   B +  0 B   E2010480
/dev/nvme4n1          /dev/ng4n1            INTEL SSDPE2KX040T8                      0x1          4.00  TB /   4.00  TB    512   B +  0 B   VDV10152
/dev/nvme5n1          /dev/ng5n1            INTEL SSDPE2KX040T8                      0x1          4.00  TB /   4.00  TB    512   B +  0 B   VDV10152
/dev/nvme6n1          /dev/ng6n1            INTEL SSDPE2KX040T8                      0x1          4.00  TB /   4.00  TB    512   B +  0 B   VDV10152
/dev/nvme7n1          /dev/ng7n1            INTEL SSDPE21D960GA                      0x1        960.20  GB / 960.20  GB    512   B +  0 B   E2010480
/dev/nvme8n1          /dev/ng8n1            INTEL SSDPE21D960GA                      0x1        960.20  GB / 960.20  GB    512   B +  0 B   E2010480
/dev/nvme9n1          /dev/ng9n1            INTEL SSDPE21D960GA                      0x1        960.20  GB / 960.20  GB    512   B +  0 B   E2010480

 

[Rand__]

Maybe "wasted" space for slog, but metadata devices can use a lot of space, totally depends on the number of files/objects you have on your regular pool

Edit - Have you played around with NVMEof from Truenas to Proxmox? Not really RDMA but maybe an improvement

 

[IamSpartacus]

I've been testing NFS-oRDMA between the SAN I built and multiple Proxmox hosts that I want to access that storage. I'm hitting a wall with write speeds as they are atrocious and I'm not having any luck troubleshooting why. I've tested both connecting the SAN (bonded and unbonded 100Gb NICs) and PVE hosts (single 100Gb NIC) through a Mikrotik switch and direct connecting the PVE hosts to the SAN. There is no difference in the write speeds I'm seeing.

Read speeds are able to saturate the 100Gb link in testing (roughly 11GB/s from my zfs pool of nvme drives). However when I do write tests the performance is abysmal. Usually sub 200MB/s and often sub 50MB/s. Everything I'm reading is that I'm up against a NFS COMMIT limitation but I've also read that people have gotten terrific results with NFS-oRDMA.

What am I missing?

 

[kapone]

Usually sub 200MB/s and often sub 50MB/s

With your hardware...that is borderline..F*ck this, I'm out. I can get over 1GB/s sync writes on NFSoRDMA with just 18x spinning HDDs (HGST Helium SAS), OOTB, with no tuning. This is not even over jumbo frames, standard 1500 MTU. This is with no SLOG, special VDEV etc. Standard x86 host with 128GB RAM.

In my case the "SAN" is Truenas while the clients are Proxmox and RHEL (Almalinux). Something's off on your SAN end, given that your read speeds are perfectly fine. Bonds and RDMA coexist just fine, bonds are not gonna be an issue.

So, that leaves us with actual hardware in the SAN itself. Since you can benchmark them locally just fine, that leaves the networking as the likely culprit. Remind me again, what OS are you using on the SAN?

 

[IamSpartacus]

View attachment 48456

With your hardware...that is borderline..F*ck this, I'm out. I can get over 1GB/s sync writes on NFSoRDMA with just 18x spinning HDDs (HGST Helium SAS), OOTB, with no tuning. This is not even over jumbo frames, standard 1500 MTU. This is with no SLOG, special VDEV etc. Standard x86 host with 128GB RAM.

In my case the "SAN" is Truenas while the clients are Proxmox and RHEL (Almalinux). Something's off on your SAN end, given that your read speeds are perfectly fine. Bonds and RDMA coexist just fine, bonds are not gonna be an issue.

So, that leaves us with actual hardware in the SAN itself. Since you can benchmark them locally just fine, that leaves the networking as the likely culprit. Remind me again, what OS are you using on the SAN?

Baremetal Debian 13. I tried with the standard 6.12.x kernel it comes with and upgrading to 6.19 via backports with no discernable difference.

Also I just tested standard TCP NFS mounts and get 1.5-2.0GB/s sync writes. The problem there is my read speeds drop from 11GB/s to about 3.5-4GB/s. Nothing to sneeze at but it hurts haha.

 

[kapone]

The problem there is my read speeds drop from 11GB/s to about 3.5-4GB/s

Of course the read speeds will take a hit. At 100gbps or more, RDMA is pretty much essential, otherwise you're at the mercy of your CPU's single threaded performance. But...that does gives us some hints...

1. Using RDMA, your read speeds are fine, but writes are much worse.
2. Using TCP, your read speeds drop but writes are not that bad.

So...that may point to the problem being on the other end. What's the OS on the client? Proxmox?

 

[IamSpartacus]

Of course the read speeds will take a hit. At 100gbps or more, RDMA is pretty much essential, otherwise you're at the mercy of your CPU's single threaded performance. But...that does gives us some hints...

1. Using RDMA, your read speeds are fine, but writes are much worse.
2. Using TCP, your read speeds drop but writes are not that bad.

So...that may point to the problem being on the other end. What's the OS on the client? Proxmox?

Proxmox 9.1.7 running 6.17.13-2-pve on both hosts that have 100Gb connectivity.

 

[kapone]

Proxmox 9.1.7 running 6.17.13-2-pve on both hosts that have 100Gb connectivity.

How did you mount the NFS share on it?

 

[IamSpartacus]

How did you mount the NFS share on it?

Right now with TCP it's like this

10.99.9.1:/cache /mnt/san01/cache nfs vers=4.1,proto=tcp,hard,rsize=1048576,wsize=1048576 0 0

When I was attempting RDMA it was like this

10.99.9.1:/cache /mnt/san01/cache nfs vers=4.1,proto=rdma,port=20049,hard,intr,rsize=1048576,wsize=1048576 0 0