Recommended FS for Shared storage to back NVME-oF/RoCE?

[IamSpartacus]

Well...

1. Did we find the culprit?
2. Is "Above 4G Decoding" enabled in the BIOS?
3. If there's an "MMIO" setting, set it to a higher value than the default, test.
4. Double check your Bifurcation/pcie lane allocation setup. Test.

Just thinking out loud here.

I would love to confirm all of that if I could actually get into the BIOS lol. I'm pretty confident that Above 4G Decoding is enabled and that my bifurcation settings are correct because I had to set that for all my quad oculink cards.

 

[kapone]

I would love to confirm all of that if I could actually get into the BIOS lol. I'm pretty confident that Above 4G Decoding is enabled and that my bifurcation settings are correct because I had to set that for all my quad oculink cards.

yeah, you'll likely need to do some surgery i.e. remove some devices physically to try to get into the BIOS.

 

[IamSpartacus]

yeah, you'll likely need to do some surgery i.e. remove some devices physically to try to get into the BIOS.

Yea it's odd. I had 6 of the 7 pcie slots populated for a month with no issue. As soon as I did a boot with the 7th slot populated it freaked out. And now even with just 3 slots populated it won't boot even after many CMOS resets. I'll need to find some time to play with all the combinations of slots and cards but regardless I need the system to function with at lease 6 pcie cards.

 

[kapone]

just 3 slots populated it won't boot even after many CMOS resets

How many should you remove?

 

[IamSpartacus]

How many should you remove?

View attachment 48468

Haha. Yup that gets me into the BIOS. Unfortunately I've yet to figure out how to get the server boot again with the PCIe cards I actually need to run.

 

[IamSpartacus]

Ok I finally have found some time to pick the project back up. Got my motherboard issues sorted and for further testing I have direct connected two servers so that there are no switch variables in play. I've gone back down to an MTU of 1500 on both NICs but that has had no impact. I'm testing on a RAID0 pool of NVMe where I'm getting 3.3GB/s in local writes but writes over RDMA from a second host direct connected are still atrocious (sub 30MB/s).

PVEHOST

# tracepath -n 10.99.9.1
1?: [LOCALHOST]                      pmtu 1500
1:  10.99.9.1                                             0.277ms reached
1:  10.99.9.1                                             0.166ms reached
     Resume: pmtu 1500 hops 1 back 1

SAN

# tracepath -n 10.99.9.3
1?: [LOCALHOST]                      pmtu 1500
1:  10.99.9.3                                             0.924ms reached
1:  10.99.9.3                                             0.201ms reached
     Resume: pmtu 1500 hops 1 back 1

 

[TrevorH]

I suggest you fire up tshark/tcpdump and capture the raw wire traffic between the hosts and see if it sheds any light on the problem.

 

[kapone]

Ok I finally have found some time to pick the project back up. Got my motherboard issues sorted and for further testing I have direct connected two servers so that there are no switch variables in play. I've gone back down to an MTU of 1500 on both NICs but that has had no impact. I'm testing on a RAID0 pool of NVMe where I'm getting 3.3GB/s in local writes but writes over RDMA from a second host direct connected are still atrocious (sub 30MB/s).

PVEHOST

# tracepath -n 10.99.9.1
1?: [LOCALHOST]                      pmtu 1500
1:  10.99.9.1                                             0.277ms reached
1:  10.99.9.1                                             0.166ms reached
     Resume: pmtu 1500 hops 1 back 1

SAN

# tracepath -n 10.99.9.3
1?: [LOCALHOST]                      pmtu 1500
1:  10.99.9.3                                             0.924ms reached
1:  10.99.9.3                                             0.201ms reached
     Resume: pmtu 1500 hops 1 back 1

Grrr....That is not good.

Has to be something more basic. NICs good? Cables good? Connectors properly inserted? No overheating on the other host? Prayed to your favorite god yet?

I mean, I'm sorta shooting in the dark at this point. This is very confounding.

 

[TrevorH]

Duplicate ip addresses of client or server on some weird forgotten-about device?