Search results

Ok I finally have found some time to pick the project back up. Got my motherboard issues sorted and for further testing I have direct connected two servers so that there are no switch variables in play. I've gone back down to an MTU of 1500 on both NICs but that has had no impact. I'm testing...

Haha. Yup that gets me into the BIOS. Unfortunately I've yet to figure out how to get the server boot again with the PCIe cards I actually need to run.

Yea it's odd. I had 6 of the 7 pcie slots populated for a month with no issue. As soon as I did a boot with the 7th slot populated it freaked out. And now even with just 3 slots populated it won't boot even after many CMOS resets. I'll need to find some time to play with all the...

I would love to confirm all of that if I could actually get into the BIOS lol. I'm pretty confident that Above 4G Decoding is enabled and that my bifurcation settings are correct because I had to set that for all my quad oculink cards.

Every time I reset the CMOS unplug the power, press the power button a few times to discharge, then leave it off for 5 minutes with the battery out. Think it needs to sit longer?

And went to do most testing today but before doing so I went to through an Arc a40 GPU into the machine so I could run some services on this machine with local storage will I continue to test out this issue. Well, now I'm left with this screen at boot an no matter how many PCI cards I take out...

Currently a switch. However when I saw these results I tried direct connecting and the results were identical.

I'm glad I'm not the only one who's frustrated by this :D.

Right now it's like this. But I've done the same tests with async and it made no difference. /cache 10.99.9.0/28(rw,sync,no_subtree_check,no_root_squash)

Here are the results. 10.99.9.1:/cache /mnt/san01/cache nfs vers=4.1,proto=tcp,hard,rsize=1048576,wsize=1048576 0 0 time sh -c 'dd if=/dev/zero of=/mnt/san01/cache/test.bin bs=1M count=4096 conv=fdatasync status=progress' 4096+0 records in 4096+0 records out...

Right now with TCP it's like this 10.99.9.1:/cache /mnt/san01/cache nfs vers=4.1,proto=tcp,hard,rsize=1048576,wsize=1048576 0 0 When I was attempting RDMA it was like this 10.99.9.1:/cache /mnt/san01/cache nfs vers=4.1,proto=rdma,port=20049,hard,intr,rsize=1048576,wsize=1048576 0 0

Proxmox 9.1.7 running 6.17.13-2-pve on both hosts that have 100Gb connectivity.

Baremetal Debian 13. I tried with the standard 6.12.x kernel it comes with and upgrading to 6.19 via backports with no discernable difference. Also I just tested standard TCP NFS mounts and get 1.5-2.0GB/s sync writes. The problem there is my read speeds drop from 11GB/s to about 3.5-4GB/s...

I've been testing NFS-oRDMA between the SAN I built and multiple Proxmox hosts that I want to access that storage. I'm hitting a wall with write speeds as they are atrocious and I'm not having any luck troubleshooting why. I've tested both connecting the SAN (bonded and unbonded 100Gb NICs)...

I have the 4 x 960GB Optane 905p's I could use as SLOG/special devices but seems like the capacity would probably be wasted. These are the NVMe's in my system ATM. Node Generic Model Namespace Usage Format...

Record size does indeed have a huge impact. Bumping up my cache zpool from default (128K) to 1M boosts my sequential speeds to 5.6 GB/s writes and 13.7 GB/s reads.

If I want to run a mix of ZFS pools and Mergerfs pools (for spinners) on the same storage server, is there something I should be looking at over just a plain vanilla Debian Linux server?

Anything specific I need to know about optics and/or fiber cables to pair with these cards? I have a bunch of Cisco QSFP-40/100G-SRBD in storage I could use if they are compatible. If so, I just need to figure out which patch cables to pair with them.

Nice. But now that I think about it, it would be more cost effective to just use two single port cards in my storage server. I have enough free slots.

These will suffice? IBM Mellanox CX556N 2-Port EDR 100GB IB ConnectX-5 Adapter (dual port) MELLANOX MCX455A-ECAT 100GB Single Port VPI QSFP28 CONNECTX-4 (single port)