I have two boxes (running Mint 19.2 on both) using 10Gbps Mellanox X3 cards connected via a DAC cable and iPerf reports 9.8Gbps both ways. No networking switch between these boxes.... just the DAC cable.
Server A is a 24 bay, SuperMicro box. I think I have 12, 4TB drives in RAID 6 via MDADM. The backplane in the SuperMicro is SAS2 and the SFF cable goes into a 9207-8i HBA. When I copy a 10 gig file from the SSD to the RAID 6 array or from the SSD to the RAID 6 array, I'm getting a hair over 500MB/s.
Server B is a 12 bay Rosewill box. It has a bunch of drives in RAID 5 (MDADM, 9207-8i HBA) and a SSD for booting. Same deal. When I copy to/from the RAID 5 array in this box to/from the SSD, seeing 500+MB/s.
At this point, the sub-system in both servers is capable of maxing out the read/write performance of my SSD drives. But when copying over the 10Gbps link, I'm only seeing about 300+MB/s. I did some digging around and found references to adjusting the MTU. I did that for the 10Gb NICs (from 1500 to 9000) in both systems and the first file copy test got me speeds around 350MB/s and then every test copy after that........ roughly 70MB/s! I restarted both boxes and the MTU is back to 1500 and speeds are consistent at 300+MB/s.
Both cards have the latest firmware and are using v4.00 of the driver (Mint default). I can't apply the absolute latest driver because Mellanox's installer is hard-coded to look for only "officially" supported operating systems and Mint isn't supported.
In case it matters, both systems are using their 1Gbps NICs to route out through the switch and get on the internet and talk to the LAN so in case I need to make some universal kernel boot change, I need to make sure it doesn't negatively impact the 1Gbps NIC.
Server A is a 24 bay, SuperMicro box. I think I have 12, 4TB drives in RAID 6 via MDADM. The backplane in the SuperMicro is SAS2 and the SFF cable goes into a 9207-8i HBA. When I copy a 10 gig file from the SSD to the RAID 6 array or from the SSD to the RAID 6 array, I'm getting a hair over 500MB/s.
Server B is a 12 bay Rosewill box. It has a bunch of drives in RAID 5 (MDADM, 9207-8i HBA) and a SSD for booting. Same deal. When I copy to/from the RAID 5 array in this box to/from the SSD, seeing 500+MB/s.
At this point, the sub-system in both servers is capable of maxing out the read/write performance of my SSD drives. But when copying over the 10Gbps link, I'm only seeing about 300+MB/s. I did some digging around and found references to adjusting the MTU. I did that for the 10Gb NICs (from 1500 to 9000) in both systems and the first file copy test got me speeds around 350MB/s and then every test copy after that........ roughly 70MB/s! I restarted both boxes and the MTU is back to 1500 and speeds are consistent at 300+MB/s.
Both cards have the latest firmware and are using v4.00 of the driver (Mint default). I can't apply the absolute latest driver because Mellanox's installer is hard-coded to look for only "officially" supported operating systems and Mint isn't supported.
In case it matters, both systems are using their 1Gbps NICs to route out through the switch and get on the internet and talk to the LAN so in case I need to make some universal kernel boot change, I need to make sure it doesn't negatively impact the 1Gbps NIC.