Jumbo frames - some testing

[MountainBofh]

I recently got a pair of Mellanox ConnectX-4 Lx cards and after flashing them to the newest firmware decided to do some quick and dirty performance testing. Both test boxes are running a fully updated but otherwise stock Debian 12.5 configuration. I linked the machines directly together using a SFP28 AOC cable.

MTU was left stock at 1500.

Testbox1 - 10.0.0.1
I7-9700K
Gigabyte Z370 Aorus motherboard
16GB (2x8GB) DDR4 3200 ram

Testbox2 - 10.0.0.2
I5-12500
Gigabyte B660M DS3H motherboard
16GB (2x8GB) DDR4 3200 ram

I ran iperf3 and got the following result.




Pleased with that, I setup a NFS server on one machine, had it export a ram drive, and mounted it on my other machine. To eliminate any possible disk i/o issues, I also created a ram drive on the client machine. Using about 8 gigabytes of ISO files, I ran a copy from testbox1 to testbox2, and then in the other direction. Forgot to take a screen shot here, but copy performance was above 21 GB/sec.

Another poster mentioned that he was seeing lessor performance on his X520's unless he used jumbo frames. I should be getting 2 X520's in tomorrow, so I'll repeat the test using those cards and see if jumbo frames help or not.

 

[MountainBofh]

Here's the result of using mtu 900 on the ConnectX-4 Lx cards. I did use Parted Magic with kernel 6.5 for this test, but I figured that wouldn't really impact the results. Conclusion, about a 5% boost in speed.




I'll test the Intel X520's soon.

 

[MountainBofh]

And here's with the X520's. Using Parted Magic with kernel 6.5 again. Machines are direct connected, no switch involved.

MTU 1500



MTU 9000




Once again, about a 5% gain.

 

[nabsltd]

These pretty much match my 10Gbps test results. Jumbo frames can help a bit, but they don't really shine until latency gets higher.

I got about 30% faster speed on a San Diego to DC link over Internet2 when I worked with all the ISPs and the internal network engineers at both end sites to be 9000 MTU on every link.