Hey,
I purchased a few mellanox x-2 ethernet sfp+ adapters and hooked them up to my 10gb switch. When running iperf from ESXi 6.7 using iperf built into the esxi shell, i am only getting about 1.2 gb/s (gigabit) bandwidth. I get the same 1.2 gb/s speed from iperf inside a VM using vmxnet3 adapter, so all the bandwidth is able to be used in the VM guests, albeit slower than expected.
Is there some configuration i am missing. I went through the mellanox tuning book and set all bios options to recommended settings. I have confirmed that all cards are running 2.9.1200 firmware on them.
So far i have tried iperf from:
Workstation << >> ESXi Host (~1.2 gb/s)
ESXi Host << >> baremetal Freenas Box (~1.2 gb/s)
and Workstation << >> Freenas (~6.0 gb/s)
What leads me to believe there is an issue with ESXi is the fact all other speeds are greater when not speed testing to VMWare. Although, i am curious why i can only get 6.0 gb/s to Freenas and not the full 10 gb/s.
I see the same speeds when linking each test direct and bypassing the switch, so i do not believe it to be the switch (MikroTik CRS309-1G-8S+IN)
The ESXi and freenas boxes each have an E5-16xx V4 processor and my workstation has a Ryzen 1700x processor, so there should be plenty of processor/pcie bandwidth.
Any help is appreciated
-Cheers
I purchased a few mellanox x-2 ethernet sfp+ adapters and hooked them up to my 10gb switch. When running iperf from ESXi 6.7 using iperf built into the esxi shell, i am only getting about 1.2 gb/s (gigabit) bandwidth. I get the same 1.2 gb/s speed from iperf inside a VM using vmxnet3 adapter, so all the bandwidth is able to be used in the VM guests, albeit slower than expected.
Is there some configuration i am missing. I went through the mellanox tuning book and set all bios options to recommended settings. I have confirmed that all cards are running 2.9.1200 firmware on them.
So far i have tried iperf from:
Workstation << >> ESXi Host (~1.2 gb/s)
ESXi Host << >> baremetal Freenas Box (~1.2 gb/s)
and Workstation << >> Freenas (~6.0 gb/s)
What leads me to believe there is an issue with ESXi is the fact all other speeds are greater when not speed testing to VMWare. Although, i am curious why i can only get 6.0 gb/s to Freenas and not the full 10 gb/s.
I see the same speeds when linking each test direct and bypassing the switch, so i do not believe it to be the switch (MikroTik CRS309-1G-8S+IN)
The ESXi and freenas boxes each have an E5-16xx V4 processor and my workstation has a Ryzen 1700x processor, so there should be plenty of processor/pcie bandwidth.
Any help is appreciated
-Cheers