ESXi only getting 10Gbit but link at 40Gbit...

Sniper_X

Member
Mar 11, 2021
67
9
8
My vSAN isn't seeing the 40Gb speed that the cards and the switch report.
ESXi, using ConnecctX-3 cards with in-box driver...

Thoughts?
1616788854941.png
 

Rand__

Well-Known Member
Mar 6, 2014
5,600
1,228
113
Have the same on my CX3 Pro's. I think its a vSAN limitation
1616838832982.png

There are some updates in ESXi 7 including RDMA for vSan support that might alleviate this, but I have not updated yet (from 6.7)
 

Sniper_X

Member
Mar 11, 2021
67
9
8
probably got the nic plugged into a gen 2 pcie slot

Negative.

All available slots in the T41S-2U are PCIe GEN 3.

(1) PCIe Gen3 x16 Low profile MD-2
(1) PCIe Gen3 x8 mezzanine slot
(1) PCIe Gen3 x8 OCP LAN mezzanine slot
 

Rand__

Well-Known Member
Mar 6, 2014
5,600
1,228
113
no idea, as mentioned i am not on 7 yet.
Originally i thought rdma (roce) needed switch support, but ESXi 7 seems to run it without... so not sure
 

i386

Well-Known Member
Mar 18, 2016
2,654
773
113
32
Germany
Originally i thought rdma (roce) needed switch support, but ESXi 7 seems to run it without... so not sure
Roce (v1 & 2) can work without any switch configuration, but in an environment where the switches buffer capacity is maxed out you want to take precautions that no frames (v1) or packets (v2) are dropped.
 
  • Like
Reactions: Rand__

Sniper_X

Member
Mar 11, 2021
67
9
8
Yea, I think that is a big problem right now.

1616878620367.png

Granted, I have just hooked this up and ran it to see what would happen, so....

I could really use some assistance in real-time if someone wanted to do so... (i386?) :)

I would eagerly compensate you for your time if you can help show me how to optimize and stabilize this.
 

Sniper_X

Member
Mar 11, 2021
67
9
8
I got this to happen via server2016 to a ramdrive over the network.
40Gbit.

Now to try for 56gbit!!!!

1617244836885.png

1617244850537.png
 

Sniper_X

Member
Mar 11, 2021
67
9
8
Why does it require that I increase the queue depth to get to the maximum speed?

I have copied movies and other things on 1Gbit networks and that single file maxed out at full wire speed (112 mbytes/second)

I can’t get 40Gbit on a file fee unless it’s running benchmark and the queue depth is super high >16
 

Rand__

Well-Known Member
Mar 6, 2014
5,600
1,228
113
It certainly would help to understand what you're testing here?
Initially you mentioned vSAN, now you're running a Ramdrive somewhere...

No explanation, no config details, nothing - makes it quite difficult to answer.
 

Sniper_X

Member
Mar 11, 2021
67
9
8
It certainly would help to understand what you're testing here?
Initially you mentioned vSAN, now you're running a Ramdrive somewhere...

No explanation, no config details, nothing - makes it quite difficult to answer.
Yes I was running vSAN, but i blew that away for now to run Windows servers on the nodes to cut through the confusion and experiment with configurations.

I will go back to vSAN soon, but, as I mentioned, for now I am back in Windows 2016.
 

Rand__

Well-Known Member
Mar 6, 2014
5,600
1,228
113
Cant help you if you dont provide details what you're testing where and how.