Advice for upgrading from IPoIB CX3 -> ETH/VPI CX4?

[Skuzzi]

Hi all

Currently I'm using 40gb IPoIB with ESXi 6.5 and a NFS NAS - I want to upgrade to something a bit more supported.

I have the Mellanox CX3 cards and a EMC SX6012 that I've remanufactured to a generic Mellanox OS from a post here (Thanks!)

I'm able to get a 56gb connection between an ESXi 7 host and an ubuntu server - both Dell R740s

I only get around 20gbit between them using IPerf3 from a VM, and when benchmarking over NFS. I haven't tried iperf between NAS and host yet.

Anyway, I know CX3 cards aren't supported officially, and I've read there are some weird timeout issues with CX3 on ESXi 7 so I'm looking at CX4 cards.

Seems the LX cards can only do 25gb and the more expensive MCX455A-ECAT version will do up to 100gb.

My question is, are they compatible with the SX6012? I assume I won't get the full 100gb but can I at least get 56? or 50?

Thanks

 

[Rand__]

Nope.
You will fall back to 10 with LX's and to 40 with the regular CX4's.

 

[Skuzzi]

So CX4s will run at 40gb. Will I get the full 40gb speed?

 

[i386]

That depends on your storage...

 

[Skuzzi]

Well I'm currently testing with a RAM disk, at the moment I get about 20Gbps on a 56gb link, if I can get 40gbps on a 40gbps link then I'll be happy.

 

[Rand__]

That depends on your storage...

And use case (blocksize, concurrent users, iodepth...)