Lab network suggestions?

[kujinke]

Hi,

My lab setup consists 3 hp dl380p gen8 servers.
There is 1 used for storage with 16 ssds connected to the other 2 over direct 10gbe lan(intel x540-t2) sharing the storage over nfs to each of the esxi servers that are used for a 2 node VMware cluster.

I was thinking of getting rid of my Achilles heel(bottleneck) which is the 10 gig direct attached coppers.
I really need to move over to 40gbe network only this time I'm considering using a switch on top of everything so I could expand later.

Any suggestions regarding a cheap second hand 40gig switch and nics?
I saw a lot of mellanox gear on ebay but I'm not sure about its compatibility because the older 40gbe switches and I might say cheap ones only support ib.

 

[i386]

Mellanox connectx3 + Arista 7050 Switch
$60 per nic, about $700 for the switch depending on the qsfp/sfp configurations

 

[kujinke]

Mellanox connectx3 + Arista 7050 Switch
$60 per nic, about $700 for the switch depending on the qsfp/sfp configurations

Nic exact model should be CX354A-FCBT right ?
if so, havent seen those for 60$ or so, if you see any ill be more then happy to know about it
Thanks!

 

[i386]

Search for "649281-B21" on ebay, that are hp branded connectx-3 vpi (infiniband + ethernet) cards that can be flashed with stock mellanox firmware.

 

[kujinke]

Search for "649281-B21" on ebay, that are hp branded connectx-3 vpi (infiniband + ethernet) cards that can be flashed with stock mellanox firmware.

Found it, thanks!!
is there a special reason the flash them to stock firmware ?
after bit of googling ive seen that they should fit the gen 8 ..

 

[i386]

is there a special reason the flash them to stock firmware ?

The stock mellanox firmware gets more often updates that the hp version, and if you use the official driver package it will always check if newer firmware is available and offer you a checkbox to update the firmware

 

[Evan]

On the other hand since you have a HP server I would stick to the HP firmware.

 

[kujinke]

Recommend FS N5850-48S6Q 40G switch for you, one of my partner have got one, seems to be work great. Besides, you also can refer to other places like reddit to get more similar answers about your problem.

Yeah only downside with the fs switch is it costs 5k .its the total cost of my entire lab including the ssds.

 

[Connorise]

I'll vote for Mellanox in whatever situation I would be.

And a bit obvious fact but the 40GbE switch will cost a fortune.

 

[whitey]

MLX 40GbE will cost serious $$$, Arista or Juniper are good recommendations.