HPE SN2700M 100gb 16 port switch for $4250

[Frank173]

I saw on ebay an HPE SN2700M 16port 100gb switch which is the OEM of Mellanox's SN2700 which goes new for around 30k or so. I was wondering whether anyone uses this one and whether there is a catch? I look to connect several GPU compute nodes and found this switch to meet all my needs. But am wondering why this oem version sells for so much less given its identical to the much more expensive Mellanox SN2700.

HPE / Mellanox StoreFabric SN2700M 100GbE 16QSFP28 Switch Q6M26A MSN2700-CS2F | eBay

 

[i386]

I think HPE is cheaper because they sell more units than Mellanox.

With older switches only part numbers were different in mellanox-os, can't say if this is true for the newer switches.

 

[Frank173]

I agree this makes a difference but the pricing delta is quite large hence I wondered. Well, I bought this particular unit and will report back once I get it.

I think HPE is cheaper because they sell more units than Mellanox.

With older switches only part numbers were different in mellanox-os, can't say if this is true for the newer switches.

 

[Frank173]

I received the switch and could not be happier. 16x 100Gb ports for 4.2k is not bad for one of the top performing switches in its class. I had some trouble initially to get HPE to honor the warranty as this is a pre-owned unit but eventually they sent me the most up to date Onyx OS to upgrade my switche's OS and firmware. Everything works as expected...

 

[T_Minus]

I received the switch and could not be happier. 16x 100Gb ports for 4.2k is not bad for one of the top performing switches in its class. I had some trouble initially to get HPE to honor the warranty as this is a pre-owned unit but eventually they sent me the most up to date Onyx OS to upgrade my switche's OS and firmware. Everything works as expected...

Any chance you plan to share how it's utilized and what beasts are connected to it

 

[Rand__]

Nice, been oogling that one for a while (but way too much for me )

 

[Frank173]

I use it to connect several GPU compute servers with each other and a file server that runs multiple nvme drives in raid0 for high speed data transfer.

PS still need to get a rack and have not yet fully configured the switch...

Any chance you plan to share how it's utilized and what beasts are connected to it

 

[Rand__]

Nice.
How do you Raid0 the nvme drives? Any further details you can share re the setup?

 

[Frank173]

I use x16 PCIe cards that house 4 Nvme drives each and I bifurcate several of my x16 PCIe slots on the board to make all the drives work. I then use Storage Spaces Direct which, after some performance optimizations, provides close to 92 Gbit data throughput over the network, nearly exhausting the 100Gbit bandwidth of the 100Gb Nics (Mellanox ConnectX-4). The actual throughput of the Nvme drives in Raid0 is even higher than 92Gbit but I guess this is the bandwidth at which the network throughput becomes the bottleneck.

Hope this helps, let me know if you need even more details...the performance optimizations were very tricky and definitely took the longest among the entire setup to fine tune. I gave up on drive/folder mapping and sharing of folders/drives because Windows imposes OS bottlenecks that I have never been able to figure out over the years. I would consider Linux as OS for the file server but I run several backup applications that constantly backup changes in the Raid0 drive to mirrored backup disks and also sync changes to the cloud and to my knowledge certain apps that I use for cloud sync are not available in Linux.

Nice.
How do you Raid0 the nvme drives? Any further details you can share re the setup?

 

[Rand__]

Now that definitely is interesting- i am sure many here would love to hear how you optimized and what exactly the setup is (kind of drives, #, type of traffic, cpu used). So anything you can share...