Cheap 40GbE at home - too good to be true?

Why my 40G upgrade was less expensive than 10G upgrade less than 3 years ago?

  • It was a fluke (got lucky with my 40G choices and unlucky with 10G)

    Votes: 0 0.0%
  • It was not a fluke (QSFP+ gear is indeed less expensive than SFP+)

    Votes: 15 65.2%
  • The prices have changed in the meantime - 10G is much cheaper today

    Votes: 6 26.1%
  • Other (please explain in the thread)

    Votes: 2 8.7%

  • Total voters
    23

BeTeP

Well-Known Member
Mar 23, 2019
580
378
63
would you be willing to share the conversion guide?
As I just explained in the original conversion thread - I did not keep the copy of the original guide because I did not have any intention to actually follow through. I just used the information to build my own custom image for easy flashing.
 

ehorn

Active Member
Jun 21, 2012
342
52
28
No worries, op. I have no personal need for 40gbe in my home or lab. but I suspect others would love to set one up in their labs. And it’s a community of sharing which makes it a great site. Hence my reference to the switch mod post and poster. And also the main contributor to your low cost implementation.

As far as fluke or not, here is my take:

Earlier on, 40gbe And up had its start in hpc... mellanox was and is a major player in hpc markets and had the lead over others by far. Ib was,is cutting edge for latency and bandwidth. Just really never made it mainstream and remains the fabric of esoteric, custom systems.

the hpc guys are also never satisfied and always stretching performance, so we get loads of ‘old tech’ hitting the market with each cycle. All your gear is mellanox. No surprises there. Try to find intel gear, or any other manufacturer who has so much second hand gear... not really out there.. there is a dearth of gear to choose.

Now more recently mellanox has pivoting to Ethernet as they want to compete in that segment and Now we see more competition in the higher bandwidth tiers growing.

Best
 

herby

Active Member
Aug 18, 2013
187
53
28
TLDR: You're correct, 40GbE is just cheaper.

40GbE is a dead end, technologically, and was done with "trickery" by bonding 4x10G channels into a single link - it wasn't really a separate speed type, like 1G vs. 10G is. But, it was faster and didn't have the drawbacks of port-channels, so it got used.

Then 25GbE, which actually IS a non-trickery, separate speed type, was introduced, and was able to use the same trickery to get 50GbE and 100GbE. Further, the optics were intentionally forwards compatible with planned 100GbE tech that was still in development at the time, and THAT could do the same bonding for 400GbE.

So basically 40GbE was a dead end street, tech-wise. Once everyone saw a viable, faster forward path (that was actually not significantly more expensive, either) they dumped 40GbE hard and the market was flooded. Whereas 10GbE trickled in and continues to do so, as it has its uses for access-layer aggregation instead of datacenter-only, which is where 25/50/100/400GbE really lives.
I think this is pretty much it. I feel like it never got great traction except with people that really needed more than 10 GbE during a relatively short window. After that time passed people either continued gravitating to 10 GbE if they needed just a little more than gig, or are chasing the new higher end at 100+. 40 GbE became the middle child while both faster and lower still had a place.
 

atemik

New Member
Sep 11, 2019
3
0
1
Totally agree, 40/56Gbps can be had very cheap now (and silent!)

My experience with this is IS5022 (QDR, ~$80) or MSX6005 (FDR, ~$100), a bunch of ConnectX-3's (around $30 for HP branded) and a cables (10m fiber can be found for less than $15).
The best feature about IS5022 is on the attached photo. It's totally silent and cool. Ideal for home lab. BEWARE: pins for fans are non-standard here, you'll need to make an adapter or the fan will make some smoke!
MSX6005 has ICs on the bottom of the board so I'm unsure if this kind of cooling will be enough (and mine are in a datacenter so the noise is not an issue).
 

Attachments

Crond

Member
Mar 25, 2019
51
11
8
Mmmm did you guys update firmware on MSX6005 to convert it to Ethernet switch ?

IMHO 40GBE is not cheaper than 10GBE. the IB 40/56GB solutions on the other hand are much cheaper because the support for IB has been dropped by major players like vmware, so you can score crazy IB switch with terrabytes of switching capacity and 56Gb wire speed for under $100
A lot of OEM hardware is based on IB and thanks to their design for CX3 series a lot of IB solutions can be cross flashed to support 40GBE.

While it works for network hackers / home LABs experiments, it's not a solution for small biz. or individuals who are looking for something that will work out of the box.
finally there are plenty of options for silient or near silent 10GbE deployment while 40GBE is either expensive or for older model are quite noisy (assuming "stick back fan to open case" is not an option)

As a summary:
40/56GB IB solution or cross-flashes / open case design has much smaller market (due to DIY nature and noise/power consumption) compared to 10GBE while has a lot of hardware available due to major vendors dropping support for IB based fabric in favor RoCE

Proper 40GBE solution is still more expensive than 10GBE
 

atemik

New Member
Sep 11, 2019
3
0
1
can the IS5022 be converted to 40gbe?
AFAIK no, you'll need at least MSX6012 with "modifications" (that's for Mellanox switches).
If you need network only for file sharing/remote drives - then IBoIP can be ok.
If you are going into virtualisation then IPoIB won't to make it. You can't have many "IPoIB macs" on one card. I don't know if SR/IOV can solve this problem.