Cheapest, quietest, lowest power 40Gbe switch? (and preferably less than 55cm deep)

[solon]

I've recently set up a 40Gb SAN at home as Covid keep oneself occupied therapy, but I've come to the conclusion that I als want to run parallel 40GBE. I have Mellanox MCX354A-QCBT cards which I'll flash to FCBT, ideally using one port for the SAN and the other for 40Gbe. I have a mellanox IS5022 modded with a 200mm noctua fan (those bastards use a non-standard 3 pin fan connector by the way, which fried a cheap test fan) and a little 3D printed air guide, which works fine for the SAN and pulls about 20W.

I stil have plenty of spare QDR cables, so as far as I can tell all I really need is a 40Gbe switch.

So, I'm wondering what people's thoughts are on the best way to go with this. I've been looking at the mellanox SX6012 or SX6015 switches which seem like they'll work and can probably be made silent with a small fortune in Noctua fans.

I'm not all that enthusiastic about how much power (spec states a ludicrous 126W, but that must be less with only four nodes I think) they draw or how easy it's likely to be to make them silent. I'll probably be able to delete one or two 1gbit switches if I do all ethernet over 40gbe for the workstations, but those only pull about 7W each. The PC's are on an external watercooling loop so I could conceivably watercool the switch, though that's likely to beat least as much of hassle as noctua fans though it's considerably more likely I will be able to preserve the 1U profile that way.

Hopefully someone has a suggestion on the best of all worlds (compromise) approach to this?

 

[i386]

For 1-4 40GBE I would look at the icx switches in the megethread,
for 8+ 40GBE ports I would look at arista switches. The os of arista switches has a build in command to set the fans to 30% speed which makes them quieter than supermicro 3/4u servers.

SX6015

That's an unmanaged infinband switch, it can't be used for ethernet networks.

 

[Rand__]

If you are fine with 12 ports the 6012's are nice, they run ~ 30-50W depending on used modules (DACs vs active), # of ports etc. Probably need fan modding in your case though, but I dont think any affordable 40G switch will run silent without, thats just not what they are made for.
If you need 30+ ports there is a short version of the 6036 available (o/c more power draw, ~50W+ iirc)

 

[MichalPL]

Cisco 3064PQ you can mod for new fans and will be totally silent, but It has only 4 QSFP+ ports and 48 SFP+. if I remember correctly 65W

 

[solon]

Thanks everyone, I'll inventory all those and see what's avaiable at a fair price on ebay. First quick glances look like they're not quite at the prices I'd like. I thought there was a thread on here somewhere where someone had modded a qsfp+ infiniband switch to 40gbe. I'll have so start sifting through everything again.

In total I only really need 4 ports, server and 3 pc's. Don't seen any lcx stuff on ebay, and the arista and cisco stuff is at a price level that I'd really need to think about.

I don't mind modding the fans to make it silent, but as noctua fans aren't cheap either, it's nice if it can be avoided.

Anyway thanks for the suggestions, I'll post back here if I can find anything affordable in Europe. All this working from home really has put a different load on my home network.

 

[Rand__]

You can import a EMC based 6012/18 from the US, will be ~€200.
Sometimes you can get a 6036 for the same price in EU, depends on how urgent it is

 

[solon]

No urgency to be honest. This is almost completely surplus to requirements project to keep me occupied. It was initially caused by annoyance that it took 10 minutes to copy a game install from my PC to my GF's. That led to looking at 10gbe RJ45, which led to noticed 40gb prices, which led to infiniband.... which led to a side goal of an rdma drive on SAN for game installs. It just seems elegant to network at 40gbe now that I mostly have everything anyway.

I'll keep my eye out. To be honest a 6012 looks best for my needs, and seems most likely to be available. Not terribly enthusiastic about the prices I'm seeing at the moment, cheapest one was around 700,- euros. I may still do it, but I'll shop around unless I find myself particularly annoyed by something that would be alleviated by having 40gbe.

 

[xeonguy]

You can import a EMC based 6012/18 from the US, will be ~€200.
Sometimes you can get a 6036 for the same price in EU, depends on how urgent it is

Thought OP wanted 40Gbe, but it seems the 6012 is Infiniband? I'm also trying to setup a silent 40Gbe network, so in the same boat.

 

[Rand__]

The Mellanox 6012/18/36 can run IB, Ethernet or both depending on licenses.
Licensing is discussed in other theeads in this forum

 

[Stephan]

A 12-port SX6012 idle and empty (nothing connected) and patched in rc.local to run fans idle at >4500 rpm will draw around 28 watts. Top cover can be replaced by a PETG plate and one large fan, which will make the switch basically silent. But you will need some sort of cheat on fan TACHO signal to turn 1000 rpm into 5000 rpm, because else switch will ramp fans to high speed when it senses fans are below 4000 rpm. With right FDR cables 56 GBps is possible. ConnectX3 release notes by Mellanox have compatibility lists of these cables.

 

[solon]

I actually have never really gotten around to sorting out the licensing things, the 6012 is still sitting on my desk. Turns out that infiniband is doing everything I need at the moment, at least until >1gbit internet becomes available. I could, for downloading from steam up the size of my cable connection so that that and the DSL linke they just exceed 1gbit when concurrently downloading, but it's such a small extra convenience that I haven't bothered with it.

Something to note though, I took particular care to find a 6012 that wasn't EMC branded... it was old, apparently from the brazilian national telecoms company in the original Mellanox box but it still turned out to be running EMC firmware when I connected to it.

 

[Stephan]

EMC "branding" would be black front bezel. Only if its Mellanox blue on the front bezel, it is an original Mellanox. EMC is preferrable because you get the highest tier silicon with all features possible (Ethernet, VPI, Infiniband, 56 Gbps, dual PSU, probably forgot something).