10GBe vs. 25GBe - Cost Analysis and Part Numbers

Edd189

New Member
Oct 18, 2019
20
2
3
I am upgrading my backbone in my home and am trying to decide between 10GBe and 25GBe. Right now I only need two high speed connections, but will plan for more. Along the way, I have researched part numbers and prices and pulled everything into a spreadsheet. I figured I would share here as someone else may find this very helpful.

To the community, does anyone have a recommendation for a better 25GBe switch than the one I have listed? Also, if you see anything incorrect or incompatible, please let me know!
 

Attachments

altmind

Active Member
Sep 23, 2018
138
46
28
Will you compare 25g prices with 40gig?

I find multigig tech(new hardware) highly expensive, sometimes more expensive than next closest speed. And for case of 25g sfp, even the sfp connector is different, so theres little to reuse.
 

Edd189

New Member
Oct 18, 2019
20
2
3
Will you compare 25g prices with 40gig?
I have scoured the internet for information on 40Gbe and the conclusion I reached was that Ethernet wasn’t possible, only Infiniband. I wasn’t interested in that option. But please correct me if I’m wrong!
 

Edd189

New Member
Oct 18, 2019
20
2
3
The IS5022 switch looks perfect when paired with a ConnectX-3 Pro, but no Ethernet. I suppose it would be OK for direct connections though.
 

altmind

Active Member
Sep 23, 2018
138
46
28
What you mean 40g ethernet is impossible? There are 40gbe cards, there are 40gb sfp, there are 40gbe. They usually use MTP-12, so its basically 10x4 bonding, no gains in latency, only in bandwidth.
 

Edd189

New Member
Oct 18, 2019
20
2
3
Impossible in my price range, I think. Do you know a 40Gbe switch that works with Ethernet?
 

altmind

Active Member
Sep 23, 2018
138
46
28
Cisco N3K-C3132Q-40GE $650 used, or nexus 3064x if you are good with 4 qsfp and 48 sfp+.

used nexus 3k is in general very price competitive.
 

Rand__

Well-Known Member
Mar 6, 2014
4,418
861
113
And of course all the converted Mellanox/EMC ones (search Sx6036, sx6018, sx6012) here on StH;)
 

RageBone

Active Member
Jul 11, 2017
290
77
28
those, though i haven't found any the last few times i looked.

But since op currently only needs two connections, pear to pear directly t is still an option and would cost you the 3 cards, one dualport, and the cables.

On the other hand with 10GbE, CX2 / CX3 sfp+ cards seem to be ok, though i went the cx2/3 QSFP 40GbE rout, without a switch since i will only have less then 6 connections ever* and i can do that with spare adapters for less power and what my 18 port switch cost me.

SFP+ has the advantage in switch availability and cost i think. .
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,815
1,579
113
29
fohdeesha.com
Impossible in my price range, I think. Do you know a 40Gbe switch that works with Ethernet?

a ton of us are running 40gbE ethernet here on this forum, the ICX6610 for example is $200 and has 2x 40gbE ports (and a bunch of 10gb and 1gb): https://forums.servethehome.com/ind...s-cheap-powerful-10gbe-40gbe-switching.21107/

there's other switches like the arista 7050S-64 with 4x 40gbE (and a bunch of 10gb) ports for around 500, or if you want just a ton of 40gbE density there's stuff like the arista 7050qx-32s for around $900 with 32 40gbE ports. I would stay far away from the old cisco nexus stuff, it's generally twice the price of the aristas and 3x as loud

mellanox connectx 40gbE NICs are $40
 

funkywizard

mmm.... bandwidth.
Jan 15, 2017
615
271
63
USA
ioflood.com
I have scoured the internet for information on 40Gbe and the conclusion I reached was that Ethernet wasn’t possible, only Infiniband. I wasn’t interested in that option. But please correct me if I’m wrong!
I'm happy to tell you that you're misinformed. 40gbe is widely available and relatively inexpensive.

ConnectX-3 cards can do either IB or ethernet mode

Arista makes 7050QX32 switches (32x40g) that are routinely $500 or less on ebay

There may also be cheaper / better options.

As well, the 40g is typically backwards compatible with 10g. 24 of the 32 ports on the 7050QX32 for example, can be set to 4x10g mode
 

Edd189

New Member
Oct 18, 2019
20
2
3
Here is what I have for 40GBe fiber parts and costs.

I also added an option to buy the NIC cards at 40GBe so I could direct attach my server/workstation to play with, but run the full backbone at 10GBe. That's where I'm leaning now.
 

Attachments