How to 10GBPS?

[breakaway]

As in title pretty much.

Normally, I'd buy a Netgear XS708E/XS716E "Smart Managed" switch, some Intel x540-T1 or T2, some CAT7 patch cables and be done with it.

But when googling furiously to see if this was indeed the best option, I came across this site and it appears some of you have manged to get 10G networking for pennies on the dollar.

Can someone explain to me what I need to do in order to get this? There's so many options. DAC, Fiber, CAT6A/CAT7?

I also found that Ubiquiti is now doing a 16 port 10gbps switch with 4 x 10GBASE-T Copper interfaces + 12 x SFP+ interfaces. If I Get this, what sort of SPF+ do I need in the switch, what sort of NICs and SFP+ do I need in my servers/computers and what type of cabling? I need to check if its readily available and inexpensive since I'm guessing terminating fiber isn't as straightforward as punching down some RJ45s?

There's also many variants of pci-e SFP+ NICs on ebay, but I'm not sure which one(s) to buy. For e.g, I would hate to buy 10 NICs at a good price then getting stung because they only accept a specific type of SFP+!

What is the most cost effective way to get say, 16 computers networked up at 10 g? I'm not particularly fussed about media (be it fibre or copper) but it'd be nice if I can have longer runs (20-30m) so I can get our out buildings connected. I understand twinax DACs have quite a short length and the cables themselves are quite builky.

I will need the whole 9 yards - i.e.
- NICs
- Switches
- Cables

Recommendations on what to get would be great. I love Ubiquiti gear so if I could get that 16 port switch mentioned above that'd be fantastic.

 

[T_Minus]

Cheapest is LB6M but it's far from NetGear simplicity I have netgear new unit for sale but I went from copper to fiber, both work awesome!

Intel NICs were DIRT cheap! Now they're $$ 2-3x as much as 2+ months ago.

Cables, you can find DACs on ebay for very good prices! I have gotten 30m fiber down to 1m copper all for insane discount off "retail".

 

[whitey]

my 2 cents...

NICS - Intel X520-da1/2's or Mellanox ConnectX-2's ($20-100 depending on whatcha get/what deal ya find/how many ports ya need per host - IE: 1 or 2)

SWITCHES - Ubi switch ya mentioned, $600 now, was $300 during beta, @T_Minus has already tapped on LB6M. Former will let ya have 4 ports of cat6a/7 cabling for longer runs else go OM4 fiber and proper optics (depening on what switch ya decide on)

CABLES - Copper passive DAC's for 2 meter runs or less, OM4 fiber patch cables for longer runs paired w/ optics again (typically SR)

It's a bit early to SOLIDLY tell ya 'ohh yeah get this or get that' until ya settle in on the switch for sure, after that we can assist w/ optics/DAC's/etc. to get her all cabled up n' happy.

 

[acquacow]

I've been using Intel x540-T2 cards off ebay and cat5 to connect all my home stuff. It's been very reliable the last few years. I don't have a switch yet.

 

[Jerry Renwick]

I have read a nice article about how to deploy 10Gbps network in less than $60 10 Gbps network for less than $60

In this article, several inexpensive NICs, switches and DACs are recommended.

As to the 10G SFP+ optics, DAC twinax might not be an option for the 20-30m distance. You can choose the cat6/cat7 cables or SFP+ fiber optics as the interconnect media. However, this might depend on what type of switch or NiCs you'd like to use.

 

[pc_doom]

NIC card: Mellanox Connectx-2 or Chelsio 110-1088-30 are okay. But I would take connectX-2 over Chelsio ANY day. Chelsio cards run fantastically hot and suck like 3x the power of the Mellanox cards. In a tower server, I had to jury-rig in custom cooling to get the Chelsio card not to shut down due to the heat.

Switch: Quanta lb6m or ES-16-XG are both good. I am currently running ES-16-XG switch in my home network, that's bought in the sale for $299, but not it's $599.

Cables: I never try Quanta lb6m, so if you use UBNT switch, you have to know that the current firmware of ES-16-XG does not supprt well with DAC cable, so you'd better choose SFP+ optics and OM4 patch cables for your connection. That will not cost you much, since 3-rd party SFP+ optics is really cheap now. See Cisco SFP-10G-SR 10GBASE-SR SFP+ Transceiver | FS.COM

 

[am4593]

I'm curious as to why no infiniband recommendations when it seems to be cheaper and at least 40Gbit. Is infiniband really that dead

 

[pyro_]

I think it is mainly because 10ge just works with existing network services and integrates nicely with existing network equipment. Infiniband does not integrate as nicely and is more of a pain to setup with existing ethernet networks.

Also for the most part 10g is fast enough 40g is not really needed though can be nice to have if you have all of the other needed equipment that can feed it fast enough which is a whole other story

 

[acquacow]

Infiniband is great if you work at a national lab, are building supercomputers, and want high-bandwidth memory checkpointing.

Aside from that, it can be a science project to setup, the rdma packages are constantly changing, and most of the ethernet implementations I've seen are emulated by the drivers at 10GigE and carry some overhead.

Plus you need a host to act as your master and if something happens to that host, your network won't work.

 

[Blinky 42]

The least expensive for 16 or more hosts @ 10G is probably going to be get a quanta lb6m, a stack of SR SFP+ optics new or from eBay, and then a collection of as-needed lengths of OM4. I wouldn't bother with DAC cables for anything outside a single rack or 2 adjacent racks, and avoid active DAC's - it isn't worth the hassle and expense when you can buy new SFP+ modules and fiber and route it a lot easier for less and have something that is better supported in a mix of equipment.
The Quantas have a cli only, it and depending on the firmware you get with it, the specific commands vary a bit. If you are setting up something not too complex with a few VLANs you will be fine if you have ever used a cli interface on a switch before.
While Connectx-2's are cheap, I would try and invest in ConnectX 3's as they are newer, more capable and better supported. The Intel cards are fine also and well supported, whatever you can get at a good price that day is fine.

If you need copper or copper & fiber, then don't bother with the LB6M and get one of the Netgear or UBNT switches.

 

[breakaway]

So what kind of NICs do I need? There are a LOAD of them on ebay, some very cheap, some with SFPs already loaded, some without.

If I get a PCI-e nic with a SFP+ slot, can I basically buy any SFP (e.g. one from fs.com for example) and it will "Just work" or will I land in compatibility hell where only a specific few SPF+ can be installed in a specific type of card?

 

[pyro_]

Most cards are fairly free with what optics they accept. It gets a bit trickier when you get to the switches. Some switches are brand locked for the sfp optics what you plug into them and others don't care. Decide on what switch you are going with first and that will drive if you need sfp+ or copper based cards

 

[breakaway]

How loud is the LB6M? They seem cheap and do what I need it to do by the looks of things.

I'm OK to figure out the CLI - can always get assistance from you helpful bunch!

What PCI-e NICs are good for computers? I am happy to use single mode or multi mode fiber. Whatever's cheaper. This is a fly-by-night ghetto setup of course!

 

[Joshh]

How loud is the LB6M? They seem cheap and do what I need it to do by the looks of things.

I'm OK to figure out the CLI - can always get assistance from you helpful bunch!

What PCI-e NICs are good for computers? I am happy to use single mode or multi mode fiber. Whatever's cheaper. This is a fly-by-night ghetto setup of course!

You should look at the thread for the LB6M specifically, there is a ton of posts regarding the sound. In most folks opinion it is pretty loud. There are things you can do to make it quieter though.

 

[i386]

How loud is the LB6M? They seem cheap and do what I need it to do by the looks of things.

I'm OK to figure out the CLI - can always get assistance from you helpful bunch!

What PCI-e NICs are good for computers? I am happy to use single mode or multi mode fiber. Whatever's cheaper. This is a fly-by-night ghetto setup of course!

If you don't need more than 4 10gbe ports and want a quiet (fanless) switch get a tp link t1700g-28tq.

Mellanox connectx2 nics are probably the cheapest on ebay (<$20 each) and they support many transceiver (for fiber) and dacs.