Moving to 10gbe

doup93

New Member
Feb 26, 2013
21
2
3
Hi all,

I currently have a home lab that consists of one file server, 2 Hyper-V servers and my workstation. All servers are on Server 2012 R2 and moving at some point to Server 2016 and my workstation is running Windows 10 Enterprise.

All these are running gigE using a Netgear GS748T v5 and I'd like to move these to 10 gigE without buying a switch but having a hard time making sense of all the SFP, SFP+, intel adapters, Mellanox adapters, etc, aka complete noob in that department. Current cabling is CAT 6a for all the above.

Any useful links you guys can share where I can learn some of that stuff, how to configure and use them, recommendations, etc would be much appreciated.
 

kiteboarder

Active Member
May 10, 2016
106
46
28
42
I'm also relatively new to 10 Gb, but here's my free summary that I recently emailed to some friends. Sorry if some of the ebay links have expired.

You have two choices of connectors for "10 Gb ethernet": 10Gbase-T or SFP+

10Gbase-T is the CAT 6 connector you have now. Drawbacks are IC heat, higher latency, and the NICs are expensive. The benefit is that the newer switches aren't very loud. However, these switches can be expensive.
Amazon.com: NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Web Managed Switch (XS708E-200NES): Computers & Accessories

SFP+ is a different connector type and is the 10 Gb version (don't confuse with SFP which is 1 Gb). You have at least two choices of cables, DAC (direct attach copper, or maybe also called twin-ax) or fiber optic. DAC cables are for "short" distances, think same room/rack. Fiber is when you want to go to different rooms/buildings. A downside of fiber is that you have to buy the "transceivers" separately from the cable. Used transceivers are about 10 bucks each. Most annoying is that the transceivers must be compatible with the device - they don't all place nice with each other. For fiber cable you want LC connectors, OM3 or OM4 grade, multi mode, 50/125 size (I think). Monoprice or fs.com can hook you up.

Drawbacks of SFP+ is that you have to buy all of it from scratch. But unless you have already run CAT 6+, then it doesn't matter since you have to buy all of that from scratch too. So I've gone the SFP+ route.

Used, single port SFP+ cards are stupid cheap:
671798-001 666172-001 MNPA19-XTR HP 10GB ETHERNET NETWORK INTERFACE CARD W/CABLE | eBay

18 bucks gets you a card AND a 3m cable.

A good switch choice is the $299 ubiquity deal if you don't need that many ports.
Ubiquiti Beta US-16-XG $299

If you need a lot of ports, many on this forum have bought used Quanta LB6M:
Quanta LB6M 10GBe 24-Port SFP+ Switch L2 Dual PS w/ Rack Ears 4x Uplink | eBay

Only downside for this 24 port monster is heat and fan noise. But with my server on in the same room, it's louder than the switch so it doesn't matter much.
 
  • Like
Reactions: Tha_14 and pc_doom

RyC

Active Member
Oct 17, 2013
357
89
28
I would strongly recommend buying a proper switch if you're trying to connect all those devices to each other. The Ubiquiti US-16-XG is probably the cheapest one currently that supports both SFP+ and 10gbe ports
 

Jeff Robertson

Active Member
Oct 18, 2016
423
113
43
Chico, CA
I second the purchase of the ubiquiti switch, it is darn cheap for what you get. You may also want to look into these instead of the connectx-2 cards:

649281-B21 656089-001 661685-001 HP IB FDR/EN 10/40GB 2P 544QSFP ADAPTER | eBay

No nearly as cheap but I made an offer of $60 each and it was accepted. They are HP branded connectx-3 cards and are still being actively supported. HP just released new firmware for them a week or so ago. You will need to purchase SFP+ to QSFP+ adapters and they run $20 or less on ebay. Good luck, I just upgraded my network to 10Gb, it's expensive but fun.
 

_alex

Active Member
Jan 28, 2016
874
95
28
Bavaria / Germany
with 2x dual-port cards in each box you could put up to 5 devices in a mesh, where each device is directly connected to each other.

Pro:
no switch, not more power draw, no additional component (that can fail) in setup
less latency

Con:
requires proper configuration / pushing routes on each port
multiple adapters needed in each node
limited number of nodes

i personally would go sfp+ with dac copper for short distance/in-rack and Fibre for other room (workstation)
 

doup93

New Member
Feb 26, 2013
21
2
3
Managed to buy Whitey's 3 X520-DA2 and cables so that'll get me started for my servers. I'll setup a link between each servers so each HV server will have a link to the FS and between them.
 
Last edited:

pyro_

Active Member
Oct 4, 2013
747
164
43
There is not much in the way of cheap 10gbbase-t out there once you get past a couple of ports. Figure on 600$+ for an 8 port one and go up from there. 8 port ones like the Xs708t use 15-30w of power and are fairly quiet
 

sparx

Active Member
Jul 16, 2015
227
70
28
Sweden
There is not much in the way of cheap 10gbbase-t out there once you get past a couple of ports. Figure on 600$+ for an 8 port one and go up from there. 8 port ones like the Xs708t use 15-30w of power and are fairly quiet
Ah yes. Its 8ports of 10GBaseT and some SFPs. Would be good but a bit overkill for me still. I just need very few 10G and some 1G. Would like to stay away from optics. =)
 

sparx

Active Member
Jul 16, 2015
227
70
28
Sweden
I probably wrote in the wrong way. 2 ports of 10G is fine. Dont need more actually. What else than the Asus is there?
 

pyro_

Active Member
Oct 4, 2013
747
164
43
You could look at the netgear s3300 series switches. They have two sfp+ and two rj45 10gb ports along with either 24 or 48 gig ports. If you can work with sfp+ ports instead then there are a lot more options available on the cheaper end of the scale
 

pyro_

Active Member
Oct 4, 2013
747
164
43
Also right now you could look at the ubiquiti unifi 16xg the beta store has them for 300$. 4 ports 10gbbase-t and 12 sfp+ ports
 

sparx

Active Member
Jul 16, 2015
227
70
28
Sweden
I could, but then i have to add a second processor to my NAS. No free PCIe ports with one proc. Dont need the performance or electric bill :p .. I also saw this one with good SFP+ options. MikroTik Cloud Router Switch CRS210-8G-2S+IN.
 

Waldek

New Member
Apr 3, 2016
17
2
3
47
I would really not recommend the Ubiquiti es-16-xg. Looks great on paper, with specs and price. You can make it whisper quiet by replacing the fans with Noctuas (I actually mounted 3 Noctuas 40mm and reduced their speed). But... I do have the switch at my homelab, but has turned out to be my worst buying decision - lottery with cables (esp. DACs), lottery with ports (both the SFP+ and the 10GBase-T ones), lottery with network cards. Been there... What a waste of time.
What gives you the highest chance with the switch is to use the fiber optic cables (100+ USD per piece). But, if this is for the homelab, then they will cost you more than the switch, so no point...
There is a whole thread here dedicated to the gear. Similarly on the ubiquity forum. Lots of angry people. Ubiquiti has promised a new firmware to better cope with the DAC cables. Mid October '16. Now (January), they claim, they might have an alpha version by the end of February '17. Come on...
Today, I would rather spend the money on the D-Link ones (D-Link DXS-1100-16SC or D-Link DXS-1210-12TC or -16TC), but for a homelab where silence is really a factor, I would mount one of the thin fans on the radiators (like NF-A9x14 PWM) and leave the switch top open (unfortunately, most, if not all, of the 10G switches are 1U, which makes them really difficult to silence...
 
  • Like
Reactions: Cole

pc_doom

Member
Nov 2, 2016
50
17
8
26
I'm also relatively new to 10 Gb, but here's my free summary that I recently emailed to some friends. Sorry if some of the ebay links have expired.

You have two choices of connectors for "10 Gb ethernet": 10Gbase-T or SFP+

10Gbase-T is the CAT 6 connector you have now. Drawbacks are IC heat, higher latency, and the NICs are expensive. The benefit is that the newer switches aren't very loud. However, these switches can be expensive.
Amazon.com: NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Web Managed Switch (XS708E-200NES): Computers & Accessories

SFP+ is a different connector type and is the 10 Gb version (don't confuse with SFP which is 1 Gb). You have at least two choices of cables, DAC (direct attach copper, or maybe also called twin-ax) or fiber optic. DAC cables are for "short" distances, think same room/rack. Fiber is when you want to go to different rooms/buildings. A downside of fiber is that you have to buy the "transceivers" separately from the cable. Used transceivers are about 10 bucks each. Most annoying is that the transceivers must be compatible with the device - they don't all place nice with each other. For fiber cable you want LC connectors, OM3 or OM4 grade, multi mode, 50/125 size (I think). Monoprice or fs.com can hook you up.

Drawbacks of SFP+ is that you have to buy all of it from scratch. But unless you have already run CAT 6+, then it doesn't matter since you have to buy all of that from scratch too. So I've gone the SFP+ route.

Used, single port SFP+ cards are stupid cheap:
671798-001 666172-001 MNPA19-XTR HP 10GB ETHERNET NETWORK INTERFACE CARD W/CABLE | eBay

18 bucks gets you a card AND a 3m cable.

A good switch choice is the $299 ubiquity deal if you don't need that many ports.
Ubiquiti Beta US-16-XG $299

If you need a lot of ports, many on this forum have bought used Quanta LB6M:
Quanta LB6M 10GBe 24-Port SFP+ Switch L2 Dual PS w/ Rack Ears 4x Uplink | eBay

Only downside for this 24 port monster is heat and fan noise. But with my server on in the same room, it's louder than the switch so it doesn't matter much.
Thanks for your sharing, and I know what I exactly need for 10G switch. SFP+ DAC and modules are both good solutions. And will go to monoprice or FS.COM for more information.
 

whitey

Moderator
Jun 30, 2014
2,773
867
113
38
I would really not recommend the Ubiquiti es-16-xg. Looks great on paper, with specs and price. You can make it whisper quiet by replacing the fans with Noctuas (I actually mounted 3 Noctuas 40mm and reduced their speed). But... I do have the switch at my homelab, but has turned out to be my worst buying decision - lottery with cables (esp. DACs), lottery with ports (both the SFP+ and the 10GBase-T ones), lottery with network cards. Been there... What a waste of time.
What gives you the highest chance with the switch is to use the fiber optic cables (100+ USD per piece). But, if this is for the homelab, then they will cost you more than the switch, so no point...
There is a whole thread here dedicated to the gear. Similarly on the ubiquity forum. Lots of angry people. Ubiquiti has promised a new firmware to better cope with the DAC cables. Mid October '16. Now (January), they claim, they might have an alpha version by the end of February '17. Come on...
Today, I would rather spend the money on the D-Link ones (D-Link DXS-1100-16SC or D-Link DXS-1210-12TC or -16TC), but for a homelab where silence is really a factor, I would mount one of the thin fans on the radiators (like NF-A9x14 PWM) and leave the switch top open (unfortunately, most, if not all, of the 10G switches are 1U, which makes them really difficult to silence...
This is NOT entirely true really, if it we're I'd be $800 lighter in the pockets over the last week or two as I recently converted to optics/fiber for my 10G connects and each optic cost $16 and OM4 cabling cost $4 each over at fs.com, so for my 4 ports of 10G (accouting for both ends, switch and nic) I ended up paying a lil' over $100 ($16x8 =$128 + $16 for four OM4 cables...$144 out the door...just under 20% of what you are claiming which I'm gonna throw my vote in for inaccurate unless you are just not 'shopping smartly/doing your research' or asking us :-D

You can use DAC's and just stick to the 'known/validated' DAC's for now...nothing wrong w/ that other than it being a PITA but yeah fiber/optics may be most compatible on the US-16-XG currently. It's still a helluva lot of bang for buck when you get the connections sorted out and users on the forums here have reported pretty decent performance even driving IP SAN traffic so I don't think it's a dud...should only get better w/ firmware updates/release as frustrating as it is to wait for them.

2cents
 
Last edited:

sparx

Active Member
Jul 16, 2015
227
70
28
Sweden
I guess there is a major difference between single mode and multi mode fiber price. Multi mode beeing much cheaper. But if your home lab isn't in the km size then multi mode is great! =)
 

wildchild

Active Member
Feb 4, 2014
394
57
28
This is NOT entirely true really, if it we're I'd be $800 lighter in the pockets over the last week or two as I recently converted to optics/fiber for my 10G connects and each optic cost $16 and OM4 cabling cost $4 each over at fs.com, so for my 4 ports of 10G (accouting for both ends, switch and nic) I ended up paying a lil' over $100 ($16x8 =$128 + $16 for four OM4 cables...$144 out the door...just under 20% of what you are claiming which I'm gonna throw my vote in for inaccurate unless you are just not 'shopping smartly/doing your research' or asking us :-D

You can use DAC's and just stick to the 'known/validated' DAC's for now...nothing wrong w/ that other than it being a PITA but yeah fiber/optics may be most compatible on the US-16-XG currently. It's still a helluva lot of bang for buck when you get the connections sorted out and users on the forums here have reported pretty decent performance even driving IP SAN traffic so I don't think it's a dud...should only get better w/ firmware updates/release as frustrating as it is to wait for them.

2cents
To add.. this IS a beta product that you have been given at half price for TESTING.
Now that you find stuff that doesnt work you start complaining
Ubiquiti is aware and is working with broadcom to get this fixed.
This is NOT unique to this switch, but all switches/routers using this chipset.
Some DAC's work, use those,or be patient and wait for a FW update
 

Waldek

New Member
Apr 3, 2016
17
2
3
47
To add.. this IS a beta product that you have been given at half price for TESTING.
Now that you find stuff that doesnt work you start complaining
Ubiquiti is aware and is working with broadcom to get this fixed.
This is NOT unique to this switch, but all switches/routers using this chipset.
Some DAC's work, use those,or be patient and wait for a FW update
No, you are mixing two different products. I am talking about a released product es-16-xg, being on the market at least since August 2016. Bought mine in mid-September. My first complaint to their support was end of September. I might have an alpha version of the firmware supposedly improving the situation with DACs by end of February (already after being fooled they will have it by end of December).

Before commenting take a look at the thread regarding the device here at the STH and the ubiquity forum... It is not just about the DACs. Also, about NICs compatibility, the ports working stable (some do, some don't). Again, if the firmware was ready, I would buy a second unit to my homelab, because the specs and price look very good (on paper, but not in reality).