1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Moving to 10gbe

Discussion in 'Networking' started by doup93, Jan 9, 2017.

  1. doup93

    doup93 New Member

    Joined:
    Feb 26, 2013
    Messages:
    15
    Likes Received:
    1
    Hi all,

    I currently have a home lab that consists of one file server, 2 Hyper-V servers and my workstation. All servers are on Server 2012 R2 and moving at some point to Server 2016 and my workstation is running Windows 10 Enterprise.

    All these are running gigE using a Netgear GS748T v5 and I'd like to move these to 10 gigE without buying a switch but having a hard time making sense of all the SFP, SFP+, intel adapters, Mellanox adapters, etc, aka complete noob in that department. Current cabling is CAT 6a for all the above.

    Any useful links you guys can share where I can learn some of that stuff, how to configure and use them, recommendations, etc would be much appreciated.
     
    #1
  2. kiteboarder

    kiteboarder New Member

    Joined:
    May 10, 2016
    Messages:
    9
    Likes Received:
    2
    I'm also relatively new to 10 Gb, but here's my free summary that I recently emailed to some friends. Sorry if some of the ebay links have expired.

    You have two choices of connectors for "10 Gb ethernet": 10Gbase-T or SFP+

    10Gbase-T is the CAT 6 connector you have now. Drawbacks are IC heat, higher latency, and the NICs are expensive. The benefit is that the newer switches aren't very loud. However, these switches can be expensive.
    Amazon.com: NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Web Managed Switch (XS708E-200NES): Computers & Accessories

    SFP+ is a different connector type and is the 10 Gb version (don't confuse with SFP which is 1 Gb). You have at least two choices of cables, DAC (direct attach copper, or maybe also called twin-ax) or fiber optic. DAC cables are for "short" distances, think same room/rack. Fiber is when you want to go to different rooms/buildings. A downside of fiber is that you have to buy the "transceivers" separately from the cable. Used transceivers are about 10 bucks each. Most annoying is that the transceivers must be compatible with the device - they don't all place nice with each other. For fiber cable you want LC connectors, OM3 or OM4 grade, multi mode, 50/125 size (I think). Monoprice or fs.com can hook you up.

    Drawbacks of SFP+ is that you have to buy all of it from scratch. But unless you have already run CAT 6+, then it doesn't matter since you have to buy all of that from scratch too. So I've gone the SFP+ route.

    Used, single port SFP+ cards are stupid cheap:
    671798-001 666172-001 MNPA19-XTR HP 10GB ETHERNET NETWORK INTERFACE CARD W/CABLE | eBay

    18 bucks gets you a card AND a 3m cable.

    A good switch choice is the $299 ubiquity deal if you don't need that many ports.
    Ubiquiti Beta US-16-XG $299

    If you need a lot of ports, many on this forum have bought used Quanta LB6M:
    Quanta LB6M 10GBe 24-Port SFP+ Switch L2 Dual PS w/ Rack Ears 4x Uplink | eBay

    Only downside for this 24 port monster is heat and fan noise. But with my server on in the same room, it's louder than the switch so it doesn't matter much.
     
    #2
    pc_doom likes this.
  3. RyC

    RyC Active Member

    Joined:
    Oct 17, 2013
    Messages:
    255
    Likes Received:
    56
    I would strongly recommend buying a proper switch if you're trying to connect all those devices to each other. The Ubiquiti US-16-XG is probably the cheapest one currently that supports both SFP+ and 10gbe ports
     
    #3
  4. Jeff Robertson

    Joined:
    Oct 18, 2016
    Messages:
    124
    Likes Received:
    24
    I second the purchase of the ubiquiti switch, it is darn cheap for what you get. You may also want to look into these instead of the connectx-2 cards:

    649281-B21 656089-001 661685-001 HP IB FDR/EN 10/40GB 2P 544QSFP ADAPTER | eBay

    No nearly as cheap but I made an offer of $60 each and it was accepted. They are HP branded connectx-3 cards and are still being actively supported. HP just released new firmware for them a week or so ago. You will need to purchase SFP+ to QSFP+ adapters and they run $20 or less on ebay. Good luck, I just upgraded my network to 10Gb, it's expensive but fun.
     
    #4
  5. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    327
    Likes Received:
    27
    with 2x dual-port cards in each box you could put up to 5 devices in a mesh, where each device is directly connected to each other.

    Pro:
    no switch, not more power draw, no additional component (that can fail) in setup
    less latency

    Con:
    requires proper configuration / pushing routes on each port
    multiple adapters needed in each node
    limited number of nodes

    i personally would go sfp+ with dac copper for short distance/in-rack and Fibre for other room (workstation)
     
    #5
  6. doup93

    doup93 New Member

    Joined:
    Feb 26, 2013
    Messages:
    15
    Likes Received:
    1
    Managed to buy Whitey's 3 X520-DA2 and cables so that'll get me started for my servers. I'll setup a link between each servers so each HV server will have a link to the FS and between them.
     
    #6
    Last edited: Jan 11, 2017
  7. sparx

    sparx New Member

    Joined:
    Jul 16, 2015
    Messages:
    20
    Likes Received:
    8
    #7
    doup93 likes this.
  8. pyro_

    pyro_ Active Member

    Joined:
    Oct 4, 2013
    Messages:
    486
    Likes Received:
    115
    There is not much in the way of cheap 10gbbase-t out there once you get past a couple of ports. Figure on 600$+ for an 8 port one and go up from there. 8 port ones like the Xs708t use 15-30w of power and are fairly quiet
     
    #8
  9. sparx

    sparx New Member

    Joined:
    Jul 16, 2015
    Messages:
    20
    Likes Received:
    8
    Ah yes. Its 8ports of 10GBaseT and some SFPs. Would be good but a bit overkill for me still. I just need very few 10G and some 1G. Would like to stay away from optics. =)
     
    #9
  10. sparx

    sparx New Member

    Joined:
    Jul 16, 2015
    Messages:
    20
    Likes Received:
    8
    I probably wrote in the wrong way. 2 ports of 10G is fine. Dont need more actually. What else than the Asus is there?
     
    #10
  11. pyro_

    pyro_ Active Member

    Joined:
    Oct 4, 2013
    Messages:
    486
    Likes Received:
    115
    You could look at the netgear s3300 series switches. They have two sfp+ and two rj45 10gb ports along with either 24 or 48 gig ports. If you can work with sfp+ ports instead then there are a lot more options available on the cheaper end of the scale
     
    #11
  12. pyro_

    pyro_ Active Member

    Joined:
    Oct 4, 2013
    Messages:
    486
    Likes Received:
    115
    Also right now you could look at the ubiquiti unifi 16xg the beta store has them for 300$. 4 ports 10gbbase-t and 12 sfp+ ports
     
    #12
  13. sparx

    sparx New Member

    Joined:
    Jul 16, 2015
    Messages:
    20
    Likes Received:
    8
    I could, but then i have to add a second processor to my NAS. No free PCIe ports with one proc. Dont need the performance or electric bill :p .. I also saw this one with good SFP+ options. MikroTik Cloud Router Switch CRS210-8G-2S+IN.
     
    #13
  14. Waldek

    Waldek New Member

    Joined:
    Apr 3, 2016
    Messages:
    9
    Likes Received:
    1
    I would really not recommend the Ubiquiti es-16-xg. Looks great on paper, with specs and price. You can make it whisper quiet by replacing the fans with Noctuas (I actually mounted 3 Noctuas 40mm and reduced their speed). But... I do have the switch at my homelab, but has turned out to be my worst buying decision - lottery with cables (esp. DACs), lottery with ports (both the SFP+ and the 10GBase-T ones), lottery with network cards. Been there... What a waste of time.
    What gives you the highest chance with the switch is to use the fiber optic cables (100+ USD per piece). But, if this is for the homelab, then they will cost you more than the switch, so no point...
    There is a whole thread here dedicated to the gear. Similarly on the ubiquity forum. Lots of angry people. Ubiquiti has promised a new firmware to better cope with the DAC cables. Mid October '16. Now (January), they claim, they might have an alpha version by the end of February '17. Come on...
    Today, I would rather spend the money on the D-Link ones (D-Link DXS-1100-16SC or D-Link DXS-1210-12TC or -16TC), but for a homelab where silence is really a factor, I would mount one of the thin fans on the radiators (like NF-A9x14 PWM) and leave the switch top open (unfortunately, most, if not all, of the 10G switches are 1U, which makes them really difficult to silence...
     
    #14
    Cole likes this.
  15. pc_doom

    pc_doom New Member

    Joined:
    Nov 2, 2016
    Messages:
    24
    Likes Received:
    6
    Thanks for your sharing, and I know what I exactly need for 10G switch. SFP+ DAC and modules are both good solutions. And will go to monoprice or FS.COM for more information.
     
    #15
  16. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    1,887
    Likes Received:
    574
    This is NOT entirely true really, if it we're I'd be $800 lighter in the pockets over the last week or two as I recently converted to optics/fiber for my 10G connects and each optic cost $16 and OM4 cabling cost $4 each over at fs.com, so for my 4 ports of 10G (accouting for both ends, switch and nic) I ended up paying a lil' over $100 ($16x8 =$128 + $16 for four OM4 cables...$144 out the door...just under 20% of what you are claiming which I'm gonna throw my vote in for inaccurate unless you are just not 'shopping smartly/doing your research' or asking us :-D

    You can use DAC's and just stick to the 'known/validated' DAC's for now...nothing wrong w/ that other than it being a PITA but yeah fiber/optics may be most compatible on the US-16-XG currently. It's still a helluva lot of bang for buck when you get the connections sorted out and users on the forums here have reported pretty decent performance even driving IP SAN traffic so I don't think it's a dud...should only get better w/ firmware updates/release as frustrating as it is to wait for them.

    2cents
     
    #16
    Last edited: Jan 11, 2017
    wildchild and Jon Massey like this.
  17. sparx

    sparx New Member

    Joined:
    Jul 16, 2015
    Messages:
    20
    Likes Received:
    8
    I guess there is a major difference between single mode and multi mode fiber price. Multi mode beeing much cheaper. But if your home lab isn't in the km size then multi mode is great! =)
     
    #17
  18. Jon Massey

    Jon Massey Active Member

    Joined:
    Nov 11, 2015
    Messages:
    232
    Likes Received:
    60
    lolwut? For 100m of armoured 8-core SMF, maybe. For a few metres of OM4, not so much.
     
    #18
  19. wildchild

    wildchild Active Member

    Joined:
    Feb 4, 2014
    Messages:
    279
    Likes Received:
    31
    To add.. this IS a beta product that you have been given at half price for TESTING.
    Now that you find stuff that doesnt work you start complaining
    Ubiquiti is aware and is working with broadcom to get this fixed.
    This is NOT unique to this switch, but all switches/routers using this chipset.
    Some DAC's work, use those,or be patient and wait for a FW update
     
    #19
  20. Waldek

    Waldek New Member

    Joined:
    Apr 3, 2016
    Messages:
    9
    Likes Received:
    1
    No, you are mixing two different products. I am talking about a released product es-16-xg, being on the market at least since August 2016. Bought mine in mid-September. My first complaint to their support was end of September. I might have an alpha version of the firmware supposedly improving the situation with DACs by end of February (already after being fooled they will have it by end of December).

    Before commenting take a look at the thread regarding the device here at the STH and the ubiquity forum... It is not just about the DACs. Also, about NICs compatibility, the ports working stable (some do, some don't). Again, if the firmware was ready, I would buy a second unit to my homelab, because the specs and price look very good (on paper, but not in reality).
     
    #20
Similar Threads: Moving 10gbe
Forum Title Date
Networking Questions About Moving to 10gbe May 12, 2015
Networking pfSense moving into small switches? Dec 12, 2016
Networking Moving computers to another room. How should I connect displays and USB? Mar 13, 2016
Networking Moving from traditional router to IDS / IPS / UTM... suggestions? Jan 16, 2016
Networking 12 sfp+ + 4 10gbe rj45 - Ubiquiti Edgeswitch 16xg Aug 3, 2016

Share This Page