Questions About Moving to 10gbe

Discussion in 'Networking' started by Roman2179, May 12, 2015.

  1. Roman2179

    Roman2179 Member

    Joined:
    Sep 23, 2013
    Messages:
    48
    Likes Received:
    14
    As the title states, I am looking into moving some and maybe all of my servers over to 10gbe. Partially to help speed up file transfers and partially to simplify some of the cabling since I have a few LAGG groups that I would like to get rid of. Since I am complete newbie when it comes to anything 10gbe/fiber, I am hoping that you guys can help me and save me some money on trial and error.

    To start things off, here are the servers:
    FreeNAS
    Supermicro X9SCM-F
    Intel e3-1230 v2
    32gb RAM
    12x3TB WD RED
    6x2TB WD RED
    Will be upgrading to 9.3

    ESXi 5.5 Server #1
    Supermicro X9SCM-F
    Intel e3-1230 v2
    32gb RAM
    Quad Port 1gbe NIC

    ESXi 5.5 Server #2
    Supermicro A1SAM-2750F
    32gb RAM
    Quad Port 1gbe NIC

    So the two servers I would really like to get on 10gbe are the FreeNAS server and the first ESXi server since I move the most files between those two. I would really like to be able to speed up the archive process. Ideally, if I can get a switch that has four 10gbe ports for a reasonable price, I would like to move the second ESXi server and my workstation to 10gbe as well.

    From my research so far, it seems like the Intel x520 is an ideal choice for NICs as far as driver support goes but those get expensive fast if I were to get four of them. It seems like the Intel X520 is a good choice for FreeNAS but seems to have driver issues in some instances. Chelsio t3/t4/t5 cards work with FreeNAS as well but they don't seem to be any cheaper than the Intel cards but look like they are the preferred card by iXsystems. As far as the ESXi server go, can I use something like Connectx-2 EN cards since they go for about 35 on ebay or Brocade 1010/1020 cards?

    Next decision is the switch. I was looking at the MicroTik CRS226-24G-2S+RM since it was in the price range I am looking for and satisfies the criteria of switching two servers to 10gbe but leaves zero room to expand. With some reading on the forums here, the Netgear ProSafe GSM7328Sv2 seems like a good choice and gives me the option of adding the other servers later on. Are there any switches that have four 10gbe SFP+ ports that are near close the price range of the two switches I have listed?

    Finally, the transcievers, I have no idea which ones work with what switches and cards. Do I need to buy the 60+ dollar ones for the Intel cards or can I use the $18 ones from the the fiberstore?

    Thanks for the help in advance!

    EDIT:
    Thinking of picking up a couple of these Connectx-2 en cards on Ebay. Thoughts? Also what transceivers work with these?
     
    #1
    Last edited: May 12, 2015
  2. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,923
    Likes Received:
    1,523
    NetGear has a 4 port 10gb SFP switch with iirc 24 1gigE ports ~600$ which I think is a great deal for all you get, and would have gone that way myself had I not got their 14 Port 10gige switch and a LB4M (that im selling to get the netgear to make life easier).

    I was going with Mellanox, direct connect cables, cat6, and fiber and decided it just got too messy for home/lab/etc use, and am ditching most of my non Cat6 stuff, fiber staying for switch connections.

    I'm also using the X520 Intel cards. Those + NetGear make life about 10x easier to setup and get going.

    If you want to do more then maybe the MEllanox, etc is for you, but I decdied if I'm going that way I'm going 40gig eventually :D LOL
     
    #2
  3. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,663
    Likes Received:
    4,601
    Check FreeNAS and those Mellanox cards for compatibility.
     
    #3
  4. Roman2179

    Roman2179 Member

    Joined:
    Sep 23, 2013
    Messages:
    48
    Likes Received:
    14
    I was thinking of using the Mellanox cards for the ESXi servers and maybe the workstation. I doesn't look like FreeNAS will support Mellanox cards until 10.x. I think the safe bet for FreeNAS is an Intel x520, pricier but the seems to have the least issues.
     
    #4
  5. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,923
    Likes Received:
    1,523
    Yes, the Intel is by far the easiest and fastest to setup solution :)
     
    #5
  6. neo

    neo Well-Known Member

    Joined:
    Mar 18, 2015
    Messages:
    672
    Likes Received:
    359
    For 10 GbE - Chelsio cards seem to get the most attention from FreeBSD developers. I've used almost every brand card, and do prefer the Chelsio.

    As a cheap plug, I do have a T420 for sale on my other thread.
     
    #6
  7. Hank C

    Hank C Active Member

    Joined:
    Jun 16, 2014
    Messages:
    642
    Likes Received:
    66
    this will be my thread to see. I am also in the same situation. Looking to get either all 10gb on storage traffic or getting just 10gb on just the storage server and 1gb for the storage clients
     
    #7
  8. Roman2179

    Roman2179 Member

    Joined:
    Sep 23, 2013
    Messages:
    48
    Likes Received:
    14
    Any thoughts on the the Connectx-2 cards for the ESXi servers? Looking to go as cheap as possible for least the ESXi server #2 since it has a PCIe 8x slot that is being used by the RAID card and a PCIe 4x slot available for the network card which means I would have to cut the card down to 4x. Would rather cut a ~$35 dollar card vs modifying a $300+ motherboard.
     
    #8
  9. Entz

    Entz Active Member

    Joined:
    Apr 25, 2013
    Messages:
    269
    Likes Received:
    62
    The Connectx-2 EN cards work great in ESXi . That is what I am using (5.5) and they are dirt cheap. Not sure how well they do cut though..

    With the move to 10GBe is the raid card even still required (i.e. All storage on the NAS/Fileserver)?
     
    #9
    T_Minus likes this.
  10. Hank C

    Hank C Active Member

    Joined:
    Jun 16, 2014
    Messages:
    642
    Likes Received:
    66
    but the sfp+ switch are still on the high side though...even if the nic are dirt cheap
     
    #10
    T_Minus likes this.
  11. seang86s

    seang86s Member

    Joined:
    Feb 19, 2013
    Messages:
    136
    Likes Received:
    12
    I picked up the D-Link SmartPro DGS-1510-28X which is a 24 port 1GBe switch with 4 SFP+ ports. I'm using it for a 3 node ESXi cluster and 1GBe ports for a few desktops and miscellaneous home stuff. Got it for $550 from Provantage in March, but I see the price is $30 higher now, after a $20 instant rebate. I'm using 1.5 meter twinax cables to connect the machines I got off eBay for cheap.

    Here's the link: PROVANTAGE: D-Link Systems DGS-1510-28X SmartPro 24-Port Gigabit Switch with 4 10GbE SFP+
     
    #11
    Last edited: May 12, 2015
  12. Entz

    Entz Active Member

    Joined:
    Apr 25, 2013
    Messages:
    269
    Likes Received:
    62
    Very true, switches are the problem (and optics) but it is a pretty critical part and something you build your network around and don't change very often. That being said, you can get away with a single CRS226 for 3-4 servers at the expense of some bandwidth provided you have the PCI-e slots for extra cards in the file server..

    Good price on the DGS-1510-28X!
     
    #12
  13. Roman2179

    Roman2179 Member

    Joined:
    Sep 23, 2013
    Messages:
    48
    Likes Received:
    14
    I ended up getting a couple Connectx-2 EN cards for 33 a piece. Now the question is, which transceivers work with these?

    At this point, could I got without a switch if I got a 2 port card for the FreeNAS server and just connected the two ESXi server to it directly? I think I want to hold out and pick up the D-Link switch. Has all of the features I would want and the price seems pretty reasonable compared to everything else I've seen. Power usage for it seems pretty good as well.

    EDIT:
    Looks like a need a couple of transceivers for a Chelsio T420 as well. Thanks neo!
     
    #13
    Last edited: May 12, 2015
    neo likes this.
  14. toomuchstuff

    toomuchstuff Member

    Joined:
    Nov 4, 2013
    Messages:
    35
    Likes Received:
    9
    I am using Arista sfp-10g-sr optics in my switch and Connectx-2/3 EN cards. Make an offer on the optics and you can get them pretty cheap (mine were $18, maybe you can get them lower)
     
    #14
    Roman2179 and T_Minus like this.
  15. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,923
    Likes Received:
    1,523
    #15
  16. Roman2179

    Roman2179 Member

    Joined:
    Sep 23, 2013
    Messages:
    48
    Likes Received:
    14
    The seller took $15 for the Arista optics in case anyone is curious.
     
    #16
    toomuchstuff likes this.
  17. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,923
    Likes Received:
    1,523
    He countered my offer with $15 t00, btw. Declined my low-ball for fun offer on 25 units, countered for 15 for 10 units. So I don't think he's going under $15 unless you're going to get a lot. My last offer, 13$ well see :) LOL!! I love negotiating on ebay, its fast and simple LOL.

    And they rapidly declined the $13, and didn't even re-counter with $15. Some sellers, I wonder if they get how ebay works. I'm not going to BIN For $25, and the $15 is off the table, and I'm out of offers... LOL Ah-well

    $12-15 is going rate what I paid for past units, so I'd say that's the best deal they're gonna do :)
     
    #17
  18. neo

    neo Well-Known Member

    Joined:
    Mar 18, 2015
    Messages:
    672
    Likes Received:
    359
    Enjoy the Chelsio card, you'll love it.
     
    #18
  19. Roman2179

    Roman2179 Member

    Joined:
    Sep 23, 2013
    Messages:
    48
    Likes Received:
    14
    Chelsio SM10G-SR transceivers seem to be going for about $35 each on ebay, probably a little less with best offers. Has anyone used any other transceivers for T420-CR cards?
     
    #19
    Last edited: May 12, 2015
  20. Entz

    Entz Active Member

    Joined:
    Apr 25, 2013
    Messages:
    269
    Likes Received:
    62
    I have used Mikrotik, Brocade (Fiberstore) and some generic DAC (not sure what it is lol). From what I have seen/read the ConnectX-2 cards do not really seem to care.
     
    #20
    T_Minus likes this.
Similar Threads: Questions Moving
Forum Title Date
Networking VMWare NSX - I've some basic design questions Dec 17, 2019
Networking New Networking Setup Questions Sep 25, 2019
Networking 40Gb to 10Gb breakout cable questions Jun 19, 2019
Networking Questions about 100g networking Apr 9, 2019
Networking Juniper EX4200 Questions Dec 4, 2018

Share This Page