Switch Replacement recommendations

Discussion in 'Networking' started by spali, Nov 6, 2018.

  1. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    Hi all

    before I start, thanks to all for the great infos I got from here, but now my first post is required ;)

    My Current homelab Setup:
    • 2 x Stacked SG500-28MPP
      • 26 x 1000Base-T POE+ powered for the house cabling
      • 4 x 1000Base-T -> Server 1 (Windows Server Management, VMs etc.)
      • 4 x 1000Base-T -> Server 2 (Windows Server Management, VMs etc.)
      • 4 x 1000Base-T -> Server 3 (Windows Server Management, VMs etc.)
    I use these servers for a 3-node hyper-v cluster. Where two nodes have some additional storage to share for my network.

    Last month I upgraded all 3 servers with a Dual ConnectX-3 Pro VPI which are direct attached by 40gbit DAC cables in ETH Mode with each other mainly for SMB-direct.
    Because they are only direct attached, I'm currently limited in the use of those babies with only migration and cluster communication.

    My thought and Ideas

    Now I want to "simplify" the setup with a 40Gbit switch, which I can "downlink" to the network and to have the possibility to upgrade some or all house cabling connections to 10gbit cooper.
    I would like to use the 40gbit as converged network to also share the storage to the rest of the network with more than 1Gbit transparently.
    So I need a switch for the 40Gbit to interconnect them with my network but keep the RDMA feature for SMB direct.

    So the new setup should contain a core switch with at least 6 QSFP+ ports with DCB/PFC capability.
    House cabling in general keeps 1Gbit cooper, but want the possibility to upgrade some of them to 10gbit cooper (i.e to the Office).
    Possibility to have POE on all house cabling connections, because I love how I can just hook up those cheap netgear GS105PE switches if I need some more ports somewhere in the house for TV or whatever boxes :)

    Currently I think I have to go anyway for at least two new switches.
    One core switch for the servers and uplinking the "house" switch with some SFP+ and 6 QSFP+.

    For the "house" switch it gets a bit more complicated, 1/10Gbit cooper POE+ with at least 32 ports (or even 48 to have some spare ports) uplinked with SFP+.

    The switches would be in a rack in the basement, so doesn't have to be silent, but still would like to keep the noise as low as possible. But this wouldn't be a show stopper as long I don't hear it on the next floor with one door between :)
    The power consumption should be as low as possible, currently my whole rack with 3 low power server and the switches are around 300W during idle times (at least was before the ConnectX, should probably measure it again some time).
    Due it's a homelab, I'm fine with used hardware.

    Currently I'm a bit stuck, because the "house" switch.
    I found the Juniper EX4300M which is probably the best match for my use-case as "house" switch.
    But the price is astronomical for a home network.
    Didn't found something useful that is payable to get all what I need in one switch...

    For the core switch currently I'm thinking of getting a used DCS-7050QX-32S, they are available at a reasonable price. And I could at least integrate it directly in the network.

    My budget is variable if I find the perfect match, but I think there must be something below 2/3k for my use-case.

    Interested in any suggestions about the whole network setup, but specifically have these questions too:
    • Does it make sense to get two DCS-7050QX-32S for availability and for testing in a homelab? Thinking especially on this switch because as I read on STH, you can almost do anything on them ;)
    • Should I stop watching for a "house" switch that does just anything and keep the SG500 for 1Gbit and POE and buy a cheap 10Gbit switch for my planned partial upgrade to 10Gbit?

    Sorry for the long text, but tried to bring some context for hopefully better recommendations :)
     
    #1
  2. dwright1542

    dwright1542 Active Member

    Joined:
    Dec 26, 2015
    Messages:
    338
    Likes Received:
    67
    ICX 6650 with a 6610?
     
    #2
  3. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    100
    Likes Received:
    28
    A few thoughts. Check this out: Juniper Networks EX4300-48P PoE+ QSFP+ 40GBASE 1GbE 10GbE Switch Warranty | eBay

    They keep sending me "Give me an offer" emails, so you know they want to move it. I'd offer $350 and see if they take it. That'll get you 4x 10Gbe and 4x 40Gbe.

    Or look for Arista 7050q-16. Again drop some lowball offers, you should be able to get this for $500 or less. Don't pay more then that.

    I have two Arista 7050qx-32x switches, they're incredible. If you want something that will move a LOT of packets then go this route.

    The 7050q-16 is great for what you nead, 16x 40gbe and 8x 10gbe. You can use transceivers in the 10gbe and step it down to 1gbe if you need as well. Someone posted on r/homelab on reddit how they replaced the fans to make the switch silent too. They're loud, ours live in a datacenter so we don't care. But in the office for setup you need noise cancelling headphones. But with a fan upgrade they're near silent.
     
    #3
    spali likes this.
  4. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    6610 doesn't have 10Gbit RJ45 which I require for what I would need in the house (sorry if I didn't mentioned that clearly).
    6650 does look more expensive with the license (which I would require), but do they support DCB and PFC?
    Thanks anyway for the recommendations, didn't look for ICX's in general yet.

    The linked EX4300 is a great switch, but lacks also of 10Gbit RJ45 ports. I dream of the EX4300-48MP, sorry didn't found the exact model name before ;)

    I think for the core switch I'll go the route with Arista 7050qx-32xS, do you have two of them in homelab use? Do you think it is worth to buy two if I get them cheap, even I will never need all ports of one (maybe to play with one) :) Especially thinking about docker support for core network services like DHCP etc.
     
    #4
  5. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    100
    Likes Received:
    28
    Not for a homelab, in a production environment.

    We’re using 15-16 ports on them. They’re in an MLAG configuration so 32 ports between both. But the upside is double bandwidth. Servers have 80Gbps if directly attached.

    CPU usage is pretty stable at 10-15%.

    It’s really hard to sustain 40Gbe or 80Gbe. We like it for the lower latency. Processing can be offloaded to a separate machine to work in RAM. Doing this over a 40g fabric is faster than hitting a local disk.
     
    #5
    spali likes this.
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,429
    Likes Received:
    327
    (16x 40gbe) or (15x 40gbe & 4x sfp+) or (14x 40gbe & 8 sfp+)
     
    #6
  7. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    100
    Likes Received:
    28
    Yes, good point, I forgot that you lose 40g ports when you turn on the 10g. You don't really lose anything, it's the same as using a 40g-4x 10g breakout, but with more visibility into what's happening at the port level.
     
    #7
  8. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    I'm preparing for ordering a 7050qx-32xS and give it a shot. I read in an other thread you participating about upgrading RAM. I didn't see the exact module type there, also because there was a mix of different models in the thread. Does the 32S use pc3-10600 unbuffered ECC DIMM's?
     
    #8
  9. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    100
    Likes Received:
    28
    This is the DIMM that is in there from the manufacturer:

    TriCor TRF7251U67G1600G8-MTEBP
    ASMEM-00171

    Which according to the manufacturer is:

    TRF7251U67G1600Gx-xxxP DDR3 SDRAM UDIMM 4GB 240-pin 512Mb x72 512M x 8 Unbuffered 1.35V CL = 11 1600 MT/s PC3-12800 9 ECC Single Rank Yes
     
    #9
    spali likes this.
  10. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    100
    Likes Received:
    28
    Sorry, here's the manufacturer link: DDR3
     
    #10
  11. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    Wow thanks, that's precise.
    Now I understand the commend about the rare 16GB DIMM's ;)
    But there are some around on ebay... I think I will give it a try as soon I get the switch.
     
    #11
  12. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    Because my core switch would need at least 6 QSFP+ for the servers. And the switch for my house cabling would be primary cooper.

    was wrong... they are all registered :(
     
    #12
  13. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    Wohoo... just ordered one for 825$ :)
    Has anyone mounted this switch with just the ears? I'm scared for such a big switch... but I can't find rail kits for a reasonable price in Europe... they wan't 200-300Euro :confused:
     
    #13
  14. oddball

    oddball Active Member

    Joined:
    May 18, 2018
    Messages:
    100
    Likes Received:
    28
    I have one with just ears, but it's sitting on top of another on rails, so sort of?

    You need a unique rail kit for these. There are inner rails that attach to the studs on the switch. Then an outer rail kit they slide into.
     
    #14
  15. spali

    spali New Member

    Joined:
    Nov 4, 2018
    Messages:
    14
    Likes Received:
    1
    Yes, was afraid of that, but didn't found a payable option with delivery in/to Europe.
    I think and am getting some cheap universal rails that at least hold the switch on the sides. And hope will get Arista rails later from somewhere.
     
    #15
  16. Stril

    Stril Member

    Joined:
    Sep 26, 2017
    Messages:
    88
    Likes Received:
    5
    Did you ever use the FS.com switches?
    They look promising, but nearly zero reputation...
     
    #16
Similar Threads: Switch Replacement
Forum Title Date
Networking Quiet-speed ... switches with a few QSFP ports with DIY Noctua replacement fans? Yesterday at 7:16 PM
Networking Bricked my Netgear Switch... help & opinions on replacement Oct 1, 2018
Networking Switch that allows custom firmware Thursday at 12:26 PM
Networking NAS connection on Cisco Nexus 5020 Switch Nov 13, 2018
Networking Do 1Gb switches matter? Nov 12, 2018

Share This Page