DIY 40Gigabit Switch?

Discussion in 'Networking' started by Sleyk, Oct 13, 2018.

  1. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Whatup peeps. Im back!

    Im kinda interested myself in going to 40Gb. I've been in the 10Gb game now for a while and I just finished some research on this new 25Gb standard business. I kinda like where its going, but 25Gb is hella expensive! Cuz its newer I guess.

    Anywayz, I'm thinking about building out a switch with 40Gb adapters, the same way I built a 10Gb switch with Pfsense.

    Can anyone weigh in if this has a shot at working? Those Hp mellanox's look cheap enough to throw a few in a box, then connect them up to each server.

    Anybody considered this?
     
    #1
  2. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,344
    Likes Received:
    328
    VyOS is what I would use, probably could kind of work but 40G switches don’t cost that much $$
     
    #2
  3. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Hey Evan, Yeah, I used to use Vyos for 10Gb so makes sense that it could work for 40Gb. What's your preference? VM for Vyos? I have an old lga1366 box with 7 x Chelsio dual port 10Gb nics and 48GB of ram that I could use with Vyos as a super-overkill switch and upgrade my current pcie 2.0 setup to 3.0 x 8 to use the connectx-3 40Gb cards.

    So I'm wondering if anyone is currently using a couple of connectx-3 40Gb cards in this config? (pfsense) I'm thinking of just running a connection to my media pc's at 40Gb and leaving all the servers on 10Gb through my old pfsense box. For that, I think I might just need about 2 cards and hope to goodness pfsense supports them. :D
     
    #3
  4. Juan C

    Juan C New Member

    Joined:
    Oct 9, 2018
    Messages:
    4
    Likes Received:
    1
    I've got a couple of cards on order to hopefully test a similar scenario. I'm getting a Brocade 6610 set up to act as my 40Gb bridge between the rest of the network and the VMs. Total overkill, but why not!
     
    #4
    Sleyk likes this.
  5. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    932
    Likes Received:
    679
    you're not gunna break more than 11 or 12gbps trying to bridge a bunch of 40gbE cards in a server unless you have some incredible CPU power. Even with a modern CPU and a single 40gbE card it usually takes something like an RDMA protocol to bypass a lot of the OS overhead to get anywhere near 40gbps. And even then, that's pure L2, if you introduce any kind of l3 it's going to drop to an order of magnitude less

    edit: just saw you're thinking of doing this in pfsense. pfsense (even with pf and everything disabled on an interface) has notoriously high overhead (search their forums for people on modern processors not being able to do more than 7 or 8gbps), I'd wager you wouldn't push past 10gbps

    Do you have anything (disk arrays etc) that can actually push traffic faster than 10gbps? If so I would just pick up a cheap switch with a couple 40gbE ports
     
    #5
    Last edited: Oct 13, 2018
    BackupProphet and Patrick like this.
  6. RTM

    RTM Active Member

    Joined:
    Jan 26, 2014
    Messages:
    341
    Likes Received:
    111
    I have no idea about how well it performs but Open vSwitch supports a DPDK datapath, it might be worth looking into for such a project.
    It is included in the Ubuntu repositories, so getting started should be relatively easy :cool:
     
    #6
    Sleyk likes this.
  7. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    457
    Likes Received:
    159
    #7
    Sleyk, gigatexal and Patrick like this.
  8. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,501
    Likes Received:
    441
    Yeah I’d just buy a switch and use the hardware for something else
     
    #8
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,195
    Likes Received:
    4,147
    I thought about it, but when switches are a few hundred dollars, there is very little reason to make one yourself. Even if you could do 40gbps per port switching, you cannot do dual 40gbps from one card to another. Practically, you can make an 8 port switch by spending a few hundred in 40GbE cards, then several hundred on a motherboard with PCIe lanes, power supply, chassis, CPU and RAM.

    Or you can just buy one that has more ports, more features, better performance and lower cost.

    Maybe if you just wanted 2 ports with 2 cards and already had the server you can come out ahead.
     
    #9
  10. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Yeah I was thinking just to bridge no more than 2-4 ports for a super fast connection to my media pc and server, and leave everything else on 10Gb. I think its doable, albeit I do understand I probably wont see 40Gb between cards, although that would be so nice ;)

    Fodeesh my fam! I'm using pfsense and get a consistent 9 Gbps across my 2 cards and I'm using an old 1156 mobo with an x3440 and 8gb ram so I know its doable, but you are most likely right on trying to break past 15Gbs on one of the 40Gb cards, although the tinkering spirit in me feels like I can do at least 20Gbps...

    The suggestions to just buy a switch is certainly valid, and definitely the cheaper option if you want alot of ports, but $35 x 2 cards + $20 in transceivers + maybe $50-$75 bucks in some fiber cables to start, = $165 (assuming you already got a decent mobo and cpu) isn't too bad right?
     
    #10
    gigatexal likes this.
  11. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    671
    Likes Received:
    142
    Lol Kapone! Why do you show me such a thing hmmm? I already went broke buying 2 LB6M's to play with.
    Wanna make me homeless now huh bro? ;)
     
    #11
Similar Threads: 40Gigabit Switch
Forum Title Date
Networking HP ProCurve 1800-8G J9029A switch Thursday at 11:29 AM
Networking Any reason not to buy higher port switches? Nov 26, 2018
Networking Looking for suggestions on l2 10g switch Nov 24, 2018
Networking Mellanox Switch (SX6012) QSFP Breakout Compatibility Nov 23, 2018
Networking VOIP Switch Recommendation Nov 21, 2018

Share This Page