Recommend: 10GbE, L2+/L3 switch. VLAN ACLs

Discussion in 'Networking' started by StammesOpfer, Apr 6, 2018.

  1. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    337
    Likes Received:
    100
    Ok so I am looking for the wisdom of the group here for a switch model recommendation. Does one that fits my desires exist?

    My needs:
    10GbE SFP+ (other standard 10GbE Fiber may work)
    L2+ routing
    VLAN ACLs
    reasonable power draw (not 100w+)
    not $500+ (prefer under $300)

    Nice to have:
    Cisco (Since I am most comfortable with that)
    POE (power draw obviously higher but that is fine)
    4+ SFP+ ports
    Full L3

    I currently have a stack of three Dell 5524 switches which are capable of static routes and that works fine. However they only allow ACLs to be applied to the physical interfaces not the VLAN/SVI. This doesn't work for me. I am running router on a stick with pfSense right now but I would like to do my inter VLAN routing on the switch.

    Thanks for any suggestions.
     
    #1
  2. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    241
    Likes Received:
    146
    I could have named a bajillion switches right up until the "not draw 100w" part :p

    The closest I can come up with offhand is the brocade FCX with the FCX-4XG card which gets you 4x SFP+. They come in POE and non-POE. They draw around 50 to 60w with no PoE ports enabled. warning: the PoE model of the FCX is one of the loudest pieces of gear I've ever owned -the huge 1200w power supply fans are not software controlled so they run at 100%

    ready to go with 10gbe card for $180 (no PoE, you can find one with PoE for probably $50 more) Brocade FCX 648-I 48-Port Gigabit Ethernet Managed L3 Switch w/ FCX-4XG 2 x PSU 729198041281 | eBay
    (If you buy the chassis and card separately, do not buy the fcx models ending in S - those will not take the 4x sfp card)

    There's also the ICX 6610, which is the successor to the FCX - it has 8x 10gbe ports, 4x 40gbe ports on the back (which can only be used for stacking), and the PoE version is MUCH quieter (the non-poe versions are about equal, not very loud). The downside is it's more expensive (generally around $300), and draws more power (around 90w).

    example: (non POE, even though it's listed as the "P" PoE model, it's not as you can see from the missing P in the name on the front, gotta watch out for this on ebay) Brocade ICX6610-48-PE ICX 6610 48-Port Gigabit 8-Port SFP Dual Power AC Switch | eBay

    both run the same FastIron OS, which is like 80% identical to cisco so you'd probably feel at home. They're both still under active support as well, their most recent fw release was 2 days ago. You can do anything needed with ACL's, per vlan, per port, both at the same time, route ACL-blocked packets off to a mirror port or sflow collector for analyzation, etc. They're full blown routers with ipv4 and ipv6 bgp, ospf, etc. the ICX also supports SDN with openflow, including hybrid port mode

    I'm sure there's also a little microtik or something that meets the same criteria with less power draw, but I'm not familiar with them so someone else will have to chime in

    before someone tells me you need a license to use the 10gbe ports on the ICX: nahhh fammm
     
    #2
    Last edited: Apr 6, 2018
    jkjk and StammesOpfer like this.
  3. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    337
    Likes Received:
    100
    Yeah if it wasn't for the power requirement it would be easy.
    Thanks I'll have a look at those....
    I don't need silent or even quite. Just not full on jet fighter.
    I have the CRS226 Microtik and that thing is terrible to try and do anything on so I will never touch Microtik again if I can help it.
    I have touched Brocade before and can work my way through the differences.
     
    #3
    fohdeesha likes this.
  4. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    337
    Likes Received:
    100
    Ok so this ICX looks extremely interesting to me. A little help here if you don't mind otherwise I will start diggin the best I can tomorrow.

    You say it should have a "P" on the front if it truly is a POE model? Looking at this listing it shows the label as a PE but front panel doesn't have a P. It seems like the front panel doesn't change between models at all. Am I missing something?

    Then from the listing you posted "Uplink Ports : 8 x Gigabit Ethernet SFP slots (upgradable to 10 GbE)" is this what you are talking about with licensing? And should I take it to mean licensing is not enforced, works anyway, or I need to visit some of the darker bits of the internet?

    8x SPF+ and POE would mean I don't need any other switches which would mean my power budget is a little more flexible.
     
    #4
  5. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    1,990
    Likes Received:
    283
    Not sure on the power draw and I know it will be on the higher side...

    Cisco 4948E , has 4 x SFP+ ports, still in support and can be had used on eBay less than $300
     
    #5
    imafreak likes this.
  6. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    241
    Likes Received:
    146
    They're an incredible deal at $300. Hell, the only reason the 40gbe ports on the back are restricted to stacking only is because the marketing department forced them to, after they realized they accidentally built a 1Ru switch that would do full l3 SDN at 500gbps+, which was more aggregate bandwidth than their top-end MLXe 4 slot chassis router at the time. Stupid market segmentation!

    They show up as fully configurable ports, 2 of them even support 4x 10gbe breakout, but there's a software flag somewhere stopping them from being used in non-stack config. I've dumped the entire running OS out of ram via jtag, and I've been slowly decompiling the assembly line by line, until I find the flag that's doing this and I can remove it, unlocking the 40gbe ports for normal use. It's a massive pain in the ass and might take months if I ever even finish, so pretend I didn't even mention this

    Correct. As you've noticed, a lot of sellers have listed non-P chassis as PoE, which drives me nuts. Especially when they even say upfront it has the 250w power supplies - those do not support PoE either. For PoE look for a P on the front, like this listing: Brocade ICX6610-48P 48-Port Gigabit PoE+ 8x 1GbE SFPP 4x 40GbE 1x PSU OUTWARD | eBay

    It means send me a PM before you purchase and I'll make your dreams come true
     
    #6
    Last edited: Apr 9, 2018
    StammesOpfer likes this.
  7. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    337
    Likes Received:
    100
    Looked at that but 230watts. For Cisco I think I would be better off with a 3560E and X2 10Gbe modules.

    This ICX looks pretty damn good though.
     
    #7
  8. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    1,990
    Likes Received:
    283
    Ouch and the 93108’s that using to replace them average 290w each !
    Switching power consumption is crazy except for a few :-/
     
    #8
    rchunter likes this.
  9. littleredwagen

    littleredwagen New Member

    Joined:
    Dec 8, 2016
    Messages:
    11
    Likes Received:
    2
    This thread is great. I use brocade ICX and VDX fabric switches at work love them. I am interested now in getting one of these in my home lab
     
    #9
  10. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    241
    Likes Received:
    146
    I don't want to get anyone's hopes up, but it seems I've unlocked all the 40gbe ports on the rear for general use. I am waiting on a very kind STH member to ship me his switch for further testing before I publish the procedure (a very clean procedure I must say)

    On the back there's 2x 40gbe-only ports and 2x 40gbe breakout (4x 10gbe each) ports. So including the front ports, thats 48x copper, 16x 10gbe, and 2x 40gbe with full L3 including BGP/VRFs/IPV6/etc for $350, and still under active software development to boot. Will keep you posted
     
    #10
    MellowTone likes this.
  11. mpennett

    mpennett New Member

    Joined:
    Apr 11, 2017
    Messages:
    12
    Likes Received:
    0
    Interesting indeed, I might have to consider the non-POE version. I do however have to be concerned about noise levels. The datasheet states 48.7 dB, would you know if that is the ceiling under full load or the constant?
     
    #11
  12. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    241
    Likes Received:
    146
    I would imagine that's peak, it's definitely quieter than my FCX, and that's not bad to begin with. Even if not, I've never heard the FCX or ICX scale fan speed under load. "full load" on these ASICs is 500gbps+ of l3 routing and I've never gotten past a few percent of that
     
    #12
  13. littleredwagen

    littleredwagen New Member

    Joined:
    Dec 8, 2016
    Messages:
    11
    Likes Received:
    2
    Our ICX-6450-48p are really loud at fire up. then quiet down, and the non P version is even quieter That is definitely peak, and I have never heard one ramp up to full speed
     
    #13
    fohdeesha likes this.
  14. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    337
    Likes Received:
    100
    Well I am officially convinced. I snagged a ICX6610-48P with 1 damaged port for under $200. I am very happy with that deal. Now just have to wait for it to arrive.

    Thanks for all the recommendations and @fohdeesha your work on these things is amazing.
     
    #14
    fohdeesha likes this.
  15. mpennett

    mpennett New Member

    Joined:
    Apr 11, 2017
    Messages:
    12
    Likes Received:
    0
    I picked up a non-PoE one as I only have 1 PoE device and prefer using the injector for it anyway. Don't need the potential extra noise either. Currently maxed out on 10Gbe ports so this will provide the extra I needed as well. To echo @StammesOpfer I thank you all as well.
     
    #15
  16. jmck

    jmck Member

    Joined:
    Apr 4, 2013
    Messages:
    77
    Likes Received:
    14
    Guess you convinced me as well, just picked up a 48P-PI. Many thanks @fohdeesha
     
    #16
  17. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    241
    Likes Received:
    146
    You guys are exhausting the ebay supply! Ignore what I said above, they are garbage switches

    also, the 10gbe ports will not work without a license. (hence the "PM me" message earlier)
     
    #17
  18. Jerry Renwick

    Jerry Renwick Active Member

    Joined:
    Aug 7, 2014
    Messages:
    194
    Likes Received:
    36
    The CRS226-24G-2S+IN is able to power via PoE. With it, one can deploy to nooks and crannies in homes and offices. It can also power the unit through a single Ethernet cable.
     
    #18
  19. marv

    marv Member

    Joined:
    Apr 2, 2015
    Messages:
    77
    Likes Received:
    16
    Do these ICX switches support RDMA? priority flow-control and data center bridging protocols. I didnt find it in datasheet so I assume no?
     
    #19
  20. fohdeesha

    fohdeesha Active Member

    Joined:
    Nov 20, 2016
    Messages:
    241
    Likes Received:
    146
    nope that's all in their datacenter line (VDX series)

    I should know in the middle of next week whether the stacking ports can be fully unlocked
     
    #20
    StammesOpfer likes this.

Share This Page