Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    Absolutely. Check these boxes in the DNS Resolver configuration.

    Screen Shot 2019-03-08 at 8.25.51 PM.png

    And all hostnames will be registered correctly.

    Screen Shot 2019-03-08 at 8.28.04 PM.png
     
    #1461
  2. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    I kind of get it now, still funky to me to have the WAN directly to the switch. I just have an altered version of this were WAN goes to WAN on PFSense and LAN goes to the Switch. I mean its just 2 ports and for me using gigabit I would prefer not to split that traffic over a single interface via vlans (aka router on a stick)

    All the routing happens on the switch and default gateway is the PFSense LAN so all local traffic is on the switch and only WAN traffic goes to PFSense.

    For the DHCP issues I was going to either create VLAN interfaces that do not do routing so I can create DHCP scopes or even try to create loop back interfaces in that IP range.

    But in the end it's all for fun and experimentation so why not?
     
    #1462
  3. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    Well...
    - you could run pfSense in a redundant config (with a single provider) in this way, with two boxes...
    - "For the DHCP issues I was going to either create VLAN interfaces that do not do routing so I can create DHCP scopes or even try to create loop back interfaces in that IP range." - Hence the problem.

    It's not just fun per se, while it IS fun :) don't get me wrong, the fact that my power consumption went down by ~18w by eliminating a dedicated DNS/DHCP server, AND my WAN latency went down a bit, is sure nice.
     
    #1463
    arglebargle likes this.
  4. PGlover

    PGlover Active Member

    Joined:
    Nov 8, 2014
    Messages:
    439
    Likes Received:
    48
    What is that tool you are using to capture performance?
     
    #1464
  5. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    That's the dslreports.com speed test, which measures ping, jitter, bufferbloat etc in addition to speed.
     
    #1465
  6. Blue)(Fusion

    Blue)(Fusion Member

    Joined:
    Mar 1, 2017
    Messages:
    86
    Likes Received:
    14
    @kapone,

    I took a deeper dive into the instructions for your set up. On first glance it looked like it would solve my many-VLAN on pfSense issue strictly to serve DHCP. However, it will not solve it, sadly. I will still need a VLAN interface for each VLAN I want DHCP to serve on pfSense, just to serve DHCP.

    As far as the transport goes on your setup, my understanding is your internet traffic flows:

    client <--> switch <--> pfsense <--> switch <--> modem/ONT

    I gather this because your default route on the switch it still pfsense before it hops to the internet (through the switch a second time). If this is the case, this sounds inefficient, basically having the same traffic traverse the switch twice.

    If your pfsense box has a gigabit WAN port, eliminating the WAN VLAN, transport VLAN, and plugging the modem/ONT into the WAN port should remove the inefficiency and then you'll have the same setup as I currently do (except I have 6 VLANs that terminate on pfSense).
     
    #1466
  7. acbaldwi

    acbaldwi New Member

    Joined:
    Feb 15, 2019
    Messages:
    7
    Likes Received:
    0
    Looking into ways to quite down my 6610, its fine in stage one, its when stage 2 kicks in that i feel the pain :) it seems that mac 2 is nearly always borderline temps, does anyone know where on the board that is? i would like to maybe direct a fan into that area to cool it off?
    Fan 1 ok, speed (auto): [[1]]<->2
    Fan 2 not present

    Fan controlled temperature: 82.0 deg-C

    Fan speed switching temperature thresholds:
    Speed 1: NM<----->84 deg-C
    Speed 2: 79<-----> 87 deg-C (shutdown)

    Fan 1 Air Flow Direction: Front to Back
    MAC 1 Temperature Readings:
    Current temperature : 61.0 deg-C
    MAC 2 Temperature Readings:
    Current temperature : 82.0 deg-C
    CPU Temperature Readings:
    Current temperature : 69.0 deg-C
    sensor A Temperature Readings:
    Current temperature : 61.0 deg-C
    sensor B Temperature Readings:
    Current temperature : 64.5 deg-C
    sensor C Temperature Readings:
    Current temperature : 64.5 deg-C
    stacking card Temperature Readings:
    Current temperature : 67.0 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C
     
    #1467
  8. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    Correct. There is no way to avoid that with the current DHCP server implementation in pfSense. However, in my case that's what I was shooting for, except with no compromises on the L3 routing speed. The gateway for each VLAN is still at the switch, the only traffic for that VLAN (on the tagged VLAN) with pfSense is DHCP and DNS.

    This is no different than having the WAN connected to a dedicated port on pfSense itself. The traffic will still need to hop from one port to another. However, by terminating the WAN at the switch, you get additional flexibility that you don't if it was terminated at pfSense.

    So, instead of:

    client <--> switch <--> pfsense <--> switch <--> modem/ONT

    you'd have

    client <--> switch <--> pfsense <--> WAN port <--> modem/ONT

    Hence why my point about the transit pipe being appropriately sized/quality.

    Correct. But like I said, I want to port mirror the WAN port and do additional analysis on it, and I can't do that by terminating it at pfSense.
     
    #1468
  9. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    That switch is running way way too hot. Are you sure it's in a well ventilated space with airflow?
     
    #1469
  10. acbaldwi

    acbaldwi New Member

    Joined:
    Feb 15, 2019
    Messages:
    7
    Likes Received:
    0
    im working on cooling the room it's in.... but the switch is in fact the hottest part of then environment.....so to protect it until the cooling is in i was thinking f adding a fan or 2 in there
     
    #1470
  11. acbaldwi

    acbaldwi New Member

    Joined:
    Feb 15, 2019
    Messages:
    7
    Likes Received:
    0
    unfortunately the fan only came with 1 fan unit in it as you can see im not sure if the second one would actually make that much of a difference or not....
     
    #1471
  12. Blue)(Fusion

    Blue)(Fusion Member

    Joined:
    Mar 1, 2017
    Messages:
    86
    Likes Received:
    14
    Ahh, I see. So this isn't for performance, per se. It's for flexability, and that I understand now.

    I too have all my gateways set as the switch ve interfaces. I now figured out all of my day-to-day ACLs required to make everything work how I want it (essentially pinholing the firewall). The last one I want to change is instead of the last ACL in a group be permit ip any any to deny....but my understanding is that would block internet-bound packets as well, no?
     
    #1472
  13. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    Correct. However there is something to said for having a single big pipe to the firewall. Now, I don't need to worry about onboard LAN ports, their quality etc etc. I know I'm just going to throw in a 10g NIC and a single one is more than sufficient. You could do that with a lot of SFF type systems that only have a single expansion slot... ;)

    In addition, for e.g., I can now have two separate WAN connections coming into the house, terminate both of them at the switch, add two WAN interfaces to pfSense (over the same transit pipe) and do WAN load balancing and/or redundancy, without adding more ports to the firewall. Or do HA with pfSense with a single or dual WAN connection etc etc.

    That's an interesting scenario. I think if you allow all traffic to the transit gateway as the second to last rule, and then deny all as last, that should work? I actually haven't tried this, I may just do that.
     
    #1473
  14. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    618
    Likes Received:
    247
    As long as the switch is not shutting down..you're not hurting it as bad, but it's still running at fried chicken temps..
    It will most certainly help.

    Just to give you an idea, my home switch (6610-24P, but the POE isn't active, single PSU, single fan tray.) is in an unfinished part of the basement, no active cooling in that part of the basement, in fact it's closer to the utility/heater in the basement than I'd prefer, and it runs at these temps.

    Code:
    Power supply 1 (AC - Regular) present, status ok
            Model Number:   23-0000144-01
            Serial Number:  091   
            Firmware Ver:    B
    Power supply 1 Fan Air Flow Direction:  Front to Back
    Power supply 2 not present
    
    Fan 1 ok, speed (auto): [[1]]<->2
    Fan 2 not present
    
    Fan controlled temperature: 48.0 deg-C
    
    Fan speed switching temperature thresholds:
                    Speed 1: NM<----->78       deg-C
                    Speed 2:       73<-----> 87 deg-C (shutdown)
    
    Fan 1 Air Flow Direction:  Front to Back
    MAC 1 Temperature Readings:
            Current temperature : 35.5 deg-C
    CPU Temperature Readings:
            Current temperature : 38.0 deg-C
    sensor A Temperature Readings:                                 
            Current temperature : 23.0 deg-C
    sensor B Temperature Readings:
            Current temperature : 29.0 deg-C
    sensor C Temperature Readings:
            Current temperature : 16.0 deg-C
    sensor D Temperature Readings:
            Current temperature : 15.0 deg-C
    stacking card Temperature Readings:
            Current temperature : 48.0 deg-C
            Warning level.......: 84.0 deg-C
            Shutdown level......: 87.0 deg-C
    
     
    #1474
  15. Blue)(Fusion

    Blue)(Fusion Member

    Joined:
    Mar 1, 2017
    Messages:
    86
    Likes Received:
    14
    Those are very low temps @kapone . Mine is PoE and only powering a single PoE device but I have quite a bit higher temps, but still only first fan speed. My CPU load is 1% and the unfinished basement is rather cold, say around 65F and the rack is not in a corner.

    Code:
    SSH@brocore>show cha
    The stack unit 1 chassis info:
    
    Power supply 1 (AC - PoE) present, status ok
            Model Number:   23-0000142-02
            Serial Number:  BBY     
            Firmware Ver:    A
    Power supply 1 Fan Air Flow Direction:  Front to Back
    Power supply 2 (AC - PoE) present, status ok
            Model Number:   23-0000142-02
            Serial Number:  FRW     
            Firmware Ver:    A
    Power supply 2 Fan Air Flow Direction:  Front to Back
    
    Fan 1 ok, speed (auto): [[1]]<->2
    Fan 2 ok, speed (auto): [[1]]<->2
    
    Fan controlled temperature: 60.5 deg-C
    
    Fan speed switching temperature thresholds:
                    Speed 1: NM<----->76       deg-C
                    Speed 2:       71<-----> 80 deg-C (shutdown)
    
    Fan 1 Air Flow Direction:  Front to Back
    Fan 2 Air Flow Direction:  Front to Back                         
    MAC 1 Temperature Readings:
            Current temperature : 45.5 deg-C
    MAC 2 Temperature Readings:
            Current temperature : 50.5 deg-C
    CPU Temperature Readings:
            Current temperature : 60.5 deg-C
    sensor A Temperature Readings:
            Current temperature : 53.5 deg-C
    sensor B Temperature Readings:
            Current temperature : 48.5 deg-C
    sensor C Temperature Readings:
            Current temperature : 34.5 deg-C
    stacking card Temperature Readings:
            Current temperature : 52.5 deg-C
            Warning level.......: 77.0 deg-C
            Shutdown level......: 80.0 deg-C
     
    #1475
  16. acbaldwi

    acbaldwi New Member

    Joined:
    Feb 15, 2019
    Messages:
    7
    Likes Received:
    0

    I was able to get it here.... by removing the blank plate for the missing fan and the missing power supply and placing a small desk fan blowing into it, i tried pulling but it didnt work as well as blowing....

    The stack unit 1 chassis info:

    Power supply 1 not present
    Power supply 2 (AC - PoE) present, status ok
    Model Number: 23-0000142-02
    Serial Number: CM6
    Firmware Ver: C
    Power supply 2 Fan Air Flow Direction: Front to Back

    Fan 1 not present
    Fan 2 ok, speed (auto): [[1]]<->2

    Fan controlled temperature: 71.0 deg-C

    Fan speed switching temperature thresholds:
    Speed 1: NM<----->84 deg-C
    Speed 2: 79<-----> 87 deg-C (shutdown)

    Fan 2 Air Flow Direction: Front to Back
    MAC 1 Temperature Readings:
    Current temperature : 51.0 deg-C
    MAC 2 Temperature Readings:
    Current temperature : 71.0 deg-C
    CPU Temperature Readings:
    Current temperature : 40.5 deg-C
    sensor A Temperature Readings:
    Current temperature : 54.5 deg-C
    sensor B Temperature Readings:
    Current temperature : 40.0 deg-C
    sensor C Temperature Readings:
    Current temperature : 58.0 deg-C
    stacking card Temperature Readings:
    Current temperature : 58.0 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C
     
    #1476
  17. Ouraing

    Ouraing New Member

    Joined:
    Dec 31, 2018
    Messages:
    25
    Likes Received:
    21
    So the thought of paying $55 for a RMK + $10 more in shipping on eBay didn't sit well with me and today I made these (not 100% finished here).

    [​IMG]

    I made it from 1/8" steel flat stock, which was way more material than I actually needed for just 1 switch.

    [​IMG]

    Material:
    • 1/8" x 2" x 4' flat steel piece ($12)
    • Pack of 12 M4x.70 x 12 flat philips machine screws ($2)
    • Flat Black paint
    The tools used were
    • 5" Vise
    • 4lb sledge hammer
    • 3/8" drill bit
    • 3/8" counter-sink
    • Hacksaw
    • Angle Grinder
    • Torch
    • Center Punch
    I made a template with card stock and used that to transfer over the proper length and hole locations. I cut the pieces to length with the hack saw (could have also used the angle grinder I guess) then clamped them in the vise to cut off the excess width. After that I marked where the bend needed to be and placed the piece in the vise, heated it with the torch and used the sledge hammer to bend it over. I then threw it in a bucket of water to cool off and center punched the mounting holes and drilled them out. After test fitting to make sure the holes were properly placed, I counter sunk the screw holes. I cleaned the pieces off with some degreaser and painted them with some flat black paint so they won't rust.

    Since I already had all the tools and the paint, my total investment was only $14. It took me about 90 minutes to get it all done, plus a few hours for the paint to dry completely.
     
    #1477
    mathiastro and itronin like this.
  18. Callan05

    Callan05 New Member

    Joined:
    Nov 8, 2018
    Messages:
    12
    Likes Received:
    5
    I'm late to the conversion here on pfsense and vlans, but wanted to add quickly:

    I use pfsense to route between my vlans, mostly because that's where I want to control the security.
    I have a DMZ with a reverse proxy.
    So incoming internet facing traffic is:
    Wan, Pfsense DMZ-vlan interface, switch, Hyper-V, reverse proxy VM, switch, pfsense (in dmz-vlan out lan-vlan), switch, Hyper-V, web server VM.

    Could I do routing in my switch? Sure, and it's likely faster, but I prefer to use a firewall to do at least some segregation of the traffic. (Not that vlans are bullet proof by any means. The traffic isn't that important, I just want to keep my windows box off the internet)

    My dmz vlan has almost no access to my lan.

    If your vlans are all trusted, then sure use the switch, but pfsense and other firewalls can still play a part for certain use cases.
     
    #1478
    mathiastro likes this.
  19. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    634
    Likes Received:
    209
    Can anyone think of a reason only one of my 6450s would be disallowing remote management access without ACLs? Both have an identical (very simple) configuration (basically just a static IP on ve1 plus dns and ntp servers) but one refuses to allow ssh or https connections without explicitly setting allowed IP ranges in an ACL.

    Here's the config from the picky switch: hastebin -- the other is identical minus ACLs, hostname and static IP.
     
    #1479
  20. Zervun

    Zervun Member

    Joined:
    Feb 2, 2019
    Messages:
    43
    Likes Received:
    6
    That looks the same as mine (minus the ACLs and the SSH access group) - eyeballing it I would think it would work as long as you are coming from a 192.168.17.0/24 address (I do all my ACLs on my untangle firewall)
     
    #1480
Similar Threads: Brocade Series
Forum Title Date
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019

Share This Page