Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kapone]

Did you just assume I'm hooman?

Really nice write up! I know what I'm going to be spending my Saturday doing. Just when I think I've figured it out......destroy it all and build it better!

Just for clarification. Do you have device hostnames registered in DNS through DHCP somehow with this setup? So far that's the only thing keeping me from ditching pfSense almost entirely. Although I'm slowly just adding A records in my local DNS and using more and more static IPs. Your setup still looks like what I'm aiming for.

Absolutely. Check these boxes in the DNS Resolver configuration.



And all hostnames will be registered correctly.

 

[ViciousXUSMC]

I kind of get it now, still funky to me to have the WAN directly to the switch. I just have an altered version of this were WAN goes to WAN on PFSense and LAN goes to the Switch. I mean its just 2 ports and for me using gigabit I would prefer not to split that traffic over a single interface via vlans (aka router on a stick)

All the routing happens on the switch and default gateway is the PFSense LAN so all local traffic is on the switch and only WAN traffic goes to PFSense.

For the DHCP issues I was going to either create VLAN interfaces that do not do routing so I can create DHCP scopes or even try to create loop back interfaces in that IP range.

But in the end it's all for fun and experimentation so why not?

 

[kapone]

Well...
- you could run pfSense in a redundant config (with a single provider) in this way, with two boxes...
- "For the DHCP issues I was going to either create VLAN interfaces that do not do routing so I can create DHCP scopes or even try to create loop back interfaces in that IP range." - Hence the problem.

It's not just fun per se, while it IS fun don't get me wrong, the fact that my power consumption went down by ~18w by eliminating a dedicated DNS/DHCP server, AND my WAN latency went down a bit, is sure nice.

 

[PGlover]

Re-did my home network based on what I was thinking... and it actually turned out pretty good. I even removed the standalone DHCP/DNS server and got pfSense to play nice with a layer 3 switch and doing DHCP/DNS without compromising on the routing speed between VLANs.

Also terminated my WAN directly on the switch (on a VLAN, no SVI), so no there is a single 10gb pipe to pfSense.

View attachment 10634

The "performance" of the WAN seems to have improved slightly, although it could just be placebo. (I have symmetric gigabit at home)

View attachment 10633

What is that tool you are using to capture performance?

 

[kapone]

What is that tool you are using to capture performance?

That's the dslreports.com speed test, which measures ping, jitter, bufferbloat etc in addition to speed.

 

[Blue)(Fusion]

@kapone,

I took a deeper dive into the instructions for your set up. On first glance it looked like it would solve my many-VLAN on pfSense issue strictly to serve DHCP. However, it will not solve it, sadly. I will still need a VLAN interface for each VLAN I want DHCP to serve on pfSense, just to serve DHCP.

As far as the transport goes on your setup, my understanding is your internet traffic flows:

client <--> switch <--> pfsense <--> switch <--> modem/ONT

I gather this because your default route on the switch it still pfsense before it hops to the internet (through the switch a second time). If this is the case, this sounds inefficient, basically having the same traffic traverse the switch twice.

If your pfsense box has a gigabit WAN port, eliminating the WAN VLAN, transport VLAN, and plugging the modem/ONT into the WAN port should remove the inefficiency and then you'll have the same setup as I currently do (except I have 6 VLANs that terminate on pfSense).

 

[acbaldwi]

Looking into ways to quite down my 6610, its fine in stage one, its when stage 2 kicks in that i feel the pain it seems that mac 2 is nearly always borderline temps, does anyone know where on the board that is? i would like to maybe direct a fan into that area to cool it off?
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 82.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->84 deg-C
Speed 2: 79<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 61.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 82.0 deg-C
CPU Temperature Readings:
Current temperature : 69.0 deg-C
sensor A Temperature Readings:
Current temperature : 61.0 deg-C
sensor B Temperature Readings:
Current temperature : 64.5 deg-C
sensor C Temperature Readings:
Current temperature : 64.5 deg-C
stacking card Temperature Readings:
Current temperature : 67.0 deg-C
Warning level.......: 84.0 deg-C
Shutdown level......: 87.0 deg-C

 

[kapone]

I took a deeper dive into the instructions for your set up. On first glance it looked like it would solve my many-VLAN on pfSense issue strictly to serve DHCP. However, it will not solve it, sadly. I will still need a VLAN interface for each VLAN I want DHCP to serve on pfSense, just to serve DHCP.

Correct. There is no way to avoid that with the current DHCP server implementation in pfSense. However, in my case that's what I was shooting for, except with no compromises on the L3 routing speed. The gateway for each VLAN is still at the switch, the only traffic for that VLAN (on the tagged VLAN) with pfSense is DHCP and DNS.

As far as the transport goes on your setup, my understanding is your internet traffic flows:

client <--> switch <--> pfsense <--> switch <--> modem/ONT

I gather this because your default route on the switch it still pfsense before it hops to the internet (through the switch a second time). If this is the case, this sounds inefficient, basically having the same traffic traverse the switch twice.

This is no different than having the WAN connected to a dedicated port on pfSense itself. The traffic will still need to hop from one port to another. However, by terminating the WAN at the switch, you get additional flexibility that you don't if it was terminated at pfSense.

So, instead of:

client <--> switch <--> pfsense <--> switch <--> modem/ONT

you'd have

client <--> switch <--> pfsense <--> WAN port <--> modem/ONT

Hence why my point about the transit pipe being appropriately sized/quality.

If your pfsense box has a gigabit WAN port, eliminating the WAN VLAN, transport VLAN, and plugging the modem/ONT into the WAN port should remove the inefficiency and then you'll have the same setup as I currently do (except I have 6 VLANs that terminate on pfSense).

Correct. But like I said, I want to port mirror the WAN port and do additional analysis on it, and I can't do that by terminating it at pfSense.

 

[kapone]

Looking into ways to quite down my 6610, its fine in stage one, its when stage 2 kicks in that i feel the pain it seems that mac 2 is nearly always borderline temps, does anyone know where on the board that is? i would like to maybe direct a fan into that area to cool it off?
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 82.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->84 deg-C
Speed 2: 79<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 61.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 82.0 deg-C
CPU Temperature Readings:
Current temperature : 69.0 deg-C
sensor A Temperature Readings:
Current temperature : 61.0 deg-C
sensor B Temperature Readings:
Current temperature : 64.5 deg-C
sensor C Temperature Readings:
Current temperature : 64.5 deg-C
stacking card Temperature Readings:
Current temperature : 67.0 deg-C
Warning level.......: 84.0 deg-C
Shutdown level......: 87.0 deg-C

That switch is running way way too hot. Are you sure it's in a well ventilated space with airflow?

 

[acbaldwi]

That switch is running way way too hot. Are you sure it's in a well ventilated space with airflow?

im working on cooling the room it's in.... but the switch is in fact the hottest part of then environment.....so to protect it until the cooling is in i was thinking f adding a fan or 2 in there

 

[acbaldwi]

im working on cooling the room it's in.... but the switch is in fact the hottest part of then environment.....so to protect it until the cooling is in i was thinking f adding a fan or 2 in there

unfortunately the fan only came with 1 fan unit in it as you can see im not sure if the second one would actually make that much of a difference or not....

 

[Blue)(Fusion]

This is no different than having the WAN connected to a dedicated port on pfSense itself. The traffic will still need to hop from one port to another. However, by terminating the WAN at the switch, you get additional flexibility that you don't if it was terminated at pfSense.

So, instead of:

client <--> switch <--> pfsense <--> switch <--> modem/ONT

you'd have

client <--> switch <--> pfsense <--> WAN port <--> modem/ONT

Hence why my point about the transit pipe being appropriately sized/quality.


Correct. But like I said, I want to port mirror the WAN port and do additional analysis on it, and I can't do that by terminating it at pfSense.

Ahh, I see. So this isn't for performance, per se. It's for flexability, and that I understand now.

I too have all my gateways set as the switch ve interfaces. I now figured out all of my day-to-day ACLs required to make everything work how I want it (essentially pinholing the firewall). The last one I want to change is instead of the last ACL in a group be permit ip any any to deny....but my understanding is that would block internet-bound packets as well, no?

 

[kapone]

Ahh, I see. So this isn't for performance, per se. It's for flexability, and that I understand now.

Correct. However there is something to said for having a single big pipe to the firewall. Now, I don't need to worry about onboard LAN ports, their quality etc etc. I know I'm just going to throw in a 10g NIC and a single one is more than sufficient. You could do that with a lot of SFF type systems that only have a single expansion slot...

In addition, for e.g., I can now have two separate WAN connections coming into the house, terminate both of them at the switch, add two WAN interfaces to pfSense (over the same transit pipe) and do WAN load balancing and/or redundancy, without adding more ports to the firewall. Or do HA with pfSense with a single or dual WAN connection etc etc.

I too have all my gateways set as the switch ve interfaces. I now figured out all of my day-to-day ACLs required to make everything work how I want it (essentially pinholing the firewall). The last one I want to change is instead of the last ACL in a group be permit ip any any to deny....but my understanding is that would block internet-bound packets as well, no?

That's an interesting scenario. I think if you allow all traffic to the transit gateway as the second to last rule, and then deny all as last, that should work? I actually haven't tried this, I may just do that.

 

[kapone]

im working on cooling the room it's in.... but the switch is in fact the hottest part of then environment.....so to protect it until the cooling is in i was thinking f adding a fan or 2 in there

As long as the switch is not shutting down..you're not hurting it as bad, but it's still running at fried chicken temps..

unfortunately the fan only came with 1 fan unit in it as you can see im not sure if the second one would actually make that much of a difference or not....

It will most certainly help.

Just to give you an idea, my home switch (6610-24P, but the POE isn't active, single PSU, single fan tray.) is in an unfinished part of the basement, no active cooling in that part of the basement, in fact it's closer to the utility/heater in the basement than I'd prefer, and it runs at these temps.

Power supply 1 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  091   
        Firmware Ver:    B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 48.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->78       deg-C
                Speed 2:       73<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 35.5 deg-C
CPU Temperature Readings:
        Current temperature : 38.0 deg-C
sensor A Temperature Readings:                                 
        Current temperature : 23.0 deg-C
sensor B Temperature Readings:
        Current temperature : 29.0 deg-C
sensor C Temperature Readings:
        Current temperature : 16.0 deg-C
sensor D Temperature Readings:
        Current temperature : 15.0 deg-C
stacking card Temperature Readings:
        Current temperature : 48.0 deg-C
        Warning level.......: 84.0 deg-C
        Shutdown level......: 87.0 deg-C

 

[Blue)(Fusion]

Those are very low temps @kapone . Mine is PoE and only powering a single PoE device but I have quite a bit higher temps, but still only first fan speed. My CPU load is 1% and the unfinished basement is rather cold, say around 65F and the rack is not in a corner.

SSH@brocore>show cha
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  BBY     
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  FRW     
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 60.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->76       deg-C
                Speed 2:       71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                         
MAC 1 Temperature Readings:
        Current temperature : 45.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 50.5 deg-C
CPU Temperature Readings:
        Current temperature : 60.5 deg-C
sensor A Temperature Readings:
        Current temperature : 53.5 deg-C
sensor B Temperature Readings:
        Current temperature : 48.5 deg-C
sensor C Temperature Readings:
        Current temperature : 34.5 deg-C
stacking card Temperature Readings:
        Current temperature : 52.5 deg-C
        Warning level.......: 77.0 deg-C
        Shutdown level......: 80.0 deg-C

 

[acbaldwi]

Those are very low temps @kapone . Mine is PoE and only powering a single PoE device but I have quite a bit higher temps, but still only first fan speed. My CPU load is 1% and the unfinished basement is rather cold, say around 65F and the rack is not in a corner.

SSH@brocore>show cha
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  BBY    
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  FRW    
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 60.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->76       deg-C
                Speed 2:       71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                        
MAC 1 Temperature Readings:
        Current temperature : 45.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 50.5 deg-C
CPU Temperature Readings:
        Current temperature : 60.5 deg-C
sensor A Temperature Readings:
        Current temperature : 53.5 deg-C
sensor B Temperature Readings:
        Current temperature : 48.5 deg-C
sensor C Temperature Readings:
        Current temperature : 34.5 deg-C
stacking card Temperature Readings:
        Current temperature : 52.5 deg-C
        Warning level.......: 77.0 deg-C
        Shutdown level......: 80.0 deg-C

I was able to get it here.... by removing the blank plate for the missing fan and the missing power supply and placing a small desk fan blowing into it, i tried pulling but it didnt work as well as blowing....

The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - PoE) present, status ok
Model Number: 23-0000142-02
Serial Number: CM6
Firmware Ver: C
Power supply 2 Fan Air Flow Direction: Front to Back

Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 71.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->84 deg-C
Speed 2: 79<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 51.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 71.0 deg-C
CPU Temperature Readings:
Current temperature : 40.5 deg-C
sensor A Temperature Readings:
Current temperature : 54.5 deg-C
sensor B Temperature Readings:
Current temperature : 40.0 deg-C
sensor C Temperature Readings:
Current temperature : 58.0 deg-C
stacking card Temperature Readings:
Current temperature : 58.0 deg-C
Warning level.......: 84.0 deg-C
Shutdown level......: 87.0 deg-C

 

[Ouraing]

So the thought of paying $55 for a RMK + $10 more in shipping on eBay didn't sit well with me and today I made these (not 100% finished here).

I made it from 1/8" steel flat stock, which was way more material than I actually needed for just 1 switch.

Material:

• 1/8" x 2" x 4' flat steel piece ($12)
  
• Pack of 12 M4x.70 x 12 flat philips machine screws ($2)
  
• Flat Black paint

The tools used were

• 5" Vise
• 4lb sledge hammer
• 3/8" drill bit
• 3/8" counter-sink
• Hacksaw
• Angle Grinder
• Torch
• Center Punch

I made a template with card stock and used that to transfer over the proper length and hole locations. I cut the pieces to length with the hack saw (could have also used the angle grinder I guess) then clamped them in the vise to cut off the excess width. After that I marked where the bend needed to be and placed the piece in the vise, heated it with the torch and used the sledge hammer to bend it over. I then threw it in a bucket of water to cool off and center punched the mounting holes and drilled them out. After test fitting to make sure the holes were properly placed, I counter sunk the screw holes. I cleaned the pieces off with some degreaser and painted them with some flat black paint so they won't rust.

Since I already had all the tools and the paint, my total investment was only $14. It took me about 90 minutes to get it all done, plus a few hours for the paint to dry completely.

 

[Callan05]

I'm late to the conversion here on pfsense and vlans, but wanted to add quickly:

I use pfsense to route between my vlans, mostly because that's where I want to control the security.
I have a DMZ with a reverse proxy.
So incoming internet facing traffic is:
Wan, Pfsense DMZ-vlan interface, switch, Hyper-V, reverse proxy VM, switch, pfsense (in dmz-vlan out lan-vlan), switch, Hyper-V, web server VM.

Could I do routing in my switch? Sure, and it's likely faster, but I prefer to use a firewall to do at least some segregation of the traffic. (Not that vlans are bullet proof by any means. The traffic isn't that important, I just want to keep my windows box off the internet)

My dmz vlan has almost no access to my lan.

If your vlans are all trusted, then sure use the switch, but pfsense and other firewalls can still play a part for certain use cases.

 

[arglebargle]

Can anyone think of a reason only one of my 6450s would be disallowing remote management access without ACLs? Both have an identical (very simple) configuration (basically just a static IP on ve1 plus dns and ntp servers) but one refuses to allow ssh or https connections without explicitly setting allowed IP ranges in an ACL.

Here's the config from the picky switch: hastebin -- the other is identical minus ACLs, hostname and static IP.

 

[Zervun]

Can anyone think of a reason only one of my 6450s would be disallowing remote management access without ACLs? Both have an identical (very simple) configuration (basically just a static IP on ve1 plus dns and ntp servers) but one refuses to allow ssh or https connections without explicitly setting allowed IP ranges in an ACL.

Here's the config from the picky switch: hastebin -- the other is identical minus ACLs, hostname and static IP.

That looks the same as mine (minus the ACLs and the SSH access group) - eyeballing it I would think it would work as long as you are coming from a 192.168.17.0/24 address (I do all my ACLs on my untangle firewall)