Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

klui

Well-Known Member
Feb 3, 2019
824
453
63
"With a layer 3 switch, the general recommendation is to either let the switch do DHCP duties (it can), or have a dedicated DHCP server that can handle VLANs (pfSense cannot, at this time...)."

My head is spinning from all the reading I've done the last few days; but am I nuts - doesn't pfSense current version 2.4.4 offer DHCP Server built in ?
In order for a switch to perform L3 duties it has to have an interface for your subnet that acts as a gateway. Then the switch can perform ACLs and perform routing to other subnets. Yeah, pfSense has DHCP functions but most how-tos for the configuration will set the pfSense interface as the gateway and you need to either change that or define a DHCP server on an L3 switch and use that and add an additional route to pfSense for your upstream gateway.
 
  • Like
Reactions: tommybackeast

tommybackeast

Active Member
Jun 10, 2018
286
105
43
In order for a switch to perform L3 duties it has to have an interface for your subnet that acts as a gateway. Then the switch can perform ACLs and perform routing to other subnets. Yeah, pfSense has DHCP functions but most how-tos for the configuration will set the pfSense interface as the gateway and you need to either change that or define a DHCP server on an L3 switch and use that and add an additional route to pfSense for your upstream gateway.
first thank you for replying. Given your language, you have knowledge and experience.

Bluntly, my own lack of knowledge prevents me from deeply understanding your above comments. (sorry)

I ask my question as a newbie to all of this who is truly trying to learn but also maintain good security on the home network (but lacks knowledge when stepping away from the asus router/AP and 192.168.1.0/24 I have used for a long time.

If I am understanding you correctly, current version pfSense does have DHCP Server but does it in a poor manner? is that accurate?

How much "harder" (for a newbie) is letting the pfsense router just be a router with some packages like suricata , pfblockerng -and- DHCP Server, VLANS + L3 being done on the Brocade 7250?

I suspect you do everything CLI on the brocade but might you know how 'easy' the brocade WebGUI is to setup VLANS + L3 ?
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
In order for a switch to perform L3 duties it has to have an interface for your subnet that acts as a gateway. Then the switch can perform ACLs and perform routing to other subnets. Yeah, pfSense has DHCP functions but most how-tos for the configuration will set the pfSense interface as the gateway and you need to either change that or define a DHCP server on an L3 switch and use that and add an additional route to pfSense for your upstream gateway.
Allow me to ask a sideways question: I now will have a 10GBe Network : Brocade 7250, 2 Synology NAS with 10GBe NIC, Dell Server / ESXi with 10GB ; and two PC computers with 10GB NICs.

Pfsense router has two 1GB NICs

Nothing is really set up yet. If I run pfSense with DHCP Server, VLANs and L3 switching : does this mean all the LAN traffic will pass through the pfsense 1GB NICs?

Say I move a 5GB file from Synology NAS (on STORAGE-VLAN) to PC Computer [on LAN-VLAN] (10GB in both boxes). Will that 5GB file LAN transfer pass through the 1GB NIC in the pfsense router? (thus making have 10GB NICs useless) ? (note above two VLAN names are just made up for the example, still in process of trying to figure out how to start this process)
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
Allow me to ask a sideways question: I now will have a 10GBe Network : Brocade 7250, 2 Synology NAS with 10GBe NIC, Dell Server / ESXi with 10GB ; and two PC computers with 10GB NICs.

Pfsense router has two 1GB NICs

Nothing is really set up yet. If I run pfSense with DHCP Server, VLANs and L3 switching : does this mean all the LAN traffic will pass through the pfsense 1GB NICs?
Yes, that is one of the reasons why you want to use an L3 switch. Because typically your DHCP server will have a scope and the gateway will point to the interface where the DHCP server is listening to. The other benefit of using an L3 switch like the ones discussed in this thread is rules are performed in hardware so you will get line rate (1G, 10G, 40G) performance. Under pfSense installed in white-box hardware, your firewall rules will most likely be done in software so you will need a robust CPU to achieve line rate.

Having your pfSense firewall as the gateway won't make your 10G NICs useless. But they will need to have access to the pfSense interface if the destination isn't already known by the switch. If it is known by the switch then those transactions will be routed properly; however, if you have FW rules then your packets must go back to pfSense first.

pfSense doesn't have a bad implementation but you need to look at what HW you have and what kind of performance you require.
 
  • Like
Reactions: tommybackeast

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Wanted to post an update on my infinite quest to silence an ICX 7250-48P (I run this thing in a fairly warm closet with bad circulation). Call me obsessive, but I wasn't a fan (no pun intended) of my temps creeping into the low 70s after my previous fan swap. I believe I have come up with the perfect silent solution for this rig.

Replace all 3 rear fans with the Mechatronics MR4020X12B1-RSR fan. They will read just fine in CLI. Then add a Sunon MF60101V3-1000U-A99 fan to the ASIC heat sink (you'll still have around 1/3" to spare in the case, plenty). Wire it in with Fan 3. All the fans are reduced load vs stock so they won't draw too much and the Sunon only has power and ground, so doesn't affect the read on Fan 3.

The switch now runs next-to-dead silent; temperatures on the ASIC dropped ~20 degrees over my previous configuration into the low to mid 50s degrees and the PSU is in the upper 20s which is great. Been running around a week, everything has been perfect. I'm in love.
Could you post some pics?

Also, where did you source these from? I found the Mechatronics only at digikey and the Sunon at Mouser - not both at one. Then again, I do have some parts I have to get some Mouser anyways... :)
 

Fallen Kell

Member
Mar 10, 2020
57
23
8
Sorry for asking something that might have already been answered. I just picked up a 6610-24 (non-PoE). I have seen others asking about quieting the 6610 before and it was stated not to try. Now I can understand that for the PoE versions (which have ~1000W power supplies) but I find it surprising that a switch which nominally draws 120W can not be cooled quietly. Are there really no methods to quiet this switch with modifications?

It is by far the loudest item in my rack (even over my supermicro cse-846).
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Sorry for asking something that might have already been answered. I just picked up a 6610-24 (non-PoE). I have seen others asking about quieting the 6610 before and it was stated not to try. Now I can understand that for the PoE versions (which have ~1000W power supplies) but I find it surprising that a switch which nominally draws 120W can not be cooled quietly. Are there really no methods to quiet this switch with modifications?

It is by far the loudest item in my rack (even over my supermicro cse-846).
no, and it shouldn't be that loud, I can't hear mine over an R720. do the fan spins way down after boot, and what revision are the power supplies?
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
no, and it shouldn't be that loud, I can't hear mine over an R720. do the fan spins way down after boot, and what revision are the power supplies?
The loudest thing in my XrackPro2 enclosed cabinet are the rear XrackPro2 fans themselves lol.

I"m not home; and forget if its 3 80mm or 120mm fans, but can anyone suggest a Mfg of DC fans that are quiet but still move air?
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
Yes, that is one of the reasons why you want to use an L3 switch. Because typically your DHCP server will have a scope and the gateway will point to the interface where the DHCP server is listening to. The other benefit of using an L3 switch like the ones discussed in this thread is rules are performed in hardware so you will get line rate (1G, 10G, 40G) performance. Under pfSense installed in white-box hardware, your firewall rules will most likely be done in software so you will need a robust CPU to achieve line rate.

Having your pfSense firewall as the gateway won't make your 10G NICs useless. But they will need to have access to the pfSense interface if the destination isn't already known by the switch. If it is known by the switch then those transactions will be routed properly; however, if you have FW rules then your packets must go back to pfSense first.

pfSense doesn't have a bad implementation but you need to look at what HW you have and what kind of performance you require.
Thank you for explaining : I got a bit confused from reading /r/homelab; where lots of guys mix PROD and LAB; and got used to seeing them talk about a large of VLANs.

so I know understand why (for me, a newbie) keep all the 10GB devices in the same, primary VLAN as my two Workstation computers.

until 15min ago, I was planning on DHCP server on Brocade but just read foodeesha doesn't really suggest doing that, given the understandable comment that Brocade's real enterprise clients are doing DHCP Server on a different dedicated box; so their implementation is not perfect. (of course, I have just spent an hour reading the Brocade manual on DHCP Server before visiting STH.com, lol

Question: is your own DHCP Server on your Brocade? any problems?
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
Wanted to post an update on my infinite quest to silence an ICX 7250-48P (I run this thing in a fairly warm closet with bad circulation). Call me obsessive, but I wasn't a fan (no pun intended) of my temps creeping into the low 70s after my previous fan swap. I believe I have come up with the perfect silent solution for this rig.

Replace all 3 rear fans with the Mechatronics MR4020X12B1-RSR fan. They will read just fine in CLI. Then add a Sunon MF60101V3-1000U-A99 fan to the ASIC heat sink (you'll still have around 1/3" to spare in the case, plenty). Wire it in with Fan 3. All the fans are reduced load vs stock so they won't draw too much and the Sunon only has power and ground, so doesn't affect the read on Fan 3.

The switch now runs next-to-dead silent; temperatures on the ASIC dropped ~20 degrees over my previous configuration into the low to mid 50s degrees and the PSU is in the upper 20s which is great. Been running around a week, everything has been perfect. I'm in love.
now please teach me how to quiet the 3 fans on my 12U XrackPro2 server cabinet please, lol

My own 7250 is ok for me, but compliments to your work.
 
  • Like
Reactions: acpatel

snclawson

Member
Feb 7, 2013
51
22
8
So it seems that I've found yet another rabbit hole to fall into care of STH!

This time my home network is getting a workover. I've already got the pfSense box. A pair of R500 access points are showing up today and I bought a cheap Mokerlink PoE unmanaged switch before I ran into this thread. As it turns out, I've also been moving a bunch of data between my main PC and my NAS recently and the single 1G link that it's currently connected to the network with is becomming a real annoyance.

Sooo...a cheap(ish), quiet box that does PoE and has at least two 10G SFP+ ports would be perfect.

Other than it being a little more than I was hoping to spend, the ICX7150-C12P seems to be it. =) But the ICX6450 is intriguing, especially if it comes around at $100 + shipping often enough, since it's got 2 extra 10G ports.


Two quick questions though, since I haven't be able to read the entire thread yet and either I'm incompetent with`search' or it's not mentioned:

- On the switches with Broadcom ASICs (ICX7150/ICX7250), is `bshell' available? Maybe through the console/debug terminal?

- I saw mention of what Marvell chip was in the ICX6450 as the CPU (although I can't find that again either!), but what switching ASIC is it using? I've seen mention of Prestera for the FCX line, but not the ICX6xxx?
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
So it seems that I've found yet another rabbit hole to fall into care of STH!

This time my home network is getting a workover. I've already got the pfSense box. A pair of R500 access points are showing up today and I bought a cheap Mokerlink PoE unmanaged switch before I ran into this thread. As it turns out, I've also been moving a bunch of data between my main PC and my NAS recently and the single 1G link that it's currently connected to the network with is becomming a real annoyance.

Sooo...a cheap(ish), quiet box that does PoE and has at least two 10G SFP+ ports would be perfect.

Other than it being a little more than I was hoping to spend, the ICX7150-C12P seems to be it. =) But the ICX6450 is intriguing, especially if it comes around at $100 + shipping often enough, since it's got 2 extra 10G ports.


Two quick questions though, since I haven't be able to read the entire thread yet and either I'm incompetent with`search' or it's not mentioned:

- On the switches with Broadcom ASICs (ICX7150/ICX7250), is `bshell' available? Maybe through the console/debug terminal?

- I saw mention of what Marvell chip was in the ICX6450 as the CPU (although I can't find that again either!), but what switching ASIC is it using? I've seen mention of Prestera for the FCX line, but not the ICX6xxx?
all icx6xxx series are marvell, icx7xxx is broadcom. If I recall correctly up-n-atom and I found the hidden debug shell, it required a bunch of u-boot args to get the system to boot in a state where it was allowed. What are you trying to do that's not exposed through the regular cli?
 
  • Like
Reactions: tommybackeast

snclawson

Member
Feb 7, 2013
51
22
8
all icx6xxx series are marvell, icx7xxx is broadcom. If I recall correctly up-n-atom and I found the hidden debug shell, it required a bunch of u-boot args to get the system to boot in a state where it was allowed. What are you trying to do that's not exposed through the regular cli?
Just interested in poking around and seeing how they've implemented things. My`day job' has consisted of writing software for switches that have mainly had Broadcom ASICs in them (including the Helix4 that the 7250 uses), so I'm somewhat familiar with poking around the ASIC registers via bshell (`bshell' being the `Broadcom shell' that comes as part of their SDK). =)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
If you're familiar with IDA Pro or Ghidra, the fastiron binary extracted from their firmware decompiles really nicely in them - can see pretty much all the broadcom registers and a ton of the SDK etc (this is how we found the shell enable commands in the first place)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
from our old notes:

Code:
FastIron or ICX
doesn't utilize config.bcm but instead has the config embedded in the
binary, I've at least found the code responsible at 0x0036CE24 and
will be able to parse out a config.bcm for SONiC. Fan, Temp, and PSU
should be straightforward since the kernel already contains a NCT7802Y
driver and it would just be a matter of probing the 3x PCA9557, 2x
DS100DF41 and also the CPLD.

I've also found the u-boot env check at 0x00014818 and the vars are:

nopolicer
debugoncrash
nocoredump
nofiapp
memdebug
mgmtdebug
mgmtpromisc
remotedebug
noautostart
nomod
nosoftwatchdog
enabletelnet
skiperror
storeforward
sildebug
disablefpga
disableautouboot
no_cpldauto_upd
disable-pkttest
enable-pkttest
en-pkttest-log
enable-tnls-reboot
enable-gpio-debug
enable-i2c-debug

They can shed a lot of info, for ex. sildebug enables all the nitty
gritty printouts for the FastIron binary itself
 
  • Like
Reactions: tommybackeast

OKGolombRuler

New Member
Mar 13, 2020
21
6
3
@fohdeesha - Licensing question. I see "Premium", "Advanced", and "Premium to Advanced", but after searching this forum ("premium AND advanced") and some modestly-enthusiastic googling, all I've been able to figure as the *potential* difference between licensing levels is
"
• BGP4, BGP4+(IPv6)
• GRE
• IPv6 over IPv4 tunnels
• VRF (IPv4 and IPv6)
" which seem to be in the 'advanced' but not(?) the premium.

Can you confirm my understanding, or expand on what functionality your free licenses do/not enable?