Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[m4r1k]

Make sure that you've configured the correct options in the bonding interfaces... try something like "mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3".
If you don't specify the 802.3d mode it won't send LACP control frames down the wire and thus the switch drops the LAG 'cause it doesn't detect a correct configured peer and it won't risk dropping traffic or worse creating a network loop.

Thanks, indeed LACP options are correct [1]. The problem doesn't happen when Linux is happily up and running but during the PXE phase (hence also the force-up)

I cannot find anywhere meaning for LagExpiry (the issue only persist with RHEL7 while RHEL8 is fine but of course it might be a timing problem and having a faster PXE like the one in RHEL8 won't trigger it)


[1] m4r1k/nfvi_lab

 

[infoMatt]

Thanks, indeed LACP options are correct [1]. The problem doesn't happen when Linux is happily up and running but during the PXE phase (hence also the force-up)

I don't think it's a safe choice to boot from a LACP... or at least, you could boot with just one interface of the bond up and then hot plug the other.

"By feeling", I've assumed that a LagExpired means that it couldn't receive the LACP renew control packet before the refresh timer expires... It could be that it cannot apply the LACP configuration correctly to a interface that is already UP and from where it booted...

EDIT: @m4r1k take a look at another user with a similar problem using FreeNAS: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-223952

 

[m4r1k]

I don't think it's a safe choice to boot from a LACP... or at least, you could boot with just one interface of the bond up and then hot plug the other.

PXE boot over LACP might not be common with Brocade but is done by plenty of people, you just the correct config, like NXOS LACP set in passive mode or the force-up of Brocade and Juniper ;-)
The config is also described in the ICX manuals, so nothing esoteric here

"By feeling", I've assumed that a LagExpired means that it couldn't receive the LACP renew control packet before the refresh timer expires... It could be that it cannot apply the LACP configuration correctly to a interface that is already UP and from where it booted...

Good point! In the PXE process, nothing that I know tried to create LACP, everything happens after this phase and yet it goes down.

Spoiler: Here the full logs just generated

May 27 16:17:29:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is down.
May 27 16:17:28:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is down.
May 27 16:17:28:I:System: Logical link on dynamic lag interface ethernet 1/1/14 is down.
May 27 16:17:25:I:System: dynamic lag 10 peer info (priority=65535,id=ecf4.bbdd.9628,key=1) remove (LagExpiry) <= the issue
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is down.
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is down.
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/14 is down.
May 27 16:17:25:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is down.
May 27 16:17:22:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is down.
May 27 16:17:22:I:Trunk: Group (1/1/13) removed by 802.3ad link-aggregation module.
May 27 16:17:22:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is up.<= starting of the PXE process
May 27 16:17:19:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is down.
May 27 16:17:19:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is down.
May 27 16:17:19:I:System: Logical link on dynamic lag interface ethernet 1/1/14 is down.
May 27 16:15:18:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is up.
May 27 16:15:18:I:System: Interface ethernet 1/1/13, state up
May 27 16:15:16:I:System: Interface ethernet 1/1/13, state down
May 27 16:15:13:I:System: Logical link on dynamic lag interface ethernet 1/1/13 is up.
May 27 16:15:13:I:System: Interface ethernet 1/1/13, state up <= during the POST
May 27 16:15:10:I:System: Interface ethernet 1/1/16, state down
May 27 16:15:10:I:System: Interface ethernet 1/1/15, state down
May 27 16:15:09:I:System: Interface ethernet 1/1/14, state down

Spoiler: Here the LAG details at the beginning of the PXE process

SSH@diablo(config-lag-R630)#show lag R630
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / cc4e.24f8.98d0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3

=== LAG "R630" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/13 to 1/1/16
Port Count: 4
Primary Port: 1/1/13
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/13 Up Forward Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/14 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/15 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/16 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/1/13 1 1 20010 Yes L Agg Syn Col Dis No No Ope
1/1/14 1 1 20010 Yes L Agg Syn No No Def Exp Ina
1/1/15 1 1 20010 Yes L Agg Syn No No Def Exp Ina
1/1/16 1 1 20010 Yes L Agg Syn No No Def Exp Ina


Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/1/13 65535-ecf4.bbdd.9628 1 4 20
1/1/14 1-0000.0000.0000 13 0 91
1/1/15 1-0000.0000.0000 14 0 91
1/1/16 1-0000.0000.0000 15 0 92

Spoiler: Here the LAG details at the beginning of the PXE process

SSH@diablo(config-lag-R630)#show lag R630
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / cc4e.24f8.98d0
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3

=== LAG "R630" ID 10 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/1/13 to 1/1/16
Port Count: 4
Primary Port: 1/1/13
Trunk Type: hash-based
LACP Key: 20010
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/1/13 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/14 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/15 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0
1/1/16 Up Blocked Full 1G 10 Yes 100 0 cc4e.24f8.98d0

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/1/13 1 1 20010 Yes S Agg Syn No No Def Exp Frc
1/1/14 1 1 20010 Yes S Agg Syn Col Dis Def No Ina
1/1/15 1 1 20010 Yes S Agg Syn Col Dis Def No Ina
1/1/16 1 1 20010 Yes S Agg Syn Col Dis Def No Ina


Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/1/13 1-0000.0000.0000 12 4 437
1/1/14 1-0000.0000.0000 13 0 576
1/1/15 1-0000.0000.0000 14 0 576
1/1/16 1-0000.0000.0000 15 0 578

 

[m4r1k]

Well, weird enough, specifying 'lacp-timeout short' the lag actually stays up.
I'll now run the entire deployment process and report back. Maybe somebody else ran into the same issue

 

[m4r1k]

Well, weird enough, specifying 'lacp-timeout short' the lag actually stays up.
I'll now run the entire deployment process and report back. Maybe somebody else ran into the same issue

Somehow the issue was related to the lack of timeout. Probably the unspecified default results in some timing issue if for "too long" LACP doesn't go up. Thanks guys!

 

[John Francini]

odd question, did you happen to dust the thing out previously by blowing canned air into the fan trays?

I opened it up to see if there was anything obvious (dust, parts visible from the top, etc.) - switch was clean as a whistle. I don't feel confident enough to disassmble the switch entirely to get the motherboard out and look for a cold solder joint or broken trace on the motherboard.

During the time the switch was off, at one point it did drop about 3 feet from the shelf it was on to the concrete floor. I expect that something's physically damaged. Sigh. As I said, I have a ICX 7250 on order.

Thanks for the ideas, folks.

 

[fohdeesha]

at one point it did drop about 3 feet from the shelf it was on to the concrete floor

OUCH. rest in peace 6610

 

[infoMatt]

I opened it up to see if there was anything obvious (dust, parts visible from the top, etc.) - switch was clean as a whistle. I don't feel confident enough to disassmble the switch entirely to get the motherboard out and look for a cold solder joint or broken trace on the motherboard.

During the time the switch was off, at one point it did drop about 3 feet from the shelf it was on to the concrete floor. I expect that something's physically damaged. Sigh. As I said, I have a ICX 7250 on order.

Thanks for the ideas, folks.

Well... yea.. that was really no bueno for sure.

OUCH. rest in peace 6610

Indeed.
Buut... broken it is... so... you can actually open it and try to look at dislodged components or broken traces/board without fear of breaking it... I mean...

 

[noise850]

I searched online, found GDSTIME Quiet 80mm Fan, 12V 3PIN 80mm x 80mm x 10mm Brushless DC Fan for CPU Coolers , put on top of the chip heat sink and it fits perfectly.

Do you mind posting some more pics / details on how you did this? I've got a 6450-48 swapped with a Sunon that is running super hot even with the fan on high. Under the original stock fan it would switch to low speed at boot but with the new fan it stays on high at all times and is fairly loud.

 

[epicurean]

Is this the right thread to ask for help on configuring vlans on a ICX6450?

 

[Jiaxinxi]

Do you mind posting some more pics / details on how you did this? I've got a 6450-48 swapped with a Sunon that is running super hot even with the fan on high. Under the original stock fan it would switch to low speed at boot but with the new fan it stays on high at all times and is fairly loud.

Already put the switch on my rack. Will see if I can take it out and take some more close up picture.
Here is some picture I took before I put it on rack.



My switch is ICX7240-48, NON-POE version, so there is some space between the main chip heat sink and the top shelf. I measured about 15MM

The 10MM fit into that space perfectly. I just use some drywall screw to fix the fan on the heat sink.

This screw is too long. You can by some short one. I just cut it short.

It is still not cool. But it is stablized at 77C. Below the temperature threshold (98C) of Speed 2 so it won't ask the fan to run at full speed which is noisy.

If you want really good cooling, you can cut a hole on the top shelf and install 120MM fan. or like this to cool off a POE switch. It is very quiet and temperature is very good.

 

[tozmo]

You'll have to verify that it is not used at the moment (ie. no DHCP lease active), and it is outside of the DHCP pool

Reviving page 124 from November. I'm a newbie+ on networking and CLI. If anyone has problems figuring out the config part of, "Now we need to assign that virtual interface an address. Choose an IP that is unused in your subnet, and out of your DHCP server range (ping it first to be sure it's unused)"

I use 192.168.0.1 for my main network. I used an old asus router that uses 192.168.1.1 in order to config my switch. Fohdeesha's doc made sense because he uses 192.168.1.1.

Once I put the switch on my network, I assigned it a static 192.168.0.xxx on my router, and then had to redo the command on the CLI to change the ip to the xxx I chose. (the interface ve 1 | ip address 192.168.0.xxx/24 commands)

My only other question is that the uplink ports "don't support half-duplex". Why does this matter, why would I want half-duplex? If I connect sfp+ connections (either directly to a nic or to a stack), the rj45 uplinks still function full speed, right? Some prosumer switches allow the sfp+ or the sister rj45's to be used, but not both at the same time.

My OCD may end up cleaning this thread up..., or creating a wiki like I did with the HP N40L (multi-hundred pages on an Australian forum, STH and homeservershow, and then Hardforum got confusing). So much good information/discussion here scattered amongst 180+ pages.

Thanks to all who have contributed to this thread, and to Fohdeesha

 

[tangofan]

Hi all, new forum user here,

I've read through this thread a few months ago, and just searched it for further info on my question, so I hope that my questions isn't too much of a repeat of previous ones and that this is indeed the appropriate place to ask.

I'm looking for a new main switch for my home network and I'm ready to dip my toes into the sea of enterprise-grade equipment (or at least something close to it) and am currently contemplating either the ICX 7150-24 or the ICX 7150-C12P for that role.

Background: I already run 10G on my NAS and my PC and have a MikroTik CRS305 switch to connect both and use the 1G uplink to connect to my router. As a router I'm currently running Untangle on a Protectli FWB6 Mini PC and the rest of my gear is repurposed consumer-grade equipment (including old Wifi routers running in AP mode).

The goal for that new switch would be to be to act as the main switch with the following requirements:
* 10G downlink to my MikroTik switch
* Provide extra SFP+ ports for future 10G devices (optinality)
* Provide 12+ 1G RJ45 ports for my networking equipment
* Be silent or near-silent (since the switch will be located in my office, near to my desk)
* PoE is nice to have, but I don't really need it right not (optionality)

At a later point I'm planning to retire the current APs and go with something more solid.

It seems that the ICX 7150-C12P and the ICX 7150-24 (non "P") would both fit the bill. Unfortunately the ICX 714024P (edit: I mean of course the 7150-24P) would not, since - according to the data sheet - it has a minimum noise level of 41.4 dbA, which appears to be too loud for my use case.

The prices of the 7150-C12P and the 7150-24 on eBay's US-site are in a pretty similar range. I like the optionality of PoE, but I like the optionality of more SFP+ ports quite a bit more. I don't really need 24 RJ45 ports, the 12 ports of the -C12P would be fine, it's just the exta SFP+ ports and the similar price range to the -C12P that make me lean towards the -24 model.

Before I pull the plug (or push the "Buy" button) I'd like to ask the following: Does the 7150-C12P have any advantage over the 7150-24 other than the smaller form factor and PoE or would (at it appears to me right now) the 7150-24 be the better choice for my use-case?

Thanks in advance for any advice on this matter.

 

[infoMatt]

If you want really good cooling, you can cut a hole on the top shelf and install 120MM fan. or like this to cool off a POE switch. It is very quiet and temperature is very good.

Oh for $santa sake, please don't... Those boxes are very well engineered, don't ruin countless hours of airflow engineering for a redneck solution.. simply put, if the replacement fans can't keep the unit cool enough, they can't move enough air.. Instead of cutting holes on the top and disturbing the flows with random fans, all @noise850 has to do is to replace the replacement fans with others slightly more powerful.. and maybe it won't be any louder because they will actually be moving the air through the unit, instead of stalling and choking due to insufficient static pressure.

My only other question is that the uplink ports "don't support half-duplex". Why does this matter, why would I want half-duplex? If I connect sfp+ connections (either directly to a nic or to a stack), the rj45 uplinks still function full speed, right? Some prosumer switches allow the sfp+ or the sister rj45's to be used, but not both at the same time.

I am not sure what you mean with "half duplex"... SFP/SFP+ ports can't negotiate 100Mbps or lower, so there isn't any specification for a gigabit speed (or higher) with half duplex connection.
What you might wanted to say was "dual personality" or "dual media" ports... if that's the case no, there is no dual media ports.. every interface has it's own physical connection and all can be used at the same time.

 

[infoMatt]

Before I pull the plug (or push the "Buy" button) I'd like to ask the following: Does the 7150-C12P have any advantage over the 7150-24 other than the smaller form factor and PoE or would (at it appears to me right now) the 7150-24 be the better choice for my use-case?

Generally speaking, the "C" model doesn't have any advantages other than being passively cooled, smaller in size and possibly reverse-PoE powered from the upstream port.
Other than that it's pretty much the same as the non-C version, but keep in mind that the compact tend to run fairly hotter than the rack version.

i think that you can also swap the fans on the 7250 with other models, but don't count on my word for it, have a look at the thread if this is what fulfils your needs.

 

[Jiaxinxi]

Oh for $santa sake, please don't... Those boxes are very well engineered, don't ruin countless hours of airflow engineering for a redneck solution.. simply put, if the replacement fans can't keep the unit cool enough, they can't move enough air.. Instead of cutting holes on the top and disturbing the flows with random fans,

Agree with you on the airflow engineering part. It is definitely very sophisticated. But it is designed for data center. Noise is not an issue, density matters. For folks want to us at home, noise is a major issue. Like me, many people put the gear in their living space. 40dBA is probably the max noise level while in Data center, 70+dBA is common. Physical law determines that smaller fan will generate higher noise. I have tried numerous 40mm fan including Noctua, I haven't found one that has good air flow yet silent yet.

 

[tangofan]

Generally speaking, the "C" model doesn't have any advantages other than being passively cooled, smaller in size and possibly reverse-PoE powered from the upstream port.
Other than that it's pretty much the same as the non-C version, but keep in mind that the compact tend to run fairly hotter than the rack version.

i think that you can also swap the fans on the 7250 with other models, but don't count on my word for it, have a look at the thread if this is what fulfils your needs.

Thanks so much for responding. Actually the data sheet seems to say that the 7150-24 (not the 7250-24!) is also passively cooled or did I misread that?

 

[infoMatt]

Thanks so much for responding. Actually the data sheet seems to say that the 7150-24 (not the 7250-24!) is also passively cooled or did I misread that?

Yes, according to the datasheet it's fanless. Even the 7150-48 doesn't have a fan.

 

[LodeRunner]

The 7150-C12P is not PoE powered, and it only has 2 SFP+ ports, the only ports on it that can do 10G. There are 12 PoE ports, then 2 non-PoE gigabit ports, and the 2 SFP+ ports.

If you need more than 2 10G ports, the 7150-24, or a 7250 would be decent choices, or the 6610. A 7250-24 or a 6610 would give you 8 SFP+ ports, the 6610 also has 4 QSFP+ ports and I believe that 2 of them support 4x10G breakout.

 

[infoMatt]

The 7150-C12P is not PoE powered

Yea, sorry, I was thinking at the 6450-C12-PD which can act as a PoE powered device.

Agree with you on the airflow engineering part. It is definitely very sophisticated. But it is designed for data center. Noise is not an issue, density matters. For folks want to us at home, noise is a major issue. Like me, many people put the gear in their living space. 40dBA is probably the max noise level while in Data center, 70+dBA is common. Physical law determines that smaller fan will generate higher noise. I have tried numerous 40mm fan including Noctua, I haven't found one that has good air flow yet silent yet.

Sorry I've missed your post before. You are right about the noise, but those aren't datacenter grade switches (trust me, you don't want to stand next to a Cisco 9300 series units at startup; at full tilt those screams at 100dB or thereabout), but access grade switches, they are ment to be installed in some networking racks/closet near the offices.
I've seen a video about a fan swapping of a 6450 with some noctuas and it became basically inaudible at 30% PWM... But yea, you need to find the right fans capable of moving air through the unit.
Don't expect it to be at 30°C/90F, those are older process silicon, they are designed to run at 50-60°C/120-140F basically all the time, and hardly ever be shut down.