Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[jasonwc]

also, if his switches were really pulling 200w or above like he states, the fans would NOT be on fan speed 1 still as his output indicates - that is an insane amount of heat for these fans to exhaust out of 1ru, they would have ramped up to fan speed 2. How is he measuring power draw?

I asked him the same question. He said he used this meter -https://www.amazon.com/Digital-Wattmeter-Consumption-Frequency-Electricity/dp/B0828QWQZP

He noted he’s using a 230V Shucko plug (Poland).

It didn’t make sense to me that the switch would really pull that much power so I told him to test by running with a single power supply. He said it wouldn’t boot in that configuration.

 

[jasonwc]

these switches were never counterfeited (that I'm aware of), so it's certainly genuine, but it sounds like it had a pretty serious fault to begin with. if a 250w PSU is pulling 400w, I'm assuming the PSU(s) themselves had a pretty bad fault, which could also explain why their manuf data EEPROM couldn't be read (reporting all FFF)

He says he has three of the 250W power supplies. Could all be defective?

 

[fohdeesha]

He says he has three of the 250W power supplies. Could all be defective?

if he's tried three separate supplies then there's a 99% chance the failure mode is inside the switch itself. especially if it only boots with two power supplies

 

[jasonwc]

if he's tried three separate supplies then there's a 99% chance the failure mode is inside the switch itself. especially if it only boots with two power supplies

So, it sounds like the switch itself is defective, and his best bet is simply to buy a new ICX6610-24. Is there any chance old firmware or a bad configuration could cause these issues? He’s looking at paying over $200 in shipping to buy a new switch from the US due to limited availability in Poland.

 

[RobstarUSA]

Has anyone got RANCID workin with backups for the 6610s? It fails for me when configured as type foundry. I've seen a few other posts like that on the RANCID mailing list but no solution. If anyone has an ieas......(I have a stack of 2)

 

[danb35]

I'm using Oxidized to back up mine, and it's working fine--don't know if information there would help though.

 

[fohdeesha]

So, it sounds like the switch itself is defective, and his best bet is simply to buy a new ICX6610-24. Is there any chance old firmware or a bad configuration could cause these issues? He’s looking at paying over $200 in shipping to buy a new switch from the US due to limited availability in Poland.

if he's tried 3 different PSUs and the switch always shows this behavior I'm not sure what else it could be. no way a config or firmware issue is going to make it draw 3x it's rated power

 

[seatrope]

Please note that DHCP is not a normal application protocol which can be handled by VRRP or similar techniques.

Clients locate the DHCP server using broadcast messages, which don't even have normal IP addresses in them (they can't since the client doesn't have an address at that point). Once the DHCP transaction has been completed the client will use unicast messages for renewal/release (unless that fails, in which case it will go back to broadcast). if the IP address of the DHCP server is handled via VRRP, the secondary/failover DHCP server will need to have all of the lease state from the primary one in order to be able to respond to these messages properly. It also needs that state in order to ensure that it doesn't hand out duplicate addresses.

HA DHCP is really quite off-topic for this thread (although not this forum!) but it certainly can be done. I've got ISC Kea DHCP running on two boxes on my LAN, using ICX 7150s for traffic, and it works really well. The ICXs are configured with two helper addresses for forwarding DHCP broadcast traffic to both Kea boxes in parallel, and then Kea handles the HA aspects itself.

@kpfleming thanks for this valuable nugget. I've been trying to understand the underpinnings of this, and went and learned a bit more about how DHCP is performed. As DHCPREQUEST is being broadcast (I assume across only the level 2 subnet and not further unless we enable some form of multicast forwarding) - will VRRP with keepalived work IF:
- the pair of HA (active-passive, not active-active) raspi's are on a separate subnet containing only those two hosts; and
- iphelper is configured in the ICX VLANs to point to the VIP in that VLAN; and
- no multicast passthrough to that "DHCP server" VLAN is active?

If so, the DHCPREQUEST broadcast should only be directed by the 6610 to the iphelper addess which should be the VIP (and not the specific IPs of the primary or secondary DHCP servers?

Thoughts before I go down this rabbit hole? I started configuring isc-dhcp then got sucked back into thinking about this. I really like having everything under the pihole interface for ease of use

 

[kpfleming]

Yes, that configuration should work, although you will need to be certain to configure the DHCP server instance to use the VIP as their identity address, so that unicast replies from them will use the proper source address.

 

[seatrope]

@kpfleming thanks! Trialing it right now. Got one Rpi and VIP setup. Ip helper pointed only to the VIP. Second one will be proof of the pudding later this week. Fingers crossed. Thanks for all the help!

 

[fohdeesha]

@kpfleming thanks! Trialing it right now. Got one Rpi and VIP setup. Ip helper pointed only to the VIP. Second one will be proof of the pudding later this week. Fingers crossed. Thanks for all the help!

there's no need to dick around with VRRP, fastiron (and almost every other enterprise switch) supports forwarding dhcp broadcasts to multiple IPs/DHCP servers directly, then the HA component of ISC-DHCP handles which one is active (replies to the forwarded broadcasts) and which one is standby (ignores them):

interface ve 10
 ip address 192.168.1.1 255.255.255.0
 ip helper-address 1 172.16.110.2
 ip helper-address 2 172.16.110.3
 ipv6 address xxxx::1/64
 ipv6 enable
 ipv6 dhcp-relay destination xxxx::2
 ipv6 dhcp-relay destination xxxx::3
 ipv6 dhcp-relay include-options interface-id remote-id
 ipv6 nd managed-config-flag

If you're going through the time to set this up from scratch as well, I would highly recommend using ISC Kea, which is replacing ISC-DHCP. you can also then set up ISC Stork which is a nice web UI for ISC-KEA clusters:

 

[seatrope]

there's no need to dick around with VRRP, fastiron (and almost every other enterprise switch) supports forwarding dhcp broadcasts to multiple IPs/DHCP servers directly, then the HA component of ISC-DHCP handles which one is active (replies to the forwarded broadcasts) and which one is standby (ignores them):

interface ve 10
ip address 192.168.1.1 255.255.255.0
ip helper-address 1 172.16.110.2
ip helper-address 2 172.16.110.3
ipv6 address xxxx::1/64
ipv6 enable
ipv6 dhcp-relay destination xxxx::2
ipv6 dhcp-relay destination xxxx::3
ipv6 dhcp-relay include-options interface-id remote-id
ipv6 nd managed-config-flag

If you're going through the time to set this up from scratch as well, I would highly recommend using ISC Kea, which is replacing ISC-DHCP. you can also then set up ISC Stork which is a nice web UI for ISC-KEA clusters:

Thanks @fohdeesha Jon, if I wasn't so fixated on having an integrated interface between pihole DNS and DHCP I would have gone down this road for sure. Looked into building kea DHCP for raspi and it was not straightforward (for me) either.. I don't want to depend on a VM for this. If VRRP/keepalived doesn't work I'll bite the bullet and go either Kea or ISC.

 

[beren]

Hey @fohdeesha great guide! I just got my 6610. I was thinking of splunking for old licenses on the flash just because, and wondered if upping the baudrate in the bootloader would make it take less time. Anyone ever tried that?

 

[fohdeesha]

Hey @fohdeesha great guide! I just got my 6610. I was thinking of splunking for old licenses on the flash just because, and wondered if upping the baudrate in the bootloader would make it take less time. Anyone ever tried that?

if you can figure out the hidden var to actually do that go for it lmao

 

[beren]

if you can figure out the hidden var to actually do that go for it lmao

damn you're right. just got home with my serial cable. I thought you were the hacker!

BTW what's the current recommended JTAG? Seems BDI2000 doesn't exist anymore.

 

[Vatharian]

Hello!
I am the reddit person with said failed switch. I have just registered to reply here.

Original post:

A user on Reddit posted an interesting problem with his ICX6610-24 (non PoE). He said his switch idles at 180-200W and when under load, can hit 400W. I told him this makes no sense given that the spec sheet indicates it only requires a single 250W power supply. The specs say the second power supply is optional, for redundancy. Also, this thread indicates it should idle at 80W or so.
[...]
Based on this information, does the switch appear to be genuine? Why would the power supply show a blank model number with no revision?

these switches were never counterfeited (that I'm aware of), so it's certainly genuine, but it sounds like it had a pretty serious fault to begin with. if a 250w PSU is pulling 400w, I'm assuming the PSU(s) themselves had a pretty bad fault, which could also explain why their manuf data EEPROM couldn't be read (reporting all FFF)

I assume you are right on every account. They HAVE been tampered with.

also, if his switches were really pulling 200w or above like he states, the fans would NOT be on fan speed 1 still as his output indicates - that is an insane amount of heat for these fans to exhaust out of 1ru, they would have ramped up to fan speed 2. How is he measuring power draw?

I asked him the same question. He said he used this meter -https://www.amazon.com/Digital-Wattmeter-Consumption-Frequency-Electricity/dp/B0828QWQZP

He noted he’s using a 230V Shucko plug (Poland).

It didn’t make sense to me that the switch would really pull that much power so I told him to test by running with a single power supply. He said it wouldn’t boot in that configuration.

I linked the power meter very closely resembling the one I have, as I couldn't find the my Poland/Germany only model on US Amazon. I also have DIN rail mounted current meter.

He says he has three of the 250W power supplies. Could all be defective?

if he's tried three separate supplies then there's a 99% chance the failure mode is inside the switch itself. especially if it only boots with two power supplies

So, it sounds like the switch itself is defective, and his best bet is simply to buy a new ICX6610-24. Is there any chance old firmware or a bad configuration could cause these issues? He’s looking at paying over $200 in shipping to buy a new switch from the US due to limited availability in Poland.

if he's tried 3 different PSUs and the switch always shows this behavior I'm not sure what else it could be. no way a config or firmware issue is going to make it draw 3x it's rated power

After reading the replies out there and looking into all the photos and other threads I could find of the switch and its insides, I came to conclusion, that I either "scored" engineering sample or qualification sample or got a result of some not-exactly-professional-but-still-somewhat-competent person frankensteining this device from scrapped or stolen lot.

It can be either, since it's impossible to modify fan curve, and knowing devices like that usually turn into "survival mode" (fan speed to max) when some sensors are off, it behaved surprisingly well, suggesting either someone found a way to alter it, to fool the sensors or it came with pre-release fw or client-targeted one.

I should have opened it right after receiving, but I only did coursory look to see of there are no loose components or screws, threw cables in and it worked, so I left it as it is. Only thing I did was swap the PSUs around. Switch and extra PSU were bought from same place (an eBay auction, and it was private person as far as I remember).

Using this photo:

https://people.ucsc.edu/~warner/Bufs/icx-6610.JPG

as reference, I found:

- switch has no serial number, neither on the device itself or on the motherboard, and all stickers were removed. Can't say anything about the firmware now, but you saw the zeros. Only markings I could find are laser etched "AR2054-01-011" between CPU and stack connector card and "Brocade MV1194V-0 / AB 1 026-3" under the card;
- memory stick was kaptoned to the slot from all sides and all over, and I have Smart sg57a648bro535y1sj EP2-5300c-555-13-zz. This is the only thing that has intact serial number in whole device, but I didn't find anything about this particular stick.
- all PSUs had a sticker with revision saying S5, where there would be A, B or C originally, underneath it the original rev has been scratched off. Stickers with QR and s/n are missing. They all bear marks of being opened multiple times (a lot of scratches around the screws and tabs were clearly abused). I missed this since for some time I used to handle device scrapping at my workplace so I got resistant to noticing scratches.
- I went over the motherboard with good light and found solder flux residue around almost all power components;
- almost all of electrolytic capacitors are random brands;
- Boot flash chip has "fused!" handwritten with a marker pen and flux residue around its legs; This probably explains the bricking and null s/n.
- Battery socket, after removing the battery shows signs of cleaned up corrosion (I don't ever recall seeing Lithium battery leak!); Current battery reads 2.9 V.
- headers J2, J10 and U6 had been clearly removed;
- place where POE headers go in PoE-equipped models were clearly soldered on and cleaned up;
- One of the fan tray connectors on the board had mangled pins and its mounting screw was held in place by copious amount of Locktite or similar glue. I had to wrestle it off. Fan modules look okay. There is a trace of s/n stickers being removed from them on the inside between the fans.

One thing that stands to me now, is that I never noticed the switch kicking up the fans to speeds anywhere close those heard during boot sequence, they did went up but not much - but at the same time it really did pull that much power off the wall. On idle the exhaust was moderately warm to very warm, but I wouldn't describe it as hot, like for example Dell R640 going full tilt can get, and with max traffic I could put on it it got really, really hot, enough to make keeping hand in the airstream very uncomfortable to painful.

I should also note that I misread the specifications! I assumed that both PSUs work in unison and share power, and to boot it only from one I need higher tier, 750 ow 1000 W ones. I am used to moving around 2 kW+ switches at work, so this is really why I paid no mind to inability of my device to turn on with one and the power consumption.

With all of that, and switch now bricked, even if I managed to find someone actually competent who would repair it I don't exactly feel safe putting it back into my homelab, little late perhaps, but oh well.

If anything I consider this to be a warning not to blindly trust enterprise gear. It never dawned on me someone would just go over a device like this to fix it. It's not that big blow financially, these switches go under $100 routinely, and I have other 10G gear, 40G was very cool addition, but I can live without it.

 

[NablaSquaredG]

@Vatharian Can you please post (detailed) pictures of your switch?

 

[seatrope]

Yes, that configuration should work, although you will need to be certain to configure the DHCP server instance to use the VIP as their identity address, so that unicast replies from them will use the proper source address.

Seems to be working well now. HA piholes/DHCP/unbound DNS with keepalived VIP. 6610 iphelper is pointed only to the VIP address and there are no other hosts in the small subnet that the piholes are in.

Tested failover and worked well.

Thanks for the help, all!

 

[LemonheadST]

Hello,

I'm having an odd problem with the rear 10gb ports on my ICX6610 (1/2/2-1/2/5 & 1/2/7-1/2/10). I'm attempting to connect 4 of these ports to a host (using a breakout cable), and bonding them using 802.3ad LACP.

On a fresh boot of the switch I can get this fully working on either QSFP+ port, all ports are up, and it works for a time. However, after several reboots of the other device (a Honeycomb LX2K), most or all of the 10gbe interfaces on the QSFP+ port just stop passing traffic, 'show lag' reports that they are LACP-BLOCKED.

The problem persists when I remove the ports from the trunk - they just won't pass traffic anymore even as standalone ports. Yes I did re-enable them after running 'no lag' . I can connect the port to a completely different interface or machine and it still won't pass traffic. I even tried a different breakout cable - no luck there either. There is nothing unusual about the interface when running 'show interface' - it's shows enabled at 10gbe and in a FORWARDING state.
I can move the QSFP+ module to the other 4x10gb port and those will work fine for a time, but eventually these ports get the same issue.
So far the only 'fix' for the issue is to completely reload the switch. After a reload, all of the ports function normally, for a time.
Has anyone run into a similar issue with the rear 10gbe ports? If so, is there a way to prevent this? Or at least get them working without a switch reload?

10GigabitEthernet 1/2/7 is up, line protocol is up
Port up for 1 hour(s) 18 minute(s) 37 second(s)
Hardware is 10GigabitEthernet , address is [removed]
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Configured mdi mode AUTO, actual none
Member of 8 L2 VLANs, port is dual mode in Vlan 1, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 5080 bits/sec, 5 packets/sec, 0.00% utilization
7093 packets input, 851519 bytes, 0 no buffer
Received 824 broadcasts, 6246 multicasts, 23 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
330955 packets output, 36155642 bytes, 0 underruns
Transmitted 79531 broadcasts, 248621 multicasts, 2803 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

 

[fohdeesha]

damn you're right. just got home with my serial cable. I thought you were the hacker!

BTW what's the current recommended JTAG? Seems BDI2000 doesn't exist anymore.

A while back I scoured through the bootloader binary and couldn't find any evidence whatsoever of an adjustable baudrate so I'm pretty sure it's stuck on 9600. The baud is easily changeable however on the newer switches that run u-boot (because it's just u-boot, use the baudrate env variable)

as for jtag, that's your only option. Not many PowerPC JTAGs existed for this particular generation, the bdi2000 and the bdi3000 were the only real models and the bdi3000 is even more rare/expensive. There were also a couple Freescale Codewarrior models but they're useless without the (very expensive) CW software package