Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[donedeal19]

I stopped bidding after they were over 400$. If it’s not fully functional then I recommend a full refund. If it’s for business then I would not trust it. However for home use you might not even notice it.
Put some data down the 1,2,10,40 ports and some Poe devices for 24 hours and then check out the logs for any weird.

 

[TonyArrr]

I don't think I touched anything high voltage (120v), instead I think I hit something around 12v as my hand kinda just felt funny for a few seconds and felt fine after removing contact.

Yeah it’s not really the voltage I’d be worried about. 3 volts at 1 amp is enough to kill a person. 240v at 0.01 amp will be fine.

If you had dry skin, relative humidity was medium or low, and you had no open wounds (paper cuts, even) then that all works in your favour.

Glad to see you’re still up and posting through!

 

[TonyArrr]

CRAP

What does this even mean?

1/2/2   374945528       374945528       0               ##PASS
1/2/3   371165213       371165213       0               ##PASS
1/2/4   366856907       366856907       0               ##PASS
1/3/1   [  409.240204] [BrcdSoftlockup]: Tracker queued..
416501837       416501837       0               ##PASS
1/4/1   419099154       419099154       0               ##PASS


FATAL ERROR: System did not pass diagnosis
SYSTEM STOPPED, Please Check with Ruckus Networks Support
Typing 'r' would reset the system
##FAILED DIAG
[  411.030781] [BrcdSoftlockup]: sim_softwatchdog thread is detached on core=1

I've also been seeing this during boot and now wonder if it is significant?

                        SOC unit 0 attached to PCI device BCM56548_A0
0:soc_do_init: soc_do_init: total cpu and arm cosq 0030 unexpected
Initializing the Port pair for the P2 board
u=0 p=50 Using SDK PHY driver for BCM84756/7/9
u=0 p=51 Using SDK PHY driver for BCM84756/7/9
Setting the CMIC registers for the link scan
SOC unit 1 attached to PCI device BCM56548_A0
1:soc_do_init: soc_do_init: total cpu and arm cosq 0030 unexpected
u=1 p=42 Using SDK PHY driver for BCM84756/7/9
u=1 p=43 Using SDK PHY driver for BCM84756/7/9
Setting the CMIC registers for the link scan
Info: SendEventCommon: scp task Not inited by FI

Sigh. I've wanted this switch for so long....

Yeah this is the decision I most hate facing. Will this partially working through work well enough for me at least long enough that I can get another one to replace it, or do I yeet my cash back and do some more waiting.

As a in between measure, you might be able to talk the seller into a partial refund. If they were selling it as “tested and working”, but the built in self diagnostics say it’s faulty, then it’s not really as described.
If you can get a bit of your money back for the reduced functionality, it certainly makes it easier to keep in use knowing you need to be ready to replace it at potentially short notice..

 

[jei]

^ is that the last or the first error in a series? It broke a bit and it's smooth sailing from now on? Don't cope, just return it for good money if there's a return window..

 

[CIR-Engineering]

^ is that the last or the first error in a series? It broke a bit and it's smooth sailing from now on? Don't cope, just return it for good money if there's a return window..

That’s the first, last, and only error. Got it during both diagnostics.

Does anyone know what it means?

I’ll probably get a refund. I paid about $640 with tax and shipping.

Darn.

 

[CIR-Engineering]

^Well, kinda like dating a dime if the mental stability isn't there. Do you REALLY want it, and if so how long can you last before you lose it...

In the past I’d have said “yes” and I don’t care.

 

[CIR-Engineering]

So one thing. While running the diagnostics, the only lights that acted odd were on one of the 40 GbE cards. Could this affect just that connection? I doubt I will ever use it.

 

[CIR-Engineering]

And another thing. Is this a CPU issue, a PoE issue, both? I'm trying to narrow down where to look on the board for problems. So often these are easy to fix. I have a few caps (but they are near the power supply) that are measuring as shorted so possibly there is one bad one somewhere (could also be normal). I may need to remove the heat sinks but then I'll probably also need to replace the thermal pads... Anyone else do SMD rework here? I'm not very familiar with the 7450 topography so I am looking for help on where to look on the PCB. The way I read the diagnostic error is that watchdog is restarting or shutting down one core of the CPU?

 

[nutsnax]

I picked up a 7250 and can't use the 10gbe SFP+ ports as uplinks to other switches only as standard ports for servers. Is there some mode that I need to set on the switch to allow me to uplink to other switches?

 

[jei]

I picked up a 7250 and can't use the 10gbe SFP+ ports as uplinks to other switches only as standard ports for servers. Is there some mode that I need to set on the switch to allow me to uplink to other switches?

Do you mean stacking? All 7250 models support SFP+ uplinks (data), all models except 7250-24G support stacking.

 

[nutsnax]

Do you mean stacking? All 7250 models support SFP+ uplinks (data), all models except 7250-24G support stacking.

Ahh, so Uplinking is called Stacking in Brocade-land. Got it. Is that enabled on a per-port basis? If so can I plug a standard NIC into it afterward or would stacking need to be disabled?

 

[jei]

Ahh, so Uplinking is called Stacking in Brocade-land. Got it. Is that enabled on a per-port basis? If so can I plug a standard NIC into it afterward or would stacking need to be disabled?

I'm by no means stacking expert but from reading the manuals and this thread, stacking needs to be disabled for "normal" switching on the port. (edit: It's port based, yes.)

 

[CIR-Engineering]

Yeah this is the decision I most hate facing. Will this partially working through work well enough for me at least long enough that I can get another one to replace it, or do I yeet my cash back and do some more waiting.

As a in between measure, you might be able to talk the seller into a partial refund. If they were selling it as “tested and working”, but the built in self diagnostics say it’s faulty, then it’s not really as described.
If you can get a bit of your money back for the reduced functionality, it certainly makes it easier to keep in use knowing you need to be ready to replace it at potentially short notice..

Well for now partial refund it is! They gave me back $337 + tax (11% here) so now instead of about $640, I paid $246.

^Well, kinda like dating a dime if the mental stability isn't there. Do you REALLY want it, and if so how long can you last before you lose it...

For $246 I'll date the dime any time and see how crazy she is in the long run. It's from eBay Fastcrap whom I've bought from many times; they are awesome and have a hilarious name. It was advertised as fully functional and they immediately offered to take it back, but I asked about a partial refund, they offered $250. I sent a note and a counteroffer of $337 back, they said OK. I'll install the switch tonight in a working environment and make sure she's not completely dysfunctional. With eBay I can still get a full refund for 30 days if she's totally crazy.

I am still hoping to fix this switch. I don't have a thermal camera now and that would help, [EDIT: pulled camera out of packing for move to new house], but I am quite good at tracking down problems. It could be a problem with an important chip, but there is a very high chance it's just a shorted cap or fried resistor someplace. After I run it for a week and have some free time (like that ever happens) I'll try and find what's fried again.

Thanks to you all for your technical help, opinions, and encouragement.

Kind regards to fohdeesha and all of you who have been there for me. I truly appreciate all of your assistance!

 

[TonyArrr]

Ahh, so Uplinking is called Stacking in Brocade-land. Got it. Is that enabled on a per-port basis? If so can I plug a standard NIC into it afterward or would stacking need to be disabled?

Stacking logically combines the two or more switches so when you admin them, you just connect to one of them and can apply config to any port on any of the stacked switches. If you have more than two switches, you basically make a ring connection going through all of them.

You can also only stack compatible models. For example, I can’t stack my 7150-24 with my 7250-24.

Take a look at the stacking guide in the docs that come with @fohdeesha’s download

 

[fohdeesha]

Well for now partial refund it is! They gave me back $337 + tax (11% here) so now instead of about $640, I paid $246.



For $246 I'll date the dime any time and see how crazy she is in the long run. It's from eBay Fastcrap whom I've bought from many times; they are awesome and have a hilarious name. It was advertised as fully functional and they immediately offered to take it back, but I asked about a partial refund, they offered $250. I sent a note and a counteroffer of $337 back, they said OK. I'll install the switch tonight in a working environment and make sure she's not completely dysfunctional. With eBay I can still get a full refund for 30 days if she's totally crazy.

I am still hoping to fix this switch. I don't have a thermal camera now and that would help, [EDIT: pulled camera out of packing for move to new house], but I am quite good at tracking down problems. It could be a problem with an important chip, but there is a very high chance it's just a shorted cap or fried resistor someplace. After I run it for a week and have some free time (like that ever happens) I'll try and find what's fried again.

Thanks to you all for your technical help, opinions, and encouragement.

Kind regards to fohdeesha and all of you who have been there for me. I truly appreciate all of your assistance!

I would just forget all this happened and use it like it's brand new, because if I'm understanding you correctly, it operates perfectly. I wouldn't give too much credence to the output/results of dm commands, they weren't designed for end user use, a lot have to be ran in very specific environments to give reliable/believable output, and a ton of them just outright don't work on certain platforms. Regarding the PoE error that sometimes comes up, granted it's only sometimes, that leads me to believe even more what I said earlier, it's probably the PoE board trying to come up to soon before the PHY has or some other weird race condition. You can try updating to a newer codetrain like v9 to see if it goes away

 

[CIR-Engineering]

I would just forget all this happened and use it like it's brand new, because if I'm understanding you correctly, it operates perfectly. I wouldn't give too much credence to the output/results of dm commands, they weren't designed for end user use, a lot have to be ran in very specific environments to give reliable/believable output, and a ton of them just outright don't work on certain platforms. Regarding the PoE error that sometimes comes up, granted it's only sometimes, that leads me to believe even more what I said earlier, it's probably the PoE board trying to come up to soon before the PHY has or some other weird race condition. You can try updating to a newer codetrain like v9 to see if it goes away

Thanks for that.

EDIT 2: I don't know what I don't know or know. That AP has given trouble before and now everything is running fine all day. I'm just going to deploy the switch tonight while everyone is in bed and we'll see what happens. Then I'll know for sure.

I have discovered that there is something wrong with PoE on some of the ports. There are two PoE chips for the regular PoE (High PoE is separate on ports 1-8) that cover 16x of the 1 GB ports (17-24) and the 8x 2.5 GbE ports (25-32). The PoE seems to cut in and out on the right-side chip (closest to the outside of the board) and right side 1 GbE ports (theory now). I monitored my PoE 1 GbE WiFi access points on port 11 and 19 all night. Eleven was perfect, but port 19 keeps going up and down. I have a bad feeling the flaky chip covers the 2.5 GbE ports (but I have nothing here now to test PoE that is 2.5 GbE. I'm hoping it's not actually the PoE IC that's the problem, but instead its power supply chip. I'll know more when I dig.

But this could put a large obstacle in my deployment of 2.5 GbE WiFi access points.

EDIT: I broke my own rule and posted before coffee.

I also got an idea that I am trying now. I have PoE extractors that I am going to plug into the 2.5 GbE ports and power some things even if they aren't 2.5 GbE. I have one WiFi access point plugged into the 2.5 GbE ports now. It can't connect because it's only a 1 GbE AP, but it is drawing PoE. I'll monitor through serial and see if the PoE stays up.

 

[Wolfcastle]

This has maybe been asked in this thread but has anyone tried to connect a 10Gbps Wifi 7 AP on the 10G ports for a 6610?

I am specifically looking at the TP-Link EAP773. I know it's not ideal in the sense that I would need to use a 10G port with a 10GBase-T SFP+ and a POE injector but I assume it should work right?

 

[NablaSquaredG]

but I assume it should work right?

Yep

 

[Craig Curtin]

Having trouble with my 10g ports on a 6610. I had a dual lag connection with my storage server and it was working fine. Left the original 1g connection on it. Came back an hour later and the 10g ports were not connected anymore. Reverted back to the 1g connection and it works but nothing I do can make the 2 10g connections come back up. Moved the sfp adapters around, tried a 10g connection to another switch, works fine. A patch cable looped works fine. Tried clearing everything in cache, rebooting warm and cold. Any suggestions?

Edit. I am using an HP branded mellanox connectx3 dual card. Seems there might be issues with that card.

I had something very similar to this (search under my name in this thread) - 3 x ESXi hosts in a cluster - trying to use dual 10Gb ethernet ports (on a variety of different cards to the same switch - could not consistently get both connections to run - tried across various versions of ESXi (6.7, 7,8) and still the same - each time a restarted a host i was a fair chance of losing one of the 10GB connections and the only fix was a switch reset. Happened across multip 6610 as well.

In the end i gave up and use the 40GB ports for two of the hosts.

IN the other one i had to place two single port 10GB cards- dual port just would not do it

Craig

 

[CIR-Engineering]

I would just forget all this happened and use it like it's brand new, because if I'm understanding you correctly, it operates perfectly. I wouldn't give too much credence to the output/results of dm commands, they weren't designed for end user use, a lot have to be ran in very specific environments to give reliable/believable output, and a ton of them just outright don't work on certain platforms. Regarding the PoE error that sometimes comes up, granted it's only sometimes, that leads me to believe even more what I said earlier, it's probably the PoE board trying to come up to soon before the PHY has or some other weird race condition. You can try updating to a newer codetrain like v9 to see if it goes away

Updated to 09010j and so far no PoE restarts... fingers crossed.

Web GUI is even more useless, but much prettier