Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

Well then, after all this time I finally have the necessary tools (at least I think I do) and knowledge to even attempt to connect to my Brocade FCX648-HPOE (picking up an old Cisco 887VA-K9 V2 (the old version) that was thrown out by a local medical center branch has helped in that regard).

Unfortunately, I cannot seem to be able to connect to it, no matter what I do (I was able to connect to the Cisco router without any issues, although I am still struggling to understand how to get it to acknowledge what is connected to it, and not just that something is connected).

I have tried using serial RS232 from my motherboard (started with an ASRock H97M, now with an Asus Z97 Pro Gamer), however have been unsuccessful in connecting to anything with the onboard serial controller (I tried both passthrough and crossover cables), so switched over to CH340G-based USB serial controllers (the USB to RJ-45 patch cable works well with the Cisco router, still need to test the USB to RS232 adapter in conjunction with the crossover cable), however even then I am unable to get any output from the console (I use Picocom, and I have verified that my settings are correct: 9600/N/N/8/1 with /dev/ttyUSB0, as dmesg and ls /dev/ report), whether I try using the front RS232 port pr the rear RJ-45 console port.

It appears as though the switch is working (despite it not serving DHCP) from the LEDs that remain on after initialization (PS1, PS2 and Stack ID 1), so I am really stumped here.

there's no rj45 console port on the fcx, that's the ethernet management port. for the front rs232 console port, you need a straight through serial cable. if you just have a usb to db9 serial cable, it will be male, and the rs232 on the fcx is male, so I typically just use a female to female couple (that is straight through, NOT null modem): https://www.amazon.com/WOVTE-Changer-Coupler-Adapter-Connector/dp/B00W7BZ31G/

good luck with the noise the poe fcx is the loudest switch I've ever heard

 

[moriel5]

there's no rj45 console port on the fcx, that's the ethernet management port. for the front rs232 console port, you need a straight through serial cable. if you just have a usb to db9 serial cable, it will be male, and the rs232 on the fcx is male, so I typically just use a female to female couple (that is straight through, NOT null modem): https://www.amazon.com/WOVTE-Changer-Coupler-Adapter-Connector/dp/B00W7BZ31G/

good luck with the noise the poe fcx is the loudest switch I've ever heard

That is what I had meant. RJ-45 is just the connector.
And thanks for the link, however I already have a female to female RS232 cable, which is what I had attempted to use, would there be any benefit to using a male to female (I also have one like that) or male to male cable with the couplers over what I have already tried?

And thanks, but while noticeably loud, the FCX648-HPOE's PSU fans are nowhere near as loud as what it seems like from the comments here (it certainly drowns completely my desktop's fans, understandably), I could easily sleep with it working right outside my window (and I generally listen to music at levels quiet enough that others can't make out the sounds, so this is not due to bad hearing).

 

[moriel5]

Alright, I have no idea whether I am doing something wrong, or that all of my RS232 hardware is faulty, however I am unable to get any console output with the CH340G USB to RS232 adapter and an RS232 to RJ-45 adapter, even with the Cisco router (the CH340G USB to RJ45 cable is working fine with the same Cisco router, however I am getting no output from the Brocade switch with it).

Add to that the fact that I have never been able to get serial over RS232 working with any hardware (I have successfully used serial over USB directly to contacts on boards, and now to RJ-45), and I have absolutely no idea anymore.

 

[Whattteva]

Update from my earlier post about the loud ICX 6450-48 (NO PoE) fan.

I bought this Sunon fan (MB40201V2) and it's working well and while it's not dead silent, it's a HUGE improvement over the stock fan and I'm pretty happy with it. Thermals still look solid (increase of 3 C). I would like to point out a couple issues for anyone else that may want to attempt this:

• Fan requires the two outer wires to be swapped to function properly. Without swapping, the fan will both spin very slowly and be detected as "failed" by the switch.
• The stock screws are too small for the new fan so you must buy new screws. You can either buy really short 10-24 screws or long 8-32 screws + nuts. I opted to go with long 8-32 + nuts since the 10-24 feels very very snug on the fan.

If you don't want the headache of dealing with swapping wires and screws, you can just go buy this fan instead for a more plug'n'play experience for a bit more money.

 

[fohdeesha]

Alright, I have no idea whether I am doing something wrong, or that all of my RS232 hardware is faulty, however I am unable to get any console output with the CH340G USB to RS232 adapter and an RS232 to RJ-45 adapter, even with the Cisco router (the CH340G USB to RJ45 cable is working fine with the same Cisco router, however I am getting no output from the Brocade switch with it).

Add to that the fact that I have never been able to get serial over RS232 working with any hardware (I have successfully used serial over USB directly to contacts on boards, and now to RJ-45), and I have absolutely no idea anymore.

you're not plugging the rj45 serial cable into the fcx are you? the fcx has no rj45 serial ports, the rj45 port on the back is an ethernet port

 

[Just2Bad]

Hello one quick question on th eicx6610 and the setup I want to do. , if I have 2x icx6610 and I have 4 proxmox hosts that I want to connect at 1x40gbe.

is it possible to stack the 2 switch via the 2 ports that can be use as break out but at full 1x40gbe in stkaing mode and use non breakout port to connect to the proxmox hosts ?

I want full 1x 40gbe to the host I want to avoid bond on the proxmox hosts cause you never get the 40gbe form a single thread connection



or would I end up with 4x10 + 4x10 for the staking link on the right ? can those be 1x40 + 1x40 in staking mode if there's one ?

 

[fohdeesha]

Hello one quick question on th eicx6610 and the setup I want to do. , if I have 2x icx6610 and I have 4 proxmox hosts that I want to connect at 1x40gbe.

is it possible to stack the 2 switch via the 2 ports that can be use as break out but at full 1x40gbe in stkaing mode and use non breakout port to connect to the proxmox hosts ?

I want full 1x 40gbe to the host I want to avoid bond on the proxmox hosts cause you never get the 40gbe form a single thread connection

View attachment 28153

or would I end up with 4x10 + 4x10 for the staking link on the right ? can those be 1x40 + 1x40 in staking mode if there's one ?

as the guide said, those two ports are fixed 4x10gbe, even in stacking mode

 

[Just2Bad]

as the guide said, those two ports are fixed 4x10gbe, even in stacking mode

thanks I wanted to be sure and have a confirmation form someone else before doing the move and buy a 2nd one, I'll will find another solution

 

[NablaSquaredG]

is it possible to stack the 2 switch via the 2 ports that can be use as break out but at full 1x40gbe in stkaing mode and use non breakout port to connect to the proxmox hosts ?

If you find out how, let me know. I've tried it multiple times, but never got it to work. I doubt it is possible.

You can either stack using the 40G non-breakout ports, or the 40G non-breakout + 40G breakout together, but not only with the 40G breakout ports... Or maybe I'm stupid?

 

[itronin]

You can either stack using the 40G non-breakout ports, or the 40G non-breakout + 40G breakout together, but not only with the 40G breakout ports... Or maybe I'm stupid?

I have only tried a two switch setup and I have only been successful with the first two scenarios. I may also be stupid though - WK?

 

[Just2Bad]

we all stupid lolll, but thanks for feedback now I'm sure it's not possible
....

 

[NablaSquaredG]

I still hope that @fohdeesha will do some magic and come up with a guide that allows the third scenario (stacking only with the 40G breakout ports)

 

[Just2Bad]

I still hope that @fohdeesha will do some magic and come up with a guide that allows the third scenario (stacking only with the 40G breakout ports)

would have been nice for sure, now I have to find another switch with more 40gbe ports to stake with my 6610

 

[DerTom]

Hi!

During configuration of my new ICX6610 I do get the following error:

ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files.........
error in reading i2c device 1
eeprom_read: reading i2c device 1 failed
error in writing i2c device 1, error code: 5
eeprom_write: writing i2c device 1 failed
error in reading i2c device 1
ONBOARD_EEPROM read failure 1.
Backplane MAC address is not configured.
Use default 0004.80a0.4000
BOOT INFO: load image from primary copy...

error in reading i2c device 1, error code 7
eeprom_read: reading i2c device 1 failed
error in writing i2c device 1, error code: 5
eeprom_write: writing i2c device 1 failed
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
..............................
Firmware integrity checksum passed
.......
Starting Main Task .------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - chow_led_ch_mux_enable, error in seting up mux
..CPSS DxCh Version: cpss3.4p1 release
------------------------------------------------------------------
M:9 L:0 - PS EEPROM I2C Open: i2c cmd failed, error 7, ps 1
------------------------------------------------------------------
M:9 L:0 - PS EEPROM Offset Read: eeprom mux open error, ps 2
Fatal Error: No FANTRAY fan is present or fan has failed
System is shutting down!!!
Rebooting(0)...

The device is on a bootloop...

The led of the power supply is on but the fan is still 'spinning'!? As there is the message 'No FANTRAY fan present' I assume there is something wrong with the fan... Is this correct?

And most of all which way do I get the device to boot properly?

Kind regards,

tom

 

[frogtech]

Checking in after a while, has anyone made a final determination on the best fans for icx7250 24 port poe? I've tried a couple diff but they're still the loudest item in my rack. The hard part is keeping the switch below 60c still. I think the best solution someone posted was using a heat pipe and some extra heatsink attached.

 

[MannyLNJ]

I recently acquired a FCX648S-HPOE and need to know what is the correct part to get to add a QSFP+ Port so I can add 2.5GB Ethernet ports? I already purchased a XFP module which I believe is the wrong item

 

[DerTom]

Hi!

During configuration of my new ICX6610 I do get the following error:



The device is on a bootloop...

The led of the power supply is on but the fan is still 'spinning'!? As there is the message 'No FANTRAY fan present' I assume there is something wrong with the fan... Is this correct?

And most of all which way do I get the device to boot properly?

Kind regards,

tom

I do have replaced the power supply but there is still this error and the device on a bootloop... Mr. Google doesn't seem to know how to get rid of this.

If you have any hints, let me know, please.

 

[msg7086]

Just bought a 6450-48p on ebay with a $100 offer. They seem to have more than 100 units available. I'm not linking to it to reduce STH effect but it should be very easy to find. In case anyone is looking to buy one.

 

[oxoyoz]

Hi I got an ICX6610-48 recently and want to know:

1. Is it beneficial to upgrade its DDR2 memory from 512M to 1G/2G stick? These memory sell for 5 bucks on Ebay including shipping. But I am not sure it’s better used for a beer instead.

2. How much idle power can it reduce if I remove its 40gbps QSFP module on its back? I don’t have a use case for it yet. Granted I would lose the MGMT ports too but after initial setup I can SSH on the front ports to access it. Trying to save on utility bills a bit.

Thanks.

 

[NablaSquaredG]

These memory sell for 5 bucks on Ebay including shipping.

Are you sure? iirc the DIMMs are ECC SODIMMs and they're very hard to come by (I contemplated a memory upgrade for my SX6036)
NVM, according to @fohdeesha they're RDIMMs, and some pages ago he posted a link to this offer Centon Lot of 2 RD703G01 2GB PC2-5300 Reg ECC CL5 DDR2 667Mhz 1 Rank SODIMM 6-2 | eBay

2. How much idle power can it reduce if I remove its 40gbps QSFP module on its back? I don’t have a use case for it yet. Granted I would lose the MGMT ports too but after initial setup I can SSH on the front ports to access it. Trying to save on utility bills a bit.

Just a wild guess, but probably not much (if any power saving at all). Maybe it would make more sense to get an ICX6450-48 and sell the ICX6610?