Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[tommybackeast]

Nope, out of sheer contempt they had to change it yet again, and on the 7150 it has an actual usb-c port that speaks actual USB, so if you have a usb-c cable you can plug it directly into your PC. The good news is it also has a regular oldschool rj45 serial port, like the 6610 and 6450 etc, so you can use that

Thanks. My 6450 "normal old--school" RJ45 console cable has broken; do have a 7250 console cable which you indicate wont work; but great to know a USB-C cable will work. Of course, there's many different kinds of USB-C cables

Some days I miss a basic Serial Port and DVI , lol

 

[LodeRunner]

A standard USB-C data cable will do it. The 7150-C12 has the USB to serial device inside. The 7250's console port was a mini-USB wired for direct serial instead of being an actual USB device. Plugging a regular USB cable into a 7250 will most likely fry the console port.

 

[tommybackeast]

A standard USB-C data cable will do it. The 7150-C12 has the USB to serial device inside. The 7250's console port was a mini-USB wired for direct serial instead of being an actual USB device. Plugging a regular USB cable into a 7250 will most likely fry the console port.

Thank you - I confess to being ignorant of all the different types of USB-c Cables.

Is a USB-C charging cable the same as a USB-C data cable?

 

[LodeRunner]

Not always, and boy ever does that make things fun.

Since your serial cable for the 6450 is dead, you might consider buying a Cisco DB9 > RJ45 serial cable since it will work with the 6450 and the 7150.

 

[dodgy route]

Well like we discussed in PM (at least I'm pretty sure that was you), even if it's another rev a, having two of them plugged in at the same time will make it quiet wayyy down. Rev A's only stay in full speed if there's 1 PSU active - with 2 psu's they behave like any other revision. I honestly dunno how anyone can stand the full boot fan speed. You only need rev b or rev c if you want it to quiet down with a single PSU plugged in

I know it would be quite easy to overwrite the revision string in the PSU EEPROM so the switch thinks it's a rev B and runs it at normal slow speed even with 1 PSU, but then I realized that probably wouldn't be a good guide to post. Like everything else on this switch, there's probably a very good reason those revisions run at a full speed unless there's two of them - the switch relies mainly on the PSU's for chassis cooling, the revision A psu's probably had a slower fan model or something so it runs them as fast as it can unless there's two of them working together

speaking of icx6610 noise, I think I had an epiphany. my 6610's have always been quiet, certainly not loud enough to hear through a closet door let alone in another room like some of you are reporting. I thought a lot of you just had WILDLY different definitions of "loud"

However I just picked up another ICX6610, and it's LOUD. From what I can tell, it's the rev A power supplies. That's the only thing different in this chassis from the other 5 or 6 I have

As previously mentioned, with Rev A's it won't even spin down after boot unless 2 are plugged in - but I realized even after they do spin down, their definition of "idle" is at least twice as loud (if not more) than my other ICX6610's with rev B and rev C supplies. It seems no matter how you cut it the rev A supplies are loud

That would certainly explain the huge range of reactions people have had to the 6610 noise level. If you have rev a supplies and want to make it twice as quiet, search around ebay for rev b or rev c (make sure their airflow is in the right direction)

I am not sure if I missed it but my rev A does drop after it loads (2.5 minutes? from powering on).

Before I continue doing much more to the switch or consider hacking it big fans (to reduce noise, not airflow) and make controllers..
Is the rev B/C PSU on 6610-48P quieter then rev A once the fans spin down?

Maybe I cant tell what has spun down actually, maybe the PSU are spinning loudly and it is only the fan tray that calms down?
I did my wrist in a bit so things are lightly on hold from any modding perspective so I am investigating PSU options.

While my voice sounds terrible on camera, here is a quick booted up vs running fan noise of my switch. Even through a shut solid door its loud in the next room/s after its loaded.
This is what I am trying to stop with much bigger fans that still spin enough to provide allllll the airflow. Would a single REV B/C be quieter?

I am also unable to find a REVB PSU for under 2/3 of the cost of what I picked up the switch, but maybe if someone gives me hope its quieter... may stop me making a smart controller heh

EDIT: Dont hate my "servers" setups... it will be changing shortly, including custom made rack mounted 5-6U cases to house huge fans for the i7 9700's
Based off Antec P101 Silent cases, here is one where all the metal work is completed, just the front remains, and then another 2 to do as soon as wrist is better...


Thanks!

 

[fohdeesha]

It's hard to tell from a video but they're definitely spinning way down...you can still hear it through a door? My posts you quoted about rev A supplies being way louder, if I recall correctly I ended up finding out that unit was much louder not because of the rev A PSUs, but because it was a reverse airflow (intake) model. Those models spin all the fans faster than the regular airflow version. I don't *think* a spun down rev A is quieter than a spun down rev B, but I could be wrong

 

[klui]

I offer one data point from my 6610-48P with rev B and C PSUs. https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-172#post-262048

My phone was roughly 6-8 inches away from the rear of the switch, right behind the cluster of QSFP ports in the rear, maybe 5" above the switch. The phone was on top of my monitor where I could read the values from the app. The microphone, at the bottom edge of the phone, was facing away from the switch.

 

[fohdeesha]

Got bored and did some more digging through source code, found some interesting (but more or less useless) information. If you remember the "no password" command you can run in the bootloader that makes it temporarily boot without a user password, this is the routine it runs: Paste ofCode

It's just reading the value stored in the chassis EEPROM at offset 8, and adding 0x1000000 to it. This is the same EEPROM that stores the chassis MAC, serial, license ID, etc. Booting into the hidden dev bootloader mode so I had access to i2c commands, I could watch the EEPROM value change:

ICX64XX-boot>> i2cread 52 8 1 4
Read: dev_addr=0x52, addr=0x8, data=0x80000000, (al=1, dl=4)

ICX64XX-boot>> no password
OK! Skip password check when the system is up.

ICX64XX-boot>> i2cread 52 8 1 4
Read: dev_addr=0x52, addr=0x8, data=0x81000000, (al=1, dl=4)

Then when linux boots, then launches the fastiron binary, the fastiron binary has a routine that checks this EEPROM offset for certain values. if it sees that value, it temporarily removes password checks. Interestingly while digging around I found a hidden command I haven't seen mentioned anywhere, "use default-configuration" - Paste ofCode

this works the exact same way as the "no password" routine above, except it adds 0x800000 to that EEPROM location instead. And again, after boot the fastiron binary checks this location, and if it sees this value, runs with the default configuration. Kind of redundant though, as we already have the "factory set-default" command, which does the same thing but is more thorough. Here's the factory set-default routine: ICX7150 u-boot factory default routine

This one is even simpler, it just sets a u-boot environment variable named fips_reset to a value of "fipsreset" (the value is usually blank). In u-boot this is one of many environment variables that get appended to the boot argument that gets sent to linux (/proc/cmdline). So by default when it's blank, nothing is appended, but when it's set, the word "fipsreset" is appended to what gets passed to /proc/cmdline - then if we look in the linux OS, at the main init script that brings the system and then fastiron up, we see a routine that specifically searches for "fipsreset" in the cmdline, and if it's found, deletes pretty much everything (except licenses) - FI 8080b fips_reset routine (from /etc/init.sh)

 

[safado]

Thanks to this incredible thread and the wealth of information! Just took delivery of a ICX6610-48p to replace a Cisco 3750-48 POE switch and now looking at transceivers for purchase. I see that FS.com has lots of options at decent/good prices. I work in IT but not on the networking side that much--always used manufacturers GBIC's. I'm assuming I'll see no difference in performance/capabilities with a compatible off brand? Or would I be better off just grabbing some used Brocade 57-00000075-10 SFP+ from ebay? Anyone know an equivalent part number for a 10Gbase-T module? I would like to attempt a 10G connection to my workstation over an existing Cat5e run (maybe 25-40M long) just for fun--can you recommend me a PCIe card that would be best to attempt this with?

Thanks again to the OP for this incredible resource! Looking forward to 10Gb between my servers!

 

[Spearfoot]

Just took delivery of a ICX6610-48p to replace a Cisco 3750-48 POE switch and now looking at transceivers for purchase.

These switches don't seem to be very picky about transceivers. One of advantage of using 'supported' transceivers, though, would be optical monitoring. I use these cheap $8 Finisar transceivers in my 6610-48P, and they work fine:

Genuine FInisar FTLX8571D3BNL-E5 10GbE 850nm SFP+ Transceiver Module | eBay

Genuine FInisar FTLX8571D3BNL-. E5 10GbE 850nm SFP+ Transceiver Module.

www.ebay.com

 

[Wesumat]

...
Anyone know an equivalent part number for a 10Gbase-T module? I would like to attempt a 10G connection to my workstation over an existing Cat5e run (maybe 25-40M long) just for fun--can you recommend me a PCIe card that would be best to attempt this with?

Thanks again to the OP for this incredible resource! Looking forward to 10Gb between my servers!

Hi,

i‘m using this one from fs.com without any issues in an ICX6450 to connect to my PC which is equipped with an Intel X550.

fs.com - 10G-SFPP-T

 

[blinkenlights]

Anyone know an equivalent part number for a 10Gbase-T module? I would like to attempt a 10G connection to my workstation over an existing Cat5e run (maybe 25-40M long) just for fun--can you recommend me a PCIe card that would be best to attempt this with?

I am using this transceiver with an Intel X550-T1 on the other end:

ICX7450-24 Switch>sh med eth 1/4/2  
Port   1/4/2: Type  : 10GE SR 300m (SFP+)
             Vendor: UBNT               Version: 2   
             Part# : UF-RJ45-10G        Serial#: XXXXXXXXXXXX

Around $50 from reputable sellers.

My original plan was to use a Chelsio SFP+ card with an SR transceiver... unfortunately, the Linux kernel would panic when trying to suspend/resume with the Chelsio cards (including RJ45 versions). In my experience, the Intel cards just work.

 

[klui]

This one is even simpler, it just sets a u-boot environment variable named fips_reset to a value of "fipsreset" (the value is usually blank). In u-boot this is one of many environment variables that get appended to the boot argument that gets sent to linux (/proc/cmdline). So by default when it's blank, nothing is appended, but when it's set, the word "fipsreset" is appended to what gets passed to /proc/cmdline - then if we look in the linux OS, at the main init script that brings the system and then fastiron up, we see a routine that specifically searches for "fipsreset" in the cmdline, and if it's found, deletes pretty much everything (except licenses) - Paste ofCode

I was going to ask you the difference between set-default vs setting fips_reset but they're just the same thing. set-default just calls the function that sets fips_reset.

I guess these things you posted about won't work on a 6610 because it's not based on Linux?

 

[fohdeesha]

I was going to ask you the difference between set-default vs setting fips_reset but they're just the same thing. set-default just calls the function that sets fips_reset.

I guess these things you posted about won't work on a 6610 because it's not based on Linux?

the commands all work on the 6610, it's just implemented slightly differently (well, except for " use default-configuration ", haven't tried that one). If you remember the ICX6610 guide still has you use "factory set-default" to wipe the thing.

regarding manually setting the fips_reset environment variable, I tried that a few times to wipe it so it wouldn't actually wipe my config on boot, and I noticed the variable wouldn't actually change even though the setenv command took just fine. After looking, it's because they have a routine specifically to block altering the fips_reset variable directly:

+    /* user is not allowed to set fips related environment variables */
+    if( check_setenv && ( (strncmp(name, "fips_status", 11) == 0) || (strncmp(name, "fips_reset", 10) == 0) ) )
+    {
+        return 1;
+    }

 

[klui]

The sneaky bastards! Enforcing FIPS policies. Where's a good backdoor when you need it?

 

[sowilo]

Hello,

My ICX 6610 is failing to boot up.

All the SFP+ lights are staying solid orange, and the fans are spinning at max like it normally does when it first boots up. I've waited at least an hour for it to boot up but nothing changes.

Plugging on console it gets to the press b to stop at boot monitor prompt and then nothing happens after that. I can stop at memory test and get into that prompt, but stopping at the boot monitor doesn't work.

This started after something happened with my UPS during the night (replace battery light was on, which went away after a self test, 50% of the battery charge gone, no power outage at the time of the incident.), before that the switch was running fine for almost a year now.

I know its probably fried but I am reaching out in case this is a fixable issue so I don't have to buy another one.

 

[infoMatt]

My ICX 6610 is failing to boot up.

All the SFP+ lights are staying solid orange, and the fans are spinning at max like it normally does when it first boots up. I've waited at least an hour for it to boot up but nothing changes.


I know its probably fried but I am reaching out in case this is a fixable issue so I don't have to buy another one.

I think that it might have a corrupt flash and so it can't load the OS... @fohdeesha can you help sowilo diagnose any furthest?

 

[kache]

The ICX 6450 arrived today!
A few hours to get it updated, configured in the same way as the 2960S is replacing and installing the licenses (Thank you, @foohdeesha! ) and I swapped it in.

The fans will have to be replaced though, the noise is quite high even at idle. Tempted to get the Noctua as I usually do but the feedback here was quite negative on them so guess I'll scour ebay for Sunon fans.

Now, time to hit Google to find out how to configure an ipv6 helper to get dhcpv6-pd working from the Edgerouter!

EDIT:

Actually, it doesn't seem to be possible:
"SSH@10G(config-if-e1000-1/1/33)#ipv6 dhcp-relay
Invalid input -> dhcp-relay
Type ? for a list"

Is this possible only on the new devices?

Commscope Technical Content Portal

docs.ruckuswireless.com

EDIT2:

Apparently not:
SW: Version 08.0.30tT313

DHCPv6 relay agent​

​

08.0.01​

Commscope Technical Content Portal

docs.ruckuswireless.com

Par contre Prefix Delegation is on No:

DHCPv6 prefix delegation notification​

No​

No​

EDIT Final?

"DHCPv6 Server"

"DHCPv6 is supported on the following Ruckus ICX platforms for both Layer 2 and Layer 3 software images:

• Ruckus ICX 7150
• Ruckus ICX 7250
• Ruckus ICX 7450
• Ruckus ICX 7650
• Ruckus ICX 7750
• Ruckus ICX 7850"

Which basically means that if I want to have ipv6 I have to get one of the new switches. Pity since they're so expensive here in EU.

 

[fohdeesha]

Are you just trying to relay IPv6 DHCP to an actual ipv6 DHCP server? that's doable on 8030:

interface ve 10
 ip address 192.168.1.1 255.255.255.0
 ip helper-address 1 172.16.110.2
 ipv6 address 2001:470:c15e:8000::1/64
 ipv6 enable
 ipv6 dhcp-relay destination 2001:470:c15e::2
 ipv6 dhcp-relay include-options interface-id remote-id
 ipv6 nd managed-config-flag

if you want to do prefix delegation from the ICX itself, yeah that's gunna be an issue

 

[kache]

Are you just trying to relay IPv6 DHCP to an actual ipv6 DHCP server? that's doable on 8030:

interface ve 10

ip address 192.168.1.1 255.255.255.0

ip helper-address 1 172.16.110.2

ipv6 address 2001:470:c15e:8000::1/64

ipv6 enable

ipv6 dhcp-relay destination 2001:470:c15e::2

ipv6 dhcp-relay include-options interface-id remote-id

ipv6 nd managed-config-flag

if you want to do prefix delegation from the ICX itself, yeah that's gunna be an issue

It doesn't seem to be possible even with SLAAC though:

Brocade:

interface ve 1
 ip address 192.168.2.254 255.255.255.0
 ipv6 address some:thing:else:3901::2/64
 ipv6 enable
!
interface ve 10
 ip address 192.168.10.1 255.255.255.0
 ip helper-address 1 192.168.10.31
 ip helper-address 2 192.168.10.21
 ipv6 address some:thing:else:3902::2/64
 ipv6 enable
 ipv6 dhcp-relay destination some:thing:else:3901::2
 ipv6 dhcp-relay include-options interface-id remote-id
 ipv6 nd managed-config-flag
!
interface ve 20
 ip address 192.168.20.1 255.255.255.0
 ip helper-address 1 192.168.10.31
 ip helper-address 2 192.168.10.21
 ipv6 address some:thing:else:3903::3/64
 ipv6 enable
 ipv6 dhcp-relay destination some:thing:else:3901::2
 ipv6 dhcp-relay include-options interface-id remote-id
 ipv6 nd managed-config-flag

Edgerouter:

ubnt@EdgeRouter-Pro-8-Port# show
 no-dns
 pd 0 {
     interface eth1 {
         host-address :1
         no-dns
         prefix-id 1
         service slaac
     }
     prefix-length 56
 }
 prefix-only
 rapid-commit enable

The problem is that the edgerouter is only assigning a /64 to the interface so it doesn't identify the presence of the other subnets on the other side:

ubnt@EdgeRouter-Pro-8-Port:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                        S/L  Description
---------    ----------                        ---  -----------
eth0         -                                 u/u  Internet (PPPoE)
eth1         192.168.2.1/24                    u/u  Local
             some:thing:else:3901::1/64
eth2         -                                 u/D
eth3         -                                 u/D
eth4         -                                 u/D
eth5         -                                 u/D
eth6         -                                 u/D
eth7         -                                 u/D
lo           127.0.0.1/8                       u/u
             ::1/128
pppoe0       some.thing.v4                     u/u
             some:thing:v6/64

Funny thing is, pinging from default interface works:

SSH@10G#ping ipv6 some:thing:else:3901::1
Sending 1, 16-byte ICMPv6 Echo to some:thing:else:3901::1
timeout 5000 msec, Hop Limit 64
Type Control-c to abort
Reply from some:thing:else:3901::1: bytes=16 time<1ms Hop Limit=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.
SSH@10G#

But not from the ve 10:

SSH@10G#ping ipv6 some:thing:else:3901::1 source some:thing:else:3902::2
Sending 1, 16-byte ICMPv6 Echo to some:thing:else:3901::1
timeout 5000 msec, Hop Limit 64
Type Control-c to abort
Request timed out.
No reply from remote host.

And that's with the static route configured:

ipv6 unicast-routing
ipv6 route ::/0 some:thing:else:3901::1

There is something missing but I can't wrap around my head on what.