Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

infoMatt

Active Member
Apr 16, 2019
183
77
28
I can't remember exactly where I read it... a post stating the 1kW power supplies are really 250W (switch) + 750W (PoE), and not 1kW (shared). Does that sound right? And will the RPS16 will work just fine in a non-PoE switch?
Yes, they are a dual supply, a 12V 250W for the board itself and a ~51V 750W for the PoE devices.
A PoE capable supply should work on a non-PoE board, but I think it's wiser to ask @fohdeesha

EDIT: Yay, too slow... ;)
 

liberty

New Member
Nov 4, 2019
4
2
3
Good find. I'm assuming you mean this guy?



I was wondering what that's for, A while back I saw the psu_sense silkscreen on the PCB and figured it was just a simple presence sense for the optional external power supply or something. Missed the T. Interestingly it's not present at all or even populated on the PCB on the non-PoE models, I guess they don't monitor PSU temps at all (they'd have no way to now that I look at the internals, there's only 2 wires coming off the PSU itself, 12V DC)
Yep, that's the one. Here's another view of the connector nearest the ASIC.

20200821_102233.jpg
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
Console Cable Question: would a console cable from 7250 work on a ixc7150-c12p ?
Nope, out of sheer contempt they had to change it yet again, and on the 7150 it has an actual usb-c port that speaks actual USB, so if you have a usb-c cable you can plug it directly into your PC. The good news is it also has a regular oldschool rj45 serial port, like the 6610 and 6450 etc, so you can use that
 

tommybackeast

Active Member
Jun 10, 2018
253
83
28
Nope, out of sheer contempt they had to change it yet again, and on the 7150 it has an actual usb-c port that speaks actual USB, so if you have a usb-c cable you can plug it directly into your PC. The good news is it also has a regular oldschool rj45 serial port, like the 6610 and 6450 etc, so you can use that
Thanks. My 6450 "normal old--school" RJ45 console cable has broken; do have a 7250 console cable which you indicate wont work; but great to know a USB-C cable will work. Of course, there's many different kinds of USB-C cables :(

Some days I miss a basic Serial Port and DVI , lol
 

LodeRunner

Member
Apr 27, 2019
67
34
18
A standard USB-C data cable will do it. The 7150-C12 has the USB to serial device inside. The 7250's console port was a mini-USB wired for direct serial instead of being an actual USB device. Plugging a regular USB cable into a 7250 will most likely fry the console port.
 
  • Like
Reactions: tommybackeast

tommybackeast

Active Member
Jun 10, 2018
253
83
28
A standard USB-C data cable will do it. The 7150-C12 has the USB to serial device inside. The 7250's console port was a mini-USB wired for direct serial instead of being an actual USB device. Plugging a regular USB cable into a 7250 will most likely fry the console port.
Thank you - I confess to being ignorant of all the different types of USB-c Cables.

Is a USB-C charging cable the same as a USB-C data cable?
 

dodgy route

Member
Aug 12, 2020
9
26
13
Australia
Well like we discussed in PM (at least I'm pretty sure that was you), even if it's another rev a, having two of them plugged in at the same time will make it quiet wayyy down. Rev A's only stay in full speed if there's 1 PSU active - with 2 psu's they behave like any other revision. I honestly dunno how anyone can stand the full boot fan speed. You only need rev b or rev c if you want it to quiet down with a single PSU plugged in

I know it would be quite easy to overwrite the revision string in the PSU EEPROM so the switch thinks it's a rev B and runs it at normal slow speed even with 1 PSU, but then I realized that probably wouldn't be a good guide to post. Like everything else on this switch, there's probably a very good reason those revisions run at a full speed unless there's two of them - the switch relies mainly on the PSU's for chassis cooling, the revision A psu's probably had a slower fan model or something so it runs them as fast as it can unless there's two of them working together
speaking of icx6610 noise, I think I had an epiphany. my 6610's have always been quiet, certainly not loud enough to hear through a closet door let alone in another room like some of you are reporting. I thought a lot of you just had WILDLY different definitions of "loud"

However I just picked up another ICX6610, and it's LOUD. From what I can tell, it's the rev A power supplies. That's the only thing different in this chassis from the other 5 or 6 I have

As previously mentioned, with Rev A's it won't even spin down after boot unless 2 are plugged in - but I realized even after they do spin down, their definition of "idle" is at least twice as loud (if not more) than my other ICX6610's with rev B and rev C supplies. It seems no matter how you cut it the rev A supplies are loud

That would certainly explain the huge range of reactions people have had to the 6610 noise level. If you have rev a supplies and want to make it twice as quiet, search around ebay for rev b or rev c (make sure their airflow is in the right direction)
I am not sure if I missed it but my rev A does drop after it loads (2.5 minutes? from powering on).

Before I continue doing much more to the switch or consider hacking it big fans (to reduce noise, not airflow) and make controllers..
Is the rev B/C PSU on 6610-48P quieter then rev A once the fans spin down?

Maybe I cant tell what has spun down actually, maybe the PSU are spinning loudly and it is only the fan tray that calms down?
I did my wrist in a bit so things are lightly on hold from any modding perspective so I am investigating PSU options.

While my voice sounds terrible on camera, here is a quick booted up vs running fan noise of my switch. Even through a shut solid door its loud in the next room/s after its loaded.
This is what I am trying to stop with much bigger fans that still spin enough to provide allllll the airflow. Would a single REV B/C be quieter?

I am also unable to find a REVB PSU for under 2/3 of the cost of what I picked up the switch, but maybe if someone gives me hope its quieter... may stop me making a smart controller heh

EDIT: Dont hate my "servers" setups... it will be changing shortly, including custom made rack mounted 5-6U cases to house huge fans for the i7 9700's
Based off Antec P101 Silent cases, here is one where all the metal work is completed, just the front remains, and then another 2 to do as soon as wrist is better...
1598517533941.png

Thanks!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
It's hard to tell from a video but they're definitely spinning way down...you can still hear it through a door? My posts you quoted about rev A supplies being way louder, if I recall correctly I ended up finding out that unit was much louder not because of the rev A PSUs, but because it was a reverse airflow (intake) model. Those models spin all the fans faster than the regular airflow version. I don't *think* a spun down rev A is quieter than a spun down rev B, but I could be wrong
 
  • Like
Reactions: dodgy route

klui

Active Member
Feb 3, 2019
194
82
28
I offer one data point from my 6610-48P with rev B and C PSUs. https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-172#post-262048

My phone was roughly 6-8 inches away from the rear of the switch, right behind the cluster of QSFP ports in the rear, maybe 5" above the switch. The phone was on top of my monitor where I could read the values from the app. The microphone, at the bottom edge of the phone, was facing away from the switch.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
Got bored and did some more digging through source code, found some interesting (but more or less useless) information. If you remember the "no password" command you can run in the bootloader that makes it temporarily boot without a user password, this is the routine it runs: Paste ofCode

It's just reading the value stored in the chassis EEPROM at offset 8, and adding 0x1000000 to it. This is the same EEPROM that stores the chassis MAC, serial, license ID, etc. Booting into the hidden dev bootloader mode so I had access to i2c commands, I could watch the EEPROM value change:

Code:
ICX64XX-boot>> i2cread 52 8 1 4
Read: dev_addr=0x52, addr=0x8, data=0x80000000, (al=1, dl=4)

ICX64XX-boot>> no password
OK! Skip password check when the system is up.

ICX64XX-boot>> i2cread 52 8 1 4
Read: dev_addr=0x52, addr=0x8, data=0x81000000, (al=1, dl=4)
Then when linux boots, then launches the fastiron binary, the fastiron binary has a routine that checks this EEPROM offset for certain values. if it sees that value, it temporarily removes password checks. Interestingly while digging around I found a hidden command I haven't seen mentioned anywhere, "use default-configuration" - Paste ofCode

this works the exact same way as the "no password" routine above, except it adds 0x800000 to that EEPROM location instead. And again, after boot the fastiron binary checks this location, and if it sees this value, runs with the default configuration. Kind of redundant though, as we already have the "factory set-default" command, which does the same thing but is more thorough. Here's the factory set-default routine: Paste ofCode

This one is even simpler, it just sets a u-boot environment variable named fips_reset to a value of "fipsreset" (the value is usually blank). In u-boot this is one of many environment variables that get appended to the boot argument that gets sent to linux (/proc/cmdline). So by default when it's blank, nothing is appended, but when it's set, the word "fipsreset" is appended to what gets passed to /proc/cmdline - then if we look in the linux OS, at the main init script that brings the system and then fastiron up, we see a routine that specifically searches for "fipsreset" in the cmdline, and if it's found, deletes pretty much everything (except licenses) - Paste ofCode
 

safado

New Member
Aug 21, 2020
18
0
1
Thanks to this incredible thread and the wealth of information! Just took delivery of a ICX6610-48p to replace a Cisco 3750-48 POE switch and now looking at transceivers for purchase. I see that FS.com has lots of options at decent/good prices. I work in IT but not on the networking side that much--always used manufacturers GBIC's. I'm assuming I'll see no difference in performance/capabilities with a compatible off brand? Or would I be better off just grabbing some used Brocade 57-00000075-10 SFP+ from ebay? Anyone know an equivalent part number for a 10Gbase-T module? I would like to attempt a 10G connection to my workstation over an existing Cat5e run (maybe 25-40M long) just for fun--can you recommend me a PCIe card that would be best to attempt this with?

Thanks again to the OP for this incredible resource! Looking forward to 10Gb between my servers!
 

Spearfoot

Member
Apr 22, 2015
68
26
18
62
Just took delivery of a ICX6610-48p to replace a Cisco 3750-48 POE switch and now looking at transceivers for purchase.
These switches don't seem to be very picky about transceivers. One of advantage of using 'supported' transceivers, though, would be optical monitoring. I use these cheap $8 Finisar transceivers in my 6610-48P, and they work fine:
 

Wesumat

New Member
Oct 16, 2019
3
1
3
Germany
...
Anyone know an equivalent part number for a 10Gbase-T module? I would like to attempt a 10G connection to my workstation over an existing Cat5e run (maybe 25-40M long) just for fun--can you recommend me a PCIe card that would be best to attempt this with?

Thanks again to the OP for this incredible resource! Looking forward to 10Gb between my servers!
Hi,

i‘m using this one from fs.com without any issues in an ICX6450 to connect to my PC which is equipped with an Intel X550.

fs.com - 10G-SFPP-T
 
  • Like
Reactions: 3nodeproblem

blinkenlights

Active Member
May 24, 2019
143
64
28
Anyone know an equivalent part number for a 10Gbase-T module? I would like to attempt a 10G connection to my workstation over an existing Cat5e run (maybe 25-40M long) just for fun--can you recommend me a PCIe card that would be best to attempt this with?
I am using this transceiver with an Intel X550-T1 on the other end:
Code:
ICX7450-24 Switch>sh med eth 1/4/2  
Port   1/4/2: Type  : 10GE SR 300m (SFP+)
             Vendor: UBNT               Version: 2   
             Part# : UF-RJ45-10G        Serial#: XXXXXXXXXXXX
Around $50 from reputable sellers.

My original plan was to use a Chelsio SFP+ card with an SR transceiver... unfortunately, the Linux kernel would panic when trying to suspend/resume with the Chelsio cards (including RJ45 versions). In my experience, the Intel cards just work.
 

klui

Active Member
Feb 3, 2019
194
82
28
This one is even simpler, it just sets a u-boot environment variable named fips_reset to a value of "fipsreset" (the value is usually blank). In u-boot this is one of many environment variables that get appended to the boot argument that gets sent to linux (/proc/cmdline). So by default when it's blank, nothing is appended, but when it's set, the word "fipsreset" is appended to what gets passed to /proc/cmdline - then if we look in the linux OS, at the main init script that brings the system and then fastiron up, we see a routine that specifically searches for "fipsreset" in the cmdline, and if it's found, deletes pretty much everything (except licenses) - Paste ofCode
I was going to ask you the difference between set-default vs setting fips_reset but they're just the same thing. set-default just calls the function that sets fips_reset.

I guess these things you posted about won't work on a 6610 because it's not based on Linux?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,823
113
29
fohdeesha.com
I was going to ask you the difference between set-default vs setting fips_reset but they're just the same thing. set-default just calls the function that sets fips_reset.

I guess these things you posted about won't work on a 6610 because it's not based on Linux?
the commands all work on the 6610, it's just implemented slightly differently (well, except for " use default-configuration ", haven't tried that one). If you remember the ICX6610 guide still has you use "factory set-default" to wipe the thing.

regarding manually setting the fips_reset environment variable, I tried that a few times to wipe it so it wouldn't actually wipe my config on boot, and I noticed the variable wouldn't actually change even though the setenv command took just fine. After looking, it's because they have a routine specifically to block altering the fips_reset variable directly:

Code:
+    /* user is not allowed to set fips related environment variables */
+    if( check_setenv && ( (strncmp(name, "fips_status", 11) == 0) || (strncmp(name, "fips_reset", 10) == 0) ) )
+    {
+        return 1;
+    }
 
Last edited:
  • Like
Reactions: klui