Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

PGlover

Active Member
Nov 8, 2014
499
64
28
57
The fact that it worked on the guest VLAN (which is still going through the brocade) 100% rules out STP, and means it's most likely a DHCP issue. The DHCP implementation on the 6610 is 100% RFC 2131 compliant (and I've used it with hundreds of devices), so either the sonos DHCP implementation is broken, or you have a misconfig somewhere

My first guess would be it doesn't like getting handed a gateway address that isn't the first usable IP in the subnet (your DHCP config has it handing out 192.168.1.2 for a gateway instead of 192.168.1.1). A compliant ethernet device should have no issue with this, but again we're talking IoT stuff

does the VE interface in that vlan actually have an IP of 192.168.1.2? because that's the gateway your DHCP config is giving out to clients. In the config you sent me in pm a few days ago the VE had an address of 192.168.1.1, so if you're telling devices to use 192.168.1.2, that's going to fail. Try changing it to 192.168.1.1, and change your DHCP config to hand out 192.168.1.1 for the default gateway.

The global STP config line is just because STP was previously enabled in a vlan, it doesn't turn STP on witohut a vlan config as well. if you reboot it'll go away from the config, but I wouldn't worry about it

After making these changes, turn all of the sonos off, and turn them on one at a time, starting with just the wired controller, and see if the wired controller gets an address
Let me clarify.. It did not work when using the guest VLAN. The Sonos device was able to get a DHCP address from the DHCP server which is my pfSense appliance for that Vlan, but it failed trying to add the device to the guest network. So, there is still something about the ICX 6610 that is not compatible with Sonos. Once again, I will do some further setting on my spare ICX 6610 in the next couple weeks.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,726
3,066
113
33
fohdeesha.com
Generic monoprice "cisco compatible" breakout DAC got here, and I've confirmed it works perfectly fine with the ICX6610 rear QSFP breakout ports when the ICX6610 is NOT in a stack. I have not had time yet to test with a stacking config as well. Given other reports, I would imagine ANY breakout DAC will work just fine

this one - 2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable 889028000663 | eBay

Also @PGlover you weren't kidding - comparing the QSFP connector on this cable to my others, they look identical, however this one did NOT want to come out on it's own. It took a little prying with a screwdriver (using the method I posted in a video a few pages back) but that did indeed get it free after a couple tries. it must be a millimeter thicker than usual or something

Code:
telnet@ICX5#sh int br e 1/2/2 to 1/2/5

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/2      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/3      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/4      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/5      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,726
3,066
113
33
fohdeesha.com
I noticed that with the monoprice DAC breakout, it was showing up as absolutely nothing with the "show media" command. Even though it doesn't affect the behavior of the links, I figured something was up with the I2C EEPROM in the DAC cable (this is where vendor/serial/cable type info is stored for optics and DACs).

You can see the show media command can't find any vendor or cable type info for the monoprice DAC, so it shows EMPTY for the 4 breakout slots:

Code:
telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : EMPTY
Port 1/2/3:  Type : EMPTY
Port 1/2/4:  Type : EMPTY
Port 1/2/5:  Type : EMPTY
So I dropped down to the debug console (over serial, press ctrl+y, let go, then press m, then enter) to use some I2C read and write commands.

First, let's read the EEPROM from a proper official brocade DAC cable:

Code:
OS>i2c read 3f 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......
You can see the first half is mostly empty, and the vendor information does not begin until the second half of the EEPROM, where you get the byte specifying the cable type, the vendor string, a serial number, etc. This is as it should be.

Now let's read the Monoprice DAC that doesn't show up properly:

Code:
OS>i2c read 41 0 256
0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g...
0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE .
0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 ..........
0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co
0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C
0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA   .........
0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
You can see they must have been in a rush programming these, as they stuck all the vendor and serial information in the incorrect section. No wonder it shows up as empty/unknown media type! Upon closer inspection, they programmed it like an SFP+ module. This first half of the EEPROM should be populated with these values like this in an SFP+, but for a QSFP+ module, it should be in the second half.

So I used a series of i2c write commands to copy byte for byte the contents of the official brocade DAC, to the monoprice DAC, overwriting the incorrectly placed vendor info with properly placed brocade info.

The monoprice DAC EEPROM contents now looks like this:
Code:
OS>i2c read 41 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......
Look familiar? Our monoprice DAC now has identical EEPROM contents, compared to our actual official brocade DAC.

And as expected, after unplugging and replugging the cable, the switch now ID's it as a DAC, and thinks it's an official brocade part:

Code:
telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : 40GE-Passive Copper
Port 1/2/3:  Type : 40GE-Passive Copper
Port 1/2/4:  Type : 40GE-Passive Copper
Port 1/2/5:  Type : 40GE-Passive Copper

##even more details - still the monoprice DAC
telnet@ICX3#show media e 1/2/2
Port   1/2/2:Type  : 40GBASE-Passive Copper
Vendor Name: BROCADE          Serial Num: PAE114103000876 Revision: A
I wouldn't think this has any effect on functionality, as I said the links worked perfectly fine beforehand. Re-coding it into a brocade part like this was only possible because the monoprice DAC did not have the EEPROM write protect bit set. This exact same procedure can be used to re-code optics to different vendors to bypass vendor locks, but many non-OEM optics have a bit set that prevents EEPROM writing without special unlock strings

A little more information:

The SFP+ EEPROM data location layout, you can see the identity and vendor etc strings are at the beginning (like our QSFP+ monoprice incorrectly had) - http://files.siemon.com/sis/application-guide/sfp-plus-to-sfp-plus_eeprom_summary.pdf

The QSFP+ EEPROM layout, you can see that the vendor and cable type etc now does not start until around the middle. This is where the ICX was trying to read to figure out the cable type, and of course on the monoprice there was nothing there - http://files.siemon.com/sis/application-guide/qsfpplus_to_qsfpplus_eeprom_summary.pdf

Upon closer inspection, this could have potentially caused it refusing to link up, as it could not even read the ident bit to figure out what kind of cable it was. For QSFP modules, it reads byte number 128 (first byte on line 80 in the dump above). This byte is set to "0d" hex to tell the host it's a QSFP+ device. On the monoprice dac, when it read that byte, it was completely empty. You can see in the brocade dump, that bit is correctly set to 0d.

here's a table of possible values for that byte:

https://i.imgur.com/2Id6xp9.png

You can see that if we set that byte to 03 instead, we could make the switch think it's an SFP module. It would probably not like this, given it's a QSFP+ slot (or on second thought, it would just assume it's an SFP+ optic in an SFP+ to QSFP+ adapter)

And upon even further digging, the EEPROM also contains signal attenuation values for the given cable at both 2.5ghz and 5ghz (bytes 186 and 187 respectively), and the host (the switch) uses these values to set up it's transmitter and receiver EQ appropriately. Without it being able to read these on the stock monoprice DAC, the transmitter and receiver probably defaulted to "fallback" values, which could have also resulted in the not quite working links.

Again, copying over the brocade DAC contents should fix this, as it contains these values (4db and 5db, respectively). I'd imagine the attenuation values there are very close to what the monoprice DAC is supposed to have, given they are both the same length, and both 30AWG twinax. Given how much stuff the monoprices come with wrong from factory, I'm surprised they ever worked, especially with cisco equipment (which is what they're advertised for). Thankfully it's not write-locked, so all this is easily fixed with a few commands
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,726
3,066
113
33
fohdeesha.com
@PGlover if you're interested in re-coding the monoprice DAC with proper values to see if it fixes the 1/2/2 link issue, follow these instructions (i'm curious to see if it fixes it)

Facing the rear of the switch, put the monoprice dac in the top right QSFP slot. (breakout port 1/2/2 - 1/2/5)

Connect to the switch over serial, and let it boot fully. When it boots fully, over serial, press ctrl+y, let go, press m, then enter. you should be at an OS console.

#first read the i2c EEPROM contents of the top right QSFP slot. this ensures you have it in the right place. if you'd like, copy and paste all the output of this command to a text file to back up the original eeprom contents. The output should have "monoprice" somewhere in it like the below example. To read it just run "i2c read 41 0 256"

Code:
OS>i2c read 41 0 256
0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g...
0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE .
0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 ..........
0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co
0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C
0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA   .........
0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
Now you can start writing. The monoprice dac seems to only take 8 byte writes at a time, so it's split up into a bunch of write commands in a row, copy/paste and run each line one at a time, should only take a couple minutes:

Code:
i2c write 41 0 0d00060000000000 1
i2c write 41 8 0000000000000000 1
i2c write 41 10 0000000000000181 1
i2c write 41 18 0000000000000000 1
i2c write 41 20 0000000000000000 1
i2c write 41 28 0000000000000000 1
i2c write 41 30 0000000000000000 1
i2c write 41 38 0000000000000000 1
i2c write 41 40 0000000000000000 1
i2c write 41 48 0000000000000000 1
i2c write 41 50 0000000000000000 1
i2c write 41 58 0000000000000000 1
i2c write 41 60 0000000000000000 1
i2c write 41 68 0000000000000000 1
i2c write 41 70 0000000000000000 1
i2c write 41 78 0000000000000000 1
i2c write 41 80 0d00230800000000 1
i2c write 41 88 0000000067000000 1
i2c write 41 90 000001a042524f43 1
i2c write 41 98 4144452020202020 1
i2c write 41 a0 202020200700051e 1
i2c write 41 a8 35382d3030303030 1
i2c write 41 b0 33332d3031202020 1
i2c write 41 b8 4120040500004608 1
i2c write 41 c0 0000000050414531 1
i2c write 41 c8 3134313033303030 1
i2c write 41 d0 3837362031343033 1
i2c write 41 d8 30352020000000c2 1
i2c write 41 e0 3131313034303930 1
i2c write 41 e8 3731202020202020 1
i2c write 41 f0 2031202020202020 1
i2c write 41 f8 2020000000000000 1
Now unplug the DAC and plug it back in. If you can't because it's still stuck, just reboot the switch. When it comes back up, run "show media e 1/2/2" - it should say you have a brocade passive copper link now, then see if all 4x breakouts work
 

PGlover

Active Member
Nov 8, 2014
499
64
28
57
@PGlover if you're interested in re-coding the monoprice DAC with proper values to see if it fixes the 1/2/2 link issue, follow these instructions (i'm curious to see if it fixes it)

Facing the rear of the switch, put the monoprice dac in the top right QSFP slot. (breakout port 1/2/2 - 1/2/5)

Connect to the switch over serial, and let it boot fully. When it boots fully, over serial, press ctrl+y, let go, press m, then enter. you should be at an OS console.

#first read the i2c EEPROM contents of the top right QSFP slot. this ensures you have it in the right place. if you'd like, copy and paste all the output of this command to a text file to back up the original eeprom contents. The output should have "monoprice" somewhere in it like the below example. To read it just run "i2c read 41 0 256"

Code:
OS>i2c read 41 0 256
0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g...
0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE .
0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 ..........
0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co
0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C
0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA   .........
0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
Now you can start writing. The monoprice dac seems to only take 8 byte writes at a time, so it's split up into a bunch of write commands in a row, copy/paste and run each line one at a time, should only take a couple minutes:

Code:
i2c write 41 0 0d00060000000000 1
i2c write 41 8 0000000000000000 1
i2c write 41 10 0000000000000181 1
i2c write 41 18 0000000000000000 1
i2c write 41 20 0000000000000000 1
i2c write 41 28 0000000000000000 1
i2c write 41 30 0000000000000000 1
i2c write 41 38 0000000000000000 1
i2c write 41 40 0000000000000000 1
i2c write 41 48 0000000000000000 1
i2c write 41 50 0000000000000000 1
i2c write 41 58 0000000000000000 1
i2c write 41 60 0000000000000000 1
i2c write 41 68 0000000000000000 1
i2c write 41 70 0000000000000000 1
i2c write 41 78 0000000000000000 1
i2c write 41 80 0d00230800000000 1
i2c write 41 88 0000000067000000 1
i2c write 41 90 000001a042524f43 1
i2c write 41 98 4144452020202020 1
i2c write 41 a0 202020200700051e 1
i2c write 41 a8 35382d3030303030 1
i2c write 41 b0 33332d3031202020 1
i2c write 41 b8 4120040500004608 1
i2c write 41 c0 0000000050414531 1
i2c write 41 c8 3134313033303030 1
i2c write 41 d0 3837362031343033 1
i2c write 41 d8 30352020000000c2 1
i2c write 41 e0 3131313034303930 1
i2c write 41 e8 3731202020202020 1
i2c write 41 f0 2031202020202020 1
i2c write 41 f8 2020000000000000 1
Now unplug the DAC and plug it back in. If you can't because it's still stuck, just reboot the switch. When it comes back up, run "show media e 1/2/2" - it should say you have a brocade passive copper link now, then see if all 4x breakouts work
I made the changes and still no communication on port 1/2/2 based on my stack configuration. I hope this problem is not unique to me based on my stack configuration.

SSH@ICX6610-48P Router#show media | include 1/2/[2-5]
Port 1/2/2: Type : 40GE-Passive Copper
Port 1/2/3: Type : 40GE-Passive Copper
Port 1/2/4: Type : 40GE-Passive Copper
Port 1/2/5: Type : 40GE-Passive Copper
 

kapone

Well-Known Member
May 23, 2015
1,095
641
113
I made the changes and still no communication on port 1/2/2 based on my stack configuration. I hope this problem is not unique to me based on my stack configuration.

SSH@ICX6610-48P Router#show media | include 1/2/[2-5]
Port 1/2/2: Type : 40GE-Passive Copper
Port 1/2/3: Type : 40GE-Passive Copper
Port 1/2/4: Type : 40GE-Passive Copper
Port 1/2/5: Type : 40GE-Passive Copper
Well, I guess we have no choice but back to basics, if we want to troubleshoot this. But you have a live environment...so, your call.

- NO ROUTER in this test. Use static IPs.
- NO VLANS for this test, we're just testing. Port connection testing does not require VLANs
- Take one 6610, reset it to factory default. Connect breakout cable to 1/2/2 and connect at least two 10g devices. Confirm that both work at 10g.
- Take a second 6610, reset it to factory default. Connect breakout cable to 1/2/2 and connect at least two 10g devices. Confirm that both work at 10g.
- Stack both of them on the 40G ports. Connect the two devices to 1/2/2 and see what happens. Connect them to 2/2/2 and see what happens.
 
  • Like
Reactions: fohdeesha

PGlover

Active Member
Nov 8, 2014
499
64
28
57
Generic monoprice "cisco compatible" breakout DAC got here, and I've confirmed it works perfectly fine with the ICX6610 rear QSFP breakout ports when the ICX6610 is NOT in a stack. I have not had time yet to test with a stacking config as well. Given other reports, I would imagine ANY breakout DAC will work just fine

this one - 2M Cisco QSFP-4SFP10G-CU2M Compatible QSFP+ to 4SFP+ Breakout Cable 889028000663 | eBay

Also @PGlover you weren't kidding - comparing the QSFP connector on this cable to my others, they look identical, however this one did NOT want to come out on it's own. It took a little prying with a screwdriver (using the method I posted in a video a few pages back) but that did indeed get it free after a couple tries. it must be a millimeter thicker than usual or something

Code:
telnet@ICX5#sh int br e 1/2/2 to 1/2/5

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/2/2      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/3      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/4      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
1/2/5      Up      Forward Full 10G   None  No  1    0   cc4e.2484.f218
I still have not been able to get the stuck QSFP+ out of the slot and this switch is the active controller in my stack. I would like to replace it in my production with the one of the 2 spare ICX 6610 I have. What is the best method to replace the active controller switch in my production setup with minimum interuption? Once again, I have 2 extra spares. Maybe the best option is to set those up and then move them into the production environment to replace the 2 currently in production. Just looking for the best option.
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
I noticed that with the monoprice DAC breakout, it was showing up as absolutely nothing with the "show media" command. Even though it doesn't affect the behavior of the links, I figured something was up with the I2C EEPROM in the DAC cable (this is where vendor/serial/cable type info is stored for optics and DACs).

You can see the show media command can't find any vendor or cable type info for the monoprice DAC, so it shows EMPTY for the 4 breakout slots:

Code:
telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : EMPTY
Port 1/2/3:  Type : EMPTY
Port 1/2/4:  Type : EMPTY
Port 1/2/5:  Type : EMPTY
So I dropped down to the debug console (over serial, press ctrl+y, let go, then press m, then enter) to use some I2C read and write commands.

First, let's read the EEPROM from a proper official brocade DAC cable:

Code:
OS>i2c read 3f 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......
You can see the first half is mostly empty, and the vendor information does not begin until the second half of the EEPROM, where you get the byte specifying the cable type, the vendor string, a serial number, etc. This is as it should be.

Now let's read the Monoprice DAC that doesn't show up properly:

Code:
OS>i2c read 41 0 256
0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g...
0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE .
0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 ..........
0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co
0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C
0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA   .........
0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
You can see they must have been in a rush programming these, as they stuck all the vendor and serial information in the incorrect section. No wonder it shows up as empty/unknown media type! Upon closer inspection, they programmed it like an SFP+ module. This first half of the EEPROM should be populated with these values like this in an SFP+, but for a QSFP+ module, it should be in the second half.

So I used a series of i2c write commands to copy byte for byte the contents of the official brocade DAC, to the monoprice DAC, overwriting the incorrectly placed vendor info with properly placed brocade info.

The monoprice DAC EEPROM contents now looks like this:
Code:
OS>i2c read 41 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......
Look familiar? Our monoprice DAC now has identical EEPROM contents, compared to our actual official brocade DAC.

And as expected, after unplugging and replugging the cable, the switch now ID's it as a DAC, and thinks it's an official brocade part:

Code:
telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : 40GE-Passive Copper
Port 1/2/3:  Type : 40GE-Passive Copper
Port 1/2/4:  Type : 40GE-Passive Copper
Port 1/2/5:  Type : 40GE-Passive Copper

##even more details - still the monoprice DAC
telnet@ICX3#show media e 1/2/2
Port   1/2/2:Type  : 40GBASE-Passive Copper
Vendor Name: BROCADE          Serial Num: PAE114103000876 Revision: A
I wouldn't think this has any effect on functionality, as I said the links worked perfectly fine beforehand. Re-coding it into a brocade part like this was only possible because the monoprice DAC did not have the EEPROM write protect bit set. This exact same procedure can be used to re-code optics to different vendors to bypass vendor locks, but many non-OEM optics have a bit set that prevents EEPROM writing without special unlock strings

A little more information:

The SFP+ EEPROM data location layout, you can see the identity and vendor etc strings are at the beginning (like our QSFP+ monoprice incorrectly had) - http://files.siemon.com/sis/application-guide/sfp-plus-to-sfp-plus_eeprom_summary.pdf

The QSFP+ EEPROM layout, you can see that the vendor and cable type etc now does not start until around the middle. This is where the ICX was trying to read to figure out the cable type, and of course on the monoprice there was nothing there - http://files.siemon.com/sis/application-guide/qsfpplus_to_qsfpplus_eeprom_summary.pdf

Upon closer inspection, this could have potentially caused it refusing to link up, as it could not even read the ident bit to figure out what kind of cable it was. For QSFP modules, it reads byte number 128 (first byte on line 80 in the dump above). This byte is set to "0d" hex to tell the host it's a QSFP+ device. On the monoprice dac, when it read that byte, it was completely empty. You can see in the brocade dump, that bit is correctly set to 0d.

here's a table of possible values for that byte:

https://i.imgur.com/2Id6xp9.png

You can see that if we set that byte to 03 instead, we could make the switch think it's an SFP module. It would probably not like this, given it's a QSFP+ slot (or on second thought, it would just assume it's an SFP+ optic in an SFP+ to QSFP+ adapter)

And upon even further digging, the EEPROM also contains signal attenuation values for the given cable at both 2.5ghz and 5ghz (bytes 186 and 187 respectively), and the host (the switch) uses these values to set up it's transmitter and receiver EQ appropriately. Without it being able to read these on the stock monoprice DAC, the transmitter and receiver probably defaulted to "fallback" values, which could have also resulted in the not quite working links.

Again, copying over the brocade DAC contents should fix this, as it contains these values (4db and 5db, respectively). I'd imagine the attenuation values there are very close to what the monoprice DAC is supposed to have, given they are both the same length, and both 30AWG twinax. Given how much stuff the monoprices come with wrong from factory, I'm surprised they ever worked, especially with cisco equipment (which is what they're advertised for). Thankfully it's not write-locked, so all this is easily fixed with a few commands
I understood a fraction of this post; but playing with EEPROM is ultra-geek old-school, very well done.
 
  • Like
Reactions: Yunia and fohdeesha

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Just a couple of quick notes on silencing the 6450-24-

There's some extremely high frequency power circuitry noise from the switch when running the stock fan at <100% and after replacing the stock fan with a Sunon maglev 5200RPM. You really can't hear it over the stock fan, but it's definitely there with a quieter replacement.

The switch is virtually silent apart from the high frequency and I'm not even sure I can pick it out from my other machines from across the room.

The switch also runs a bit hotter than I expected with a silent fan (see below) so I put a higher CFM Sunon on order. All in all this is an extremely nice piece of hardware to add to the homelab.

Code:
ICX6450-24 Router>sh chass
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 71 deg-C
        1 <- 2 @ 66 deg-C

Sensor B Temperature Readings:
        Current temperature : 52.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 58.5 deg-C
        Warning level.......: 73.0 deg-C
        Shutdown level......: 83.0 deg-C
Boot Prom MAC : cc4e.2460.88a0
Management MAC: cc4e.2460.88a0
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,726
3,066
113
33
fohdeesha.com
Those temps are still quite cool, I wouldn't worry about it. that's a full 15C lower than the idle temp of the ICX6610 with the stock fans. It'll ramp the fans up itself automatically if it thinks it's getting too hot, as you can see it's still just running along happily at the lowest fan speed setting - it doesn't even bother ramping up until you break 71C
 

pcmoore

Active Member
Apr 14, 2018
138
48
28
New England, USA
After watching this forum and listening to @fohdeesha 's sales pitch over the past few months it is getting hard to ignore these switches ;)

I'm looking around for a cheap 6450-24, doesn't need to be the PoE model, and the couple I've seen don't have ears. Does anyone know the part number for the ears, or better yet, does anyone know a good source of ears?
 
  • Like
Reactions: fohdeesha

nthu9280

Well-Known Member
Feb 3, 2016
1,627
498
83
San Antonio, TX
Did an impulse buy on a ICX6610-48P and paid little more than some of the good deals. But it did come with all the advanced licenses and didn't have any trouble activating 40G ports. Haven't actually gotten around to plugging those yet. Idles over 100W. Now debating whether I need this or go with the lower powered but quiter ICX6450-24. Choices, Choices...
 

istamov

New Member
Jul 31, 2015
17
5
3
Also, this is a shot in the dark, but for $7.99 I thought I'd just buy it and test.

NetApp X6559-R6 SAS expansion Cable 5m 5 meter QSFP-QSFP 112-00178 15ft 11200178 | eBay

Anyone know if these are capable of 40Gb between two C-X3s?
I was also wondering this, so thank you for asking - it made me go find a spare cable and test.
For the moment I just tried to loop the two ports of the only 40 GbE device I currently have (Mellanox ConnectX-3 card) with NetApp QSFP-QSFP 2m/6ft SAS cable (P/N: 112-00177) and at least the physical link layer seems to be up:
Code:
4: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 24be05ffffb24101 state UP group default qlen 1000
    link/ether 24:be:05:b2:41:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::26be:5ff:feb2:4101/64 scope link 
       valid_lft forever preferred_lft forever
5: enp2s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 24be05ffffb24102 state UP group default qlen 1000
    link/ether 24:be:05:b2:41:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::26be:5ff:feb2:4102/64 scope link 
       valid_lft forever preferred_lft forever

# ethtool enp2s0
Settings for enp2s0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full 
                            10000baseKX4/Full 
                            10000baseKR/Full 
                            40000baseCR4/Full 
                            40000baseSR4/Full 
                            56000baseCR4/Full 
                            56000baseSR4/Full 
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseKX/Full 
                            10000baseKX4/Full 
                            10000baseKR/Full 
                            40000baseCR4/Full 
                            40000baseSR4/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Link partner advertised link modes:  40000baseCR4/Full 
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Speed: 40000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000014 (20)
                   link ifdown
    Link detected: yes
Once I receive my switches will try the full setup (as in passing traffic) and will confirm if it is working.
 

AlphaG

Active Member
Jun 8, 2017
102
26
28
53
I need 3-4 24 port switches with 10G uplinks. How does this compare to the Mikrotik 326-24G-2S+RM?

I was considering the Mikrotik because of price and quiet operation, but the ICX6450 are tempting. Could even get rid of some POE injectors by getting a POE version of the switch.
 

pcmoore

Active Member
Apr 14, 2018
138
48
28
New England, USA
I need 3-4 24 port switches with 10G uplinks. How does this compare to the Mikrotik 326-24G-2S+RM?

I was considering the Mikrotik because of price and quiet operation, but the ICX6450 are tempting. Could even get rid of some POE injectors by getting a POE version of the switch.
While I don't currently have a Brocade, I did recently pick up a Mikrotik CRS326 to play with and decide if RouterOS/SwOS was a reasonable option for me. If you've used the Mikrotik network OSes before and you know what you are getting into, great. However, if you are assuming that RouterOS and/or SwOS are just slightly different takes on a more conventional switch UI then I think you are in for a rude awakening. I personally would consider the Mikrotik CRS326 to be more comparable to an entry level smart/web managed Netgear switch than a Brocade ICX.

As an aside, if you do go with the CRS326 and want to ditch the wall wart (a personal pet peeve), it is incredibly easy to add an internal power supply. I'm almost considering using the CRS326 in a limited capacity simply because it doesn't have a wall wart and isn't much worse than the other "smart" switches I've used.