Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Hmm, I'm not sure why you are having issues. I purchased two HP 649281-B21 NICs (Mellanox MCX354A-QCBT) from 649281-B21 HP Infiniband FDR/Ethernet 10Gb/40Gb 2-port 544QSFP Adapter 886111482092 | eBay with both high and low brackets for $30 ea. with free shipping. I then followed fohdeesha's instructions for cross-flashing the MCX354A-FCBT firmware, manually setting ethernet mode on both ports, and removing the FlexBoot ROM.
I'm using four of the KAIAM XQX2502 40G-LR4 Lite transceivers to connect my ICX6610-48p to my storage server (Debian Buster) and my primary Windows 10 desktop. Both systems immediately established a link over 20M of OS2 SMF and the digital optical monitoring shows a strong signal for both machines.
I suppose it's possible that one or more of your KAIAM transceivers is defective. Do you just have two? I assume you don't have any other 40G-LR4 optics to test. There are also some cheap AQI optics if you want a different option - AOI QSFP+ 40G-LR4 Lite 2km 1301nm 4xCWDM Daul LC SMF AQOLBCQ4EDMA0878 | eBay
Last edited:
Well, that's very strange. I don't see any traffic on my unused ports. I seem to have the opposite issue with the LEDs. I have both 1/2/1 and 1/2/6 connected and active, but neither XL1 or XL6 LEDs are illuminated.
Last edited:
Not sure if this offers a hint, but the two ports on the nic do report differently in >ifconfig:
Code:
mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:8a:07:76:38:91
media: Ethernet autoselect
status: no carrier
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TNot sure if related, but when I run
[evan@router ~]$ mst status it shows this:
Code:
MST devices:
------------
pci0:1:0:0 - MT27500 Family [ConnectX-3][evan@router ~]$ mst status -v it shows this:
Code:
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMASo perhaps something isn't right with the card?
When I run
[evan@router ~]$ mst server start it just seems to hang indefinitely.When I run
[evan@router ~]$ mst cable add it returns:
Code:
-I- Searching for connected cables
-W- No cables were found!sudo mlxfwmanager which gave a little more meat:
Code:
[evan@router ~/mft-4.18.0-106]$ sudo mlxfwmanager
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: 050-0050-02_01
Description: ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6
PSID: ISL1090110018
PCI Device Name: pci0:1:0:0
Port1 MAC: 248a07763891
Port2 MAC: 248a07763892
Versions: Current Available
FW 2.40.5030 N/A
PXE 3.4.0746 N/A
Status: No matching image found
Last edited:
Well, there you go! If you're running the correct firmware, it should show 40GbE. Since the NIC only supports 10GbE, it will never link up at 40GbE. Unlike 1000Base-T, these fiber NICs have no auto-negotiation. The only way to run it at 10GbE would be to breakout to 4x 10Gb and use a QSFP to SFP+ adapter. Here's what I see on my correctly flashed NIC:
root@storage-server:~# mlxfwmanager
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: /dev/mst/mt4099_pci_cr0
Port1 MAC: 0002c93fda20
Port2 MAC: 0002c93fda21
Versions: Current Available
FW 2.42.5000 N/A
Status: No matching image found
Code:
root@storage-server:~# mst status -v
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX3(rev:1) /dev/mst/mt4099_pciconf0
ConnectX3(rev:1) /dev/mst/mt4099_pci_cr0 03:00.0 mlx4_0 net-enp3s0d1,net-enp3s0 0
Last edited:
Aha! Based on this, I plugged it into the 4x10 port on the 6610 and it's working perfectly, albeit at 10g instead of the desired 40. Let's be honest though, I'm not likely not see any real world difference any time soon, as 10g should even saturate my zfs NAS.
I'd love to be able to flash new firmware, but I can't figure out how to force it to do so. All of the mellanox utils seem to reject it due to the odd PSID.
Awesome, I'm glad you got it working. fohdeesha mentioned that when he connected the 40G-LR4 transceivers to the QSFP breakout ports between two ICX6610 switches, it showed all four lanes in use. I didn't realize it would work when connecting to a client that only supports a single 10G lane.
However, the downside of using the QSFP breakout port with a 40G-LR transceiver for a single 10G lane is additional power and heat at the client and switch, as well as the loss of 3 out of the 4 brekaout ports. The dual port 40G Mellanox NICs use 8-9W whereas a single 10G Mellanox NIC ( CONNECTX-3 EN CX311A) uses around ~3.3W and the dual uses 4.44W. I find the CX311A great for enabling 10G connectivity on desktop machines. It's tiny, low power, can be purchased with a short or long bracket for around $25, and works with SFF machines as long as there's a x4 PCI-E slot.
The 40G-LR4 Lite transceiver uses ~2.5W and gets reasonably warm (~48-50C) whereas the 10G-LR and 10G-SR transceivers use 1W or less and run near ambient temp (30-32C). If you only need 10G, you can use a 10G-LR SFP+ transceiver with the existing SMF cable ($5 on Ebay if you don't care about digital optical monitoring). I was able to purchase a lot of 4 genuine Brocade 10G-LR4 transceivers for $20, but I don't see any cheap ones at the moment. If you run OM3, you can use $5 original Brocade 10G-SR transceivers and get digital optical monitoring.
If instead you want working 40G, and your current card can't be flashed, many folks here can confirm that the HP 649281-B21 can be flashed easily using the above instructions. I've already done two cards myself, and they only cost $30. Unlike the CX311A NICs, these have working digital optical monitoring, and it seems to work with any manufacturer, since it works with the KAIAM transceivers with detailed info.
I'm basically using all of the above options for different needs, as well as a FS 40G to 4 x 10G SFP+ breakout DAC for my pfsense router.
SSH@ICX6610-48p> show media validation
1/2/1 Yes KAIAM CORP 40G QSFP Module
1/2/2 Yes BROCADE 40GBASE-Passive Copper
1/2/3 Yes BROCADE 40GBASE-Passive Copper
1/2/4 Yes BROCADE 40GBASE-Passive Copper
1/2/5 Yes BROCADE 40GBASE-Passive Copper
1/2/6 Yes KAIAM CORP 40G QSFP Module
1/3/1 Yes BROCADE Type : 10GE SR 300m (SFP +)
1/3/2 Yes BROCADE Type : 10GE SR 300m (SFP +)
1/3/3 Yes BROCADE Type : 10GE LR 10km (SFP +)
1/3/4 Yes BROCADE Type : 10GE SR 300m (SFP +)
1/3/5 Yes BROCADE Type : 10GE LR 10km (SFP +)
1/3/7 Yes BROCADE Type : 10GE LR 10km (SFP +)
Last edited:
One question about fan modding on the ICX6450:
Ive read something about newer firmware which doesnt support alternative fans anymore here:
brocade.fandom.com
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
Ive read something about newer firmware which doesnt support alternative fans anymore here:
7250
"Medium beef" data sheet 24/48 1gbE copper (PoE available) 8x 10gbE SFP+ Full layer 3, IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, all the usual, but also adding VRFs and tunnels (but no BGP) 50w power draw single built in PSU Aggregate capacity: 256gbps / 190Mpps (wirespeed...
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
before you go down this path you might want to listen to one first - decide whether you really need or want to replace the fans and if so then perform a search here... maybe something like "6450 replace fan" and see what pops. At 327 pages of information - reading this particular thread may feel like summit-ting a mountain but it really isn't, especially if you are digging for specific information.One question about fan modding on the ICX6450:
Ive read something about newer firmware which doesnt support alternative fans anymore here:
7250
"Medium beef" data sheet 24/48 1gbE copper (PoE available) 8x 10gbE SFP+ Full layer 3, IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, all the usual, but also adding VRFs and tunnels (but no BGP) 50w power draw single built in PSU Aggregate capacity: 256gbps / 190Mpps (wirespeed...
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
OK, I will do that after I got one. I have some 6450 and also some 7250 in my Watch list.
Hopefully I can Grab one of the 7250 for a reasonable price. Downside is i would have to pay 70$ of parcel. But maybe I can get one of the 7250 for 180$ including parcel.
Hopefully I can Grab one of the 7250 for a reasonable price. Downside is i would have to pay 70$ of parcel. But maybe I can get one of the 7250 for 180$ including parcel.
In case anyone else has been struggling to get ssh public key auth working on the 09 release, I did some digging and here's what I found:
First of all, the documented bit is that they changed the command from (conf mode) "ip ssh pub-key-file tftp 192.168.1.8 public.key" to (exec mode) "copy tftp flash 10.168.1.234 pkeys.txt ssh-pub-key-file". Second, it's not documented, but now it accepts openssh format keys instead of SSH2.
However, this still isn't enough to get it working. I couldn't find any useful logs or debug information from the normal CLI, but I eventually found the "debug linux" command, which lets you poke around the linux environment. You can view the sshd logs with "debug linux cat /logmgr/logs/management/sshd_low_tmp.log".
The culprit is the message "Authentication refused: bad ownership or modes for directory /". The root filesystem seems to have been assembled with some strange permissions:
drwxr-sr-x 19 251 1011 0 Jan 3 02:56 /
The "debug linux" command has some rudimentary command checking, but it appears to only check the first token, so it's trivial to fix this:
router#debug linux ls ; chmod 755 / ; chown 0:0 /
Unfortunately, / is a tmpfs, so this doesn't persist across reboots.
First of all, the documented bit is that they changed the command from (conf mode) "ip ssh pub-key-file tftp 192.168.1.8 public.key" to (exec mode) "copy tftp flash 10.168.1.234 pkeys.txt ssh-pub-key-file". Second, it's not documented, but now it accepts openssh format keys instead of SSH2.
However, this still isn't enough to get it working. I couldn't find any useful logs or debug information from the normal CLI, but I eventually found the "debug linux" command, which lets you poke around the linux environment. You can view the sshd logs with "debug linux cat /logmgr/logs/management/sshd_low_tmp.log".
The culprit is the message "Authentication refused: bad ownership or modes for directory /". The root filesystem seems to have been assembled with some strange permissions:
drwxr-sr-x 19 251 1011 0 Jan 3 02:56 /
The "debug linux" command has some rudimentary command checking, but it appears to only check the first token, so it's trivial to fix this:
router#debug linux ls ; chmod 755 / ; chown 0:0 /
Unfortunately, / is a tmpfs, so this doesn't persist across reboots.
Configuration changes won't stick via GUI...
I set up an ICX 6430 using the most recent files found here: Brocade Overview - Fohdeesha Docs
I was using the web GUI to configure inline power settings. After I made changes (and Saved), the desired settings took effect. But the next time I booted up the switch, the inline power settings went back to their previous state.
I went into the CLI and made the desired inline power changes and after the next rebooting those changes held.
Is there something I'm missing about saving changes from the GUI? Or some way to resolve this issue? My skill level says the web GUI is more my speed than the CLI.
I set up an ICX 6430 using the most recent files found here: Brocade Overview - Fohdeesha Docs
I was using the web GUI to configure inline power settings. After I made changes (and Saved), the desired settings took effect. But the next time I booted up the switch, the inline power settings went back to their previous state.
I went into the CLI and made the desired inline power changes and after the next rebooting those changes held.
Is there something I'm missing about saving changes from the GUI? Or some way to resolve this issue? My skill level says the web GUI is more my speed than the CLI.
Yes, you only made the changes to the running config. To make it persist, you need to run "write mem". So, you'll need to do the following from a new terminal window:
enable
conf t
write mem
exit
I think you can save it by going to the very bottom on the left pane and select Command->Save to Flash
I recently picked up a couple 7250s and transplanted my aftermarket fans (I haven't been able to find a record of which specific model, but I believe it's one of the Sunon maglev models discussed early in the thread) from my 6450s. My 6450s and 7250s are running the latest firmware on @fohdeesha 's site and don't throw any errors.One question about fan modding on the ICX6450:
Ive read something about newer firmware which doesnt support alternative fans anymore here:
7250
"Medium beef" data sheet 24/48 1gbE copper (PoE available) 8x 10gbE SFP+ Full layer 3, IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, all the usual, but also adding VRFs and tunnels (but no BGP) 50w power draw single built in PSU Aggregate capacity: 256gbps / 190Mpps (wirespeed...
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
To my ear, the 6450s were still borderline too loud with the new fans. When I moved the fans into my 7250s, the temps are quite high and send the fans into the "speed 2" regime every 5 minutes or so. This is with ~20W of POE and 6 10G devices in each switch (half DAC, half RJ45 transceivers).
While I still have both my 6450s and 7250s I may make a separate thread about fans with noise samples since it seems like a frequent topic.
Regarding fan noise, I noticed that my ICX6610-48p was running at 55dB with either a single or dual PSU (2 x Rev C) and dual fans. I couldn't figure out why the fan noise didn't drop with two power supplies, as that was widely reported by others. After restarting the switch for an unrelated reason, the two fans significantly reduced RPM, and my sound meter now measures 50dB or slightly below. The only difference was that I had both power supplies connected to power when I reloaded the switch.
I am stacking 2x 7250s, do I need to dedicate 4x 10G ports (2x per switch) to the stack or can I just use 2x (1x per switch)?
Last edited:
I ran "stack interactive-setup" and it seems like it automatically made a ring stack. Do I need to reset things or is there a way to transform it into a linear stack?
Do you have two cables connecting the switches (using four ports)? If so, that's why it built a ring stack, because that's the preferred configuration (since it provides redundancy).