The interface statistics show about 130GB of traffic on 1/2/1. Did you have it connected previously using a different DAC or with fiber?Thanks for the reply. I had found that and added the commands, which worked just fine. The interfaces show up in OpnSense, just as disconnected.
I just tried connecting the cable in a loop from interface 1 to interface 2 on the mellanox nic, and it lit up just fine, so it seems like there's something on the switch side that's not recognizing or utilizing the cable.
EDIT2: Here's what the most relevant parts of my network setup look like, in case it's relevant (open to feedback here as well). Note that there wasn't a good 6610 equivalent in draw.io, so ignore the positional matchups. I added the interface ids for clarity.
EDIT:
Code:telnet@ICX6610-24P Router#show interfaces ethernet 1/2/1 40GigabitEthernet1/2/1 is down, line protocol is down Port down for 17 minute(s) 55 second(s) Hardware is 40GigabitEthernet, address is 748e.f8e8.5d7a (bia 748e.f8e8.5d93) Interface type is 40Gig Fiber Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown Configured mdi mode AUTO, actual unknown Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled Link Error Dampening is Disabled STP configured to ON, priority is level0, mac-learning is enabled Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is enabled Mirror disabled, Monitor disabled Mac-notification is disabled Not member of any active trunks Not member of any configured trunks No port name MTU 1500 bytes, encapsulation ethernet 300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization 300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization 607855000 packets input, 67135060808 bytes, 0 no buffer Received 210154120 broadcasts, 394335559 multicasts, 3365321 unicasts 15 input errors, 1 CRC, 0 frame, 0 ignored 0 runts, 0 giants 601682076 packets output, 65644112862 bytes, 0 underruns Transmitted 214559764 broadcasts, 386674611 multicasts, 447701 unicasts 0 output errors, 0 collisions Relay Agent Information option: Disabled Egress queues: Queue counters Queued packets Dropped Packets 0 64811133 0 1 0 0 2 0 0 3 0 0 4 0 0 5 30 0 6 0 0 7 1 0
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Hi guys,
i wish you a nice new years eve
So, i just found this awesome thread about those nice affordable switches and immediately started searching for some on ebay.
I've found some icx6450-48 switches, but in the product description they say the switches only have 2x10gbe sfp+, so i assume the other two are only 1gbe sfp ports.
Are there different versions of the icx6450 switch with either 4xsfp+/10gbe or 2xsfp+/10gbe and 2xsfp/1gbe?
i wish you a nice new years eve
So, i just found this awesome thread about those nice affordable switches and immediately started searching for some on ebay.
I've found some icx6450-48 switches, but in the product description they say the switches only have 2x10gbe sfp+, so i assume the other two are only 1gbe sfp ports.
Are there different versions of the icx6450 switch with either 4xsfp+/10gbe or 2xsfp+/10gbe and 2xsfp/1gbe?
I did indeed. Even re-enabled and re-disabled to be sure. show run snippet:
Code:
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-24p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
stack mac 748e.f8e8.5d7aThat's one weird thing - the activity light for 1/2/1 is on solid whether I have this cable plugged into it or not, and that number keeps going up...
Yes but by way of magick you can get what you want.
Incantation:
Before you buy any switches please take a look at the first post in this thread. Please Read it then go back to the top.
Click on the first link in the first post of the thread. give that page a read.
Look to the left. There's a clicky called Brocade Setup. click there for the switch you seek.
At that bottom of that page there is a link that talks about licensing incantations and that should tell you all you need to know to create magick.
dang - 5 edits to get that right. clearly I started drinking too early.
Thanks for the hint! I will do it after finishing my Dominican cigar and Champagne
Wish you a nice New years eve!
Edit: Damn, i could kick my ass that i didnt find this earlier.... Going to setup a Dell R210ii as my Router and a 6450 as my home network setup. So i can get rid of my pfsense vm, run it bare metal and get my nas box and my main PC running on 10gbps
Last edited:
Hmm, I'm not sure why you are having issues. I purchased two HP 649281-B21 NICs (Mellanox MCX354A-QCBT) from 649281-B21 HP Infiniband FDR/Ethernet 10Gb/40Gb 2-port 544QSFP Adapter 886111482092 | eBay with both high and low brackets for $30 ea. with free shipping. I then followed fohdeesha's instructions for cross-flashing the MCX354A-FCBT firmware, manually setting ethernet mode on both ports, and removing the FlexBoot ROM.
I'm using four of the KAIAM XQX2502 40G-LR4 Lite transceivers to connect my ICX6610-48p to my storage server (Debian Buster) and my primary Windows 10 desktop. Both systems immediately established a link over 20M of OS2 SMF and the digital optical monitoring shows a strong signal for both machines.
I suppose it's possible that one or more of your KAIAM transceivers is defective. Do you just have two? I assume you don't have any other 40G-LR4 optics to test. There are also some cheap AQI optics if you want a different option - AOI QSFP+ 40G-LR4 Lite 2km 1301nm 4xCWDM Daul LC SMF AQOLBCQ4EDMA0878 | eBay
Last edited:
Well, that's very strange. I don't see any traffic on my unused ports. I seem to have the opposite issue with the LEDs. I have both 1/2/1 and 1/2/6 connected and active, but neither XL1 or XL6 LEDs are illuminated.
Last edited:
Not sure if this offers a hint, but the two ports on the nic do report differently in >ifconfig:
Code:
mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:8a:07:76:38:91
media: Ethernet autoselect
status: no carrier
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TNot sure if related, but when I run
[evan@router ~]$ mst status it shows this:
Code:
MST devices:
------------
pci0:1:0:0 - MT27500 Family [ConnectX-3][evan@router ~]$ mst status -v it shows this:
Code:
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMASo perhaps something isn't right with the card?
When I run
[evan@router ~]$ mst server start it just seems to hang indefinitely.When I run
[evan@router ~]$ mst cable add it returns:
Code:
-I- Searching for connected cables
-W- No cables were found!sudo mlxfwmanager which gave a little more meat:
Code:
[evan@router ~/mft-4.18.0-106]$ sudo mlxfwmanager
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: 050-0050-02_01
Description: ConnectX-3 VPI adapter card; dual-port QSFP; QDR IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6
PSID: ISL1090110018
PCI Device Name: pci0:1:0:0
Port1 MAC: 248a07763891
Port2 MAC: 248a07763892
Versions: Current Available
FW 2.40.5030 N/A
PXE 3.4.0746 N/A
Status: No matching image found
Last edited:
Well, there you go! If you're running the correct firmware, it should show 40GbE. Since the NIC only supports 10GbE, it will never link up at 40GbE. Unlike 1000Base-T, these fiber NICs have no auto-negotiation. The only way to run it at 10GbE would be to breakout to 4x 10Gb and use a QSFP to SFP+ adapter. Here's what I see on my correctly flashed NIC:
root@storage-server:~# mlxfwmanager
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: /dev/mst/mt4099_pci_cr0
Port1 MAC: 0002c93fda20
Port2 MAC: 0002c93fda21
Versions: Current Available
FW 2.42.5000 N/A
Status: No matching image found
Code:
root@storage-server:~# mst status -v
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX3(rev:1) /dev/mst/mt4099_pciconf0
ConnectX3(rev:1) /dev/mst/mt4099_pci_cr0 03:00.0 mlx4_0 net-enp3s0d1,net-enp3s0 0
Last edited:
Aha! Based on this, I plugged it into the 4x10 port on the 6610 and it's working perfectly, albeit at 10g instead of the desired 40. Let's be honest though, I'm not likely not see any real world difference any time soon, as 10g should even saturate my zfs NAS.
I'd love to be able to flash new firmware, but I can't figure out how to force it to do so. All of the mellanox utils seem to reject it due to the odd PSID.
Awesome, I'm glad you got it working. fohdeesha mentioned that when he connected the 40G-LR4 transceivers to the QSFP breakout ports between two ICX6610 switches, it showed all four lanes in use. I didn't realize it would work when connecting to a client that only supports a single 10G lane.
However, the downside of using the QSFP breakout port with a 40G-LR transceiver for a single 10G lane is additional power and heat at the client and switch, as well as the loss of 3 out of the 4 brekaout ports. The dual port 40G Mellanox NICs use 8-9W whereas a single 10G Mellanox NIC ( CONNECTX-3 EN CX311A) uses around ~3.3W and the dual uses 4.44W. I find the CX311A great for enabling 10G connectivity on desktop machines. It's tiny, low power, can be purchased with a short or long bracket for around $25, and works with SFF machines as long as there's a x4 PCI-E slot.
The 40G-LR4 Lite transceiver uses ~2.5W and gets reasonably warm (~48-50C) whereas the 10G-LR and 10G-SR transceivers use 1W or less and run near ambient temp (30-32C). If you only need 10G, you can use a 10G-LR SFP+ transceiver with the existing SMF cable ($5 on Ebay if you don't care about digital optical monitoring). I was able to purchase a lot of 4 genuine Brocade 10G-LR4 transceivers for $20, but I don't see any cheap ones at the moment. If you run OM3, you can use $5 original Brocade 10G-SR transceivers and get digital optical monitoring.
If instead you want working 40G, and your current card can't be flashed, many folks here can confirm that the HP 649281-B21 can be flashed easily using the above instructions. I've already done two cards myself, and they only cost $30. Unlike the CX311A NICs, these have working digital optical monitoring, and it seems to work with any manufacturer, since it works with the KAIAM transceivers with detailed info.
I'm basically using all of the above options for different needs, as well as a FS 40G to 4 x 10G SFP+ breakout DAC for my pfsense router.
SSH@ICX6610-48p> show media validation
1/2/1 Yes KAIAM CORP 40G QSFP Module
1/2/2 Yes BROCADE 40GBASE-Passive Copper
1/2/3 Yes BROCADE 40GBASE-Passive Copper
1/2/4 Yes BROCADE 40GBASE-Passive Copper
1/2/5 Yes BROCADE 40GBASE-Passive Copper
1/2/6 Yes KAIAM CORP 40G QSFP Module
1/3/1 Yes BROCADE Type : 10GE SR 300m (SFP +)
1/3/2 Yes BROCADE Type : 10GE SR 300m (SFP +)
1/3/3 Yes BROCADE Type : 10GE LR 10km (SFP +)
1/3/4 Yes BROCADE Type : 10GE SR 300m (SFP +)
1/3/5 Yes BROCADE Type : 10GE LR 10km (SFP +)
1/3/7 Yes BROCADE Type : 10GE LR 10km (SFP +)
Last edited:
One question about fan modding on the ICX6450:
Ive read something about newer firmware which doesnt support alternative fans anymore here:
brocade.fandom.com
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
Ive read something about newer firmware which doesnt support alternative fans anymore here:
7250
"Medium beef" data sheet 24/48 1gbE copper (PoE available) 8x 10gbE SFP+ Full layer 3, IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, all the usual, but also adding VRFs and tunnels (but no BGP) 50w power draw single built in PSU Aggregate capacity: 256gbps / 190Mpps (wirespeed...
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
before you go down this path you might want to listen to one first - decide whether you really need or want to replace the fans and if so then perform a search here... maybe something like "6450 replace fan" and see what pops. At 327 pages of information - reading this particular thread may feel like summit-ting a mountain but it really isn't, especially if you are digging for specific information.One question about fan modding on the ICX6450:
Ive read something about newer firmware which doesnt support alternative fans anymore here:
7250
"Medium beef" data sheet 24/48 1gbE copper (PoE available) 8x 10gbE SFP+ Full layer 3, IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, all the usual, but also adding VRFs and tunnels (but no BGP) 50w power draw single built in PSU Aggregate capacity: 256gbps / 190Mpps (wirespeed...
So, is this true or is there a workaround for it? There are those Sunon silent fans ive seen in another video about this switch.
Maybe someone can tell me something
Edit: NVM this is for the 7250, but maybe the same problem exists on the 6450?
OK, I will do that after I got one. I have some 6450 and also some 7250 in my Watch list.
Hopefully I can Grab one of the 7250 for a reasonable price. Downside is i would have to pay 70$ of parcel. But maybe I can get one of the 7250 for 180$ including parcel.
Hopefully I can Grab one of the 7250 for a reasonable price. Downside is i would have to pay 70$ of parcel. But maybe I can get one of the 7250 for 180$ including parcel.
In case anyone else has been struggling to get ssh public key auth working on the 09 release, I did some digging and here's what I found:
First of all, the documented bit is that they changed the command from (conf mode) "ip ssh pub-key-file tftp 192.168.1.8 public.key" to (exec mode) "copy tftp flash 10.168.1.234 pkeys.txt ssh-pub-key-file". Second, it's not documented, but now it accepts openssh format keys instead of SSH2.
However, this still isn't enough to get it working. I couldn't find any useful logs or debug information from the normal CLI, but I eventually found the "debug linux" command, which lets you poke around the linux environment. You can view the sshd logs with "debug linux cat /logmgr/logs/management/sshd_low_tmp.log".
The culprit is the message "Authentication refused: bad ownership or modes for directory /". The root filesystem seems to have been assembled with some strange permissions:
drwxr-sr-x 19 251 1011 0 Jan 3 02:56 /
The "debug linux" command has some rudimentary command checking, but it appears to only check the first token, so it's trivial to fix this:
router#debug linux ls ; chmod 755 / ; chown 0:0 /
Unfortunately, / is a tmpfs, so this doesn't persist across reboots.
First of all, the documented bit is that they changed the command from (conf mode) "ip ssh pub-key-file tftp 192.168.1.8 public.key" to (exec mode) "copy tftp flash 10.168.1.234 pkeys.txt ssh-pub-key-file". Second, it's not documented, but now it accepts openssh format keys instead of SSH2.
However, this still isn't enough to get it working. I couldn't find any useful logs or debug information from the normal CLI, but I eventually found the "debug linux" command, which lets you poke around the linux environment. You can view the sshd logs with "debug linux cat /logmgr/logs/management/sshd_low_tmp.log".
The culprit is the message "Authentication refused: bad ownership or modes for directory /". The root filesystem seems to have been assembled with some strange permissions:
drwxr-sr-x 19 251 1011 0 Jan 3 02:56 /
The "debug linux" command has some rudimentary command checking, but it appears to only check the first token, so it's trivial to fix this:
router#debug linux ls ; chmod 755 / ; chown 0:0 /
Unfortunately, / is a tmpfs, so this doesn't persist across reboots.
Configuration changes won't stick via GUI...
I set up an ICX 6430 using the most recent files found here: Brocade Overview - Fohdeesha Docs
I was using the web GUI to configure inline power settings. After I made changes (and Saved), the desired settings took effect. But the next time I booted up the switch, the inline power settings went back to their previous state.
I went into the CLI and made the desired inline power changes and after the next rebooting those changes held.
Is there something I'm missing about saving changes from the GUI? Or some way to resolve this issue? My skill level says the web GUI is more my speed than the CLI.
I set up an ICX 6430 using the most recent files found here: Brocade Overview - Fohdeesha Docs
I was using the web GUI to configure inline power settings. After I made changes (and Saved), the desired settings took effect. But the next time I booted up the switch, the inline power settings went back to their previous state.
I went into the CLI and made the desired inline power changes and after the next rebooting those changes held.
Is there something I'm missing about saving changes from the GUI? Or some way to resolve this issue? My skill level says the web GUI is more my speed than the CLI.
Yes, you only made the changes to the running config. To make it persist, you need to run "write mem". So, you'll need to do the following from a new terminal window:
enable
conf t
write mem
exit