Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AgentXXL]

Just confirm that I bought a Supermicro Aquantia AQS-107-B0C2-CX 10G on ebay for $125 , and it's working great at 2.5Gbps from my main PC with a ASUS TUF X570-Pro motherboard, so far iPerf got 2.2 and 2.3 Gbps sustained,

since that was the last one from that vendor, now in the hunt for another one at a reasonable price

Thanks again for the reply

Glad it worked out for you... I ended up grabbing a Mikrotik CRS305 and one of their S+RJ10 SFP+ to RJ45 modules as I couldn't find a supplier that had stock and/or would ship to Canada. I'm still shopping for one of the Supermicro modules but I suspect I have another issue to fix - specifically the onboard 5Gbps NIC itself. The Mikrotik is really revealing the issues with the onboard 5Gbps NIC (Aquantia AQC111C) on my Asus Prime x299 Deluxe II. Specifically the random disconnects... when it happens I have to either unplug/re-attach the cable or reboot the Mikrotik. Thankfully under SwOS it takes less than 20 secs to reboot, but it's still a major annoyance.

I have a case open with Asus regarding a firmware updater for the NIC. The official updater from Marvell (who bought Aquantia) doesn't work on my motherboard, and neither does the community modded version which has added IDs for some Asus motherboards. Asus themselves state there's no known problem with the NIC and no firmware updater is available - they want me to RMA the board. But a simple Google search shows many with similar issues to what I'm experiencing, and most are resolved by updating the NIC firmware. And many of the posts about the issues are in Asus's own ROG forums. They've used the Aquantia AQC111C on a number of their products in the regular and ROG lines.

I provided a bunch of these links to Asus and am waiting to hear back. If I could figure out how to find the required ID info for my board, I'd add it myself to the XML config file in the updater. That XML file contains the IDs of systems that the updater works for. Hopefully I can get mine updated and it resolves my issues.

In the meantime, I have a LOT of learning to do about configuring my home network. I'm having difficulties with VLAN and/or port isolation but I'm sure it's just a lack of knowledge issue. Back to my reading...

 

[prdtabim]

Glad it worked out for you... I ended up grabbing a Mikrotik CRS305 and one of their S+RJ10 SFP+ to RJ45 modules as I couldn't find a supplier that had stock and/or would ship to Canada. I'm still shopping for one of the Supermicro modules but I suspect I have another issue to fix - specifically the onboard 5Gbps NIC itself. The Mikrotik is really revealing the issues with the onboard 5Gbps NIC (Aquantia AQC111C) on my Asus Prime x299 Deluxe II. Specifically the random disconnects... when it happens I have to either unplug/re-attach the cable or reboot the Mikrotik. Thankfully under SwOS it takes less than 20 secs to reboot, but it's still a major annoyance.

I have a case open with Asus regarding a firmware updater for the NIC. The official updater from Marvell (who bought Aquantia) doesn't work on my motherboard, and neither does the community modded version which has added IDs for some Asus motherboards. Asus themselves state there's no known problem with the NIC and no firmware updater is available - they want me to RMA the board. But a simple Google search shows many with similar issues to what I'm experiencing, and most are resolved by updating the NIC firmware. And many of the posts about the issues are in Asus's own ROG forums. They've used the Aquantia AQC111C on a number of their products in the regular and ROG lines.

I provided a bunch of these links to Asus and am waiting to hear back. If I could figure out how to find the required ID info for my board, I'd add it myself to the XML config file in the updater. That XML file contains the IDs of systems that the updater works for. Hopefully I can get mine updated and it resolves my issues.

In the meantime, I have a LOT of learning to do about configuring my home network. I'm having difficulties with VLAN and/or port isolation but I'm sure it's just a lack of knowledge issue. Back to my reading...

I have 3 mainboards with Aquantia chipset ( 2x aqc-107 and 1x aqc-108 ). In any case the last firmware updates from Marvell website NEEDs the driver 2.2 or newer to firmware updater detect and update. I have spended hours in this and discovered by chance after swaping the system ssd ( newer system = newer driver download ).

 

[AgentXXL]

I have 3 mainboards with Aquantia chipset ( 2x aqc-107 and 1x aqc-108 ). In any case the last firmware updates from Marvell website NEEDs the driver 2.2 or newer to firmware updater detect and update. I have spended hours in this and discovered by chance after swaping the system ssd ( newer system = newer driver download ).

My system normally runs under unRAID but I booted to my trusty WinToGo Win10 x64 and installed the latest driver from Marvell - for my AQC111C that's driver 3.1.6.0. Alas still no joy for the firmware updater. I'm more than certain it's the missing system IDs in the XML file that's preventing the updater from working. It sees my NIC and shows the outdated firmware - v. 3.1.50 vs the 3.1.121 on both Marvell's site and the community modded version from station-drivers.com.

 

[NablaSquaredG]

Is it possible to use the 4*10G QSFP+ breakout ports on the ICX6610 for stacking without using the regular QSFP+ ports for stacking too? i.e. stacking via the 4*10G QSFP+ breakout ports and using the other two QSFP+ port for regular connection?

Bump

Any ideas?

 

[crackelf]

yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

Anything I need to do on the Brocade side specifically to get these working? Connecting two cards together = connection, connecting the brocade ports together = connection, but I can't get the card to talk to the Brocade at all.

From the Brocade side we get

1/3/1      Down    None    None None  None  No  1    0   cc4e.2488.3380        
1/4/1      Down    None    None None  None  No  1    0   cc4e.2488.3380

Tagging @LodeRunner since you seem to be the 7450 guy on here

@jasonwc you seem to have a good thing going

I'm using four of the KAIAM XQX2502 40G-LR4 Lite transceivers to connect my ICX6610-48p to my storage server (Debian Buster) and my primary Windows 10 desktop. Both systems immediately established a link over 20M of OS2 SMF and the digital optical monitoring shows a strong signal for both machines.

 

[LodeRunner]

Anything I need to do on the Brocade side specifically to get these working? Connecting two cards together = connection, connecting the brocade ports together = connection, but I can't get the card to talk to the Brocade at all.

From the Brocade side we get

1/3/1      Down    None    None None  None  No  1    0   cc4e.2488.3380      
1/4/1      Down    None    None None  None  No  1    0   cc4e.2488.3380

Tagging @LodeRunner since you seem to be the 7450 guy on here

@jasonwc you seem to have a good thing going

As far as I'm aware, the Brocades in general don't care who the transceiver MFR is. I'm using DACs though for my 40G LAG (7450 to Arista 7050), so I haven't used any 40G optics in it.

What's the output of "sh med e 1/3/1" and "sh int e 1/3/1"

Edit: saw you direct connected the cards and it came up, and you looped the Brocade interfaces and it came up, but card to port isn't. That's really odd.

 

[crackelf]

Edit: saw you direct connected the cards and it came up, and you looped the Brocade interfaces and it came up, but card to port isn't. That's really odd.

The more I look at this the more it feels like I haven't got these Mellanox cards working correctly given their MAC addresses are all 00:00:00:00:00:00... See my Mellanox thread over here. My other pet theory is that I need to be using the mlx5_en driver instead of the mlx5_core driver, but I also don't know how to do that.

As far as I'm aware, the Brocades in general don't care who the transceiver MFR is. I'm using DACs though for my 40G LAG (7450 to Arista 7050), so I haven't used any 40G optics in it.

What's the output of "sh med e 1/3/1" and "sh int e 1/3/1"

sh med e 1/3/1

Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209

sh int e 1/3/1

40GigabitEthernet1/3/1 is down, line protocol is down
  Port down for 3 minute(s) 16 second(s)
  Hardware is 40GigabitEthernet, address is cc4e.2488.3380 (bia cc4e.2488.33b5)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Untagged member of L2 VLAN 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  0 packets input, 0 bytes, 0 no buffer                       
  Received 0 broadcasts, 0 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  0 packets output, 0 bytes, 0 underruns
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

 

[LodeRunner]

Yeah, it sees the transceiver but no carrier. If the card is in Auto or IB mode that would probably be the issue. IB mode for sure would be a problem; I've seen reports that Mellanox cards in Auto mode does not play nice with Ethernet switches.

mlxconfig -d <device id> set LINK_TYPE_P1=2 (LINK_TYPE_P2 for port 2 if present)
It will show you the current current config, proposed new config, and a confirm prompt.

I have CX354 (CX3-Pro) cards, so older, running in Windows.

More mlxconfig stuff; including SR-IOV VF configuration on the card itself: Using mlxconfig - MFT v4.11.0 - NVIDIA Networking Docs

 

[crackelf]

I have mstconfig as a part of the mstflint package. Does that make a difference?

-E- The Device doesn't support LINK_TYPE_P1 parameter

It looks like it's in ethernet mode though when I look at ip link show. Here is my current config

I have mstconfig -d 03:00.0 set SRIOV_EN=1 and mstconfig -d 03:00.0 set NUM_OF_VFS=8 set. I can see all the VFs, but no dice.

 

[LodeRunner]

Oh right, a -LX card. That appears to be Ethernet only. So makes sense setting the port mode is not supported, it only has Ethernet mode. Given the litany of issues in the other thread, I'm not entirely sure which end of the link the problem is on.

 

[crackelf]

Oh right, a -LX card. That appears to be Ethernet only. So makes sense setting the port mode is not supported, it only has Ethernet mode. Given the litany of issues in the other thread, I'm not entirely sure which end of the link the problem is on.

Hahaha I hear you. I'm about to pick up some CX354A and try those out. These things have been nothing but trouble.

 

[fohdeesha]

@fohdeesha

looks like the latest firmware for the fastiron on ICX platforms has a completely revamped GUI, version 9.0.10a

i know in your informational posts you commended the ICX7250 platform because it used an honor based licensing system, but i was never sure if that was only because it was on v8080 firmware (which i assume translates to 8.0.80 in ruckus firmware notation)

was it ever planned by ruckus/brocade to phase out the honor based licensing for these switches?

i'm aware you may not be the biggest fan of gui for switches but it will be nice to integrate these with my ruckus r500 which have a similar interface aesthetic

we covered the 9x releases a few pages back, they're super buggy alpha releases, you'll see what I mean just watching the boot output of them. the new web UI is pretty, but provides even less actual functionality than the old school one. licenses will always be honor based

 

[fohdeesha]

Is there any way to find out the manifacturing date of an ICX7450 by running a cli command or opening the case?

at enable cli level, show pid-prom

SSH@sw-01-ash#show pid-prom
Version Number: 0003
Part Number: 84-1002981-04
Serial Number: CYR3313N036
Bench test: Passed
Burnin test: Passed
Test Date: 04/09/17
Date of manufacture at CM: 04/09/17
Date received at factory: 04/10/17
Date of burn in: 04/09/17
Mfg Test: Passed
RMA Date: 00/00/00
RMA Info:
LID info: pattern=b6cd, LID in EEPROM=eatIIGIpFIL, LID str=eatIIGIpFIL

 

[crackelf]

@fohdeesha any ideas before I return these ConnectX-4s? Happy to put a bounty on getting these working if that encourages anyone...

Anything I need to do on the Brocade side specifically to get these working? Connecting two cards together = connection, connecting the brocade ports together = connection, but I can't get the card to talk to the Brocade at all.

From the Brocade side we get

1/3/1      Down    None    None None  None  No  1    0   cc4e.2488.3380      
1/4/1      Down    None    None None  None  No  1    0   cc4e.2488.3380

 

[fohdeesha]

@fohdeesha any ideas before I return these ConnectX-4s? Happy to put a bounty on getting these working if that encourages anyone...

once everything is plugged in and modules are installed etc, try rebooting the icx and see if it comes up after its done booting. if the cards work connected to eachother the cards are definitely fine, assuming you don't have them locked in mellanox's special 56gbE mode or something

 

[crackelf]

once everything is plugged in and modules are installed etc, try rebooting the icx and see if it comes up after its done booting. if the cards work connected to eachother the cards are definitely fine, assuming you don't have them locked in mellanox's special 56gbE mode or something

No luck after a good old reload. These LX series are allegedly ethernet only, how would I check these aren't outputting @50GbE instead of 40 (they're apparently capable of both). I've disabled stacking & the 40G ports are happy being plugged into themselves, so it doesn't feel like a Brocade side thing at this point.

       description: Ethernet interface
       product: MT27710 Family [ConnectX-4 Lx Virtual Function]
       vendor: Mellanox Technologies
       physical id: 0.7
       bus info: pci@0000:03:00.7
       logical name: enp3s0v6
       version: 00
       serial: ce:a2:52:84:1f:cb
       capacity: 40Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pciexpress msix bus_master cap_list ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 firmware=14.31.1014 (MT_2430110032) latency=0 link=no multicast=yes port=fibre
       resources: irq:0 memory:ea600000-ea6fffff

#ethtool enp3s0v6
Settings for enp3s0v6:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no

 

[fohdeesha]

No luck after a good old reload. These LX series are allegedly ethernet only, how would I check these aren't outputting @50GbE instead of 40 (they're apparently capable of both). I've disabled stacking & the 40G ports are happy being plugged into themselves, so it doesn't feel like a Brocade side thing at this point.

       description: Ethernet interface
       product: MT27710 Family [ConnectX-4 Lx Virtual Function]
       vendor: Mellanox Technologies
       physical id: 0.7
       bus info: pci@0000:03:00.7
       logical name: enp3s0v6
       version: 00
       serial: ce:a2:52:84:1f:cb
       capacity: 40Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pciexpress msix bus_master cap_list ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 firmware=14.31.1014 (MT_2430110032) latency=0 link=no multicast=yes port=fibre
       resources: irq:0 memory:ea600000-ea6fffff

#ethtool enp3s0v6
Settings for enp3s0v6:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no

is this a 6610? I've seen this happen with these exact reload on 6610s on the 40gbe ports when used for non-stacking, but a reboot always brought them up

 

[crackelf]

is this a 6610? I've seen this happen with these exact reload on 6610s on the 40gbe ports when used for non-stacking, but a reboot always brought them up

ICX 7450-48P

 

[crackelf]

Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!

yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

 

[fohdeesha]

ICX 7450-48P

that's even weirder, it's a stretch but what fw are you on? 8095? try going down to 8090mc (its in the website guide package)

also paste me the output of "show run" and "show mod", there might be some stacking port definitions you can try to remove. lastly maybe try forcing the port speed if it'll let you, something like int e 1/3/1 then type speed and hit tab a bunch to see the options