Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

Yeah, it sees the transceiver but no carrier. If the card is in Auto or IB mode that would probably be the issue. IB mode for sure would be a problem; I've seen reports that Mellanox cards in Auto mode does not play nice with Ethernet switches.

mlxconfig -d <device id> set LINK_TYPE_P1=2 (LINK_TYPE_P2 for port 2 if present)
It will show you the current current config, proposed new config, and a confirm prompt.

I have CX354 (CX3-Pro) cards, so older, running in Windows.

More mlxconfig stuff; including SR-IOV VF configuration on the card itself: Using mlxconfig - MFT v4.11.0 - NVIDIA Networking Docs

 

[crackelf]

I have mstconfig as a part of the mstflint package. Does that make a difference?

-E- The Device doesn't support LINK_TYPE_P1 parameter

It looks like it's in ethernet mode though when I look at ip link show. Here is my current config

I have mstconfig -d 03:00.0 set SRIOV_EN=1 and mstconfig -d 03:00.0 set NUM_OF_VFS=8 set. I can see all the VFs, but no dice.

 

[LodeRunner]

Oh right, a -LX card. That appears to be Ethernet only. So makes sense setting the port mode is not supported, it only has Ethernet mode. Given the litany of issues in the other thread, I'm not entirely sure which end of the link the problem is on.

 

[crackelf]

Oh right, a -LX card. That appears to be Ethernet only. So makes sense setting the port mode is not supported, it only has Ethernet mode. Given the litany of issues in the other thread, I'm not entirely sure which end of the link the problem is on.

Hahaha I hear you. I'm about to pick up some CX354A and try those out. These things have been nothing but trouble.

 

[fohdeesha]

@fohdeesha

looks like the latest firmware for the fastiron on ICX platforms has a completely revamped GUI, version 9.0.10a

i know in your informational posts you commended the ICX7250 platform because it used an honor based licensing system, but i was never sure if that was only because it was on v8080 firmware (which i assume translates to 8.0.80 in ruckus firmware notation)

was it ever planned by ruckus/brocade to phase out the honor based licensing for these switches?

i'm aware you may not be the biggest fan of gui for switches but it will be nice to integrate these with my ruckus r500 which have a similar interface aesthetic

we covered the 9x releases a few pages back, they're super buggy alpha releases, you'll see what I mean just watching the boot output of them. the new web UI is pretty, but provides even less actual functionality than the old school one. licenses will always be honor based

 

[fohdeesha]

Is there any way to find out the manifacturing date of an ICX7450 by running a cli command or opening the case?

at enable cli level, show pid-prom

SSH@sw-01-ash#show pid-prom
Version Number: 0003
Part Number: 84-1002981-04
Serial Number: CYR3313N036
Bench test: Passed
Burnin test: Passed
Test Date: 04/09/17
Date of manufacture at CM: 04/09/17
Date received at factory: 04/10/17
Date of burn in: 04/09/17
Mfg Test: Passed
RMA Date: 00/00/00
RMA Info:
LID info: pattern=b6cd, LID in EEPROM=eatIIGIpFIL, LID str=eatIIGIpFIL

 

[crackelf]

@fohdeesha any ideas before I return these ConnectX-4s? Happy to put a bounty on getting these working if that encourages anyone...

Anything I need to do on the Brocade side specifically to get these working? Connecting two cards together = connection, connecting the brocade ports together = connection, but I can't get the card to talk to the Brocade at all.

From the Brocade side we get

1/3/1      Down    None    None None  None  No  1    0   cc4e.2488.3380      
1/4/1      Down    None    None None  None  No  1    0   cc4e.2488.3380

 

[fohdeesha]

@fohdeesha any ideas before I return these ConnectX-4s? Happy to put a bounty on getting these working if that encourages anyone...

once everything is plugged in and modules are installed etc, try rebooting the icx and see if it comes up after its done booting. if the cards work connected to eachother the cards are definitely fine, assuming you don't have them locked in mellanox's special 56gbE mode or something

 

[crackelf]

once everything is plugged in and modules are installed etc, try rebooting the icx and see if it comes up after its done booting. if the cards work connected to eachother the cards are definitely fine, assuming you don't have them locked in mellanox's special 56gbE mode or something

No luck after a good old reload. These LX series are allegedly ethernet only, how would I check these aren't outputting @50GbE instead of 40 (they're apparently capable of both). I've disabled stacking & the 40G ports are happy being plugged into themselves, so it doesn't feel like a Brocade side thing at this point.

       description: Ethernet interface
       product: MT27710 Family [ConnectX-4 Lx Virtual Function]
       vendor: Mellanox Technologies
       physical id: 0.7
       bus info: pci@0000:03:00.7
       logical name: enp3s0v6
       version: 00
       serial: ce:a2:52:84:1f:cb
       capacity: 40Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pciexpress msix bus_master cap_list ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 firmware=14.31.1014 (MT_2430110032) latency=0 link=no multicast=yes port=fibre
       resources: irq:0 memory:ea600000-ea6fffff

#ethtool enp3s0v6
Settings for enp3s0v6:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no

 

[fohdeesha]

No luck after a good old reload. These LX series are allegedly ethernet only, how would I check these aren't outputting @50GbE instead of 40 (they're apparently capable of both). I've disabled stacking & the 40G ports are happy being plugged into themselves, so it doesn't feel like a Brocade side thing at this point.

       description: Ethernet interface
       product: MT27710 Family [ConnectX-4 Lx Virtual Function]
       vendor: Mellanox Technologies
       physical id: 0.7
       bus info: pci@0000:03:00.7
       logical name: enp3s0v6
       version: 00
       serial: ce:a2:52:84:1f:cb
       capacity: 40Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pciexpress msix bus_master cap_list ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 firmware=14.31.1014 (MT_2430110032) latency=0 link=no multicast=yes port=fibre
       resources: irq:0 memory:ea600000-ea6fffff

#ethtool enp3s0v6
Settings for enp3s0v6:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKR/Full
                            40000baseKR4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            40000baseLR4/Full
                            25000baseCR/Full
                            25000baseKR/Full
                            25000baseSR/Full
                            50000baseCR2/Full
                            50000baseKR2/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no

is this a 6610? I've seen this happen with these exact reload on 6610s on the 40gbe ports when used for non-stacking, but a reboot always brought them up

 

[crackelf]

is this a 6610? I've seen this happen with these exact reload on 6610s on the 40gbe ports when used for non-stacking, but a reboot always brought them up

ICX 7450-48P

 

[crackelf]

Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!

yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

 

[fohdeesha]

ICX 7450-48P

that's even weirder, it's a stretch but what fw are you on? 8095? try going down to 8090mc (its in the website guide package)

also paste me the output of "show run" and "show mod", there might be some stacking port definitions you can try to remove. lastly maybe try forcing the port speed if it'll let you, something like int e 1/3/1 then type speed and hit tab a bunch to see the options

 

[crackelf]

that's even weirder, it's a stretch but what fw are you on? 8095? try going down to 8090mc (its in the website guide package)

also paste me the output of "show run" and "show mod", there might be some stacking port definitions you can try to remove. lastly maybe try forcing the port speed if it'll let you, something like int e 1/3/1 then type speed and hit tab a bunch to see the options

Will try the fw downgrade - give me a few moments to run through it. This whole thing is massively weird to me too. I've tried slamming through the speed-duplex menu before

speed-duplex 40g-full
This CLI operation is not applicable to port 1/4/1

show run

Current configuration:
!
ver 08.0.95fT213
!
stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
stack disable
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
!
!                                                                
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
chassis fanless all
enable aaa console
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
!
!
no web-management http
!
manager disable                                                  
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!
interface ethernet 1/2/4
no optical-monitor                                              
speed-duplex 10G-full
!
interface ve 1
ip address [redacted let me know if you actually need this]
no ip dhcp-client enable
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end

show mod

       Module                                         Status Ports Starting MAC 
U1:M1  ICX7450-48P POE 48-port Management Module        OK     48   cc4e.2488.3380
U1:M2  ICX7400-4X10GF 4-port 40G Module                 OK     4    cc4e.2488.3380
U1:M3  ICX7400-1X40GQ 1-port 40G Module                 OK     1    cc4e.2488.3380
U1:M4  ICX7400-1X40GQ 1-port 40G Module                 OK     1    cc4e.2488.3380

 

[fohdeesha]

hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):

show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1

 

[crackelf]

hmm maybe try removing the "stack disable" option, (either "no stack disable" or "stack enable" can't remember which) then rebooting. then if you can paste the output of (assuming 1/3/1 is actually plugged into a card):

show int e 1/3/1
show media ethernet 1/3/1
show optic 1/3/1

Writing this down as I go here
stack enable

Error! stack enable fails due to no stack port. Please configure stack-port/trunk on unit 1. (default: stack-port 1/3/1 stack-port 1/4/1)

stack-port 1/2/3

T=27m26.1: Module 2 was inited with MACsec. Its ports cannot be stack-ports without a reload.
Reload required to form a stack with Module 2 ports. Please write memory and then reload or power cycle.

write mem
reload
show run

stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/2/3
stack disable

stack enable

Success - Enable stacking. This unit actively participates in stacking
Stacking is enabled. Optical Monitoring is not available for the port(s) 1/2/3

show run

stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/2/3
stack enable

write mem
reload
sh int e 1/3/1

  Port down for 7 minute(s) 49 second(s) 
  Hardware is 40GigabitEthernet, address is cc4e.2488.3380 (bia cc4e.2488.33b5)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Untagged member of L2 VLAN 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  0 packets input, 0 bytes, 0 no buffer                           
  Received 0 broadcasts, 0 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  0 packets output, 0 bytes, 0 underruns
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

sh med e 1/3/1

Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209

show optic 1/3/1 returns nothing oddly enough

 

[crackelf]

that's even weirder, it's a stretch but what fw are you on? 8095? try going down to 8090mc (its in the website guide package)

I haven't done the "Load The Latest UFI Image" step yet. Do I need to? I don't want to brick this if copy tftp flash 192.168.1.8 ICX7xxx/SPR08095fufi.bin primary conflicts with 8090mc.
show run

Current configuration:
!
ver 08.0.90mcT213
!
stack unit 1
  module 1 icx7450-48p-poe-management-module
  module 2 icx7400-xgf-4port-40g-module
  module 3 icx7400-qsfp-1port-40g-module
  module 4 icx7400-qsfp-1port-40g-module
  stack-port 1/2/3

sh med e 1/3/1

Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209

ICX7450-48P Router(config-if-e40000-1/3/1)#show int e 1/3/1

40GigabitEthernet1/3/1 is down, line protocol is down
  Port down for 9 minute(s) 46 second(s)
  Hardware is 40GigabitEthernet, address is cc4e.2488.3380 (bia cc4e.2488.33b5)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Untagged member of L2 VLAN 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
[/ICODE]

 

[AgentXXL]

For the 'who the heck cares' crowd: my inability to find an actual Supermicro Aquantia AQS-107-B0C2-CX SFP module led me to purchase a Mikrotik CRS305 and S+RJ10 module. Alas that still seems to have issues with my onboard Aquantia 5Gbps NIC - the AQC111C. Asus support has been useless in my quest to find a firmware updater that works for my onboard NIC. They instead want me to RMA my motherboard. I'm more than certain they'll just say they find nothing wrong and send it back.

So, instead of banging my head against my lack of knowledge, I decided to throw more money at the issue. Anyone using the SFP+ ports on the 6610 with an Intel X550 NIC? As I'm going to follow-through and proceed with the RMA, I needed an option to keep my system running. When I originally built this CoreX system, I wanted to go with the Asus WS x299 SAGE/10 motherboard - this one has dual onboard 10Gbps NICs based on the Intel X550 controller. It goes for about $1K new here in Canada, but there were none to be found so I went with the Asus Prime x299 Deluxe II with the now known to be buggy Aquantia.

Yesterday I found someone selling a used WS x299 SAGE/10 on eBay and took the plunge. I did do some research on the Intel X550 and it seems to be pretty well supported, but there are always going to be setups that have issues. Hopefully those 10G ports will work well with the ICX6610. I'm holding onto the Mikrotik so I'll still have that if the 6610 doesn't like the X550. The replacement motherboard won't be here for at least 10 days so in the meantime I'll continue fighting with the Aquantia issues.

Sigh...

 

[LodeRunner]

The X550 supports 10Gb so there’s no reason it shouldn’t work with the 6610.

 

[AgentXXL]

The X550 supports 10Gb so there’s no reason it shouldn’t work with the 6610.

Yes, that's why I'm more hopeful it'll work. One of my other systems has an Aquantia 10Gbps NIC onboard and the 6610 has no discernible issues with it. I just didn't find anyone posting about issues between the Intel X550 and the 6610 but thought I'd ask.

EDIT: Are there any recommended sites/programs for 'free' learning more about networking? Google/DuckDuckGo list many, but was hoping to find something that uses the Brocade switches for their lessons.

EDIT2: This one? Seems to be SAN but probably contains some network training. Brocade Education