Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

crackelf

Member
Apr 11, 2021
71
2
8
on any OS or just with debian? If the latter, this issue seems to be better discussed in one of the debian/linux forums..
Windows, FreeBSD, Debian. Link me an ISO I'll install it.
Because there's nothing to figure out if it just works for everyone...
Oh except for the $2k worth of 40G equipment I'm staring at that doesn't work, but I guess that's not *your* problem though so it doesn't matter, does it? Really lovely forum we have here guys.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,664
113
31
fohdeesha.com
I haven't noticed that. In fact two hours ago:
I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
Since we're reading together (I love group activities), you'll note the part where he's talking about mellanox switches, and infiniband - you have neither. You left out the part where even he himself said:

In cards in Ethernet everything works fine
I'll help you out: you're using a card, and ethernet. We have a match!

because literally nothing in observable science is 100%.
Other than the chances of you no longer getting any help in this thread after you've blamed everyone for your purchase of unknown vendor origin cards lmao
 

crackelf

Member
Apr 11, 2021
71
2
8
If you want to talk about wasting time, try crossflashing, modifying, or recovering bricked intel, qlogic, broadcom, etc NICs on boards with no rescue bootloader (one of Mellanox's best features, to be honest)
That's a nice chip on your shoulder that you spend your spare time debugging NICs. I prefer spending time with loved ones, but to each their own.
 

crackelf

Member
Apr 11, 2021
71
2
8
Since you've all confirmed that the issue you are debugging is unrelated to any Brocade ICX product, can you move this discussion to a different thread?
Sorry hall monitor, didn't know I needed my hall pass around here. Will update my other thread and dip. Enjoy STH!
 
Last edited:

i386

Well-Known Member
Mar 18, 2016
2,971
943
113
33
Germany
take a picture of the nics (not the stock pictures from the seller on ebay), front and back side with labels/sticker visible (including product/serial numbers). could increase the chances to identify your nics/problems.
 

i386

Well-Known Member
Mar 18, 2016
2,971
943
113
33
Germany
I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"
 
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
71
2
8
I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)
I agree. I was fully expecting HP flashed NICs when I bought these and got these instead.
Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"
Thank you very very much for the insight and references. I would never have known this. I appreciate your help. I've flopped between it being the cards, the fiber, the transceivers and the switch about 100 times, but something is definitely up here that doesn't seem to be software related. I have some ConnectX-3's coming in along with a few other cables / transcievers and will update my post over here if I ever get to the bottom of this.
 
  • Like
Reactions: kpfleming

NablaSquaredG

Well-Known Member
Aug 17, 2020
604
259
63
BTW, I have just ordered a bunch of transceivers from the USA, including the recommended XQX2502 because I really want to see how they behave.
 

crackelf

Member
Apr 11, 2021
71
2
8
after reading through all of "that" - I find myself using the ignore member function for the first time. :eek: :(
XDDDDD LAWL ROFLCOPTER don't worry I'll let you get back to your beloved men's support forum.
Mod edited for crimes against vision.
 
Last edited by a moderator:

RobstarUSA

Active Member
Sep 15, 2016
200
63
28
Just had my first real weirdness with my 6610. I had a unifi access point go offline. Weird. I power cycled the POE port on my 6610, and while the port has 2 lights on, the cli says the port is down. I try the port next to it (19)...same issue. I grab a new spare out of the box, same issue. I test the RJ45 cable with a tester (all pairs good). I replace the cable anyway, same issue.

I reload the POE stack member these are attached into and everything is perfect again. ARGHH
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,664
113
31
fohdeesha.com
Just had my first real weirdness with my 6610. I had a unifi access point go offline. Weird. I power cycled the POE port on my 6610, and while the port has 2 lights on, the cli says the port is down. I try the port next to it (19)...same issue. I grab a new spare out of the box, same issue. I test the RJ45 cable with a tester (all pairs good). I replace the cable anyway, same issue.

I reload the POE stack member these are attached into and everything is perfect again. ARGHH
the spare you grabbed and tested, was it a spare AP, or a space ICX? if you had the problem even on a separate newly booted ICX, that def sounds like the unifi device was having issues. if it was a spare AP, and the same icx with the prob was still having the issue on the different AP, then yeah it sounds like a strange icx bug. what OS and PoE fw version are you on?
 
  • Like
Reactions: RobstarUSA

RobstarUSA

Active Member
Sep 15, 2016
200
63
28
the spare you grabbed and tested, was it a spare AP, or a space ICX? if you had the problem even on a separate newly booted ICX, that def sounds like the unifi device was having issues. if it was a spare AP, and the same icx with the prob was still having the issue on the different AP, then yeah it sounds like a strange icx bug. what OS and PoE fw version are you on?
Sorry for the confusion. I tried spare ap, cable & port in the same stack member. I have a 2 stack setup using the 40G QSFP connectors on back. I don't have another ICX6610 that has POE, so as a last resort I kicked just the POE stackmember which has my access points (only) in it.

I did everything according to the guide you setup (much thanks btw!).

SSH@BCORE#sh ver | i Version
SW: Version 08.0.30uT7f3
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)

Not sure offhand how to get the poe firmware version, but I remember upgrading it as per your guide.

I kinda wish I had waited for a deal, as I found the 48 port POE model with 2 PS for about $150 shipped on eBay which I thought was superb.
For my first one, non-poe, 48 port, It was $90 but only with one power supply.

To be fair I use Cisco all day at my day job and have had, at one time, 3 Cisco TAC Teams working for 24 hours straight on a multicast over GRE over ipsec issue that was not solvable by them. I found a workaround as I was on a tight time schedule & we eventually ended up removing that equipment as it was older (although still supported). I"m kinda famous @ work for finding issues :) There is an HP G6 or G7 BIOS out there that has a bios memory setting bug I found. HP Later released an updated bios and the only fix was the bug I found. So maybe it's just me & not the equpment :)
 
Last edited:

crackelf

Member
Apr 11, 2021
71
2
8
@fohdeesha The Infallible, confirmed fallible - ConnectX-4 Lx DO NOT work with KAIAM modules:
Another user is saying the fiber I have & transceivers won't work with the card...but I was convinced otherwise:
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
Code:
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core
Code:
show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
Code:
Settings for enp1s0np0:
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no
Proof that Mellanox programmed transceivers work:
Code:
show media ethernet 1/4/1
Port   1/4/1: Type  : 40GBASE-Passive Copper  2m (QSFP+)
         Vendor: Mellanox             Version: A3
         Part# : MC2207130-002        Serial#: MT1433VS02614
Code:
Settings for enp1s0np0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: yes
Proof that KAIAM transceivers work properly with ConnectX-3 cards:
Code:
02:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
    Kernel driver in use: mlx4_core
    Kernel modules: mlx4_core
Code:
show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
Code:
Settings for enp2s0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes
but as you've noticed, you're the only one in this thread having these issues. You could have bad fiber, you could have newer or older firmware, your motherboard might not like supplying extra power to the NIC, who knows.
I haven't noticed that. In fact two hours ago:
I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...
I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"
He's wrong like most people on the internet
So you're infallible? To me everyone here is just text on a screen. It's all equal in my eyes until I see something functional.