Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[andreathing]

Has anyone done any progress on this error?

brocade02(config)#M:9 L:0 - chow_qsfp_read, qsfp 0, error in reading qsfp
M:9 L:0 - link_40G_4x10G_get_media: qsfp 0, port 1/2/1 error in reading qsfp

I thought i had faulty 40g ports, but it turns out the link work perfectly using dac cables. I've tried 4 different xbr-00232 modules so far on both ports of two different switches and i get the same error every time. It seems highly unlikely to me that all 4 modules are broken and since the link work perfectly with the dac cable (even connecting to a mellanox sx6036) i doubt the port are fried either. Btw config is vanilla as per @fohdeesha guide, just installed latest firmware and removed stacking.

 

[klui]

Has anyone done any progress on this error?

brocade02(config)#M:9 L:0 - chow_qsfp_read, qsfp 0, error in reading qsfp
M:9 L:0 - link_40G_4x10G_get_media: qsfp 0, port 1/2/1 error in reading qsfp

Although I get something similar during insertion, show media ethernet .... shows the vendor and serial and the port links up. Those are the important behaviors. I suppose it's due to the type of transceiver.

ICX6610-48P Router>sh media eth 1/2/1
Port 1/2/1:Type : 40G QSFP Module
Vendor Name: BROCADE Serial Num: HTA11402000010N Revision: A

 

[andreathing]

Although I get something similar during insertion, show media ethernet .... shows the vendor and serial and the port links up. Those are the important behaviors. I suppose it's due to the type of transceiver.

ICX6610-48P Router>sh media eth 1/2/1
Port 1/2/1:Type : 40G QSFP Module
Vendor Name: BROCADE Serial Num: HTA11402000010N Revision: A

Ok, i get the same result. I must confess i panicked at the error and stopped there, afraid to have just bought 4 not-so-expensive paperweights... I have the same result as you:

SSH@brocade02#show media eth 1/2/6
Port   1/2/6:Type  : 40G QSFP Module
Vendor Name: BROCADE          Serial Num: HTA214320000041 Revision: A

Now i have to understand why the link doesn't come up... I thought the cable was good (b polaryty MTP).

Edit: just a bad cable, the error has been a red herring. As soon as i changed it i got the link up.

Thank you again @klui

 

[klui]

Now i have to understand why the link doesn't come up... I thought the cable was good (b polaryty MTP).

Edit: just a bad cable, the error has been a red herring. As soon as i changed it i got the link up.

Bad cable or bad part #? I purchased the exact model @datanet recommended at https://forums.servethehome.com/index.php?threads/mto-questions.31800/post-298142.

 

[rootwyrm]

So, one of my 6450's is behaving ODDLY. Let me show you:

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 57.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 46.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

SSH@6450-2d#dm fan-speed
All fans in High speed (0xff)

"Then your fans should be in high speed." Yes, except check the A/B differential. It's 11C lower. And I'm measuring the air at the exhaust side at 34C, so there's no way it's 58C. B isn't changing even when deliberately fed hot or cold air, but A is updating. Anyone ever seen this before?

 

[andreathing]

Bad cable or bad part #? I purchased the exact model @datanet recommended at https://forums.servethehome.com/index.php?threads/mto-questions.31800/post-298142.

I think it can be the latter, i didn't buy the exact one you linked but one that was readily available on amazon where i live (Europe). I think i'll just send it back. That error really made me go on the wrong path but luckily the optics seem to work just fine.

 

[fortunamatada]

First, thanks to fohdeesha for the work on these units!

Have an issue that I am slamming my head against the wall trying to figure out: an icx 7250 in router mode with everything working as expected except for one vlan on which NONE of the hosts can talk to one another. They can talk outside of the VLAN, they can reach the internet, hosts on other vlans can talk to the hosts inside of this vlan. All of the devices are untagged. No vlan tagging is configured on any of the hosts. All ports are forwarding. Devices in other vlans (i.e vlan 100) have no problem communicating with each other. Disabled all firewalls to make sure this was not an issue. No ACLs enabled yet.

Spoiler: sh run

Current configuration:
!
ver 08.0.95caT213
!
stack unit 1
  module 1 icx7250-24p-poe-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
  stack-port 1/2/1
stack disable
!
global-stp
!
default-vlan-id 1001
!
vlan 31 by port
 tagged ethe 1/2/8
 router-interface ve 31
 spanning-tree
!
vlan 95 by port
 untagged ethe 1/1/5 to 1/1/8                                     
 router-interface ve 95
 spanning-tree 802-1w
!
vlan 100 by port
 tagged ethe 1/2/1 to 1/2/2 ethe 1/2/4 to 1/2/5 ethe 1/2/7
 untagged ethe 1/2/3
 router-interface ve 100
 spanning-tree 802-1w
!
vlan 250 by port
 untagged ethe 1/1/1 ethe 1/1/3 ethe 1/1/17 to 1/1/24
 router-interface ve 250
 spanning-tree 802-1w
 loop-detection
!
vlan 1001 name DEFAULT-VLAN by port
 router-interface ve 1001
!
system-max ip-route-default-vrf 9000
system-max ip-route-vrf 128
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
enable aaa console
hostname unimatrix001
ip dhcp-client disable
ip dns server-address 192.168.100.2
ip route 0.0.0.0/0 172.21.31.1
!
no telnet server
route-only
!
clock summer-time
!
!
ntp
 server 172.21.31.1
!
!
web-management https
web-management enable vlan 250
!
manager disable
!
manager port-list 987
!
interface ve 31
 ip address 172.21.31.2 255.255.255.252
!
interface ve 95
 ip address 192.168.95.1 255.255.255.0
 ip helper-address 1 192.168.100.2
!
interface ve 100
 ip address 192.168.100.1 255.255.255.0
 ip helper-address 1 192.168.100.2
!
interface ve 250
 ip address 192.168.250.1 255.255.255.0 dynamic
 ip helper-address 1 192.168.100.2
!
interface ve 1001                                                 
 ip address 192.168.148.1 255.255.255.0
 ip helper-address 1 192.168.100.2
!

end

Spoiler: sh vlan

Total PORT-VLAN entries: 5
Maximum PORT-VLAN entries: 1024

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 31, Name [None], Priority level0, On
 Untagged Ports: None
   Tagged Ports: (U1/M2)   8 
 Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 95, Name [None], Priority level0, On
 Untagged Ports: (U1/M1)   5   6   7   8 
   Tagged Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 100, Name [None], Priority level0, On
 Untagged Ports: (U1/M2)   3 
   Tagged Ports: (U1/M2)   1   2   4   5   7 
 Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 250, Name [None], Priority level0, On                   
 Untagged Ports: (U1/M1)   1   3  17  18  19  20  21  22  23  24 
   Tagged Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

PORT-VLAN 1001, Name DEFAULT-VLAN, Priority level0, Off
 Untagged Ports: (U1/M1)   2   4   9  10  11  12  13  14  15  16 
 Untagged Ports: (U1/M2)   1   2   4   5   6   7   8 
   Tagged Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

Spoiler: sh ip route

Total number of IP routes: 6
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          172.21.31.1     ve 31         1/1           S    2h13m 
2       172.21.31.0/30     DIRECT          ve 31         0/0           D    2h13m 
3       192.168.95.0/24    DIRECT          ve 95         0/0           D    1h40m 
4       192.168.100.0/24   DIRECT          ve 100        0/0           D    2h14m 
5       192.168.148.0/24   DIRECT          ve 1001       0/0           D    2h14m 
6       192.168.250.0/24   DIRECT          ve 250        0/0           D    2h14m

Spoiler: sh 820.1w

--- VLAN 250 [ STP Instance owned by VLAN 250 ] ----------------------------

Bridge IEEE 802.1W Parameters:

Bridge           Bridge Bridge Bridge Force    tx   
Identifier       MaxAge Hello  FwdDly Version  Hold 
hex              sec    sec    sec             cnt  
800078a6e127d110 20     2      15     Default  3    

RootBridge       RootPath  DesignatedBri-   Root      Max Fwd Hel 
Identifier       Cost      dge Identifier   Port      Age Dly lo  
hex                        hex                        sec sec sec 
800078a6e127d110 0         800078a6e127d110 Root      20  15  2   

Port IEEE 802.1W Parameters:

          <--- Config Params --><-------------- Current state -----------------> 
Port      Pri PortPath P2P Edge Role       State       Designa-  Designated       
Num           Cost     Mac Port                        ted cost  bridge           
1/1/1     128 20000    F   F    DESIGNATED FORWARDING  0         800078a6e127d110 
1/1/3     128 0        F   F    DISABLED   DISABLED    0         0000000000000000 
1/1/17    128 20000    F   F    DESIGNATED FORWARDING  0         800078a6e127d110 
1/1/18    128 20000    F   F    DESIGNATED FORWARDING  0         800078a6e127d110 
1/1/19    128 0        F   F    DISABLED   DISABLED    0         0000000000000000 
1/1/20    128 0        F   F    DISABLED   DISABLED    0         0000000000000000 
1/1/21    128 0        F   F    DISABLED   DISABLED    0         0000000000000000 
1/1/22    128 20000    F   F    DESIGNATED FORWARDING  0         800078a6e127d110 
1/1/23    128 20000    F   F    DESIGNATED FORWARDING  0         800078a6e127d110 
1/1/24    128 20000    F   F    DESIGNATED FORWARDING  0         800078a6e127d1

VRFs have not been enabled.
I know this has to be some absolutely simple oversight. Does anyone have a clue what I messed up?

 

[infoMatt]

Does anyone have a clue what I messed up?

Remove "route-only" from your configuration.

route-only

Enables Brocade Layer 3 switches to support Layer 2 switching.

docs.ruckuswireless.com

 

[fortunamatada]

Remove "route-only" from your configuration.

route-only

Enables Brocade Layer 3 switches to support Layer 2 switching.

docs.ruckuswireless.com

It is in layer 3 mode. That is how it is supposed to be. Hence the VEs for each vlan.

 

[fohdeesha]

It is in layer 3 mode. That is how it is supposed to be. Hence the VEs for each vlan.

n....no lol. route-only does exactly the behavior you're complaining about, it completely disables layer2 switching, so hosts in the same l2/vlan are not going to talk to each other. did u see that command in my guide? hell nah didn't think so fam QED please do the needful and remove

 

[fortunamatada]

n....no lol. route-only does exactly the behavior you're complaining about, it completely disables layer2 switching, so hosts in the same l2/vlan are not going to talk to each other. did u see that command in my guide? hell nah didn't think so fam QED please do the needful and remove

oh so sorry i misinterpreted " Enabling or disabling Layer 2 switching is not supported on virtual interfaces. "

 

[NablaSquaredG]

Eh - Stupid question:

Is there any hardware difference between front-to-back and rear-to-front airflow ICX6610s?

i.e. can I just open the fan modules and PSUs and turn the fans around?

I've got a Ports-to-PSU one here and I'd like to make it a PSU-to-Ports one. Does anyone have a PSU-to-Ports one and would mind sharing some pictures of the fan tray?

 

[Freebsd1976]

now i have icx6450-24, want to replace it with icx7150-24 , what is pros and cons?

 

[fohdeesha]

Eh - Stupid question:

Is there any hardware difference between front-to-back and rear-to-front airflow ICX6610s?

i.e. can I just open the fan modules and PSUs and turn the fans around?

I've got a Ports-to-PSU one here and I'd like to make it a PSU-to-Ports one. Does anyone have a PSU-to-Ports one and would mind sharing some pictures of the fan tray?

Same hardware

 

[kapone]

Eh - Stupid question:

Is there any hardware difference between front-to-back and rear-to-front airflow ICX6610s?

i.e. can I just open the fan modules and PSUs and turn the fans around?

I've got a Ports-to-PSU one here and I'd like to make it a PSU-to-Ports one. Does anyone have a PSU-to-Ports one and would mind sharing some pictures of the fan tray?

The fan trays are seriously difficult to take apart and turn the fans around and so are the PSU fans. Ask me how I know....

 

[NablaSquaredG]

The fan trays are seriously difficult to take apart and turn the fans around and so are the PSU fans. Ask me how I know....

Haha, thanks for the warning. Any hints I should know?

 

[kapone]

Haha, thanks for the warning. Any hints I should know?

Yup. Small hands and a low power solder gun (those wires are quite thin).

 

[NablaSquaredG]

Solder Gun? I was planning to just physically turn the fans around

 

[nerdalertdk]

Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.

edgeswitch ears works great
My 7150-48zp hangs fine in just the front ones

 

[kapone]

Solder Gun? I was planning to just physically turn the fans around

The fan trays use a custom connector with the wires (from the fans) connected straight to it. The length of those wires is just right. i.e. if you flip the fan, the wire is too short. You'll have to take apart the tray, including the connector and fans (which use rubber screws that are really hard to take out and even harder to put back in), cut the wires, splice in just a bit more extra wire, to flip them around.

Small hands and a solder gun...