Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
T

TheCodeLife

New Member
Mar 29, 2019
25
3
3
Just want to thank you fohdeesha for all you have done here! I was able to get enough SFP+ ports this way to run fiber throughout my house. Such a tremendous help and I'm sad to see this occur. I have no intention of selling my switch at this point, but is there any chance you could put up enough of the guide for removing all of the configuration for the licenses you helped us with so we can set our switches back to default prior to selling? The switch I purchased was already partially licensed and I would want to set it back to that state before selling. I'm sure a number of us on this site had partially licensed switches and would revert back to that state when selling.
 
  • Like
Reactions: maes and Hrast
D

Dave Corder

Active Member
Dec 21, 2015
333
225
43
42
I'm a little confused on something.

On the 6610-48p, can all four QSFP ports do a single 40gb link, or just 1/2/1 and 1/2/6 (with the other two being 4x10gb breakouts only)?
 
D

Dave Corder

Active Member
Dec 21, 2015
333
225
43
42
Ok, I'm having some trouble with one of my 40gb ports on my 6610-48P.

I picked up a pair of these cables to use with my FreeNAS system and my Proxmox hypervisor: HP 498385-B23 3M 4X DDR/QDR QSFP IB CU CABLE *New Bulk* - 503815-002

I am using Mellanox ConnectX-3 cards in each system (technically "HP 656089-001 Infiband 4 X 10/40GB DP adapter 544QSFP"), both flashed to the official Mellanox firmware and set to Ethernet mode).

Port 1/3/7 is currently connected to port 1 on the Proxmox system with a QSFP->SFP+ adapter and a SFP+ 10gb SR optic. Port 1/2/6 is connected to port 1 on the FreeNAS system. I'm attempting to connect the Proxmox system to 1/2/1.

Port 1/2/6 works fine. I duplicated (I think) my port config from 1/3/7 to port 1/2/1. Both the switch and Promox detect that the cable is plugged in, but I don't get a link.

I swapped cables (knowing the FreeNAS cable works fine), no change.

I swapped ports (FreeNAS to 1/2/1 and Proxmox to 1/2/6) and still 1/2/1 did not work.

Right now I've got 1/2/1 connected to port 2 on the Proxmox system while I troubleshoot. It doesn't work there, either.

Anyone have any ideas? Did I miss a step in my configuration? Are there some diagnostics I can run on the switch to test the port? Or do I just have a dead port?

Code: 
SSH@sw-core-01>show config
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30saT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-trunk 1/2/1 to 1/2/2
  stack-trunk 1/2/6 to 1/2/7
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 2 by port
!
vlan 10 name LAN by port
 tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/2/6 ethe 1/3/1 to 1/3/8
 untagged ethe 1/1/13 to 1/1/24
 router-interface ve 10
!
vlan 20 by port
 tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/3/1 to 1/3/6 ethe 1/3/8
!
vlan 30 by port
 tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/3/1 to 1/3/6 ethe 1/3/8
!
vlan 40 by port
 tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/3/1 to 1/3/6 ethe 1/3/8
!
vlan 99 name storage by port
 tagged ethe 1/2/1 ethe 1/2/6 ethe 1/3/1 ethe 1/3/7 to 1/3/8
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname sw-core-01
ip dhcp-client disable
!
username root password .....
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/45
 dual-mode
!
interface ethernet 1/1/46
 dual-mode
!
interface ethernet 1/1/47
 port-name uap-pro-01
 dual-mode
!
interface ethernet 1/1/48
 port-name uap-pro-02
 dual-mode
 inline power
!
interface ethernet 1/2/1
 port-name proxmox-01-40gb
 dual-mode
!
interface ethernet 1/2/6
 port-name lcars-40gb
 dual-mode  10
!
interface ethernet 1/3/1
 port-name pfsense
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 port-name lcars-10gb
 dual-mode  10
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 port-name proxmox-01-10gb
 dual-mode
 speed-duplex 10G-full
!
interface ve 1
 ip address 10.0.1.2 255.255.255.0
!
interface ve 10
 ip address 10.0.10.2 255.255.255.0
!
!
!
!
!
!
!
!
!
end

Code: 
root@proxmox-01:~# ethtool enp5s0d1
Settings for enp5s0d1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 40000Mb/s
    Duplex: Full
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000014 (20)
                   link ifdown
    Link detected: no

Code: 
SSH@sw-core-01>show interface eth 1/2/1
40GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 1 minute(s) 14 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f834.bc80 (bia 748e.f834.bcb1)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of 6 L2 VLANs, port is dual mode in Vlan 1, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is proxmox-01-40gb
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  10568 packets input, 1170346 bytes, 0 no buffer
  Received 7580 broadcasts, 2751 multicasts, 237 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  0 packets output, 0 bytes, 0 underruns
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0                1138           204491026
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                  13                   0
    6                   0                   0
    7                   0                   0
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,828
3,269
113
33
fohdeesha.com
try this, sometimes the rear 40gb ports can be weird the first time they link, esp if you haven't removed the stack trunks


enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
no stack-trunk 1/2/6 to 1/2/7
exit
write mem
exit
reload
 
D

DavidB

Member
Aug 31, 2018
60
19
8
mb300sd said:
Ordered some material, should have some ready to go next week for those interested.
Click to expand...
definitely interested. Right now it's resting on top of an old supermicro case and I may or may not have dropped it twice as a result of pulling out a DAC cable too enthusiastic
 
D

Dave Corder

Active Member
Dec 21, 2015
333
225
43
42
fohdeesha said:
try this, sometimes the rear 40gb ports can be weird the first time they link, esp if you haven't removed the stack trunks


enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
no stack-trunk 1/2/6 to 1/2/7
exit
write mem
exit
reload
Click to expand...
No luck with these commands. Also tried removing and reinserting the cable a few times on the switch side. I also inspected the inside of the port with a flashlight and it looked OK to me, but it's at the bottom of my rack and I was crouched on the floor, so I can't be too certain. I'm going to try blowing it out with some compressed air (as soon as I find some), just to make sure there's no dust bunnies in there or something.

It looks like QSFP SR optics are pretty cheap, too, so I'm going to order a pair and give those a try and see if I have any better luck than with the DAC.

I'm several months past the purchase date past on eBay with this switch, so I doubt I have any recourse with the seller, but if I can't get the port to work I'll at least reach out to them and see...
 
D

Dave Corder

Active Member
Dec 21, 2015
333
225
43
42
On the off chance that this is an issue on the Mellanox ConnectX-3 card side (both were flashed with the latest firmware and set to ETH mode), I flashed them following fohdeesha's guide and this is my current status of the card in the two systems:

Code: 
root@proxmox-01:~/mft-4.12.0-105-x86_64-deb# mlxconfig -d /dev/mst/mt4099_pci_cr0 query

Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                              Next Boot
fl         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        ETH(2)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
root@proxmox-01:~/mft-4.12.0-105-x86_64-deb# flint -d /dev/mst/mt4099_pci_cr0 query
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 24be05ffffa03cc0 24be05ffffa03cc1 24be05ffffa03cc2 24be05ffffa03cc3
MACs:                                       24be05a03cc1     24be05a03cc2
VSD:
PSID:                  MT_1090120019

Code: 
➜  mft-4.12.0-105 sudo mlxconfig -d pci0:3:0:0 query

Device #1:
----------

Device type:    ConnectX3
Device:         pci0:3:0:0

Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        ETH(2)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)
➜  mft-4.12.0-105 sudo flint -d pci0:3:0:0 query
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0002c9030040f720 0002c9030040f721 0002c9030040f722 0002c9030040f723
MACs:                                       0002c940f720     0002c940f721
VSD:
PSID:                  MT_1090120019
 
Last edited:
A

ambient1

New Member
Apr 10, 2019
1
0
1
Ontario, Canada
Thanks to this thread, I've had my ICX6610-24P for a few weeks. It only came with one psu and one fan, and was missing the blanks for the rest. I ended up paying $200 delivered. It was the regular front-to-back airflow, but I like my switches in the rear of the rack. I ended up turning all the fans around and that seems to have worked fine. For the psu fan I ended up not being able to reuse the silicone tabs and just using machine screws to mount the fan. I followed fohdeesha's excellent guide and got everything mostly up and running. I'm still waiting for transceivers and cabling before I move my lab to 10GB.

I also have a ICX6610-48P in the mail which was a barebones unit without fans or psus. I thought I got it for a good deal ($90 delivered) but the prices of fans and psus are ridiculous.:mad:
 
You must log in or register to reply here.
Top Bottom