Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Cobra0101]

What code are you running on the switch ?

What VLANs have you defined ? - none in the config you have posted.

What is this switch ?

Craig

ICX 6645 - running in router mode
no vlans defind yet as only got as far as set default route when i tested to see if i could access the web admin page and failed


ICX6450-24 Router#sho ve
version System status
ICX6450-24 Router#sho version
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Nov 27 2017 at 13:41:09 labeled as ICX64R08030q
(9855884 bytes) from Primary ICX64R08030q.bin
SW: Version 08.0.30qT313
Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
HW: Stackable ICX6450-24
==========================================================================
UNIT 1: SL 1: ICX6450-24 24-port Management Module
Serial #: 2ax5o2jk68e
License: ICX6450_PREM_ROUTER_SOFT_PACKAGE (LID: H4CKTH3PLN8)
P-ENGINE 0: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 3 minute(s) 29 second(s)
The system : started=cold start

 

[Cobra0101]

also when i try and create a virtual interface

ICX6450-24 Router(config-vlan-1)#Int ve 1
Error - invalid virtual ethernet interface number.
ICX6450-24 Router(config)#Error - invalid virtual ethernet interface number.
Invalid input -> Error - invalid virtual ethernet interface number.

 

[hmw]

what vendor would you like to trick your boss into thinking you bought?

 

[fohdeesha]

also when i try and create a virtual interface

ICX6450-24 Router(config-vlan-1)#Int ve 1
Error - invalid virtual ethernet interface number.
ICX6450-24 Router(config)#Error - invalid virtual ethernet interface number.
Invalid input -> Error - invalid virtual ethernet interface number.

please follow the guide more closely, that's the wrong command

 

[ServerSemi]

Is there a brocade model that support 2.5/5/10 gbit 45j ports?

 

[bitbckt]

Is there a brocade model that support 2.5/5/10 gbit 45j ports?

Yes, if you have the moolah.

 

[NablaSquaredG]

Is there a brocade model that support 2.5/5/10 gbit 45j ports?

Yes.

ICX 7650-48ZP, ICX 7550-24ZP, ICX 7550-48ZP ICX 7150-C10ZP to name a few.

But all are $$$$$

 

[ramicio]

Is anyone able to help me get to the bottom of the issues I'm having with [probably] the switch? I need to know if I should be asking for a refund for the switch, or other parts, or if there's really nothing wrong anywhere and I'm just an idiot.

 

[itronin]

Is anyone able to help me get to the bottom of the issues I'm having with [probably] the switch? I need to know if I should be asking for a refund for the switch, or other parts, or if there's really nothing wrong anywhere and I'm just an idiot.

1) Did you follow the setup config licensing guide v2 in the first post of this thread?
2) Did you follow the steps to disable stacking - you'll need to do that in order to use the rear 40Gbe and 40Gbe breakout ports (they are different - are you aware of this?)
3) Have you learned how to pull the config from your switch? (you'll need to be able to ssh into the switch or use the console) I believe you've previously been asked to provide the config. The config is super duper helpful (really critical) for remote T/S.

If you haven't done item 1 above then at a minimum you're switch probably isn't where it needs to be for steps 2 and 3. if you purchased the switch from ebay and expected it to work out of the box its VERY UNLIKELY that it will.

One other bit of information, the intel (and oem intel) 10/40Gbe nics can be very picky about DACS and how they self identify. not saying that's an issue here but it is something to be aware of.

there's a world of difference between being ignorant and being an idiot and the first step is figuring out what you know and don't know before you decide to describe yourself as one or the other.

 

[ramicio]

1. Yes
2. Yes
3. No. I was asked to do this, and I do not know how. I asked how to do this, and got no reply. I can SSH into the switch. What do I need to do to supply that information?

Regarding Intel being picky about DACs. I did the xl710_unlocker thing (which is supposed to be for optics, right?). Regardless, I think people keep glossing over the fact that only 1 of the 40 gig ports show any sign of life, and the one that does is riddled with nothing but errors. I don't mean to be rude about that last part, but please tell me what I specifically need to do on the switch to provide the required information.

 

[itronin]

1. Yes
2. Yes
3. No. I was asked to do this, and I do not know how. I asked how to do this, and got no reply. I can SSH into the switch. What do I need to do to supply that information?

Regarding Intel being picky about DACs. I did the xl710_unlocker thing (which is supposed to be for optics, right?). Regardless, I think people keep glossing over the fact that only 1 of the 40 gig ports show any sign of life, and the one that does is riddled with nothing but errors. I don't mean to be rude about that last part, but please tell me what I specifically need to do on the switch to provide the required information.

k.

In step 2 near the end is the following:

Spoiler: Now show the config:

show run

There should no longer be any stack-trunk commands, or any commands referencing stack ports. That's pretty much it! You can now use the ports on the rear as normal. However, see the next section on how they are laid out.

you will need to cut and paste the output and post it here. my advice is to wrap it in "code" and in a "spoiler" so you don't post a huge reply and force people to scroll through it.

Yes I saw your comments and yeah you may have a bad switch - however I don't want to be rude and ASSUME something so I recommend starting at the beginning and provide the config. You have missed going through items 1 and 2 how to pull the config so I provided the linkage on how to do that.

 

[ramicio]

Excellent, thank you. Here it is:

Spoiler

Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-24p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname switcheroo
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/3
inline power
!
interface ethernet 1/1/6
inline power
!
interface ethernet 1/1/7
inline power
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.5 255.255.255.0
!
!
!
!
!
!
!
!
!
end

 

[itronin]

Excellent, thank you. Here it is:

with the QSFP DAC plugged into each of the left side connections please provide the switch's show interface output. It may also be helpful to show status output (from your OS) as well for each port but that is up to you.

recommend using the same cable for each interface test.

 

[ramicio]

with the QSFP DAC plugged into each of the left side connections please provide the switch's show interface output. It may also be helpful to show status output (from your OS) as well for each port but that is up to you.

recommend using the same cable for each interface test.

Spoiler: Plugged into 1/2/1

SSH@switcheroo>show interface ethernet 1/2/1
40GigabitEthernet1/2/1 is up, line protocol is up
  Port up for 1 hour(s) 17 minute(s) 51 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f8fe.93f6 (bia 748e.f8fe.940f)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 224 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1976 bits/sec, 2 packets/sec, 0.00% utilization
  7685 packets input, 826324 bytes, 0 no buffer
  Received 272 broadcasts, 7404 multicasts, 9 unicasts
  33 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 30 giants
  502846 packets output, 56757049 bytes, 0 underruns
  Transmitted 181923 broadcasts, 319738 multicasts, 1185 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0              502861                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   2                   0
    6                   0                   0
    7                   0                   0

tim@fileserver:~$ ethtool enp98s0
Settings for enp98s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   40000baseCR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  40000baseCR4/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Spoiler: Plugged into 1/2/6

SSH@switcheroo>show interface ethernet 1/2/6
40GigabitEthernet1/2/6 is up, line protocol is up
  Port up for 32 second(s)
  Hardware is 40GigabitEthernet, address is 748e.f8fe.93f6 (bia 748e.f8fe.9414)
  Interface type is 40Gig Fiber
  Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 216 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 1416 bits/sec, 1 packets/sec, 0.00% utilization
  10 packets input, 964 bytes, 0 no buffer
  Received 0 broadcasts, 10 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  64 packets output, 6290 bytes, 0 underruns
  Transmitted 20 broadcasts, 44 multicasts, 0 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0                  68                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   0                   0
    6                   0                   0
    7                   0                   0

tim@fileserver:~$ ethtool enp98s0
Settings for enp98s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   40000baseCR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  40000baseCR4/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

I don't really know what's going on and I don't know how to articulate it.

What was in the machine before was a 10G SFP+ NIC. It was enp98s0. So I put this thing in (XL710) and it takes the same name. I've since put in a different 10G NIC and it took the name of enp129s0. All is fine and works there. Got its MAC address in my router for a static lease (192.168.1.2 is the IP of the machine I desire). When I mess around with this XL710 on enp98s0 and I go to kill it with ifdown, it says "DHCPRELEASE of 192.168.1.2 on enp98s0 to 192.168.1.1 port 67 (xid=0x7d5d72e3)" This is after assigning it a static IP in /etc/network/interfaces. I really don't get what's going on, and I am about to just wipe the OS install (which I've had maintained for like 14 years now, and would be a HUGE pain in the butt to start over).

 

[itronin]

I don't really know what's going on and I don't know how to articulate it.

...


STOP! Take a deep breath.

I reviewed your previous posts. I'd like to get some clarity first. I'm going to give you some suggestions in a bit so bear with me.

Your interface output looks "fine" even the small number of giants (hoping they're just 1522 and not something else - which could be coming form something else on your network). I'm going with the test on the 1/2/6 was simply brief as the uptime on it is short. - please confirm

based on the show int I'd say both 40gbe ports are fine.

for each interface test did you have good link / status "blinky lights" on your NIC?
You previously reported you had no electrical activity on the bottom port - is that still the case?

What was in the machine before was a 10G SFP+ NIC. It was enp98s0. So I put this thing in (XL710) and it takes the same name. I've since put in a different 10G NIC and it took the name of enp129s0. All is fine and works there. Got its MAC address in my router for a static lease (192.168.1.2 is the IP of the machine I desire). When I mess around with this XL710 on enp98s0 and I go to kill it with ifdown, it says "DHCPRELEASE of 192.168.1.2 on enp98s0 to 192.168.1.1 port 67 (xid=0x7d5d72e3)" This is after assigning it a static IP in /etc/network/interfaces. I really don't get what's going
on, and I am about to just wipe the OS install (which I've had maintained for like 14 years now, and would be a HUGE pain in the butt to start over).

STOP again! Please don't wipe your 14 year maintained system. If it has been maintained that long I'd either consider it "prod" or of significant sentimental value. Put it back on your network the way it was and leave it alone until you've finished testing the switch.

You are experimenting and troubleshooting - use something else for this process.

Beg, borrow, steal another system. Throw an OS on it (clean install) and let's troubleshoot the rest of the ports on your switch. yeah?

seriously. Don't EFF around with the system you've been using. Use something else to finish testing out the switch.

Next, I saw a comment that made me think you want to rely on DHCP for testing. Bad idea right now.
80% of most network issues are physical layer.
If you rely on DHCP you may question whether DHCP is behaving or you have something else going on. Remove DHCP from the equation. Use static IP's for the remainder of the testing and on a CLEAN system.
that way you can focus on do I have a bad port, do i have a bad cable.
simplify your troubleshooting by simplifying the number of variables in any given test.

 

[kh78]

Hi all,

Hoping you might be able to offer some guidance on moving from L2 to L3 code (I'm doing this really just as a learning exercise, actually do my routing in a VM).

I have a batch of VLANs, with a separate management VLAN (VLAN 10). That VLAN had the management IP on it for inband access, and I also have a separate physical switch for the mgmt network, cabled to the OOB management port on the back.

I've just moved to running the routing l3 image code and figured I'd learn about setting up Multi-VRF and also management-vrf.

I setup a vrf (MGMT), and a router-interface ve 10 on VLAN 10. The router-interface ve 10, is forwarding into vrf MGMT. All is well thus far.

I then applied management-vrf MGMT which was accepted fine, and when I check with 'show management-vrf' it does indeed appear to have worked.

MY QUESTION
The OOB Management port, does not report itself as belonging to my newly defined management vrf (MGMT). I thought that was part of what the 'management-vrf MGMT' command was meant to do, i.e. forward the OOB physical management ethernet interface in to whatever VRF you had defined as the management-[B]vrf?

Basically everything below points to my inband ve 10 interface being in the managment vrf correctly, but not the OOB interface. How do I get the OOB to forward into the defined managment-vrf?

show who[/B]

SSH connections (inbound):
1      established, client ip address 192.168.1.50, server hostkey RSA, user is USER, privilege super-user, in config mode
        using vrf default-vrf.
        you are connecting to this session
        10 second(s) in idle

^^^ That particular SSH session is to the IP address that is configured on the OOB management port, and as you can see, it reports it's in the default-vrf.

sh ip int

sh ip int
Interface           IP-Address      OK?  Method    Status             Protocol   VRF
Eth mgmt1           192.168.1.171 YES  manual    up                 up         default-vrf
Ve 10               192.168.1.173 YES  manual    up                 up         MGMT

show vrf MGMT

VRF MGMT, default RD 4:10, Table ID 1
Configured as management-vrf
IP Router-Id: 10.10.10.10
  Interfaces:
    ve10

  Address Family IPv4
    Max Routes: 200
    Number of Unicast Routes: 1

 

[kh78]

A seperate question:
For a management VRF, is it better practice to put an IP address on a loopback and forward that to the VRF, rather than a VE?

I ask because I have seen posts elsewhere, with people implying it's somehow better, but I fail to see how in any practical sense. Different if you are using it for keeping a routing protocol up or something I guess?

 

[ramicio]

...


STOP! Take a deep breath.

I reviewed your previous posts. I'd like to get some clarity first. I'm going to give you some suggestions in a bit so bear with me.

Your interface output looks "fine" even the small number of giants (hoping they're just 1522 and not something else - which could be coming form something else on your network). I'm going with the test on the 1/2/6 was simply brief as the uptime on it is short. - please confirm

based on the show int I'd say both 40gbe ports are fine.

for each interface test did you have good link / status "blinky lights" on your NIC?
You previously reported you had no electrical activity on the bottom port - is that still the case?



STOP again! Please don't wipe your 14 year maintained system. If it has been maintained that long I'd either consider it "prod" or of significant sentimental value. Put it back on your network the way it was and leave it alone until you've finished testing the switch.

You are experimenting and troubleshooting - use something else for this process.

Beg, borrow, steal another system. Throw an OS on it (clean install) and let's troubleshoot the rest of the ports on your switch. yeah?

seriously. Don't EFF around with the system you've been using. Use something else to finish testing out the switch.

Next, I saw a comment that made me think you want to rely on DHCP for testing. Bad idea right now.
80% of most network issues are physical layer.
If you rely on DHCP you may question whether DHCP is behaving or you have something else going on. Remove DHCP from the equation. Use static IP's for the remainder of the testing and on a CLEAN system.
that way you can focus on do I have a bad port, do i have a bad cable.
simplify your troubleshooting by simplifying the number of variables in any given test.

When I tell it to use a static IP is when I can actually form a connection with it, but it's then doing all of those RX errors. And using a static IP is also when it says that thing with the old IP address and DHCP when I go to ifdown the connection. I am going to force it to use the old "eth" names and do some more poking. Later I will stick the card in another server I just got and installed 22.04 onto. Honestly, the 14 year old install has been a pain when I've gone and upgraded it. It's pretty janky at this point, which is probably why I'm having so many issues. I will report back some stuff later. Thank you.

 

[itronin]

When I tell it to use a static IP is when I can actually form a connection with it, but it's then doing all of those RX errors. And using a static IP is also when it says that thing with the old IP address and DHCP when I go to ifdown the connection. I am going to force it to use the old "eth" names and do some more poking.

this. Makes me think you may have a conflict on your network with your static or some other thing unrelated to the switch (L2).

Later I will stick the card in another server I just got and installed 22.04 onto. Honestly, the 14 year old install has been a pain when I've gone and upgraded it. It's pretty janky at this point, which is probably why I'm having so many issues. I will report back some stuff later. Thank you.

yep.

please do report back as to whether the blinkies were working correctly on your NIC when connected to BOTH 40Gbe interfaces on the switch.

 

[ramicio]

this. Makes me think you may have a conflict on your network with your static or some other thing unrelated to the switch (L2).



yep.

please do report back as to whether the blinkies were working correctly on your NIC when connected to BOTH 40Gbe interfaces on the switch.

Changed the naming to eth*. No change. I don't understand why, when I use a static IP address (under /etc/network/interfaces) and I go to ifdown it, it mentions releasing the old IP address of 192.168.1.2. The static IP I'm assigning it is 192.168.1.17. When I do an ifconfig, I get something like the following:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.17  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::3efd:feff:feb2:cd30  prefixlen 64  scopeid 0x20<link>
        ether 3c:fd:fe:b2:cd:30  txqueuelen 1000  (Ethernet)
        RX packets 3392  bytes 504447 (504.4 KB)
        RX errors 3819  dropped 0  overruns 0  frame 3819
        TX packets 5184  bytes 637121 (637.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Going to throw the card in my other server now.