Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[abster]

Are the servers reporting a link-level connection?

I assume the switches have a static IP address configured? Is there a DHCP server running, or else, do the servers work if you manually configure their IP address?

The switches are on a fixed IP address.

The DHCP server is running on my firewalla device.

If I connect the servers using copper, they get an IP and connect fine.

For some reason the 10 Gb ports are not getting an IP!

 

[itronin]

The switches are on a fixed IP address.

The DHCP server is running on my firewalla device.

If I connect the servers using copper, they get an IP and connect fine.

For some reason the 10 Gb ports are not getting an IP!

its pretty helpful to provide your configs, without it everything is just SWAG'ing.

 

[dbvader]

Configs are useful, yes.

For some reason the 10 Gb ports are not getting an IP!

At the very least:

show interfaces ethernet 1/2/3

and the same for 1/2/4 could be revealing.

 

[abster]

Configs are useful, yes.



At the very least:

show interfaces ethernet 1/2/3

and the same for 1/2/4 could be revealing.

Here you go:

telnet@barnrouter>show interfaces ethernet 1/2/3
10GigabitEthernet1/2/3 is up, line protocol is up
Port up for 9 hour(s) 19 minute(s) 19 second(s)
Hardware is 10GigabitEthernet, address is d4c1.9e6a.5810 (bia d4c1.9e6a.5843)
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 24 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 27112 bits/sec, 20 packets/sec, 0.00% utilization
1128 packets input, 112203 bytes, 0 no buffer
Received 5 broadcasts, 1123 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
6762428 packets output, 6263556790 bytes, 0 underruns
Transmitted 1203401 broadcasts, 725967 multicasts, 4833060 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

telnet@barnrouter>show interfaces ethernet 1/2/4
10GigabitEthernet1/2/4 is up, line protocol is up
Port up for 8 hour(s) 54 minute(s) 39 second(s)
Hardware is 10GigabitEthernet, address is d4c1.9e6a.5810 (bia d4c1.9e6a.5844)
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 24 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 26496 bits/sec, 20 packets/sec, 0.00% utilization
810 packets input, 80190 bytes, 0 no buffer
Received 0 broadcasts, 810 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
6532160 packets output, 6160652507 bytes, 0 underruns
Transmitted 1082854 broadcasts, 664381 multicasts, 4784925 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

 

[abster]

Config:

SSH@barnrouter>show config
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-48p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
stack-port 1/2/1 1/2/3
stack enable
stack mac d4c1.9e6a.5810
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname barnrouter
ip dhcp-client disable
ip dns server-address 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
username root password .....
snmp-server community ..... ro
!
!
clock timezone gmt GMT-05
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
hitless-failover enable
!
!
!
interface ethernet 1/1/47
inline power
!
interface ethernet 1/1/48
inline power
!
interface ve 1
ip address 192.168.1.3 255.255.255.0
!
!
!
!
!
!
!
!
!
end

 

[dbvader]

Not sure if the MTU (10200) may be contributing to the issue. Would want to compare that to the MTU on the working copper connection. I would just set it to 1500, at least for diagnostic purposes.

Also, if the server is working on its copper connection, the SFP nic may be in a different firewall zone?

 

[dbvader]

And should 1/2/3 be a stack port if it's connected to one of the servers?

 

[itronin]

@abster
You have 3 switches or three switches configured in a stack?

from previous post:

1/2/1 Down None None None None No 1 0 d4c1.9e6a.5810
1/2/2 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810
1/2/3 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server1
1/2/4 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server2

from your config:

stack unit 1
 module 1 icx6450-48p-poe-port-management-module
 module 2 icx6450-sfp-plus-4port-40g-module
 stack-port 1/2/1 1/2/3
stack enable

so you have port 1/2/3 configured for stacking but you are using it to connect to server1?


concur with @dbvader - at a minimum for troubleshooting mtu should be 1500 - honestly there is no really good reason to run anything BUT a 1500 mtu... However if you insist going down the jumbo path - did you set your 10g interface on server 2 (connected to 1/2/4) to 10200? again - consider using the default mtu of 1500

 

[abster]

Corrected a few things (servers are connected to port 2 and 4 not 3 and 4):

1/2/1 Down None None None None No 1 0 d4c1.9e6a.5810
1/2/2 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810<------ Connected to server1
1/2/3 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810<------- connected to 2nd switch
1/2/4 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server2

Turned off stacking (I was playing with it).

Also, changed MTU to 1500

SSH@barnrouter#show interfaces ethernet 1/2/4
10GigabitEthernet1/2/4 is up, line protocol is up
Port up for 3 minute(s) 16 second(s)
Hardware is 10GigabitEthernet, address is d4c1.9e6a.5810 (bia d4c1.9e6a.5844)
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 32 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 53528 bits/sec, 33 packets/sec, 0.00% utilization
8 packets input, 792 bytes, 0 no buffer
Received 0 broadcasts, 8 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
6602 packets output, 1308368 bytes, 0 underruns
Transmitted 2890 broadcasts, 3544 multicasts, 168 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

 

[dbvader]

Still not working? If so, I would move to the connected server to see what it's reporting. I still can imagine its SFP nic is in some kind of zone that's firewalling its connection to the 6450. What kind of server (linux, windows, vms, etc.)?

 

[abster]

2 Servers:

1. Dell R730XD with a BroadCom 10GbE 2P 57810S Adapter running Proxmox 8.1.4
2. Dell R430 with a BroadCom 10GbE 2P 57810S Adapter running Unraid 6.12.6

The copper connection on the R730XD works fine and at full 1Gb speed.

All the SFP transceivers are 10Gtek AXS85-192-M3

They work fine switch to switch!

 

[dbvader]

I really have no experience with proxmox or unraid.

I'm guessing you always had copper working and then added a nic with an SFP+ port? I would check whether there's a firewall running on the servers which is blocking traffic, especially if the nic with SFP was added later on. Seems like the 6450 is effectively not receiving any traffic, and the port config seems to be as plain vanilla as it should be.

To rule out any dhcp propagation idiosyncrasies you could also consider manually configuring the server's ip address, just for testing purposes.

 

[itronin]

2 Servers:

1. Dell R730XD with a BroadCom 10GbE 2P 57810S Adapter running Proxmox 8.1.4
2. Dell R430 with a BroadCom 10GbE 2P 57810S Adapter running Unraid 6.12.6

The copper connection on the R730XD works fine and at full 1Gb speed.

All the SFP transceivers are 10Gtek AXS85-192-M3

They work fine switch to switch!

Given the issues I'd suggest reducing the complexity of your testing and going component by component.

from your post it sounds like you tested 1gbe into your server nics and used some of the 1/1 ports and that worked but I'd go ahead and make sure 10gb is working.

test the servers 10gbe back to back which will require assigning a different subnet than you have on your LAN and static IP'ing the servers. If you don't want to mess with your unraid and prox configurations you should be able to boot a live ubuntu or something like that for the testing.

Use the same network cable that you were running between the NIC 10gb-t interface and the copper SFP+ to run back to back. ping test and if you can move some data back and forth.

Not sure which DELL broadcom those are but I have had challenges in the past with some of them if they are set up for virtual interfaces inside the NIC. fixing that required going into the iDRAC then into the card to nuke the configuration. quick google makes me think those are NOT the virtual NIC broadcom's...

I'm not an expert with prox but if the 10gb interface came up and was configured for VM's then I would not expect the prox host to necessarily be trying to pull DHCP.

On Unraid, should just be a "tick box" to configure DHCP on that interface.

I'm going to make an observation. running a pair of 10gb-t SFP+ adapters is going to make that SFP cage area VERY VERY WARM if not VERY HOT. If you have the funds and the distance works - really suggest DAC and changing out your NIC's. If you have the funds but distance is an issue then change your NICs and 10Gb MM optics and fiber patch.

 

[jei]

Yes been through the documentation, and does not explicitly say recommended. And in case it is recommended over 09 I would like to know why.
Anyways I'll try my luck on 09, have room for some trial, error and tinkering.

It is recommended by Ruckus. 09 is not considered stable.

Ruckus ICX software currently has two recommended release types; Stability and Technology.

Stability Release: This is for customers where stability is of utmost importance. This release may not contain every feature available for your product.

Technology Release: This is for customers looking to utilize the maximum feature set available for your product.

We recommend most customers utilize the Stability Release if it contains all needed features for your network. A Technology Release is recommended if your network requires newer features not available in the Stability Release.

 

[Sealside]

I do like the webgui in 09. Even though you configure and use the cli, it is nice to have a webui to get an overview.



Another question is, my firmware update failed for the boot

SH@beef1>show version
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on Jan 12 2024 at 09:59:12 labeled as SPR09010j
      (33554432 bytes) from Primary SPR09010j.bin (UFI)
        SW: Version 09.0.10jT213
      Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
       Compiled on Tue Nov 29 15:43:15 2022

spz10126 is probably not the same what's included in the SPR09010-release. I fail when I try to update it using manifest, any ideas how I can update it?

Best Regards, S

 

[Sealside]

Trying to get the 7250 to pass the WAF, I had to do something with the fans.
I went with the same approach as https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-331607

3xMechatronic MR4020C12B1-RSR and 1x M5210E12C-RSR
I reused extension cables from my previous Noctua fans to make connectors for the fans.
I used solder seal wire connectors to attach the connectors with a heat gun.

The noise level is more than my Juniper Ex3300 (Which has 3xNoctuas), but it is ok and a lot better than Stock fans.
Noctua fan are delivered with noise adapters, probably could use those on stock, but I have not tested.

 

[Vesalius]

Another question is, my firmware update failed for the boot

SH@beef1>show version
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on Jan 12 2024 at 09:59:12 labeled as SPR09010j
      (33554432 bytes) from Primary SPR09010j.bin (UFI)
        SW: Version 09.0.10jT213
      Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
       Compiled on Tue Nov 29 15:43:15 2022

spz10126 is probably not the same what's included in the SPR09010-release. I fail when I try to update it using manifest, any ideas how I can update it?

Best Regards, S

I have a similar Boot Code version on an icx-7150-zp using SPR09010j.bin (UFI), Version:10.1.26T225 (mnz10126), curious why you think this incorrect? BTW Ruckus released a minor update RUCKUS ICX FastIron 09.0.10j_cd1 (GA) Software Release (.zip) you can try.

 

[Jason Antes]

For those wondering about 2.5Gb connectivity, I recently bought 2 FS.com Intel coded 10GbaseT transceivers to use with my 6610. 1 works and the other does not (due to the branding encoding being botched). At any rate, the one that does work connects in as 10Gb and allows my desktop to connect at 2.5Gb. SFP-10G-T. I've been doing large amounts of file transfers without issues (consistent copies across network to my NAS server to the tune of 8-10 hours straight) and it seems to be stable.

 

[abster]

I was able to get both servers connected by assigning fixed IP's to both. Once I was able to get the link going on Unraid, I added to network connect and it's working fine at 10GB speeds. In Promox, assigning a fixed IP allowed the link to work but I am not getting DNS name resolution.

I'll look in DAC cables instead of the SFP transceivers.

Thanks!

Given the issues I'd suggest reducing the complexity of your testing and going component by component.

from your post it sounds like you tested 1gbe into your server nics and used some of the 1/1 ports and that worked but I'd go ahead and make sure 10gb is working.

test the servers 10gbe back to back which will require assigning a different subnet than you have on your LAN and static IP'ing the servers. If you don't want to mess with your unraid and prox configurations you should be able to boot a live ubuntu or something like that for the testing.

Use the same network cable that you were running between the NIC 10gb-t interface and the copper SFP+ to run back to back. ping test and if you can move some data back and forth.

Not sure which DELL broadcom those are but I have had challenges in the past with some of them if they are set up for virtual interfaces inside the NIC. fixing that required going into the iDRAC then into the card to nuke the configuration. quick google makes me think those are NOT the virtual NIC broadcom's...

I'm not an expert with prox but if the 10gb interface came up and was configured for VM's then I would not expect the prox host to necessarily be trying to pull DHCP.

On Unraid, should just be a "tick box" to configure DHCP on that interface.

I'm going to make an observation. running a pair of 10gb-t SFP+ adapters is going to make that SFP cage area VERY VERY WARM if not VERY HOT. If you have the funds and the distance works - really suggest DAC and changing out your NIC's. If you have the funds but distance is an issue then change your NICs and 10Gb MM optics and fiber patch.

 

[kpfleming]

I was able to get both servers connected by assigning fixed IP's to both. Once I was able to get the link going on Unraid, I added to network connect and it's working fine at 10GB speeds. In Promox, assigning a fixed IP allowed the link to work but I am not getting DNS name resolution.

I'll look in DAC cables instead of the SFP transceivers.

Thanks!

If you have connectivity using statically assigned addresses then the transceivers/cables are not the problem. It sounds like those machines are on a VLAN which does not have access to (or does not include) your DHCP/DNS/etc. server.