Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[daleq]

@rheaalleen

For the Delta FFB0412HHN-F00, I used the same color coding you show.
Green -> Blue
Yellow -> Red
Black -> Black

I don't know if this also applies to your EFB0412VHD-F00, but is seems possible.

 

[Sealside]

What is the recommended firmware version?
Running 08.0.95 currently, but tempted to install 09.0.10a before i start to set up everything. That way i will be able to test much more on 09, rather than configure everything in 08 and then later install 09.

Any known issues with 09?

7250 48P

 

[Gerhen]

What is the recommended firmware version?
Running 08.0.95 currently, but tempted to install 09.0.10a before i start to set up everything. That way i will be able to test much more on 09, rather than configure everything in 08 and then later install 09.

Any known issues with 09?

7250 48P

Got the same model and running the latest 09 in router mode. Have not experienced any bugs or any behavior that made me wish I was on 08. The only thing I’d say is some of the command syntax is different on 09 so you’ll have to look up those differences if you follow guides written with 08 in mind. Things like dual-mode and specifying the interface ve within the vlan aren’t specified in 09 but instead implied.

 

[jei]

What is the recommended firmware version?

You can find the recommended versions from post #1

Brocade Overview - Fohdeesha Docs

fohdeesha.com

 

[Sealside]

You can find the recommended versions from post #1

Yes been through the documentation, and does not explicitly say recommended. And in case it is recommended over 09 I would like to know why.
Anyways I'll try my luck on 09, have room for some trial, error and tinkering.

 

[abster]

Hi,

I just added 3 ICX 6450 switches to my network using SFP+ fiber cables. All three are talking to each other at 10GB and working great overall.

Now the problem: To take advantage of the 10GB network, I also upgraded the network cards in my servers to SFP+ 10gb. For some reason the 2 servers will not connect to the 6450 using the fiber connection. The connection lights are on and blinking but no IP address is being pulled by them.

Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/1 Down None None None None No 1 0 d4c1.9e6a.5810
1/2/2 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810
1/2/3 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server1
1/2/4 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server2

Any idea what I am missing?

 

[dbvader]

For some reason the 2 servers will not connect to the 6450 using the fiber connection.

Are the servers reporting a link-level connection?

I assume the switches have a static IP address configured? Is there a DHCP server running, or else, do the servers work if you manually configure their IP address?

 

[abster]

Are the servers reporting a link-level connection?

I assume the switches have a static IP address configured? Is there a DHCP server running, or else, do the servers work if you manually configure their IP address?

The switches are on a fixed IP address.

The DHCP server is running on my firewalla device.

If I connect the servers using copper, they get an IP and connect fine.

For some reason the 10 Gb ports are not getting an IP!

 

[itronin]

The switches are on a fixed IP address.

The DHCP server is running on my firewalla device.

If I connect the servers using copper, they get an IP and connect fine.

For some reason the 10 Gb ports are not getting an IP!

its pretty helpful to provide your configs, without it everything is just SWAG'ing.

 

[dbvader]

Configs are useful, yes.

For some reason the 10 Gb ports are not getting an IP!

At the very least:

show interfaces ethernet 1/2/3

and the same for 1/2/4 could be revealing.

 

[abster]

Configs are useful, yes.



At the very least:

show interfaces ethernet 1/2/3

and the same for 1/2/4 could be revealing.

Here you go:

telnet@barnrouter>show interfaces ethernet 1/2/3
10GigabitEthernet1/2/3 is up, line protocol is up
Port up for 9 hour(s) 19 minute(s) 19 second(s)
Hardware is 10GigabitEthernet, address is d4c1.9e6a.5810 (bia d4c1.9e6a.5843)
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 24 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 27112 bits/sec, 20 packets/sec, 0.00% utilization
1128 packets input, 112203 bytes, 0 no buffer
Received 5 broadcasts, 1123 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
6762428 packets output, 6263556790 bytes, 0 underruns
Transmitted 1203401 broadcasts, 725967 multicasts, 4833060 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

telnet@barnrouter>show interfaces ethernet 1/2/4
10GigabitEthernet1/2/4 is up, line protocol is up
Port up for 8 hour(s) 54 minute(s) 39 second(s)
Hardware is 10GigabitEthernet, address is d4c1.9e6a.5810 (bia d4c1.9e6a.5844)
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 24 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 26496 bits/sec, 20 packets/sec, 0.00% utilization
810 packets input, 80190 bytes, 0 no buffer
Received 0 broadcasts, 810 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
6532160 packets output, 6160652507 bytes, 0 underruns
Transmitted 1082854 broadcasts, 664381 multicasts, 4784925 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

 

[abster]

Config:

SSH@barnrouter>show config
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-48p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
stack-port 1/2/1 1/2/3
stack enable
stack mac d4c1.9e6a.5810
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
hostname barnrouter
ip dhcp-client disable
ip dns server-address 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
username root password .....
snmp-server community ..... ro
!
!
clock timezone gmt GMT-05
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
!
hitless-failover enable
!
!
!
interface ethernet 1/1/47
inline power
!
interface ethernet 1/1/48
inline power
!
interface ve 1
ip address 192.168.1.3 255.255.255.0
!
!
!
!
!
!
!
!
!
end

 

[dbvader]

Not sure if the MTU (10200) may be contributing to the issue. Would want to compare that to the MTU on the working copper connection. I would just set it to 1500, at least for diagnostic purposes.

Also, if the server is working on its copper connection, the SFP nic may be in a different firewall zone?

 

[dbvader]

And should 1/2/3 be a stack port if it's connected to one of the servers?

 

[itronin]

@abster
You have 3 switches or three switches configured in a stack?

from previous post:

1/2/1 Down None None None None No 1 0 d4c1.9e6a.5810
1/2/2 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810
1/2/3 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server1
1/2/4 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server2

from your config:

stack unit 1
 module 1 icx6450-48p-poe-port-management-module
 module 2 icx6450-sfp-plus-4port-40g-module
 stack-port 1/2/1 1/2/3
stack enable

so you have port 1/2/3 configured for stacking but you are using it to connect to server1?


concur with @dbvader - at a minimum for troubleshooting mtu should be 1500 - honestly there is no really good reason to run anything BUT a 1500 mtu... However if you insist going down the jumbo path - did you set your 10g interface on server 2 (connected to 1/2/4) to 10200? again - consider using the default mtu of 1500

 

[abster]

Corrected a few things (servers are connected to port 2 and 4 not 3 and 4):

1/2/1 Down None None None None No 1 0 d4c1.9e6a.5810
1/2/2 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810<------ Connected to server1
1/2/3 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810<------- connected to 2nd switch
1/2/4 Up Forward Full 10G None No 1 0 d4c1.9e6a.5810 <------ Connected to server2

Turned off stacking (I was playing with it).

Also, changed MTU to 1500

SSH@barnrouter#show interfaces ethernet 1/2/4
10GigabitEthernet1/2/4 is up, line protocol is up
Port up for 3 minute(s) 16 second(s)
Hardware is 10GigabitEthernet, address is d4c1.9e6a.5810 (bia d4c1.9e6a.5844)
Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
Member of L2 VLAN ID 1, port is untagged, port state is FORWARDING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 32 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 53528 bits/sec, 33 packets/sec, 0.00% utilization
8 packets input, 792 bytes, 0 no buffer
Received 0 broadcasts, 8 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
6602 packets output, 1308368 bytes, 0 underruns
Transmitted 2890 broadcasts, 3544 multicasts, 168 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0

 

[dbvader]

Still not working? If so, I would move to the connected server to see what it's reporting. I still can imagine its SFP nic is in some kind of zone that's firewalling its connection to the 6450. What kind of server (linux, windows, vms, etc.)?

 

[abster]

2 Servers:

1. Dell R730XD with a BroadCom 10GbE 2P 57810S Adapter running Proxmox 8.1.4
2. Dell R430 with a BroadCom 10GbE 2P 57810S Adapter running Unraid 6.12.6

The copper connection on the R730XD works fine and at full 1Gb speed.

All the SFP transceivers are 10Gtek AXS85-192-M3

They work fine switch to switch!

 

[dbvader]

I really have no experience with proxmox or unraid.

I'm guessing you always had copper working and then added a nic with an SFP+ port? I would check whether there's a firewall running on the servers which is blocking traffic, especially if the nic with SFP was added later on. Seems like the 6450 is effectively not receiving any traffic, and the port config seems to be as plain vanilla as it should be.

To rule out any dhcp propagation idiosyncrasies you could also consider manually configuring the server's ip address, just for testing purposes.

 

[itronin]

2 Servers:

1. Dell R730XD with a BroadCom 10GbE 2P 57810S Adapter running Proxmox 8.1.4
2. Dell R430 with a BroadCom 10GbE 2P 57810S Adapter running Unraid 6.12.6

The copper connection on the R730XD works fine and at full 1Gb speed.

All the SFP transceivers are 10Gtek AXS85-192-M3

They work fine switch to switch!

Given the issues I'd suggest reducing the complexity of your testing and going component by component.

from your post it sounds like you tested 1gbe into your server nics and used some of the 1/1 ports and that worked but I'd go ahead and make sure 10gb is working.

test the servers 10gbe back to back which will require assigning a different subnet than you have on your LAN and static IP'ing the servers. If you don't want to mess with your unraid and prox configurations you should be able to boot a live ubuntu or something like that for the testing.

Use the same network cable that you were running between the NIC 10gb-t interface and the copper SFP+ to run back to back. ping test and if you can move some data back and forth.

Not sure which DELL broadcom those are but I have had challenges in the past with some of them if they are set up for virtual interfaces inside the NIC. fixing that required going into the iDRAC then into the card to nuke the configuration. quick google makes me think those are NOT the virtual NIC broadcom's...

I'm not an expert with prox but if the 10gb interface came up and was configured for VM's then I would not expect the prox host to necessarily be trying to pull DHCP.

On Unraid, should just be a "tick box" to configure DHCP on that interface.

I'm going to make an observation. running a pair of 10gb-t SFP+ adapters is going to make that SFP cage area VERY VERY WARM if not VERY HOT. If you have the funds and the distance works - really suggest DAC and changing out your NIC's. If you have the funds but distance is an issue then change your NICs and 10Gb MM optics and fiber patch.