Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Wolfcastle]

Do someone have audio/video comparing noise from 6610/7450/6450/7250 24P models ? I'm trying to decide which switch will provide best price/performance/features ratio and for now I'm steering to 6610 but can figure out how noisier will be compared to the others...

6610 and 7450 24P both are the same fans/power supplies and quite loud. Power supply revision makes no difference to my ears. Can’t speak to the other models but anything that has Delta 40mm 23k RPM fans will be loud.

 

[paxswill]

So, I recently switched from my Brocade switch to a Cisco one. I was having problems mostly with multicast, mDNS/Bonjour stuff specifically. I would have Google Homes or Chromecasts not showing up in lists to cast to, light switches which wouldn't turn on and off, Airprint printers which would show up sometimes and then not others, and other weird behavior I otherwise couldn't explain. It was not all the time, and sometimes it worked fine.

Initially, I thought this was a result of how Brocade had implemented PIM, as I had phones and PCs in a separate L2 segment. For science, I merged all of my devices into the same vlan and the problem persisted. I tried turning PIM off and just used IGMP. I tried using Ubiquiti, Aruba and finally Cisco APs as I thought maybe one brand or another's WiFi would solve the problem with their various multicast features. The same weird behavior still existed no matter the vendor. Aruba and Cisco worked better and was more stable than Ubiquiti, but that is probably a result of tunneling/CAPWAPing the traffic to a controller, so the MAC addresses of all of the various devices were literally on the same port on the switch, However, if I ran them in Flexconnect or Bridging mode, the multicast behavior was just as unreliable as the Ubiquiti deployment. I even thought my Android phone's IP stack was just hokey (I was having other problems, too. Bluetooth and Wifi calling never worked reliably, as an example), so I switched to an iPhone. Problems persisted.

Were you using 09.0.00 or later for this? There's a poorly documented change (and possibly a bug) in multicast behavior starting with 09.0.00. From what I can tell, previous versions would by default flood unregistered multicast packets, while 09.0.00 and 09.0.10 default to the opposite. The only documentation I found for this was in the list of new and deprecated commands, where ip multicast disable-flooding has been deprecated, and there's a new ip multicast flood-unregistered command (with corresponding changes for the ipv6 commands as well).

Running the new command fixed a bunch of weird multicast issues I was having that I'm pretty sure boiled down to the new default being the opposite of the old default (which is also the default for other devices I've used so far).

 

[kpfleming]

ip multicast flood-unregistered command (with corresponding changes for the ipv6 commands as well)

Could this affect general IPv6 operation, since it uses subnet-local multicast for Neighbor Discovery and other mechanisms?

 

[Roelf Zomerman]

that would be NAT, which L3 switches don't do. also the 6450 does not support bgp

ah okay... yeah.. hmmm.. any idea on how to achieve this? perhaps pfsense or something?

 

[brob]

Hi

I am having issues with my ICX6450, I have 5 of these in a stack, and I have STP and dynamic arp inspection enabled. I have been getting constant ARP inspection failures and STP Blocking coming up. I have this happening on multiple ports, anywhere from every 10 mins to every hour on the same port. this is not happening to every port, just seems like certain ones. the ports this is happening to are on different VLAN's as well, mostly VLAN 2 and some VLAN 6. but most of the computers are on VLAN 2 and they are connected to just single computers.


2022-02-09 09:17:45 User.Info 192.168.1.1 Feb 9 09:17:45 STP: VLAN 2 Port 2/1/31 STP State -> FORWARDING (PortDown)
2022-02-09 09:17:45 User.Info 192.168.1.1 Feb 9 09:17:45 STP: VLAN 2 Port 2/1/31 STP State -> BLOCKING (DOT1wTransition)
2022-02-09 09:17:45 User.Info 192.168.1.1 Feb 9 09:17:45 STP: VLAN 2 Port 2/1/31 STP State -> DISABLED (PortDown)
2022-02-09 09:17:45 User.Info 192.168.1.1 Feb 9 09:17:45 System: Interface ethernet 2/1/31, state down
2022-02-09 09:17:47 User.Info 192.168.1.1 Feb 9 09:17:47 STP: VLAN 2 Port 2/1/31 STP State -> BLOCKING (DOT1wTransition)
2022-02-09 09:17:47 User.Info 192.168.1.1 Feb 9 09:17:47 System: Interface ethernet 2/1/31, state up
2022-02-09 09:17:52 User.Info 192.168.1.1 Feb 9 09:17:52 STP: VLAN 2 Port 2/1/31 STP State -> LEARNING (DOT1wTransition)
2022-02-09 09:17:52 User.Info 192.168.1.1 Feb 9 09:17:52 STP: VLAN 2 Port 2/1/31 STP State -> FORWARDING (DOT1wTransition)
2022-02-09 09:18:18 User.Info 192.168.1.1 Feb 9 09:18:18 STP: VLAN 2 Port 2/1/31 STP State -> FORWARDING (PortDown)
2022-02-09 09:18:18 User.Info 192.168.1.1 Feb 9 09:18:18 STP: VLAN 2 Port 2/1/31 STP State -> BLOCKING (DOT1wTransition)
2022-02-09 09:18:18 User.Info 192.168.1.1 Feb 9 09:18:18 STP: VLAN 2 Port 2/1/31 STP State -> DISABLED (PortDown)
2022-02-09 09:18:18 User.Info 192.168.1.1 Feb 9 09:18:18 System: Interface ethernet 2/1/31, state down
2022-02-09 09:18:20 User.Info 192.168.1.1 Feb 9 09:18:20 STP: VLAN 2 Port 2/1/31 STP State -> BLOCKING (DOT1wTransition)
2022-02-09 09:18:20 User.Info 192.168.1.1 Feb 9 09:18:20 System: Interface ethernet 2/1/31, state up
2022-02-09 09:18:25 User.Info 192.168.1.1 Feb 9 09:18:25 STP: VLAN 2 Port 2/1/31 STP State -> LEARNING (DOT1wTransition)
2022-02-09 09:18:25 User.Info 192.168.1.1 Feb 9 09:18:25 STP: VLAN 2 Port 2/1/31 STP State -> FORWARDING (DOT1wTransition)

I also have ARP inspection failures as below. it picks up on the DHCP and maps it, but still shows inspection failure. and they have the same mac address

2022-02-09 08:21:46 User.Info 192.168.1.1 Feb 9 08:21:47 DHCP: snooping on trusted port 2/2/2, type 5, VRF 0, 192.168.1.36->e04f.43e8.****
2022-02-09 08:21:46 User.Info 192.168.1.1 Feb 9 08:21:47 DHCP: snooping mapped to outgoing port 3/1/7 Interface v2 vlan 2 VRF 0
2022-02-09 08:21:47 User.Info 192.168.1.1 Feb 9 08:21:47 ARP: inspection failure, invalid src ip 0.0.0.0, (L2 src mac e04f.43e8.****, port 3/1/7)
2022-02-09 08:21:48 User.Info 192.168.1.1 Feb 9 08:21:48 ARP: inspection failure, invalid src ip 0.0.0.0, (L2 src mac e04f.43e8.****, port 3/1/7)
2022-02-09 08:21:49 User.Info 192.168.1.1 Feb 9 08:21:49 ARP: inspection failure, invalid src ip 0.0.0.0, (L2 src mac e04f.43e8.****, port 3/1/7)

Any help on what would be causing this would be appreciated

Thanks

ver 08.0.30uT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  priority 128
  stack-port 1/2/1 1/2/3
stack unit 2
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  priority 128
  stack-port 2/2/1 2/2/3
stack unit 3
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  priority 62
  stack-port 3/2/1 3/2/3
stack unit 4
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  stack-port 4/2/1 4/2/3
stack unit 5
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
  stack-port 5/2/1 5/2/3
stack enable
stack mac ****.****.****
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 2 name RM by port
tagged ethe 1/1/15 ethe 1/1/30 ethe 1/1/32 ethe 1/1/45 ethe 1/2/2 ethe 2/1/38 ethe 2/2/2 ethe 3/1/26 ethe 3/1/34 ethe 5/1/5 to 5/1/6
untagged ethe 1/1/1 to 1/1/4 ethe 1/1/6 to 1/1/14 ethe 1/1/16 to 1/1/29 ethe 1/1/31 ethe 1/1/33 ethe 1/1/35 ethe 1/1/37 ethe 1/1/40 ethe 1/1/42 ethe 2/1/1 to 2/1/7 ethe 2/1/9 ethe 2/1/11 to 2/1/16 ethe 2/1/18 ethe 2/1/21 ethe 2/1/23 ethe 2/1/25 to 2/1/26 ethe 2/1/28 to 2/1/32 ethe 2/1/34 to 2/1/37 ethe 2/1/39 to 2/1/46 ethe 3/1/1 ethe 3/1/3 to 3/1/7 ethe 3/1/9 to 3/1/17 ethe 3/1/19 ethe 3/1/23 ethe 3/1/25 ethe 3/1/28 to 3/1/32 ethe 3/1/35 to 3/1/36 ethe 3/1/38 ethe 3/1/40 to 3/1/44 ethe 3/1/46 to 3/1/47 ethe 4/1/1 to 4/1/47 ethe 5/1/2 to 5/1/4 ethe 5/1/9 to 5/1/10 ethe 5/1/13 ethe 5/1/16 to 5/1/18 ethe 5/1/22 to 5/1/23 ethe 5/1/26 ethe 5/1/28 ethe 5/1/30 ethe 5/1/32 ethe 5/1/34 ethe 5/1/36 to 5/1/48
router-interface ve 2
spanning-tree 802-1w
spanning-tree 802-1w priority 1
!
vlan 3 name WAN by port
tagged ethe 1/1/48
router-interface ve 3
!
vlan 4 name E2 by port
tagged ethe 1/1/30
untagged ethe 1/1/34 ethe 2/1/10 ethe 2/1/19 ethe 3/1/8 ethe 3/1/21 ethe 5/1/14 to 5/1/15
router-interface ve 4
!
vlan 5 name secure by port
tagged ethe 1/1/30
untagged ethe 2/1/8
router-interface ve 5
!
vlan 6 name CR by port
tagged ethe 1/1/30 ethe 1/2/2 ethe 2/2/2
untagged ethe 1/1/5 ethe 1/1/36 ethe 1/1/38 to 1/1/39 ethe 2/1/17 ethe 2/1/27 ethe 2/1/33 ethe 2/1/47 to 2/1/48 ethe 3/1/33 ethe 3/1/37 ethe 3/1/39 ethe 3/1/45 ethe 5/1/11 to 5/1/12 ethe 5/1/21
router-interface ve 6
spanning-tree
!
vlan 15 name management by port
tagged ethe 1/1/15 ethe 1/1/30 ethe 1/1/32 ethe 1/1/45 ethe 1/2/2 ethe 2/1/38 ethe 2/2/2 ethe 3/1/26 ethe 3/1/34 ethe 5/1/5 to 5/1/6
untagged ethe 1/1/41 ethe 1/1/43 to 1/1/44 ethe 2/1/20 ethe 2/1/22 ethe 2/1/24 ethe 3/1/18 ethe 3/1/20 ethe 3/1/22 ethe 3/1/24 ethe 3/1/48 ethe 4/1/48 ethe 5/1/7 to 5/1/8 ethe 5/1/19 to 5/1/20
router-interface ve 15
!
vlan 20 name Brents by port
tagged ethe 1/1/15 ethe 1/1/30 ethe 1/1/32 ethe 1/1/45 ethe 2/1/38 ethe 3/1/26 ethe 3/1/34 ethe 5/1/5 to 5/1/6
untagged ethe 1/1/47
spanning-tree 802-1w
!
vlan 50 name guest by port
tagged ethe 1/1/15 ethe 1/1/30 ethe 1/1/32 ethe 1/1/45 ethe 1/1/48 ethe 2/1/38 ethe 3/1/26 ethe 3/1/34 ethe 5/1/5 to 5/1/6
spanning-tree 802-1w
spanning-tree 802-1w priority 1
!
vlan 100 name voice by port
tagged ethe 1/1/30 ethe 1/1/46 ethe 1/2/2 ethe 2/2/2
untagged ethe 3/1/2 ethe 3/1/27 ethe 5/1/1 ethe 5/1/24 to 5/1/25 ethe 5/1/27 ethe 5/1/29 ethe 5/1/31 ethe 5/1/33 ethe 5/1/35
!
!
!
!
!
optical-monitor
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
enable acl-per-port-per-vlan
ip arp inspection vlan 2
ip arp inspection vlan 5
ip arp inspection vlan 15
ip dhcp snooping vlan 2
ip dhcp snooping vlan 4
ip dhcp snooping vlan 5
ip dhcp snooping vlan 6
ip dhcp snooping vlan 15
ip dhcp-client disable
ip dhcp-server enable
ip dns server-address 192.168.1.110 192.168.3.1 8.8.8.8
ip route 0.0.0.0/0 192.168.3.1
!
logging host 192.168.1.5
no telnet server
username root password .....

!
!
clock timezone gmt GMT-06
!
!
ntp
server 192.168.1.110
!
!
web-management frame bottom
web-management page-menu
hitless-failover enable

interface ve 2
ip access-group RM in
ip address 192.168.1.1 255.255.255.0
!
interface ve 3
ip address 192.168.3.2 255.255.255.0
!
interface ve 4
ip access-group E2 in
ip address 192.168.4.1 255.255.255.0
ip helper-address 1 192.168.1.110
!
interface ve 5
ip access-group secure in
ip address 192.168.5.1 255.255.255.0
ip helper-address 1 192.168.1.110
!
interface ve 6
ip access-group CR in
ip address 172.18.1.254 255.255.255.0
ip helper-address 1 192.168.1.110
!
interface ve 15
ip address 192.168.15.1 255.255.255.0
ip helper-address 1 192.168.1.110
source-guard enable e 1/1/41
source-guard enable e 2/1/20
source-guard enable e 2/1/22
source-guard enable e 2/1/24
source-guard enable e 3/1/18
source-guard enable e 3/1/20
source-guard enable e 3/1/22
source-guard enable e 3/1/24
source-guard enable e 4/1/48
source-guard enable e 5/1/7
source-guard enable e 5/1/8
!


!

 

[bpye]

Does anyone know why the power draw in the Brocade literature is noticably less than that listed in the first post? https://webresources.ruckuswireless.com/pdf/datasheets/ds-icx-6430-6450.pdf

They claim 21W idle for the 6450-24P and 31W for the 6450-48P vs the 25W and 50W numbers listed here?

 

[LodeRunner]

Does anyone know why the power draw in the Brocade literature is noticably less than that listed in the first post? https://webresources.ruckuswireless.com/pdf/datasheets/ds-icx-6430-6450.pdf

They claim 21W idle for the 6450-24P and 31W for the 6450-48P vs the 25W and 50W numbers listed here?

It could be that’s the internal power draw and is not the at-receptical draw which would include PSU efficiency and power factor (if the PSUs are not PFC type).

 

[NablaSquaredG]

I suppose that the IDLE numbers are really IDLE - No network traffic on the non-management ports at all, so more like a marketing number...

If you look at the second column, "5% Throughput", you see significantly higher numbers compared to the "IDLE" column which are quite close to the numbers in this thread (29W for -24, 51W for -48)

 

[blademan]

Do someone have audio/video comparing noise from 6610/7450/6450/7250 24P models ? I'm trying to decide which switch will provide best price/performance/features ratio and for now I'm steering to 6610 but can figure out how noisier will be compared to the others...

6610 and 7450 24P both are the same fans/power supplies and quite loud. Power supply revision makes no difference to my ears. Can’t speak to the other models but anything that has Delta 40mm 23k RPM fans will be loud.

Based on sensor from my wearable:
7250 at boot: 75 dB
7250 running: 55 dB
Old Cisco Catalyst 3750 I've had running for years in my wiring closet: 42 dB front, 45 dB back

Anyone tried these low speed/noise fans from this seller? 1x Quiet Replacement Fan (18dBA) for Brocade ICX 6430 ICX 6450 ICX 7250

 

[jei]

So, I recently switched from my Brocade switch to a Cisco one. I was having problems mostly with multicast, mDNS/Bonjour stuff specifically. I would have Google Homes or Chromecasts not showing up in lists to cast to, light switches which wouldn't turn on and off, Airprint printers which would show up sometimes and then not others, and other weird behavior I otherwise couldn't explain. It was not all the time, and sometimes it worked fine.

Initially, I thought this was a result of how Brocade had implemented PIM, as I had phones and PCs in a separate L2 segment. For science, I merged all of my devices into the same vlan and the problem persisted. I tried turning PIM off and just used IGMP. I tried using Ubiquiti, Aruba and finally Cisco APs as I thought maybe one brand or another's WiFi would solve the problem with their various multicast features. The same weird behavior still existed no matter the vendor. Aruba and Cisco worked better and was more stable than Ubiquiti, but that is probably a result of tunneling/CAPWAPing the traffic to a controller, so the MAC addresses of all of the various devices were literally on the same port on the switch, However, if I ran them in Flexconnect or Bridging mode, the multicast behavior was just as unreliable as the Ubiquiti deployment. I even thought my Android phone's IP stack was just hokey (I was having other problems, too. Bluetooth and Wifi calling never worked reliably, as an example), so I switched to an iPhone. Problems persisted.

Finally, I kicked the Brocade 7450 switch (which had replaced an HP E3800 switch because it had better 10GbE density) for a Cisco 3850 switch. I turned PIM on the Cisco and enabled the mdns-sd gateway. The problem is more-or-less solved now. Things populate properly for casting all of the time, the AirPrint printers always show up, I can be in a L3 hop away and things still work, and I can turn my wifi lights on and off reliably. The only problem I have is that Apple Home occasionally does not like when I roam between APs and thinks things are disconnected on my iPhone, but then my iPad is working fine and It is connected to the same AP. I may just work towards CAPWAPing my traffic again and calling this a success.

In addition, the local network is performing measurably better. Take a look at this graph, as an example. This is a SmokePing chart monitoring my core switch. Can you see where I transitioned from the Brocade to the Cisco?
View attachment 21491

And just so you don't think something is wrong with my SmokePing instance, PRTG tells a similar story, and is running on different metal:
View attachment 21492

I've kept my ICX 7150 in production as an access switch (L2 only) in my Garage and it seems to do that job fairly well. I have an AP running on it and it seems to be behaving fine for that purpose. It's latency graphs looks better than the 7450 I had in production with L3 services (OSPF, PIM, etc) running.
View attachment 21493
But the latency is so much better on my Cisco switch that I even took my VDX out of production. This is an enterprise access/aggregation switch performing 40-50% better than a Datacenter switch in terms of latency, if Ping times are to be believed. The VDX always did this weird ping spike thing every day or so that I could never explain...pinging THROUGH it didn't correlate to an increase in latency so I never really gave it much mind but it was an annoying anomaly nonetheless.
View attachment 21495


As much as I like the value of what Brocade/Ruckus has to offer, clearly it can be worth paying more money to get better stability and performance...

Figured I would share, maybe my experience will help someone make a purchasing decision later.

If anyone wants a Brocade ICX 7450, 6450 or a VDX 6740 (SFP+, not copper, lower power consumption version) I have one of each sitting in a pile right now I would be willing to get rid of for a reasonable price (shipping costs are the issue really).

ICMP pings to the switch itself are answered by the management CPU, which prioritizes ICMP replies very low compared to other activities, so some fluctuation is normal. That combined with the fact LibreNMS runs the ping during the same time as the scheduled SNMP polling, which puts a decent amount of load on the CPU to answer. If you just open up a command prompt and ping the switch, it should stay around 1-2ms until libreNMS's scheduled polling happens

^^If I understand the ping issue correctly, it's expected bahavior and not a problem.

 

[WampaCow]

In case you guys were wondering, I seem to have fixed my ICX 7250-24 temps by adding a 60mm fan (Sunon MF60101V3-1000U-A99) on the ASIC (blowing down), restoring the stock exhaust fan, and relegating it to a closet where I won't hear the stock fan

I was able to wire both in the same header connector, but a Y-splitter would achieve the same thing. Someone can correct me if I'm wrong, but because the 60mm fan uses so little power, it should be okay also running off the same header, especially because the fans aren't running anywhere near full speed. I secured the fan to the ASIC heatsink with some twist ties wrapped around the heatsink mounting posts (see photo). I'm sure there are more elegant ways of doing it, but this works.

Temps in various configurations:

Fan Configuration​	Temp (C)​
Stock	79.0
Sunon KDE1204PKVX in place of stock fan	87.4
Stock exhaust fan, 60mm fan on ASIC	62.4

Hopefully this info will help someone trying to solve the same problem. Appreciate all the help and suggestions from everyone here.

 

[fohdeesha]

yall mfs crazy with them fans gahdamnbah

 

[Wolfcastle]

Totally worth it when you have a switch in your office!

yall mfs crazy with them fans gahdamnbah

 

[Freebsd1976]

In case you guys were wondering, I seem to have fixed my ICX 7250-24 temps by adding a 60mm fan (Sunon MF60101V3-1000U-A99) on the ASIC (blowing down), restoring the stock exhaust fan, and relegating it to a closet where I won't hear the stock fan

I was able to wire both in the same header connector, but a Y-splitter would achieve the same thing. Someone can correct me if I'm wrong, but because the 60mm fan uses so little power, it should be okay also running off the same header, especially because the fans aren't running anywhere near full speed. I secured the fan to the ASIC heatsink with some twist ties wrapped around the heatsink mounting posts (see photo). I'm sure there are more elegant ways of doing it, but this works.

Temps in various configurations:

Fan Configuration​	Temp (C)​
Stock	79.0
Sunon KDE1204PKVX in place of stock fan	87.4
Stock exhaust fan, 60mm fan on ASIC	62.4

Hopefully this info will help someone trying to solve the same problem. Appreciate all the help and suggestions from everyone here.

same as you , mine is 65C . consider the Temp (65C)and power consume (46w on idle ), now i am looking icx7150 replace icx7250

 

[Drewy]

Just update my small 2x 7250 stack to 8095. Upgrade went very smooth.

I've noticed that although my pre-existing extended ACL config shows up via a show running etc. It doesn't show up in the web gui. Neither do they appear to be tagged to the ve's that they were.
Trying to tag them to the original ve's via the CLI results in an error saying that they need to be tagged to vlans not ve's. I think I saw someone post about a while back but can't find it now. Anyway I tagged them to the vlan's that hold the ve's and that seems to work BUT they still don't show up on the web GUI. Not a big issue but curious if anyone else has seen this?

 

[bpye]

I've been thinking through my config, and I've just realised one potential snag... It may be an artifact of the ISP router currently (which I will replace) but my IPv6 prefix does not appear constant. I am hoping that it's just the router and if I run my own I may be able to get the same prefix back through renew/rebind, but there is still a chance that I won't.

Is there any way to deal with this in ACLs? One (terrible) idea I've had is to have my router machine push the ACLs to the switch via SSH so any time the address changes, realistically just on a power cycle I think, it'll get updated, but I'd love there to be a better way? It looks like for IPv4 we can use hostname, but I don't see that for IPv6?

 

[blademan]

I am concerned with the switch sagging.

Make sure the bolts are tight, and the rack frame is of high enough build quality to support it. I’ve seen plenty of gear mounted with just the front bottom bolts mounted, with no issues, gear much heavier and longer than these. The only time I’ve ever seen gear sag is when some clown installed something with only the top 2 bolts.

 

[AgentXXL]

The TL;DR summary: I’m looking for assistance configuring my ICX6610 switch to make use of the 5Gbps and 10Gbps onboard NICs in my 2 servers. My experience working with managed switches is limited, but I do have basic understanding of the functionality from working with IT folks at my last place of employment. Skip down to the section: HELP PLEASE for my questions.

I’m sure questions like mine get asked (and answered) numerous times throughout this thread, but my search skills often leave me frustrated. I’m slowly trying to read the entire thread and have only made it to page 25 so far… only another 313 to go! In the meantime I thought I’d post my issue(s) and see if I can get some assistance.

I’ve been a long-time lurker of the STH site and over the last 2 years I’ve been building out my homelab. So far I have 2 unRAID servers, a pfSense system (physical), and the newest addition to the computer room, a Brocade ICX6610 (48 port POE). For WiFi I use an Ubiquiti UAC-Pro AP powered by the POE on the new (to me) switch. My ISP provides a DOCSIS 3.1 connection offering 1Gbps down, 100Mbps up.

I’ve been playing with cheap 10G gear to learn on - some old HP/QLogix dual SFP 10G NICs and DAC cables were my 1st step in trying to improve my network performance between the 2 servers. They worked off and on under unRAID but eventually were more trouble than they were worth so I decided to move to something newer. I’ve since upgraded the much older CPU/motherboards to more modern gear. My 9yr old dual X5650 Xeon setup that came with my Supermicro CSE-847 has been replaced by a i9-10940X on an Asus Prime x299 Deluxe II. My 10yr old i7-980 hexacore system was replaced by a new 12th gen i9-12900K on a ROG Maximus z690 Formula just a few days ago.

My main unRAID server (x299 system) has an onboard Aquantia 5Gbps NIC and my 2nd unRAID server (z690) has an onboard 10Gbps Aquantia NIC. These are connected to the ICX6610 on two of the front SFP+ ports (licensed to 10G). As the onboard NICs both use RJ45, I’m using Cat6 and Cat 7 RJ45 cables with Wiitek SFP-10G-T modules at the switch end.

Alas I’m very much a rookie at configuring the switch and so far my speeds between the 2 unRAID systems have gotten worse than they were when I was only using 1Gbps NICs. I’m obviously doing something wrong, so here’s my ask:

HELP PLEASE: (for the TL;DR folks)

How should I configure the 5G and 10G ports on my unRAID servers to maximize throughput? I thought it might be as simple as enabling jumbo frames at the switch and setting the NICs and switch ports to use a higher MTU. This certainly hasn’t worked, or I’ve missed something in the configuration. I’ve also seen a few posts in this thread that say jumbo frames aren’t worth the trouble. If so, how else can I try to improve my speeds?

The system with the 5Gbps NIC also has an onboard 1Gbps NIC (currently disabled in the BIOS), but the other system only has the single 10Gbps NIC. I could add a 1Gbps if I had to, but that’s only a temporary workaround to my speed issues. When I was using the older systems with only 1Gbps NICs, I could saturate my ISP connection at the full 1Gbps rate - 110 - 120MB/sec.

Now that I’m using the 5G and 10G NICs, I’m getting about 50-70 MB/sec, and transfers between the 2 servers seem to max out around 40 MB/sec. Worse than it was when using 1G NICS. I’d appreciate any assistance or input and will try to answer questions to the best of my knowledge and abilities. TIA for the help!

 

[brb78]

I suspect my ICX6610 recently acquired from ebay has a bad console management port. When I connect a known-good usb-to-rj45 and view the serial output over minicom it either stops at

(if no connection to LAN)

ICX6610-48 Switch>

Power supply 2  detected.


Power supply 2  detected.

Power supply 2  is up.

or just repeats tftp session timed out after that (if connected to LAN)

I'm able to view serial output, but none of my keystrokes register. So no show version or show license commands seem posible through serial-to-rj45 over minicom

I can telnet in and receive the below

telnet@ICX6610-48 Switch>show license
License record empty
telnet@ICX6610-48 Switch>show version
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on May 06 2017 at 08:15:28 labeled as FCXS08030n
        (7762230 bytes) from Primary FCXS08030n.bin
        SW: Version 08.0.30nT7f1
  Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)
  HW: Stackable ICX6610-48
==========================================================================
UNIT 1: SL 1: ICX6610-48 48-port Management Module
      Serial  #: BXxxxxxxxx
      License: BASE_SOFT_PACKAGE   (LID: xxxxxxxxx)
      P-ENGINE  0: type E02B, rev 01
      P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 23 minute(s) 1 second(s)
The system : started=cold start

Is the console port bad? is there a way to check or some suggested troubleshooting?

serial output below:

ICX Boot Code Version 7.3.02 (grz07302)                                   

Enter 'a' to stop at memory test                                           

Enter 'b' to stop at boot monitor                                         

BOOT INFO: load monitor from boot flash, cksum = 1fef                     

BOOT INFO: verify flash files....                                         

BOOT INFO: load image from primary copy...                                 

                                                                          

platform type = 9                                                         

PCIE-1 LTSSM status: 22                                                   

PCIE Switch status: 0                                                     

......................                                                     

Firmware integrity checksum passed

.....

Starting Main Task .Applying factory defaults..

INFO: startup config data is not available, try to read from backup

INFO: startup config data in the backup area is not available

..CPSS DxCh Version: cpss3.4p1 release

Pre Parsing Config Data ...

INFO: empty config data in the primary area, try to read from backup

INFO: empty config data in the backup area also


Parsing Config Data ...

INFO: empty config data in the primary area, try to read from backup

INFO: empty config data in the backup area also


System initialization completed...console going online.

  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.

    UNIT 1: compiled on May 06 2017 at 08:15:28 labeled as FCXS08030n

                (7762230 bytes) from Primary FCXS08030n.bin

        SW: Version 08.0.30nT7f1

  Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)

  HW: Stackable ICX6610-48

==========================================================================

UNIT 1: SL 1: ICX6610-48 48-port Management Module

         Serial  #: BXxxxxxxxxx

         License: BASE_SOFT_PACKAGE   (LID: xxxxxxxxxxx)

         P-ENGINE  0: type E02B, rev 01

         P-ENGINE  1: type E02B, rev 01

==========================================================================

UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module

==========================================================================

UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module

==========================================================================

  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus

65536 KB flash memory

  512 MB DRAM

STACKID 1  system uptime is 2 minute(s) 30 second(s)

The system : started=cold start


ICX6610-48 Switch>

Power supply 2  detected.


Power supply 2  detected.

Power supply 2  is up.

TFTP session timed out

 

[kpfleming]

ICX6610 switch to make use of the 5Gbps and 10Gbps

Does the 6610 even support 5GbE? It's not listed in the datasheet.