Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kevindd992002]

it's almost 2024 don't use jumbo frames

Why? I was looking into enabling jumbo frames as I thought they're good?

 

[fohdeesha]

in 2003

 

[bwahaha]

in 2003

I totally profess my ignorance on all things layer 2 and up, but how so? I'm not running them, the effort isn't worth it for my home lab, but it's often recommended, especially when discussions of poor 10g performance are brought up.

 

[fohdeesha]

I totally profess my ignorance on all things layer 2 and up, but how so? I'm not running them, the effort isn't worth it for my home lab, but it's often recommended, especially when discussions of poor 10g performance are brought up.

because in 2023 if you are having performance issues with 10gbE, you have much deeper issues with your setup and jumbo frames will be a bandaid. I can saturate a 10gbps link with default 1500 MTU on a $10 15 year old xeon, if you need jumbo frames to make 10gbe work on something newer you have bigger fish to fry. 40gbe and up is a different story depending on hardware and traffic type

 

[kevindd992002]

because in 2023 if you are having performance issues with 10gbE, you have much deeper issues with your setup and jumbo frames will be a bandaid. I can saturate a 10gbps link with default 1500 MTU on a $10 15 year old xeon, if you need jumbo frames to make 10gbe work on something newer you have bigger fish to fry. 40gbe and up is a different story depending on hardware and traffic type

I am in the process of revamping my home network to 10GbE. I installed a duplex foc from my server room to one of the bedrooms in the house. The bedroom has a windows 11 pc with a mellanox cx3 card. The server room has a windows 11 laptop with a qnap 10gbe sfp+ to thunderbolt adapter. To test, I used iperf3. I cannot saturate the link until I enabled jumbo frames. So is this a limitation from the qnap adapter or iperf3?

 

[fohdeesha]

Hi all, I have an ICX6650 and I am attempting to connect a Mellanox MSX6720. I am attempting to use ports 1/2/3 and ports 1/2/4 to connect to the Mellanox switch configured in ethernet mode for ports 1/35 and 1/36. Both systems when I run show media, both systems show the type of connected cable.
Below is the output from the Mellanox side

Spoiler: Mellanox Interface

Eth1/36:
Admin state : Enabled
Operational state : Up
Last change in operational status : 14:40:24 ago (77 oper change)
Boot delay time : 0 sec
Description : UpLink-ICX6650
Mac address : 24:8a:07:d1:05:4f
MTU : 9216 bytes (Maximum packet size 9238 bytes)
Fec : auto
Flow-control : receive off send off
Actual speed : 40 Gbps
Auto-negotiation : Enabled
Width reduction mode : Unknown
Switchport mode : trunk
MAC learning mode : Enabled
Last clearing of "show interface" counters: Never
60 seconds ingress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec
60 seconds egress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec

Rx:
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 discard packets
0 error packets
0 fcs errors
0 undersize packets
0 oversize packets
0 pause packets
0 unknown control opcode
0 symbol errors

Tx:
7 packets
0 unicast packets
7 multicast packets
0 broadcast packets
570 bytes
2789797 discard packets
0 error packets
0 hoq discard packets

Spoiler: Connected Cable Mellanox Side

Identifier :		QSFP+
Cable/ Module type :		Passive copper, unequalized
Ethernet speed and type :		56GigE
Vendor :		Mellanox

Cable length :		3 m
Part number :		670759-B25
Revision :		A1
Serial number :		6C2634028J

Spoiler: ICX Switch Interface

show interfaces ethernet 1/2/3
40GigabitEthernet1/2/3 is down, line protocol is down
Port down for 14 hours 47 minutes 37 seconds
Hardware is 40GigabitEthernet, address is 748e.f8fb.1c00 (bia 748e.f8fb.1c3b)
Interface type is 40Gig Copper
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 9 L2 VLANs, port is tagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Enabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10240 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
145 packets input, 19778 bytes, 0 no buffer
Received 0 broadcasts, 145 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
228 packets output, 30627 bytes, 0 underruns
Transmitted 0 broadcasts, 228 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 228 0

Spoiler: ICX Interface Media

show media ethernet 1/2/3
Port 1/2/3: Type : 40GBASE-Passive Copper
Vendor Name: Mellanox Serial Num: 6C2634028J Revision: A1

I am not getting any traffic across, it seems as if the ICX switch just is not happy with the link. I have tried a Plain Jane non-Mellanox 40gb cable as well and it has the same issue. The Mellanox switch lights the port, but the ICX switch will not.

Any pointers on making this work would be great.
Thanks

what's the output of "show pod"

 

[Jason Antes]

Why? I was looking into enabling jumbo frames as I thought they're good?

The only place I use jumbo frames is with storage, generally FCIP doing replication traffic out of an FC network. Not something I've had to deal with much for years though since many storage arrays that I've dealt with more recently have ethernet ports for replication. Some of them strongly suggest using jumbo frames, though they'll settle for standard frames and complain about it.

 

[linuxsrc]

what's the output of "show pod"

Spoiler: Show Pod

show pod

Unit-Id: 1
PoD 10G license capacity: 24
PoD 10G license capacity used: 24
PoD 40G license capacity: 24
PoD 40G license capacity used: 6

PoD-ports Lic-Available Lic-Used
1/1/33 Yes Yes
1/1/34 Yes Yes
1/1/35 Yes Yes
1/1/36 Yes Yes
1/1/37 Yes Yes
1/1/38 Yes Yes
1/1/39 Yes Yes
1/1/40 Yes Yes
1/1/41 Yes Yes
1/1/42 Yes Yes
1/1/43 Yes Yes
1/1/44 Yes Yes
1/1/45 Yes Yes
1/1/46 Yes Yes
1/1/47 Yes Yes
1/1/48 Yes Yes
1/1/49 Yes Yes
1/1/50 Yes Yes
1/1/51 Yes Yes
1/1/52 Yes Yes
1/1/53 Yes Yes
1/1/54 Yes Yes
1/1/55 Yes Yes
1/1/56 Yes Yes
1/2/1 Yes Yes
1/2/2 Yes Yes
1/2/3 Yes Yes
1/2/4 Yes Yes
1/3/1 Yes Yes
1/3/2 Yes Yes
1/3/3 Yes Yes
1/3/4 Yes Yes
1/3/5 Yes Yes
1/3/6 Yes Yes
1/3/7 Yes Yes
1/3/8 Yes Yes

What is strange about this port, normally you see an insertion in the logs when a cable is inserted or removed. I have turned debug on via debug port. And usually, I will see

2023 Nov 21 14:05:19:I:System: Interface ethernet 1/1/3, state up
2023 Nov 21 14:05:18:I:System: Interface ethernet 1/1/3, state down

But these 2 ports on the ICX 1/2/3 and 1/2/4 don't do that

It acts like the are no changes at all.

I just put a breakout on the 1/3/1 and it links up fine and shows LLDP data on the port. So I don't think it is a backplane issue.
My next test will be to get a server connected to that port.

I have tested this now with a dual-port ConenctX-3 Pro. I plugged one into the Mellanox switch, works fine and passes data. The other to the ICX switch and nothing. It does the same, detects the media but does not start the interface.

 

[notdeadyet]

upgrading from 30t -> 30u solved the problem. Hat tip to a post I found on reddit here: https://www.reddit.com/r/homelab/comments/eyvny7 I sent the guy a pm and he let me know 30u solved the issue. Flashed that, issue fixed.

@Juggie I know that was a very old post of yours, but do you think you could share your switches config? I am really struggling to get multicast working on my network and I've tried everything.

 

[Andydude]

Wanted to plug my post on my fan changes to a 7250-48p.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-159#post-256375

Hey RoachedCoach, big fan of your fan mod and the stock fans definitely are not passing my wife-test noise-wise ("is it always that loud??" after the fans spun down).

I can get a Sunon MF60101V3-1000U-A99 ok but I'm having no luck at all getting the Mechatronics MR4020X12B1-RSR in the UK and the digikey.com US site is also saying 0 in stock so maybe there is a stock issue or they're discontinuing them...?

Do you or anyone else know a replacement you can recommend that is similarly quiet and monitorable that I can source in the UK...?

Thanks!

 

[Ralph_IT]

Hey RoachedCoach, big fan of your fan mod and the stock fans definitely are not passing my wife-test noise-wise ("is it always that loud??" after the fans spun down).

I can get a Sunon MF60101V3-1000U-A99 ok but I'm having no luck at all getting the Mechatronics MR4020X12B1-RSR in the UK and the digikey.com US site is also saying 0 in stock so maybe there is a stock issue or they're discontinuing them...?

Do you or anyone else know a replacement you can recommend that is similarly quiet and monitorable that I can source in the UK...?

Thanks!

I, too, tried to follow @RoachedCoach mod but was not able to get the recommended Mechatronics.
At Mouser you cannot buy them if you are not a reseller and Digikey wanted 100€ + VAT + custom fees for 3 of them some weeks ago.
I bought the MF60101V3-1000U-A99 and 3 x MF40201VX-G99-A which are detected by the switch and keep it reasonably cool, but there are some drawbacks, at least for me.
The MF4020 "resonate" if there are 2 or more installed. I read @rootwyrm talking about this issue previously on this very thread, but don't know if he was talking about this model or just about all Sunon fans.
I wouldn't say the switch is quiet, but it's beareable.
If I can source some Mechatronics I'll try them to see if I can make it quieter.

 

[shpitz461]

I, too, tried to follow @RoachedCoach mod but was not able to get the recommended Mechatronics.
At Mouser you cannot buy them if you are not a reseller and Digikey wanted 100€ + VAT + custom fees for 3 of them some weeks ago.

If you are in the UK, I found this guy on eBay located in the UK:

FAN AXIAL 40X20MM 12VDC WIRE | eBay

Fan Type: Tubeaxial. Features: Auto Restart Speed Sensor (Tach). Weight: 0.053 lb (24.04 g). Approvals: CE cUL TUV UL. Ingress Protection: -. RPM: 12000 RPM.

www.ebay.com

 

[Andydude]

I, too, tried to follow @RoachedCoach mod but was not able to get the recommended Mechatronics.
At Mouser you cannot buy them if you are not a reseller and Digikey wanted 100€ + VAT + custom fees for 3 of them some weeks ago.
I bought the MF60101V3-1000U-A99 and 3 x MF40201VX-G99-A which are detected by the switch and keep it reasonably cool, but there are some drawbacks, at least for me.
The MF4020 "resonate" if there are 2 or more installed. I read @rootwyrm talking about this issue previously on this very thread, but don't know if he was talking about this model or just about all Sunon fans.
I wouldn't say the switch is quiet, but it's beareable.
If I can source some Mechatronics I'll try them to see if I can make it quieter.

180 of them just came in stock today at Digikey so I've ordered 4x which worked out to be €43 delivered with free shipping and Digikey are paying custom fees(!) so we'll see what happens.

Thanks for the recommendation!

 

[ivans89]

I ordered them too from digikey, but 5 to get free delivery with fedex.

 

[Andydude]

I ordered them too from digikey, but 5 to get free delivery with fedex.

Good luck, I'm putting 3x in a 7250-48 and 1x in a 6450-48.

Anyone know if just 1x MR4020X12B1-RSR in a 6450-48 will cool it ok? It's not PoE and I'm only probably plugging in 7-8 devices...

 

[Juggie]

@Juggie I know that was a very old post of yours, but do you think you could share your switches config? I am really struggling to get multicast working on my network and I've tried everything.

Multicast config is all default no changes. Once I upgraded the firmware it worked. What switch do you have? Does it have the latest firmware?

 

[Glynax]

For the Brocade ICX6610 is it possible to use all the rear ports as 40g? I'm guessing no but I thought I'd ask.

 

[Dave Corder]

For the Brocade ICX6610 is it possible to use all the rear ports as 40g? I'm guessing no but I thought I'd ask.

Nope

 

[bwahaha]

For the Brocade ICX6610 is it possible to use all the rear ports as 40g? I'm guessing no but I thought I'd ask.

no, two are 1x40g, two are 4x10g. You can LAG them together, but it'll still only be 10g.

 

[Gerhen]

I picked up a 7250-48 after finding this thread and reading through. Thanks to all who have contributed their time/insight here.

Curious to know how many are running 09.0.10x branch rather than the stable 08.0.95x branch ?