because in 2023 if you are having performance issues with 10gbE, you have much deeper issues with your setup and jumbo frames will be a bandaid. I can saturate a 10gbps link with default 1500 MTU on a $10 15 year old xeon, if you need jumbo frames to make 10gbe work on something newer you have bigger fish to fry. 40gbe and up is a different story depending on hardware and traffic typeI totally profess my ignorance on all things layer 2 and up, but how so? I'm not running them, the effort isn't worth it for my home lab, but it's often recommended, especially when discussions of poor 10g performance are brought up.
I am in the process of revamping my home network to 10GbE. I installed a duplex foc from my server room to one of the bedrooms in the house. The bedroom has a windows 11 pc with a mellanox cx3 card. The server room has a windows 11 laptop with a qnap 10gbe sfp+ to thunderbolt adapter. To test, I used iperf3. I cannot saturate the link until I enabled jumbo frames. So is this a limitation from the qnap adapter or iperf3?because in 2023 if you are having performance issues with 10gbE, you have much deeper issues with your setup and jumbo frames will be a bandaid. I can saturate a 10gbps link with default 1500 MTU on a $10 15 year old xeon, if you need jumbo frames to make 10gbe work on something newer you have bigger fish to fry. 40gbe and up is a different story depending on hardware and traffic type
what's the output of "show pod"Hi all, I have an ICX6650 and I am attempting to connect a Mellanox MSX6720. I am attempting to use ports 1/2/3 and ports 1/2/4 to connect to the Mellanox switch configured in ethernet mode for ports 1/35 and 1/36. Both systems when I run show media, both systems show the type of connected cable.
Below is the output from the Mellanox side
Admin state : Enabled
Operational state : Up
Last change in operational status : 14:40:24 ago (77 oper change)
Boot delay time : 0 sec
Description : UpLink-ICX6650
Mac address : 24:8a:07:d1:05:4f
MTU : 9216 bytes (Maximum packet size 9238 bytes)
Fec : auto
Flow-control : receive off send off
Actual speed : 40 Gbps
Auto-negotiation : Enabled
Width reduction mode : Unknown
Switchport mode : trunk
MAC learning mode : Enabled
Last clearing of "show interface" counters: Never
60 seconds ingress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec
60 seconds egress rate : 0 bits/sec, 0 bytes/sec, 0 packets/sec
0 unicast packets
0 multicast packets
0 broadcast packets
0 discard packets
0 error packets
0 fcs errors
0 undersize packets
0 oversize packets
0 pause packets
0 unknown control opcode
0 symbol errors
0 unicast packets
7 multicast packets
0 broadcast packets
2789797 discard packets
0 error packets
0 hoq discard packets
Identifier : QSFP+ Cable/ Module type : Passive copper, unequalized Ethernet speed and type : 56GigE Vendor : Mellanox
Cable length : 3 m Part number : 670759-B25 Revision : A1 Serial number : 6C2634028Jshow interfaces ethernet 1/2/3
40GigabitEthernet1/2/3 is down, line protocol is down
Port down for 14 hours 47 minutes 37 seconds
Hardware is 40GigabitEthernet, address is 748e.f8fb.1c00 (bia 748e.f8fb.1c3b)
Interface type is 40Gig Copper
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 9 L2 VLANs, port is tagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Enabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 10240 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
145 packets input, 19778 bytes, 0 no buffer
Received 0 broadcasts, 145 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
228 packets output, 30627 bytes, 0 underruns
Transmitted 0 broadcasts, 228 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 228 0show media ethernet 1/2/3
Port 1/2/3: Type : 40GBASE-Passive Copper
Vendor Name: Mellanox Serial Num: 6C2634028J Revision: A1
I am not getting any traffic across, it seems as if the ICX switch just is not happy with the link. I have tried a Plain Jane non-Mellanox 40gb cable as well and it has the same issue. The Mellanox switch lights the port, but the ICX switch will not.
Any pointers on making this work would be great.
The only place I use jumbo frames is with storage, generally FCIP doing replication traffic out of an FC network. Not something I've had to deal with much for years though since many storage arrays that I've dealt with more recently have ethernet ports for replication. Some of them strongly suggest using jumbo frames, though they'll settle for standard frames and complain about it.Why? I was looking into enabling jumbo frames as I thought they're good?
what's the output of "show pod"
@Juggie I know that was a very old post of yours, but do you think you could share your switches config? I am really struggling to get multicast working on my network and I've tried everything.
Hey RoachedCoach, big fan of your fan mod and the stock fans definitely are not passing my wife-test noise-wise ("is it always that loud??" after the fans spun down).Wanted to plug my post on my fan changes to a 7250-48p.
I, too, tried to follow @RoachedCoach mod but was not able to get the recommended Mechatronics.Hey RoachedCoach, big fan of your fan mod and the stock fans definitely are not passing my wife-test noise-wise ("is it always that loud??" after the fans spun down).
I can get a Sunon MF60101V3-1000U-A99 ok but I'm having no luck at all getting the Mechatronics MR4020X12B1-RSR in the UK and the digikey.com US site is also saying 0 in stock so maybe there is a stock issue or they're discontinuing them...?
Do you or anyone else know a replacement you can recommend that is similarly quiet and monitorable that I can source in the UK...?
If you are in the UK, I found this guy on eBay located in the UK:
180 of them just came in stock today at Digikey so I've ordered 4x which worked out to be €43 delivered with free shipping and Digikey are paying custom fees(!) so we'll see what happens.I, too, tried to follow @RoachedCoach mod but was not able to get the recommended Mechatronics.
At Mouser you cannot buy them if you are not a reseller and Digikey wanted 100€ + VAT + custom fees for 3 of them some weeks ago.
I bought the MF60101V3-1000U-A99 and 3 x MF40201VX-G99-A which are detected by the switch and keep it reasonably cool, but there are some drawbacks, at least for me.
The MF4020 "resonate" if there are 2 or more installed. I read @rootwyrm talking about this issue previously on this very thread, but don't know if he was talking about this model or just about all Sunon fans.
I wouldn't say the switch is quiet, but it's beareable.
If I can source some Mechatronics I'll try them to see if I can make it quieter.
Multicast config is all default no changes. Once I upgraded the firmware it worked. What switch do you have? Does it have the latest firmware?