No traffic is flowing. I have two WAN interfaces, one is dedicated for my homelab stuff and one is for general users. When I power cycle one of the modems only the WAN that I didn't touch stays up.
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Fantastic thank you for the resources and info.
I'll move this over to this thread if needed, but I just got these ConnectX-4 Lx in the mail, and as you mentioned
edit: I moved this over to another thread
quick answer for anyone looking:
echo 8 > /sys/devices/pci0000\:00/0000\:00\:1d.0/0000\:03\:00.0/sriov_numvfs
Last edited:
Got a small problem where I can't get the 10GB breakout interfaces to come up. I'm trying to brake out the 40GB to 10GB LCs using some FS.com gear. I've followed the setup guide for removing the ports from the stack.
To get the 10G LCs on the front of the rack, I've purchased the following:
- QSFP-SR4-40G Brocade 40G-QSFP-SR4 Compatible 40GBASE-SR4 QSFP+ 850nm 150m DOM MTP/MPO MMF Optical Transceiver Module
- 12FMTPOM4 1m (3ft) MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta
- FHD-1MTP4LCDOM4 FHD MTP®-8 Cassette, 8 Fibers OM4 Multimode, Universal Polarity, MTP® to 4 x LC Duplex (Aqua), 0.35dB max
I'm using FS.com 10GB 850nm SR transceiver's on the other end with out any problems over OM4 cables.
1/3/7 is an optic chucked into the front port used to test the LC cables. The other 1/3's are LAGs go to other switchgear.
FWIW, I set the 1/2/2 to 1/2/5 and 1/2/7 to 1/2/10 to speed-duplex 10G-full also left blank.
Here's the interface summary
Here's the interface that has a plugged in LC cable:
I've double checked that the cables are plugged in. I have 2 x of the above kit and I get the same behaviour on each 40GQSFP, MTP cable and breakout cassette.
Any ideas on what to try next?
To get the 10G LCs on the front of the rack, I've purchased the following:
- QSFP-SR4-40G Brocade 40G-QSFP-SR4 Compatible 40GBASE-SR4 QSFP+ 850nm 150m DOM MTP/MPO MMF Optical Transceiver Module
- 12FMTPOM4 1m (3ft) MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta
- FHD-1MTP4LCDOM4 FHD MTP®-8 Cassette, 8 Fibers OM4 Multimode, Universal Polarity, MTP® to 4 x LC Duplex (Aqua), 0.35dB max
I'm using FS.com 10GB 850nm SR transceiver's on the other end with out any problems over OM4 cables.
Code:
SSH@sw-core(config)#show media validation
Port Supported Vendor Type
----------------------------------------------------------------------
1/2/2 Yes BROCADE 40G QSFP-SR4
1/2/3 Yes BROCADE 40G QSFP-SR4
1/2/4 Yes BROCADE 40G QSFP-SR4
1/2/5 Yes BROCADE 40G QSFP-SR4
1/2/7 Yes BROCADE 40G QSFP-SR4
1/2/8 Yes BROCADE 40G QSFP-SR4
1/2/9 Yes BROCADE 40G QSFP-SR4
1/2/10 Yes BROCADE 40G QSFP-SR4
1/3/3 Yes FS Type : 10GE Passive Twinax 1m (SFP +) (Not supported)
1/3/4 Yes FS Type : 10GE Passive Twinax 1m (SFP +) (Not supported)
1/3/5 Yes FS Type : 10GE Passive Twinax 5m (SFP +) (Not supported)
1/3/6 Yes FS Type : 10GE Passive Twinax 5m (SFP +) (Not supported)
1/3/7 Yes OEM Type : 10GE SR 300m (SFP +)1/3/7 is an optic chucked into the front port used to test the LC cables. The other 1/3's are LAGs go to other switchgear.
Code:
SSH@sw-core(config)#show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disableFWIW, I set the 1/2/2 to 1/2/5 and 1/2/7 to 1/2/10 to speed-duplex 10G-full also left blank.
Here's the interface summary
Code:
1/2/1 Down None None None None Yes 1 0 748e.f8fe.c148
1/2/2 Down None None None None No 1 0 748e.f8fe.c148
1/2/3 Down None None None None Yes 1 0 748e.f8fe.c148
1/2/4 Down None None None None No 1 0 748e.f8fe.c148
1/2/5 Down None None None None No 1 0 748e.f8fe.c148
1/2/6 Down None None None None No 1 0 748e.f8fe.c148
1/2/7 Down None None None None Yes 1 0 748e.f8fe.c148 SW-XXX1
1/2/8 Down None None None None Yes 1 0 748e.f8fe.c148 SW-XXX2
1/2/9 Down None None None None Yes 1 0 748e.f8fe.c148 SW-XXX3
1/2/10 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/1 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/2 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/3 Up Forward Full 10G 3 Yes 1 0 748e.f8fe.c148 SW-DXX-1
1/3/4 Up Forward Full 10G 3 Yes 1 0 748e.f8fe.c148 SW-DXX-2
1/3/5 Up Forward Full 10G 2 Yes 1 0 748e.f8fe.c148 SW-RXX-1
1/3/6 Up Forward Full 10G 2 Yes 1 0 748e.f8fe.c148 SW-RXX-2
1/3/7 Down None None None None Yes 1 0 748e.f8fe.c148
1/3/8 Down None None None None Yes 1 0 748e.f8fe.c148
Code:
SSH@sw-core(config)#show int ethe 1/2/7
10GigabitEthernet 1/2/7 is down, line protocol is down
Port down for 2 hour(s) 25 minute(s) 13 second(s)
Hardware is 10GigabitEthernet , address is 748e.f8fe.c148 (bia 748e.f8fe.c17f)
Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of 11 L2 VLANs, port is dual mode in Vlan 1, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
Port name is SW-XXX1
MTU 10200 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
54035 packets input, 15271970 bytes, 0 no buffer
Received 2649 broadcasts, 3205 multicasts, 48181 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
129068 packets output, 41755954 bytes, 0 underruns
Transmitted 11005 broadcasts, 55975 multicasts, 62088 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Egress queues:
Queue counters Queued packets Dropped Packets
0 115303 0
1 0 0
2 0 0
3 0 0
4 0 0
5 1 0
6 0 0
Code:
SSH@sw-core>show ver
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
(10545591 bytes) from Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #: 2ax5o2jk68e
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: H4CKTH3PLN8)
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 5 day(s) 1 hour(s) 38 minute(s) 27 second(s)
The system started at 12:06:39 GMT+10 Sun Feb 20 2022
The system : started=cold startAny ideas on what to try next?
It's most likely a bug (or just...a not implemented feature) in your modem's firmware that's making it not send out a dhcp force renew broadcast (FORCERENEW ) on fresh boot, which gives clients new fresh leases even when the end client doesn't explicitly ask for one.
The reason you don't run into the bug/issue when the modem is directly connected to your pfsense box is because when you reboot the modem and its directly connected to your pfsense box, pfsense sees the physical link completely drop and come back up - this will obviously force a dhcp release, then a renew request on pfsense's behalf when the link is back up - it doesn't need the dhcp server side to broadcast a force renew.
when a switch is in between the two and you reboot the modem, the pfsense box doesn't see any link flaps, and will gladly hold onto the DHCP lease it was given earlier until either the lease timer expires, it sees a FORCERENEW broadcast, or you manually bounce the lease. A nicely coded modem immediately sends a "dhcp force renew" on fresh boot, so any clients on its connected l2 with old slowly expiring leases will immediately drop them and grab a new lease. I had an old docsis modem that didn't have forcerenew implemented, and yes it was annoying. my current DOCSIS modem (CM600) has it implemented and works great running through an ICX then to a virtual pfsense instance when rebooted, as an example
Another question for the network gurus here... probably more of a Mikrotik question but since it's connected to the ICX6610 I'm posting it in this thread.
Installing the Mikrotik CRS305 as a 'rate converter' for the onboard Aquantic 5Gbps NIC in one of my systems has provided better transfer speeds, although it's still not as good as I expect. That's likely due to the older firmware on the onboard NIC - I'm still waiting on Asus to provide a working firmware updater.
Alas I'm seeing frequent 'disconnects' from the system with the 5Gbps NIC. I notice it when watching content via Plex and randomly playback will halt. I sometimes get the familiar 'buffering' message from the Plex client about my network not being fast enough, other times it just sits there. If I try to ping the media server, it won't respond. Going to the management page for the Mikrotik, it shows the link as up, but I can't seem to figure out why it stops responding.
My solution has been to reboot the Mikrotik, which in SwOS takes less than 20 seconds. After the reboot completes, the system in question 'reconnects' automatically and Plex playback resumes. I've also had this happen during file transfers between systems. With those I usually have to restart the transfer from where it left off.
While my speeds were much worse when directly connected to the ICX6610, I wasn't seeing these disconnects. I went with the Mikrotik due to the 6610 not supporting 2.5 and 5Gbps NBase-T with most of the common SFP+ to RJ45 modules. I purchased a Mikrotik S+RJ10 module with the CRS305 as recommended, and it shows as a 5Gbps link, but so did my Wiitek modules when I tried them.
If anyone has any suggestions on how to troubleshoot this, or know if there are options I can configure on the Mikrotik to try and mitigate this issue, please let me know. Thanks!
Installing the Mikrotik CRS305 as a 'rate converter' for the onboard Aquantic 5Gbps NIC in one of my systems has provided better transfer speeds, although it's still not as good as I expect. That's likely due to the older firmware on the onboard NIC - I'm still waiting on Asus to provide a working firmware updater.
Alas I'm seeing frequent 'disconnects' from the system with the 5Gbps NIC. I notice it when watching content via Plex and randomly playback will halt. I sometimes get the familiar 'buffering' message from the Plex client about my network not being fast enough, other times it just sits there. If I try to ping the media server, it won't respond. Going to the management page for the Mikrotik, it shows the link as up, but I can't seem to figure out why it stops responding.
My solution has been to reboot the Mikrotik, which in SwOS takes less than 20 seconds. After the reboot completes, the system in question 'reconnects' automatically and Plex playback resumes. I've also had this happen during file transfers between systems. With those I usually have to restart the transfer from where it left off.
While my speeds were much worse when directly connected to the ICX6610, I wasn't seeing these disconnects. I went with the Mikrotik due to the 6610 not supporting 2.5 and 5Gbps NBase-T with most of the common SFP+ to RJ45 modules. I purchased a Mikrotik S+RJ10 module with the CRS305 as recommended, and it shows as a 5Gbps link, but so did my Wiitek modules when I tried them.
If anyone has any suggestions on how to troubleshoot this, or know if there are options I can configure on the Mikrotik to try and mitigate this issue, please let me know. Thanks!
Does anyone know if you can put 1x40G modules in all three of the module slots on the 7450? I'm not seeing an option to set the front module slot to 1x40G qsfp, only 4x10g sfp. Tagging @LodeRunner but any 7450 owners might have ran into this already too.
on the 24 port chassis yes on the 48 port chassis no. 7450 big dumb doodoo switch with 9 million dollar 4x10g modules
Biggest face palm ever. Why?! I...nevermind. Thank you hahaha I guess I'm on the hunt for a 24p variant of this bad boy. Really liking this switch otherwise though.on the 24 port chassis yes on the 48 port chassis no. 7450 big dumb doodoo switch with 9 million dollar 4x10g modules
Last edited:
@fohdeesha
looks like the latest firmware for the fastiron on ICX platforms has a completely revamped GUI, version 9.0.10a
i know in your informational posts you commended the ICX7250 platform because it used an honor based licensing system, but i was never sure if that was only because it was on v8080 firmware (which i assume translates to 8.0.80 in ruckus firmware notation)
was it ever planned by ruckus/brocade to phase out the honor based licensing for these switches?
i'm aware you may not be the biggest fan of gui for switches but it will be nice to integrate these with my ruckus r500 which have a similar interface aesthetic
looks like the latest firmware for the fastiron on ICX platforms has a completely revamped GUI, version 9.0.10a
i know in your informational posts you commended the ICX7250 platform because it used an honor based licensing system, but i was never sure if that was only because it was on v8080 firmware (which i assume translates to 8.0.80 in ruckus firmware notation)
was it ever planned by ruckus/brocade to phase out the honor based licensing for these switches?
i'm aware you may not be the biggest fan of gui for switches but it will be nice to integrate these with my ruckus r500 which have a similar interface aesthetic
Just confirm that I bought a Supermicro Aquantia AQS-107-B0C2-CX 10G on ebay for $125 , and it's working great at 2.5Gbps from my main PC with a ASUS TUF X570-Pro motherboard, so far iPerf got 2.2 and 2.3 Gbps sustained,
since that was the last one from that vendor, now in the hunt for another one at a reasonable price
Thanks again for the reply
Thanks for confirming the issue.
I've finally decided to downgrade fw to 8.0.90mc where it's still working.
This is great info. Thanks for the insight. I had a feeling that pfsense not detecting a down link had something to do with it.
I’m going to play around with my setup a bit more with this knowledge in mind.
I’m going to play around with my setup a bit more with this knowledge in mind.
Glad it worked out for you... I ended up grabbing a Mikrotik CRS305 and one of their S+RJ10 SFP+ to RJ45 modules as I couldn't find a supplier that had stock and/or would ship to Canada. I'm still shopping for one of the Supermicro modules but I suspect I have another issue to fix - specifically the onboard 5Gbps NIC itself. The Mikrotik is really revealing the issues with the onboard 5Gbps NIC (Aquantia AQC111C) on my Asus Prime x299 Deluxe II. Specifically the random disconnects... when it happens I have to either unplug/re-attach the cable or reboot the Mikrotik. Thankfully under SwOS it takes less than 20 secs to reboot, but it's still a major annoyance.
I have a case open with Asus regarding a firmware updater for the NIC. The official updater from Marvell (who bought Aquantia) doesn't work on my motherboard, and neither does the community modded version which has added IDs for some Asus motherboards. Asus themselves state there's no known problem with the NIC and no firmware updater is available - they want me to RMA the board. But a simple Google search shows many with similar issues to what I'm experiencing, and most are resolved by updating the NIC firmware. And many of the posts about the issues are in Asus's own ROG forums. They've used the Aquantia AQC111C on a number of their products in the regular and ROG lines.
I provided a bunch of these links to Asus and am waiting to hear back. If I could figure out how to find the required ID info for my board, I'd add it myself to the XML config file in the updater. That XML file contains the IDs of systems that the updater works for. Hopefully I can get mine updated and it resolves my issues.
In the meantime, I have a LOT of learning to do about configuring my home network. I'm having difficulties with VLAN and/or port isolation but I'm sure it's just a lack of knowledge issue. Back to my reading...
I have 3 mainboards with Aquantia chipset ( 2x aqc-107 and 1x aqc-108 ). In any case the last firmware updates from Marvell website NEEDs the driver 2.2 or newer to firmware updater detect and update. I have spended hours in this and discovered by chance after swaping the system ssd ( newer system = newer driver download
).My system normally runs under unRAID but I booted to my trusty WinToGo Win10 x64 and installed the latest driver from Marvell - for my AQC111C that's driver 3.1.6.0. Alas still no joy for the firmware updater. I'm more than certain it's the missing system IDs in the XML file that's preventing the updater from working. It sees my NIC and shows the outdated firmware - v. 3.1.50 vs the 3.1.121 on both Marvell's site and the community modded version from station-drivers.com.I have 3 mainboards with Aquantia chipset ( 2x aqc-107 and 1x aqc-108 ). In any case the last firmware updates from Marvell website NEEDs the driver 2.2 or newer to firmware updater detect and update. I have spended hours in this and discovered by chance after swaping the system ssd ( newer system = newer driver download
Anything I need to do on the Brocade side specifically to get these working? Connecting two cards together = connection, connecting the brocade ports together = connection, but I can't get the card to talk to the Brocade at all.
From the Brocade side we get
Code:
1/3/1 Down None None None None No 1 0 cc4e.2488.3380
1/4/1 Down None None None None No 1 0 cc4e.2488.3380@jasonwc you seem to have a good thing going
Last edited:
As far as I'm aware, the Brocades in general don't care who the transceiver MFR is. I'm using DACs though for my 40G LAG (7450 to Arista 7050), so I haven't used any 40G optics in it.
What's the output of "sh med e 1/3/1" and "sh int e 1/3/1"
Edit: saw you direct connected the cards and it came up, and you looped the Brocade interfaces and it came up, but card to port isn't. That's really odd.
The more I look at this the more it feels like I haven't got these Mellanox cards working correctly given their MAC addresses are all 00:00:00:00:00:00... See my Mellanox thread over here. My other pet theory is that I need to be using the mlx5_en driver instead of the mlx5_core driver, but I also don't know how to do that.
sh med e 1/3/1
Code:
Port 1/3/1: Type : 40GE-LR4 2km (QSFP+ LC)
Vendor: KAIAM CORP Version: 1A
Part# : XQX2502 Serial#: KD60315209sh int e 1/3/1
Code:
40GigabitEthernet1/3/1 is down, line protocol is down
Port down for 3 minute(s) 16 second(s)
Hardware is 40GigabitEthernet, address is cc4e.2488.3380 (bia cc4e.2488.33b5)
Interface type is 40Gig Fiber
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Untagged member of L2 VLAN 1, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
MACsec is Disabled
Openflow is Disabled, Openflow Hybrid mode is Disabled, Flow Control is config enabled, oper enabled, negotiation disabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
VLAN-Mapping is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
IPG MII 96 bits-time, IPG GMII 96 bits-time
MTU 1500 bytes, encapsulation ethernet
MMU Mode is Store-and-forward
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
0 packets output, 0 bytes, 0 underruns
Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled
Protected: No
MAC Port Security: Disabled
Last edited: