Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
cpu on what? traffic on the ICX is not handled by a CPUSeems like the CPU isn´t handling high speeds
if you're actually pushing (or trying to push) more than 10gb or 1gb to a 10gb or 1gb port, then you're going to drop packets - it's physics. If you have flow control enabled the switch can try to send pause frames to the sending host saying the receiving port/client can't receive that fast, but flow control is hit and miss. then you have other complications like the ASIC buffer which is gunna fill during this, and a bunch of other stuff. sending data from a fast port to a slow port is not gunna turn the slow port into a fast one, it can only fit so many packets
this sounds like weird throttling/tcp window scaling/etc on the opnsense side, especially if setting the opnsense interface to 1gbe so it can't send any faster than that fixes it
@fohdeesha
Thank you for replying.
I understand that I wont get the higher speed while sending to a slower port but if it's dropping packets and I'm getting ~%20 slower rate than the slowest port is getting then there's something wrong.
I had an ASUS RT-89AX sitting around (Has a 10Gbase-T port and a 10GB SFP+ port). Both can be used for WAN or LAN. However with OPNSense out of the picture and with completely different hardware and software the speed test results did not change.
There's something in your hardware/config that's not quite right.
I run a 6610 and Opnsense as well (Opnsense is on bare-metal with a 10gb connection to the 6610). Although my internet is only 1gb symmetric, I think given that our hardware/software is almost similar, it's relevant. I can do almost wire speed NAT from any machine on my network, regardless of its connection to the 6610 (I've tested with both 1gb and 10gb links.)
But...I don't have any 2.5g devices/adapters in the mix. Not sure if that's what might be an issue here.
I've got some good new. Just browsing around in the cli I found the symmetric-flow-control enable command. After I enabled it my speedtests on 1gb links shot up to where they should be ~940. At least on PCs with 1G NICs. I still need to test speeds more than 1GB, but I did notice there weren't any dropped packets too.
The 'GUI' doesn't have any configuration, you probably mean the management interface which is typically reached via VLAN 1 (using a virtual ethernet interface on that VLAN). If that's what you mean, yes, you can configure the VE on VLAN 1 to use DHCP to get address/gateway/etc. information, although that means if your DHCP server is unavailable for any reason you may lose the ability to manage the ICX over the network.
Thanks for the excellent documentation on this.
Short of cutting holes in the top for 120mm fans (which I'm considering), are there any workarounds for a dead fan MOSFET? I did replace the stock fans, but at full speed, they're still fairly loud.
Short of cutting holes in the top for 120mm fans (which I'm considering), are there any workarounds for a dead fan MOSFET? I did replace the stock fans, but at full speed, they're still fairly loud.
Thank you for your response. I realize the implication if the DHCP server is unavailable but I presently cannot access the management interface on a static IP through the existing wireguard VPN tunnel.
Hi, I havent seen it mentioned,
would a 6510 work with licensing trick as well? It's not on the compatibility list
would a 6510 work with licensing trick as well? It's not on the compatibility list
that is not an ethernet switch, it's fibrechannel. so nope, this thread is for the ICX ethernet line
Hello, everyone. I am setting up my second ICX-6610. The first one had zero issues, but the second one gave me a weird issue.
"Error! Stack unit 1 does not have POE module"
Both switches seem identical ICX 6610-48 (the second one has 2 PSU tho).
Any information to the right direction would be appreciated.
"Error! Stack unit 1 does not have POE module"
Both switches seem identical ICX 6610-48 (the second one has 2 PSU tho).
Any information to the right direction would be appreciated.
POE model's have a P at the end of their model name so for a POE ICX 6610-48 it would be ICX 6610-48 for non POE, and ICX 6610-48P for the POE model
I have an ICX 6610 with the 10G SFP+ ports enabled, and I'm trying to use these http://fs.com/products/42385.html transceivers, but when I plug them in nothing is detected. Are these incompatible?
Not sure about compatibility, but are you sure you want to connect to a device up to 10km away from your switch?
Most commonly in home lab and data centers 850nm SR (=short range; up to 300m) transceivers are used, such as these https://www.fs.com/products/31443.html?attribute=94972&id=3752829
The good news here: they are cheaper and use less power.
LR has no minimum distance though, so it should be fine in that regard. Yeah, SR LC UPC Duplex does make more sense but I needed white cables and happened to have some white simplex fiber lying around from a previous installation, so decided to buy the BiDi transceivers. I couldn't really find reputable sources for white single-mode fiber, but I didn't spend too much time looking since I had this fiber already and so assumed it would work.
Hoping I'm just missing something in the configs though - does FS.com do returns?
Last edited:
I'm using only single mode fibre. In fact, my in-wall installation is 2*cat6a cables +4*single mode fiber.LR has no minimum distance though, so it should be fine in that regard. Yeah, SR LC UPC Duplex does make more sense but I needed white cables and happened to have some white simplex fiber lying around from a previous installation, so decided to buy the BiDi transceivers. I couldn't really find reputable sources for white single-mode fiber, but I didn't spend too much time looking since I had this fiber already and so assumed it would work.
Hoping I'm just missing something in the configs though - does FS.com do returns?
Even in-rack, i'm using LR sfp/sfp+, and haven't had any issues with any sfp's so far, with some 3m cables.
Supposedly, the transceiver EEPROM can be reprogrammed to report as a duplex transceiver and the switch would detect it. My only board with an SFP+ connection is a BPi-R4 running OpenWRT and it doesn't look like the MTK driver supports dumping or modifying the EEPROM or registers:LR has no minimum distance though, so it should be fine in that regard. Yeah, SR LC UPC Duplex does make more sense but I needed white cables and happened to have some white simplex fiber lying around from a previous installation, so decided to buy the BiDi transceivers. I couldn't really find reputable sources for white single-mode fiber, but I didn't spend too much time looking since I had this fiber already and so assumed it would work.
Hoping I'm just missing something in the configs though - does FS.com do returns?
Code:
root@OpenWrt:~# ethtool -e eth1
Cannot get EEPROM data: Not supported
root@OpenWrt:~# ethtool -i eth1
driver: mtk_soc_eth
version: 6.6.63
firmware-version:
expansion-rom-version:
bus-info: 15100000.ethernet
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: noethtool on desktop x86 Linux,or are there any cheap ways to get access to an SFP+ programmer?
Update: Apparently the FS BOX does not support changing tranceiver type like this.
Update 2: The FS Box does indeed support this, I changed the brand to Brocade and the part number to SFP-10GLR-31 and it works perfectly now with optical monitoring.
Last edited:
I've got an ICX6650, attempting to use the QSFP ports on the rear and I'm encountering an odd problem. Ive gotten a few "QSFP-40G-SR-BD" modules (some Proline and a couple Cisco) and for some reason only ports 1/2/3, 1/3/1-4 and 1/3/5-8 seem to work. Of the slot 2 ports only 1/2/3 gives light (and only when putting in the Proline optic, the Cisco one gives no light). If I take an LC cable and loop it back the port shows active and lights up, but even with the loopback the other 40gb ports dont work. The modules do appear to work in the 2 4x10Gb ports though, I can both loopback and connect the ports to each other and they show active and give light.
I also have a couple HPE QSFPs (817040-B21), and they seem to give light on all ports show show up on
If I have the Proline modules plugged into 1/2/1, 1/2/2, 1/2/3 and 1/2/4 this is some of the output I get (lines referring to 10Gb SFP ports removed):
For the Cisco module that doesnt work in any of the ports:
A DAC I was able to find (no link light when plugging both ends into the switch):
Does anyone have any idea why this is bahaving this way? The most advanced switch I've used before this was a Microtik, so I'm still attempting to wade through the CLI. I dont know if I'm missing a command or if this is a compatibility issue. Are these BiDi modules just not supposed to work? If thats the case why does it work in port 3 and both the 4x10Gb ports? Any help is appreciated.
I also have a couple HPE QSFPs (817040-B21), and they seem to give light on all ports show show up on
show media. I dont have the MPO cables to connect to them to test whether they can show active though.If I have the Proline modules plugged into 1/2/1, 1/2/2, 1/2/3 and 1/2/4 this is some of the output I get (lines referring to 10Gb SFP ports removed):
show pod
Code:
Unit-Id: 1
PoD 10G license capacity: 24
PoD 10G license capacity used: 24
PoD 40G license capacity: 24
PoD 40G license capacity used: 6
PoD-ports Lic-Available Lic-Used
1/2/1 Yes Yes
1/2/2 Yes Yes
1/2/3 Yes Yes
1/2/4 Yes Yes
1/3/1 Yes Yes
1/3/2 Yes Yes
1/3/3 Yes Yes
1/3/4 Yes Yes
1/3/5 Yes Yes
1/3/6 Yes Yes
1/3/7 Yes Yes
1/3/8 Yes Yesshow media
Code:
Port 1/2/1: Type : EMPTY
Port 1/2/2: Type : EMPTY
Port 1/2/3: Type : EMPTY
Port 1/2/4: Type : EMPTYshow media validation
Code:
Port Supported Vendor Type
----------------------------------------------------------------------
1/2/1 Yes CISCO-PROLINE 40G QSFP Module
1/2/2 Yes CISCO-PROLINE 40G QSFP Module
1/2/3 Yes CISCO-PROLINE 40G QSFP Module
1/2/4 Yes CISCO-PROLINE 40G QSFP Moduleshow media ethernet 1/2/X
Code:
Port 1/2/1: Type : 40G QSFP Module
Vendor Name: CISCO-PROLINE Serial Num: PROXXXXXXXXX Revision: D1
Port 1/2/2: Type : 40G QSFP Module
Vendor Name: CISCO-PROLINE Serial Num: PROXXXXXXXXX Revision: D1
Port 1/2/3: Type : 40G QSFP Module
Vendor Name: CISCO-PROLINE Serial Num: PROXXXXXXXXX Revision: D1
Port 1/2/4: Type : 40G QSFP Module
Vendor Name: CISCO-PROLINE Serial Num: PROXXXXXXXXX Revision: D1
Code:
Port 1/2/3: Type : EMPTY
Port Supported Vendor Type
----------------------------------------------------------------------
1/2/3 Yes CISCO-AVAGO 40G QSFP Module
Port 1/2/3: Type : 40G QSFP Module
Vendor Name: CISCO-AVAGO Serial Num: AVMXXXXXXXX Revision: 01
Code:
Port 1/2/1: Type : 40GBASE-Passive Copper
Port 1/2/2: Type : 40GBASE-Passive Copper
Port Supported Vendor Type
----------------------------------------------------------------------
1/2/1 Yes Mellanox 40G QSFP Module
1/2/2 Yes Mellanox 40G QSFP Module
Port 1/2/1: Type : 40G QSFP Module
Vendor Name: Mellanox Serial Num: 6CFXXXXXXX Revision: A1
Port 1/2/2: Type : 40G QSFP Module
Vendor Name: Mellanox Serial Num: 6CFXXXXXXX Revision: A1