Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
You would almost certainly need to use the special type of SFP+ transceiver that negotiates 10G to the switch and then does speed limiting itself. You want an Aquantia based transceiver if you want that to work properly. :) I highly doubt these old switches have native support for anything other than 1G and 10G.
Thats why I am asking, I cant get a clear answer anywhere. I figured it depends on the switch and the SFP, but other corners of the internet say its only the SFP that determins if it will work or not.

Not sure if what I have is Aquantia based but so far every SFP I have tried has worked on my switches.

I think my Aurba switches were the only ones I hunted down Finistar or something for.
 

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
Well, fohdeesha definitely doesn't say the following in the original post :)
I am running 10gb copper now, and back then when we were talking about these regularly that was the first 10gb copper, multi gig was not really a thing yet. I'll personally test it soon and let everyone know.

I have a RJ45 SPF that I know works with Brocade as my desktop is using it now on 10gb copper, and I get another tomorrow and my R730 has a 5gb port.

So can see what happens.
 
Last edited:

blunden

Active Member
Nov 29, 2019
742
243
43
Thats why I am asking, I cant get a clear answer anywhere. I figured it depends on the switch and the SFP, but other corners of the internet say its only the SFP that determins if it will work or not.

Not sure if what I have is Aquantia based but so far every SFP I have tried has worked on my switches.

I think my Aurba switches were the only ones I hunted down Finistar or something for.
Yes, it depends on the switch as well. Older 10 Gig equipment basically never support those newer lower speeds, which is why those special SFP+ modules exist.

The reason you'll want an Aquantia based SFP+ module is because their chips are the only ones that handle the rate limiting properly according to others on this forum. It's not because of vendor locking or anything like that.

If I recall correctly, the cheapest module with such a chip is the 10Gtek ASF-10G2-T (note the "2" in there). I'd double check that by searching this forum though. :) It uses PAUSE frames to limit the speed so you need to have Flow Control enabled on the switch.
 
Last edited:

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
Yes, it depends on the switch as well. Older 10 Gig equipment basically never support those newer lower speeds, which is why those special SFP+ modules exist.

The reason you'll want an Aquantia based SFP+ module is because their chips are the only ones that handle the rate limiting properly according to others on this forum. It's not because of vendor locking or anything like that.

If I recall correctly, the cheapest module with such a chip is the 10Gtek ASF-10G2-T (note the "2" in there). I'd double check that by searching this forum though. :) It uses PAUSE frames to limit the speed to you need to have Flow Control enabled on the switch.
Just grabbed the last one I saw on Amazon for testing, since I already ordered a multi gig switch I dont need this to work, but if it works I can just use the Brocade by itself but would need to upgrade to the 7250 for the extra 10gb ports. Still cheaper than adding a 2nd switch though-ish.

But will also be good to see if my cheaper SFP works.
 
  • Like
Reactions: blunden

blademan

New Member
Jan 7, 2022
7
2
3
Looking at the latest release notes (p.33):
Known Issues in 08.0.95m. Issue FI-279139: Symptom 4x10GE Active Copper 1m optics is not supported.
  1. 4 x 10Gb DAC 1m are not supported in 8.0.95m
  2. 4 x 10Gb DAC 1m are not supported in all releases prior to 8.0.95m?
  3. Does that also mean >4 x 10Gb 1m DAC are not supported?
  4. Are 4 x 10Gb DAC >1m supported?
 

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
In the name of science I bought like 5 different SFP's the first one got here today.

This was one of the cheaper ones:

Really nicely boxed in a metal tin, overnight shipping

I used my Ruckus R730 (Flashed to R850) as my test subject.

The AP detected its full 5gb backplane connection:
5gb on R730.png

The switch itself still saw the connection as 10gb:
ICX6450.png

So it very much looks like there are ways to use these with Cheap SFP and get multi gig support.

My oncly concern would be the long ethernet run to my furthest AP in the house, I am not sure how these SFP hold up and deal with longer distance runs.

I'll post back with the test for the other SFP soon.
 
  • Like
Reactions: Jason Antes

ManoftheSea

Member
Apr 18, 2023
41
16
8
Sorry to interrupt the current discussion, but I'd like a sanity check: When setting up port monitoring, how are you supposed to turn off the monitor?

I've got an ICX6450 on 08.0.30. I configure e.g. 1/1/23 to be a mirror, I configure 1/1/48 to be monitored. When I issue "no mirror-port 1/1/23" or "no monitor both", the traffic doesn't stop. Am I missing a command, is this a known issue? I have to reboot the router to stop the mirroring.
 

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
I also have a 2nd question too lol.
I am about to go ahead and buy a ICX 7250-48 to expand my 10gb capabilities

I have the ICX 6610 but its a bit too much for 24/7 use I want to keep noise and power down as much as possible.

So I am using the ICX 6450 and its good and all, I was about to add a 2nd smaller switch with multi-gig ethernet to my 6450, but now that I have tested and confirmed I can do that with SFP I'll just upgrade the switch.

My question is this, the PoE vs Non-PoE models.
Our first post does not make any mention of additional power/heat from the PoE model and I just wanted to confirm there is no difference.

Most PoE switches tend to draw additional power and produce a bit more heat, and it seems everything I need PoE for these days is PoE++ so these switches are not able to do that, and thus I never use the PoE.

There was also an issue with SFP's in a switch I was looking at recently where they worked on the PoE model but not the non PoE model, though that is probably entirelly related to that switch and not the norm.

When hunting down switches on ebay if I see a good looking PoE model cheaper this makes the difference in buying or waiting for something else.


TLDR:
I am looking for an ICX 7250 and some of the PoE versions are cheaper or better looking than the standard non-poe model, but if the PoE model uses more power and generates more heat/noise than the standard version I would pay a bit extra to get the NON-PoE version as I have no current benefits to having PoE on my switch unless its PoE++
 
Last edited:

blunden

Active Member
Nov 29, 2019
742
243
43
In the name of science I bought like 5 different SFP's the first one got here today.

So it very much looks like there are ways to use these with Cheap SFP and get multi gig support.

My oncly concern would be the long ethernet run to my furthest AP in the house, I am not sure how these SFP hold up and deal with longer distance runs.
There are several 10GBase-T transceivers that will appear to work in a single direction at these NBASE-T speeds but won't handle bidirectional traffic correctly. That's what sets the Aquantia based ones apart as far as I remember. :)

Most 10GBase-T transceivers are rated for a maximum of 30 meters. There are also a few relatively new ones that are rated for 80 and 100 meters, but they generally cost significantly more since they need to use more advanced process nodes to keep the power draw down enough to make the longer range possible. If you need one of those 100 meter ones, it appears as though Ubiquiti has the cheapest one by far, at roughly half the price of the module on FS.com.
 

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
There are several 10GBase-T transceivers that will appear to work in a single direction at these NBASE-T speeds but won't handle bidirectional traffic correctly. That's what sets the Aquantia based ones apart as far as I remember. :)

Most 10GBase-T transceivers are rated for a maximum of 30 meters. There are also a few relatively new ones that are rated for 80 and 100 meters, but they generally cost significantly more since they need to use more advanced process nodes to keep the power draw down enough to make the longer range possible. If you need one of those 100 meter ones, it appears as though Ubiquiti has the cheapest one by far, at roughly half the price of the module on FS.com.
What do you mean bi-directonal?
The test from earlier was bi-directional had my phone and tablet hooked to the AP and did a speed test between them.
 

blunden

Active Member
Nov 29, 2019
742
243
43
What do you mean bi-directonal?
The test from earlier was bi-directional had my phone and tablet hooked to the AP and did a speed test between them.
Sending data in both directions at the same time. You can use iperf3 with the --bidir flag.

 

ViciousXUSMC

Active Member
Nov 27, 2016
277
147
43
42
Sending data in both directions at the same time. You can use iperf3 with the --bidir flag.

Ok let me test later, it was actually iPerf that I had used for my speed test.

To me two devices on my AP doing a transfer would be bi-directional, but never have I used that flag and it might be a good thing to add to my future testing.

Edit: seems that bidir is not in the version I have on my devices.
I was running 10 paralell streams though.


Have another way to test this and see if its actually an issue? So far all real world testing is fine.
Also this makes me excited to try the new iperf sometime lol.

I can get the new version on a VM on my hypervisor, but that means not able to use my phone/tablet and other devices to be the other side (good for testing wireless and seeing the generational difference in different wireless technologies)
 
Last edited:

blunden

Active Member
Nov 29, 2019
742
243
43

jode

Member
Jul 27, 2021
65
57
18
Took a little while, but this works.
I want to provide a quick update on progress of my effort of enabling automated testing and deployment to ICX switches.

At this point I have created scripts that test accessibility against an increasing number of end points on my home network and report success/fail for each VLAN. This provides me an objective measure of security and I can compare that to my (VLAN design) goals.
I keep these in my git repo and run them via a CI/CD pipeline that will provide feedback on any change I implement on the switch. At this point each change triggers >250 tests with more expected to come online, soon. I already found and addressed a bunch of issues identified that fall into categories from service misconfiguration to switch/router/network misconfiguration.

I plan on breaking up the content of a switch configuration by implementation goals allowing me to easier remember what parts of the switch configuration were enabled. Also, I plan on adding some basic performance tests to the framework, allowing me to programmatically identify misconfigurations.

While this sort of quality assessment is common in programming I have not (yet) seen open source tools or capabilities applied to switch/network configurations. Well, I am sure enterprise class and -priced tools exist, but these don't apply to my home network ;)

Let me know if I missed anything or if you're interested in me sharing more about this.
 
  • Like
Reactions: r0adawg and pcmoore

ManoftheSea

Member
Apr 18, 2023
41
16
8
I want to provide a quick update on progress of my effort of enabling automated testing and deployment to ICX switches.
Very cool. I would think that OpenTofu would be a tool to deploy "parts of configuration", though there's no provider that I know of to do it to ICX Switches. The first thing that springs to my mind would be being able to deploy firewall rules (access control lists).

Or are you describing something more like Nessus, where you've got sensors that are trying to connect to endpoints?

Shared anywhere on github/gitlab?
 

jode

Member
Jul 27, 2021
65
57
18
Or are you describing something more like Nessus, where you've got sensors that are trying to connect to endpoints?
No. That's really cool, but not the thing I try to solve. Lemme explain.

I manage a somewhat suped-up home network, full of enthusiasm, but otherwise very not professional. It has multiple VLANs (say, one for untrustworthy IoT devices, one for management e.g. to prevent the kids hacking into firewall for circumventing their time limit, one for trusted family use). I like my privacy and deploy a bunch of (micro-)services that provide modern creature comforts without having to rely on cloud; e.g. pihole dns, file share, jellyfin, gitlab, etc. I want some services accessible from all vlans (dns, ntp, etc.), but make others only available to specific VLANs.

At this point I use the most basic way to test if a service accessible, netcat to a port (UDP/TCP depending on service). In case of a failure I want the script to collect some troubleshooting info. For example dns:
Bash:
test="$ssh_cmd nc -zw1 192.168.100.253 53 -s $local_ip"
if [ "$1" = "debug" ] ; then
  $ssh_cmd ping -4c2 192.168.100.253
  echo
  echo $test
  $($test -v)
else
  $($test)
fi
I want to run this test once for each vlan, hence it is encapsulated in its own script. Also, as I said earlier, at this point I have a couple dozen of these tests.

The general idea is to use a test client connected to an interface on my ICX switch, submit all the tests and collect the results, then change the VLAN association on the ICX interface.
I use a raspberry pi with two network interfaces. The first (eth0) to connect and submit commands via SSH ($ssh_cmd), the second (eth1) as test interface. The script above uses the following command to figure out what IP the test interface currently has.
Bash:
local_ip=$($ssh_cmd ip a show dev eth1 | grep  "inet \(.*\)/" -o | grep -o "[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*")
To change the VLAN, I not only need to assign the ICX interface to the correct VLAN, but I also need to change the NIC address on the pi. The following script does all that
Bash:
# Figure out what VLAN the interface is currently assigned to.
current_vlan=$($ssh_cmd ip a show dev eth1 | grep  "inet \(.*\)/" -o | sed 's/inet [[:digit:]]*\.[[:digit:]]*\.\([[:digit:]]*\)\.[[:digit:]]*\//\1/')
echo Current vlan is $current_vlan.

# Generate ICX script to switch interface to new VLAN
cat > vlan.script <<EOF
conf t vlan ${current_vlan}
no untagged ethernet $2
vlan $1
untagged ethernet $2
int e $2
exit
EOF

# Run above script on the ICX
./run_script.exp 192.168.10.14 vlan.script "SSH@icx6450"

# change ip on pi test interface
$ssh_cmd sudo nmcli c modify test_connection ipv4.addresses "192.168.$1.250/24"
$ssh_cmd sudo nmcli c modify test_connection ipv4.gateway "192.168.$1.1"
$ssh_cmd sudo nmcli c down test_connection
$ssh_cmd sudo nmcli c up test_connection
A parent script sets a switch port to the desired VLAN on my ICX (well, associates the interface with the correct VLAN), then executes all the tests sequentially recording the results in industry-standard Junit format.

Bash:
# script is called with ${vlan} as a parameter

# load and cycle through all test scripts in tests folder
tests=0
failures=0

testsuitestart=$(date +%s%N)
for testfile in tests/*.test ; do
   tests=$((tests+1))
   testname=$(basename "$testfile")
   echo -n "<testcase id=\"net.basic.$testfile\" name=\"VLAN${vlan} $testname\" " >> $tmpfile
   teststart=$(date +%s%N)
   if source "./$testfile" ; then
     echo "time=\"$(timespent $teststart $(date +%s%N))\"> " >>$tmpfile
   else
     echo "time=\"$(timespent $teststart $(date +%s%N))\"> " >>$tmpfile
     failures=$((failures+1))
     echo "<failure message=\"$testfile: No connectivity.\" type=\"WARNING\"> $testfile: No connectivity." >> $tmpfile
     source "./$testfile" debug >> $tmpfile 2>&1
     echo "</failure>" >> $tmpfile
   fi
   echo "</testcase>" >> $tmpfile
done

# Printing results for testsuite into resultsfile
printf "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n  <testsuites id=\"20240120_vlan\" name=\"VLAN Net Connectivity\" tests=\"$tests\" failures=\"$failures\" time=\"$(timespent $testsuitestart $(date +%s%N))\">\n" > $resultsfile
printf "    <testsuite id=\"${vlan}\" name=\"VLAN${vlan}\" tests=\"$tests\" failures=\"$failures\" time=\"$(timespent $testsuitestart $(date +%s%N))\">" >> $resultsfile
cat $tmpfile >> $resultsfile
echo "    </testsuite>" >> $resultsfile
echo "  </testsuites>" >> $resultsfile
GitLab calls the parent script as part of project pipeline, finds the generated test results and makes them available:
Screenshot from 2024-01-25 16-16-23.png

Click on Job name to review details for all tests:
Screenshot from 2024-01-25 16-17-23.png

Debug details in case of failure:
Screenshot from 2024-01-25 16-17-47.png
 
  • Like
Reactions: jdawg

ManoftheSea

Member
Apr 18, 2023
41
16
8
I manage a somewhat suped-up home network, full of enthusiasm, but otherwise very not professional. It has multiple VLANs (say, one for untrustworthy IoT devices, one for management e.g. to prevent the kids hacking into firewall for circumventing their time limit, one for trusted family use). I like my privacy and deploy a bunch of (micro-)services that provide modern creature comforts without having to rely on cloud; e.g. pihole dns, file share, jellyfin, gitlab, etc. I want some services accessible from all vlans (dns, ntp, etc.), but make others only available to specific VLANs.
Me too. "Full of enthusiasm, but not very professional".

The general idea is to use a test client connected to an interface on my ICX switch, submit all the tests and collect the results, then change the VLAN association on the ICX interface.
This seems excessively complex. Couldn't you tag all of your VLANs to a single port, and set up multiple interfaces on the probe computer? At that point, it seems like you're doing something similar to NAGIOS/Icinga with running the checks.

Instead of switching the VLAN on the untagged port and connecting from eth0, you have the script originate from eth0.200 which is a tagged connection.