Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

cuco

Member
Feb 13, 2018
51
2
8
34
So i made my deal. 250$ for the 6610-24p-e. Plus shipping and taxes to germany 400$.
Now i need some good transceivers.
I picked this cables so i need LR transceivers. right? what brocade transceiver would you recommend ?
 

Hakujou

New Member
Apr 28, 2019
19
6
3
So i made my deal. 250$ for the 6610-24p-e. Plus shipping and taxes to germany 400$.
Now i need some good transceivers.
I picked this cables so i need LR transceivers. right? what brocade transceiver would you recommend ?
I buy them from fs.com
 

cuco

Member
Feb 13, 2018
51
2
8
34
ok so when optical monitoring with link strength and module temp work, i go with fs.com transceivers.
 

wings2004

New Member
May 16, 2019
2
0
1
So I've finally got my 6450-24P running, took a lot of trial and errors and painful driver configs w/ some connectx-2 cards in windows 10. I've got one system running drivers that are version 5.1 and the other running 4.8 and I'm getting some weird results in iperf tests.

PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.87.237
Connecting to host 192.168.87.237, port 5201
[ 4] local 192.168.87.229 port 51691 connected to 192.168.87.237 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec
[ 4] 1.00-2.00 sec 428 MBytes 3.59 Gbits/sec
[ 4] 2.00-3.00 sec 470 MBytes 3.94 Gbits/sec
[ 4] 3.00-4.00 sec 488 MBytes 4.09 Gbits/sec
[ 4] 4.00-5.00 sec 452 MBytes 3.80 Gbits/sec
[ 4] 5.00-6.00 sec 291 MBytes 2.44 Gbits/sec
[ 4] 6.00-7.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 7.00-8.00 sec 468 MBytes 3.93 Gbits/sec
[ 4] 8.00-9.00 sec 414 MBytes 3.47 Gbits/sec
[ 4] 9.00-10.00 sec 460 MBytes 3.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec receiver
iperf Done.
PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.87.237, port 50911
[ 5] local 192.168.87.229 port 5201 connected to 192.168.87.237 port 50912
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 648 MBytes 5.44 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 666 MBytes 5.59 Gbits/sec
[ 5] 3.00-4.00 sec 845 MBytes 7.09 Gbits/sec
[ 5] 4.00-5.00 sec 868 MBytes 7.28 Gbits/sec
[ 5] 5.00-6.00 sec 881 MBytes 7.39 Gbits/sec
[ 5] 6.00-7.00 sec 876 MBytes 7.35 Gbits/sec
[ 5] 7.00-8.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 8.00-9.00 sec 856 MBytes 7.18 Gbits/sec
[ 5] 9.00-10.00 sec 890 MBytes 7.47 Gbits/sec
[ 5] 10.00-10.05 sec 39.2 MBytes 7.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.05 sec 7.93 GBytes 6.78 Gbits/sec receiver

I'm using a 7M DAC from my gaming system and a 3M DAC from Fatal. I'm not really super confident in the 3M DAC being 100% good as it came in a with a connectx-2 set where one of those cards was DOA. In general getting drivers to play nice on Win 10 has been painful but it could be the dead card and bad cables. I'm open to suggestions as I think this might be on the PC side and not the switch side. I would try other ports on the 6450 but I am waiting on licenses there (I don't think it's the ports and I'm in 1 and 3 which should be 10Gbps without licenses.

Thanks in advance!

Jason
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,907
1,718
113
29
fohdeesha.com
I think I replied to everyone's PMs from the last week but it was multiple pages and i may have missed one so if you haven't heard from me yet, bump the PM
 
Last edited:

DedoBOT

Member
Dec 24, 2018
38
5
8
So I've finally got my 6450-24P running, took a lot of trial and errors and painful driver configs w/ some connectx-2 cards in windows 10. I've got one system running drivers that are version 5.1 and the other running 4.8 and I'm getting some weird results in iperf tests.

PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.87.237
Connecting to host 192.168.87.237, port 5201
[ 4] local 192.168.87.229 port 51691 connected to 192.168.87.237 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec
[ 4] 1.00-2.00 sec 428 MBytes 3.59 Gbits/sec
[ 4] 2.00-3.00 sec 470 MBytes 3.94 Gbits/sec
[ 4] 3.00-4.00 sec 488 MBytes 4.09 Gbits/sec
[ 4] 4.00-5.00 sec 452 MBytes 3.80 Gbits/sec
[ 4] 5.00-6.00 sec 291 MBytes 2.44 Gbits/sec
[ 4] 6.00-7.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 7.00-8.00 sec 468 MBytes 3.93 Gbits/sec
[ 4] 8.00-9.00 sec 414 MBytes 3.47 Gbits/sec
[ 4] 9.00-10.00 sec 460 MBytes 3.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec receiver
iperf Done.
PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.87.237, port 50911
[ 5] local 192.168.87.229 port 5201 connected to 192.168.87.237 port 50912
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 648 MBytes 5.44 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 666 MBytes 5.59 Gbits/sec
[ 5] 3.00-4.00 sec 845 MBytes 7.09 Gbits/sec
[ 5] 4.00-5.00 sec 868 MBytes 7.28 Gbits/sec
[ 5] 5.00-6.00 sec 881 MBytes 7.39 Gbits/sec
[ 5] 6.00-7.00 sec 876 MBytes 7.35 Gbits/sec
[ 5] 7.00-8.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 8.00-9.00 sec 856 MBytes 7.18 Gbits/sec
[ 5] 9.00-10.00 sec 890 MBytes 7.47 Gbits/sec
[ 5] 10.00-10.05 sec 39.2 MBytes 7.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.05 sec 7.93 GBytes 6.78 Gbits/sec receiver

I'm using a 7M DAC from my gaming system and a 3M DAC from Fatal. I'm not really super confident in the 3M DAC being 100% good as it came in a with a connectx-2 set where one of those cards was DOA. In general getting drivers to play nice on Win 10 has been painful but it could be the dead card and bad cables. I'm open to suggestions as I think this might be on the PC side and not the switch side. I would try other ports on the 6450 but I am waiting on licenses there (I don't think it's the ports and I'm in 1 and 3 which should be 10Gbps without licenses.

Thanks in advance!

Jason
Pretty common results for windows 7-10 10gb networking "out of the box".
Play with interrupts, offloading, and flow control settings in the nic drivers, both ends. Interrupts from adaptive to mid/low was the thing which was most helpful in my situation.
 

wings2004

New Member
May 16, 2019
2
0
1
Are there recommendations on cables/optics? I am thinking of getting the HP QSFP nics but worry about a conversion. I was feeling that a DAC would be more stable than fiber but looks like I was wrong there. Never played with fiber before and I rent so that's been my hesitation there.
 

DedoBOT

Member
Dec 24, 2018
38
5
8
Are there recommendations on cables/optics? I am thinking of getting the HP QSFP nics but worry about a conversion. I was feeling that a DAC would be more stable than fiber but looks like I was wrong there. Never played with fiber before and I rent so that's been my hesitation there.
I have limited view in 10gbe hardware, but what i found is to avoid DACs and third party intel based nics. Very bad experience with supermicro and smalltree /expensive shit but macos support/ 10gbe lan cards. Best results with second hand oem intels nics - sun, cisco, ibm ...same for the sfp+ modules. Ebay's melanox rocks too. My switch is EdgeSwitch 16 XG, works flawlessly but i regret didn't go after used melanox 40/10 interfaces.
Plain intel nics are good too but works only with intel's sfps.
 

Jenfil82

New Member
May 9, 2019
18
4
3
Alright, finally joined the select ICX6610 club (thank you fohdeesha). So here comes my first question:

The one I bought got 2 PSUs, is it useful to plug both PSUs in if:

- don't really care if a PSU fails, nothing critical if my switch turns off for a couple of minutes, I can't put it on my UPS anyway (only a 750Va). Plus here in NY, we got monthly power outages
- it seems to make more noise when both are plugged in, even if both fans slow down after boot
- will 2 PSUs not consume a lot more electricity? It's theoretically 2x1000w I am guessing?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,907
1,718
113
29
fohdeesha.com
there's not really a reason to plug both in, in your case. It won't consume noticeably more electricity, the switch only draws around 100w under normal load, nowhere near the full rating of the PSU. It also load shares when you plug in both PSU's, so each PSU would supply 50 watts. You're incurring the 5% or whatever efficiency loss of the second supply, so add maybe 5 or 10 watts more, but nothing crazy
 

WeekendWarrior

Active Member
Apr 2, 2015
299
96
28
53
Alright, finally joined the select ICX6610 club (thank you fohdeesha). So here comes my first question:

The one I bought got 2 PSUs, is it useful to plug both PSUs in if:

- don't really care if a PSU fails, nothing critical if my switch turns off for a couple of minutes, I can't put it on my UPS anyway (only a 750Va). Plus here in NY, we got monthly power outages
- it seems to make more noise when both are plugged in, even if both fans slow down after boot
- will 2 PSUs not consume a lot more electricity? It's theoretically 2x1000w I am guessing?
Running both PSUs will increase your power bill slightly and will increase noise due to fans from both PSUs contributing noise but will not double your power consumption. Only one PSU is supplying power at any time although the standby PSU consumes some power while it remains in standby. An explanation of power supply efficiency issues across loads is here: Power Supply Buyer's Guide: Efficiency. The tldr; for that article is that your total power draw is the active power draw by your switch and the overhead for the power supply delivering the power to your switch. Simplistically, if your power supply was 50% efficient (which would be terrible but could occur at very low power consumption levels), your supply would consume 100W if your switch drew 50W. The amount drawn is independent of the maximum power that your PSU can supply (1000W in your example).

You may see 10-20% higher power if the second PSU is connected - my estimate based on dual-PSU situations not related to this switch.

Given your insensitivity to short outages, I would unplug the 2nd PSU but keep it handy if the first one goes bad (unlikely but it could happen); i.e., a "cold spare"

Hope that helps - good luck
WW
 

Jenfil82

New Member
May 9, 2019
18
4
3
Took the "kill a watt" out today to put some numbers behind fohdeesha and WeekendWarrior statements :

Note that I have disconnected everything to make this test and all POE ports are turned off.

PSU1: 86w at boot and 127.9 after initialization. Stays at 127.8-127.9 for 6hours
PSU2:86w at boot and 127.6 after initialization. Stays at 127.5-127.6 for 6hours

PSU1+2: 125w at boot and 137.2 after initialization. Stays at 137.2 for 6hours

So that's a 7.43% increase with 2 PSUs
 

WeekendWarrior

Active Member
Apr 2, 2015
299
96
28
53
Took the "kill a watt" out today to put some numbers behind fohdeesha and WeekendWarrior statements :

Note that I have disconnected everything to make this test and all POE ports are turned off.

PSU1: 86w at boot and 127.9 after initialization. Stays at 127.8-127.9 for 6hours
PSU2:86w at boot and 127.6 after initialization. Stays at 127.5-127.6 for 6hours

PSU1+2: 125w at boot and 137.2 after initialization. Stays at 137.2 for 6hours

So that's a 7.43% increase with 2 PSUs
Thanks for sharing the data - data always carries the day
 

Thorzeen

New Member
Jun 15, 2018
17
7
3
Hi Everyone,

I have 2 "RPS16-E 1,000 W power supply for Brocade ICX 6610 PoE models, power-supply-side exhaust airflow REV B" PSU's

I am looking to trade them for

2 "RPS16-I 1,000 W power supply for Brocade ICX 6610 PoE models, power-supply-side intake airflow " PSU's

Please send me a PM if your interested.

Have a Great Weekend!
 
Last edited: