Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
So I've finally got my 6450-24P running, took a lot of trial and errors and painful driver configs w/ some connectx-2 cards in windows 10. I've got one system running drivers that are version 5.1 and the other running 4.8 and I'm getting some weird results in iperf tests.
I'm using a 7M DAC from my gaming system and a 3M DAC from Fatal. I'm not really super confident in the 3M DAC being 100% good as it came in a with a connectx-2 set where one of those cards was DOA. In general getting drivers to play nice on Win 10 has been painful but it could be the dead card and bad cables. I'm open to suggestions as I think this might be on the PC side and not the switch side. I would try other ports on the 6450 but I am waiting on licenses there (I don't think it's the ports and I'm in 1 and 3 which should be 10Gbps without licenses.
Thanks in advance!
Jason
PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.87.237
Connecting to host 192.168.87.237, port 5201
[ 4] local 192.168.87.229 port 51691 connected to 192.168.87.237 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec
[ 4] 1.00-2.00 sec 428 MBytes 3.59 Gbits/sec
[ 4] 2.00-3.00 sec 470 MBytes 3.94 Gbits/sec
[ 4] 3.00-4.00 sec 488 MBytes 4.09 Gbits/sec
[ 4] 4.00-5.00 sec 452 MBytes 3.80 Gbits/sec
[ 4] 5.00-6.00 sec 291 MBytes 2.44 Gbits/sec
[ 4] 6.00-7.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 7.00-8.00 sec 468 MBytes 3.93 Gbits/sec
[ 4] 8.00-9.00 sec 414 MBytes 3.47 Gbits/sec
[ 4] 9.00-10.00 sec 460 MBytes 3.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec receiver
iperf Done.
PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.87.237, port 50911
[ 5] local 192.168.87.229 port 5201 connected to 192.168.87.237 port 50912
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 648 MBytes 5.44 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 666 MBytes 5.59 Gbits/sec
[ 5] 3.00-4.00 sec 845 MBytes 7.09 Gbits/sec
[ 5] 4.00-5.00 sec 868 MBytes 7.28 Gbits/sec
[ 5] 5.00-6.00 sec 881 MBytes 7.39 Gbits/sec
[ 5] 6.00-7.00 sec 876 MBytes 7.35 Gbits/sec
[ 5] 7.00-8.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 8.00-9.00 sec 856 MBytes 7.18 Gbits/sec
[ 5] 9.00-10.00 sec 890 MBytes 7.47 Gbits/sec
[ 5] 10.00-10.05 sec 39.2 MBytes 7.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.05 sec 7.93 GBytes 6.78 Gbits/sec receiver
Connecting to host 192.168.87.237, port 5201
[ 4] local 192.168.87.229 port 51691 connected to 192.168.87.237 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec
[ 4] 1.00-2.00 sec 428 MBytes 3.59 Gbits/sec
[ 4] 2.00-3.00 sec 470 MBytes 3.94 Gbits/sec
[ 4] 3.00-4.00 sec 488 MBytes 4.09 Gbits/sec
[ 4] 4.00-5.00 sec 452 MBytes 3.80 Gbits/sec
[ 4] 5.00-6.00 sec 291 MBytes 2.44 Gbits/sec
[ 4] 6.00-7.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 7.00-8.00 sec 468 MBytes 3.93 Gbits/sec
[ 4] 8.00-9.00 sec 414 MBytes 3.47 Gbits/sec
[ 4] 9.00-10.00 sec 460 MBytes 3.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.14 GBytes 3.56 Gbits/sec receiver
iperf Done.
PS C:\Users\Fatal\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.87.237, port 50911
[ 5] local 192.168.87.229 port 5201 connected to 192.168.87.237 port 50912
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 648 MBytes 5.44 Gbits/sec
[ 5] 1.00-2.00 sec 719 MBytes 6.03 Gbits/sec
[ 5] 2.00-3.00 sec 666 MBytes 5.59 Gbits/sec
[ 5] 3.00-4.00 sec 845 MBytes 7.09 Gbits/sec
[ 5] 4.00-5.00 sec 868 MBytes 7.28 Gbits/sec
[ 5] 5.00-6.00 sec 881 MBytes 7.39 Gbits/sec
[ 5] 6.00-7.00 sec 876 MBytes 7.35 Gbits/sec
[ 5] 7.00-8.00 sec 830 MBytes 6.96 Gbits/sec
[ 5] 8.00-9.00 sec 856 MBytes 7.18 Gbits/sec
[ 5] 9.00-10.00 sec 890 MBytes 7.47 Gbits/sec
[ 5] 10.00-10.05 sec 39.2 MBytes 7.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.05 sec 7.93 GBytes 6.78 Gbits/sec receiver
I'm using a 7M DAC from my gaming system and a 3M DAC from Fatal. I'm not really super confident in the 3M DAC being 100% good as it came in a with a connectx-2 set where one of those cards was DOA. In general getting drivers to play nice on Win 10 has been painful but it could be the dead card and bad cables. I'm open to suggestions as I think this might be on the PC side and not the switch side. I would try other ports on the 6450 but I am waiting on licenses there (I don't think it's the ports and I'm in 1 and 3 which should be 10Gbps without licenses.
Thanks in advance!
Jason
Last edited:
I think I replied to everyone's PMs from the last week but it was multiple pages and i may have missed one so if you haven't heard from me yet, bump the PM
Last edited:
Pretty common results for windows 7-10 10gb networking "out of the box".
Play with interrupts, offloading, and flow control settings in the nic drivers, both ends. Interrupts from adaptive to mid/low was the thing which was most helpful in my situation.
Are there recommendations on cables/optics? I am thinking of getting the HP QSFP nics but worry about a conversion. I was feeling that a DAC would be more stable than fiber but looks like I was wrong there. Never played with fiber before and I rent so that's been my hesitation there.
I have limited view in 10gbe hardware, but what i found is to avoid DACs and third party intel based nics. Very bad experience with supermicro and smalltree /expensive shit but macos support/ 10gbe lan cards. Best results with second hand oem intels nics - sun, cisco, ibm ...same for the sfp+ modules. Ebay's melanox rocks too. My switch is EdgeSwitch 16 XG, works flawlessly but i regret didn't go after used melanox 40/10 interfaces.
Plain intel nics are good too but works only with intel's sfps.
Alright, finally joined the select ICX6610 club (thank you fohdeesha). So here comes my first question:
The one I bought got 2 PSUs, is it useful to plug both PSUs in if:
- don't really care if a PSU fails, nothing critical if my switch turns off for a couple of minutes, I can't put it on my UPS anyway (only a 750Va). Plus here in NY, we got monthly power outages
- it seems to make more noise when both are plugged in, even if both fans slow down after boot
- will 2 PSUs not consume a lot more electricity? It's theoretically 2x1000w I am guessing?
The one I bought got 2 PSUs, is it useful to plug both PSUs in if:
- don't really care if a PSU fails, nothing critical if my switch turns off for a couple of minutes, I can't put it on my UPS anyway (only a 750Va). Plus here in NY, we got monthly power outages
- it seems to make more noise when both are plugged in, even if both fans slow down after boot
- will 2 PSUs not consume a lot more electricity? It's theoretically 2x1000w I am guessing?
there's not really a reason to plug both in, in your case. It won't consume noticeably more electricity, the switch only draws around 100w under normal load, nowhere near the full rating of the PSU. It also load shares when you plug in both PSU's, so each PSU would supply 50 watts. You're incurring the 5% or whatever efficiency loss of the second supply, so add maybe 5 or 10 watts more, but nothing crazy
Running both PSUs will increase your power bill slightly and will increase noise due to fans from both PSUs contributing noise but will not double your power consumption. Only one PSU is supplying power at any time although the standby PSU consumes some power while it remains in standby. An explanation of power supply efficiency issues across loads is here: Power Supply Buyer's Guide: Efficiency. The tldr; for that article is that your total power draw is the active power draw by your switch and the overhead for the power supply delivering the power to your switch. Simplistically, if your power supply was 50% efficient (which would be terrible but could occur at very low power consumption levels), your supply would consume 100W if your switch drew 50W. The amount drawn is independent of the maximum power that your PSU can supply (1000W in your example).
You may see 10-20% higher power if the second PSU is connected - my estimate based on dual-PSU situations not related to this switch.
Given your insensitivity to short outages, I would unplug the 2nd PSU but keep it handy if the first one goes bad (unlikely but it could happen); i.e., a "cold spare"
Hope that helps - good luck
WW
album of the brocade powered 2k19 lan: Imgur
Took the "kill a watt" out today to put some numbers behind fohdeesha and WeekendWarrior statements :
Note that I have disconnected everything to make this test and all POE ports are turned off.
PSU1: 86w at boot and 127.9 after initialization. Stays at 127.8-127.9 for 6hours
PSU2:86w at boot and 127.6 after initialization. Stays at 127.5-127.6 for 6hours
PSU1+2: 125w at boot and 137.2 after initialization. Stays at 137.2 for 6hours
So that's a 7.43% increase with 2 PSUs
Note that I have disconnected everything to make this test and all POE ports are turned off.
PSU1: 86w at boot and 127.9 after initialization. Stays at 127.8-127.9 for 6hours
PSU2:86w at boot and 127.6 after initialization. Stays at 127.5-127.6 for 6hours
PSU1+2: 125w at boot and 137.2 after initialization. Stays at 137.2 for 6hours
So that's a 7.43% increase with 2 PSUs
Hi Everyone,
I have 2 "RPS16-E 1,000 W power supply for Brocade ICX 6610 PoE models, power-supply-side exhaust airflow REV B" PSU's
I am looking to trade them for
2 "RPS16-I 1,000 W power supply for Brocade ICX 6610 PoE models, power-supply-side intake airflow " PSU's
Please send me a PM if your interested.
Have a Great Weekend!
I have 2 "RPS16-E 1,000 W power supply for Brocade ICX 6610 PoE models, power-supply-side exhaust airflow REV B" PSU's
I am looking to trade them for
2 "RPS16-I 1,000 W power supply for Brocade ICX 6610 PoE models, power-supply-side intake airflow " PSU's
Please send me a PM if your interested.
Have a Great Weekend!
Last edited:
Wow, I just want to say thank you to all who have contributed to this thread. What a great community - I've been messing around with stuff like this at home for years, but for some reason never really stumbled onto STH forums! This thread has made me a believer...
I'm really excited since I'm moving into a new house in a few weeks and it has some CAT5e (hey, beggars can't be choosers; most homes have zilch for structured wiring!)... So I enthusiastically read every single post up to this point, and somewhere around page 50 I couldn't take it any longer and purchased two 6450-24P's on eBay @$175/ea. (shipped, USA).
...Unfortunately, that was about 20 or 30 pages before people started posting reports of busted switches from eBay seller "g-electronic," which of course is who I purchased mine from. They haven't arrived yet, but when they do it looks like I'll be putting them through a VERY intense round of testing and burn-in, especially the PoE stuff. Ugh. I'll be sure to report back on their condition once they arrive.
Thanks again to all, especially @fohdeesha!
EDIT: Forgot to ask, does anyone know if a 6450 and a 6430 can be stacked provided there are no L3 features enabled on the 6450? I'm guessing the answer is no, but I am curious. Thanks.
I'm really excited since I'm moving into a new house in a few weeks and it has some CAT5e (hey, beggars can't be choosers; most homes have zilch for structured wiring!)... So I enthusiastically read every single post up to this point, and somewhere around page 50 I couldn't take it any longer and purchased two 6450-24P's on eBay @$175/ea. (shipped, USA).
...Unfortunately, that was about 20 or 30 pages before people started posting reports of busted switches from eBay seller "g-electronic," which of course is who I purchased mine from. They haven't arrived yet, but when they do it looks like I'll be putting them through a VERY intense round of testing and burn-in, especially the PoE stuff. Ugh. I'll be sure to report back on their condition once they arrive.
Thanks again to all, especially @fohdeesha!
EDIT: Forgot to ask, does anyone know if a 6450 and a 6430 can be stacked provided there are no L3 features enabled on the 6450? I'm guessing the answer is no, but I am curious. Thanks.
Last edited:
About g-electronics.
I think the initial post from Hakujou was offset with the same experience with another seller's 6610. rgunner obtained a full refund. Most likely they obtained some bad switches that had damaged PoE modules.
My 6610-48P came from them and I have to state that it works fine for the lab. All ports work with PoE. All SFP+/QSFP+ ports work with the Mellanox DACs I have access to. The only issue is one of the link LEDs doesn't work. Prior to buying I asked them what PSU versions they had on hand and whether they had ears because their photos didn't show any. They allocated one with Rev B PSU along with with ears. Their customer support is pretty awesome, considering what rgunner experienced.
I think the initial post from Hakujou was offset with the same experience with another seller's 6610. rgunner obtained a full refund. Most likely they obtained some bad switches that had damaged PoE modules.
My 6610-48P came from them and I have to state that it works fine for the lab. All ports work with PoE. All SFP+/QSFP+ ports work with the Mellanox DACs I have access to. The only issue is one of the link LEDs doesn't work. Prior to buying I asked them what PSU versions they had on hand and whether they had ears because their photos didn't show any. They allocated one with Rev B PSU along with with ears. Their customer support is pretty awesome, considering what rgunner experienced.