Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[maes]

and some rack ears...

I'll check tomorrow if the generic set fits and let you know asap if you're interested.

 

[Dave Corder]

I'll check tomorrow if the generic set fits and let you know asap if you're interested.

Much appreciated. No rush, though...this switch is going to be the core switch for the next house anyway (planning to move early next year).

 

[PnoT]

A little bit of an update after my last switch came in and the POE board was bad. The seller shipped me another one...

It seems that a switch curse has been put on me =/

 

[fohdeesha]

Got my 6610 all flashed to the latest and greatest thanks to fohdeesha's guide, but it turns out one of the PSU fans is broken in some way:

sw-core-03(config)#show chassis
The stack unit 1 chassis info:

Power supply 1 present, status failed
Power Supply 1 Fan has failed
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  64K
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Front to Back

Waiting to see what the seller can do for me. I'm open to a partial refund so I can but a working fan or PSU and some rack ears...

maybe obvious, but just in case: have you plugged that PSU into the wall? if it has no power, it'll say exactly that

 

[Dave Corder]

maybe obvious, but just in case: have you plugged that PSU into the wall? if it has no power, it'll say exactly that

It's been a long couple of days at work troubleshooting database performance gremlins, so I'll gladly take obvious suggestions at this point.

And, as it turns out, the great fohdessha is correct...I didn't have the power cord seated firmly in that PSU. The switch is happy now:

SSH@sw-core-03>show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
    Model Number:    23-0000142-02
    Serial Number:    CC4
    Firmware Ver:     A
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
    Model Number:    23-0000142-02
    Serial Number:    64K
    Firmware Ver:     A
Power supply 2 Fan Air Flow Direction:  Front to Back

 

[mattaw]

A little bit of an update after my last switch came in and the POE board was bad. The seller shipped me another one..

I'm so sorry. Do you have the original with failed poe or did you ship it back? Swapping the Poe board takes about 20 mins if you still have it ...

 

[fohdeesha]

@PnoT jeez you have not had good luck with these things! I believe via pm he did indeed get the PoE board swapped over

 

[u238]

Can anyone give me any insight on multicast? I have an experimental proxmox cluster that relies on multicast, and it seems to work fine in the default vlan. On a new vlan I created, multicast didn't work at all until I enabled multicast active and multicast version 3. Am I doing this right or should I be choosing other settings?

 

[mattaw]

Racked my icx6610, disabled dual mode and moved to vlans only. Benchmarked by ZFS array from my Windows 10 desktop via 40gbps ethernet. This is cache testing only, btw, not spinning disk performance.

None too shabby. Next phases are kernel buffer tuning, IRQ binding and windows 10 buffer tuning to get those numbers up. I hope my PCIe 2.0 slots don't damage the throughput though.

 

[mattaw]

Basic networking irq affinity set and tcp tuning. 1.75GBs write and 2.65GBs read! Biggest improvement was banning numa node 0 of interrupts, and then pinning the mlx4 irqs to node 0. Leave irqbalance running for the rest of the system.

 

[maes]

Hm, after testing, the Noctua NF-A4x20 runs so slow at the 'low speed' setting that the 6450-24 throws a 'fan failed' error.
The 2nd fan header RPM doesn't seem to be monitored at all but its driving voltage varies in sync with the 1st fan header.

Even hardwired for 12V, it's still slow enough to throw fan failed errors. Any quiet fan recommendations that don't throw that error message, or can it be casually ignored?

 

[lambda]

Hm, after testing, the Noctua NF-A4x20 runs so slow at the 'low speed' setting that the 6450-24 throws a 'fan failed' error.
The 2nd fan header RPM doesn't seem to be monitored at all but its driving voltage varies in sync with the 1st fan header.

Even hardwired for 12V, it's still slow enough to throw fan failed errors. Any quiet fan recommendations that don't throw that error message, or can it be casually ignored?

I just ordered two of these for my ICX6450-24P and still waiting them to arrive:

SUNON 4020 KDE1204PKVX MS.AF.GN FAN DC12V 1.4W 3Pin Super Silence | eBay

https://www.alliedelec.com/m/d/efd14bdd843dd6f08d20002435c34e64.pdf

 

[PnoT]

Be lucky you don't have an A PSU on the 6610!

I'm looking into converting the Air Flow of the fan around and it looks pretty simple except for the 4 rubber fastening tubes. Testing a few methods out right now on how to get those back in after flipping the fan around.

 

[mattaw]

@fohdeesha - tried lifting a heatsink, and got no-where. Even though they have pins, the heatsinks are glued down with probably the same glue as the smaller ones used throughout the design.

I have not tried freezing them off, not sure whether to go that far with a working switch.

 

[fohdeesha]

Hm, after testing, the Noctua NF-A4x20 runs so slow at the 'low speed' setting that the 6450-24 throws a 'fan failed' error.
The 2nd fan header RPM doesn't seem to be monitored at all but its driving voltage varies in sync with the 1st fan header.

Even hardwired for 12V, it's still slow enough to throw fan failed errors. Any quiet fan recommendations that don't throw that error message, or can it be casually ignored?

it can be casually ignored, it's just not seeing the minimum RPM it expects from the stock fans

 

[fohdeesha]

@fohdeesha - tried lifting a heatsink, and got no-where. Even though they have pins, the heatsinks are glued down with probably the same glue as the smaller ones used throughout the design.

I have not tried freezing them off, not sure whether to go that far with a working switch.

no worries at all, it's not very important. thanks for trying though! I agree, not worth it on a working switch

 

[maes]

It would take me some time to develop it since I haven't coded in just about a decade (I'm beyond rusty), but it just came to mind that with a voltage regulator, a voltage divider, an attiny4/5/9/10 and maybe a transistor, it should be possible to make a 'tachometer adjuster' to fool a switch into thinking it has a fan running at the speed it expects.

Even the tiny SOT23 6-pin package should be enough;

fan power goes to the voltage regulator, dropped to 3.3V to power the attiny
fan power also goes to the voltage divider, the output of which goes to an ADC pin. This lets the attiny know when the ICX switch expects the fan to be running at full speed (85oorpm)
fan tach wire goes to an input on the attiny
ouput on the attiny goes to the fan header tach pin

(overall pins used: VCC, GND, ADC, Tach in, Tach out.)

On boot, when the fan is running at full speed (attiny sees 'higher' voltage on ADC pin)

short delay for fan to reach full speed
use 16-bit timer to count 'ticks' between two tach pulses from the fan
repeat 7x, average result (power-of-2 samples makes it easy, just bitshift instead of dividing)
this gives the microcontroller the 'reference' full speed (or rotation period, to be more accurate)
from there, calculate the difference between the 'reference' full speed and the 'expected' full speed. This multiplier is important.​

After boot and initialization:

Maybe once a second, repeat count of 'ticks' between tach pulses from the fan
Use said count and the multiplier obtained during the initialization phase to generate fake 'higher rpm' tach pulse train
output to router tach sensor​

This would still allow the switch to throw an alert if the fan actually stops. It's not a tach offset but a multiplier.

 

[lambda]

RoCE "technically" requires switches to support PFC (priority flow control) to run over them, and the ICX line does not support pfc as they are the "campus" product line. For RoCE feature support you'd need the data center line, which is the VDX series (which I really don't like).

PFC support on the switch ensures the Ethernet transport layer is lossless - eg no dropped frames, which is how RoCE guaruntees it's speed/very low latency. That said, RoCE has its own error correction and retransmission faculties, it will run just fine over a "lossy" network (eg a switch without PFC support), it won't corrupt your data or anything malicious like that. It will just impact performance.

On a non-oversubbed switch with only a few clients, how often will you get dropped frames and therefore how much will it kill RoCE performance? I haven't the slightest clue, I've been wondering that myself. I've seen tests showing plain RoCE (eg the type intended for lossless networks only) working plenty fast without PFC in smaller networks, but I've also heard from engineers "don't ever do that, it'll be slower than non-rdma protocols" so I really have no clue.

Maybe someone here with an existing pfc-enabled switch and RoCE clients and some spare time can run a quick benchmark, then disable PFC on the switch and run it again. I honestly can't even begin to guess the performance impact, I would assume the biggest impact would be to latency, and you might get dropped frames that need to be recovered (costly time wise) just from one nic outpacing the switch or vice versa. What I do know is many people will tell you "just don't do it without PFC support"

The performance test done by Brocade indicates that the ICX6610-48P is capable of sustaining 100% load with all the 10g ports fully saturated without dropping any frame. Combining the test result from Brocade and @fohdecsha's actual observation, I think it is pretty save to assume that ICX6610-48p is capable of handling RoCE v1 at least in a non-mission critical setting. The test result is on page 9 of the following Brocade document kindly provided by @fohdecsha.

Architecture Brief

 

[Scrubs]

I've been following this thread for awhile now and I was finally able to sell off some of my old networking gear and submitted some offers on some Brocade gear today. So far I've scored an ICX 6450-48 and a 6430-48p (needed a POE switch, no need for giant uplinks) for $220 shipped. I've got an offer out on a 6610-24 for around $110. Planning on using the 6610 for my TOR and to play with SDN and the rest for port distribution (my homelab is a little excessive).

I just wanted to say thanks to all the people that contributed to this thread. I have/had a mismash of Cisco, Dell, and Ubiquiti gear. I'm looking forward to being on a single network stack (well, plus my CRS317). It makes it especially easy when STH has a great community surrounding it. Thanks everyone!

 

[Jannis Jacobsen]

Just got an icx6610-24 for $150 on buy it now on ebay.
only 1 psu and 1 fan module.
Hopefully the noise will be bearable...

Anyone got a 250w psu and fan module for sale?
Is the correct psu model rps15-e, or are there other models too?

-jannis