Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[BobTB]

The option "1000-full-master" is the correct one, as I have tried it on my 6450 with a Cisco 1G SX, and the link went up...

It does not work for me. Is there any way to "debug" the port connectivity, does anyone know how?

 

[e97]

If you can find a way to defeat all the safty/security of this switch that prevents it from being quieter I'd love to know.

You bet. My main concern is getting the thing quiet!

Mine has a high pitched noise too, maybe it was a batch of Foxconn fans that does this... I'd swap fans on mine too.

Please, don't do it. There are many components beside the main ASIC that rely on airflow to cool themselves... power stages, port driver, PoE board if fitted... Those things are pretty well engineered, I wouldn't bother to """"optimize"""" it...

Oh, and for the power efficiency: in this case, provided that you cannot remove all the fans (otherwise the switch won't boot), and still there's a fan inside the power supply, you'll be adding power requirement for the pump

Good info. I'm aware there are other "hot" components in this pizza box IR camera is a quick way to get a heat map of what needs cooling.

Water cooling the components in a switch isnt revolutionary [WARNING: some parts may induce severe cringe ] :

Theres also immersion cooling at HPC facilities.

A modern 12V water cooling pump rarely needs to be run at full load, I run mine at 30-40% and it using <5W

I use DDC clones:


or D5 clones:



If the generic ebay blocks dont work, I can mill my own with the HAAS CNC at the makerspace a couple towns over

if you want to be halfway serious about 3 and 4 percent changes in PSU efficiency, you need to throw away your killawatt. It is not true that it "propagates the same error", if that were true the factory could just program in a static offset and they would match up with a reference grade meter. It's error varies wildly depending on the actual percent of it's full scale measurement running through it (eg 500w versus 900w), the ambient temperature, how long it's been powered up and the resistor network inside it has been allowed to warm up, etc. I've been using a reference grade power meter ( https://ctlsys.com/support/wattnode-modbus/ ) and I have caught killawatts anywhere from 5 to 15 percent off depending on environmental conditions. I would actually consider 5% pretty good for those things based on how bad I've seen them, and obviously that's not going to be of much help when measuring much smaller variances. You simply can't use them for anything other than "yeah, this thing is drawing so me rough amount of power". And while a reference grade multimeter is nice to have, and can be used to measure DC power accurately (and even then, you need two of them), they cannot be used to accurately measure AC. They'll show you apparent power, but not real power. You need a proper power meter that will actually measure and show you AC power factor

All good things to be aware of! ..but I dont think they are relevant in this case:

I'm comparing an identical system, in an identical environment with two different PSUs, after multiple measurements.

The loads measured are between:

idle: 90W-130W
load: 325W-350W

So there's no vignette error at the extremes of the measurement range. I'm comparing idle of psu1 vs idle psu2 and the same for load.

My reference is Keysight InfiniiVision oscilloscope:



which shows my Kill-a-Watt is within 1~3%.

The opensource energy meter is also 0.5% accurate - all the energy meters use the same design, non invasive current transformer and opto-coupled voltage sensor. The more expensive ones also add temp and humidity as you mentioned. The open source one includes it as an optional addon.

If you have a NIST facility or university nearby, they'll calibrate for you if you ask nicely
If you want a certificate from NIST you'll have to pay $$

Also, while I commend the effort (thinking outside the box is good and how we got a lot of the cool hacks on this forum), I think if you spend probably 20 man-hours replacing the OEM delta's with the best efficiency power supply you find, the 3 or 4 percent efficiency gain you may find, at the barely 100w of draw, is going to equal a lot of work to reduce your draw by literally 4 watts. Keep in mind the switch talks to the PSUs via PMbus (kind of a specialized version of i2c for power supplies), and reads 30 or 40 registers as well as identifying EEPROMs with serial codes etc, so if you want to replace the power supplies you'll need to engineer a microcontroller to fake all that, so perhaps 20 man-hours of work is a low estimation

Thanks for the info! I got a USB protocol analyzer and it'll be a nice goal to practice RE. If its simple, record and playback and bit bang with Pi GPIOs if its slow enough. If all else fails, I'll replace only the fans.

It looks like its SMbus, a documented protocol and theres a python library for the Pi! Fingers crossed theres nothing encrypted or proprietary..


The main goal here is silence, increasing efficiency/decreasing power consumption is an optional bonus.

 

[fohdeesha]

which shows my Kill-a-Watt is within 1~3%

Within 1-3% of what? Voltage? Current? Phase angle? The error you'll get is the product of all of these errors integrated - also note an oscilloscope with an 8-bit ADC front end is going to give a less accurate measurement for raw voltage (or current, if you're reading voltage from a CT) than a $20 multimeter off of amazon, which is also why the "DVM" function in a lot of scopes (at least ones without 10-bit or higher front ends) only reads out to 3 digits - it's just not what they are designed to do. Scopes are great, but they are the last thing on my entire workbench I would use as a reference for this (unless I was calibrating the phase angle error for a specific CT or something, which most of the higher end energy meters will let you program in, if you can manage to measure it)

if the opensource energy meter you speak of claims 0.5% base accuracy, that would be a much better place to start. Note that that is also a function of (a slave to, really) the accuracy of the CT's you use with it, and most of the CTs I see people use with these off ebay are very poor accuracy, especially at the lower ends of the full scale rating. I've found Continental Control Systems CT's to have less than 1% error while still being affordable, however they don't show up for sale very often

 

[e97]

Within 1-3% of what? Voltage? Current? Phase angle? The error you'll get is the product of all of these errors integrated - also note an oscilloscope with an 8-bit ADC front end is going to give a less accurate measurement for raw voltage (or current, if you're reading voltage from a CT) than a $20 multimeter off of amazon, which is also why the "DVM" function in a lot of scopes (at least ones without 10-bit or higher front ends) only reads out to 3 digits - it's just not what they are designed to do. Scopes are great, but they are the last thing on my entire workbench I would use as a reference for this (unless I was calibrating the phase angle error for a specific CT or something, which most of the higher end energy meters will let you program in, if you can manage to measure it)

if the opensource energy meter you speak of claims 0.5% base accuracy, that would be a much better place to start. Note that that is also a function of (a slave to, really) the accuracy of the CT's you use with it, and most of the CTs I see people use with these off ebay are very poor accuracy, especially at the lower ends of the full scale rating. I've found Continental Control Systems CT's to have less than 1% error while still being affordable, however they don't show up for sale very often

Watts! I've been talking about power but its good to be explicit.

If the probes are "dumb" probes that straight up use the ADC, then they're a huge rip off for how much they cost!

The probes are calibrated for voltage (~1mV) and current (0.4% 50mA - 5A) and approved for certification testing.. i'll switch them with an amazon multimeter and see if anyone complains

Re: the CTs, two sets of copper wire wound around a conductor. The eBay ones are built the same way with the nearly the same materials as the expensive ones, The difference is post mfg testing, calibration and certification. If the CT outputs mV (vs mA) add in the quality of the passives, like you said compounding errors. There are techniques to reduce error such as Kalman filtering using measurements over time or multiple "noisy" sensors.

Re: the open source meter, it must be calibrated. With a good reference you can get to revenue grade (0.5%). If you're building one get the mA CTs and use 1% passives for the voltage divider with a 16-bit ADC. That'll get you 0.5% with a good reference.

A certain energywall v1 (look up neurio) uses the same mfg as the eBay CTs. the difference being testing and calibration to meet revenue grade vs selling off the assembly line after basic testing.

No matter how good you are, its currently very very very hard to manufacture the EXACT same CT that wouldn't require only calibrating a reference design due to composition of material, lengths, injection over molding, impedance, and more fun things to think about.

At least until we get atomic level replicators

edit: here's my power meter for my test systems:



Since the systems are on 24/7, every 100W I shave off = $170/yr I dont pay for electricity

I calibrated it with the incandescent light bulb method:
get various watt light bulbs, 20W, 40W, 60W, 100W, 200W, etc..
measure
plot expected vs actual
fit a curve
adjust meter.

Its <2% vs what the energy company reports

 

[DRAGONKZ]

2.5 months after I bought a 6610-48P it’s finally arrived from the USA (shipped to AUS) but it’s been damaged in transit and arrived with a bent rack mount ear and broken heat sink retainer pin

(I could hear a slight rattle inside and opened it up to find the pin broken on one of the larger 2 heat sinks)

Everything else luckily seems ok so far!

No idea how a single retainer pin has snapped, but if someone has a dead unit and would be willing to send me a heat sink retainer pin I’d be happy to pay a reasonable amount!

Same goes if someone wants to sell a set of 2 post or 4 post rack mounts

Cheers

I’ve managed to salvage the mangled rack ear!

I still need a heat sink pin if someone has a dead unit they are willing to take one from? (I’m happy to pay!)

thanks

 

[fohdeesha]

I seriously doubt you'll ever find the retention pin, they're also "one use" so when they are removed they tend to be worn down so they don't retain as well. What I've done for issues like this in the past (motherboard chipset heatsinks) is just go to the hardware store and find an appropriate sized nut and bolt - if the PCB doesn't have a nice grounded ring around where the screw will contact (eg metal contacting the PCB looks like it might cause damage), just pick up a plastic washer as well. stick the screw upside down so the head is underneath the pcb, then if the screw has extra length (it will) it'll just be sticking up in the air on the heatsink side and not impacting the bottom of the case. Just don't over tighten it! barely a turn past finger tight

 

[Necrotyr]

Alternatively, take two zip ties that fit through the hole and use the locks as stoppers.

 

[fohdeesha]

I think I might like that even better

 

[fohdeesha]

I’ve managed to salvage the mangled rack ear!

I still need a heat sink pin if someone has a dead unit they are willing to take one from? (I’m happy to pay!)

thanks

oh, while you're in there and have the heatsink (kind of) off, can you grab a high res pic of the top of the ASIC? (or otherwise get me the part number). I'm pretty sure through PCI vendor IDs and such I've got it narrowed down to Marvell Cheetah 5, but not sure if that's the 98CX8297 or what

 

[MaxCFM]

Well late to the party but got me some 6610's for our upcoming LAN party's and a 2nd project. Working though getting them cleaned and updated. Thanks to everyone before me for all the info and fohdeesha for letting the cat out of the bag and sticking with it.

 

[fohdeesha]

Well late to the party but got me some 6610's for our upcoming LAN party's and a 2nd project. Working though getting them cleaned and updated. Thanks to everyone before me for all the info and fohdeesha for letting the cat out of the bag and sticking with it.

if you need licenses just let me know

 

[Zombielinux]

So I think my 6610 just completely died on me. Was running just fine one minute, then went down. Havn't been able to get it back up since. No data from serial console (but not sure it ever worked). Cracked open the case, had two leds near the silver heatsink on solid, then another led about an inch towards the right and back about 1/4" blinking slowly (like every 15s). Fans going full bore.

 

[juey]

Just pulled the trigger for that: Brocade ICX6450-24P 24 port Gigabyte PoE Switch + 4 X SFP ports | eBay
Pretty good price for people in the EU, seller accepted 150 €.
The last one i buyed from him was in absolutely perfect condition, even without scratches.

 

[Zombielinux]

@fohdeesha Since mine is officially dead (new powersupply didn't fix it, still shows no sign of life other than raging fans)

Have an image https://i.imgur.com/xPiiEMk.jpg
Any other chips you need a high res shot of? I think imgur is downscaling them, and until the replacement switch is in on monday, I won't have a place to host the full res ones.

 

[MaxCFM]

Well 2 of 6 6610's I got have bad POE boards, doing the ebay return dance. @Zombielinux was yours a POE version? and what all did you have it running if you don't mind sharing?

 

[Zombielinux]

@MaxCFM yea, it was the 48port PoE model.

I had it running 3 unifi WAPS, 2 1440p cameras and 1 4k camera on PoE

Beyond that I had some desktops, a few raspis, TVs, a NAS on the 40G, and servers each with 2x bonded 10G.

Bummer about the bad boards though.

 

[MaxCFM]

Well it was a risk with the seller I used as a few here have had bad luck with them, but I needed more than one switch at sub $200 per. Still need 4 more but those can wait a bit.

That doesn't sound like much in terms of power draw on the POE side wonder if it was a marginal VRM and just gave up and took something else with it?

Good luck with the new one @Zombielinux

 

[Zombielinux]

Good lord thats a lot of ports. It definitely wasn't the poe side that died. It just stopped switching.

Its not a voltage issue I can tell. Might be some little chip defect.

Good luck with yours too.

 

[MaxCFM]

Yeah trying something diff with the lan setup. I have a few ports sitting around at the moment...
Hmm can't seem to upload an image...

 

[fohdeesha]

@fohdeesha Since mine is officially dead (new powersupply didn't fix it, still shows no sign of life other than raging fans)

Have an image https://i.imgur.com/xPiiEMk.jpg
Any other chips you need a high res shot of? I think imgur is downscaling them, and until the replacement switch is in on monday, I won't have a place to host the full res ones.

Thank you! that should be all I need, the management PPC chip and everything else was quite easy to identify