Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Aluminat

Member
Jul 5, 2019
54
23
8
Hello all, I'm looking for managed layer 2 switch 16 port gigabit. Requirements very simple tag VLAN for wireless AP, no need PoE and can do routing local.
Currently, I have local offer 75$ for Used Allied Telesis AT-GS950/16.
Can you guys comment on this? Or recommence on used one?
 

matthew5025

Member
Mar 21, 2016
27
25
13
definitely not, sounds like a bad optic or cabling (or host). I would get rid of the mikrotik from the path and retest to see if the issue goes away
Thanks for the reply! Further testing and I think it's the connenection between the host and the switch that has the problem.
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Ruckus ICX 7250 Switch Technical Specifications
^- scroll down towards the bottom

Serial port specifications (pinout mini-USB)
Pin - Signal - Description
--------------------------------------
1 - Reserved - Not used
2 - UART_RX - Receive data by Ruckus ICX 7250
3 - UART_TX - Transmit data by Ruckus ICX 7250
4 - Reserved - Not used
5 - GND - Logic ground

Serial port specifications (protocol)
Parameter - Value
----------------
Baud - 9600 bps
Data bits - 8
Parity - None
Stop bits - 1
Flow control - None


Also, created a PR against @fohdeesha 's github repo so we don't have to keep looking it up.

Add 7250/7450 Serial Pinout Details by eduncan911 · Pull Request #5 · Fohdeesha/lab-docu

:)
RE: 7250/7450 Mini-USB-B Serial Cables

So, I found this cable on eBay for HAM radio programming.

(Removed link to bad cable, don't get)

I confirmed with the seller the pinouts, which seem to match that of the Ruckus website for the 7250/7450 serial cable.

The only addition is that they supply 5V+ on pin 1 - where the Ruckus website says Pin 1 is unused.

Unfortunately, since I am remodeling, I don't have a workbench to disassemble my ICX 7250 48P to see if pin one is connected to anything.

What do y'all think? if Ruckus says Pin 1 is reserved, I'm guessing that means not in use. Thoughts?
 
Last edited:
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
RE: 7250/7450 Mini-USB-B Serial Cables

So, I found this cable on eBay for HAM radio programming.

FTDI USB Programming Cable TH-7800 TH-8600 TH-9800 TYT Serial Mini | eBay

I confirmed with the seller the pinouts, which seem to match that of the Ruckus website for the 7250/7450 serial cable.

The only addition is that they supply 5V+ on pin 1 - where the Ruckus website says Pin 1 is unused.

Unfortunately, since I am remodeling, I don't have a workbench to disassemble my ICX 7250 48P to see if pin one is connected to anything.

What do y'all think? if Ruckus says Pin 1 is reserved, I'm guessing that means not in use. Thoughts?
I was going to say just ship me one and I'll give it a rip, but after a little digging it seems the TYT's programming port isn't RS232, it uses 3.3v TTL logic levels - (this guy says TTY but means TTL, as indicated by the TTL adapter he used, last post) TYT HT-7800 Data port pin outs

so connecting that to the ~12v logic levels on the ICX RS232 port, the 3.3v TTL signals sent to the thing won't be enough to trigger anything, and then the ICX talking back to the 3.3v TTL adapter in that cable with ~12V logic levels will probably blow out the ftdi adapter in the thing
 

Bocephus

New Member
Feb 12, 2020
9
4
3
I'm new here, and stumbled across this thread while researching an inexpensive 10gb switch options for my home lab. I'm real close to ordering two ICX6610's but before I do, I'm trying to figure out the license bit.

If I've read this thread correctly, the license is irrelevant, and can be solved?

Just wanted to clarify, because prem license price jumps from ~$200 to ~$350ish.

Thanks in advance!
 
Last edited:

ewer0012

Member
Feb 10, 2019
81
25
18
If I've read this thread correctly, the license is irrelevant, and can be solved with help from @fohdeesha?

Just wanted to clarify, because prem license price jumps from ~$200 to ~$350ish.

Thanks in advance!
Basically, yes.

If you don't mind me asking, what's the use case for two 6610s?
 

Bocephus

New Member
Feb 12, 2020
9
4
3
Basically, yes.

If you don't mind me asking, what's the use case for two 6610s?
Thanks for the quick reply.

As for the use case, my Lab currently consists of 8 physical servers, and I'm getting ready to split them between two half racks, and I wanted a top of rack switch for each rack.

I currently have 2 SG line Cisco switches, that i"m quite happy with, except for the fact they are only 1GB. For 10GB I'm using a MicroTik CRS309, and I really am not a fan.

As for all the 10GB ports, currently on my CRS309 they are used strictly for storage replication for my 3 node cluster. (Previously Hyper-V/S2D, currently running Proxmox/Ceph). I'd also like to add 10GB to a few other boxes as well over time.

Kind of long winded, but hopefully that answer the question.
 

ewer0012

Member
Feb 10, 2019
81
25
18
Why split between two half-racks? If you only have 8 servers, that's not too much space unless they're all 3u-4u. One 6610 would have 16x 10Gbe if you use the two breakout ports on the back. Would be enough for 2 connections per server, then you could utilize one or both of the QSFP ports for your storage.

The 6610s are really nice for the 2 breakout ports, giving you 16x 10Gbe ports total. I'm using the QSFP ports for my access to NAS and iSCSI respectively. Works well.

One thing I would caution about buying two (unless your power bill isn't a big deal) is that you're probably looking at about $30-$35+ for 30 days of running 2 of them 24/7. That's about what it's costing me now for 2x 48P that are mostly populated.
 

Bocephus

New Member
Feb 12, 2020
9
4
3
Most of my servers are actually desktops. I have a PowerEdge R420, 3 Precision T7600's, a Precision T5400, and the rest are re-purposed gaming rigs. So with the exception of the R420, everything is 3-4U.

As for two TOR switches, versus just the one, I was wanting to minimize the cables running between the two racks. Port count I'm sure I'd be fine with the ICX6610-48.
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Most of my servers are actually desktops. I have a PowerEdge R420, 3 Precision T7600's, a Precision T5400, and the rest are re-purposed gaming rigs. So with the exception of the R420, everything is 3-4U.

As for two TOR switches, versus just the one, I was wanting to minimize the cables running between the two racks. Port count I'm sure I'd be fine with the ICX6610-48.
^- this person is a clone of me. A real "home lab" user repurposing things to learn, that can't afford the pretty $10,000 setups in r/homelab. Mine were old mining and gaming rigs myself. Though I've since downsized, got rid of most, but let only a single purpose Godzilla server running 5 VMs plus K8s for a dozen Tor and NZB docker images mixed with a dozen UP Boards as part of the K8s cluster.

Only real non-k8s machine running is my UP board router that supports AES-NI for my openvpn, which is getting replaced soon with Frankenstein Qubes-like router I've been tinkering with soon for better isolation.

I just wished AMD had IoT solutions like these UP boards, as Ive gotten rid of most Intel from my household.
 

Bocephus

New Member
Feb 12, 2020
9
4
3
^- this person is a clone of me. A real "home lab" user repurposing things to learn, that can't afford the pretty $10,000 setups in r/homelab. Mine were old mining and gaming rigs myself. Though I've since downsized, got rid of most, but let only a single purpose Godzilla server running 5 VMs plus K8s for a dozen Tor and NZB docker images mixed with a dozen UP Boards as part of the K8s cluster.

Only real non-k8s machine running is my UP board router that supports AES-NI for my openvpn, which is getting replaced soon with Frankenstein Qubes-like router I've been tinkering with soon for better isolation.

I just wished AMD had IoT solutions like these UP boards, as Ive gotten rid of most Intel from my household.
Yeah, my lab keeps getting bigger, and so does the electric bill. I often wonder if it wouldn't be cheaper in the long term to pick up ~5 R630's and get rid of my 7 Desktops/Servers.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Yeah, my lab keeps getting bigger, and so does the electric bill. I often wonder if it wouldn't be cheaper in the long term to pick up ~5 R630's and get rid of my 7 Desktops/Servers.
From a pure energetic standpoint, nothing beats consolidation and virtualization: it's better to have a pretty big machine with lots of idling cores and RAM than to have half a dozen or more of smaller servers, all of them underutilized. A "modern" dual socket server can idle below 75ish W, and it can perform better than 3 or 4 "desktop" PCs.
The break-even point however will depend of course on the price of the components that you can find vs. the increased power bills.
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Yeah, my lab keeps getting bigger, and so does the electric bill. I often wonder if it wouldn't be cheaper in the long term to pick up ~5 R630's and get rid of my 7 Desktops/Servers.
To echo @infoMatt, this was one of the primary reasons I consolidated into a single BFM (Big Fracking Machine). All of the machines I had running was something like 400 to 500W idle, and a LOT of heat in the closet. I'm on my third build of the BFM right now and finally got the power levels decent (~75W).

The two other primary reasons was heat generation and exploit/attack surface (I'm in CyberSec, and just can't stand to run unsecure hardware/ports no matter the "low threat model".

Previously, I build an Intel 2690V3 (12C) with an engineering sample and 128 GB of ECC DDR4 Buffered ram. Idled with 14 HDDs at 75W due to the 9W LSI card and 12W Supermicro SC846 chassis backplane. It wasn't enough cores.

Then I went with dual 2640LV4s (14C each, 28C/56T total) ES in a Supermicro board with no IPMI (to remove those CVEs from my scans, rigged up a RPi with serial console instead to power on/off remotely on an airgapped VLAN). Got the CPUs cheap on ebay since they were ES. Same 128 GB DDR4 ECC Buffered ram. Idle was now 81W, due to the chipset and dual CPUs, along with a Q2000 nvidia card for hardware encoding/decoding of movie streams. If I removed a single CPU, I could get it down to around 70W idle. Which led me to my collection of SBC UP Boards to now handle most of my home infra and apps needs.

Finally, I got sick of the Intel exploits, Intel ME disable from each BIOS update, etc... And went all in with AMD.

Current server is AMD 2950X (16C, 32T), more than double the Mhz of the engineering sample versions of the Intel chips above (and more than double, sometimes 3x the CPU Mark ratings per core!). And, I replaced my Port Expander backplane in the SC846 chassis with a JBOD version (saving 12W, 24 SATA hookups now)), dumped the LSI card (~9W saving) for 10 onboard SATA ports along with replacing the 12x 4TB drives with 8x 10TB and 12TB drives (this was the most expensive upgrade of them all - I've done it piece meal over the last two years, slowly...). And, I had to switch to Unbuffered ECC 64 GB DDR4 ram as the AMD X399 chipset only supports Unbuffered, not the Buffered ECC I had in the Intel boards. :( But i did notice I wasn't using that much ram any longer, with all the UP Boards I have running in clusters.

I'm now down to ~75W idle with that beast. AMD Threadripper and X399 isn't as power efficient at idle as the Intel chipsets. In a few years, I may downscale to the Ryzen consumer models with 16C as this was kinda overkill, and the newer 3000 series is just about as powerful - with much lower TDP. The UP Boards, per 4x board cluster of the UP Squared Pentium versions hovers around ~8W when idle. But they are hardly idle, always processing some NZB, Tor relay and services, and constant metric/logs/SNMP collections. All in all, I'd say per cluster averages around 15 to 20W per 24 hours - remember, this is running like two dozen docker images and services, log collections and parsing, metric compaction nightly, etc - they get heavy heavy use. I have two clusters I switch between as I am constantly tearing down and rebuilding the k8s clusters in my homelab for work. I have a pretty good immutable setup now using PXE boot images created from Packer - so I can easily replace a single bare metal UP board at any time (I really need to blog this setup).

I run the latest Xen on ArchLinux on everything for the latest CVE fixes.

Which lead me now to this thread about Brocade switches... I need to get a handle on my VLANs and get some POEs for new security cameras, as well as lowering even more wattage. I'm now building up my single ICX 7250 48P for that task, thanks to this thread, to replace a number of POE, managed/unmanaged, and dumb switches.
 
Last edited:

matthew5025

Member
Mar 21, 2016
27
25
13
Thanks for the reply! Further testing and I think it's the connenection between the host and the switch that has the problem.
Just placing this here to help anyone who might run into the same issue:

The switch gave itself an IP address from DHCP and assigned it to the uplink port, and effectively did not forward ARP messages, causing the issue. Running no ip address solved the issue
 

SRussell

Active Member
Oct 7, 2019
327
152
43
US
A "modern" dual socket server can idle below 75ish W, and it can perform better than 3 or 4 "desktop" PCs.
The break-even point however will depend of course on the price of the components that you can find vs. the increased power bills.
What socket and CPU would you consider to be the bottom of the line for a modern dual socket server?
 

SRussell

Active Member
Oct 7, 2019
327
152
43
US
Looking for some help deciding between two switches:
Brocade BR-VDX6740T: https://www.dataswitchworks.com/datasheets/switches/brocade-vdx-6740-switches-ds.pdf
Brocade ICX7750-48C: https://www.dataswitchworks.com/datasheets/brocade-icx-7750-ds.pdf

I currently have 8x servers:
6x: 2x 10GbE-T, 2x 1GbE-T
2x: 2x 10GbE-T, 2X 1GbE-T, 2x 40GbE

  • Both switches have more than enough ports for current and future growth.
  • The BR switch has 4 QSFP+ vs the 6 QSFP+ ports on the ICX. I am unsure if the QSFP+ ports on the BR are uplink only.
  • The biggest unknown is power usage. The ICX quotes 511W-typical to 586W-maximum. The BR quotes 460W-maximum but also states low power consumption, consuming less than 5 watts per 10 GbE port. That would put the switch at 240W + 4x QSFP+ power: where is the 220W being consumed when at maximum draw?
  • The difference in price, BR=$500 and ICX=$895

My current goals are setting up a VSAN and KVM lab. Down the road adding in NSX.

The least of my concerns is price. My main concern is to not bottleneck myself with features I may want down the road but my naivety does not see. 100W+/- is not a deal breaker.
 
Last edited:

UKEE

New Member
Jan 2, 2020
10
9
3
Lexington, KY
If this post is not allowed, I'll edit it. With that said, if anyone is looking for a lower price ICX6450-24, one popped up on eBay for $145 shipped OR BEST OFFER (free shipping and includes console cable according to listing). These ARE NOT my listing, just thought I would pass it on. You will have to search eBay for the listing. They have 5 available. I would guess that you should be able to get these for $135 shipped or maybe lower (haven't tried).

Best price I've seen since I've joined this site. If I hadn't just bought a 7250, I would be all over these.
 

BecauseScience

New Member
Feb 3, 2016
19
2
3
Would it be a bad idea to run a 6450 or 6610 in a uninsulated residential attic? What about a 7150-C12P? I have no idea what the temperatures are up there. Typical weather here runs -10F to 95F at the extremes. The majority of the year is much milder.

I have an attic exhaust fan so I doubt it goes much above ambient even on sunny summer days. I'm hoping the attic won't see the full outdoor temperature swing due to heating and cooling from the living space leaking up there too.

My motivation is laziness. It would be a lot easier to run a few pairs of fiber from the basement to a switch in the attic and do all of my cat6 drops from there. I don't have a good spot to get a bunch of cat6 from the basement to the attic and the attic is the only practical place to bring drops down into individual rooms.