Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

heromode

Active Member
May 25, 2020
139
83
28
Part of why I'm doing this is a learning exercise. The other part is I'm trying to figure out whether or not I can solve some issues I've been experiencing with NAT and some games. I don't actually have very high hopes that the games support IPv6, but on the off chance they do it could alleviate some spouse related annoyances.

That said, thank you for the article. It was informative and I do agree IPv6 is fairly insane. It's the best we currently have to help alleviate some of the IPv4 related issues and it would have been nice if the IETF had designed something else , with lessons learned, 10-15 years ago.
we need ipv5 :)
 

DavidRa

Infrastructure Architect
Aug 3, 2015
303
137
43
Central Coast of NSW
www.pdconsec.net
You literally cannot have something that's completely compatible with IPv4 as well as providing increased address spaces and changing all the code, all the tools, all the IPv4-aware hardware etc.

All the IPv4 code in the world has 32 little bit-sized boxes to put the IP address into. The moment you have more bits, no matter what you do, that code is insufficient.

IMO IPv6 isn't actually that hard, the big problems are:
  • It seems like the designers wanted the network people to control everything - which is why DHCPv6 doesn't actually do much without the routers "permitting" it;
  • The whole shebang is "design by committee" where no-one had the ability or control to stop adding stuff;
  • Almost all the corporate network people threw their hands in the air because it was "change" and "different" and "we don't need it", and combined with the afore-mentioned "routers control everything" the result that IPv6 just can't get deployed;
  • IT people seem to hate DNS and "Waaah I can't remember IPv6 addresses".
In short, people are stupid and shortsighted. Most of the problems in that blog rant are either unsolvable, irrelevant or "thinking is hard". I have had working IPv6 for nearly a decade - using a HE.net tunnel, because most ISPs won't provision IPv6 in Australia. And I'm by no means a network engineer.
 
Last edited:

tfran1990

New Member
Jul 19, 2019
3
1
1
I used the guide in the very beginning of this thread and i was able to configure everything as far a ve and vlans with the attached ports. my problem is i cant ping a laptop connected to one of the vlans. is this the right place to post a config?
 

kpfleming

Active Member
Dec 28, 2021
230
106
43
Pelham NY USA
In short, people are stupid and shortsighted. Most of the problems in that blog rant are either unsolvable, irrelevant or "thinking is hard". I have had working IPv6 for nearly a decade - using a HE.net tunnel, because most ISPs won't provision IPv6 in Australia. And I'm by no means a network engineer.
Same here, I've had IPv6 in my home network (in multiple variations) for years and it works well and solves actual problems.

Back to the OP's question: with the configuration you have, you'll need to setup IPv6 address configuration and routing at each hop, you can't expect RAs to *pass through* layer 3 network elements (routers). RAs are *local* configuration, they don't pass through routers.

Assuming that FIOS allows you to use DHCPv6-PD to get a prefix delegation of reasonable size (/56 or larger), which they don't yet in my FIOS neighborhood, you'll need to do various things:

* The OPNsense firewall will need to obtain a delegation from FIOS using DHCPv6-PD. It may also get its own address (via SLAAC or DHCPv6) but that is not mandatory, as it will already have a link-local address that can be used for IPv6 transit between itself and the FIOS network. The delegation will need to be at least a /56 in size, although a /48 would be better given your configuration; if it's a /64, you're stuck because that can't be broken down further.

* The 7250 'core router' will need to be given a block of addresses it can use on the VLANs it manages. Just like the FIOS-OPNsense link there is no need for addresses to be assigned to the transit LAN, that LAN can use the automatic link-local addresses. The block can be given to the 7250 via DHCPv6-PD *from OPNsense* (not FIOS) if OPNsense is able to do that; if not, it can be manually configured on both ends.

* Each of the VLANs on the 7250 will need to be manually configured with a sub-block out of the 7250's block (but no smaller than /64). The 7250 would then be configured to emit RAs on the LANs so that hosts can get IPv6 addresses and have routing information. It can also provide DHCPv6 services if you desire, but that is not necessary.

* There is no need for any dynamic routing protocols for this to work; the subnet addresses form a 'tree' starting from the block delegated by FIOS, and each router in the tree knows about the downstream routers, but there's no need for them to know about 2nd or 3rd level routers/routes below them.

* "passing through" DHCPv6 traffic via the OPNsense firewall isn't likely to work out well. That sort of thing can work OK for host addresses (not prefix delegation) if the device doing the passthrough supports NDP proxying (similar to ARP proxying in IPv4), but that isn't going to work with DHCPv6-PD.
 

Oodaloop

New Member
Jul 28, 2021
3
0
1
I am having connectivity problems hooking up my ICX-7250 to a Dell Poweredge R740 server . The Dell has an Intel X520/I350 daughter card. My connection uses a DAC cable from one of the 10G ports on the Intel card to one 10G port on the ICX-7250.
My connection is very instable , with frequent loss of connectivity. Not sure how to troubleshoot this - change the DAC cable? I previously had stp setup on the brocade, so I turned it off because I thought the Dell Poweredge was bringing down the network with a broadcast storm. The network was rock solid until I included the Dell server. What should I be looking for?
Did you ever get this figured out? I'm having a similar issue with a R710 server. The 10G ports on the ICX work when initially booted then just shutdown eventually and I can't get them to turn back on until after a reboot.

Anyone else have this experience or could walk me through some troubleshooting steps to figure out the problem or atleast a way to get the ports back on without having to power cycle the entire switch?
 

Vesalius

Active Member
Nov 25, 2019
224
160
43

LodeRunner

Active Member
Apr 27, 2019
457
199
43
I'm trying to hunt down packet loss in a stack of 7450's. Work has a stack of 6, full ring using QSFP modules in the back, and up until a few days ago everything was fine.

Now, despite a full stack reload and reflash to 8095g we're getting measurable packet loss to many hosts across all the units in the stack. Ruckus has a document about using the MAC filter to identify packet loss, but it's locked behind paid support. Any advice?

Edit: sh int and sh stats on the stack ports show all zeroes in the error counters. Ports used for testing area also showing no errors.

Edit 2: symptoms may have actually begun a week ago and presented as a reporting service generating blank reports (reporting service finalizes the file after receiving data or timing out). In addition to a full stack reload yesterday and a ISSU upgrade today, we reloaded all downstream switches in the building.


We traced it all to a workstation; once that port was disabled, we stopped losing traffic. Plugging that workstation into a different port immediately created the issue again. So diagnosing the home run and the workstation NIC will be fun.
 
Last edited:
  • Like
Reactions: Lone Wolf

heromode

Active Member
May 25, 2020
139
83
28
I'm not happy with the chinese fans on the ASIC's, they're actually creating noise that can be heard outside the case. Onto the next option, 3x Noctua nf-a4x10-flx fans with low noise adapter on the asic's, spinning at 3700 RPM, and fed from the 12V rail of the PSU. Then maybe 1x Noctua NF-A4x20 FLX at the rear also fed with 12V and a low noise adapter, to slightly increase the outbound airflow, because the Arctic S4028-6K's barely move any air at 4.5V.

I'm bummed that i wasn't able to get the 24 port ICX6450, thermal management would have been so much easier with 25W. I ordered the last one but had to pay with bank transfer, and while the seller was waiting for that transfer, some
yahoo came along and bought the last 24 version with paypal, and the seller then offered me the 48 at the same price.

Now looking at the markets i see the 48 port models are plentiful and selling for cheap, but the 24 port models are few and far between, and much more expensive. Man i wish i could lose that extra 25W idle :(

ATM at idle and the summer heatwave gone, i'm running at 55 degrees in a small room with lots of old gear producing heat, which will all soon be gone. But i'm determined to make my 48p model absolutely silent, it's just the 25 Watts running 24/7 is gonna cost me about 25 EUR annually until december 2024, and after that, who knows..

edit anyone have a spare good condition 6450-24, poe or no poe, and wanna swap with a -48p in excellent condition in europe, lemme know.
 
Last edited:

heromode

Active Member
May 25, 2020
139
83
28
Does anyone know if a 2.5Ggbe SFP+ to RJ45 adapter will work in the ICX6450?
@zanechua It seems that the modules with the real Aquantia chipset have special capabilities for allowing the intermediate speeds of NBase-T on switches like the Brocade ICX that do not clock at those intermediate speeds. See https://forums.servethehome.com/ind...t-marvell-88x3300-v-s-aquantia-aqs-107.30004/ for a more thorough explanation. Also https://forums.servethehome.com/index.php?threads/mix-10g-2-5g-slow-speed-high-retr.33217/.

Using your HiFiber tranceiver, do you get full 2.xGbps throughput in BOTH directions between a 10Gbe endpoint and a 2.5Gbe endpoint?
@PANiCnz i don't know the answer to your question, but the post above should give you some idea..

edit i remember from the thread that any standard 2.5Gbe will NOT work, it needs to be some special transceiver that presents itself as a 10gbit to the switch, and does the speed conversion internally.

The ICX6450 itself does NOT support 2.5gbe.
 
Last edited:

tdk

New Member
Aug 31, 2022
2
1
3
Hi,

Just made it through 379 pages... skimming most of it...
Thank you @fohdeesha for your generosity.

As a result I got some 6610, 6450s. I am looking forward to playing with them.

Has anyone mounted the 6610 sideways? Because of space constraint, the rack and patch panel will not accommodate the 6610-48 (I can do 6450-24 x 2.. but would prefer 6610-48 x 1). I can mount it sideways in something like this:


Yes, no?

Thank you.
 

donedeal19

Member
Jul 10, 2013
37
12
8
@PANiCnz I have an 7250-48p, but I have an qsfptek sfp+ for ubiquiti. Usb type c to 2.5gbe from pc to sfp+ shows up connected.
Also shows up on the 7150-c12p as a quick test. Not sure on how the performances just plug and play.
 

bluez

New Member
Sep 4, 2022
2
0
1
I was eyeing the ICX6610-48P-E for home setup. I have a few 10GbE capable devices and this port has PoE+ so it fits the bill well. However, I need help to understand how to use the QSFP+ ports to add 10GbE RJ45 copper? I am a newbie and a bit lost here.
 

SlothCroissant

New Member
Sep 4, 2022
1
0
1
Hey folks, long time lurker, first time poster. I've scanned through this thread and have gotten mixed results (some people saying their 6610s ramp the fans to a million RPM all the time, others who never see north of 40C), and I think I just need to ask flat out:

Got an ICX6610 that I've been running for a few months now, and most of my learning has been from this thread (came from a Cisco C3750X-48P to a ICX6610-48P). Quick question - what do people see as far as temps are concerned with these? Mine seems to sit at 75C in a room that is around 72F-74F, and I can't figure out why it's so hot. I've heard it runs a bit spicy, but I didn't think it lived at its fan speed switching point.

Code:
SSH@ff-icr01.lan>sh chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
     Model Number:    23-0000142-02
    Serial Number:    DPZ
    Firmware Ver:      B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
     Model Number:    23-0000142-02
    Serial Number:    F3N
    Firmware Ver:      B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 72.0 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->76       deg-C
        Speed 2:       71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 53.5 deg-C
MAC 2 Temperature Readings:
    Current temperature : 63.5 deg-C
CPU Temperature Readings:
    Current temperature : 72.0 deg-C
sensor A Temperature Readings:
    Current temperature : 53.0 deg-C
sensor B Temperature Readings:
    Current temperature : 58.5 deg-C
sensor C Temperature Readings:
    Current temperature : 43.5 deg-C
stacking card Temperature Readings:
    Current temperature : 51.5 deg-C
    Warning level.......: 77.0 deg-C
    Shutdown level......: 80.0 deg-C

My PSUs are both Rev A, if that helps. I'm also using a handful (8, I think offhand?) PoE powered devices.

Would it be worthwhile to rip the thing apart and put a new thermal pad on the CPU? I've looked, and the switch is spotless inside (I originally suspected dust, etc).

See below for my temps over the last 24h (ignore the bit where my Zabbix instance was down for maintenance). You can clearly see it bouncing between fan speed 1 and speed 2:

Screen Shot 2022-09-04 at 23.49.56.png

Thoughts?

EDIT: Just glanced, my R720XD two rack-Us away from this switch sits at 21C.
 
Last edited: