Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Rain]

I've still been playing around and trying to get stacking working with 10GBASE-T transceivers between two 7250-48p switches. Nothing I've tried seems to work. ...

I meant to follow up on my attempts to stack with 10GBASE-T transceivers a while back but I've been busy.

To anyone else thinking of stacking with 10GBASE-T transceivers: It doesn't appear to be possible on the ICX 7250 switches (and likely all other ICX switches as well). I imagine the desire to stack over CAT6 is rather limited, but hopefully this saves someone a headache.

It isn't a design limitation with the switches, it's an issue with the transceivers. The 10GBASE-T transceivers can only send/receive ethernet data/frames (IEEE 802.3) whereas fiber transceivers and/or DACs are raw data connections that can be used for other, non-ethernet connections (Fiber Channel, for example). As far as I can tell, this is true of every current 10GBASE-T transceiver on the market (understandably; they're RJ45 adapters, after all).

The ICX switches appear to negotiate via ethernet and then switch to communicating between stack members via a raw connection. This is why the interfaces appear to flap when attempting to stack with 10GBASE-T transceivers. The switches bring the stacking interfaces up, figure out who's on the other end, and then switch to a raw data connection. Since the transceivers don't support non-802.3 operation, they promptly disconnect. The switch the resets the interfaces and the cycle repeats.

One of my searches while trying to figure this out rendered this internal Brocade slide-deck on stacking: stack-unit - Brocade Community Forums (edit: Despite the title, this isn't a link to the Brocade forums; it's to "studylib.net") -- Notable slides that support this theory are 30 & 31.

 

[MelnorMelvin]

Does anyone know what the difference between these AWF-2DC-1000W-E part numbers is? I was thinking at first it might tie into the hardware revision, but that didn't pan out.

23-0000142-01
23-0000142-02
23-0000142-04

More or less I'm trying to find the best deal for a Rev B/C power supply for an ICX6610-48P-PE.

 

[MelnorMelvin]

One more question. I am needing to pick up 2x ICX-FAN10-E for my ICX6610-48P-PE, and was hoping not to spend a fortune on fans.

Pretty much as cheap as I can find for an item actually tagged as ICX-FAN10-E is ~$49.50, some with a P/N of 80-1008308-02 on the factory box.

However, I did come across an item which looks physically identical with part number 60-1002274-03 (sticker on the bottom of the fans) at a much more reasonable ~$14.99, with bulk discounts (qty 2 at $13.49/ea).

Does anyone know if these two items are actually the same?

 

[klui]

Better hurry. Only 1 left.

 

[MelnorMelvin]

I’ll be honest, I already bought two because I was afraid of missing out on the price. I’m just hoping they will work for the ICX6610.

 

[eduncan911]

One more question. I am needing to pick up 2x ICX-FAN10-E for my ICX6610-48P-PE, and was hoping not to spend a fortune on fans.

Pretty much as cheap as I can find for an item actually tagged as ICX-FAN10-E is ~$49.50, some with a P/N of 80-1008308-02 on the factory box.

However, I did come across an item which looks physically identical with part number 60-1002274-03 (sticker on the bottom of the fans) at a much more reasonable ~$14.99, with bulk discounts (qty 2 at $13.49/ea).

Does anyone know if these two items are actually the same?

If you could remember to ping, ping us (or me) when you get the fans and let us know. I am in need of 1 or 2 as well.

 

[Mithril]

For the 6450 I seem to recall someone having instructions on how to change the PWM values used for low and high fan speed, but I can't find the post again

 

[sdyoung]

Assign one. That's the way I set up my switch, with a desktop plugged directly into the management port of the switch.

In Windows 11, type the start button and search for "view network connections". Right click Ethernet and double-click "Internet Protocol Version 4 (TCP/IPv4). You can then "Use the following IP address" and set whatever address you want there. Say 10.1.1.50 with a subnet mask of 255.255.255.0. Then follow fodheesas guide EXACTLY, setting a compatible IP:

factory set-default
setenv ipaddr 10.1.1.99
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 10.1.1.50
setenv image_name ICX64xx/ICX64R08
...

Once you are done following the guide, set Win 11 back to "Obtain an IP Address automatically" and plug it back into your network.

@Lone Wolf, plugging the switch directly into my desktop, instead of another switch on my network, and manually assigning a new IP to the desktop did the trick. Was able to run update_uboot with no issues and am now on my way through the rest of the guide! Thanks again for the help man!

 

[Lone Wolf]

Awesome, I'm glad I was able to help! Enjoy your new switch!

 

[Pitchfork]

Greetings from sunny Brisbane

Before I start let me say that despite working hand-on in the IT industry for over 40 years, my experience of networking previously has been limited to plugging RJ45 cables into dumb unmanaged switches. If I sound like a total noob it's because, well, I am.

Having got that out of the way, we recently bought 4 x ICX6610s and I am currently working my way through learning about them. We have "just plugged one in" to connect our servers temporarily and am working through another one of them using the guide on fohdeesha.com.

My first issue is that we need to transfer a lot of data (umpteen TB) from one server to another as we transition hardware and at the moment everything plugged into the switch is running by default at 1gbE. I understand that the 8 ports on the left hand side of the front of the switch are SPF+ and can run at 10gbE. I'm guessing that I need some kind of adapter to plug RJ45 cables into these. So my first question is, what do I look for when I go searching for them? Are there any that work better than others? Any that should be avoided?

The second question is regarding the breakout ports at the back of the unit. The diagram I am looking at says that the rightmost pair of 40gbE ports can break out into 8 x 10gbE. Again, what do I need to do this.

Third question is about the uplink ports (the left most two on the back of the unit). I have cables that came with the units. Is there any special way that these need to be connected together or are they just daisy chained from one switch to another?

Apologies for all these dumb questions - I've been reading and playing (and going deaf from the noise) for a week or so and I could really use the input.

Thanks and Regards

Nigel.

 

[Rand__]

My first issue is that we need to transfer a lot of data (umpteen TB) from one server to another as we transition hardware and at the moment everything plugged into the switch is running by default at 1gbE. I understand that the 8 ports on the left hand side of the front of the switch are SPF+ and can run at 10gbE. I'm guessing that I need some kind of adapter to plug RJ45 cables into these. So my first question is, what do I look for when I go searching for them? Are there any that work better than others? Any that should be avoided?

Look at these (other models might be available too) https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

The second question is regarding the breakout ports at the back of the unit. The diagram I am looking at says that the rightmost pair of 40gbE ports can break out into 8 x 10gbE. Again, what do I need to do this.

40GB -> 4x10 GB needs a breakout QSFP cable (and potentially configuration at interface level, have not done it on the brocades)

Third question is about the uplink ports (the left most two on the back of the unit). I have cables that came with the units. Is there any special way that these need to be connected together or are they just daisy chained from one switch to another?

Check the manual to get all the information you need
Look for " Dedicated stacking ports and trunks in slot 2 on an ICX 6610 "

 

[kpfleming]

Look at these (other models might be available too) https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

Also, keep in mind that 'plugging RJ45 cables into these' may not be the actual question you want to ask

If the devices you are using have 10GbE copper (RJ-45) NICs, or NICs with SFP+ that are equivalent, then using Cat-6 RJ-45 cables to connect them to the switch could provide 10GbE links. If those devices do not have 10GbE NICs, or they don't have copper ports, then you'll need to upgrade both ends (device and switch) in order to obtain 10GbE speeds.

 

[Pitchfork]

Also, keep in mind that 'plugging RJ45 cables into these' may not be the actual question you want to ask

If the devices you are using have 10GbE copper (RJ-45) NICs, or NICs with SFP+ that are equivalent, then using Cat-6 RJ-45 cables to connect them to the switch could provide 10GbE links. If those devices do not have 10GbE NICs, or they don't have copper ports, then you'll need to upgrade both ends (device and switch) in order to obtain 10GbE speeds.

Thanks for the response - yes the ones in question have 10GbE NICs.

I should have lifted the veil a little higher. It's all a complex (to me) jigsaw - we are connecting together 3 x Dell R730's, a stack of 3 x Isilon X210's (with a Mellanox IS4022 switch) and a rack mounted QNAP plus the ICX-6610s (we have two 6610 and 2 x 6610P to pick from) . The idea is that the 3 x R730's will form a VMWare cluster and will use the Isilons for main storage (they have none of their own). The main QNAP will us used for backups and for copying the whole lot up into the cloud for a secondary backup.

Two of the R730's have network connections that look like this. The two RJ-45 NICs are 10/100/1000. I have no idea what the other connections are (like I said - this is my first networking rodeo).



The third R730 looks a little different. This one has 4 x RJ45 NICs but I have no idea what the heck the second interfaces are.


So far as the Isilon X210s are concerned, I believe that the three of them are connected to the Mellanox switch via their own fibre connections to form their own "internal" network for the Isilons to talk to each other. How the Isilons (or the Mellanox switch) connect into the main network is (so far) beyond me.


The main QNAP in question is a new QNAP TS-1273AU-RP-8G with a QXG-10G2T-107 Dual Port Expansion card (10GbE / 5GbE / 2.5GbE / 1GbE / 100MbE)

We have a network engineer coming out from Dell / Unisys early next week to recommission the Isilons and I am hoping to get his advice on optimal connections of all the parts in this jigsaw but I was hoping that the collected wisdom here might be able to make some suggestions as to "how do we connect all these bits of disparate kit together to get the best network throughput speed overall at the minimum (of course) cost".

If anyone with any suggestions happens to be here in or close to Bris-Vegas I'd be happy to pour vast quantities of beer down your throat and stuff you full of BBQ in exchange for advice, knowledge, suggestions, wisdom etc.

Apologies for the length of the post and the multitude of pictures. I hope I'm not inadvertently breaking any of the forum rules.

Thanks again

Nigel.

 

[LodeRunner]

Your first image is a bog standard dual gigabit Ethernet card. The next image 4x LC fiber connectors on it, in the form of Small Form Pluggable (SFP) optics. It looks like the quad 10 Gb rNDC.

Next a quad ported RJ45 rNDC, a 2x 10Gbe + 2x 1Gbe model. The add-in card in with the protective covers is a dual 8 Gb Fibre Channel card; a lot of SANs use/used it; performant iSCSI is relatively recent, comparatively.

You'll need to either pull the Dell part numbers off the cards, or boot the system with a Linux live CD or equivalent to pull the models and get more information. For example, some 10 Gb network cards are 'converged' in that they can do Ethernet and Fibre Channel, rather than having dedicated cards for each.

Isilon X210's apparently can be configured as iSCSI (Ethernet) or Fibre Channel.

Your Mellanox IS5022's are 8x 40 Gb QSFP interfaces; I don't see any QSFP interfaces in your pictures beyond those. It's a managed Infiniband switch; yet another protocol. The IS5022 does not support any other protocols.

I do believe your first step should be to trace and tag each cable; I would be surprised if the Isilon storage servers were using Infiniband.

 

[Rand__]

You'll need to either pull the Dell part numbers off the cards, or boot the system with a Linux live CD or equivalent to pull the models and get more information.

I'd start with running the service tags at Dell's - unless somebody modified the hardware aftersales (depends on the history of the boxes, larger companies usually dont) u can see what the box was sold with - the depicted server was sold with a ( G8RPD ) "QLogic 57800 2x10Gb BT + 2x1Gb BT Network Daughter Card " which is a " QLogic 57800 2x10Gb DA/SFP+ "


That box is still under support by the way so you also can simply ask Dell to help you out (unless you intended to buy the needed components aftermarket)

 

[heromode]

ⓣⓐⓟⓔ ⓜⓞⓓ

Hi, finally got to flash and license and setup my icx6450-48p, thanks to @fohdeesha documentation and everyone in thread..

On the ICX6450 the most basic fan mod is by removing two of the three fans. You should definitely cover the slots on the two empty fan locations with a piece of tape. Without the tape a lot of the air pushed out by the single fan is just recirculated via the adjacent empty fan slots, ruining the negative pressure needed to pull air out from the front of the case:

Adding the tape creates the correct negative pressure, pulling out the air from the rest of the case. This results in a dramatic drop in temps, in my test the temp dropped 5.5 6.5 degrees 5.5 Celsius in about 30 minutes 60 minutes!



2x SFP+ ports connected, 2x ethernet ports connected, all idle, single stock fan in the slot closest to corner
Room ambient temperature 25 Celsius:


Before TapeMod

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 60.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 54.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 60.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

After TapeMod

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 55.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 55.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

edit: now another degree after 60 mins:

Fan controlled temperature: 54.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->65 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
Current temperature : 54.0 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-C

edit2: now stabilized at 55, a 5.5 degree reduction.

 

[BobTB]

My ISP gave me a /30 with a routed /27 available for my use. I was using pfsense in pure router mode to connect the /30 with /27 network, with one WAN and one "local wan" going back to ICX VLAN, where I then connect the other routers to it.

Now I got the idea to go and use my icx-6610 as a router for the above and just connect my other routers to the ports directly with completely eliminating this first pfsense doing the /30 to /27.

Will this work? Just add route and I am done?

 

[kpfleming]

My ISP gave me a /30 with a routed /27 available for my use. I was using pfsense in pure router mode to connect the /30 with /27 network, with one WAN and one "local wan" going back to ICX VLAN, where I then connect the other routers to it.

Now I got the idea to go and use my icx-6610 as a router for the above and just connect my other routers to the ports directly with completely eliminating this first pfsense doing the /30 to /27.

Will this work? Just add route and I am done?

Yes, that is just basic IPv4 routing and should work fine.

 

[BobTB]

Yes, that is just basic IPv4 routing and should work fine.

Thx, will try to do this.

Now the question I have is how can I go to a specific VLAN and ping from there when I am SSHed to the switch. I would like to "go to" the VLAN my ISP is connected to and try to ping their gateway. Or am I thinking about this wrong?

I can do it from pfsense, when it is connected to this VLAN, I was just trying to see if I can do this from the ICX console somehow.

 

[fohdeesha]

use the ping command on the switch like normal and specify the IP to source the ping from, the ip being the IP on the VE in that vlan

ping [vrf <vrf-name>] <ip addr> [source <ip addr>]

so just ping 8.8.8.8 69.69.69.69