Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Pitchfork]

Greetings from sunny Brisbane

Before I start let me say that despite working hand-on in the IT industry for over 40 years, my experience of networking previously has been limited to plugging RJ45 cables into dumb unmanaged switches. If I sound like a total noob it's because, well, I am.

Having got that out of the way, we recently bought 4 x ICX6610s and I am currently working my way through learning about them. We have "just plugged one in" to connect our servers temporarily and am working through another one of them using the guide on fohdeesha.com.

My first issue is that we need to transfer a lot of data (umpteen TB) from one server to another as we transition hardware and at the moment everything plugged into the switch is running by default at 1gbE. I understand that the 8 ports on the left hand side of the front of the switch are SPF+ and can run at 10gbE. I'm guessing that I need some kind of adapter to plug RJ45 cables into these. So my first question is, what do I look for when I go searching for them? Are there any that work better than others? Any that should be avoided?

The second question is regarding the breakout ports at the back of the unit. The diagram I am looking at says that the rightmost pair of 40gbE ports can break out into 8 x 10gbE. Again, what do I need to do this.

Third question is about the uplink ports (the left most two on the back of the unit). I have cables that came with the units. Is there any special way that these need to be connected together or are they just daisy chained from one switch to another?

Apologies for all these dumb questions - I've been reading and playing (and going deaf from the noise) for a week or so and I could really use the input.

Thanks and Regards

Nigel.

 

[Rand__]

My first issue is that we need to transfer a lot of data (umpteen TB) from one server to another as we transition hardware and at the moment everything plugged into the switch is running by default at 1gbE. I understand that the 8 ports on the left hand side of the front of the switch are SPF+ and can run at 10gbE. I'm guessing that I need some kind of adapter to plug RJ45 cables into these. So my first question is, what do I look for when I go searching for them? Are there any that work better than others? Any that should be avoided?

Look at these (other models might be available too) https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

The second question is regarding the breakout ports at the back of the unit. The diagram I am looking at says that the rightmost pair of 40gbE ports can break out into 8 x 10gbE. Again, what do I need to do this.

40GB -> 4x10 GB needs a breakout QSFP cable (and potentially configuration at interface level, have not done it on the brocades)

Third question is about the uplink ports (the left most two on the back of the unit). I have cables that came with the units. Is there any special way that these need to be connected together or are they just daisy chained from one switch to another?

Check the manual to get all the information you need
Look for " Dedicated stacking ports and trunks in slot 2 on an ICX 6610 "

 

[kpfleming]

Look at these (other models might be available too) https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

Also, keep in mind that 'plugging RJ45 cables into these' may not be the actual question you want to ask

If the devices you are using have 10GbE copper (RJ-45) NICs, or NICs with SFP+ that are equivalent, then using Cat-6 RJ-45 cables to connect them to the switch could provide 10GbE links. If those devices do not have 10GbE NICs, or they don't have copper ports, then you'll need to upgrade both ends (device and switch) in order to obtain 10GbE speeds.

 

[Pitchfork]

Also, keep in mind that 'plugging RJ45 cables into these' may not be the actual question you want to ask

If the devices you are using have 10GbE copper (RJ-45) NICs, or NICs with SFP+ that are equivalent, then using Cat-6 RJ-45 cables to connect them to the switch could provide 10GbE links. If those devices do not have 10GbE NICs, or they don't have copper ports, then you'll need to upgrade both ends (device and switch) in order to obtain 10GbE speeds.

Thanks for the response - yes the ones in question have 10GbE NICs.

I should have lifted the veil a little higher. It's all a complex (to me) jigsaw - we are connecting together 3 x Dell R730's, a stack of 3 x Isilon X210's (with a Mellanox IS4022 switch) and a rack mounted QNAP plus the ICX-6610s (we have two 6610 and 2 x 6610P to pick from) . The idea is that the 3 x R730's will form a VMWare cluster and will use the Isilons for main storage (they have none of their own). The main QNAP will us used for backups and for copying the whole lot up into the cloud for a secondary backup.

Two of the R730's have network connections that look like this. The two RJ-45 NICs are 10/100/1000. I have no idea what the other connections are (like I said - this is my first networking rodeo).



The third R730 looks a little different. This one has 4 x RJ45 NICs but I have no idea what the heck the second interfaces are.


So far as the Isilon X210s are concerned, I believe that the three of them are connected to the Mellanox switch via their own fibre connections to form their own "internal" network for the Isilons to talk to each other. How the Isilons (or the Mellanox switch) connect into the main network is (so far) beyond me.


The main QNAP in question is a new QNAP TS-1273AU-RP-8G with a QXG-10G2T-107 Dual Port Expansion card (10GbE / 5GbE / 2.5GbE / 1GbE / 100MbE)

We have a network engineer coming out from Dell / Unisys early next week to recommission the Isilons and I am hoping to get his advice on optimal connections of all the parts in this jigsaw but I was hoping that the collected wisdom here might be able to make some suggestions as to "how do we connect all these bits of disparate kit together to get the best network throughput speed overall at the minimum (of course) cost".

If anyone with any suggestions happens to be here in or close to Bris-Vegas I'd be happy to pour vast quantities of beer down your throat and stuff you full of BBQ in exchange for advice, knowledge, suggestions, wisdom etc.

Apologies for the length of the post and the multitude of pictures. I hope I'm not inadvertently breaking any of the forum rules.

Thanks again

Nigel.

 

[LodeRunner]

Your first image is a bog standard dual gigabit Ethernet card. The next image 4x LC fiber connectors on it, in the form of Small Form Pluggable (SFP) optics. It looks like the quad 10 Gb rNDC.

Next a quad ported RJ45 rNDC, a 2x 10Gbe + 2x 1Gbe model. The add-in card in with the protective covers is a dual 8 Gb Fibre Channel card; a lot of SANs use/used it; performant iSCSI is relatively recent, comparatively.

You'll need to either pull the Dell part numbers off the cards, or boot the system with a Linux live CD or equivalent to pull the models and get more information. For example, some 10 Gb network cards are 'converged' in that they can do Ethernet and Fibre Channel, rather than having dedicated cards for each.

Isilon X210's apparently can be configured as iSCSI (Ethernet) or Fibre Channel.

Your Mellanox IS5022's are 8x 40 Gb QSFP interfaces; I don't see any QSFP interfaces in your pictures beyond those. It's a managed Infiniband switch; yet another protocol. The IS5022 does not support any other protocols.

I do believe your first step should be to trace and tag each cable; I would be surprised if the Isilon storage servers were using Infiniband.

 

[Rand__]

You'll need to either pull the Dell part numbers off the cards, or boot the system with a Linux live CD or equivalent to pull the models and get more information.

I'd start with running the service tags at Dell's - unless somebody modified the hardware aftersales (depends on the history of the boxes, larger companies usually dont) u can see what the box was sold with - the depicted server was sold with a ( G8RPD ) "QLogic 57800 2x10Gb BT + 2x1Gb BT Network Daughter Card " which is a " QLogic 57800 2x10Gb DA/SFP+ "


That box is still under support by the way so you also can simply ask Dell to help you out (unless you intended to buy the needed components aftermarket)

 

[heromode]

ⓣⓐⓟⓔ ⓜⓞⓓ

Hi, finally got to flash and license and setup my icx6450-48p, thanks to @fohdeesha documentation and everyone in thread..

On the ICX6450 the most basic fan mod is by removing two of the three fans. You should definitely cover the slots on the two empty fan locations with a piece of tape. Without the tape a lot of the air pushed out by the single fan is just recirculated via the adjacent empty fan slots, ruining the negative pressure needed to pull air out from the front of the case:

Adding the tape creates the correct negative pressure, pulling out the air from the rest of the case. This results in a dramatic drop in temps, in my test the temp dropped 5.5 6.5 degrees 5.5 Celsius in about 30 minutes 60 minutes!



2x SFP+ ports connected, 2x ethernet ports connected, all idle, single stock fan in the slot closest to corner
Room ambient temperature 25 Celsius:


Before TapeMod

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 60.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 54.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 60.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

After TapeMod

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 55.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 55.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

edit: now another degree after 60 mins:

Fan controlled temperature: 54.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->65 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
Current temperature : 54.0 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-C

edit2: now stabilized at 55, a 5.5 degree reduction.

 

[BobTB]

My ISP gave me a /30 with a routed /27 available for my use. I was using pfsense in pure router mode to connect the /30 with /27 network, with one WAN and one "local wan" going back to ICX VLAN, where I then connect the other routers to it.

Now I got the idea to go and use my icx-6610 as a router for the above and just connect my other routers to the ports directly with completely eliminating this first pfsense doing the /30 to /27.

Will this work? Just add route and I am done?

 

[kpfleming]

My ISP gave me a /30 with a routed /27 available for my use. I was using pfsense in pure router mode to connect the /30 with /27 network, with one WAN and one "local wan" going back to ICX VLAN, where I then connect the other routers to it.

Now I got the idea to go and use my icx-6610 as a router for the above and just connect my other routers to the ports directly with completely eliminating this first pfsense doing the /30 to /27.

Will this work? Just add route and I am done?

Yes, that is just basic IPv4 routing and should work fine.

 

[BobTB]

Yes, that is just basic IPv4 routing and should work fine.

Thx, will try to do this.

Now the question I have is how can I go to a specific VLAN and ping from there when I am SSHed to the switch. I would like to "go to" the VLAN my ISP is connected to and try to ping their gateway. Or am I thinking about this wrong?

I can do it from pfsense, when it is connected to this VLAN, I was just trying to see if I can do this from the ICX console somehow.

 

[fohdeesha]

use the ping command on the switch like normal and specify the IP to source the ping from, the ip being the IP on the VE in that vlan

ping [vrf <vrf-name>] <ip addr> [source <ip addr>]

so just ping 8.8.8.8 69.69.69.69

 

[Lone Wolf]

Hey all,

I spent the day doing a lot of re-wiring. When I booted up my ICX6450-48P, two of the SFP+ ports didn't work. When I run 'show license' I get:

Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-PREM-LIC-SW H4CKTH3PLN8 Normal Invalid Unlimited 1
2 Node Lock ICX6450-10G-LIC-POD H4CKTH3PLN8 Normal Invalid Unlimited 2


Everything was working fine for the last couple of weeks. I've gone through the guide again, tried deleting the licenses, rebooting, adding them through tftp again, but always I get the above. I've tried searching through the thread but have come up empty. Suggestions?

 

[klui]

Did you change the serial to the one required by licensing v2?

 

[Lone Wolf]

Hey all,

I spent the day doing a lot of re-wiring. When I booted up my ICX6450-48P, two of the SFP+ ports didn't work. When I run 'show license' I get:

Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-PREM-LIC-SW H4CKTH3PLN8 Normal Invalid Unlimited 1
2 Node Lock ICX6450-10G-LIC-POD H4CKTH3PLN8 Normal Invalid Unlimited 2

I'm not sure why, but it seems to be working again :/ I copy/pasted the whole change-the-serial-reload- import-the-licenses dance from the guide a bunch of times, and now it started working. I don't know why it stopped, and I don't know why it started, but for the moment the status is Active again. Sun spots, maybe. Mars was transiting the house of Venus? I forgot to wiggle my little toe when I turned it all back on? Who knows. Computers. Pffft.

 

[Lone Wolf]

Did you change the serial to the one required by licensing v2?

I did, several times. The guide says:

enable
hw pid-prom serial 2ax5o2jk68e
hw pid-prom clear-sw-lid
reload

I did that, then:

enable
copy tftp license 192.168.1.8 ICX64xx/1-6450-prem.xml unit 1
copy tftp license 192.168.1.8 ICX64xx/2-6450-ports.xml unit 1
(substituting my tftps servers ip)

Then show license gave me what I posted above. I tried 'license delete unit 1 all' to clear the licenses first, which cleared the licenses but obviously didn't help. I tried 'write mem' before the reload. I ran through all that a number of times, and it just started working. Argh.

 

[klui]

Seeing how you're performing spring cleaning it's possible dust accumulation might have caused marginal reads of the serial numbers/flash. Inspect your fan in/outlets for signs of dust buildup. Take apart your 6450 and blow dust out.

 

[Lone Wolf]

Seeing how you're performing spring cleaning it's possible dust accumulation might have caused marginal reads of the serial numbers/flash. Inspect your fan in/outlets for signs of dust buildup. Take apart your 6450 and blow dust out.

I've had my 6450 for three weeks or so. It's spotless inside and out. My rewiring didn't kick up much dust since there wasn't any on the equipment and shelves around it either - it had all been cleaned before I got the switch. That's always good to keep in mind for the future tho!

 

[BobTB]

use the ping command on the switch like normal and specify the IP to source the ping from, the ip being the IP on the VE in that vlan

ping [vrf <vrf-name>] <ip addr> [source <ip addr>]

so just ping 8.8.8.8 69.69.69.69

Thank you, this works if I assign an IP to the VE. Obviously I am lacking in my understanding how this works.

What I am trying to do is have one /30 form ISP to /27 assigned to me. I was thinking to make two VLANS with one having the /30 network with its isp assigned GW and then another with my /27 routed to the /30 VLAN. I will need VE on both, right?

As what I have now is just a VLAN with nothing in it, connected to ISP and then pfsense with two interfaces doing the routing.

I am just learning this, and hope not to make some too complicated setup if a simple and fast one is better.

 

[fohdeesha]

I'm not sure why, but it seems to be working again :/ I copy/pasted the whole change-the-serial-reload- import-the-licenses dance from the guide a bunch of times, and now it started working. I don't know why it stopped, and I don't know why it started, but for the moment the status is Active again. Sun spots, maybe. Mars was transiting the house of Venus? I forgot to wiggle my little toe when I turned it all back on? Who knows. Computers. Pffft.

are they in a stack? if not, sounds like your chassis eeprom might be marginal

 

[heromode]

@fohdeesha were you not impressed by my tape mod?

drops temps 10% on the 6450

edit: i wanted to impress you now that your picture shows how pretty you are, and i thought maybe we could be an item? ;]