Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

Your first image is a bog standard dual gigabit Ethernet card. The next image 4x LC fiber connectors on it, in the form of Small Form Pluggable (SFP) optics. It looks like the quad 10 Gb rNDC.

Next a quad ported RJ45 rNDC, a 2x 10Gbe + 2x 1Gbe model. The add-in card in with the protective covers is a dual 8 Gb Fibre Channel card; a lot of SANs use/used it; performant iSCSI is relatively recent, comparatively.

You'll need to either pull the Dell part numbers off the cards, or boot the system with a Linux live CD or equivalent to pull the models and get more information. For example, some 10 Gb network cards are 'converged' in that they can do Ethernet and Fibre Channel, rather than having dedicated cards for each.

Isilon X210's apparently can be configured as iSCSI (Ethernet) or Fibre Channel.

Your Mellanox IS5022's are 8x 40 Gb QSFP interfaces; I don't see any QSFP interfaces in your pictures beyond those. It's a managed Infiniband switch; yet another protocol. The IS5022 does not support any other protocols.

I do believe your first step should be to trace and tag each cable; I would be surprised if the Isilon storage servers were using Infiniband.

 

[Rand__]

You'll need to either pull the Dell part numbers off the cards, or boot the system with a Linux live CD or equivalent to pull the models and get more information.

I'd start with running the service tags at Dell's - unless somebody modified the hardware aftersales (depends on the history of the boxes, larger companies usually dont) u can see what the box was sold with - the depicted server was sold with a ( G8RPD ) "QLogic 57800 2x10Gb BT + 2x1Gb BT Network Daughter Card " which is a " QLogic 57800 2x10Gb DA/SFP+ "


That box is still under support by the way so you also can simply ask Dell to help you out (unless you intended to buy the needed components aftermarket)

 

[heromode]

ⓣⓐⓟⓔ ⓜⓞⓓ

Hi, finally got to flash and license and setup my icx6450-48p, thanks to @fohdeesha documentation and everyone in thread..

On the ICX6450 the most basic fan mod is by removing two of the three fans. You should definitely cover the slots on the two empty fan locations with a piece of tape. Without the tape a lot of the air pushed out by the single fan is just recirculated via the adjacent empty fan slots, ruining the negative pressure needed to pull air out from the front of the case:

Adding the tape creates the correct negative pressure, pulling out the air from the rest of the case. This results in a dramatic drop in temps, in my test the temp dropped 5.5 6.5 degrees 5.5 Celsius in about 30 minutes 60 minutes!



2x SFP+ ports connected, 2x ethernet ports connected, all idle, single stock fan in the slot closest to corner
Room ambient temperature 25 Celsius:


Before TapeMod

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 60.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 54.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 60.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

After TapeMod

#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 55.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 55.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

edit: now another degree after 60 mins:

Fan controlled temperature: 54.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->65 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
Current temperature : 49.0 deg-C
Sensor A Temperature Readings:
Current temperature : 54.0 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-C

edit2: now stabilized at 55, a 5.5 degree reduction.

 

[BobTB]

My ISP gave me a /30 with a routed /27 available for my use. I was using pfsense in pure router mode to connect the /30 with /27 network, with one WAN and one "local wan" going back to ICX VLAN, where I then connect the other routers to it.

Now I got the idea to go and use my icx-6610 as a router for the above and just connect my other routers to the ports directly with completely eliminating this first pfsense doing the /30 to /27.

Will this work? Just add route and I am done?

 

[kpfleming]

My ISP gave me a /30 with a routed /27 available for my use. I was using pfsense in pure router mode to connect the /30 with /27 network, with one WAN and one "local wan" going back to ICX VLAN, where I then connect the other routers to it.

Now I got the idea to go and use my icx-6610 as a router for the above and just connect my other routers to the ports directly with completely eliminating this first pfsense doing the /30 to /27.

Will this work? Just add route and I am done?

Yes, that is just basic IPv4 routing and should work fine.

 

[BobTB]

Yes, that is just basic IPv4 routing and should work fine.

Thx, will try to do this.

Now the question I have is how can I go to a specific VLAN and ping from there when I am SSHed to the switch. I would like to "go to" the VLAN my ISP is connected to and try to ping their gateway. Or am I thinking about this wrong?

I can do it from pfsense, when it is connected to this VLAN, I was just trying to see if I can do this from the ICX console somehow.

 

[fohdeesha]

use the ping command on the switch like normal and specify the IP to source the ping from, the ip being the IP on the VE in that vlan

ping [vrf <vrf-name>] <ip addr> [source <ip addr>]

so just ping 8.8.8.8 69.69.69.69

 

[Lone Wolf]

Hey all,

I spent the day doing a lot of re-wiring. When I booted up my ICX6450-48P, two of the SFP+ ports didn't work. When I run 'show license' I get:

Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-PREM-LIC-SW H4CKTH3PLN8 Normal Invalid Unlimited 1
2 Node Lock ICX6450-10G-LIC-POD H4CKTH3PLN8 Normal Invalid Unlimited 2


Everything was working fine for the last couple of weeks. I've gone through the guide again, tried deleting the licenses, rebooting, adding them through tftp again, but always I get the above. I've tried searching through the thread but have come up empty. Suggestions?

 

[klui]

Did you change the serial to the one required by licensing v2?

 

[Lone Wolf]

Hey all,

I spent the day doing a lot of re-wiring. When I booted up my ICX6450-48P, two of the SFP+ ports didn't work. When I run 'show license' I get:

Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-PREM-LIC-SW H4CKTH3PLN8 Normal Invalid Unlimited 1
2 Node Lock ICX6450-10G-LIC-POD H4CKTH3PLN8 Normal Invalid Unlimited 2

I'm not sure why, but it seems to be working again :/ I copy/pasted the whole change-the-serial-reload- import-the-licenses dance from the guide a bunch of times, and now it started working. I don't know why it stopped, and I don't know why it started, but for the moment the status is Active again. Sun spots, maybe. Mars was transiting the house of Venus? I forgot to wiggle my little toe when I turned it all back on? Who knows. Computers. Pffft.

 

[Lone Wolf]

Did you change the serial to the one required by licensing v2?

I did, several times. The guide says:

enable
hw pid-prom serial 2ax5o2jk68e
hw pid-prom clear-sw-lid
reload

I did that, then:

enable
copy tftp license 192.168.1.8 ICX64xx/1-6450-prem.xml unit 1
copy tftp license 192.168.1.8 ICX64xx/2-6450-ports.xml unit 1
(substituting my tftps servers ip)

Then show license gave me what I posted above. I tried 'license delete unit 1 all' to clear the licenses first, which cleared the licenses but obviously didn't help. I tried 'write mem' before the reload. I ran through all that a number of times, and it just started working. Argh.

 

[klui]

Seeing how you're performing spring cleaning it's possible dust accumulation might have caused marginal reads of the serial numbers/flash. Inspect your fan in/outlets for signs of dust buildup. Take apart your 6450 and blow dust out.

 

[Lone Wolf]

Seeing how you're performing spring cleaning it's possible dust accumulation might have caused marginal reads of the serial numbers/flash. Inspect your fan in/outlets for signs of dust buildup. Take apart your 6450 and blow dust out.

I've had my 6450 for three weeks or so. It's spotless inside and out. My rewiring didn't kick up much dust since there wasn't any on the equipment and shelves around it either - it had all been cleaned before I got the switch. That's always good to keep in mind for the future tho!

 

[BobTB]

use the ping command on the switch like normal and specify the IP to source the ping from, the ip being the IP on the VE in that vlan

ping [vrf <vrf-name>] <ip addr> [source <ip addr>]

so just ping 8.8.8.8 69.69.69.69

Thank you, this works if I assign an IP to the VE. Obviously I am lacking in my understanding how this works.

What I am trying to do is have one /30 form ISP to /27 assigned to me. I was thinking to make two VLANS with one having the /30 network with its isp assigned GW and then another with my /27 routed to the /30 VLAN. I will need VE on both, right?

As what I have now is just a VLAN with nothing in it, connected to ISP and then pfsense with two interfaces doing the routing.

I am just learning this, and hope not to make some too complicated setup if a simple and fast one is better.

 

[fohdeesha]

I'm not sure why, but it seems to be working again :/ I copy/pasted the whole change-the-serial-reload- import-the-licenses dance from the guide a bunch of times, and now it started working. I don't know why it stopped, and I don't know why it started, but for the moment the status is Active again. Sun spots, maybe. Mars was transiting the house of Venus? I forgot to wiggle my little toe when I turned it all back on? Who knows. Computers. Pffft.

are they in a stack? if not, sounds like your chassis eeprom might be marginal

 

[heromode]

@fohdeesha were you not impressed by my tape mod?

drops temps 10% on the 6450

edit: i wanted to impress you now that your picture shows how pretty you are, and i thought maybe we could be an item? ;]

 

[fohdeesha]

@fohdeesha were you not impressed by my tape mod?

drops temps 10% on the 6450

edit: i wanted to impress you now that your picture shows how pretty you are, and i thought maybe we could be an item? ;]

 

[kpfleming]

Thank you, this works if I assign an IP to the VE. Obviously I am lacking in my understanding how this works.

What I am trying to do is have one /30 form ISP to /27 assigned to me. I was thinking to make two VLANS with one having the /30 network with its isp assigned GW and then another with my /27 routed to the /30 VLAN. I will need VE on both, right?

As what I have now is just a VLAN with nothing in it, connected to ISP and then pfsense with two interfaces doing the routing.

I am just learning this, and hope not to make some too complicated setup if a simple and fast one is better.

Yes, you will need a VE in both VLANs. One will have your ISP-assigned endpoint address from the /30 subnet, and the other will have one of the addresses from the /27 (which will be the gateway address for the other devices on that VLAN).

 

[Lone Wolf]

are they in a stack? if not, sounds like your chassis eeprom might be marginal

I only have the one, so no stack. If the eeprom in my 6450 is marginal, is it replaceable? I'd hate to have to fight with it every time it gets unplugged.

 

[fohdeesha]

I only have the one, so no stack. If the eeprom in my 6450 is marginal, is it replaceable? I'd hate to have to fight with it every time it gets unplugged.

it's soldered to the board so not easily, but I doubt the EEPROM itself is going bad. Any writes to it (like serial eeprom commands) don't take effect until next boot, part of me wonders if you or someone else had ran a bad serial command in the past while they were running, and it didn't **** up your license until you just now unplugged and replugged it. I'd power cycle them again to check, I have a feeling it'll be perfectly fine