Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
A

anlin

New Member
Dec 8, 2016
29
8
3
30
Skud said:
Got my 7250-24P today and initially I powered it up without the console plugged in. This unit takes way longer to boot than the 6610 and I thought it was DOA since the fans weren't coming down.

I built a mini-usb to RJ45 so I could use my standard Cisco cable and then I saw that it was in fact booting, it just sits at "Starting Kernel" for a long time before moving along..

Here is the pinout for the contraption I made. Seems to be working well.
Code: 
Mini-USB           Keystone (568B)
========            ===============
1 (VCC)            N/C
2 (UART RX)        6 (Grn)
3 (UART TX)        3 (Grn/Wht)
4 (Reserved)       N/C
5 (GND)            4 (Blue)
Off to tinker!!

Thanks!!
Riley
Click to expand...
Thanks for confirming the pinout (not that I ever doubted you @fohdeesha :p)! I found a Brocade micro-USB cable listed on Ebay for less than $20 shipped, so I don't technically need to build a cable anymore but might anyways due to the adapter hell that is the official console cable kit. The cable I received didn't exactly match the pictures I found of the official Brocade cable kit, so I mapped out the connections:

Code: 
(ICX)  mUSB -> RJ45 --> DE9  (RS232)     [RJ45 wire colour]
       1       NC       NC
 (RX)  2  -->  3  --->  3    (TX)        [red]
 (TX)  3  -->  6  --->  2    (RX)        [white]
       4       NC       NC
(GND)  5  -->  5,4  ->  5,8  (GND, CTS)  [black, green]
Does this look right to you guys? Quickly going over the connections it seems RX/TX pins are correctly connected and the only difference to @fohdeesha's and @Skud's pinout is the CTS pin pulled to ground.

My switch is due to arrive on Friday, so I can't test out the cable just yet.

Edit:
For reference, here are all the identifiers I could find on the cable, the adapter and on the packaging:
B371.00842.005
60-1003495-01
FOXCONN K1607

The micro USB end of the cable and the RJ45->DE9 adapter seem visually identical to the Brocade SKU, but the RJ45 connector is clear and crimped on rather than moulded.
 
Last edited:
  • Like
Reactions: fohdeesha
S

Skud

Active Member
Jan 3, 2012
141
72
28
anlin said:
Thanks for confirming the pinout (not that I ever doubted you @fohdeesha :p)! I found a Brocade micro-USB cable listed on Ebay for less than $20 shipped, so I don't technically need to build a cable anymore but might anyways due to the adapter hell that is the official console cable kit. The cable I received didn't exactly match the pictures I found of the official Brocade cable kit, so I mapped out the connections:

Code: 
(ICX)  mUSB -> RJ45 --> DE9  (RS232)     [RJ45 wire colour]
       1       NC       NC
 (RX)  2  -->  3  --->  3    (TX)        [red]
 (TX)  3  -->  6  --->  2    (RX)        [white]
       4       NC       NC
(GND)  5  -->  5,4  ->  5,8  (GND, CTS)  [black, green]
Does this look right to you guys? Quickly going over the connections it seems RX/TX pins are correctly connected and the only difference to @fohdeesha's and @Skud's pinout is the CTS pin pulled to ground.

My switch is due to arrive on Friday, so I can't test out the cable just yet.

Edit:
For reference, here are all the identifiers I could find on the cable, the adapter and on the packaging:
B371.00842.005
60-1003495-01
FOXCONN K1607

The micro USB end of the cable and the RJ45->DE9 adapter seem visually identical to the Brocade SKU, but the RJ45 connector is clear and crimped on rather than moulded.
Click to expand...
That’s correct. Pin 4and 5 on the RJ45 side should be tied together on ground. I didnt on mine because it’s already done internally on the Cisco console cable. So, I just needed the one.
 
P

PnoT

Active Member
Mar 1, 2015
625
146
43
Texas
Has anyone seen something similar to this or know what might be causing the issue?

I currently don't have any POE devices to test with, probably will order something fast I guess, but wanted to know if this is normal.

The switch just arrived yesterday and I've went through the entire initial setup and flashing to current firmware.

Code: 
PoE: Stack unit 1 Power supply 2  with 748000 mwatts capacity is up
PoE Info: Adding new 55V capacity of 748000 mW, total capacity is 1496000, total free capacity is 1496000
PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/48 detected. Initializing....
PoE Event Trace Log Buffer for 2000 log entries allocated
PoE Event Trace Logging enabled...
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,676
2,923
113
32
fohdeesha.com
Cutha said:
Thanks for all the fantastic information @fohdeesha ! It saved me from buying a pair of MicroTik's.

My kids computers are in the basement and when I seriously started down this rabbit hole my server rack was was a rigged up 2 post with 2x4's for rear supports and a couple of comparatively quiet servers and a switch. It is now a full size rack with multiple Supermicro servers and an Infiniband switch. The heat and noise proved to much and everything is being relocated. I was trying to figure out how to bridge between the workstations and the server without sacrificing bandwidth when I saw this post and I have now purchased 2* "Brocade ICX 6610-48P 48-port 1 GbE RJ45 PoE+ Ethernet Switch ICX6610-48P".

I plan on using a "Mellanox FDR 4X Infiniband& 40G Ethernet QSFP+ 15M cable MC2207310-015" to connect the two ICX6610's. Will that work?

I could use a bit of help figuring out what to purchase to finish this up. The workstations had 1GB Cat6 connecting them to the regular network and 40GB IB connecting them to the storage on the servers. I have a bunch of Mellanox MHQH29B-XTR (flashed for RDMA) and varied lengths of QDR QSFP DAC's. What is going to be the cheapest route to connect 6 workstations to the 10GB and 40GB ports? I think the Mellanox cards I have are IB only.

The cable run for the workstations to the switch are between 15' and 30'. I am assuming that I will need to purchase new cabling and cards. My current thought is buy 4 cheap Mellanox EN cards and use a QSFP to 4x10GB SFP+ cable and for the other two workstations get 2 Mellanox 40GBE cards and use my existing QSFP DAC cables. Does this make sense? Is there a cheaper way? Any suggestion for NIC part numbers to search for?

Thanks again for all the great information.
Click to expand...

15M might be pushing it length wise for a 40gbE DAC, you'll just have to try it. Regarding the workstations, probably cheaper to just use the front regular 10gbE ports instead of using the qsfp ports with a breakout (unless they're very close or you can find a deal on a breakout)

for the 40gbE nics, search 649281-B21 on ebay (about $30) then use this post to flash them with the 40gbE ethernet firmware https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

for 10gb nics, connectx2 or connectx3 are good
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,676
2,923
113
32
fohdeesha.com
The board can be swapped relatively easily, the hard part will be finding one to replace it with (I've certainly never seen one for sale)
 
P

PnoT

Active Member
Mar 1, 2015
625
146
43
Texas
fohdeesha said:
The board can be swapped relatively easily, the hard part will be finding one to replace it with (I've certainly never seen one for sale)
Click to expand...
I was just curious if that would be an option to talk to the seller about instead of having to ship me a new switch.
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,676
2,923
113
32
fohdeesha.com
I would probably insist on a full switch replacement - if it took a bad enough zap or static discharge to damage the PoE board, who knows what condition that port or PHY is in
 
P

PnoT

Active Member
Mar 1, 2015
625
146
43
Texas
fohdeesha said:
I would probably insist on a full switch replacement - if it took a bad enough zap or static discharge to damage the PoE board, who knows what condition that port or PHY is in
Click to expand...
You're probably right and I'm trying to work out something with the seller so I can get something shipped back to me by Thanksgiving. I also took a look at the PSU and one is "A" while the other "S3".
 
Last edited:
  • Like
Reactions: fohdeesha
A

anlin

New Member
Dec 8, 2016
29
8
3
30
@fohdeesha: UPS just dropped off my -24P today. What kind of temperatures are you getting on your 7250s? After 2-3h of idling the management thermal plane on mine reads 62.4C and the POE thermal plane 33.5C. I think I might leave the fan configuration as is if the management plane temperatures are normally that high even with the fairly powerful stock fans running.

Show chassis:
Code: 
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (PoE THERMAL PLANE): 33.5 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<----->93       deg-C
                Speed 2:       82<----->105 deg-C (shutdown)
        Rule 2/2 (PoE THERMAL PLANE):
                Speed 1: NM<----->58       deg-C
                Speed 2:       49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 62.4 deg-C (Sensor 1), 33.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
 
S

Skud

Active Member
Jan 3, 2012
141
72
28
I decided to swap out the fans in my 7250-24P with something a little less "whiny". I'm not bothered by "rushing air", but I am bothered by whine - and these fans whine. It's not loud, but it's "there".. Anyways..

I had some extra fans that I put into my Aruba S2500-48P left over and they are Delta P/N:FFB0412VHN-TP03. Specs are here: http://www.delta-fan.com/download/spec/ffb0412vhn-tp03.pdf

These have about half the top-speed (9500RPM vs 18,000RPM stock) and when the switch is idling they're damn near silent even though they still move a decent amount of air. Standing 3 -4 feet from the switch you'd be hard-pressed to hear it.

I'm going to leave the switch idling for a while and see how they do (I'll pay attention to the management plane), but I think they'll do nicely.

Code: 
SSH@ICX7250-24P Router#sho chassis 
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
    Rule 1/2 (MGMT THERMAL PLANE): 69.0 deg-C
    Rule 2/2 (AIR OUTLET NEAR PSU): 26.0 deg-C

Fan speed switching temperature thresholds:
    Rule 1/2 (MGMT THERMAL PLANE):
        Speed 1: NM<-----> 93       deg-C
        Speed 2:        82<----->105 deg-C (shutdown)
    Rule 2/2 (AIR OUTLET NEAR PSU):
        Speed 1: NM<-----> 58       deg-C
        Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 69.5 deg-C (Sensor 1), 26.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA                                   
    Warning level.......: 100.0 deg-C
    Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fad.4f14
Management MAC: 609c.9fad.4f14
SSH@ICX7250-24P Router#dm fan-speed
Fan 1 Speed at 3006 RPM.
Fan 2 Speed at 2954 RPM.
SSH@ICX7250-24P Router#
 
S

Skud

Active Member
Jan 3, 2012
141
72
28
After ~30 mins of idling the management plane was 77 degrees C. I assume this is the switch ASIC? Doing some research it seems that switch ASICs tend to run warm. I mean, Brocade keeps the fans on low until it gets to 93C - which means they would feel confident in this thing running 92C all day everyday with low speed fans.

However, unless someone can tell me that this thing is really designed to live a long life that hot I'm not sure how comfortable I feel running it like that.

Thanks!!
Riley
 
mattaw

mattaw

Member
Jul 30, 2018
56
17
8
@fohdeesha - how many switches do you have now?

I have two old D-Link DXS-3250s which can take XFP 10gbit ports, the rest 1gb. No I have 3x working icx6610s, of which I only need one.
 
  • Like
Reactions: fohdeesha
You must log in or register to reply here.
Top Bottom