Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

Dave Corder

Active Member
Dec 21, 2015
209
116
43
40
fohdeesha said:
that's a bad PSU
Click to expand...
Drat, I was afraid you would confirm that. I guess I can contact the eBay seller I bought it from, but it was so long ago I doubt I have any recourse there.

Anyone have a 6610 PoE PSU they are willing to part with? :)

Not that I need it (I don't have separate feeds for them or anything; both of 'em would run off the same UPS), but it'd be nice to have.
 
arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
6610's arrived this morning and I just experienced full fan speed startup. I don't think I'll ever complain about the noise level from a 6450 again. It looks like someone was using these to run infrastructure for a Cisco training operation, I find that really amusing for some reason.
 
  • Like
Reactions: Lambdariver and fohdeesha
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
capn13 said:
One thing that confuses me is how one arrives at the optics part numbers from the official Brocade part numbers.

For example, referring to @fohdeesha's datasheet on the 6450, I see Brocade part number 10G-SFPP-SR (a 10GBASE-SR, SFP+ optic, LC, 300 m over multimode fibre optic).

And perusing STH that is equivalent to a Brocade 33011-000 transceiver....put where to those latter part numbers come from? I had a search thru ruckus but perhaps by search-fu isn't strong enough..

Can anyone enlighten me please?
Click to expand...
they started as foundry, got acquired by brocade, then by ruckus, the part numbers for optics have changed about a thousand times over the last 10 years, I wouldn't worry about it. just search ebay for "brocade 10gb sfp", they'll all work. for instance: 10G-SFPP-SR - Brocade 57-0000075-01 10Gb 850nm SWL SFP+ XBR-000180 62xAvailable | eBay
 
  • Like
Reactions: capn13
arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
What extra data are you able to pull using their optics? I think I have a couple of theirs mixed in with the pile of finisar I've received with various 10Gb NICs.
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
arglebargle said:
What extra data are you able to pull using their optics? I think I have a couple of theirs mixed in with the pile of finisar I've received with various 10Gb NICs.
Click to expand...
light levels and optic health, not a huge deal in home use:

Code: 
telnet@ICX1>sh optic 1/3/1
 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/3/1   34.7382 C  -001.5976 dBm -001.8104 dBm    7.264 mA
        Normal      Normal        Normal         Normal
 
L

Lambdariver

New Member
Mar 24, 2019
23
17
3
itronin said:
Dunno. the blue paint job looks pretty cool. Really cool overall actually! Harkens back to the day when google and symantec painted their chassis' yellow. did you connect these fans to the interior sensors? Mine's in the basement so I honestly don't care. I have to say though add a le mans racing strip and you're all set! nice job!
Click to expand...
Thanks!
I connected the side fans to each of the power supplies, the middle fan is connected to one of the fan trays, but I think I miswired the sensor as it's registering as failed and puts the switch into a bootloop unless I plug in the other fan tray. Also possible it's registering as too low of an RPM. If anyone has any ideas, I'm all ears.

I had to take off the dust filters, impeding too much flow and causing some of the temperatures to rise a bit too much.

Dreece said:
LOL... I thought for a sec you were going to flesh out the upgrade path to a "put it on 22s, and throw in an rgb strip while you're at it".... :D

If I was going to go for beauty I'd take the front front off and take it in for rose gold electro plating, chamaeleon powder coat the rest of the shell and possibly even replace the port leds with blues and shove a small tft screen with a switch to hook into the console port... now we have functional bling. :cool:
Click to expand...
That sounds amazing... hmmmmmm :)
 
T

ThreeFN

New Member
May 17, 2017
4
4
3
Figured I'd post this as an FYI.

Spent a fair few days banging my head against a LAG problem, seems to trace back to a Freenas problem.

Long and short of it is after setting up the lag on the freenas box I HAD TO RESTART. Otherwise it would be bad times. I swear I read a post on this thread nonchalantly mentioning that fact (probably from grand master fohdeesha) but I can't for the life of me find the post.

Here are the log dumps (truncated and ham-fisted sterilized so time codes are a mess). What would happen is some IP next hop flapping of the pfsense firewall (10.x.x.1) as well as my laptop (ssh to switch, 10.x.x.206) across pretty much any interface that carried more than a single device (another switch, unifi AP, pfsense, etc). Flapping the next hop between ports 'in the flow path' including the firewall port I could sort of understand but when the freenas ports were included I knew something was really, truely wrong. I'm sure this is a fun intellectual exercise for why the switch would do this, but at this point I'm in the 'glad it's fixed/not broken any more' category.

May or may not have had something to do with my jails, I swear in the first example below that the flapping started once I started to bring the jails back up. Jails are plex, syncthing, unifi controller, all using VNET.

Scenario 1
Code: 
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:33:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:32:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:32:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
Apr  1 16:45:31:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:31:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:10:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:09:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
Apr  1 16:44:56:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:44:55:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:44:51:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:44:50:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:31:03:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/1/1
Apr  1 16:29:14:I:System: Interface ethernet 1/1/1, state up      
Apr  1 16:29:12:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:17:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
Apr  1 16:29:07:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to PRIVILEGED EXEC mode using RSA as Server Host Key.
Apr  1 16:29:00:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
Apr  1 16:28:49:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:47:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:24:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:22:I:NTP: System clock is synchronized to 10.X.X.1.
Apr  1 16:28:19:I:System: PoE: Power enabled on port 1/1/1.        #setting up for upstairs Unifi AP
Apr  1 16:28:09:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
Apr  1 16:28:09:I:System: Interface ethernet 1/2/4, state up
Apr  1 16:28:08:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
Apr  1 16:28:08:I:System: Interface ethernet 1/2/5, state up
Apr  1 16:28:08:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
Apr  1 16:28:08:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
Scenario 2
Code: 
00 days 00h10m20s:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
00 days 00h10m19s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h10m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h09m54s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h09m54s:I:System: Interface ethernet 1/2/4, state up
00 days 00h09m53s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h09m53s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h09m50s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is down.
00 days 00h09m47s:I:System: dynamic lag 2 peer info (priority=32768,id=XXX,key=338) remove (LagExpiry)
00 days 00h09m43s:I:Trunk: Group (1/2/4, 1/2/5) removed by 802.3ad link-aggregation module.
00 days 00h07m59s:I:System: Interface ethernet 1/2/5, state down
00 days 00h07m57s:I:System: Interface ethernet 1/2/4, state down
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m42s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
...
00 days 00h05m17s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m14s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/3
00 days 00h05m13s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m11s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m10s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m05s:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
00 days 00h04m07s:I:System: Interface ethernet 1/1/1, state up
00 days 00h04m04s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m42s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m40s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m17s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m13s:I:System: PoE: Power enabled on port 1/1/1.
00 days 00h03m10s:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m04s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m02s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m01s:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
00 days 00h03m01s:I:System: Interface ethernet 1/2/5, state up
00 days 00h03m00s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h03m00s:I:System: Interface ethernet 1/2/4, state up
00 days 00h03m00s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h03m00s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h02m58s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/37, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/39, state up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 1  with 748000 mwatts capacity is up
00 days 00h02m57s:I:System: Interface ethernet 1/1/3, state up
00 days 00h02m57s:I:System: Interface ethernet 1/1/40, state down
00 days 00h02m57s:I:System: Interface ethernet 1/1/38, state up
00 days 00h02m56s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m55s:I:System: Interface ethernet 1/2/2, state up
00 days 00h02m55s:I:System: Interface ve 1, state up
00 days 00h02m55s:I:System: Warm start
(N/A)
In the above:
10.x.x.1 is pfsense firewall
10.x.x.206 is laptop, on wifi (yes i know) so could be coming through 1/1/1 or 1/1/3
1/1/1 is unifi ap, poe from switch
1/1/3 is link to dgs1100 switch that runs downstairs (tv area, consoles, another ap, etc)
1/2/2 is link to pfsense that firewalls wan
1/2/4 & 1/2/5 are lag to freenas
log starts at warm start, so flapping happens almost immediately. This log dump was basically after a 'reload' command of the first scenario. tried unplugging the lag as can be seen in the log, unconfig and reconfig the lag, starts right back up with the flapping.

Config file wasn't captured in my terminal dumps, but it was basically fohdeesha's guide:
unstack 1/2/1 etc, ntp, gateway, etc
plus the basic lag config:
lag XXX dynamic, ports eth 1/2/4 eth 1/2/5, primary 1/2/4, deploy
no layer 3 stuff yet, haven't gotten that far.

So yah, restart your freenas box after configuring lag, if you lag. And if you have ideas on what my logs are saying, I'm all ears and very curious, but I think I learned my lessons from this which are the usual:
start simple
test first
don't tear down what's working, add to it first
don't orphan your config interfaces
read this thread
read your logs
read your logs
by the way your switch has logs
 
  • Like
Reactions: arglebargle
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
ThreeFN said:
Figured I'd post this as an FYI.

Spent a fair few days banging my head against a LAG problem, seems to trace back to a Freenas problem.

Long and short of it is after setting up the lag on the freenas box I HAD TO RESTART. Otherwise it would be bad times. I swear I read a post on this thread nonchalantly mentioning that fact (probably from grand master fohdeesha) but I can't for the life of me find the post.

Here are the log dumps (truncated and ham-fisted sterilized so time codes are a mess). What would happen is some IP next hop flapping of the pfsense firewall (10.x.x.1) as well as my laptop (ssh to switch, 10.x.x.206) across pretty much any interface that carried more than a single device (another switch, unifi AP, pfsense, etc). Flapping the next hop between ports 'in the flow path' including the firewall port I could sort of understand but when the freenas ports were included I knew something was really, truely wrong. I'm sure this is a fun intellectual exercise for why the switch would do this, but at this point I'm in the 'glad it's fixed/not broken any more' category.

May or may not have had something to do with my jails, I swear in the first example below that the flapping started once I started to bring the jails back up. Jails are plex, syncthing, unifi controller, all using VNET.

Scenario 1
Code: 
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:33:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:32:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:32:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
Apr  1 16:45:31:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:31:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:10:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:09:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
Apr  1 16:44:56:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:44:55:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:44:51:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:44:50:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:31:03:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/1/1
Apr  1 16:29:14:I:System: Interface ethernet 1/1/1, state up     
Apr  1 16:29:12:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:17:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
Apr  1 16:29:07:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to PRIVILEGED EXEC mode using RSA as Server Host Key.
Apr  1 16:29:00:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
Apr  1 16:28:49:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:47:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:24:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:22:I:NTP: System clock is synchronized to 10.X.X.1.
Apr  1 16:28:19:I:System: PoE: Power enabled on port 1/1/1.        #setting up for upstairs Unifi AP
Apr  1 16:28:09:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
Apr  1 16:28:09:I:System: Interface ethernet 1/2/4, state up
Apr  1 16:28:08:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
Apr  1 16:28:08:I:System: Interface ethernet 1/2/5, state up
Apr  1 16:28:08:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
Apr  1 16:28:08:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
Scenario 2
Code: 
00 days 00h10m20s:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
00 days 00h10m19s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h10m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h09m54s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h09m54s:I:System: Interface ethernet 1/2/4, state up
00 days 00h09m53s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h09m53s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h09m50s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is down.
00 days 00h09m47s:I:System: dynamic lag 2 peer info (priority=32768,id=XXX,key=338) remove (LagExpiry)
00 days 00h09m43s:I:Trunk: Group (1/2/4, 1/2/5) removed by 802.3ad link-aggregation module.
00 days 00h07m59s:I:System: Interface ethernet 1/2/5, state down
00 days 00h07m57s:I:System: Interface ethernet 1/2/4, state down
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m42s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
...
00 days 00h05m17s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m14s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/3
00 days 00h05m13s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m11s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m10s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m05s:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
00 days 00h04m07s:I:System: Interface ethernet 1/1/1, state up
00 days 00h04m04s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m42s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m40s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m17s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m13s:I:System: PoE: Power enabled on port 1/1/1.
00 days 00h03m10s:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m04s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m02s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m01s:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
00 days 00h03m01s:I:System: Interface ethernet 1/2/5, state up
00 days 00h03m00s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h03m00s:I:System: Interface ethernet 1/2/4, state up
00 days 00h03m00s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h03m00s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h02m58s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/37, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/39, state up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 1  with 748000 mwatts capacity is up
00 days 00h02m57s:I:System: Interface ethernet 1/1/3, state up
00 days 00h02m57s:I:System: Interface ethernet 1/1/40, state down
00 days 00h02m57s:I:System: Interface ethernet 1/1/38, state up
00 days 00h02m56s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m55s:I:System: Interface ethernet 1/2/2, state up
00 days 00h02m55s:I:System: Interface ve 1, state up
00 days 00h02m55s:I:System: Warm start
(N/A)
In the above:
10.x.x.1 is pfsense firewall
10.x.x.206 is laptop, on wifi (yes i know) so could be coming through 1/1/1 or 1/1/3
1/1/1 is unifi ap, poe from switch
1/1/3 is link to dgs1100 switch that runs downstairs (tv area, consoles, another ap, etc)
1/2/2 is link to pfsense that firewalls wan
1/2/4 & 1/2/5 are lag to freenas
log starts at warm start, so flapping happens almost immediately. This log dump was basically after a 'reload' command of the first scenario. tried unplugging the lag as can be seen in the log, unconfig and reconfig the lag, starts right back up with the flapping.

Config file wasn't captured in my terminal dumps, but it was basically fohdeesha's guide:
unstack 1/2/1 etc, ntp, gateway, etc
plus the basic lag config:
lag XXX dynamic, ports eth 1/2/4 eth 1/2/5, primary 1/2/4, deploy
no layer 3 stuff yet, haven't gotten that far.

So yah, restart your freenas box after configuring lag, if you lag. And if you have ideas on what my logs are saying, I'm all ears and very curious, but I think I learned my lessons from this which are the usual:
start simple
test first
don't tear down what's working, add to it first
don't orphan your config interfaces
read this thread
read your logs
read your logs
by the way your switch has logs
Click to expand...
yeah it's weird, you probably did see me mention something along those lines half a year ago. I was helping out another member on teamviewer debug his freenas issues (he wasn't even using brocade, was using an lb4m if I remember right). Nothing I did would remedy the flapping, but asking him to do a simple reboot of the freenas machine (he hadn't done so since setting it up) and it's worked perfectly ever since
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
Lambdariver said:
but I think I miswired the sensor as it's registering as failed and puts the switch into a bootloop unless I plug in the other fan tray. Also possible it's registering as too low of an RPM. If anyone has any ideas, I'm all ears
Click to expand...
as mentioned in previous posts, there's a hardware watchdog IC that monitors the fan trays (and everything else) and will mark the fans as failed if they're not reading like +/- 20% of what it expects (and on the 6650 fan trays that's something like 10k RPM on fan speed 1, 18k on fan speed 2). as you noticed, if it marks the fans failed it won't allow a boot
 
arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Re: freenas and pfsense and LAG/VLAN.

I've run into issues with the networking stack in freebsd that were completely baffling like this. What I've learned to do as a mitigation on every freebsd derivative is reboot after changing anything related to low level networking. Making a change that involves a LAG? Reboot. Making a change that involves a VLAN? Reboot.

I spent a couple of days diagnosing connectivity issues when I switched from using a flat non-vlan home lan layout to multi-vlan a couple of years ago. I eventually rebooted pfsense for some other reason and everything immediately started working. The same thing happened when I started playing with LAGs.

I never really thought about posting this before but it's probably something that should be spread around because it seems to bite a lot of people.

Even something simple like changing the LAN interface on pfsense from the nic to nic:vlan1 (with the connected port untagged 1 on the switch) will cause pfsense to lose connectivity to VLAN 1 until you reboot.
 
Last edited:
  • Like
Reactions: capn13 and fohdeesha
arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Hey @fohdeesha - do you remember what voltage the jtag on the 6610 uses? I have a "bus blaster" jtag debugger in the mail for another project, I thought I'd have a go at dumping the flash and recovering the shipped licenses on my switches too.
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
arglebargle said:
Hey @fohdeesha - do you remember what voltage the jtag on the 6610 uses? I have a "bus blaster" jtag debugger in the mail for another project, I thought I'd have a go at dumping the flash and recovering the shipped licenses on my switches too.
Click to expand...
the 6610 is PowerPC architecture, you'll need a PPC specific JTAG unit for it. The ARM/CPLD/etc generics like bus blaster won't be able to do anything. Pretty much the cheapest PPC unit you'll find is the BDI2000
 
fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
arglebargle said:
Hey @fohdeesha - do you remember what voltage the jtag on the 6610 uses? I have a "bus blaster" jtag debugger in the mail for another project, I thought I'd have a go at dumping the flash and recovering the shipped licenses on my switches too.
Click to expand...

however there's a trick to dump licenses without a JTAG unit, you can send a command in the bootloader to dump the entirety of flash to terminal, and set putty to log to a file. run it overnight, search for the lic string beginnings in the resulting file
 
T

ThreeFN

New Member
May 17, 2017
4
4
3
So did a fan mod on the 6450-24P that's going run the downstairs. Waffled back and forth but decided to give some NF-A4x20 FLX a try and work up from there if I had cooling issues since the switch is in an acoustically sub-optimal location. No issues with the fans not starting/stopping (I know that some others on the thread had experienced the opposite), my unit's fan speed 1 was clocking 7V on both fans. You don't even have to change the fan pinout like you do with the Sunon units, the chassis pinout was the same as PC fan pinout. Used the NA-SAV4 rubber fan mounts hopping to kill vibration, may have been a bit overkill. The fans are actually more audible at speed 1 than speed 2, due to PWM noise (must be an awfully chunky PWM waveform).Casing is a little bent from the SAV4's not being pan-head screw flush, so I didn't tighten the snot out of the lid screws. Working just fine, but there's no PoE load yet and there won't be much of one once deployed. Baked the switch in my noise isolation enclosure with the rest of the rack overnight, got a high of 61.5C. So yah, thumbs up for NF-A4x20 FLX fan swap on the 6450-24P, but YMMV.


My 6610 also in rack, left in show chassis below for reference:
Code: 
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000141-02
        Serial Number:  217
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Back to Front
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000141-02
        Serial Number:  1Y0
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Back to Front

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 54.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->77       deg-C
                Speed 2:       72<-----> 85 deg-C (shutdown)

Fan 1 Air Flow Direction:  Back to Front
Fan 2 Air Flow Direction:  Back to Front
MAC 1 Temperature Readings:
        Current temperature : 46.0 deg-C
MAC 2 Temperature Readings:
        Current temperature : 54.0 deg-C
CPU Temperature Readings:
        Current temperature : 51.0 deg-C
sensor A Temperature Readings:
        Current temperature : 51.0 deg-C
sensor B Temperature Readings:
        Current temperature : 42.0 deg-C
sensor C Temperature Readings:
        Current temperature : 39.0 deg-C
stacking card Temperature Readings:
        Current temperature : 49.0 deg-C
        Warning level.......: 82.0 deg-C
        Shutdown level......: 85.0 deg-C
Boot Prom MAC : XXX
Management MAC: XXX

The stack unit 2 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present

Fan 1 failed
Fan 2 failed

Fan controlled temperature: 61.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->64       deg-C
                Speed 2:       59<-----> 88 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 57.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 61.5 deg-C
        Warning level.......: 78.0 deg-C
        Shutdown level......: 88.0 deg-C
Also mix-stacked the 6450 to the 6610 (1/3/1 to 2/2/1) via secure-stack/auto/manual, in all cases 1/2/1 and 1/2/6 get turned into stack ports unnecessarily. could unstack one of the 40gb, but get 'can't unstack last stacking port' error so looks like even if you're mix stacking and don't need the 40gb to stack to another 6610, you still lose at least one of these ports to the job. It's not a problem in my current setup, and when I want that second 40gb port back, I'll config the 6450 and do the trunk by hand. Just didn't want to have to traffic manage the uplink between the switches if I didn't have to.
Code: 
T=13h17m57.8: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  xxxx.xxxx.xxxx   0 local   Ready
2  S ICX6450-24P   member  xxxx.xxxx.xxxx   0 remote  Ready

    active
     +---+
     | 1 |2/6
     +---+

    active
      ---         +---+
     ( 1 )3/1--2/1| 2 |
      ---         +---+
Current stack management MAC is xxx

Image-Auto-Copy is Enabled.

      Stack Port Status                        Neighbors
Unit# Stack-port1         Stack-port2          Stack-port1         Stack-port2
1     none                dn (1/2/6)           none                none
2     up (2/2/1)          none                 U1 (1/3/1)          none

Unit# System uptime
1     13 hour(s) 17 minute(s) 57 second(s)
2     12 hour(s) 48 minute(s) 49 second(s)
The system started at 18:15:20 GMT-05 Wed Apr 03 2019
 
Last edited:
  • Like
Reactions: arglebargle
arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
ThreeFN said:
So did a fan mod on the 6450-24P that's going run the downstairs. Waffled back and forth but decided to give some NF-A4x20 FLX a try and work up from there if I had cooling issues since the switch is in an acoustically sub-optimal location. No issues with the fans not starting/stopping (I know that some others on the thread had experienced the opposite), my unit's fan speed 1 was clocking 7V on both fans. You don't even have to change the fan pinout like you do with the Sunon units, the chassis pinout was the same as PC fan pinout. Used the NA-SAV4 rubber fan mounts hopping to kill vibration, may have been a bit overkill. The fans are actually more audible at speed 1 than speed 2, due to PWM noise (must be an awfully chunky PWM waveform).
Click to expand...
I went back and double checked my 6450 non-poe in my office and I completely forgot to swap pins on the Sunon in this switch when I installed it. Whoops, no wonder it was completely silent. High temp over the last month was only 62°C so no harm done but I swapped pins so the fan will actually function.

Is there any way to address the pwm noise at low speed on these? I'm willing to cobble together some simple electronics inside the case to quiet things back down, the pwm noise is actually really irritating in a quiet room.
 
You must log in or register to reply here.
Top Bottom