Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
What extra data are you able to pull using their optics? I think I have a couple of theirs mixed in with the pile of finisar I've received with various 10Gb NICs.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
What extra data are you able to pull using their optics? I think I have a couple of theirs mixed in with the pile of finisar I've received with various 10Gb NICs.
light levels and optic health, not a huge deal in home use:

Code:
telnet@ICX1>sh optic 1/3/1
 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/3/1   34.7382 C  -001.5976 dBm -001.8104 dBm    7.264 mA
        Normal      Normal        Normal         Normal
 

Lambdariver

New Member
Mar 24, 2019
22
15
3
Dunno. the blue paint job looks pretty cool. Really cool overall actually! Harkens back to the day when google and symantec painted their chassis' yellow. did you connect these fans to the interior sensors? Mine's in the basement so I honestly don't care. I have to say though add a le mans racing strip and you're all set! nice job!
Thanks!
I connected the side fans to each of the power supplies, the middle fan is connected to one of the fan trays, but I think I miswired the sensor as it's registering as failed and puts the switch into a bootloop unless I plug in the other fan tray. Also possible it's registering as too low of an RPM. If anyone has any ideas, I'm all ears.

I had to take off the dust filters, impeding too much flow and causing some of the temperatures to rise a bit too much.

LOL... I thought for a sec you were going to flesh out the upgrade path to a "put it on 22s, and throw in an rgb strip while you're at it".... :D

If I was going to go for beauty I'd take the front front off and take it in for rose gold electro plating, chamaeleon powder coat the rest of the shell and possibly even replace the port leds with blues and shove a small tft screen with a switch to hook into the console port... now we have functional bling. :cool:
That sounds amazing... hmmmmmm :)
 

ThreeFN

New Member
May 17, 2017
4
4
3
Figured I'd post this as an FYI.

Spent a fair few days banging my head against a LAG problem, seems to trace back to a Freenas problem.

Long and short of it is after setting up the lag on the freenas box I HAD TO RESTART. Otherwise it would be bad times. I swear I read a post on this thread nonchalantly mentioning that fact (probably from grand master fohdeesha) but I can't for the life of me find the post.

Here are the log dumps (truncated and ham-fisted sterilized so time codes are a mess). What would happen is some IP next hop flapping of the pfsense firewall (10.x.x.1) as well as my laptop (ssh to switch, 10.x.x.206) across pretty much any interface that carried more than a single device (another switch, unifi AP, pfsense, etc). Flapping the next hop between ports 'in the flow path' including the firewall port I could sort of understand but when the freenas ports were included I knew something was really, truely wrong. I'm sure this is a fun intellectual exercise for why the switch would do this, but at this point I'm in the 'glad it's fixed/not broken any more' category.

May or may not have had something to do with my jails, I swear in the first example below that the flapping started once I started to bring the jails back up. Jails are plex, syncthing, unifi controller, all using VNET.

Scenario 1
Code:
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:33:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:32:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:32:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
Apr  1 16:45:31:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:31:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:10:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:09:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
Apr  1 16:44:56:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:44:55:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:44:51:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:44:50:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:31:03:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/1/1
Apr  1 16:29:14:I:System: Interface ethernet 1/1/1, state up      
Apr  1 16:29:12:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:17:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
Apr  1 16:29:07:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to PRIVILEGED EXEC mode using RSA as Server Host Key.
Apr  1 16:29:00:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
Apr  1 16:28:49:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:47:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:24:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:22:I:NTP: System clock is synchronized to 10.X.X.1.
Apr  1 16:28:19:I:System: PoE: Power enabled on port 1/1/1.        #setting up for upstairs Unifi AP
Apr  1 16:28:09:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
Apr  1 16:28:09:I:System: Interface ethernet 1/2/4, state up
Apr  1 16:28:08:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
Apr  1 16:28:08:I:System: Interface ethernet 1/2/5, state up
Apr  1 16:28:08:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
Apr  1 16:28:08:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
Scenario 2
Code:
00 days 00h10m20s:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
00 days 00h10m19s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h10m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h09m54s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h09m54s:I:System: Interface ethernet 1/2/4, state up
00 days 00h09m53s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h09m53s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h09m50s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is down.
00 days 00h09m47s:I:System: dynamic lag 2 peer info (priority=32768,id=XXX,key=338) remove (LagExpiry)
00 days 00h09m43s:I:Trunk: Group (1/2/4, 1/2/5) removed by 802.3ad link-aggregation module.
00 days 00h07m59s:I:System: Interface ethernet 1/2/5, state down
00 days 00h07m57s:I:System: Interface ethernet 1/2/4, state down
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m42s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
...
00 days 00h05m17s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m14s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/3
00 days 00h05m13s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m11s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m10s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m05s:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
00 days 00h04m07s:I:System: Interface ethernet 1/1/1, state up
00 days 00h04m04s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m42s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m40s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m17s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m13s:I:System: PoE: Power enabled on port 1/1/1.
00 days 00h03m10s:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m04s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m02s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m01s:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
00 days 00h03m01s:I:System: Interface ethernet 1/2/5, state up
00 days 00h03m00s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h03m00s:I:System: Interface ethernet 1/2/4, state up
00 days 00h03m00s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h03m00s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h02m58s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/37, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/39, state up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 1  with 748000 mwatts capacity is up
00 days 00h02m57s:I:System: Interface ethernet 1/1/3, state up
00 days 00h02m57s:I:System: Interface ethernet 1/1/40, state down
00 days 00h02m57s:I:System: Interface ethernet 1/1/38, state up
00 days 00h02m56s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m55s:I:System: Interface ethernet 1/2/2, state up
00 days 00h02m55s:I:System: Interface ve 1, state up
00 days 00h02m55s:I:System: Warm start
(N/A)
In the above:
10.x.x.1 is pfsense firewall
10.x.x.206 is laptop, on wifi (yes i know) so could be coming through 1/1/1 or 1/1/3
1/1/1 is unifi ap, poe from switch
1/1/3 is link to dgs1100 switch that runs downstairs (tv area, consoles, another ap, etc)
1/2/2 is link to pfsense that firewalls wan
1/2/4 & 1/2/5 are lag to freenas
log starts at warm start, so flapping happens almost immediately. This log dump was basically after a 'reload' command of the first scenario. tried unplugging the lag as can be seen in the log, unconfig and reconfig the lag, starts right back up with the flapping.

Config file wasn't captured in my terminal dumps, but it was basically fohdeesha's guide:
unstack 1/2/1 etc, ntp, gateway, etc
plus the basic lag config:
lag XXX dynamic, ports eth 1/2/4 eth 1/2/5, primary 1/2/4, deploy
no layer 3 stuff yet, haven't gotten that far.

So yah, restart your freenas box after configuring lag, if you lag. And if you have ideas on what my logs are saying, I'm all ears and very curious, but I think I learned my lessons from this which are the usual:
start simple
test first
don't tear down what's working, add to it first
don't orphan your config interfaces
read this thread
read your logs
read your logs
by the way your switch has logs
 
  • Like
Reactions: arglebargle

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Figured I'd post this as an FYI.

Spent a fair few days banging my head against a LAG problem, seems to trace back to a Freenas problem.

Long and short of it is after setting up the lag on the freenas box I HAD TO RESTART. Otherwise it would be bad times. I swear I read a post on this thread nonchalantly mentioning that fact (probably from grand master fohdeesha) but I can't for the life of me find the post.

Here are the log dumps (truncated and ham-fisted sterilized so time codes are a mess). What would happen is some IP next hop flapping of the pfsense firewall (10.x.x.1) as well as my laptop (ssh to switch, 10.x.x.206) across pretty much any interface that carried more than a single device (another switch, unifi AP, pfsense, etc). Flapping the next hop between ports 'in the flow path' including the firewall port I could sort of understand but when the freenas ports were included I knew something was really, truely wrong. I'm sure this is a fun intellectual exercise for why the switch would do this, but at this point I'm in the 'glad it's fixed/not broken any more' category.

May or may not have had something to do with my jails, I swear in the first example below that the flapping started once I started to bring the jails back up. Jails are plex, syncthing, unifi controller, all using VNET.

Scenario 1
Code:
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:35:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:34:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:33:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:32:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:32:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
Apr  1 16:45:31:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:31:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:45:25:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:45:10:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:45:09:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
Apr  1 16:45:01:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
Apr  1 16:44:56:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
Apr  1 16:44:55:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
Apr  1 16:44:51:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
Apr  1 16:44:50:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
Apr  1 16:31:03:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/1/1
Apr  1 16:29:14:I:System: Interface ethernet 1/1/1, state up     
Apr  1 16:29:12:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:17:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
Apr  1 16:29:07:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to PRIVILEGED EXEC mode using RSA as Server Host Key.
Apr  1 16:29:00:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
Apr  1 16:28:49:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:47:I:System: Interface ethernet 1/1/1, state down
Apr  1 16:28:24:I:System: Interface ethernet 1/1/1, state up
Apr  1 16:28:22:I:NTP: System clock is synchronized to 10.X.X.1.
Apr  1 16:28:19:I:System: PoE: Power enabled on port 1/1/1.        #setting up for upstairs Unifi AP
Apr  1 16:28:09:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
Apr  1 16:28:09:I:System: Interface ethernet 1/2/4, state up
Apr  1 16:28:08:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
Apr  1 16:28:08:I:System: Interface ethernet 1/2/5, state up
Apr  1 16:28:08:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
Apr  1 16:28:08:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
Scenario 2
Code:
00 days 00h10m20s:D:next hop router 10.X.X.1 moved from port 1/2/4 to port 1/2/2
00 days 00h10m19s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/4
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h10m13s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h10m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h09m54s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h09m54s:I:System: Interface ethernet 1/2/4, state up
00 days 00h09m53s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h09m53s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h09m50s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is down.
00 days 00h09m47s:I:System: dynamic lag 2 peer info (priority=32768,id=XXX,key=338) remove (LagExpiry)
00 days 00h09m43s:I:Trunk: Group (1/2/4, 1/2/5) removed by 802.3ad link-aggregation module.
00 days 00h07m59s:I:System: Interface ethernet 1/2/5, state down
00 days 00h07m57s:I:System: Interface ethernet 1/2/4, state down
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m43s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m42s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m41s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/4
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/1
00 days 00h07m40s:D:next hop router 10.X.X.206 moved from port 1/1/1 to port 1/2/5
...
00 days 00h05m17s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m14s:D:next hop router 10.X.X.206 moved from port 1/2/5 to port 1/1/3
00 days 00h05m13s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/5
00 days 00h05m11s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m10s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m09s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/2/4 to port 1/1/3
00 days 00h05m08s:D:next hop router 10.X.X.206 moved from port 1/1/3 to port 1/2/4
00 days 00h05m05s:I:Security: SSH login by public key from src IP 10.X.X.206 from src MAC XXX to USER EXEC mode using RSA as Server Host Key.
00 days 00h04m07s:I:System: Interface ethernet 1/1/1, state up
00 days 00h04m04s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m42s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m40s:I:System: Interface ethernet 1/1/1, state down
00 days 00h03m17s:I:System: Interface ethernet 1/1/1, state up
00 days 00h03m13s:I:System: PoE: Power enabled on port 1/1/1.
00 days 00h03m10s:I:System: PoE: Allocated power of 30000 mwatts on port 1/1/1.
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m05s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m04s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m03s:D:next hop router 10.X.X.1 moved from port 1/2/2 to port 1/2/5
00 days 00h03m02s:D:next hop router 10.X.X.1 moved from port 1/2/5 to port 1/2/2
00 days 00h03m01s:I:System: Logical link on dynamic lag interface ethernet 1/2/5 is up.
00 days 00h03m01s:I:System: Interface ethernet 1/2/5, state up
00 days 00h03m00s:I:System: Logical link on dynamic lag interface ethernet 1/2/4 is up.
00 days 00h03m00s:I:System: Interface ethernet 1/2/4, state up
00 days 00h03m00s:I:Trunk: Group (1/2/4, 1/2/5) created by 802.3ad link-aggregation module.
00 days 00h03m00s:I:System: dynamic lag 2, has new peer info (priority=32768,id=XXX,key=338) (N/A)
00 days 00h02m58s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/37, state up
00 days 00h02m58s:I:System: Interface ethernet 1/1/39, state up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 2  with 748000 mwatts capacity is up
00 days 00h02m58s:I:System: Stack unit 1 POE  Power supply 1  with 748000 mwatts capacity is up
00 days 00h02m57s:I:System: Interface ethernet 1/1/3, state up
00 days 00h02m57s:I:System: Interface ethernet 1/1/40, state down
00 days 00h02m57s:I:System: Interface ethernet 1/1/38, state up
00 days 00h02m56s:I:System: Interface ethernet 1/1/40, state up
00 days 00h02m55s:I:System: Interface ethernet 1/2/2, state up
00 days 00h02m55s:I:System: Interface ve 1, state up
00 days 00h02m55s:I:System: Warm start
(N/A)
In the above:
10.x.x.1 is pfsense firewall
10.x.x.206 is laptop, on wifi (yes i know) so could be coming through 1/1/1 or 1/1/3
1/1/1 is unifi ap, poe from switch
1/1/3 is link to dgs1100 switch that runs downstairs (tv area, consoles, another ap, etc)
1/2/2 is link to pfsense that firewalls wan
1/2/4 & 1/2/5 are lag to freenas
log starts at warm start, so flapping happens almost immediately. This log dump was basically after a 'reload' command of the first scenario. tried unplugging the lag as can be seen in the log, unconfig and reconfig the lag, starts right back up with the flapping.

Config file wasn't captured in my terminal dumps, but it was basically fohdeesha's guide:
unstack 1/2/1 etc, ntp, gateway, etc
plus the basic lag config:
lag XXX dynamic, ports eth 1/2/4 eth 1/2/5, primary 1/2/4, deploy
no layer 3 stuff yet, haven't gotten that far.

So yah, restart your freenas box after configuring lag, if you lag. And if you have ideas on what my logs are saying, I'm all ears and very curious, but I think I learned my lessons from this which are the usual:
start simple
test first
don't tear down what's working, add to it first
don't orphan your config interfaces
read this thread
read your logs
read your logs
by the way your switch has logs
yeah it's weird, you probably did see me mention something along those lines half a year ago. I was helping out another member on teamviewer debug his freenas issues (he wasn't even using brocade, was using an lb4m if I remember right). Nothing I did would remedy the flapping, but asking him to do a simple reboot of the freenas machine (he hadn't done so since setting it up) and it's worked perfectly ever since
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
but I think I miswired the sensor as it's registering as failed and puts the switch into a bootloop unless I plug in the other fan tray. Also possible it's registering as too low of an RPM. If anyone has any ideas, I'm all ears
as mentioned in previous posts, there's a hardware watchdog IC that monitors the fan trays (and everything else) and will mark the fans as failed if they're not reading like +/- 20% of what it expects (and on the 6650 fan trays that's something like 10k RPM on fan speed 1, 18k on fan speed 2). as you noticed, if it marks the fans failed it won't allow a boot
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Re: freenas and pfsense and LAG/VLAN.

I've run into issues with the networking stack in freebsd that were completely baffling like this. What I've learned to do as a mitigation on every freebsd derivative is reboot after changing anything related to low level networking. Making a change that involves a LAG? Reboot. Making a change that involves a VLAN? Reboot.

I spent a couple of days diagnosing connectivity issues when I switched from using a flat non-vlan home lan layout to multi-vlan a couple of years ago. I eventually rebooted pfsense for some other reason and everything immediately started working. The same thing happened when I started playing with LAGs.

I never really thought about posting this before but it's probably something that should be spread around because it seems to bite a lot of people.

Even something simple like changing the LAN interface on pfsense from the nic to nic:vlan1 (with the connected port untagged 1 on the switch) will cause pfsense to lose connectivity to VLAN 1 until you reboot.
 
Last edited:

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Hey @fohdeesha - do you remember what voltage the jtag on the 6610 uses? I have a "bus blaster" jtag debugger in the mail for another project, I thought I'd have a go at dumping the flash and recovering the shipped licenses on my switches too.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Hey @fohdeesha - do you remember what voltage the jtag on the 6610 uses? I have a "bus blaster" jtag debugger in the mail for another project, I thought I'd have a go at dumping the flash and recovering the shipped licenses on my switches too.
the 6610 is PowerPC architecture, you'll need a PPC specific JTAG unit for it. The ARM/CPLD/etc generics like bus blaster won't be able to do anything. Pretty much the cheapest PPC unit you'll find is the BDI2000
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Hey @fohdeesha - do you remember what voltage the jtag on the 6610 uses? I have a "bus blaster" jtag debugger in the mail for another project, I thought I'd have a go at dumping the flash and recovering the shipped licenses on my switches too.

however there's a trick to dump licenses without a JTAG unit, you can send a command in the bootloader to dump the entirety of flash to terminal, and set putty to log to a file. run it overnight, search for the lic string beginnings in the resulting file
 

ThreeFN

New Member
May 17, 2017
4
4
3
So did a fan mod on the 6450-24P that's going run the downstairs. Waffled back and forth but decided to give some NF-A4x20 FLX a try and work up from there if I had cooling issues since the switch is in an acoustically sub-optimal location. No issues with the fans not starting/stopping (I know that some others on the thread had experienced the opposite), my unit's fan speed 1 was clocking 7V on both fans. You don't even have to change the fan pinout like you do with the Sunon units, the chassis pinout was the same as PC fan pinout. Used the NA-SAV4 rubber fan mounts hopping to kill vibration, may have been a bit overkill. The fans are actually more audible at speed 1 than speed 2, due to PWM noise (must be an awfully chunky PWM waveform).Casing is a little bent from the SAV4's not being pan-head screw flush, so I didn't tighten the snot out of the lid screws. Working just fine, but there's no PoE load yet and there won't be much of one once deployed. Baked the switch in my noise isolation enclosure with the rest of the rack overnight, got a high of 61.5C. So yah, thumbs up for NF-A4x20 FLX fan swap on the 6450-24P, but YMMV.


My 6610 also in rack, left in show chassis below for reference:
Code:
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000141-02
        Serial Number:  217
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Back to Front
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000141-02
        Serial Number:  1Y0
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Back to Front

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 54.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->77       deg-C
                Speed 2:       72<-----> 85 deg-C (shutdown)

Fan 1 Air Flow Direction:  Back to Front
Fan 2 Air Flow Direction:  Back to Front
MAC 1 Temperature Readings:
        Current temperature : 46.0 deg-C
MAC 2 Temperature Readings:
        Current temperature : 54.0 deg-C
CPU Temperature Readings:
        Current temperature : 51.0 deg-C
sensor A Temperature Readings:
        Current temperature : 51.0 deg-C
sensor B Temperature Readings:
        Current temperature : 42.0 deg-C
sensor C Temperature Readings:
        Current temperature : 39.0 deg-C
stacking card Temperature Readings:
        Current temperature : 49.0 deg-C
        Warning level.......: 82.0 deg-C
        Shutdown level......: 85.0 deg-C
Boot Prom MAC : XXX
Management MAC: XXX

The stack unit 2 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present

Fan 1 failed
Fan 2 failed

Fan controlled temperature: 61.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->64       deg-C
                Speed 2:       59<-----> 88 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 57.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 61.5 deg-C
        Warning level.......: 78.0 deg-C
        Shutdown level......: 88.0 deg-C
Also mix-stacked the 6450 to the 6610 (1/3/1 to 2/2/1) via secure-stack/auto/manual, in all cases 1/2/1 and 1/2/6 get turned into stack ports unnecessarily. could unstack one of the 40gb, but get 'can't unstack last stacking port' error so looks like even if you're mix stacking and don't need the 40gb to stack to another 6610, you still lose at least one of these ports to the job. It's not a problem in my current setup, and when I want that second 40gb port back, I'll config the 6450 and do the trunk by hand. Just didn't want to have to traffic manage the uplink between the switches if I didn't have to.
Code:
T=13h17m57.8: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  xxxx.xxxx.xxxx   0 local   Ready
2  S ICX6450-24P   member  xxxx.xxxx.xxxx   0 remote  Ready

    active
     +---+
     | 1 |2/6
     +---+

    active
      ---         +---+
     ( 1 )3/1--2/1| 2 |
      ---         +---+
Current stack management MAC is xxx

Image-Auto-Copy is Enabled.

      Stack Port Status                        Neighbors
Unit# Stack-port1         Stack-port2          Stack-port1         Stack-port2
1     none                dn (1/2/6)           none                none
2     up (2/2/1)          none                 U1 (1/3/1)          none

Unit# System uptime
1     13 hour(s) 17 minute(s) 57 second(s)
2     12 hour(s) 48 minute(s) 49 second(s)
The system started at 18:15:20 GMT-05 Wed Apr 03 2019
 
Last edited:
  • Like
Reactions: arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
So did a fan mod on the 6450-24P that's going run the downstairs. Waffled back and forth but decided to give some NF-A4x20 FLX a try and work up from there if I had cooling issues since the switch is in an acoustically sub-optimal location. No issues with the fans not starting/stopping (I know that some others on the thread had experienced the opposite), my unit's fan speed 1 was clocking 7V on both fans. You don't even have to change the fan pinout like you do with the Sunon units, the chassis pinout was the same as PC fan pinout. Used the NA-SAV4 rubber fan mounts hopping to kill vibration, may have been a bit overkill. The fans are actually more audible at speed 1 than speed 2, due to PWM noise (must be an awfully chunky PWM waveform).
I went back and double checked my 6450 non-poe in my office and I completely forgot to swap pins on the Sunon in this switch when I installed it. Whoops, no wonder it was completely silent. High temp over the last month was only 62°C so no harm done but I swapped pins so the fan will actually function.

Is there any way to address the pwm noise at low speed on these? I'm willing to cobble together some simple electronics inside the case to quiet things back down, the pwm noise is actually really irritating in a quiet room.
 

ThreeFN

New Member
May 17, 2017
4
4
3
I went back and double checked my 6450 non-poe in my office and I completely forgot to swap pins on the Sunon in this switch when I installed it. Whoops, no wonder it was completely silent. High temp over the last month was only 62°C so no harm done but I swapped pins so the fan will actually function.

Is there any way to address the pwm noise at low speed on these? I'm willing to cobble together some simple electronics inside the case to quiet things back down, the pwm noise is actually really irritating in a quiet room.
Hackaday just refers to another guys post, but they've earned, and deserve the love.

Short version, a low pass filter tuned to the pwm freq, ie a resistor and capacitor. You'll waste some power in the resistor but oh well.
[SCIENCE]
However the fan itself is brushless so is already doing some 'pwm' of it's own to drive the coils, which is speed dependent and therefore isn't constant. That's a reason PWM to a fans power pin is a 'suboptimal solution' and 4 pin fans are a thing (4th pin being a command signal that the fan uses to adjust itself, rather than via incoming power manipulation).

Your incoming PWM of the power to the fan 'fights' with the frequency of the coils turning on and off in the brushless motor, and you get some 'beating' constructive/destructive interference. Basically, if the fan is in a position to require power to the one motor coil, and the incoming power PWM is 'on', all is good. If those are out of sync, no power is provided, fan slows down. The increased noise is probably times when the incoming pwm cycles in the middle of a coil activation, 'suddenly' providing or dropping power, jerking the motor/fan. Will it hurt the fan? probably not as much as eventual lubrication loss, and the fan probably has enough inertial and a low enough power/torque in the motor to smooth things out. It just ends up causing noise.
[/SCIENCE]

Trying to figure out how to balance the filter to the fan is probably a fools errand (given the fan has at least 2 speeds) but filtering the PWM should be easy. If you multimeter has a frequency counter feature (I think my $35 extech does) you can get the pwm freq from that, run a low pass filter calc on that, and size your cap/resistor pair. But the results to sound may vary and I'd be a bit surprised if there wasn't some filtering already on the mainboard. Hooking a oscilloscope to the fan header, it would be obvious if it was pure pwm or if there was some filtering, but I haven't had an o-scope to hand for a fair few years.
 
  • Like
Reactions: arglebargle

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Hackaday just refers to another guys post, but they've earned, and deserve the love.

Short version, a low pass filter tuned to the pwm freq, ie a resistor and capacitor. You'll waste some power in the resistor but oh well.
...
Trying to figure out how to balance the filter to the fan is probably a fools errand (given the fan has at least 2 speeds) but filtering the PWM should be easy. If you multimeter has a frequency counter feature (I think my $35 extech does) you can get the pwm freq from that, run a low pass filter calc on that, and size your cap/resistor pair. But the results to sound may vary and I'd be a bit surprised if there wasn't some filtering already on the mainboard. Hooking a oscilloscope to the fan header, it would be obvious if it was pure pwm or if there was some filtering, but I haven't had an o-scope to hand for a fair few years.
Wow, thank you! This is a great breakdown. I'll spend some time reading about this and see if I can't get a friend to bring a scope by the next time he's on my side of the bay, he usually jumps at the chance to put a scope on electronics. I've got a meter with a frequency counter function but no scope unfortunately.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
Whoops, no wonder it was completely silent. High temp over the last month was only 62°C so no harm done but I swapped pins so the fan will actually function.
Do you guys believe me now that these things need basically zero cooling at homelab usage levels? :p

as for the PWM noise, without cobbling a bunch of shit together the easiest thing would probably be just tap off the raw 12V from the PSU and drive a fan with it, and size the fan so the full speed CFM is your ideal CFM/noise (eg a very small/slow fan)
 
  • Like
Reactions: svtkobra7

svtkobra7

Active Member
Jan 2, 2017
362
87
28
Do you guys believe me now that these things need basically zero cooling at homelab usage levels? :p
NO! ;)

I was in my dc the other day (server closet) and pulled both servers (also turned off the ceiling exhaust) and I finally heard the ICX-6450 ...

At first I was like what is that ever so faint noise?
Then I realized that the 6450 does indeed have fans in it! (had never heard them before)
I LOL when ppl bitch about the fan noise on that switch.