[Solved] Brocade ICX6610-48P PoE Errors

Discussion in 'Networking' started by KFJ, Apr 2, 2019.

  1. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    I bought a used ICX-6610 from eBay but it turns out the PoE board was shot (had a hole in an IC and a blown transistor on the PoE board). Despite this, I was able to update the firmware and image and PoE firmware but the error persisted and of course the error blocked the output of commands like 'show chassis'.

    The switch was just displaying the following in a loop:

    Code:
    PoE Error: Device 0 failed to start on PoE module.
    PoE Error: Device 1 failed to start on PoE module.
    Resetting module in slot 1 again to recover from dev fault
    The seller graciously sent another used PoE board which I installed yesterday. The switch powers on and I can get a console, but the console doesn't load past "Enter 'b' to stop at boot monitor" regardless if I press b per usual or not. The switch never fully boots.

    Any tips or tricks to reflash the PoE board or reset the switch from the current state? I know how to reflash when the switch actually boots, but I have no idea what to do if pressing 'b' does nothing or if I cannot get a prompt.

    I still have the old PoE board - at least that let me boot the switch, but I'd kinda like to have the PoE working, but it is optional (I'll have to get another PoE switch).
     
    #1
  2. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    199
    Likes Received:
    48
    Have you tried to update the switch f/w: FCX / ICX6610 - Fohdeesha Docs ?
    pinging @fohdeesha

    PoE f/w updating is curious. It's detailed in the guide, but you should know that it happens in the background and takes 10-15 mins 6610 and the only way to check the progress is to do a "show log"
     
    #2
  3. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    Thanks boredSysadmin.

    Yes, I did update the switch, including the PoE firmware using @Fodeesha's docs. However, there was a blown transistor on the original PoE board and an IC with a burn hole in it (not me, bought used). Surprisingly, the switch was functional and I was able to run commands even with the blown board - but the console was flooded with the PoE errors I posted above and of course the PoE did not supply any Power over Ethernet due to the blown board. I configured the switch and was using it with a couple of dev laptops and all was fine except no PoE.

    So I replaced the PoE board and when I power on the switch and hook a laptop to the console port with Putty I get the familiar console prompt and I can press 'i' or 'b' but then nothing happens, the switch does absolutely nothing new: displays the press 'i' and press 'b' prompts and nothing else but fans. If I don't press 'i' or 'b' then the switch still does nothing (except keep the fans on full blast). I have had it powered on for 2 hours and the console never updates past press 'i' or 'b'.

    My thinking is the 'new' PoE board needs to be flashed to the latest PoE firmware version as was the 'old' damaged board that I had updated the PoE firmware on. But now I can't flash anything without a functioning switch or console - the console is not doing anything now, just displaying "press 'b' for boot" no matter if I press 'b' or not.
     
    #3
  4. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    199
    Likes Received:
    48
    does the switch boots normally with new PoE board removed?
     
    #4
  5. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    Hold my beer...
     
    #5
  6. Dreece

    Dreece Master Tinker

    Joined:
    Jan 22, 2019
    Messages:
    169
    Likes Received:
    12
    you still holding it @BoredSysadmin ?

    reading about all these problems with the poe brocades just warrants a sticky on this sub-forum "Health Warning - Check used POE Brocades before purchasing"
     
    #6
  7. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    199
    Likes Received:
    48
    I've bought a used Brocade 6610-24P PoE from the bay and so far no issues to report. I've witnessed fohdeesha performed small miracles with these switches I and I'm a believer that he'll guide KFJ towards salvation fixing his switch
     
    #7
    Dreece likes this.
  8. Dreece

    Dreece Master Tinker

    Joined:
    Jan 22, 2019
    Messages:
    169
    Likes Received:
    12
    It isn't so much to do with being fixable, just people should be aware that there are known concerns regarding used PoE Brocades and if people arent into ripping open hardware then they should check their potential purchase over before handing over the cash.
     
    #8
    fohdeesha likes this.
  9. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    I shutdown the switch, and removed the 'new' PoE board. Same issue as before - console won't load past "Enter 'b'" or "Interrupted by entering 'b'" I let it each attempt (not pressing b and pressing b) run for 20 minutes - no change just fan at full blast.

    I then shutdown the switch and inserted the 'old' PoE board - same issue as before - console won't load past "Enter 'b'" or "Interrupted by entering 'b'" .

    I think she might be bricked :(

    Attached image shows the three attempts separated by about 20 minutes each. The first two are with the 'new' PoE board removed. The last one is with the new PoE board.
     

    Attached Files:

    #9
  10. Dreece

    Dreece Master Tinker

    Joined:
    Jan 22, 2019
    Messages:
    169
    Likes Received:
    12
    bugger :(
     
    #10
  11. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    199
    Likes Received:
    48
    I assume that you're pressing b really really quickly? it shows up in prompt very soon after power on and the window is easy to miss.
    but I guess you already know this since you've upgraded it per the guide.
     
    #11
  12. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    Yes, in the attached image it shows 'Interrupted by pressing b'. Prior to installing the 'new' board the switch would boot per usual (without pressing 'b') but throw a loop of PoE errors (every 1 to 10 seconds) and I could press 'b' and enter the boot prompt (at startup only of course). I configured the switch, it could reach my router and the network but I wanted PoE to work (as advertised and as paid for), so I contacted the seller, we agreed on replacing the board after I opened up the switch and saw the blown transistor and burnt IC. The first time I booted the switch with only the console port used I received the PoE errors and did not smell smoke, I do not believe it is anything I did. Regardless of the constant PoE errors I was able to update the firmware and image and the PoE firmware to latest (803t?) and the switch was usable other than the PoE errors. After I replaced the board, the switch will not boot past the 'Enter 'b' prompt if I do not press 'b' or 'Interrupted by pressing 'b' if I do press 'b'.

    I am going to do a close inspection of the other boards to see if I missed something or if I accidentally left a shovel on the main board or something likely non-obvious.

    Edit: NICE! I noticed the RAM wasn't fully seated, so I re-seated it. Maybe I nudged it, but I am now in the boot monitor! Gonna try the new PoE board in a few minutes.
     
    #12
  13. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    PIR: Replacing the PoE board resolved the PoE errors experienced with the original PoE board (see attached). The issue with the switch not booting with the new board (or any board or no board) was due to the DRAM module not being fully seated, likely caused by OP when removing the connector from the bottom of the original PoE board. Reseating the DRAM module resolved the issue and the switch is currently running with the configuration and firmware updates applied prior to replacing the PoE board.

    Thank you STH members for your help and insights. I guess it wasn't a shovel or a brick after all.
     

    Attached Files:

    #13
    fohdeesha and BoredSysadmin like this.
  14. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    199
    Likes Received:
    48
    Would you terribly mind posting a photo of this dram module?
     
    #14
  15. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    199
    Likes Received:
    48
    Looks like not all Brocade PoE switches are to blame for issues after all :)
     
    #15
    Dreece likes this.
  16. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,196
    Likes Received:
    975
  17. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,196
    Likes Received:
    975
    it's a RAM stick just like it sounds (well, laptop style DDR2 ECC if I remember right), here's a spare pic I had: https://i.imgur.com/l8QLkoH.jpg
     
    #17
    Dreece and BoredSysadmin like this.
  18. KFJ

    KFJ New Member

    Joined:
    Mar 11, 2019
    Messages:
    9
    Likes Received:
    5
    Exactly. Thanks @fohdeesha. That is the RAM near the middle of your pic.

    I had noticed the DRAM wasn't seated properly because the latch that is supposed to hold the DIMM in place was not in the correct position and one side of the DIMM was showing like 1/16th of an inch more pins. Simply applying pressure reset the DRAM. I have no idea if it was loose before or not and I just made it looser or not, but that was one of the issues.

    In case someone has a similar issue with the PoE errors and has a replacement board: the PoE board is the larger one near the bottom of the pic with the wire harness in the middle. I recommend removing the 4 screws holding the PoE board down, and carefully prying/lifting the board from multiple positions prior to removing the wire harness from the bottom of the PoE board. There are 4 sets of double-ended header pins that are about 1 inch long on the bottom side near the bottom edge of the board that need to not get bent when removing the PoE board. The header pins are not soldered down on either end, just loosely fit is enough I guess.
     
    #18
    BoredSysadmin, Dreece and fohdeesha like this.
Similar Threads: [Solved] Brocade
Forum Title Date
Networking [solved] Quanta LB6M connection to Mellanox ConnectX-4 /5 Nov 19, 2018
Networking [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G Nov 16, 2018
Networking [Solved] BiDi fiber question Sep 7, 2016
Networking [SOLVED]Slow speeds between two Connectx-2 machines Mar 1, 2016
Networking Guest VLAN on Brocade 6450 Thursday at 11:51 AM

Share This Page