VRTX and M630 woes

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

ihateire

New Member
Dec 25, 2019
7
0
1
hey friends!

i was recently blessed to find a VRTX and M630 in my budget, and after sourcing a few parts that weren't included (namely: backplane expander, CMC, SPERC8, and power supplies), i've found that i'm having trouble actually getting things to work.

i've had two M630s now exhibit similar issues- pulsing green power LED, non-ready iDRAC in CMC, no indicator LEDs for the iDRAC on the blade itself, even after 30+ minutes left alone. i was previously told the culprit was motherboard failure, but since the replacement blade is having the same trouble, is it possible that it's something else? my main concern is that i'm missing another part i haven't identified yet.

i have both mezzanine cards (Dell P/N 3N9XX) installed in the blade, and i've tried all four blade slots in the VRTX to no avail.

greatly appreciate any guidance you guys can provide.
 

ihateire

New Member
Dec 25, 2019
7
0
1
i grabbed some logs from the CMC- not sure how much help these will be.

Code:
Jan 22 22:37:20 CMC- thermald: HandleChassisPowerStateChange: chassisState [2] - read blade[1] power state
Jan 22 22:37:20 CMC- thermald: HandleChassisPowerStateChange(): FAILED to query Blade-1 power state. rc=20744 CompCode=255
Jan 22 22:37:20 CMC- thermald: HandleChassisPowerStateChange: chassisState [2] - read blade[1] power state
Jan 22 22:37:20 CMC- thermald: HandleChassisPowerStateChange(): FAILED to query Blade-1 power state. rc=20744 CompCode=255
Code:
Jan 22 22:42:10 CMC- bmd: Blade 1. IMC is not ready. The requested action  cannot be carried out.
Jan 22 22:42:10 CMC- lcdmgrd: bm_blade_actions_clt: receive_mms_response() blade 1 bad response code 0x5203
Jan 22 22:42:10 CMC- lcdmgrd: bm_blade_actions_clt: blade 1 returning rc 0x5203
Jan 22 22:46:22 CMC- bmd: Blade 1 virtual reseat is initiated.
Jan 22 22:47:40 CMC- systemd[1]: Starting Cleanup of Temporary Directories...
Jan 22 22:47:40 CMC- systemd[1]: Started Cleanup of Temporary Directories.
Jan 22 22:48:57 CMC- pwrmgmtd: iDRAC-1 AUX power turned OFF: status 0
Jan 22 22:48:57 CMC- pwrmgmtd: iDRAC-1 AUX power turned OFF: status 0
Jan 22 22:48:58 CMC- thermald: HandleBladeRemove(): FAILED to find Blade-(1,0)
Jan 22 22:48:58 CMC- Severity: Critical, Category: System Health, MessageID: HWC3002, Message: Server 1 is removed.
Jan 22 22:48:58 CMC- pwrmgmtd: iDRAC-1 AUX power turned ON: status 0
Jan 22 22:48:59 CMC- Severity: Informational, Category: System Health, MessageID: HWC3003, Message: Server 1 was inserted.
Jan 22 22:49:28 CMC- cfgmgrd[686]: cfgmgr: config backup 'save' operation succeeded
Jan 22 22:49:59 CMC- bmd: imc_boot_timer_expired: iDRAC for blade 1 did not boot within the expected time.
Jan 22 22:50:21 CMC- racadm: Error in powerControlChassis, rc=5511
Jan 22 22:50:21 CMC- racadm: racadm chassisaction failed, STATUS = 0x5511
 

DaSaint

Active Member
Oct 3, 2015
282
79
28
Colorado
howdy, i have a VRTX + M630s as well, but do not have the same issue, but i do also have all the components in my VRTX (CMC, SPERC, etc.)

I would recommend getting a CMC as its very possible that the blades do not know how to really communicate to the underlying system as the CMC is the brain of the entire chassis. Also there have been firmware updates that have had an impact on how other systems interoperate inside of the VRTX.

they can be a little pricy but usually i could find one under 100$ and it will give you a lot more information, but based on the logs its hard to tell if its b/c the blades are expecting a CMC or its a bad mobo as previously stated... but an active CMC might be able to tell u.


I would say the biggest investment are the PSUS and the CMC, you can go without the backplane expanders and the SPERC but you will get errors about that in the CMC as well, and trust me those expanders are not easy to find (i had to help a buddy get his from Europe for like 120$/ea)

Ive run M620s and M630s in my VRTX chassis, just been starting to migrate into 630s for my ESXi Hosts and running the M620s as backups and Desktop box for Vid Editing
 

ihateire

New Member
Dec 25, 2019
7
0
1
howdy, i have a VRTX + M630s as well, but do not have the same issue, but i do also have all the components in my VRTX (CMC, SPERC, etc.)

I would recommend getting a CMC as its very possible that the blades do not know how to really communicate to the underlying system as the CMC is the brain of the entire chassis. Also there have been firmware updates that have had an impact on how other systems interoperate inside of the VRTX.

they can be a little pricy but usually i could find one under 100$ and it will give you a lot more information, but based on the logs its hard to tell if its b/c the blades are expecting a CMC or its a bad mobo as previously stated... but an active CMC might be able to tell u.


I would say the biggest investment are the PSUS and the CMC, you can go without the backplane expanders and the SPERC but you will get errors about that in the CMC as well, and trust me those expanders are not easy to find (i had to help a buddy get his from Europe for like 120$/ea)

Ive run M620s and M630s in my VRTX chassis, just been starting to migrate into 630s for my ESXi Hosts and running the M620s as backups and Desktop box for Vid Editing
i currently have a single active CMC installed and on the latest firmware- all i get from the CMC is that the blade is present and healthy, but i don't get any other information.
 

DaSaint

Active Member
Oct 3, 2015
282
79
28
Colorado
make sure there arent any bent pins inside the back of the blade board, cause if any are missing/bent and not connecting that would equal some bad stuff... the other possibility is that the backplane those connect into could be bad as well.

Have you tried in different slots and how many PSUs do you have hooked to the back? any pictures?
 

ihateire

New Member
Dec 25, 2019
7
0
1
make sure there arent any bent pins inside the back of the blade board, cause if any are missing/bent and not connecting that would equal some bad stuff... the other possibility is that the backplane those connect into could be bad as well.

Have you tried in different slots and how many PSUs do you have hooked to the back? any pictures?
i'm running dual 1100w PSUs at the moment until i can grab two more, and there don't seem to be any bent pins. my suspicion is that the midplane may be wonky and since replacements are inexpensive (i got mine for $33 on ebay), i'm gonna give replacing that a shot next week when it arrives.
 

DaSaint

Active Member
Oct 3, 2015
282
79
28
Colorado
i have heard stories of the midplane going bad before, i had a chat with a Dell Service Tech a few years back told him about my VRTX and he mentioned that it was common for them to go bad and he had replaced many.
 

ihateire

New Member
Dec 25, 2019
7
0
1
i have heard stories of the midplane going bad before, i had a chat with a Dell Service Tech a few years back told him about my VRTX and he mentioned that it was common for them to go bad and he had replaced many.
that definitely gives me hope. i've dreamed of owning one of these for years now, and i'd be heartbroken if i had to get rid of it.
 

frogtech

Well-Known Member
Jan 4, 2016
1,482
272
83
35
You should try to fill in all of the PSU slots and get all of the PSUs plugged in. That will most likely solve your problem.

If you need a midplane, I have couple I could sell in the event that yours is not serviceable under warranty.

The minimum amount of parts required for a blade to actually power on is minimal, all you need is the midplane, the mainboard, a CMC, power supplies, and front LCD /w USB control board for KVM functionality. Really don't even need the PCIe bypass cards.
 
  • Like
Reactions: NightHawk

ihateire

New Member
Dec 25, 2019
7
0
1
You should try to fill in all of the PSU slots and get all of the PSUs plugged in. That will most likely solve your problem.

If you need a midplane, I have couple I could sell in the event that yours is not serviceable under warranty.

The minimum amount of parts required for a blade to actually power on is minimal, all you need is the midplane, the mainboard, a CMC, power supplies, and front LCD /w USB control board for KVM functionality. Really don't even need the PCIe bypass cards.
no luck even with four PSUs, unfortunately- still seeing the same behavior. i have a midplane replacement lying around already, thankfully, and it's looking like that's the most likely explanation for what's going on.

if midplane replacement turns out pointless, is there anything else i should consider short of completely swapping the motherboard on the blade? i'm not all too familiar with blades and i greatly appreciate all of your guidance.
 
Last edited:

frogtech

Well-Known Member
Jan 4, 2016
1,482
272
83
35
It's hard to say. A midplane swap will be easy enough to perform. Make sure you check all of the contact pins on blades, the underside of the VRTX mainboard. They're probably fine though. It's interesting to me that the chassis detects the insertion and removal of the blade. It's not impossible I guess that both blades could have defective iDRACs? I've definitely had M620s that had bad management engines that needed to be swapped.

If a midplane swap doesn't work, I might try a different model like an M620 or just another M630 I guess if you don't have any E5 v1/v2 parts laying around.

Barring that, the next things to consider might be the CMC or VRTX mainboard. But again I wouldn't really think it would be those since the chassis logs that the iDRAC didn't boot within the specified time frame.

Could you try a CMC reset to factory defaults? I believe you need to SSH into the CMC using something like putty, and simply issue command racresetcfg to do this.

As for mobo swaps on blades, it's easy enough, though it's definitely easier in the M620 which is weird.




 

frogtech

Well-Known Member
Jan 4, 2016
1,482
272
83
35
i sadly haven't had time to swap the midplane lately, but i'll update as soon as i'm able!
Once you do, if it works out, kindly report back on the M630 power consumption as reported by iDRAC if you don't mind! Not the total enclosure / chasiss input power.
 

jsenecal

New Member
Sep 21, 2020
1
0
1
Did replacing the midplane ended up solving your issues ?

I have an M1000e with similar behavior.