Long shot - Issues with Asrock Rack C226ws

[5teve]

Hi guys

Bit of a long shot but just canvassing some opinions on an issue I am having with the above board as part of my recent build. I have contacted Asrock Rack, but have not heard back yet

System is:-
Asrock Rack C226ws Motherboard
Xeon E3 1280V3
Coolermaster Hyper 612 cooler
32gb ECC uDIMMs 4x8gb - mix of kingston 1333 and 1600 - running at 1333
Gigabyte R7 250x 2gb GDDR5 Graphics Card
HP P822 Raid controller - 2gb fbwc
12x Hitachi 2tb Ultrastar 7k3000
2x Sandisk Extreme Pro 256gb in Raid O via motherboard
Corsair cx600 PSU
Fractal Designs Define R2 XL Case
Optical Drive
Lots of 140mm fans
windows 8.1 64bit

Essentially The system will not warm boot. The dr Debug internal LED display stops on 9A and then the system restarts, the same happens continuously. If i turn the PSU / Mains off wait for the system to de-energise, the system will generally then boot and run normally and in a stable fashion, although some cold boots the asrockrack logo appears the screen goes blank and then after a short while windows blue screen appears. I then have to switch mains off and on again and then it generally will boot.

The dr debug code indicates that the last boot event is to do with USB initialisation. after this it reboots and continues with the reboot cycle. At no point in this process does it POST or show anything on monitor.

I have tried extensive troubleshooting - including

Unplug all USB - including headers front panel etc - No improvement
Unplug HP P822 Raid card - Same issue
Reduce Ram to 1x stick in A1 - Same issue with multiple sticks tried
Moved Graphics Card to new PCIE slot - Same issue
Changed bios settings - multiple different setting including disable of INtel USB3 - Same issue.
Reflashed Bios 2.30 - Same issue.

Has anyone encountered this kind of issue previously? I initially thought a USB device wasn't playing nicely with the Intel USB3, as it seems not to be a good device with windows 8.1, but when everything is unplugged and it still does it.. I guess I can discount that.

Once the machine has booted it is stable - with the exception of some weird slow down issues with the HP P822 according to benchmarks but not actual usage..

Thanks

Steve

 

[MikeC]

Next thing I'd try is PSU.

 

[5teve]

Thanks for the suggestion - I've thought about it, but its a hell of a job to put a psu in properly.. The Sata power cables are all custom to accommodate the 12 drives - I may be able to strip the system down again (ie remove all add ons) and jerry rig a PSU to it - Maybe with all the drives starting i'm just pulling the 12v line too low, but i'm not seeing anything more than 200-220w startup via the UPS display. Its strange that it is so stable once running tho.

Steve

 

[MikeC]

Can you do staggered spin up on the drives? Or possibly disconnect the drives at the PSU as that PSU looks like a modular one.

 

[5teve]

Cant do staggered spin up, well not that I know as I cant get into the HP ORCA bios. It may do it by default - or maybe I can raid my old motherboards and nick enough jumpers to jumper the drives to start up in standby?

The PSU isnt modular.. but I can unplug the drives so may try that later if i have time..

Steve

 

[5teve]

OK Some more testing...

Removed all power cables from the spinners, leaving just 2x SSD and the motherboard etc. Same issue 9A - Cold boot OK (unless I change the Bios in which case it blue screens going into windows - Reboots and starts the 9A issue again - mains power off - on and boots ok)

Jerry rigged another 550w antec modular PSU to the machine (from my old workstation) - drives still unpowered - same issue

Swapped Graphics card out for older AMD - drives still unpowered - same issue

Checked voltages in the bios and they are very close to actuals, probably closer than I have seen on a motherboard previously

I'm still angled toward the motherboard having something wrong with it.. but only on boot.. once booted.. its fine.

Any other ideas?

Steve

 

[andrewbedia]

i'm not seeing anything more than 200-220w startup via the UPS display. Its strange that it is so stable once running tho.

That display doesn't poll/update nearly fast enough to see that initial power-on draw.

 

[MikeC]

Try reseating the CPU and / or buying a cheap LGA1150 Pentium from eBay to prove the point? I'm fast running out of ideas

 

[EffrafaxOfWug]

I ran into a similar issue on linux a while ago (also on an ASRock C22x) that was caused by the intel management engine not being (de?)initialised properly; is it possible for you to disable it in the BIOS to rule that out or check whether it's registered correctly in devmgmt.msc?

 

[5teve]

I ran into a similar issue on linux a while ago (also on an ASRock C22x) that was caused by the intel management engine not being (de?)initialised properly; is it possible for you to disable it in the BIOS to rule that out or check whether it's registered correctly in devmgmt.msc?

An interesting opinion

I have scoured the Bios for something to do with this.. and I can find nothing - I have reset the bios to show legacy bios for the 2x intel 210 NICS and the graphics card and tried pressing ctrl-p to get into the intel management engine (very little info with regard to how to access this) upon pressing ctrl-p i get a message flash up in the top left corner saying FW Status Recovery Error so possibly not a good sign.

In device manager however Intel Management engine Interface is sitting nicely in system devices and using driver 9.0.0.1287 from 17/12/12 - so appears to be working ok on here.

There is an intel Manageability engine firmware recovery agent program also that loads but when checking for updates errors out - looks like it isnt supported anymore so thats now gone..

I'll keep digging and report back..

Steve

 

[5teve]

OK - I have no idea re the advanced management engine stuff.. no info on Asrock rack products and no info on their site or even the manual - just a picture and no words...

It has a bios setting for console redirection - com0 and doesnt look like any SOL settings - so i'm lost already.

There is definitely no option for IME or AMT or any other intel based stuff in the bios. So the management engine seems completely hidden. Unfortunately we don't have the generous return policies of the US stores, so i'm going to have to wait for a reply from Asrockrack on this. I cannot see how its a knackered motherboard though when it runs so stable, once I have jumped through hoops to get it started.

I may talk to my supplier and see what they say. I'm almost out of patience and I dont have the time to research the issue all day..

Steve

 

[5teve]

OK and the final test - CPU swap - I spoke to the distributor and they suggested it could have a dirty pad or something, plus its been suggested on here (thank you) so I dropped on an ES i7 s1150 cpu.. it booted (surprised Asrock havent locked them out) and on restart - same issue - I cannot think of anything else it can be now - apart from the Intel ME which looks like it may be screwed on boot up (still getting the FW status recovery error on post after the NICs bios displays

No my supplier says they need it back - they dont have any in stock so will have to send it back to asrock and then can send me a new one - about 4-5 weeks normally but as it took them 3 months to get the board in stock - I think i'm out of luck! - Not really server / workstation support is it?

Steve

 

[EffrafaxOfWug]

It's probably not related to the IME but if you want to rule it out you can disable the MEI drivers from loading in devmgmt.msc (hecix64.sys IIRC) or simply uninstall them. If the drivers are never loaded via the OS then it doesn't present a problem when it comes to shutdown and that should rule out the problem.

Missed in your first post that you're actually getting a BSOD; is windows leaving a memory dump in system32 that you can pop open in windbg?

 

[5teve]

Thanks guys... I'll try disabling the MEI drivers, but as it happens long before windows loads or even the bios posts.. but it has to be worth a try. The FW status recovery flashing up on screen is worth looking up tho.

I have had a reply from asrock however... a little insulting - but probably copy and pasted but item 4 is interesting

Dear Steve,
Sorry for any inconvenience caused.
Here we give you some suggestions to try
1. Clear CMOS.
2. Disable Legacy USB Support in BIOS -> Advanced -> USB Configuration to try.
3. Simplify environment just plug one DIMM, CPU and VGA to verify.
4. Exchange BIOS ROM chip, could you provide us your address, we will send you a BIOS ROM chip to replace.
Thank you!

I'll be taking them up on the bios chip exchange.. and seeing what effect that has.

Re the BSOD - it only generally happens if the bios has been entered and something changed - save the bios - it restarts - bootloops - i switch it off at mains, switch it on and then it sometimes blue screens when trying to boot. You can see its going to happen as normally there is an asrockrack logo with the circle of dots as soon as the machine has posted and waited for the boot menu to timeout, when a blue screen occurs, the bios boot menu timeout waits longer than the specified 6 seconds, then the logo fades out and the screen goes black, then blue about 10 - 20 seconds later. it always blue screens with dpc_watchdog_violation - but think its a generic blue screen.

Event viewer shows the last one as
The computer has rebooted from a bugcheck. The bugcheck was: 0x00000133 (0x0000000000000000, 0x0000000000000501, 0x0000000000000500, 0x0000000000000000). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 052715-56984-01

the dump (i have just been trying to set up windbg - possibly not right) is as follows with a analyse -v added for more detail - Microsoft (R) Windows Debugger Version 6.3.9600.17336 AMD64 Copyright (c) Micro - Pastebin.com

If you have any hints and tips on how to read the dump file better let me know as its nearly 1gb in size! I can upload it if required. It looks like the blue screen is caused by the P822 driver but I dont think that is the issue for the boot loop.. I think thats an additional.

Let me know what you think..

Steve

 

[5teve]

OK another update.. Im feeling like my windows install is hosed somehow.. I just uninstalled the drivers for the P822 to revert back to the standard windows driver, checked the box to delete driver files.. rebooted.. no HP822 showing in device manager, nothing could be found.. HP SSA could not see the card.. but the drive and volume etc was available.. I had to delete all hidden HP822 (assume they are previous attempts at moving PCI E) in device manager, pull the card physically, reboot, shut down, plug the card back in and then re-install the driver.. I cannot for the life of me work out how it was working.

I also think that windows is pretty tightly integrated with the bios now.. so I may do a full image of the whole drive and then try a full reinstall of windows - again!

I have a crap load of work to do first tho... so its going to have to wait.

Steve

 

[5teve]

Well another update.. I have received the bios chip from asrock and from my initial testing it actually looks like it has fixed the boot loop issue, which i'm pretty surprised about. I'll keep testing but so far.. so good with several restarts.

The only thing is now.. the read write speeds on the p822 seem even lower! i'm definitly thinking I have a dud drive.. but as I cant pull a drive to benchmark individually.. as the array goes offline... testing is going to require some planning!

Steve