So I've mentioned some really weird issues in my build and Icy dock thread and thought they deserved their own thread as it may help others, and educate myself all at once.
I've been running this setup for over 2 weeks without any issues.
- 550w EVGA G2 Power Supply
- 8x 3.5" HDD (6x WD RED 5TB, 2x WD RED Pro 2TB)
- 4x Intel 730 240GB (in icy dock)
- 2x Intel S3700 100GB (in icy dock)
- 2x Intel S3500 80GB
- 2x HGST 200GB SAS 12Gb/s SSD
- 2x P3700 800GB 2.5" NVME
The above are running off:
- 2x M1015 PCIE HBA
- 1x LSI 3008 OnBoard HBA
- Onboard SATA (S3500 -- OmniOS/Napp-IT File Server)
- 1x SuperMicro NVME AOC
The rest of the system is just a supermicro 1P board, e5-2670 v3 (idle, no other vm running), 96GB DDR4 RDIMM, and every fan port used on motherboard from 40mm to 140mm fans -- some 3 and 4 Pin.
The issue:
Yesterday I added a 2nd Icy Dock for 4x 15MM SAS SSD utilizing 1 channel of the LSI 3008.
The 1st drive I installed was NOT detected. The other 3 ports worked fine and detected the drives. Hot-removal did not work, I had to reboot OmniOS VM to remove the drives. Thinking it was a bad cable (Adaptec brand) I swapped to a 2nd (new) cable I got from newegg (LSI Brand). The same thing occurred with this cable except it was a different port. Hot-removal did not work with this cable either. At this time one of my M1015 completely errored out, crashed the VM and I had to remove it from pass-through to get it to boot again. (I thought it over heated due to case open, and would fix it later after finding the issue with the new icydokc/cable).
Thinking I maybe got 2 bad cables I put in another new cable (Adaptec brand) and it mirrored the exact problem as the other cable except hot-removal worked for 1 port. Now, thinking maybe hot-add didn't work properly and why the same port on the same cable didn't work I shut down the VM, installed all 4x SSD and booted up. To my surprise the ENTIRE LSI 3008 stopped working, and my 12Gb/s drives were now gone as well as EVERY drive from the new icy dock.
At this point I talked to @coolrunnings82 and he urged me to investigate power usage on the 5V rail as it sounded like I was really pushing the limits with this many drives utilizing a consumer ATX power supply. Kicking myself for not using a 750 or 850 EVGA G2 I had already, thinking this system is well under 550w even at load as I saw no point... well as it turns out the 750 and 850 have the same 5V rail power, and the 550w is only 10 or 15w less... likely enough to matter with what I had under load, but still not enough (if this is the problem) for adding 4x more SSD (and wanting to add 2x more to the other 3008 channel)
The questions / thoughts:
Does this sound like a 5v RAIL issue? I can see the SSD / HDD not being detected due to not enough power but why would the HBAs completely error out and not allow the VM to boot? Do the PCIE slots require 5V for something too and I had drawn too much when I tried to boot with the 4x SSD in addition to the other drives ALL AT ONCE. (VS adding 4x ssd to a system after boot). At this point power is the only thing I can think of that would cause this.
For the record, I removed the new icy dock the 4x new sas ssd, and the 2x 12Gb/s SSD and then re-added the M1015 HBA that stopped working earlier to the VM and it booted just fine and started to work again like no problem. I did not add extra cooling, and in fact it likely got warmer while working on this. I did NOT power-down the system when I did this as I just slid out the icy dock, un-did the cables, and the LSI 3008 is onboard so nothing to remove there. The 2x SSD that were working prior were not in a location to heat the M1015 either, and I did not wait for it to cool either-way.
What are your thoughts on this being a power issue?
Other opinions?
I've had no problem the last 5+ hrs of this being 'fixed' after removing the 6x SSD and Icy Dock (2x 40mm fans). I saw no voltage 'error' in vSpere either for the record. Not too sure how accurate that is.
I've been running this setup for over 2 weeks without any issues.
- 550w EVGA G2 Power Supply
- 8x 3.5" HDD (6x WD RED 5TB, 2x WD RED Pro 2TB)
- 4x Intel 730 240GB (in icy dock)
- 2x Intel S3700 100GB (in icy dock)
- 2x Intel S3500 80GB
- 2x HGST 200GB SAS 12Gb/s SSD
- 2x P3700 800GB 2.5" NVME
The above are running off:
- 2x M1015 PCIE HBA
- 1x LSI 3008 OnBoard HBA
- Onboard SATA (S3500 -- OmniOS/Napp-IT File Server)
- 1x SuperMicro NVME AOC
The rest of the system is just a supermicro 1P board, e5-2670 v3 (idle, no other vm running), 96GB DDR4 RDIMM, and every fan port used on motherboard from 40mm to 140mm fans -- some 3 and 4 Pin.
The issue:
Yesterday I added a 2nd Icy Dock for 4x 15MM SAS SSD utilizing 1 channel of the LSI 3008.
The 1st drive I installed was NOT detected. The other 3 ports worked fine and detected the drives. Hot-removal did not work, I had to reboot OmniOS VM to remove the drives. Thinking it was a bad cable (Adaptec brand) I swapped to a 2nd (new) cable I got from newegg (LSI Brand). The same thing occurred with this cable except it was a different port. Hot-removal did not work with this cable either. At this time one of my M1015 completely errored out, crashed the VM and I had to remove it from pass-through to get it to boot again. (I thought it over heated due to case open, and would fix it later after finding the issue with the new icydokc/cable).
Thinking I maybe got 2 bad cables I put in another new cable (Adaptec brand) and it mirrored the exact problem as the other cable except hot-removal worked for 1 port. Now, thinking maybe hot-add didn't work properly and why the same port on the same cable didn't work I shut down the VM, installed all 4x SSD and booted up. To my surprise the ENTIRE LSI 3008 stopped working, and my 12Gb/s drives were now gone as well as EVERY drive from the new icy dock.
At this point I talked to @coolrunnings82 and he urged me to investigate power usage on the 5V rail as it sounded like I was really pushing the limits with this many drives utilizing a consumer ATX power supply. Kicking myself for not using a 750 or 850 EVGA G2 I had already, thinking this system is well under 550w even at load as I saw no point... well as it turns out the 750 and 850 have the same 5V rail power, and the 550w is only 10 or 15w less... likely enough to matter with what I had under load, but still not enough (if this is the problem) for adding 4x more SSD (and wanting to add 2x more to the other 3008 channel)
The questions / thoughts:
Does this sound like a 5v RAIL issue? I can see the SSD / HDD not being detected due to not enough power but why would the HBAs completely error out and not allow the VM to boot? Do the PCIE slots require 5V for something too and I had drawn too much when I tried to boot with the 4x SSD in addition to the other drives ALL AT ONCE. (VS adding 4x ssd to a system after boot). At this point power is the only thing I can think of that would cause this.
For the record, I removed the new icy dock the 4x new sas ssd, and the 2x 12Gb/s SSD and then re-added the M1015 HBA that stopped working earlier to the VM and it booted just fine and started to work again like no problem. I did not add extra cooling, and in fact it likely got warmer while working on this. I did NOT power-down the system when I did this as I just slid out the icy dock, un-did the cables, and the LSI 3008 is onboard so nothing to remove there. The 2x SSD that were working prior were not in a location to heat the M1015 either, and I did not wait for it to cool either-way.
What are your thoughts on this being a power issue?
Other opinions?
I've had no problem the last 5+ hrs of this being 'fixed' after removing the 6x SSD and Icy Dock (2x 40mm fans). I saw no voltage 'error' in vSpere either for the record. Not too sure how accurate that is.