I had to buy some a few years ago to replace a toll sensor, and I could spend $5 for a tiny package, or $8 for a giant roll.
So, now Dual Lock is my hammer for all "stick things together temporarily" nails.
SC846 system gifted to me - A full overview with questions. Replacing DVD drive with SSDs? Ideas for upgrades or keep what I got?
- Thread starter Koop
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
So, now Dual Lock is my hammer for all "stick things together temporarily" nails.
Nice on the 3M Dual Lock. I'll have to keep that in mind.
I have all the hardware now. Going through paces with testing components and all that fun stuff. BMC and BIOS updated, etc etc...
There are so many bios options in here (especially for Xeon stuff) that I'm not familiar with haha. I'll need to read. I am sure there are only a few things that really matter.
I have all the hardware now. Going through paces with testing components and all that fun stuff. BMC and BIOS updated, etc etc...
There are so many bios options in here (especially for Xeon stuff) that I'm not familiar with haha. I'll need to read. I am sure there are only a few things that really matter.
Last edited:
I'd also like to annouce that no matter how absolutely careful I have been I accidentally got a finger too close to one of the high RPM fanwall fans. Nicked the tip of my finger real good, summabeech.
Let be known that this system, as many others before it- has been given the blood sacrifice. May it be sated now and run without issue.
Let be known that this system, as many others before it- has been given the blood sacrifice. May it be sated now and run without issue.
Hm, so I am definitely having some odd behavior with my backplane I think?
I spent some time running memtest86+ on all the memory I have and all looked well there, did some stress tests and let it run to see overall system stability- no crashing or anything so all went well.
Fully loaded my BPN-SAS2-846EL1 with my drives, I have it connected to myAOC-S3008L-L8E using SM branded SFF-8643 to SFF-8087 cables. Flashed the HBA using IT Firmware directly from SuperMicro using their scrip which made it super easy.
Go into TrueNAS scale. saw all my disks, created my pool and vdevs, and all was looking normal and well.
I wanted to next destroy my pool and run a badblocks script on every drive. I know it'll probably take a long ass time so I shut down the system and called it a night to get ready to do that this weekend.
Next power up I am seeing all sorts of odd behavior. First was backplane showing no lights at all, no disks recognized in TrueNAS. Shut down. Immediately on power up all backplane LEDs are solid red. Shut down. Next power up I have a mix of solid red backplane LEDs, blinking red LEDs, and some completely off. Each reboot would either be No LEDs, all LEDs red, or a weird mix of on/off/blinking fault LEDs.
I've also fully removed both PSUs just to do a full power drain. I've tried different PSUs as well. I have a pair of PWS-920P-SQs as well as the original PWS-1K21P-1Rs that came with the chassis.
I did notice once I had a nice pattern of blinking blue LEDs going across the drive bays. bay 5 - bay 11 -bay 17 - bay 23. Blue LEDs blinking across those top 4 in order once. I think all disks were recognized just fine that time.
When I swapped to try using a single PWS-1K21P-1R everything came up normal. I thought to myself, am I not using big enough power supplies somehow? Watching the boot I never see watt usage on my kill a watt go above 400-500w. However I shut the system down and on boot up I got the same lightshow (all red) from the backplane. So it seems to happen to regardless of PSU used?
Hard to pinpont where the issue lies. I assume somewhere between the HBA to the backplane. I feel that the HBA and cables are probably good since everything worked completely fine on a few boots and it's the backplane showing issues right on first power up. As soon as the system powers up you'll see the various LED examples before the system even fully boots.
Any ideas based on the descibed behavior? I've read about flashing the backplane itself, perhaps that'll besomething to try.
Maybe reseating all the molex connection to the backplane as well. Maybe some sort of power issue to the backplane? Should try that first.
I spent some time running memtest86+ on all the memory I have and all looked well there, did some stress tests and let it run to see overall system stability- no crashing or anything so all went well.
Fully loaded my BPN-SAS2-846EL1 with my drives, I have it connected to myAOC-S3008L-L8E using SM branded SFF-8643 to SFF-8087 cables. Flashed the HBA using IT Firmware directly from SuperMicro using their scrip which made it super easy.
Go into TrueNAS scale. saw all my disks, created my pool and vdevs, and all was looking normal and well.
I wanted to next destroy my pool and run a badblocks script on every drive. I know it'll probably take a long ass time so I shut down the system and called it a night to get ready to do that this weekend.
Next power up I am seeing all sorts of odd behavior. First was backplane showing no lights at all, no disks recognized in TrueNAS. Shut down. Immediately on power up all backplane LEDs are solid red. Shut down. Next power up I have a mix of solid red backplane LEDs, blinking red LEDs, and some completely off. Each reboot would either be No LEDs, all LEDs red, or a weird mix of on/off/blinking fault LEDs.
I've also fully removed both PSUs just to do a full power drain. I've tried different PSUs as well. I have a pair of PWS-920P-SQs as well as the original PWS-1K21P-1Rs that came with the chassis.
I did notice once I had a nice pattern of blinking blue LEDs going across the drive bays. bay 5 - bay 11 -bay 17 - bay 23. Blue LEDs blinking across those top 4 in order once. I think all disks were recognized just fine that time.
When I swapped to try using a single PWS-1K21P-1R everything came up normal. I thought to myself, am I not using big enough power supplies somehow? Watching the boot I never see watt usage on my kill a watt go above 400-500w. However I shut the system down and on boot up I got the same lightshow (all red) from the backplane. So it seems to happen to regardless of PSU used?
Hard to pinpont where the issue lies. I assume somewhere between the HBA to the backplane. I feel that the HBA and cables are probably good since everything worked completely fine on a few boots and it's the backplane showing issues right on first power up. As soon as the system powers up you'll see the various LED examples before the system even fully boots.
Any ideas based on the descibed behavior? I've read about flashing the backplane itself, perhaps that'll besomething to try.
Maybe reseating all the molex connection to the backplane as well. Maybe some sort of power issue to the backplane? Should try that first.
Last edited:
Does it work with only a few disks? Also, when it isn't working right but the system still manages to boot, check if the backplane itself is being recognized as a SCSI enclosure device. e.g. on my BPN-SAS3-826EL1-N4 I see this (the SAS3x28 is the backplane):
Code:
$ lsscsi -g
[0:0:0:0] disk ATA Hitachi HUA72302 A840 /dev/sda /dev/sg0
[0:0:1:0] disk ATA HUH728080ALN600 TD40 /dev/sdd /dev/sg1
[0:0:2:0] disk IBM-ESXS PX02SMF020 S405 /dev/sdb /dev/sg2
[0:0:3:0] disk ATA HGST HUS728T8TAL W460 /dev/sdc /dev/sg3
[0:0:4:0] disk HGST HUH721008AL4201 B384 /dev/sdf /dev/sg4
[0:0:5:0] disk ATA HGST HUS728T8TAL W460 /dev/sde /dev/sg5
[0:0:6:0] disk IBM-ESXS PX02SMF020 S405 /dev/sdg /dev/sg6
[0:0:7:0] enclosu LSI SAS3x28 0601 - /dev/sg7
[1:0:0:0] disk ATA WDC WD3200BEVT-0 1A11 /dev/sdh /dev/sg8
[N:0:1:1] disk KCM51VUG1T60__1 /dev/nvme0n1 -
[N:1:1:1] disk KCM51VUG1T60__1 /dev/nvme1n1 -Even with a a single disk in I get nothing. I was only able to boot once and successfully see all the disks. Ever since that first time I've been unable to do so.
Probably going to sound like a dummy for this but I can't seem to run the lsscsi command? Getting command not found. I'm trying to run it as root right through the linux shell option directly connected to the machine.
For what it's worth, in the CLI I get the following:
Code:
>storage enclosure query
(empty list)
>storage disk query name
(only lists my two book disks)Not that I am aware of, was nothing I configured. Just as above I've even booted it with no disks to see what would happen and nada.
While going in and out on the shells I did catch this fill up the screen which appears to be relevant:
Swapped the PSUs back and forth a few times just to see if I'd get a difference. First boot up all red LEDs- then during boot, at SuperMicro splash screen they all went away- did our little blue LED dance. Captured the following on TrueNAS Core booting:
Experiment time I suppose. Removed the 4 drives TrueNAS could see. It did not like that. Results:
Last edited:
Does the strange behavior continue if you remove the SFF-8643 to SFF-8088 cables?
If no, have you tried each of the inter-connect cables individually?
Edit
also verify that the molex connectors (I believe there are 6) are firmly seated into the backplane and that none of the molex pins have popped a bit back out of their connector (this is an insidious problem with molex connectors when the pins don't quite line up and you have to use a lot of force to get the molex connectors to mate).
Last edited:
The "all red lights at power on" is completely normal. Even bays without drives will show this.
Then, red lights will disappear and blue lights will come on solid for SAS drives while SATA drives have nothing. Next, you'll see a pattern of flashing blue lights as the controller accesses the drives. Once this finishes, you'll end up with solid blue for SAS, nothing for SATA, with slowly blinking red lights on drives that are marked as hot spares in the controller (does not apply to IT mode).
I'd check power to the backplane first (maybe you knocked a cable loose), then test the HBA with direct-connect to a drive or two using a SFF-8087 to SATA fan out cable. If that goes fine, then connect the HBA back to the backplane. One thing to note is that if you are running the chassis with the top open for testing, your HBA might not be getting enough airflow and you might need a cooling fan on it. This should not affect boot, but will show as a problem when you are running badblocks.
Yeah I've tried them individually but get similar hit or miss results. I'll double check the molex pins, that's a good point. It's definitely a bit hard to see down there so that's always a possibility. I'll get my phone in there to get a better look or worst case take out the whole backplane to keep testing things. I really don't want to unscrew it all again but if I have to I shall lol.
Thanks for clarification on LED situation. When things are working fine the few times they did, once actually in TrueNAS, etc there are no LEDs lit if everything is working as normal (using all sata drives so this makes sense per your explanation)- but that's not usually how it goes.
The flashing blue light pattern is how I can tell things are going well, so that makes total sense. However it's that pattern which does or does not happen at times. Sometimes it will, sometimes they'll all just stay red the whole way through. Sometimes they'll boot with no LEDs lit and stay that way as well. So getting a variety of different behaviors on every boot.
Will definitely recheck the power first. As itronin said perhaps something got funky with the molex connections.
Since there were 1-2 times that everything booted completely normally and all drives were seen by the time I was in the TrueNAS GUI I am assuming the HBA and cables are good- was able to create a pool without issue. Can't be 100% sure though.
It's as if sometimes the backplane is good to go and has no issues on a full power reset or it'll exhibit non-responsive behavior right from first boot all the way into the backplane not being seen in TrueNAS. I feel like I can tell pretty early via the initial boot procress even before it even hits TrueNAS itself booting if things will work. All red LEDs will or will not go away during the system starting up before TrueNAS is booting. If that makes sense? Hopefully.
I've noticed that I'll get more variantion between reboots if I completely remove power/PSUs- but could just be coincidence.
First will be double checking all the molex though. They are a real pain in the ass to disconnect and reconnect, really tight connections.
appreciate the help as always.
may seem odd to suggest this since you feel the HBA is behaving - but - you might try it in a different slot on the motherboard.
Haha yeah I tried that. Also re-flashed it again too without issue.
I ran into a new change today. When booting into TrueNAS I got stuck on the Welcome to Grub screen for a good while, maybe 5-10 minutes. Not as bad as seen in this thread. In that time the LEDs on the backplane went from off to all red. No disks detected of course.
hmmm if you really feel like it... you could see if the same behavior occurs with CORE (you'd have to install it, maybe swap in a spare SSD if you have one). I'd expect the same behavior but ya never know.
I think in general something's not right with your system. I mean you aren't booting off the HBA are you? IIRC you're doing an internal SSD off a sata port right? I wonder if its just happenstance that whatever "isn't right" is being more symptomatic against the HBA than anything else.
anoither shot in the dark, go down to one stick of memory, and/or factory reset the motherboard settings?
Yeah I'm using 2x SataDOMs (SuperMicro SuperDOMs) for the boot drives.
I actually did try core as well just to see if anything would be different- same behavior though.
Hmm I mean with the backplane out of the picture the system works totally fine. Like I said I was booting into linux and windows server to try out different stress test tools, as well as boot into memtest86+ and all that went completely fine. It's only the backplane itself that seems to exhibit any sort of issue.
I think a great way to eliminate the HBA as an issue would be using the SFF-8087 to SATA fan out cable like nabsltd mentioned. Or rather SFF-8643 to SATA since that is the connection on the HBA (AOC-S3008L-L8e+). I can get a cheap cable on amazon for $12 just to test it out. I went ahead and ordered one to arrive tomorrow.
For the record, HBA -> Backplane connection right now is being handled by 2x Supermicro CBL-SAST-0507-01 SFF-8643 to SFF-8087 cables.
Still didn't touch the firmware on the backplane itself though. I can't seem to find a lot of info on how to do that. From posts I've read people had to call into SuperMicro support to be given the firmware. I may call in myself and give it a shot unless I can find a newer firmware and instructions somewhere.
Last and most expensive step will be to buy a replacement backplane.
My bad. I thought you had a SAS2 controller because of the SAS2 backplane.
Note that another way to somewhat test the HBA and backplane without all the drives coming into play is to connect the HBA to J0, then connect a SATA fan out to J1. This tests the backplane's multiplexor chip, as data has to pass through it, but removes all the drives and the drive connectors from the equation. Remember to pull out all the drive bays (far enough to avoid electrical connection is enough) when you test with the fan out this way. I have a bunch of SFF-8087 to SATA fan out cables that I will never use again...if you want one for free, send me a PM.
You might be experiencing a few different issues at the same time and not just a single one.
I have seen strange problems with SATA DOMs that resulted in similar issues as some of the pauses and failures you have seen for booting. If you can, I suggest booting from USB/DVD and see if you can run smartctl to obtain the wear level on the SATA DOMs that you have (usually called something like "Media_Wearout_Indicator" which will start at a value of 100 and then decrease to 1 or 0 at which the drive will no longer be usable).
Also, you might be experiencing issues with your power distribution board. I believe there have been people who have swapped it out and used the PDB-PT747-4648 (which is really for the 747 chassis, but the board will fit onto the mounting hardware of the board within the 846). They will run $150 new from supermicro. This will still support all the older boards like the X8 and X9, but also newer boards like the X12 and X13 (assuming form factor will let it fit in the chassis) as it has additional EPS cables. It will also provide 4x6+2pin GPU power cables, as well as 8x4pin molex and will still work with 920P-SQ power supplies and even the 1K28P-SQ power supplies.
I have seen strange problems with SATA DOMs that resulted in similar issues as some of the pauses and failures you have seen for booting. If you can, I suggest booting from USB/DVD and see if you can run smartctl to obtain the wear level on the SATA DOMs that you have (usually called something like "Media_Wearout_Indicator" which will start at a value of 100 and then decrease to 1 or 0 at which the drive will no longer be usable).
Also, you might be experiencing issues with your power distribution board. I believe there have been people who have swapped it out and used the PDB-PT747-4648 (which is really for the 747 chassis, but the board will fit onto the mounting hardware of the board within the 846). They will run $150 new from supermicro. This will still support all the older boards like the X8 and X9, but also newer boards like the X12 and X13 (assuming form factor will let it fit in the chassis) as it has additional EPS cables. It will also provide 4x6+2pin GPU power cables, as well as 8x4pin molex and will still work with 920P-SQ power supplies and even the 1K28P-SQ power supplies.
Yeah no worries. I only went with getting a new HBA so that I could keep the original setup with all HBAs/cabling/etc for potential later use. The AOC-S3008L-L8e was affordable so seemed like a good idea.
I only specified exactly just in case it caught someone's eye as "oh you missed something important here" when it comes to the HBA and backplane.
That is a great idea as well thanks for the suggestion. I actually do have SFF-8087 breakout cables from the previous triple HBA to direct backplane so I can use one from there. If I had SFF-8087 to SFF-8087 cables I could test using one of those HBAs too.
Right now though, with my current HBA I used a fan out cable and connected two SSDs. HBA and thus TrueNAS saw them without issue. I think it's safe to say there is nothing wrong with the HBA at this point.
When I reseat a drive it doesn't even spin up- it's as if the backplane is just completely non-responsive.
But what puzzles me is that's it definitly not completely dead power-wise. There are indications that it tries to function properly on boot but runs into some sort of issue. Even now there blue LEDs on the inside... side of the backplane. I can clearly see two in the middle and one on the left side below the PRI_J0 SAS connector but in documentation it doesn't say what these are to indicate. I'll have to take out this backplane to get a better look.
I'm really leaning towards there being something wrong with this backplane but I wish I could figure out what it is. Even if it means replacing it, wich is fine, it is what it is- I wish I could come to a root cause. Physical inspection before I installed it nothing seemed out of place.
I contacted SuperMicro and they actually provided me the backplane firmware and instructions for using Expander Xtools Lite to flash it. Followed the instructions as outlined in the User's Guide I updated the FW and and MFG with the files provided to me by SM support. However after doing all the steps I saw no difference in behavior. The updates given to me by SM support were the exact same versions already on the backplane. Do note that it took several reboots before the backplane was cooperative enough to show itself in Expander Xtools Lite.
They look good to me I think? Let me know if anything stands out.
Code:
SDA Info - Boot Drive Mirror
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0000 100 100 000 Old_age Offline - 0
5 Reallocated_Sector_Ct 0x0000 100 100 000 Old_age Offline - 0
9 Power_On_Hours 0x0000 100 100 000 Old_age Offline - 582
12 Power_Cycle_Count 0x0000 100 100 000 Old_age Offline - 96
15 User_Cpcty_Sector_Cnt 0x0000 100 100 000 Old_age Offline - 247463936
160 Not_In_Use 0x0000 100 100 000 Old_age Offline - 0
161 Not_In_Use 0x0000 100 100 000 Old_age Offline - 169
163 Not_In_Use 0x0000 100 100 000 Old_age Offline - 8
164 Not_In_Use 0x0000 100 100 000 Old_age Offline - 5564
165 Not_In_Use 0x0000 100 100 000 Old_age Offline - 11
166 Minimum_PE_Cycles_TLC 0x0000 100 100 000 Old_age Offline - 0
167 Not_In_Use 0x0000 100 100 000 Old_age Offline - 2
168 Maximum_PE_Cycles_TLC 0x0000 100 100 000 Old_age Offline - 3000
172 Unknown_Attribute 0x0000 100 100 000 Old_age Offline - 0
175 Program_Fail_Count_Chip 0x0000 100 100 000 Old_age Offline - 8
177 Wear_Leveling_Count 0x0000 100 100 000 Old_age Offline - 0
181 Program_Fail_Cnt_Total 0x0000 100 100 000 Old_age Offline - 0
187 Reported_Uncorrect 0x0000 100 100 000 Old_age Offline - 0
192 Power-Off_Retract_Count 0x0000 100 100 000 Old_age Offline - 13
194 Temperature_Celsius 0x0000 100 100 000 Old_age Offline - 37 (Min/Max 21/88)
195 Hardware_ECC_Recovered 0x0000 100 100 000 Old_age Offline - 252
197 Current_Pending_Sector 0x0000 100 100 000 Old_age Offline - 0
198 Offline_Uncorrectable 0x0000 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0000 100 100 000 Old_age Offline - 0
231 SSD_Life_Left 0x0000 100 100 000 Old_age Offline - 100
232 Available_Reservd_Space 0x0000 100 100 000 Old_age Offline - 100
233 NAND_Writes_1GiB 0x0000 100 100 000 Old_age Offline - 347
241 Lifetime_Writes_GiB 0x0000 100 100 000 Old_age Offline - 340
242 Lifetime_Reads_GiB 0x0000 100 100 000 Old_age Offline - 132
SDC Info - Boot Drive Mirror
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0000 100 100 000 Old_age Offline - 0
5 Reallocated_Sector_Ct 0x0000 100 100 000 Old_age Offline - 0
9 Power_On_Hours 0x0000 100 100 000 Old_age Offline - 581
12 Power_Cycle_Count 0x0000 100 100 000 Old_age Offline - 97
15 User_Cpcty_Sector_Cnt 0x0000 100 100 000 Old_age Offline - 247463936
160 Not_In_Use 0x0000 100 100 000 Old_age Offline - 0
161 Not_In_Use 0x0000 100 100 000 Old_age Offline - 161
163 Not_In_Use 0x0000 100 100 000 Old_age Offline - 16
164 Not_In_Use 0x0000 100 100 000 Old_age Offline - 3904
165 Not_In_Use 0x0000 100 100 000 Old_age Offline - 13
166 Minimum_PE_Cycles_TLC 0x0000 100 100 000 Old_age Offline - 0
167 Not_In_Use 0x0000 100 100 000 Old_age Offline - 1
168 Maximum_PE_Cycles_TLC 0x0000 100 100 000 Old_age Offline - 3000
172 Unknown_Attribute 0x0000 100 100 000 Old_age Offline - 0
175 Program_Fail_Count_Chip 0x0000 100 100 000 Old_age Offline - 16
177 Wear_Leveling_Count 0x0000 100 100 000 Old_age Offline - 0
181 Program_Fail_Cnt_Total 0x0000 100 100 000 Old_age Offline - 0
187 Reported_Uncorrect 0x0000 100 100 000 Old_age Offline - 0
192 Power-Off_Retract_Count 0x0000 100 100 000 Old_age Offline - 13
194 Temperature_Celsius 0x0000 100 100 000 Old_age Offline - 33 (Min/Max 12/85)
195 Hardware_ECC_Recovered 0x0000 100 100 000 Old_age Offline - 903
197 Current_Pending_Sector 0x0000 100 100 000 Old_age Offline - 0
198 Offline_Uncorrectable 0x0000 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0000 100 100 000 Old_age Offline - 0
231 SSD_Life_Left 0x0000 100 100 000 Old_age Offline - 100
232 Available_Reservd_Space 0x0000 100 100 000 Old_age Offline - 100
233 NAND_Writes_1GiB 0x0000 100 100 000 Old_age Offline - 244
241 Lifetime_Writes_GiB 0x0000 100 100 000 Old_age Offline - 239
242 Lifetime_Reads_GiB 0x0000 100 100 000 Old_age Offline - 24Navigating and understanding PDB compatibility is pretty annoying though. I've been able to browse through a lot of them they they sold but it's kind of hard to tell what the hell is different between them all and if they can actually be mounted without issue. I had googled this a while back and I recall reading a thread where someone replaced their PDB but had to make modifications for it to mount properly so chassis compatibility is a concern.
I also found SM sells a lot of power adapter cables themselves. (like CBL-0424L) Their store webpage is kind of a pain in the ass to navigate and find parts properly though. I was curious if they sold their own molex -> sata or other X -> sata adapters. Putting a video card in here eventually would be nice for plex transcoding.
I've already gone ahead and ordered a replacement backplane off ebay. Let's see what happens, should be here by the end of the week.
Last edited:
BTW on a completely separate note I was able to get my hands on a new top panel. However even with the new top panel I am unable to complete close the top. There is always a gap on the left side and the clip on that side can't reach/latch closed. I am going to assume there must be something off with the chassis itself somehow... Looking at it I can't tell where it's not lined up though.
Maybe the whole thing was dropped at some point and not just the panel.
Not the end of the world, it closes far enough to not let air escape as far as I can tell. Would be nice if could close all the way though like it's supposed to haha.
Maybe the whole thing was dropped at some point and not just the panel.
Not the end of the world, it closes far enough to not let air escape as far as I can tell. Would be nice if could close all the way though like it's supposed to haha.
You can always pull out an engineering or framing square to see if all the corners are square. You will then be able to see which sides have gone out of square and can attempt to shift them back into square with some wood blocking/framing and leverage.