Help with motherboard and CPU for new FreeNAS build

[itronin]

sheesh! You are having some kind of bad luck right now.

just so we understand how you have this cabled (going from previous descriptions)

Edit: Its a 4U Rosewil 4112, 4U Rosewill 4412, it has qty 3 4-Bay-Hot-Swap backplanes (each has 2 molex, and 4 sata ports) and they are cabled going from left to right as follows:

Please forgive crude ascii drawing

[4 x SATA cables from motherboard] [4 x SATA cables from motherboard] [SFF-8643 to 4xSATA forward breakout cable from motherboard]

[2 x molex] [ 2 x molex] [2x molex] <- Power feed
[S S S S] [S S S S] [L L L L] <- Cable Type S = single SATA cable, L = SFF-8643 to 4x sata breakout
[D D D D] [M D D D] [D D D M] <- D = Drive installed/appeared , M = Drive installed but missing

I don't see a pattern (1 Sata connected drive), 1 SAS connected drive, different slots in different 4 bays... Could be power.

I would definitely make sure the molex's are firmly seated in the 4 bay hot swaps. With the system turned off you might consider pulling them and plugging them back in insuring they are firmly seated. Its possibly they are not firmly seated.

What PSU are you using? Does it have modular power cables? Are you feeding 1 4 bay hot swap bay module (2 molex) with a molex cable PER modular PSU connection or you chaining all the molex connections from ONE modular PSU connection?

re. the 4112, seems to be popular with the plex/media server at home crowd (serverbuilds.net). Takes commodity parts, PSU etc. I have one of these 4 bay hot swaps (got it on sale, its for a future project that is down in the priority list). The 4 bay is plastic, plastic, and plastic, cheaply made, but fan is replaceable, It is hot swap, has blinky lights and pretty simple (no sideband connection etc) so maybe that's why so many folks seem to like it.

 

[Rand__]

Its a 4U Rosewil 4112, it has

Sorry for the OT, but WTFH - not even willing to display the page?





On topic,
its unlikely that a drive bay should fail just by removing and reinserting a drive unless it was flaky to begin with.
Maybe try playing a bit more with other slots to see if the same happens, or the drive resume regular operations in the questionable slots...

 

[rskoss]

Sorry for the OT, but WTFH - not even willing to display the page?
View attachment 17230

4412

I'm heading for the basement to review how I have the power cabled. Brb....

 

[itronin]

Sorry for the OT, but WTFH - not even willing to display the page?
View attachment 17230




On topic,
its unlikely that a drive bay should fail just by removing and reinserting a drive unless it was flaky to begin with.
Maybe try playing a bit more with other slots to see if the same happens, or the drive resume regular operations in the questionable slots...

Double WTFH...

 

[rskoss]

Double WTFH...

This is the page you want: RSV-L4412 - 4U Rackmount Server Case or Chassis, 12 SATA / SAS Hot-Swap Drives, 5 Cooling Fans Included - Rosewill

 

[rskoss]

I have two cables from the PSU to the drives, each cable has 3 molex and there are 6 power connectors across the drive bays.

The two rear case fans take a molex and that is where my last molex power cable that came with the PSU went.

IIRC, the PSU came with one cable with molex connectors and I purchased two of these. Do I need to buy another cable and only use two molex per cable?

 

[itronin]

This is the page you want: RSV-L4412 - 4U Rackmount Server Case or Chassis, 12 SATA / SAS Hot-Swap Drives, 5 Cooling Fans Included - Rosewill

I had the correct linkage wrong name.

OT but...

so much for brexit and the UK not being part of the EU (via VPN server in London, UK)

 

[itronin]

I have two cables from the PSU to the drives, each cable has 3 molex and there are 6 power connectors across the drive bays.

The two rear case fans take a molex and that is where my last molex power cable that came with the PSU went.

IIRC, the PSU came with one cable with molex connectors and I purchased two of these. Do I need to buy another cable and only use two molex per cable?

well rated PSU, well recommended and pretty std. in builds using the 4412 (serverbuilds.net).

so FWIW and troubleshooting this - My advice start simple and reduce the problem.

0) Power off the server
1) don't bother powering the rear fans for the moment. so unplug that molex cable.
2) unplug the two third party cables you purchased from the PSU and the 4 drive hot swap bays and set those aside
3) Take that Molex cable that came with the PSU (AMZ link seems to show a psu cable with 4 Molex connectors on it) and connect that to power the right hand most 4 bay hot swap module (the one that has the drive that does not show up) Make sure the molex is firmly seated in both receptacles on the 4 bay.
4) power on the server. Do the 4 drives show up?
5) Shut down, then turn off the power supply switch, wait 30 seconds
6) turn on the power supply switch and then power up the servers Do the 4 drives still show up?
repeat until you are confident the drives show up.

If all the drives do not show up on the right hand side then you may have a bad 4 bay module.

If the drives do show up and repeatedly on the right hand side repeat the above process for the middle 4 bay module using only the molex power cable that was shipped with the PSU

If all the drives do show up and repeatedly move on to the left and side and repeat the above process.

don't worry about properly routing the power cable for now.

if you can get consistent drive appearance in each of the 4 bay modules we can figure out a next step.

 

[Markess]

IIRC, the PSU came with one cable with molex connectors and I purchased two of these. Do I need to buy another cable and only use two molex per cable?

FWIW I've bought a few COMeap brand cables through Amazon and the quality has varied depending on the type. Some were fine, others not so much. I've not bought one of their Molex cables though. If, by chance it is one of the third party Molex cables, Its possible to buy extras directly from EVGA.

 

[rskoss]

well rated PSU, well recommended and pretty std. in builds using the 4412 (serverbuilds.net).

so FWIW and troubleshooting this - My advice start simple and reduce the problem.

0) Power off the server
1) don't bother powering the rear fans for the moment. so unplug that molex cable.
2) unplug the two third party cables you purchased from the PSU and the 4 drive hot swap bays and set those aside
3) Take that Molex cable that came with the PSU (AMZ link seems to show a psu cable with 4 Molex connectors on it) and connect that to power the right hand most 4 bay hot swap module (the one that has the drive that does not show up) Make sure the molex is firmly seated in both receptacles on the 4 bay.
4) power on the server. Do the 4 drives show up?
5) Shut down, then turn off the power supply switch, wait 30 seconds
6) turn on the power supply switch and then power up the servers Do the 4 drives still show up?
repeat until you are confident the drives show up.

If all the drives do not show up on the right hand side then you may have a bad 4 bay module.

If the drives do show up and repeatedly on the right hand side repeat the above process for the middle 4 bay module using only the molex power cable that was shipped with the PSU

If all the drives do show up and repeatedly move on to the left and side and repeat the above process.

don't worry about properly routing the power cable for now.

if you can get consistent drive appearance in each of the 4 bay modules we can figure out a next step.

Using the cable that came with the PSU that has 3 molex connectors, I used only 2 of them to plug into the right most drive bay. No other drive bays have power.

I toggled the passthrough of all the spinners so I could see the drives in ESXi and not have to boot TN.

ESXi shows 3 spinners. Is there a way to see serial numbers in ESXi? I've been getting that from TN. IAE, down a drive. So this really indicates a bad bay module - but, not seeing a drive in the middle module earlier gives me pause. The chance of two bad bay modules in a case? Pretty crappy QA from Rosewill - and I think we would have heard an uproar in the community if that was the case.

 

[rskoss]

FWIW I've bought a few COMeap brand cables through Amazon and the quality has varied depending on the type. Some were fine, others not so much. I've not bought one of their Molex cables though. If, by chance it is one of the third party Molex cables, Its possible to buy extras directly from EVGA.

I'll buy from EVGA if I need more (despite the convenience of Amazon). But right now I'm only using a single cable and it's the one that came with the PDU, and I'm still down a drive.

 

[Rand__]

Is there a way to see serial numbers in ESXi?

dmesg might show it, or "esxcli storage core device list" or the storage device view in the gui. Serial should be in device name like the bold part here for an old intel ssd here

Display Name: Local ATA Disk (t10.ATA_____INTEL_SSDSC2BW180A3L__________________00_CVCV224003TC180EGN__)

 

[itronin]

I'll buy from EVGA if I need more (despite the convenience of Amazon). But right now I'm only using a single cable and it's the one that came with the PDU, and I'm still down a drive.

Yes sounds like a bad bay.

You might try the same process with the evga power cable on the other bay that had a drive disappear and see if it reappears.

FWIW I find it easier to pull drive information from FreeNAS/TNC (rather than ESXI - lack of knowledge on my part). Honestly just going into the BIOS for the SAS controller or SATA ports should show the drives too. Ie. not even booting into an OS. You should be able to see the SN's via the SAS controller assuming standard LSI type BIOS - not sure if you can get that via the BIOS for the sata ports.

 

[Markess]

I'll preface this by saying that I have no idea if it matters, so someone feel free to educate me..

@rskoss is using a consumer PSU. Its a really good quality consumer PSU, half the PCs in my house have EVGA G3s in them. But, its not designed with lots of drives in mind. The Superflower Leadex II platform its based on has known current inrush stability issues on startup.

This isn't a big deal with a desktop/gaming PC, but could it be a problem with 12xHDD if they aren't staggered for spinup? I'd assume any startup issues caused by startup current would be resolved with pulling & reseating the drives (which @rskoss did), but I have no real knowledge in this area, so was wondering as much for me as for him.

 

[rskoss]

dmesg might show it, or "esxcli storage core device list" or the storage device view in the gui. Serial should be in device name like the bold part here for an old intel ssd here

Display Name: Local ATA Disk (t10.ATA_____INTEL_SSDSC2BW180A3L__________________00_CVCV224003TC180EGN__)

I see letter/numbers but nothing corresponds to the serial numbers.

 

[rskoss]

I see letter/numbers but nothing corresponds to the serial numbers.

I misspoke. Seeing them clear as can be.

I plugged in two molex connectors for the middle bay. All 4 drives show in ESXi - and I see serial numbers. I swear I didn't see serial numbers when I was looking at the 3 drives that appeared in the right bay.

 

[itronin]

I misspoke. Seeing them clear as can be.

I plugged in two molex connectors for the middle bay. All 4 drives show in ESXi - and I see serial numbers. I swear I didn't see serial numbers when I was looking at the 3 drives that appeared in the right bay.

hmmm.... SAS driver difference in ESXI? middle and left are SATA and right hand side 4 bay is ESXI...

Maybe this was asked before and I missed it, but is your 3008 in IT Mode or R Mode? Reading the product description for this motherboard, it looks like it comes by default in R Mode from the product description: "8 SAS3 (12Gbps) via Broadcom® 3008; RAID 0, 1, 10"

My apologies if this wasn't asked before...

 

[rskoss]

hmmm.... SAS driver difference in ESXI? middle and left are SATA and right hand side 4 bay is ESXI...

Maybe this was asked before and I missed it, but is your 3008 in IT Mode or R Mode? Reading the product description for this motherboard, it looks like it comes by default in R Mode from the product description: "8 SAS3 (12Gbps) via Broadcom® 3008; RAID 0, 1, 10"

My apologies if this wasn't asked before...

we assumed it came in IT mode but we never verified it.

How do I check this and what do I do about it if it isn’t?

 

[itronin]

we assumed it came in IT mode but we never verified it.

How do I check this and what do I do about it if it isn’t?

You need to go into the LSI SAS BIOS (the BIOS will display on the screen and prompt you for the key combination to go into the BIOS - Ctrl-C IIRC) and then select the the card (should only be one). Firmware version should be at the top. Might be easier once you get to that screen to go ahead and grab a picture of the screen (camera/cell camera), post that. You can also go into SAS Topology and the controller will tell you what it sees connected... get a picture of that too.

I'd go ahead and have power applied to the 4 Bay that the SAS contorller is connected to...

If you are running R mode (Raid Mode) firmware it will need to be flashed to IT Mode for TNC.

 

[rskoss]

You need to go into the LSI SAS BIOS (the BIOS will display on the screen and prompt you for the key combination to go into the BIOS - Ctrl-C IIRC) and then select the the card (should only be one). Firmware version should be at the top. Might be easier once you get to that screen to go ahead and grab a picture of the screen (camera/cell camera), post that. You can also go into SAS Topology and the controller will tell you what it sees connected... get a picture of that too.

I'd go ahead and have power applied to the 4 Bay that the SAS contorller is connected to...

If you are running R mode (Raid Mode) firmware it will need to be flashed to IT Mode for TNC.

Power is applied to right most bay. I'm going to get another power cable. I'm not comfortable with the strain trying to get the cable across bays.

Here is a link to the photo you requested:

Dropbox - File Deleted - Simplify your life

www.dropbox.com