Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server

MACscr

Member
May 4, 2011
119
3
18
So i have two nodes with the 1068e mezz card flashed to IT mode and ive recabled both nodes (two different systems, both node 1) with the official dell cables for those sas cards, yet for some reason, both servers only see 5 out 6 drives. Looks like its always the 4th sata port on the node board. Ive tried switching a sata cable a working port (port 3 on the mobo) from the controller to port 4 on the motherboard and it still doesnt see it. Any ideas?
 
Last edited:

Clownius

Member
Aug 5, 2013
85
0
6
Could it be further down the chain? Try swapping out the small board at the end of the motherboard all the cables plug into. I cant remember what its called sorry.

Oh i have 4 of these mezz cards plus all the required cabling, brackets etc if anyones interested. I havent tested them yet but i have decided to go with Raid Cards. Let me know if your interested. Be aware im based in Australia.
 

MACscr

Member
May 4, 2011
119
3
18
Exact port on two different nodes on two different chassis from two different sellers? Doubtful its a board failure issue.
 

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
So i have two nodes with the 1068e mezz card flashed to IT mode and ive recabled both nodes (two different systems, both node 1) with the official dell cables for those sas cards, yet for some reason, both servers only see 5 out 6 drives. Looks like its always the 4th sata port on the node board. Ive tried switching a sata cable a working port (port 3 on the mobo) from the controller to port 4 on the motherboard and it still doesnt see it. Any ideas?
Verify that you have the right breakout cables between the interposer board and the backplane and that they are plugged in correctly. The two connectors on the interposer are different - one wires 4 drives and one wires only two. Make sure that you used the correct cables, which have to be different too. If you used the "wires three drives" cables from another node in the chassis you'd have exactly the outcome you describe. If is a different cable set to wire 6 drives to a single node.

Less likely, if you do have the right cables: swap the 8087 end of the cables between the two connectors on the interposer board and see if the problem goes away (or moves). This will at least give you some troubleshooting hints.
 

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
Onboard works fine with 4TB drives. Onboard is plain old Intel ICH10R, which is plenty happy with advanced format drives. You might need to update the BIOS to get it working.

The 1068e-based SAS mezz cards would be limited to 2TB. That's a 1068e issue.
 

MACscr

Member
May 4, 2011
119
3
18
Ah man, that seems crazy that they make you use different cables to the backplanes as well. Man, more wasted money on cables. Any idea on the part numbers?
 

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
They are in this thread somewhere...of course, there are only 54 pages to scan back through!

You can always go to Monoprice and just get a short, thin wire 8087 forward breakout cable. It will work fine as long as you don't care about getting the lights working right (and for that you can just jumper the backplane for "drive control" and they should work again). One standard 8087 to 4-SATA forward breakout cable and then use the "3-drive" cable on the side that only wires two drives (leave the third SATA plug dangling and DONT plug in the SGPIO to the backplane - it will light the lights on the wrong drive the way its wired up).
 

techwerkz

New Member
Aug 20, 2013
12
0
0
Fort Myers, FL
www.techwerkz.com
Onboard works fine with 4TB drives. Onboard is plain old Intel ICH10R, which is plenty happy with advanced format drives. You might need to update the BIOS to get it working.

The 1068e-based SAS mezz cards would be limited to 2TB. That's a 1068e issue.
Already updated the BIOS. It's only detecting 1687GB or the 4TB. I was assuming it would work, and after researching the controller I came across this thread:

Intel Rapid Storage Manager not seeing full capacity of hdd's - [H]ard|Forum

It seems their answer was it has to be a UEFI BIOS. It's just too identical to not be a problem. Anyone else confirm they have used 4TB with the onboard controller? My 2TB drives detect just fine. It's not so much a big deal as the 4TB drives will need to used with a controller for ESX, but I just wanted to do some simple testing first. Was going to see how Hyper-V 2012 handled these type of disks. This particular node will be running Exchange 2013 and the 2012 PDC/FSMO.
 

techwerkz

New Member
Aug 20, 2013
12
0
0
Fort Myers, FL
www.techwerkz.com
They are in this thread somewhere...of course, there are only 54 pages to scan back through!

You can always go to Monoprice and just get a short, thin wire 8087 forward breakout cable. It will work fine as long as you don't care about getting the lights working right (and for that you can just jumper the backplane for "drive control" and they should work again). One standard 8087 to 4-SATA forward breakout cable and then use the "3-drive" cable on the side that only wires two drives (leave the third SATA plug dangling and DONT plug in the SGPIO to the backplane - it will light the lights on the wrong drive the way its wired up).
The Monoprice part is 8186 (just ordered some and confirmed they work great).

I didn't know about the jumper. I will go look for that now.
 

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
Already updated the BIOS. It's only detecting 1687GB or the 4TB. I was assuming it would work, and after researching the controller I came across this thread:

Intel Rapid Storage Manager not seeing full capacity of hdd's - [H]ard|Forum

It seems their answer was it has to be a UEFI BIOS. It's just too identical to not be a problem. Anyone else confirm they have used 4TB with the onboard controller? My 2TB drives detect just fine. It's not so much a big deal as the 4TB drives will need to used with a controller for ESX, but I just wanted to do some simple testing first. Was going to see how Hyper-V 2012 handled these type of disks. This particular node will be running Exchange 2013 and the 2012 PDC/FSMO.
I'm reasonably sure I tested the 4TB drives in the C6100 on the MB port - but at the time I also had some of the drives on an M1015 plugged into the node too. I know I've used 4TB drives on another 5520 chipset 1366 MB with ICH10R (same basic design architecture) and they worked great.

I happen to have a couple brand new 4TB drives that I haven't put to use yet. I'll test them on the C6100 MB ports tonight and post what I find.
 

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
Well I'll be dipped...the on-board SATA on the C6100 does appear to be limited to 2TB. First time I've ever seen this enforced on an ICH10-based controller. I guess when I was testing before the 4TB drives were on the M1015 (which makes sense when I think back - I was performance testing with ZFS and wanted the SATA-III transfers).

See image attached. Clearly a 4TB drive (Seagate STM4000DM000), but only showing 1.8TB recognized. Kinda sucks!

 
Last edited:

Clownius

Member
Aug 5, 2013
85
0
6
Well I'll be dipped...the on-board SATA on the C6100 does appear to be limited to 2TB. First time I've ever seen this enforced on an ICH10-based controller. I guess when I was testing before the 4TB drives were on the M1015 (which makes sense when I think back - I was performance testing with ZFS and wanted the SATA-III transfers).

See image attached. Clearly a 4TB drive (Seagate STM4000DM000), but only showing 1.8TB recognized. Kinda sucks!

What counts is what does the OS see?

On older systems with 3TB drives i saw similar. Full size not showing in BIOS but works fine in OS.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Well I'll be dipped...the on-board SATA on the C6100 does appear to be limited to 2TB. First time I've ever seen this enforced on an ICH10-based controller. I guess when I was testing before the 4TB drives were on the M1015 (which makes sense when I think back - I was performance testing with ZFS and wanted the SATA-III transfers).

See image attached. Clearly a 4TB drive (Seagate STM4000DM000), but only showing 1.8TB recognized. Kinda sucks!
It must be a BIOS version problem. I have a 4TB Hitachi drive in one of my c6100 nodes and it shows up as full size.
 

MACscr

Member
May 4, 2011
119
3
18
They are in this thread somewhere...of course, there are only 54 pages to scan back through!

You can always go to Monoprice and just get a short, thin wire 8087 forward breakout cable. It will work fine as long as you don't care about getting the lights working right (and for that you can just jumper the backplane for "drive control" and they should work again). One standard 8087 to 4-SATA forward breakout cable and then use the "3-drive" cable on the side that only wires two drives (leave the third SATA plug dangling and DONT plug in the SGPIO to the backplane - it will light the lights on the wrong drive the way its wired up).
So unplug the SGPIO from the 3 drive wire that is only using cables 1 and 2? Lastly, where is this jumper located? Is there one for each set of 3 that I would need to set it on? Whats the point of SGPIO if it can be done without it and less cables? Thanks for your help. I luckily already have two of these breakout cables I can use that I had accidentally bought for another system that actually had needed reverse breakout cables.
 

MACscr

Member
May 4, 2011
119
3
18
So unplug the SGPIO from the 3 drive wire that is only using cables 1 and 2? Lastly, where is this jumper located? Is there one for each set of 3 that I would need to set it on? Whats the point of SGPIO if it can be done without it and less cables? Thanks for your help. I luckily already have two of these breakout cables I can use that I had accidentally bought for another system that actually had needed reverse breakout cables.
Never mind. It think I have it figured out. Not 100% sure how to identify the drives as I dont see an identify option in lsiutil and 99% of the diagnostic options seem to be for SATA only. =/
 

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
What counts is what does the OS see?

On older systems with 3TB drives i saw similar. Full size not showing in BIOS but works fine in OS.
Yup. Dba and Clownius are correct. BIOS only sees 2TB, but once you load the OS the 4TB drives is seen full size. Thought for a minute that I was going completely crazy - I really did remember using 4TB drives on this thing. Feeling better. I'll post a screenshot tonight when I have more time.

@techwerkz: Were you trying to boot from the 4TB drive or just use it as a data drive?
 
Last edited:

techwerkz

New Member
Aug 20, 2013
12
0
0
Fort Myers, FL
www.techwerkz.com
Yup. Dba and Clownius are correct. BIOS only sees 2TB, but once you load the OS the 4TB drives is seen full size. Thought for a minute that I was going completely crazy - I really did remember using 4TB drives on this thing. Feeling better. I'll post a screenshot tonight when I have more time.

@techwerkz: Were you trying to boot from the 4TB drive or just use it as a data drive?
Just a data drive. This node will be running FreeNAS or something similar 6x4TB. I just wanted to do some testing and already had those disks in the chassis. I will be installing a 9260-8i today on that node. It was seeing all 6, just at under 2TB in the BIOS.

I just wanted to make sure I wasn't going crazy either.
 
Last edited:

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
Its just what it shows in the BIOS. Don't forget that you can't BOOT from a GPT drive unless you have UEFI BIOS, but you can certainly use it for data.