Dell C6100 XS23-TY3 2U 4-Node (8 CPU) Cloud Server

[Dragon]

Found a bunch of useful cool shits online lately, thought I'd share some with STH:

ORICO 1025ss - 2.5" to 3.5" HDD converter, hot swap compatible, US$16 each
Extra points to ORICO for having the sense of humor to make it look like a normal 3.5 SATA hdd, they look exactly the same to the casual observer:

ORICO PHC-35 - HDD/SSD protector (protection against shock & liquid), US$2 each
No more excuse to have your hotswap HDDs lying around unprotected

And no, I am not affiliated with ORICO, I am just impressed with some of their useful products and attention to details. Not going to post any spam links either, but I've provided the model numbers for anyone interested.

 

[Tsmooth3]

What is a simple and cheap way to attach 10 total drives to a single node? I have 10 open drive slots in my C6100 and an M1015 I could use, but the cabling is a mystery to me. How do I hook up extra drives to the M1015? Can it be done? My alternatives:

1) Use the M1015 with an SFF-8087 to SFF-8088 adapter and connect DAS w/ HP SAS Expander (kludge because no space for 8087->8088 bracket in C6100?)
2) Buy an LSI 2008-based 8e adapter, connect DAS with HP SAS Expander. Will the Dell 6Gbps SAS HBA work? Looks good to me. Cheap on ebay right now.

Bypass the midplane and connect directly to the backplane using SFF-8087 to 4x SATA forward breakout cable. I have two .75m cables (I think) and it's a tight fit, but it works, so 1m cables would probably be bettter. You can fish the cables along the side of the node tray (easiest if you pull the tray most of the way out first) and straight through the fan area. If you want to get the other 4 drives connected to the same node, you can use the onboard SATA ports in conjunction with the provided SFF-8087 + SGPIO drive cables that came with the chassis on the midplane (steal one from one of the other nodes). I think it's been discussed somewhere in this thread, but if you have a 4 node c6100, the SFF-8087 + SGPIO cables only have 3x SATA forward breakouts, so you'll need to use SATA ports 1-3 and SATA port 5 on the mainboard / interposer board. Onboard SATA 1-3 are on the right (looking at the front of the server) side of the midplane, and SATA 5 is on the left. I don't know the part number, but you could get one of the DELL SFF-8087 + SGPIO breakout cables that has 4x SATA connectors, or come to think of it, you might even be able to connect a simple SFF-8087 forward breakout cable to Midplane port MB1/2 HDD1~4 and connect the other 4 drives using SATA 1-4 on the interposer board, but you'll likely loose activity lights on those drives (Has anyone tested this?)

I'm also using the M1050, and can verify this works, but again as previously mentioned, you only get activity lights on the 3 or 4 drives connected to the onboard SATA ports using the DELL provided SFF-8087 + SGPIO cable. The yellow SATA cable is connecting SATA port 5 from the mainboard to the interposer board SATA 5.

Port 0 of the M1015 is connected directly to the backplane

• Breakout SATA 1 -> bottom backplane drive column 3
• Breakout SATA 2 -> top backplane drive column 4
• Breakout SATA 3 -> middle backplane drive column 4
• Breakout SATA 4 -> bottom backplane drive column 4

Port 1 of the M1015 is connected directly to the backplane

• Breakout SATA 1 -> middle backplane drive column 2
• Breakout SATA 2 -> bottom backplane drive column 2
• Breakout SATA 3 -> top backplane drive column 3
• Breakout SATA 4 -> middle backplane drive column 3

Midplane MB1/2 HDD 1~4 port connected to backplane

• Breakout SATA 1 -> top backplane drive column 1
• Breakout SATA 2 -> middle backplane drive column 1
• Breakout SATA 3 -> bottom backplane drive column 1
• SGPIO cable connected column 1

Midplane MB1/2 HDD 5~6 port connected to backplane

• Breakout SATA 1 -> top backplane drive column 2
• SGPIO cable connected column 2 (I think this will only light the top drive in column 2 .. haven't tested)

 

[PigLover]

You did that with .75m cables? That would be a tight fit indeed! When I measured it out it seemed like you'd end up needing some tight pulls and hard bends of the cable and connectors to make that fit. I've ordered 1m cables to do something similar but haven't had time to cable it up just yet.

 

[Tsmooth3]

You did that with .75m cables? That would be a tight fit indeed! When I measured it out it seemed like you'd end up needing some tight pulls and hard bends of the cable and connectors to make that fit. I've ordered 1m cables to do something similar but haven't had time to cable it up just yet.

The cable from Port 1 is .75m and has slack to spare. You're right about the other one though. I had a .75m one in port 0 and it is too short. Too much stress on the cable/card so I've ordered a 1m for that port.

 

[briancl]

This is tremendously helpful Tsmooth. Pretty simple and definitely economical. I've ordered the 1m cables and will report back. I'm actually going to attempt to wire 10 drives to Node 1 and 2 drives to Node 2. We'll see what it takes.

 

[Jeggs101]

tsmooth - did you basically put all 12 trays to 1 node using an IBM M1015 and onboard sata?

 

[legen]

This is tremendously helpful Tsmooth. Pretty simple and definitely economical. I've ordered the 1m cables and will report back. I'm actually going to attempt to wire 10 drives to Node 1 and 2 drives to Node 2. We'll see what it takes.

I too have wired my M1015 to 8 of the 3.5'' slots. I used 1m cables and connected them directly to the backplane. I will see if i can provide any picture.

 

[white-hot]

ORICO PHC-35 - HDD/SSD protector (protection against shock & liquid), US$2 each
No more excuse to have your hotswap HDDs lying around unprotected

Please direct me to where these can be found for $2. I see them for $7.99 at lowest and most places don't seem to have very many. The 3.5 only I see for $5.99.

 

[Tsmooth3]

tsmooth - did you basically put all 12 trays to 1 node using an IBM M1015 and onboard sata?

Correct. Current plan is to make node 1 a ZFS storage node, and boot esxi from usb on the other nodes and present iscsi/nfs storage to them.

 

[korban]

Correct. Current plan is to make node 1 a ZFS storage node, and boot esxi from usb on the other nodes and present iscsi/nfs storage to them.

Hi Tsmooth3, I actually have attached 6 drives to one node. If I want to power the 3 first tray I power the 1st node.
But if I wan to power the extra 3 tray I need to power the second node.

Could you explain your backplane power setup with your 12 try. How do you power them in one time ?

 

[Chuck]

For those of you that carefully watch power, just how much does it cost to run one of these with all 4 nodes? Kind of a mid-range dollar per month estimate might be nice. I realize there are varying power rates and I wish I had a utility bill available to tell you how much per Kwh. Thanks in advance for any estimate you can provide. All 4 nodes would have dual L5520's and 48GB (12x4) RAM. A single SSD in each.

I found out our rates are .0988 per Kwh. So using a calculator and some guess work I estimated a high average useage to be about 475W. If I'm correct, would that come out to around $32 a month?

 

[MACscr]

Bypass the midplane and connect directly to the backplane using SFF-8087 to 4x SATA forward breakout cable. I have two .75m cables (I think) and it's a tight fit, but it works, so 1m cables would probably be bettter. You can fish the cables along the side of the node tray (easiest if you pull the tray most of the way out first) and straight through the fan area. If you want to get the other 4 drives connected to the same node, you can use the onboard SATA ports in conjunction with the provided SFF-8087 + SGPIO drive cables that came with the chassis on the midplane (steal one from one of the other nodes). I think it's been discussed somewhere in this thread, but if you have a 4 node c6100, the SFF-8087 + SGPIO cables only have 3x SATA forward breakouts, so you'll need to use SATA ports 1-3 and SATA port 5 on the mainboard / interposer board. Onboard SATA 1-3 are on the right (looking at the front of the server) side of the midplane, and SATA 5 is on the left. I don't know the part number, but you could get one of the DELL SFF-8087 + SGPIO breakout cables that has 4x SATA connectors, or come to think of it, you might even be able to connect a simple SFF-8087 forward breakout cable to Midplane port MB1/2 HDD1~4 and connect the other 4 drives using SATA 1-4 on the interposer board, but you'll likely loose activity lights on those drives (Has anyone tested this?)

I'm also using the M1050, and can verify this works, but again as previously mentioned, you only get activity lights on the 3 or 4 drives connected to the onboard SATA ports using the DELL provided SFF-8087 + SGPIO cable. The yellow SATA cable is connecting SATA port 5 from the mainboard to the interposer board SATA 5.

Port 0 of the M1015 is connected directly to the backplane

• Breakout SATA 1 -> bottom backplane drive column 3
• Breakout SATA 2 -> top backplane drive column 4
• Breakout SATA 3 -> middle backplane drive column 4
• Breakout SATA 4 -> bottom backplane drive column 4

Port 1 of the M1015 is connected directly to the backplane

• Breakout SATA 1 -> middle backplane drive column 2
• Breakout SATA 2 -> bottom backplane drive column 2
• Breakout SATA 3 -> top backplane drive column 3
• Breakout SATA 4 -> middle backplane drive column 3

Midplane MB1/2 HDD 1~4 port connected to backplane

• Breakout SATA 1 -> top backplane drive column 1
• Breakout SATA 2 -> middle backplane drive column 1
• Breakout SATA 3 -> bottom backplane drive column 1
• SGPIO cable connected column 1

Midplane MB1/2 HDD 5~6 port connected to backplane

• Breakout SATA 1 -> top backplane drive column 2
• SGPIO cable connected column 2 (I think this will only light the top drive in column 2 .. haven't tested)

Hmm, so If i was going to do the same, but with 6 onboard sata and the 6 port mezzanine card, I would need a straight through SFF-8087 cable and a SFF-8087 to sata reverse breakout cable to connect to the mezz card?

 

[Tsmooth3]

Hi Tsmooth3, I actually have attached 6 drives to one node. If I want to power the 3 first tray I power the 1st node.
But if I wan to power the extra 3 tray I need to power the second node.

Could you explain your backplane power setup with your 12 try. How do you power them in one time ?

I believe all 12 drives are powered on when one of the nodes is powered on just like all fans are). You won't see any activity lights, but it has been my experience that the drives are on and visible to the OS.

 

[Tsmooth3]

Hmm, so If i was going to do the same, but with 6 onboard sata and the 6 port mezzanine card, I would need a straight through SFF-8087 cable and a SFF-8087 to sata reverse breakout cable to connect to the mezz card?

There might be other ways, but I believe this would work:

• 6 Cable SATA harness to connect onboard SATA ports to interposer board SATA ports (part no. is somewhere in this thread)
• Special Dell SFF-8087 + SGPIO to 4x SATA from midplane MB1/2 HDD1~4 -> backplane Drives 1-4 (part no. might be in this thread)
• Special Dell SFF-8087 + SGPIO to 2x SATA from midplane MB1/2 HDD5~6 -> backplane Drives 5-6 (part no. might be in this thread)
• SFF-8087 forward breakout 4x SATA cable from mezz card -> backplane Drives 7-9 don't connect cable 4 from breakout
• SFF-8087 forward breakout 4x SATA cable from mezz card -> backplane Drives 10-12 don't connect cable 4 from breakout

 

[Patrick]

Pretty cool note. Spoke to Steve today who is in the Dell C6220 video on the first post of this thread.

 

[sloop]

Hi, does anyone have a source for the 3.5" drive sleds, or know of an alternate part that might work? The cheapest I can find for the part # 8TV68 listed on page1 is $30/ea on ebay now.

Any info would be appreciated!

 

[Smalldog]

Hi, does anyone have a source for the 3.5" drive sleds, or know of an alternate part that might work? The cheapest I can find for the part # 8TV68 listed on page1 is $30/ea on ebay now.

Any info would be appreciated!

I ordered from Vista Computer. $15 each or less when I bought them. I dealt with Justin Bush. I think he was mentioned somewhere else around here regarding the C6100's.

Jeff

 

[Smalldog]

Is there a dell part number the cable that would hookup to the other 3 SATA ports that are hidden under the PCI-e slot? Maybe run them up front to use 6 bays internally for a 2 node setup. I know it's not HW raid, but would be fine for JBOD.

I just hooked up regular SATA cables and ran them out the back to 3 4TB drives, and the BIOS posted with all of them including the 3 from the front.

Jeff

 

[dba]

Dell part 6J3R2 is the six-port disk cable bundle. It's a custom-length cable with straight and angled connectors in just the right places, plus it is bundled into a flat cable so that it runs down the side of the motherboard sled nicely.

On a related note, the six motherboard SATA ports are good for only 650-700MB/Second of throughput combined. I found this in my c6100 testing, after which Patrick told me it was old news and pointed to Testing the Intel ICH10R: 660MB/s SATA Limit Fact or FictionServeTheHome ? Server and Workstation Reviews.
Now 700MB/S is plenty if you plan to hook up rotating drives, but throw in an SSD and you have a real bottleneck.

Is there a dell part number the cable that would hookup to the other 3 SATA ports that are hidden under the PCI-e slot? Maybe run them up front to use 6 bays internally for a 2 node setup. I know it's not HW raid, but would be fine for JBOD.

I just hooked up regular SATA cables and ran them out the back to 3 4TB drives, and the BIOS posted with all of them including the 3 from the front.

Jeff

 

[legen]

For you guys still looking for a cheap 2.5'' to 3.5'' adapter I found the SILVERSTONE SDP09 2.5 HOT-SWAPPABLE DRIVE BAY .
Over here in europe it costs ~20$ and does up to 6Gbps .

SilverStone Technology Co., Ltd.- SDP09