Dell c6145 Notes, Tips, and Stories

Patrick

Administrator
Staff member
Dec 21, 2010
12,005
4,990
113
For the record, I am jealous of everyone who got them.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
c6145 Power capping:

I am testing out different CPUs in the c6145. So far:

1) Quad AMD 6128 - runs great. 86GB/S memory bandwidth, even with only half of the DIMM slots filled. 1.22TB/S peak cache bandwidth.
2) Dual AMD 6276 ES version - runs a lower than its rated speed. The 2.3 GHz CPU runs at 1.87GHz when placed under load. I suspect that the BIOS is capping total power consumption, even though I have this feature disabled. Cache bandwidth benchmark is showing lower than expected throughput - just 1.38TB/Second peak
3) Quad AMD 62xx ES version - 12 cores per CPU. Runs at 1.7GHz under load. Lower than expected cache bandwidth, as with the 6276ES.
 
Last edited:

schro

New Member
Apr 9, 2013
9
0
1
Well drat. I got 8 copies of the CPU 3/4 heatsinks so I won't be firing mine up tonight. Emailing Justin instead....
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
My c6145s came with only CPU 1/2 style heatsinks, but Justin later sent me right right number of 3/4s to make it right. He may be able to send you your 1/2s.
Now if he does not have the right heatsinks, let me know. He didn't want the 1/2s back, so I have extras and can swap with you.

Did your c6145 come with plastic airflow baffles? If not then there really isn't any reason to use two different types of heatsinks anyway. Until you get your heatsinks sorted, you can use a driver to remove the little pins that enforce heatsink placement. It takes just a minute or two to remove them - and to put them back in place.

Well drat. I got 8 copies of the CPU 3/4 heatsinks so I won't be firing mine up tonight. Emailing Justin instead....
 

schro

New Member
Apr 9, 2013
9
0
1
He's got the 1/2s shipping out today. That is some fantastic service!

I don't see any baffles inside the box, but then again, I probably wouldn't know them from something else :)



My c6145s came with only CPU 1/2 style heatsinks, but Justin later sent me right right number of 3/4s to make it right. He may be able to send you your 1/2s.
Now if he does not have the right heatsinks, let me know. He didn't want the 1/2s back, so I have extras and can swap with you.

Did your c6145 come with plastic airflow baffles? If not then there really isn't any reason to use two different types of heatsinks anyway. Until you get your heatsinks sorted, you can use a driver to remove the little pins that enforce heatsink placement. It takes just a minute or two to remove them - and to put them back in place.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
24-bay Expander board details and performance:

My c6145s - and probably most of the 24-bay c6145s sold - include a SAS/SATA expander board attached to the disk backplane. I like the concept - twelve drives per motherboard but only one fat SAS connection to deal with. It works as advertised, but so far its performance does not impress.

Physically, the expander is a large circuit board that plugs in to the disk backplane via a PCIe-style connector that appears to be used for power only. Aside from this PCIe connector, the expander board has six SFF-8087 ports for disks and three SFF-8087 ports for the two c6145 motheboards. Oddly, one motherboard gets one SFF-8087 port while the other motherboard gets two, so one system will get four channels while the other will get eight - although the motherboard sled can only handle six. I would have expected a 6+6 arrangement, but the expander electronics probably made this impossible.
The six disk ports on the expander board each connect to four hard drives on the disk backplane via a SFF-8087 to single SAS/SATA breakout cable between the expander and the disk backplane - six such cables in total.

So the entire disk IO path looks like this: Disk drive with its sas/sata connector --> disk backplane board --> backplane to expander cable --> Expander board --> Expander to midplane cable --> Midplane board --> Motherboard sled board --> motherboard disk cable --> SAS Card.

While the "slow" motherboard gets only four 6GB/S SAS channels, that shouldn't be a big problem because we can expect 2GB/S worth of throughput from those four channels. Let's test and see.

I connected 12 Samsung 840 pro 128GB SSD drives to motherboard 2 and formatted them as separate drives. I then ran IOMeter with four workers testing 1MB random reads with a queue depth of 32. What I expected to see is 2GB/Second, limited by the performance of the single SAS x4 connection to the expander. What I actually saw was 1GB/Second - 983MB/S to be more exact. This is suspiciously close to the throughput limit for a SAS 3Gbit x4 connection, so I wonder if the c6145 expander is 3GB/S only or - more likely - if my LSI card is acting up. The card is a new-to-me LSI 9201-8i that is SAS2008-based and looks like a 9211-8i but does not exist on the LSI web site.

I'll probably have more information later, and since I'm seeking throughput, I'll probably end up re-wiring the c6145 to bypass the expander entirely.

Update: I moved the LSI card to another server to confirm that it supports 6Gbit SATA. It does. While there, I updated the firmware and BIOS from P10 to P16. Then, after moving the card back to the c6145, I saw normal 6Gbit SAS throughput - 1,990MB/S reads for a single x4 connection.
 
Last edited:
  • Like
Reactions: gigatexal

PigLover

Moderator
Jan 26, 2011
2,975
1,283
113
Do you have any idea what expander chipset is used? It's possible that the expander only negotiates SATA-II speeds for sata drives but can run at the full link speed with SAS. Other SAS expanders with non-LSI chipsets (like the HPs) have this behavior.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
I don't think it could be the drive links. With twelve drives, even SATA 1 speeds of 125MB/S would yield 1.5GB/S in aggregate - far higher than I'm seeing. It must be the connection between the backplane and the LSI card which I note is nearly exactly 4x250MB/S, the real-world throughput of a SATA2 connection. My next step is to put that LSI 9201-8i into another machine to see if it can do 6Gb/s or is limited for some unfathomable reason to 3Gb/s.

Do you have any idea what expander chipset is used? It's possible that the expander only negotiates SATA-II speeds for sata drives but can run at the full link speed with SAS. Other SAS expanders with non-LSI chipsets (like the HPs) have this behavior.
 
Last edited:

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
I have confirmed that the 9201-8i card is, as expected, a 6Gbit card. So my problem is with the expander and/or the connection between the expander and the card.

I don't think it could be the drive links. With twelve drives, even SATA 1 speeds of 125MB/S would yield 1.5GB/S in aggregate - far higher than I'm seeing. It must be the connection between the backplane and the LSI card which I note is nearly exactly 4x250MB/S, the real-world throughput of a SATA2 connection. My next step is to put that LSI 9201-8i into another machine to see if it can do 6Gb/s or is limited for some unfathomable reason to 3Gb/s.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Update: After updating the firmware and BIOS on the LSI card, I now see proper 6Gbit speeds. Maximum throughput using one x4 SAS connection is 1,990MB/Second. This is lower than the maximum throughput for a normal LSI2008 card when both x4 ports are in use, but is right in line with the expected bandwidth for a single port.

24-bay Expander board details and performance:

My c6145s - and probably most of the 24-bay c6145s sold - include a SAS/SATA expander board attached to the disk backplane. I like the concept - twelve drives per motherboard but only one fat SAS connection to deal with. It works as advertised, but so far its performance does not impress.

Physically, the expander is a large circuit board that plugs in to the disk backplane via a PCIe-style connector that appears to be used for power only. Aside from this PCIe connector, the expander board has six SFF-8087 ports for disks and three SFF-8087 ports for the two c6145 motheboards. Oddly, one motherboard gets one SFF-8087 port while the other motherboard gets two, so one system will get four channels while the other will get eight - although the motherboard sled can only handle six. I would have expected a 6+6 arrangement, but the expander electronics probably made this impossible.
The six disk ports on the expander board each connect to four hard drives on the disk backplane via a SFF-8087 to single SAS/SATA breakout cable between the expander and the disk backplane - six such cables in total.

So the entire disk IO path looks like this: Disk drive with its sas/sata connector --> disk backplane board --> backplane to expander cable --> Expander board --> Expander to midplane cable --> Midplane board --> Motherboard sled board --> motherboard disk cable --> SAS Card.

While the "slow" motherboard gets only four 6GB/S SAS channels, that shouldn't be a big problem because we can expect 2GB/S worth of throughput from those four channels. Let's test and see.

I connected 12 Samsung 840 pro 128GB SSD drives to motherboard 2 and formatted them as separate drives. I then ran IOMeter with four workers testing 1MB random reads with a queue depth of 32. What I expected to see is 2GB/Second, limited by the performance of the single SAS x4 connection to the expander. What I actually saw was 1GB/Second - 983MB/S to be more exact. This is suspiciously close to the throughput limit for a SAS 3Gbit x4 connection, so I wonder if the c6145 expander is 3GB/S only or - more likely - if my LSI card is acting up. The card is a new-to-me LSI 9201-8i that is SAS2008-based and looks like a 9211-8i but does not exist on the LSI web site.

I'll probably have more information later, and since I'm seeking throughput, I'll probably end up re-wiring the c6145 to bypass the expander entirely.
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
I used SIV software to take a peek at the expander: It shows up as a "LSI Bobcat SAS expander". That sounds like a code name, not a model number. I could not find it on the LSI site with a quick search.

Do you have any idea what expander chipset is used? It's possible that the expander only negotiates SATA-II speeds for sata drives but can run at the full link speed with SAS. Other SAS expanders with non-LSI chipsets (like the HPs) have this behavior.
 

schro

New Member
Apr 9, 2013
9
0
1
Looking at performance tuning right now and I'm just scratching my head trying to figure out what isn't right.

I have fired up Folding@Home on this box, and I'm getting frame times around 15.5 minutes on project 8105. These same chips (4x 6172) and memory (16x 1gb) would do about 12 minutes on the same project in the Tyan S8812 motherboard, so performance is nearly 30% lower than expected. I only have 1 node in use at this time.

Current setup involves Ubuntu 12.04 Server LTS as well as the [H]ard|OCP setup script that puts it into a ramdisk for folding.

With no BIOS tweaks, it appeared to be kicking the CPUs into ps2 which corresponds to 1400MHz, so I changed the power control to OS control and set the default to ps0. Started folding back up and it initially was in ps0, but after a bit, dropped back to ps2. Turion Power Control also didn't seem to push it back to ps0. 2 of the CPU's seemed to go up to about 70c with the other 2 around 60c.

This may be from the same issue dba is seeing with his bulldozers...

Still looking into it, but if anyone has any thoughts...
 

Scout255

New Member
Feb 12, 2013
58
0
0
That's pretty disappointing. With only 1 node running it really should not be throttling... Are the fans going full tilt during your testing?
 

schro

New Member
Apr 9, 2013
9
0
1
That's pretty disappointing. With only 1 node running it really should not be throttling... Are the fans going full tilt during your testing?
They're in the unfinished part of the basement (aka, I can't hear them from the office area), but when I poke my head in, they don't sound like they're running full tilt (i.e. as fast as they did after a cold boot), but then again, there's also a Supermicro 2042-G running in the same rack that is a bit whiney.

I will be trying some box fan style cooling with an open lid to see if I can push the temps down significantly to see if it is actually thermal throttling that is causing it. Whatever is happening isn't happening at the OS level... though, thinking back to dba's post, the 6128's that he tried are the only 85w TDP chips in his mix... and my 6172s are 105w TDP... I do have a set of 6166HE's (also 85w) chips that I could theoretically try if I got more ambitious...

In IPMI, it says the fans are running at 6100RPM right now..
 

schro

New Member
Apr 9, 2013
9
0
1
Interesting update on the C6145. I populated the second node with 4x 6128 and 80GB of RAM and powered it up. That forced the fans into a higher speed mode and my cooling issues on the top node seem to have resolved themselves. The top node is now running ~50c on each CPU (down from ~70c on a pair and 60c on the other pair) and at full power (no more throttling).
 

jared

New Member
Aug 22, 2013
38
0
0
Looks like this thread kinda went quiet for a bit.

hey dba

I am curious about the performance issues on this C6145 ... does it still have "sub par" performance for its hardware spec, or did you guys figure something out that brought it to what you were expecting originally?
I am looking around for documentation on the servers EOL to guess when dell plans to sell it to the secondary market.

When these things drop in price like the C6100 did, I would love to get my hands on one...but I was not sure if you are determined its efficiency and cost-effectiveness aren't really worth it?
 

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
Looks like this thread kinda went quiet for a bit.

hey dba

I am curious about the performance issues on this C6145 ... does it still have "sub par" performance for its hardware spec, or did you guys figure something out that brought it to what you were expecting originally?
I am looking around for documentation on the servers EOL to guess when dell plans to sell it to the secondary market.

When these things drop in price like the C6100 did, I would love to get my hands on one...but I was not sure if you are determined its efficiency and cost-effectiveness aren't really worth it?

Hi Jared,

I had two issues with the c6145. Both have been resolved, and they are performing up to their specs:
1) My SAS card was linking 3G instead of 6G - a SAS card firmware update fixed this
2) Some 12-core engineering sample CPUs were throttling - Non-ES CPUs do not throttle.

I have a cluster of two c6145s (four CPU boards) and it's a MONSTER in just 4U of space - 16 CPUs, 128 DIMM slots, 16 SAS controllers, 8 Infiniband ports. I'm running 8-core low power CPUs because the price was right and I don't need more than 128 total cores in this cluster! My only issue with the c6145 is the fact that it doesn't have quad IO controllers per board. Then again, ONLY the HP DL585 G7 has quad IO controllers, so how can I really complain.

Right now there are a pair of single-node C6145s on eBay for $799 each which is excellent for a quad CPU server with 24 disk bays, a 6G SAS mezzanine card, a 6G SAS expander backplane, and dual power supplies. These eBay versions include only two heatsinks, but I have spares selling on eBay right now for $39 each.
There are also some VERY CHEAP ($250) c6145 sleds on eBay, so you might be able to make one of the above into a dual-node machine, though it's a risk since you don't know if the single-node boxes have all of the cables and backplanes needed to run dual node, and the expander backplane will definitely need to be reprogrammed by Dell to split the drives over both nodes**. Even as a single node, however, it's a super cheap thread-monster machine.

**Alternatively: Let's say that you successfully add a second node to a single-node C6145. Since the expander in the single-node version serves up all 24 disks through two SFF-8087 ports, it would be very easy to run one port to each board. Both boards would see all 24 drives, which is awful... unless you are creating a cluster of course, in which case it's perfect. If everything worked - and that's a big if - then you'd have your own shared-SAS "cluster in a box" for peanuts. My c6145s are all dual-node with split expanders so I can't try it.
 
Last edited: