Intro & Built notes

[dba]

Cactus,

I have a SC216 chassis with the 900W power supply (PWS-902-1R) that puts out only 4A onto the 5V bus according to the sticker on its back. None the less, it's rated for 24 2.5" drives - and I can vouch for the fact that those can be 24 SSD drives. My conclusion is that it's the backplane that's doing the magic. It's fed by Molex connectors from the power supply, and I imagine that it's converting some 12V power to 5V power.

You are going to have a hard time finding a single PSU to supply that much 5V unless it is a dedicated 5V psu. Even looking at Supermicro's SC216 2.5 chassis, their 1200W 1+1 PSU only gives you 250W(50A) on the 5V rail. Consumer SSDs use 5V is to allow them to work with laptops which dont always provide 12V for HDD/SSD. If you look at enterprise SSDs like the STEC ZeusRam or the Seagate Pulsar, they use the 12V rail along with the 5V rail.

12V has been heavily used in higher power computer components like video cards and the CPU because you have less resistive loss. P = R*I^2 = IV IIRC Intel CPUs are powered exclusively off the 8pin 12V connector.

ehorn, the 4pin molex connector gives you 5V and 12V. The SATA/SAS power connector is spec'd to supply 3v3, 5V, and 12V with Micro SATA using only 3v3 and 5V.

 

[dba]

So you ran an IOMeter test with a single Samsung 830 drive on the LSI 9207 card and saw 440MB/S while the same drive attached to the Intel controller on the motherboard rated 510MB/S? That seems very odd. I see 540MB/S sequential reads for Vertex3 drives on an LSI 9200 card which is "full speed" for that drive. What IOMeter settings are you using?

There is one glitch with the LSI 9207-8i (in both modes IT and IR). My Samsungs loose about 20% of their peak read performance. Even if the SSD is accessed as a single drive. Read speed of the SS128GB goes down from 510MB/s to 440 MB/s. Hadn't have time to look into this deeper, but will do so. Any idea?

 

[Patrick]

dba... I think you may have two more LSI controllers to try by EOD.

Very odd... will build a test rig this weekend to see what I can find.

 

[cactus]

dba, look at the PDB specs. The PSU modules only put out 12V and 5Vsb, the latter being used for control logic that lets you turn the PSU on. The PDB then uses a DC to DC converter, like many current single rail consumer PSUs, to make 3V3 and 5V. The PDB in the SC216 is rated for 50A on the 5V rail and 30A on the 3V3 rail. The backplane supplies both 12V and 5V to the drives, but I doubt it does any DC-DC conversion.

Andreas, for 2.5", there is this. The SC216 seems like a good option, but is expensive. You could use an inexpensive PC case as your JBOD using the 4-in-1 adapters you have already. Then the SFF-8088 to SFF-8087 adapters as Patrick mentioned. You can use a JBOD control board(SM one for the SC216) or simply wire a non momentary SPST switch between ground and the green wire on the 24pin mobo connector to turn the drives on. Turn the JBOD on first then boot your computer and the drives will all be detected. I have done this with SATA DVD drives and when debugging PSU problems in my Chenbro RM31616. If you can do some metal work, the Norco RPC-4164 with a new back window to allow you to have 2+ ATX PSUs may also be another option.

You might run into a problem with cable length with SATA drives. Hopefully one of the others can chime in with their experiences.

 

[dba]

Great pointer on the power distribution board - power supplies obviously aren't my specialty!

If you do build a JBOD, Andreas, take a look at the Supermicro JBOD board that cactus describes. It's cheap and simple - your power supply connector plugs in to one end and Molex connectors come out the back, along with two or eight (V1 and V2 of the board respectively) fan connectors and wiring for an on/off switch. For around $35 you can convert any chassis to a JBOD. You even get a fan-fail alarm - though even 24 SSDs in one chassis don't generate much heat.

dba, look at the PDB specs. The PSU modules only put out 12V and 5Vsb, the latter being used for control logic that lets you turn the PSU on. The PDB then uses a DC to DC converter, like many current single rail consumer PSUs, to make 3V3 and 5V. The PDB in the SC216 is rated for 50A on the 5V rail and 30A on the 3V3 rail. The backplane supplies both 12V and 5V to the drives, but I doubt it does any DC-DC conversion.

Andreas, for 2.5", there is this. The SC216 seems like a good option, but is expensive. You could use an inexpensive PC case as your JBOD using the 4-in-1 adapters you have already. Then the SFF-8088 to SFF-8087 adapters as Patrick mentioned. You can use a JBOD control board(SM one for the SC216) or simply wire a non momentary SPST switch between ground and the green wire on the 24pin mobo connector to turn the drives on. Turn the JBOD on first then boot your computer and the drives will all be detected. I have done this with SATA DVD drives and when debugging PSU problems in my Chenbro RM31616. If you can do some metal work, the Norco RPC-4164 with a new back window to allow you to have 2+ ATX PSUs may also be another option.

You might run into a problem with cable length with SATA drives. Hopefully one of the others can chime in with their experiences.

 

[Andreas]

So you ran an IOMeter test with a single Samsung 830 drive on the LSI 9207 card and saw 440MB/S while the same drive attached to the Intel controller on the motherboard rated 510MB/S? That seems very odd. I see 540MB/S sequential reads for Vertex3 drives on an LSI 9200 card which is "full speed" for that drive. What IOMeter settings are you using?

I think I found the culprit.
It looks like the CrystalMark Software can't drive the LSI driver with the same efficiency as the X79 port. Using Io Meter with identical settings (1 MB, 16QD, 5GB file) both versions provide more or less equal performance.

All data with the same physical Samsung SSD drive.

CD Mark, drive on X79 port

CD Mark, drive on LSI 9207-8i (IR firmware)

IO Meter, drive on X79 port

IO Meter, drive on LSI port

So much for the single drive "issue".

Still need to look where I loose about 2.5 GB/sec over the whole set.

In a perfect world 32 x 520 MB/sec would get me approx 16.5 GB/sec. While the real perf as measured with IO meter of 14 GB/sec isn't bad at all, I'd like to get a better understanding where the loss takes place. Hmmm, will check that after vacation.

Potential reasons (just to name a few possibilities):

• The SAS 2308 controller (issue rate of commands to 8 drives, sweet spot with certain transfer sizes not yet matched, ...)
• The drives (non syncronized background processing like garbage collection in one drive might impact performance of the whole array) similar to the gang scheduling problem with supercomputers.
• scaling limitations of the disk driver in the OS (or settings need to be tuned)
• scaling limitations of the LSI device driver (or settings need to be tuned)
• PCIe3 port conflicts or saturation in the CPU
• memory controller limitation of DMA traffic from IO
• Limitation of the measurement software
• AOI - Any other issue

Keep to be interesting.

rgds,
Andy

 

[Andreas]

Patrick, cactus, dba,
thanks for the info on the power stuff.
Will take a closer look after vacation for a proper solution. The most compelling route for now seems to be a second case with its independent PSU for a part of the SSDs. My system doesn't need to be a production quality system, some tinkering is always possible ....

dba,
what was approx. the power consumption of the server you described above on idle?

rgds,
Andy

 

[Andreas]

A small correction:

The second CDM numbers in the post above show transfer rates for a 1 GB file, while the numbers in the first picture (on the X79 port) show the transfer rates for a 4GB file .

My mistake, I linked in the wrong image. Here is the correct one with 4 GB on the LSI port. Basically no difference, but I wanted to show numbers with identical settings.

rgds,
Andy

 

[dba]

My server won't win any power consumption contests. It's a SuperMicro 2U chassis with two 1200W power supplies plus two SC216 chassis each with two 900W power supplies. As if that isn't enough, I've got seven LSI controllers, four 12-core CPUs, 32 RAM DIMMS, and of course a total of 48 drives, mostly SSD drives. Because I'm looking for speed, all power saving is disabled - and it makes a BIG difference to database performance.
Idle power consumption is around 340 watts as measured at the wall plug. It easily hits over 750 watts when running near 100% CPU, which it often does. Luckily for my power bill, this is a development system that I can power down at night or when otherwise not needed.

One thought: Have you graphed total throughput as you add drives? I can imagine that you are nearing the throughput limit for a single CPU, regardless of how many drives and cards you throw at it. A graph of throughput for 32,30,28,26,24 drives might be a good test. Do the same for a single card (test with 6,7,8 drives) so that you understand your single-card scalability first.

Patrick, cactus, dba,
thanks for the info on the power stuff.
Will take a closer look after vacation for a proper solution. The most compelling route for now seems to be a second case with its independent PSU for a part of the SSDs. My system doesn't need to be a production quality system, some tinkering is always possible ....

dba,
what was approx. the power consumption of the server you described above on idle?

rgds,
Andy

 

[dba]

Good old IOMeter is my go-to program for very high throughput testing.

Also, have you tried six IOMeter workers instead of 12? My "rule of thumb" is to start with one worker per physical core when testing large arrays and then work up and down from there until you find maximum throughput. I did a small amount of testing on a dual-CPU Intel LGA2011 machine with 28 drives and found maximum sequential read throughput at eight workers.

I think I found the culprit.
It looks like the CrystalMark Software can't drive the LSI driver with the same efficiency as the X79 port. Using Io Meter with identical settings (1 MB, 16QD, 5GB file) both versions provide more or less equal performance.

...

rgds,
Andy

 

[Andreas]

Some comments about the different SSDs, based on tests with individual drives:

Read performance is more or less the same accross the board.
Approx. 500MB/s seem to be the current bar.

Huge variations on write performance, especially considering the time when the write happens.
I.e. writing a SSD sequentially for about 50% of its capacity works well for almost all SSDs - depending on usage state (secure erase, in GC mode, etc ..). The big difference comes when the second half of the drive is immediately written as well.
Only the Samsung drives exhibited predictable performance in this scenario. Maintaining their write speed all the way to the end. While the write speed in the first half might be a bit slower than the others (OCZ, SanDisk), no drive could match the Samsung in the second half. The Vertex 4 and Agility 4 are especially impacted. Their good write performance in the first half goes down to below 100 MB/s in the second half and does not recover during the write.

Idle power consumption:
Samsung drives (both 128 and 256GB) ca. 90mA
SandForce drives (OCZ Vertex 3, all SanDisk) ca. 100-120 mA
OCZ generation 4 drives (Agility, Vertex) ca. ca. 240-260mA

Write power:
Samsung: ca. 900mA
SanDisk Extreme 480GB: ca. 800mA
SanDisk Extreme 120GB: ca. 600mA
Vertex 3 with its SF controller has similar differences: 120GB:570mA, 240GB:750mA
Interestingly, Agility 4 drives need more power than Vertex 4: +50mA at 128GB and +100mA at 256GB

Read power consumption:
More or less all controllers are in the 450-550mA ballpark

Lesson learned:
The Samsung with the highest peak power consumption during short write tests uses the least amount of energy of all drives when writing the full SSD - thanks to it's high average write speed in the second half no other SSD can sustain. Together with it's low idle power, the energy differences betwen the drives need to be considered for the intended usage and not only on peak consumption.

There was one anomaly with my OCZ Vertex drives, both versions 128 and 256GB. While write power consumption was among the lowest observed, both drives exhibited - after the write concluded - a power consumption which was higher than during the write process itself. For a duration of about 1 minute. For the 128GB Vertex 4, the 570mA write value increased to 670mA and for the 256GB version the 590mA write value jumped to 860mA. The Agility 4 with the same Marvell controller did not show this behavior.

Initially my intention was to equip my system with a set of Vertex4 drives, but the uneven write performance made me switch to the Samsung drives. Again, this decision was based on my usage pattern, ymmv on your needs.

rgds,
Andy

 

[cactus]

How are you getting your current measurements?

 

[john4200]

Write power:
Samsung: ca. 900mA
SanDisk Extreme 480GB: ca. 800mA
SanDisk Extreme 120GB: ca. 600mA
Vertex 3 with its SF controller has similar differences: 120GB:570mA, 240GB:750mA
Interestingly, Agility 4 drives need more power than Vertex 4: +50mA at 128GB and +100mA at 256GB

What is the write current for the Vertex 4?

 

[MiniKnight]

Theory goes the vertex 4 is writing as SLC giving you about half capacity of really fast writes, then does active or idle gc to re-write data to a MLC pattern. After that the vertex 4 writes again and get full SLC like speed

 

[Andreas]

How are you getting your current measurements?

By using a SATA data/power breakout cable (about 8inch long). Cut the lines and put a power meter on each supply line (5V and 12V). As said, 12V is not used by consumer SSDs.

What is the write current for the Vertex 4?

Agility4/128GB=640mA
Agility4/256GB=680mA
Vertex4/128GB=590mA
Vertex4/256GB=590mA

Theory goes the vertex 4 is writing as SLC giving you about half capacity of really fast writes, then does active or idle gc to re-write data to a MLC pattern. After that the vertex 4 writes again and get full SLC like speed

OCZ calls it the performance mode and storage mode. This recovery take time though. In my write pattern (writing 50% of capacity first, followed by another 50% immediately therafter) the drive doesn't have the time to do so and slows down in the second write.
If the drive has more time to do its GC, it would offer again the faster write times for about 50% of the remaining capacity. I guess, in typical desktop scenarios a good approach, but didn't work out well in my write pattern.

The Sandforce based SSDs (SandDisk, Vertex3) didn't do well on the incompressible datasets. A known attribute of the SF controller.

At the end, I didn't want to deal with all the SSD internal variables of the different models impacting the performance in such a significant way, depending on state of GC, secure erase, fill%, compressible or not, etc.. and went for this reason with the Samsungs. While the 256GB has better absolute write speed (410MB/s) than the 128GB model (320 MB/s) it boiled down to 2 final decision points for me:

1) price
With prices per GB approx the same, the same total amount of money to spend on drives got me approx. 55% more write bandwidth and 100% more read bandwidth with the 128GB SSDs.
read: 32 x 500 GB/s vs. 16 x 500GB/s
write: 32 x 320 GB/s vs. 16 x 410 GB/s

2) the better I/O time versus capacity

Samsung 128GB read full SSD sequentially: ca. 250sec
Samsung 256GB read full SSD sequentially: ca. 500sec

Samsung 128GB write full SSD sequentially: ca. 400sec
Samsung 256GB write full SSD sequentially: ca. 620sec

To put things into perspective: A contemporary 3 TB hard disk might take 20000 seconds and more for this exercise.

rgds,
Andy

 

[Andreas]

A comment on memory bandwidth:
Using the stream benchmark the i7-3930K delivers with 1333MHz RAM approx 30 GB/sec main memory bandwidth on an otherwise idle system.

Using IOmeter with the previous settings to load the IO system with ca. 14 GB/s had the following (non-scientific) results. Stream only runs for a couple of seconds.

IOmeter went down to ca. 12 GB/sec and stream reported ca. 18 GB/sec back. Nice, as it indicates that the PCIe data transfer still gets it fair share of Memory bandwidth and the LGA2011 CPU still has more bandwidth available to do some processing than an idle LGA1155 system (ca. 16 GB/sec with 1333 MHz RAM).

Even if LSI would introduce 16port SAS 6G adapters with PCIe x8 Interfaces to increase the IO bandwidth per card, a fully loaded system with 4 controllers might be able accomodate the IO bandwidth with pure read performance. But it would not leave enough bandwidth to the CPU for other jobs.

Most CPU reviews focus on compute speed and the LGA2011 CPUs sometimes have a hard time to show their additional value versus their cheaper and more energy efficient LGA1155 brethrens. For an IO heavy environment the distinction between LGA2011 and LGA1155 becomes much more evident if this early measurements are a proper indication.

rgds,
Andy

 

[ehorn]

ehorn, the 4pin molex connector gives you 5V and 12V. The SATA/SAS power connector is spec'd to supply 3v3, 5V, and 12V with Micro SATA using only 3v3 and 5V.

Hey cactus,

Right you are... My assumption was similar to dba's... that the "magic" was happening at the backplane. (i.e. the backplane was stepping down the voltage). Whether this is fact I do not know without research.

It appears that the PDC's found in server PSU's provide this "transformer" mechanism.

I wonder if someone makes a tidy PDC for standard ATX PSU's? Something which would perform this transformer duty of stepping down the substantial amounts of power available on the 12V rail(s) to 5V?

Either way, I am sure Andreas will find a solution to meet his need.

Very cool build Andreas... A great project... Looking forward to more insights. Have a great vacation and thanks again for sharing.

peace,

 

[Patrick]

Andy which version of the stream benchmark did you use? I know that one is a bit sensitive.

Really cool results! dba has been seeing something very similar and just dropped by yesterday for some testing supplies.

 

[Andreas]

Andy which version of the stream benchmark did you use? I know that one is a bit sensitive.

Really cool results! dba has been seeing something very similar and just dropped by yesterday for some testing supplies.

Stream 5.8
Settings: 8 threads, every core, 64bit

 

[MiniKnight]

Stream 5.8
Settings: 8 threads, every core, 64bit

Have a link to the version you used? I want to try doing this myself.