Just a few weeks back I was asked to put together a large amount of storage for a last minute project. I usually avoid projects like this, but it's a special case and thus far it has gone as planned (knock on wood). Since I am waiting on a few external delays that are outside my control, I figured there's no better time then now to do some performance tuning. I'm working on a quick set of repeatable tests that best represent our usage. Up till now, I have relied on bonnie/iostat/dd bench/crystaldiskmark. I've read pretty much everything I can find and wanted to get an outside opinion.
Here are the three systems spec's:
Processing
(1)Supermicro 2u 24 bay Chassis
(2) Xeon E5-2620 v2's
(16) 32GB DDR3 1866
(3) Supermicro 3008 based internal HBA's flashed with IT mode firmware
(2) LSI 9300-8E HBA's
(1) Mellanox ConnectX3 Dual Port QDR IB
(22) 1TB Samsung 850 Pro's
(4) 256GB Samsung 850 Pro's
(2) Supermicro 45 Bay JBOD's (Single expander)
(90) WD RE4 4tb 7200rpm Enterprise SATA
Rpool:
(2) 256GB Mirrored
SSD Pool: (9.4TB Formatted)
(10) Mirrored 1TB vdev's
(1) 256GB ZIL Drive
(1) 1TB Spare
Spindle Pool: 157TB
(44) Mirrored 4TB vdev's
(2) 4TB Spares
(1) 256GB ZIL
(1) 1TB L2ARC
90% limited
Archival (2x)
(1)Supermicro 2u 24 bay Chassis
(2) Xeon E5-2620 v2's
(16) 32GB DDR3 1866
(1) Supermicro 3008 based internal HBA's flashed with IT mode firmware
(4) LSI 9300-8E HBA's
(1) Mellanox ConnectX3 Dual Port QDR IB
(2) 1TB Samsung 850 Pro's
(4) 256GB Samsung 850 Pro's
(4) Supermicro 45 Bay JBOD's (Single expander)
(180) WD RE4 4tb 7200rpm Enterprise SATA
Rpool:
(2) 256GB Mirrored
Spindle Pool: 475TB Formatted
(22) 8x4TB Raid Z2 vdev's
(4) 4TB Spares
(2) 256GB Mirrored ZIL
(2) 1TB L2ARC
90% Limited
OS: Solaris 11.2 (or possibly OmniOS, I'm going to play with it tomorrow.)
There are a few important notes: (1) We were limited by what we could order due to the time frame. This caused major issues- the Intel SSD's and SAS hard drives I originally specced ended up be weeks out from our deadline. The WD RE4's and Samsung 850's were the only drives available. (2) The original project requirement was 1PB of storage. After speaking with the teams using this storage and doing a quick analysis, I decided to split it in to the three systems above. Mainly because of how they move data as it is processed. The first archival machine is really a input data server, where everything from the field is uploaded and organized. The processing data server is where larger groupings are copied, broken in to smaller chunks, and processed off the SSD array by an attached 240 core/1.5TB cluster. The last machine is an output directory, where everything is QA'd and copied off for delivery. It's a complicated process with several data moves, but if you could see how they are doing it now- it's 100x better.
I'm building the arrays as I write this, let me know what you think or what benchmarks you would like to see.
Here are the three systems spec's:
Processing
(1)Supermicro 2u 24 bay Chassis
(2) Xeon E5-2620 v2's
(16) 32GB DDR3 1866
(3) Supermicro 3008 based internal HBA's flashed with IT mode firmware
(2) LSI 9300-8E HBA's
(1) Mellanox ConnectX3 Dual Port QDR IB
(22) 1TB Samsung 850 Pro's
(4) 256GB Samsung 850 Pro's
(2) Supermicro 45 Bay JBOD's (Single expander)
(90) WD RE4 4tb 7200rpm Enterprise SATA
Rpool:
(2) 256GB Mirrored
SSD Pool: (9.4TB Formatted)
(10) Mirrored 1TB vdev's
(1) 256GB ZIL Drive
(1) 1TB Spare
Spindle Pool: 157TB
(44) Mirrored 4TB vdev's
(2) 4TB Spares
(1) 256GB ZIL
(1) 1TB L2ARC
90% limited
Archival (2x)
(1)Supermicro 2u 24 bay Chassis
(2) Xeon E5-2620 v2's
(16) 32GB DDR3 1866
(1) Supermicro 3008 based internal HBA's flashed with IT mode firmware
(4) LSI 9300-8E HBA's
(1) Mellanox ConnectX3 Dual Port QDR IB
(2) 1TB Samsung 850 Pro's
(4) 256GB Samsung 850 Pro's
(4) Supermicro 45 Bay JBOD's (Single expander)
(180) WD RE4 4tb 7200rpm Enterprise SATA
Rpool:
(2) 256GB Mirrored
Spindle Pool: 475TB Formatted
(22) 8x4TB Raid Z2 vdev's
(4) 4TB Spares
(2) 256GB Mirrored ZIL
(2) 1TB L2ARC
90% Limited
OS: Solaris 11.2 (or possibly OmniOS, I'm going to play with it tomorrow.)
There are a few important notes: (1) We were limited by what we could order due to the time frame. This caused major issues- the Intel SSD's and SAS hard drives I originally specced ended up be weeks out from our deadline. The WD RE4's and Samsung 850's were the only drives available. (2) The original project requirement was 1PB of storage. After speaking with the teams using this storage and doing a quick analysis, I decided to split it in to the three systems above. Mainly because of how they move data as it is processed. The first archival machine is really a input data server, where everything from the field is uploaded and organized. The processing data server is where larger groupings are copied, broken in to smaller chunks, and processed off the SSD array by an attached 240 core/1.5TB cluster. The last machine is an output directory, where everything is QA'd and copied off for delivery. It's a complicated process with several data moves, but if you could see how they are doing it now- it's 100x better.
I'm building the arrays as I write this, let me know what you think or what benchmarks you would like to see.
Last edited: