Fun Build aka StorageRipper

clash

New Member
Feb 9, 2021
9
9
3
Dear STH community, let me introduce my Fun Build also known as Storage (and wallet :D) Ripper.

Build’s Name: Fun Build aka StorageRipper
Operating System: Windows 10 LTSC :D
Chassis: Fractal Design 7 XL, seems to be pretty popular choice for TR Pro builds
CPU: ThreadRipper Pro 3975WX -> 5975WX
CPU cooler: Thermalright Silver Arrow TR4 -> Noctua NH-U14S TR4-SP3 with second fan
Motherboard: ASUS Pro WS WRX80E-Sage SE WIFI
RAM: 256GB or 8 x Kingston Server Premier DIMM 32GB, DDR4-3200, CL22-22-22, ECC, KSM32ED8/32ME -> KSM32ED8/32HC, OCed to 3600@CL16
GPU: ASUS TUF Gaming GeForce RTX 3070 V2 OC (0.8k), just to have some moderately-power-consuming GPU
IOPs drives: 3 x Intel Optane P5800x 1.6TB, one for OS and two for data
Main storage and bandwidth drives: 32 x Samsung PM1643a SAS 12G 3.84TB
RAID controller: Microchip (Adaptec) SmartRAID Ultra 3258P-32i, 32 ports, 3258UPC32IXS
Enclosures/backplanes for PM1643a drives: 4 x IcyDock MB508SP-B
Cabling: 4 x SlimSASx8-2MiniSAS-HDx4-0.8M (2304900-R) + a couple of SFF-8643 to U.2 from aliexpress
PSU: Seasonic Prime PX-1300, 1300W
UPS: APC SMT1500IC

Usage Profile: I work as a scientist, and from time to time I deal with 3D geometries and 3D flow fields sizing from ~1GB to few TBs per file. These files are simulation output from supercomputing facilities. I.e., Fun Build is targeted for their processing and storage, as well as for development of simulation software but not for the simulation itself. I/O operations include sequential read and sometimes write with modifications of those files using own C-codes or third-party software. For this task 32 x PM1643a are targeted. In addition, I also deal with 3D computed-tomography scans of real geometries ranging from hundreds to tens of thousands of .tif files per geometry. Here a single file size ranges from ~0.1 to ~5 MB. Development and execution of reconstruction algos require simultaneous reading and writing of those files, for which Optanes were installed.

The Fun Build:
IMG_2149_ps.jpgIMG_2147_ps.jpgIMG_2158_ps.jpg

Some highlights during installation and testing:
  • I tried to overclock “the system”, but there is totally no way. CPU is non-overclockable by AMD while memory overclocking is not supported on this mobo, as asus support replied. I.e., there are frequency settings above 1600MHz (or 3200), but any small step above this frequency results in no boot. At all. I tightened timings a little bit to 18-19-18-52-74-1T@1.3V. And if there is no overclocking, let me do undervolting) I reduced Vcore and cpu SOC voltage via offset by 0.1V and 0.25V, respectively.
  • The fan noise is moderate-to-low, also at full load (Prime95).
  • 120mm fan F1 at ~1200RPM is enough to cool RAID controller (<55-57C under load), north bridge, and Optane (<35C). Room temp is ~24C.
  • GPU is supported by a small stand S1
IMG_2155_stand.jpg
  • Rear fan is off, seems to be unnecessary.
  • Bottom cables are planned to be kept as shown (out of the case), the front case door will not be installed :)
  • For better cooling of PM1643a’s, I attached rubber legs to three IcyDock enclosures to create gaps and add more air flow between enclosures from front 140mm fans. Those 140mm fans cool 32 x PM1643a and 2 x P5800x. PM1643a temperatures in 39-41C range. The default front door of Define 7 XL was removed: its closing increases SSDs temperature by ~15C.
IMG_2109_ps.jpg
  • Due to PWDIS sata “feature”, I need to tape 3rd power pin on all 32 SAS SSDs in order to make them ON inside IcyDock enclosures.
  • Plastic divider in the front frame was removed to enable 140mm fans outside installation:
ps.jpg
  • It seems that there is a mistake in WRX80E-Sage SE WIFI manual, two U.2 ports support PCIe 4.0 opposite to 3.0 stated. IOPS and bandwidth benchmarks for cable-connected Optanes are identical to the ones installed in PCIe slot. If not those two U.2 ports, I do not know how I would connect 2xP5800x. I bought several m.2 -> u.2 adapters (+ SFF-8643 to U.2 cables) from aliexpress expecting possible issues, and, as expected :), two -different- adapters did not work at all with P5800x on my system. Occupying PCIe slots is also not as straightforward: placing already installed single Optane one slot above dropped its bandwidth four times while HWiNFO64 was still reporting PCIe 4.0 connectivity with correct payload size. Additionally, the presence of two more Optanes in PCIe slots would obscure air flow from F1 fan, which is so needed for the raid controller.
u2_pcie3.png
  • “The tower” out of four IcyDocks with 32xPM1643a weighs about 10kg. (Whole build is >30kg). And while installed, it slightly bends plastic plates under it. To prevent this bending, I added some soft, low-memory material under those plates over the edges of metal box with Optanes:
IMG_2098_ps.jpgIMG_2172_ps.jpg
  • Aliexpress cables are shorter than branded-ones (2304900-R), but they resulted in a larger number of non-medium error counts in SSDs smart.
  • Power consumption according to SMT1500IC display: ~120W idle without PM1643a SSDs, ~280W with SSDs idle; +150W under IO load, +220W under Prime95 Small FFTs but without IO load. -> After CPU replacement and memory overclocking add 80W to those numbers.
Questions? Answers? :) I will post performance numbers a bit later.
 
Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,332
1,791
113
CA
Nice build!

Could you add another RAID card for increased performance too? Run 1\2 current drives on each?
 

clash

New Member
Feb 9, 2021
9
9
3
Some benchmarks. Done using fio-3.30-51-gb5f3 under ubuntu 20.04.4 LTS with ext4 file system. Kernel was upgraded to some 5.8.18-050818-lowlatency, otherwise in several tests maximum of 300k IOPs were obtained instead of 2M. Fixed fio flags for all tests:

--ioengine=io_uring --iodepth=32 --filesize=512g --direct=1 --buffered=0 --overwrite=0
--time_based=1 --runtime=10 --registerfiles --fixedbufs --gtod_reduce=1 --group_reporting


RAID partitions ranged between 1TB and 2TB, each RAID level was deployed over 32 SSDs.

perf_excel.png
 
Last edited:

clash

New Member
Feb 9, 2021
9
9
3
Nice build!

Could you add another RAID card for increased performance too? Run 1\2 current drives on each?
In theory, yes but in practice fan F1 needs to be moved somewhere most likely, and probably some other cooling solution for both controllers needs to be implemented. Also, 2nd controller can be installed vertically, next to supercapacitor. Or GPU moved to vertical position, replaced by 2nd controller. So, there are several options.

But 2nd controller is not really necessary: this build was planned from the beginning to saturate one PCIe 4.0 x16 slot (~32GB/s) with 32 SAS-3 1GB/s speedy drives, and this was achieved in practice.
 

clash

New Member
Feb 9, 2021
9
9
3
After few days of testing everything was fine until I started to experience rapid full system shutdowns within the first second of a copy operation or starting IO benchmark. Switching from Win to Linux did not change the situation. It appears that I localized the issue: originally I was powering all four IcyDock enclosures with two peripheral power cables. Under load they can draw 75W/enclosure or 150W per peripheral power cable, which seems to be overkill. After adding two more peripheral cables (now one per enclosure) these shutdowns disappeared.

I have some difficulties while trying to calculate possible max current per one SSD. The only spec I found is here

https://www.servershop-bayern.de/mediafiles/Sonstiges/Samsung PM1643A SAS SSD product brief EN.pdf

There is a total power of up to 12.5W while I observe ~9.5W under sequential write. (But it is unclear how this power is split between 5V and 12V, which is needed to calculate the current). I see lower numbers due to two possibilities: RAID controller may not create enough bandwidth to push drives to their limits, or bench in spec were done in dual-port mode, where the write speed approaches 2GB/s per SSD while I have half of that.

I appreciate any comments on the estimation of PM1643a SSD current draw.
 
Last edited:

Wasmachineman_NL

Dell Precisions FTW!
Aug 7, 2019
1,275
422
83
Holy SHIT that D7 XL is gigantic, I can easily fit a 420mm HWL GTX in the top if I wanted!

Wish I could get one for a not-retarded price though :(
 
Aug 10, 2022
1
0
1
While I don't have any experience with the Samsung SSDs in question I can say in general that SSD power consumption is a function of write bandwidth. The erase cycle takes a lot of power. If there are no writes to the SSD the charge pump doesn't have to build up the power for the next erase and overall power consumption is down 40-70%.

We build that into system designs so customers can get more capacity/rack with throttled writes that keep them under the rack max power.
 

AlexGee

New Member
Aug 3, 2022
6
0
1
Hardcore !!!!

How did you configure the storage and why ~100TB? Performance or space?
 

clash

New Member
Feb 9, 2021
9
9
3
While I don't have any experience with the Samsung SSDs in question I can say in general that SSD power consumption is a function of write bandwidth. The erase cycle takes a lot of power. If there are no writes to the SSD the charge pump doesn't have to build up the power for the next erase and overall power consumption is down 40-70%.
Well, this is totally consistent with my numbers from the first post: ~160W idle with SSD array (~5W per disk), ~310W under IO load (~9.5W per disk, +100% relative to idle or -50% relative to IO load). I use single port mode; under dual-port mode write load will be more intensive, and according to samsung each PM1643a can consume up to 12.5W. In the last case decrease 12.5W -> 5W is -60%.

Hardcore !!!!

How did you configure the storage and why ~100TB? Performance or space?
Very good question :) For all 32 SSDs, I have 37% for RAID0 (main storage) :D, 37% for RAID6 (for syncing with RAID0 every couple of days), and the rest 25% for overprovisioning. And in addition to this I do backups to HDDs from time to time.
 

clash

New Member
Feb 9, 2021
9
9
3
A few weeks ago I have added some modifications to the original system (see also the first post updated with green).
  • Original fan F1 cooling the raid controller was older Noctua model NF-P12. I replaced it with not quite standard NF-S12A PWM. This modification of their mainstream 120mm model (NF-A12x25 PWM) has slightly less static pressure, but more air flow and less noise. This combination is perfect for my scenario, where there are no obstacles on the way of air flow. As a result, slightly less noise while controller temperature dropped about 3-4 degrees.
IMG_8638_fan copy.jpg
  • CPU was replaced with 5975WX, as well as CPU fan with Noctua NH-U14S TR4-SP3. Significantly more silent compared to Silver Arrow TR4.
  • Now it was possible to overclock memory, what I actually did. Because of new CPU (Zen3 arch) and overclocking, I got whopping +20-50% performance boost. In some scenarios the increase was +100%, such as maximum achievable IOPS under windows going from ~700k to ~1.3M or AIDA64 memory tests.
  • Memory with Micron E-Die was replaced with Hynix C-Die providing slightly better overclocking result: 3600@1.42V, 16-8-19-19-34-42-1T (CL-tRCDWR-tRCDRD-tRP-tRAS-tRC-CR). Here are the additional timings I found making the performance difference: tRRDS/tRRDL/tFAW (4/4/16 in my case) and tRFC (480 cycles).
  • Because of the memory overclocking (and these are ECC modules, there are many of them located nearby), it started to overheat. After ~10–15mins of OCCT stress test, memory was steadily reaching ~75C and OS was crashing. I.e., most likely the temperature would continue to grow further. I implemented the following memory cooling solution based on the six Noctua NF-A4x10, where standard Noctua’s NA-AV3 anti-vibration mounts were cut while fans were glued together with a drop of superglue:
prep_IMG_8605_single_fan copy.jpg prep_IMG_8597_fans_legs copy.jpg
  • Glued together fans were simply placed on the memory modules. Here are the fans and CPU fan clearance:
IMG_8604_top_close_look copy.jpg IMG_8621_bottom_close_look copy.jpg IMG_8635_fans_clearance copy.jpg IMG_8624_fans_with_cpu_fan copy.jpg IMG_8628_fans_installed copy.jpg
  • Now OCCT stress test passed without any problems for ~1h or longer, with the maximum temperature reaching about 62C for the hottest module. While checking temperatures of individual RAM modules, I found 10C discrepancy between top and bottom quad of modules. I realised that RAM fans of the top quad intake hot air from the CPU fan. To prevent this, I installed just a piece of hard paper as shown below. As a result, the difference between modules dropped to ~5C, with the maximum about 57C under load :)
yellow_IMG_8644 copy.jpg yellow_IMG_8646 copy.jpg
 

EasyRhino

Active Member
Aug 6, 2019
254
151
43
so i'm curious with this build, is transfer of data on or off this platform a big chore? if so how do you do it?
 

clash

New Member
Feb 9, 2021
9
9
3
so i'm curious with this build, is transfer of data on or off this platform a big chore? if so how do you do it?
The transfers are very unsystematic, sometimes a week of intensive downloads, then several months of working with only locally available data. It is research, so one does not know at the beginning what (and if) will be found and what to do next. The connection is just a gigabit network, 110MB/s, ~0.4TB/h.