Suggestions for a ~40 TB SSD server?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Schut

New Member
Apr 13, 2020
11
2
3
Background:

I have been building TrueNAS servers from Dell R730xd's for a while now, with mostly good luck. But my main home server has been having timeout issues with drives that I can't remedy. The drives are CMR not SMR, I've triple checked, and I've replaced I think 4 drives now, but the timeouts persist. I've despaired of trying to figure out the issue, if it is drives or the server, but everything is backed up multiple times so I've been limping along.

But the timeouts are starting not just to be a minor annoyance but are now making accessing my files extremely difficult so I'm inclined to do something a bit excessive, and move to SSD for some of my key files.

Requirements:

The server I'm wanting to put together would be:

* 40 TB of usable SSD storage after 2 or 3 disks of parity.
* At least 10 gb ethernet, maybe 40gb ethernet (assume I'll supply 40gbe card).
* TrueNAS (not negotiable right now).
* I would expect to rack mount it, but will make other arrangements for something special.
* At least 128 gb RAM, more would be nice.
* Hotswap for drives strongly desired (but exceptions can be made for a system that excels in everything else).
* Used hardware is totally fine, new works if it is in budget.
* Drives used should have a healthy used market, or buying extra drives at construction is possible if drive cost is not prohibitive
* Usage will be relatively light. No massive database servers or constant VM use planned, mostly just one or two people accessing files.
* $5k - $10k budget. Could go higher for the right system but that's where I'd like to land.

I have another used R730xd still in its box, but with no drives. If there are appropriate SSDs that will plunk right into this, I'm interested, but I'll buy a new server if I have to.

Sorry for the broadness of this question, I"m just not up to speed on server SSDs. Any and all advice appreciated. I have been trying to do my own research on this but I'm still feeling lost, with no clear idea what to do.
 

ca3y6

Active Member
Apr 3, 2021
270
169
43
Not sure what you mean by "timeout"? Have you tried reinstalling the OS perhaps? Or replacing ethernet cables? Is there a device on the same network with the same IP than the NAS? What do the SMART tests and stats of the drives show?

Sounds to me that you have a technical problem and going full flash may be an overkill and will be much more expensive than large modern HDD even with used SSDs. And in term of performance you can always add a SSD cache in front of your hard drives.

But if you do want to go the SSD route, why not, SSD are fun! There is no point to buy expensive hardware if all you need is to run a NAS. Most of your budget will be buying the SSD. Most cost optimal will be used enterprise SSD. SATA if you want to keep power consumption down. SAS if you are less sensitive about power consumption, you will likely find better deals on ebay (you will need a SAS controller either way). U.2 will be more expensive to scale and you need a good server motherboard with lots of lanes.

Sweet spot in term of used SSD pricing is 3.84TB. You can find SATA from $160, sometimes less if you are lucky and patient:



7.68TB is the maximum size for SATA, they usually go for a premium to two 3.84TB so might not be cost optimal (that's the general logic for all formats, the larger the drive, the more expensive per TB). I think you will be lucky to find any under $400.


SAS drives have similar price ranges, but good deals are more frequent:




Technically the lowest cost per TB will be AIC/HHHL cards. But that's because they don't scale. Even with a server motherboard you won't reach 40TB with those.


 

i386

Well-Known Member
Mar 18, 2016
4,564
1,719
113
35
Germany
Before I would even think about new hardware I would try to find the root cause for the problem(s).

What configuration do you have? 12x disks in raidz1/z2/z3?
What controller? Is it overheating?
What disks?
Is the volume filled up to >80%?
Is the chassis modded in any way?
Does the timeout happen always or just in certain scenarios?
 
  • Like
Reactions: TRACKER

TRACKER

Active Member
Jan 14, 2019
277
116
43
Before I would even think about new hardware I would try to find the root cause for the problem(s).

What configuration do you have? 12x disks in raidz1/z2/z3?
What controller? Is it overheating?
What disks?
Is the volume filled up to >80%?
Is the chassis modded in any way?
Does the timeout happen always or just in certain scenarios?
I would add - check if power supply is sufficiently powerful, especially how many HDDs you have connected on each power line.