I would look into it further, but in monitoring my temps I simply have not had any temperature issues.
The two m.2 drives on the board stay at decent temps even under load, as does everything in the 16x risers. (The entire aluminum case on those Asus 16x risers is a heat spreader though, and they also have a built in fan that pulls it in and exhausts it out the PCI slot in the back.)
I was worried about the clearance between them at first, but in my testing here was only a degree or two difference resultant from separating them, and there were other things on the board in the way if I used the other 16x slots, so they stayed where they are. Apparently the limited couple of millimeter clearance provides sufficient airflow to stay on top of it.
If I had NVME temperature issues I'd do something about it, but I simply don't. All the temps are fine, even under load, even at my current elevated ambients due to no AC.
Of the 16 NVME devices only six of them are Gen4 though. The rest are Gen3. If I had Gen5 or more Gen4 devices I may have more of an issue, but I simply don't need Gen5 bandwidth (and the CPU and Board doesn't support it anyway)
Just to clarify and provide more data if anyone is interested, this is what my NVMe drives in my server are used for, what locations they are in, and what temps and wear that results in.
But first, in reading the below, understanding the ZFS storage configuration may help. It looks like this:
Code:
Boot Pool
data
mirror-0
Samsung 980 Pro 500GB
Samsung 980 Pro 500GB
logs
mirror-1
Intel Optane 900p 280GB Drive 0 Partition 0
Intel Optane 900p 280GB Drive 1 Partition 0
VM Datastore
data
mirror-0
Samsung 980 Pro 1TB
Samsung 980 Pro 1TB
logs
mirror-1
Intel Optane 900p 280GB Drive 0 Partition 1
Intel Optane 900p 280GB Drive 1 Partition 1
Scheduled Recordings
data
mirror-0
Inland Premium 1TB
Inland Premium 1TB
Mass File / Media Storage
data
raidz2-0
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
raidz2-1
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
Seagate Exos x18 16TB
special
mirror-0
Inland Premium 2TB
Inland Premium 2TB
Inland Premium 2TB
logs
mirror-1
Intel Optane 900p 280GB Drive 0 Partition 2
Intel Optane 900p 280GB Drive 1 Partition 2
cache
WD Black SN850x 4TB
WD Black SN850x 4TB
#1 Boot drives 2x ZFS mirrored 500GB Samsung 980 Pro
Temperature Sensor 1: 39 Celsius
Temperature Sensor 2: 43 Celsius
Temperature Sensor 1: 40 Celsius
Temperature Sensor 2: 43 Celsius
These are Gen4, but mostly idle most of the time. They are the ones directly on the motherboard, partially buried under all the junk with the shitty chinese heatsinks on them and a 40mm slim Noctua fan for good measure. 0% wear reported, but after OS install they get only limited writes, so that is expected.
#2 VM & Container Datastore drives. 2x ZFS mirrored 1TB Samsung 980 Pro
Temperature Sensor 1: 41 Celsius
Temperature Sensor 2: 52 Celsius
Temperature Sensor 1: 40 Celsius
Temperature Sensor 2: 47 Celsius
These are also Gen4, but get a little more exercise as all of the VM's and containers are running off of them, including a couple with some database activity. They are inside the 16x risers, one leftmost, other rightmost. Of all the storage pools in the system, the wear is typically the fastest on this one (or at least used to be) but since installing the Samsung drives here ~7 months ago, they still report 0% wear.
#3 Dedicated pool for scheduled MythTV PVR recordings. 2x ZFS mirrored 1TB Inland Premium (appear to be reference Phison drives manufactured by Team)
Temperature: 29 Celsius
Temperature: 30 Celsius
These are Gen3. Mostly idle unless recording scheduled shows. I used to record straight to the hard drive pool, but got some strange stutter, so decided to record to a SATA SSD pool instead, and transitioned it to NVMe when I replaced all of the SATA SSD's with NVMe drives. Cron script moves the oldest recordings to my hard drive pool every night at 4am to free up recording space. This is when they see the most activity. These are in the 16x risers. One left, one center. (one in left riser at 29C has unrestricted fan, one in center riser has that 2mm clearance, but only seems to suffer by 1C for it. After almost 3 years in this role they are reporting only 2% wear.
#4 "Special" ZFS VDEV for storing metadata and small files in hard drive pool. 3x ZFS Mirrored 2TB Inland Premium (appear to be reference Phison drives manufactured by Team)
Temperature: 29 Celsius
Temperature: 30 Celsius
Temperature: 30 Celsius
These are Gen3. They see constant slow use, as the hard drive pool is almost always active night and day. (Did a 3 way mirror as my hard drive vdevs are two RAIDz2 vdevs which each have two drive fault tolerance, and recommendation is to match fault tolerance across vdevs. One of these three is in each of the 16x risers. 29c is unobstructed on the left. 30c ones are center and right. After almost 3 years in this role they report only about 1% wear.
#5 Cache / L2ARC drives for hard drive pool. 2x Striped 4TB WD Black SN850X
Temperature: 44 Celsius
Temperature: 46 Celsius
These are Gen4. They see near constant activity as things are moved in and out of the hard drive pools read cache. The level 1 cache (ARC, 384GB) is in RAM. What gets evicted from there gets moved to the cache drives (L2ARC, 8TB) and the least desirable data in that cache gets evicted to make space for it. Since no data is lost if these fail, they are striped to maximize size. One is in th eleft 16x riser, and one is in the right one. Despite near constant write activity, these still report 0% wear after ~7 months in this role.
#6 ZFS SLOG (Separate ZFS Intent Log). 2x ZFS Mirrored 280GB Intel Optane 900p
Temperature: 48 Celsius
Temperature: 48 Celsius
These are Gen3. They see near constant activity, as they each have 4 partitions, 3 of which are in use. Because Optanes are crazy (insane high queue depth IOPS and very low write latency), there is very little drawback from partitioning them, and having each partition mirrored between the two drives, and used to support SLOG activity on three different pools (boot pool, VM Datastoore pool and Hard Drive storage pool). 2.5" U.2 drives are connected via 8x Slimsas connector on motherboard, to U.2 drives in ghetto mounted drive cage on right side of case. 0% wear as expoected despite heavy-ish load, considering they are Optanes. Crazy write endurance on Optanes.
#7 MADM Mirror for Swap Space. 2x ZFS mirrored 1TB Inland Premium (appear to be reference Phison drives manufactured by Team)
Temperature: 29 Celsius
Temperature: 30 Celsius
Used mdadm on these, as ZFS is not appropriate for swap as it can create a race condition. (RAM is low, write more to swap, Swap is on ZFS, causing ZFS RAM use to increase, reducing RAM availability, writing more to swap, etc. etc. Traditional software RAID via MDADM is more appropriate here. One on middle 16x riser, one in right riser. One on the right is hotyter for some reason, despite both having restricted airflow by roughly same amount. These drives were previously in the Cache role, but were moved to the Swap role when I picked up the WD drives ~7months ago. In the swap riole they see very very light writes. Probably negligible. But they did spend 2+ years in th ecache role, and that is probably why they are reporting 5% wear.
#8 Standalone (ext4) drive for MythTV LiveTV 1x 256GB Inland Premium (appears to be reference Phison drives manufactured by Team)
Temperature: 30 Celsius
MythTV kind of "hacks" Live TV by treating them like recordings, writing them to disk, and immediately playing htem back (with a small buffer). This allows you pause/rewind or to mark something as keep/record while watching it and it stays, otherwise they are auto-deleted within 24 hours, or if space is needed. Just like with Scheduled pool, if something is marked record, an overnight script moves it to the scheduled pool (where it remains until it ages out to the hard drive pool or is deleted) With the mother-in-law here in the house, at least one of the MythTV frontends is constantly watching live tv, so this one sees a near constant slow trickle of writes during the day, but only like 1.5 to 4MB/s somewhere. I expected to eat up this drive with writes fast, and then replace it (it was only $29, so why not?) but it has been in use in this role for almost 3 years and is only reporting 9% wear. The drive is in the middle 16x riser.
I guess, I intentionally over-dimensioned the capabilities here, in part becuase that's just what I do, but also in part because I realized I was using consumer NVMe SSD's in a server role, and I didn't want to push my luck, so I split up a lot of the loads to offload and distribute the workload a little.
Nothing runs to hot even at its highest load on a hot day without AC, and nothings really ever given me any trouble except that one 500GB Samsung boot drive which just kind of dropped offline last week. The mirror kepot it going though, and a power cycle brought it back up again. (it had to resilver 1.56GB from th eorrer mirror while it was down). I'm going to have to monitor it going forward, but I'm not too concerned. Worse comes to worse, I'll just buy a couple of other 500+GB NVMe drive and resilver them in place to replace them.
Its interesting to me that the drive that dropped out was one of the boot drives, one of the coolest running of the Gen4 drives and one of the lowest load drives as well with only 2.94TB written over 7 months, while the MUCH more highly loaded VM Datastore drives, also Samsung 980 Pro's are fine. But this is small sample size stuff, so it is difficult to draw any real conclusions.
Welp. That wound up being more of a writeup than I had planned. I hope someone finds this interesting/useful. Arent digressions fun?
