Downsides of using an m.2 enclosure in u.2 bays for servers

[robx]

I am looking into buying some servers for a very IO intensive application run on k8s where I am trying to optimize for random reads and do it on the cheap. So when I was looking at drives, I was wondering what the downsides of using m.2 drives in u.2 enclosures. Looking specifically at getting consumer pcie gen 4 drives and putting them in an [icy dock](https://www.icydock.com/goods.php?id=303) adapter.

Can anyone talk about the down sides of doing this? It is significantly cheaper (3x) than buying u.2 drives but I would think similar performance (minus some reliability I would assume). Is there anything I am missing wrt specs and is this common?

Thank you for your time.

 

[i386]

Consumer ssds are not made for sustained workloads and drop in performance[1][2], also they don't have power loss protection which can corrupt filesystems or in worst cases brick the ssd when power problems occur[3].
Enterprise ssds (usually u.2 ssd) have power loss protection, but are more expensive.

Personally I wouldn't use consumer ssds in a production environment and invest in enterprise ssds...

[1]3 reasons why you need to choose enterprise SSDs for your workloads
[2]https://www.snia.org/sites/default/files/SNIASSSI.SSDPerformance-APrimer2013.pdf
[3]Is there a way to protect SSD from corruption due to power loss?

 

[robx]

Thanks @i386, really appreciate the informative articles. Wasn't aware of the differences in sustained performance.

Please correct me if I am wrong, but do some of the more modern consumer ssds have better sustained throughput than what was noted in [1] and [2] if they have a heat sink? And also, could tuning the filesystem (will be using zfs and ceph) help with garbage collection compared to having a larger cache on enterprise ssds?

Most likely am going with enterprise ssds now but just want to be sure. Thanks.

 

[lihp]

Please correct me if I am wrong, but do some of the more modern consumer ssds have better sustained throughput than what was noted in [1] and [2] if they have a heat sink? And also, could tuning the filesystem (will be using zfs and ceph) help with garbage collection compared to having a larger cache on enterprise ssds?

Most likely am going with enterprise ssds now but just want to be sure. Thanks.

At first looking at consumer M.2 NVMe's IOPS it looks tempting. Maybe even for some RAID 5 or RAID 10, especially with "it's only for me, so not much load at all, so it works..". Turns out: that attitude is 100% wrong. So here the imho "complete" answer to your questions:

1. Yes you can use those U.2 -> M.2 brackets. Sometimes contacts may be off. In those cases cleaning the contacts with alcohol or softly with blue eraser helps to ensure proper contact.
2. Consumer NVMe's have high latency in the area of more than 300µs. Common enterprise NVMe's have latency below 90µs. If you have highly sequential reads fitting to NVMe block size, by then the consumer NVMe beats the enterprise NVMe for the first few times. The more IO happens over time, the more often the consumer NVMe needs to access different blocks or stripes. Meaning the more often the latency hits you while accessing a file. In the end the enterprise NVMe (even when shown with much lower IOPS) will beat the consumer NVMe hands down. Also most consumer NVMe come with cache trying to even out consumer "cheapish" construction. Once out of cache, the consumer NVMe plain sucks anyway.
3. Intel and AMD RAID sucks performance wise. Stripe size, placement,... usually its all off. Your best bet for some performance gains would be RAID 01 (Mirror over striped sets) and even then latency kicks in over time and usage.
4. Using consumer NVMe RAIDs increase the issue of high latency exponentially. Depending on RAID configuration and over time the consumer RAID performance can even drop below a single consumer NVMe stats. You wont be happy, even when you start off happy at first, after a few initial tests. On the long run you loose.
5. Actually NVMe RAID controllers as of now also suck big time (even in enterprise versions). For the most part they are designed for spinning disks. There are only minor adaptations like stripe size alignment, TRIM,... which are added. But so far I didnt see a 100% optimized hardware RAID for NVMe's and only recently I saw one software based NVMe RAID, which looks perfectly aligned for NVMe's. And its not mdadm, ZFS,... (its the RAIDIX).
6. When you compare eg. WD DC SN640 2DPWD (enterprise) drives in RAID 5 versus consumer NVMe's in RAID 01, by then the enterprise RAID is less expensive and outperforms the consumer NVMe. All this while the enterprise RAID is more reliable, more consistent, protects your file system against energy outages and offers a way higher lifetime (and much more).
7. In consumer PCs errors due to memory and file errors are quite common (actually more common than we perceive). OS's actually cope with it and most admins tend to reinstall their workstation after 2-3 years anyways because "it crashes so often" or is "less responsive" or "feels slow" or... Most of that is due to some weird error aka file corruption/write errors/... Thats actually one major reason for self-healing file systems like ZFS. Whereas ZFS sucks performance wise: 1.6 - 5 times slower than XFS depending on platform and configuration.
8. ... there is much more.

Bottom line: IOPS and bandwidth and reliability are a science on their own. Netapp offers some decent presentations explaining the problems properly - way better than I can.

<snip>
very IO intensive application run on k8s where I am trying to optimize for random reads and do it on the cheap.
<snip>

Cheapest, most reliable and performant option for you imho is: get 4+ WD DC SN640 2DWPD drives in RAID 5 and get your free Raidix license (free for up to 8x SN640 in one array, resulting in 4Mill IOPS). Drop XFS atop of it for maximum performance and you are good. If not satisfied or you got issues with raidix, just do a RAID 5 via mdadm or worst case ZFS, but tune them accordingly.