Whats the deal with U.2 enterprise SSDs selling so cheap?

[111alan]

U.3 Disks are cool. U.3 Backplanes are a bad idea.

Most U.3 disks also support U.2 cards/cages. Unless it said "U.3 only"

I think the entire idea of U.3 is stupid. Are they really sacrificing the backward compatibility just to save two pairs of wires in their backplane/cable? People may just choose not to upgrade the drives, because they have to replace everything.

 

[ericloewe]

Most U.3 disks also support U.2 cards/cages. Unless it said "U.3 only"

I think the entire idea of U.3 is stupid. Are they really sacrificing the backward compatibility just to save two pairs of wires in their backplane/cable? People may just choose not to upgrade the drives, because they have to replace everything.

I sort of get it, to some extent. Due to the need for airflow through the backplane, routing gets tricky, especially close to the expander. High-speed differential pairs are always a pain in the ass, so the extra two or three pairs per slot may represent a good few layers in some designs, which really inflates cost. If it were up to me, U.3 would be NVMe only, with Tri-Mode being relegated to U.2.
The real "benefit" of U.3 is that it helps Broadcom and Microchip sell overpriced Tri-Mode HBAs and expanders, under the guise of cost reductions. Still, things were mostly okay in my book (I can just avoid the U.3 backplane nonsense) up until I heard that some U.3 disks do not follow the spec and are actually not compatible with U.2 backplanes. I don't know if these offenders are real, much less who they are, but that pissed me off greatly.

 

[NablaSquaredG]

The real "benefit" of U.3 is that it helps Broadcom and Microchip sell overpriced Tri-Mode HBAs

Which are not even good!

The performance of Broadcom TriMode stuff for NVMe has always been horrible... Maybe that changed in the 9600 generation?

 

[Tech Junky]

U.3 is stupid. Are they really sacrificing the backward compatibility

The drives are backwards compatible just not the cables or backplanes.

It's a complicated mess. If building from scratch though make a plan. The idea behind U3 is a single connector for different types of disks instead of one for SATA, NVME, SAS, and so on. The issue lies with OEMs not producing the other types of disks with the 8639 connector.

 

[oneplane]

Oh come on, I don't need to know about this. Now I have to buy some U.2 hardware 16x4TB u.2 ZFS pool incoming...

 

[unwind-protect]

Keep in mind that consumers, even if they are willing to use a PCIe card, might not be able to do so.

Lots of physically small boards out there, and GPUs that are 3.5 slots wide.

And the PCIe slot might be on the southbridge, whereas the boards have one M.2 slot on the CPU. Consumers rate that highly, probably irrationally so.

 

[SnJ9MX]

Oh come on, I don't need to know about this. Now I have to buy some U.2 hardware 16x4TB u.2 ZFS pool incoming...

Hope you have the PCIe lanes to support it! 4x each x 16 = 64 lanes. You're firmly in EPYC territory.

 

[111alan]

Keep in mind that consumers, even if they are willing to use a PCIe card, might not be able to do so.

Lots of physically small boards out there, and GPUs that are 3.5 slots wide.

And the PCIe slot might be on the southbridge, whereas the boards have one M.2 slot on the CPU. Consumers rate that highly, probably irrationally so.

Use M.2 to U.2 converter and U.2 cable. Both 8643 and 8611 will do.

 

[ericloewe]

Which are not even good!

The performance of Broadcom TriMode stuff for NVMe has always been horrible... Maybe that changed in the 9600 generation?

It's supposed to have been improved, as I understand it by dragging the old MegaRAID stack kicking and screaming into a somewhat more modern setup. I mostly interpret that as "we expect the software to be stable by 2026". And NVMe disks still get to sit behind a SCSI driver, so kiss goodbye to all the latency and parallelism benefits of NVMe.

Of course, that's all a feature, since it means clueless enterprise customers can simply be told to buy more disks and more controllers and more everything to compensate.

 

[unwind-protect]

Use M.2 to U.2 converter and U.2 cable. Both 8643 and 8611 will do.

Right, but then you need a separate place to mount the drive.

 

[111alan]

Right, but then you need a separate place to mount the drive.

Just like sata drives. It's hard to find a chassis which doesn't even have a 2.5 inch drive bay.

 

[MC68000]

Just found out Qnap sells a crazy 12 bay u.2 capable NAS box with 2x 25gbe ports. TS-h1290FX
dont know how trustworthy they are compared to Synology which I use for my warm storage on spinning rust.

It does seem not much U.2 enterprise bays out there? Does the Netapp all flash systems like the A series use those? I guess it makes sense if the whole U.2 did not take off big in the market which means nice quality drives on the cheap.

 

[MC68000]

nobody has enough u.2 slots and hard to cool

hotswap is also.. well it works sometimes, maybe 5%?

I use multiple Helios 3s enclosures with locking removable drive caddys. Drives stay at 47-51 all day even on location where ambient is 82. When done I eject, pull out caddy and slide in new ones.

 

[MC68000]

that's most likely why it has almost literal zero adoption. Well they (U.2 disks) will become quite popular in homelab as lsi 9400 gets cheaper.

How good are those LSI 9400s ie 9440 8i etc. I see them for 140 new old stoc. Would like to build a dedicated machine using that.

 

[reasonsandreasons]

Just found out Qnap sells a crazy 12 bay u.2 capable NAS box with 2x 25gbe ports. TS-h1290FX
dont know how trustworthy they are compared to Synology which I use for my warm storage on spinning rust.

My impression is that Qnap has had more than its fair share of security issues, but I know many of the consumer units can boot third-party operating systems. I imagine you're out of luck for support, though.

 

[oneplane]

Just found out Qnap sells a crazy 12 bay u.2 capable NAS box with 2x 25gbe ports. TS-h1290FX
dont know how trustworthy they are compared to Synology which I use for my warm storage on spinning rust.

It does seem not much U.2 enterprise bays out there? Does the Netapp all flash systems like the A series use those? I guess it makes sense if the whole U.2 did not take off big in the market which means nice quality drives on the cheap.

It also costs 8K, for that kind of money you can also buy a supermicro server I think a 1029P-N32R configured for 32x NVMe is about 7K. Or you get a lightly used one: Supermicro SYS-1029P-MTR SuperServer 2nd Gen Silver Configurable 1U Rack Server | eBay

Now I want one for homelab VM disk storage

 

[mattventura]

I'm actually planning on making a 12xNVMe server. You can get an old 826 chassis (~$250), replace the backplane with a 12x NVMe for another ~$250, and then whatever appropriate CPU/MB.

Only question is, how am I going to get that many NVMe ports on the motherboard... H13SSL can handle 6 drives with its MCIO ports, but I don't know what adapter I would use for the other 6 drives while still having backplane management.

 

[hmw]

The main problem in utilizing all those cheap NVMe drives is where do you put them? I tried using a HPE 8x SFF NVME drive cage - the 826689-B21 can be had for ~ $100 and has space for 8 x 2.5" NVMe drives with 4 SlimSAS 8i ports. The problem is it won't work in any system I tried (and I tried a couple and also with different PCIe adapters and redrivers) I *really* want to avoid using an LSI adapter if I can. But yeah - with 8 x4 lanes, it's EPYC or Scalable Xeon all the way. Other systems simply dont have that number of PCIe lanes to spare

Oh and hotplug is ... non existent

 

[mattventura]

I must have gotten lucky with my hardware combinations. I haven't had issues with hotplugging on any server-y kind of system. Even an old X7SPA-H let me surprise insert a drive via a dumb PCIe x4 -> SFF-8654 adapter. Using a good PCIe switch (one with a fully working hotplug controller) can help with this, since it means the slot that you put the switch in doesn't need a hotplug controller.

 

[i386]

replace the backplane with a 12x NVMe

do you have a product number/sku for that?
I only know of the 24x u.2 backplane (BPN-NVMe3-216N-S4) for the 216 chassis