Whats the deal with U.2 enterprise SSDs selling so cheap?

[Tech Junky]

And this is why I opt for dumb adapters that just translate from PCIE to ____. If there's something on the PCB that sticks up then it's a no go unless it's something to do with switching like a PLX controller. There's no need to involve drivers if you want your data to be there when you reboot. Using AMD and bifurcation natively is safer in my book when it comes to these drives. Especially when you're dropping $1000/drive for the higher capacity drives at Gen4 speeds.

U drives are great in terms of price/TB vs M2 but, there are caveats that will bite you in the @$$ if you're not aware. It brings to mind the whole idea of not buying a bunch of components and then not installing or at least testing them in a timely manner before the return window closes. I've seen so many threads over the years where people buy stuff every couple of weeks instead of all at once and then stumble upon 1 piece not working and having to deal with the RMA procedures that sometimes take months.

KISS works well and saves time/money/troubleshooting.

 

[hmw]

And this is why I opt for dumb adapters that just translate from PCIE to ____. If there's something on the PCB that sticks up then it's a no go unless it's something to do with switching like a PLX controller. There's no need to involve drivers if you want your data to be there when you reboot. Using AMD and bifurcation natively is safer in my book when it comes to these drives. Especially when you're dropping $1000/drive for the higher capacity drives at Gen4 speeds.

U drives are great in terms of price/TB vs M2 but, there are caveats that will bite you in the @$$ if you're not aware. It brings to mind the whole idea of not buying a bunch of components and then not installing or at least testing them in a timely manner before the return window closes. I've seen so many threads over the years where people buy stuff every couple of weeks instead of all at once and then stumble upon 1 piece not working and having to deal with the RMA procedures that sometimes take months.

KISS works well and saves time/money/troubleshooting.

You realize that the P411W-32P is just a PCIe switch right?

 

[Tech Junky]

You realize that the P411W-32P is just a PCIe switch right?

I didn't look it up and if it needs drivers or firmware it's going to be a source of pain eventually.

 

[hmw]

I didn't look it up and if it needs drivers or firmware it's going to be a source of pain eventually.

All the PCIe devices you have need microcode, drivers and firmware. Just because you don't see them doesn't mean they don't.

You can claim that "My NVMe drive doesn't need drivers" or "I can run stuff off my mainboard PCIe slots without a driver". But that is because you're using inbox drivers that come with the OS or your mainboard & CPU runs microcode that set the PCIe slot parameters to known values that will work for 90% of the cases

I've seen problems with mainboards and M.2 NVMe drives that meant the mainboard had to get a BIOS update. Why? Because you're routing PCIe Gen4 signals to the M.2 slots and covering 90% of the cases. When you get a M.2 drive that has slightly different characteristics for impedance - your 'works fine with m.2' goes out of the window

It's the same with PCIe esp Gen4 and Gen5. Those cards that work on native slots + bifurcation? Again, they're tuned for 90% of the cases. And if you use them with longer cables or different drives - unless you have a SuperMicro, Dell or similar mainboard where you can tune PCIe signal parameters - you're out of luck

Take a look at even the driverless 'c-payne' PCIe re-driver ... it has a way to set signal gain and PCIe signal parameters, again because you need it for different cable lengths and endpoint types (ie GPU vs NVMe device etc)

 

[Tech Junky]

All the PCIe devices you have need microcode, drivers and firmware. Just because you don't see them doesn't mean they don't.

I guess I should qualify it with if it's not built into the uefi or OS it will be a pain later.

A dumb PCB adapter though shouldn't need any additional tweaking other than splitting the lanes in the uefi. Looking at them will show only the electrical traces mapped to the slot. My point being keep it as simple as possible to a oud issues later.

For instance my 5G modem and USB adapter required some old drivers to get them working in Windows but Linux worked just fine upon connecting it. When I redo windows it's going to be one extra step to remember to install them together it working again. There's just more crap to deal with when you add drivers to the equation. It should just be a simple electrical connection to the system and the device should come online and be usable.

 

[nexox]

A dumb PCB adapter though shouldn't need any additional tweaking other than splitting the lanes in the uefi.

At the frequencies that PCIe 4 and 5 run (and to a lesser extent 3) every trace, connector, wire, and end device influences the signal quality, and each generation has tighter tolerances, thus the availability of some tuning options in certain systems to expand compatibility to a wider range of configurations.

All the things you like are things I have occasionally had problems with - UEFI arbitrarily failing to load the driver for an m.2 drive in a motherboard slot that had been booting fine for months and operating systems missing drivers for super basic stuff like AHCI SATA and USB Mass Storage. The random github driver for my FusionIO card just isn't that big of a deal, especially compared to the fun that can be had if you encounter an issue in the linux-firmware repo.

 

[Tech Junky]

@nexox

All valid considerations. The point is the less moving parts the easier life is.

I did have an issue when switching from a direct PCB mount to cable+M2. The problem was the cable was too long and the attenuation caused the drive to not mount. Fix... shorter cable.

Not every system is the same. This poses an issue when selecting the HW you want to put inside. Linux is a broad term in which there's more than one base available debian / bsd / etc. If you go with debian there's about 32 flavors like Baskin Robbins but, the premise is still the same in terms of compatibility. If you go BSD though you'll likely have more issues with compatibility.

BIOS / UEFI though is about as consistent as well. Deafferent companies do things differently. For instance when I put together my current X670E setup it was like a mine field sorting through the MOBO options compared to Intel based systems. It's a hot mess.

Drives make a difference as well as some are cheaper and run hotter and others are less and cooler.

Bottom line is there's a lot of variables that come into play and minimizing your risk makes life easier.

 

[hmw]

I guess I should qualify it with if it's not built into the uefi or OS it will be a pain later.

The PCIe signal parameters are built into the UEFI
NVMe drivers are built into OS

Both can and do cause huge problems

Imagine you're using an Asus or MSI mainboard - you're using MCIO on the mainboard with 30cm cables to your shiny new Gen5 NVMe drives. Now you need 50cm and it doesn't work. On a Supermicro you can actually adjust parameters. On your Asus or MSI you suddenly have no choice but to use a PCIe Gen5 switch/retimer

And what are you going to do when your OS stops updating or providing built in/inbox drivers? Will you just throw away all the investment in existing hardware and go out and buy new stuff?

Bottom line is there's a lot of variables that come into play and minimizing your risk makes life easier.

Sure - it's called using Apple devices for home and Dell/HPE servers for work or homelab. But how many folks have the $$$$ to drop $20,000 on a Dell server because you want to use Dell branded memory/SSDs etc to get that 'minimizing of risk' ? It's a great goal and I wish I could do that - but I would rather figure out a way to spend $2,000 and accept *some* risk in return - hence the STH forums

 

[Tech Junky]

Asus or MSI

Well, that's a mistake to even consider if you're working outside of the box when it comes to hardware.

OS stops updating

Linux doesn't because it's driven by both individuals and corps.

spend $2,000

I spent $800 all in before drives on my AMD setup. So, you have a spending problem.

$20,000 on a Dell server

That's a dumb idea.

 

[hmw]

That's a dumb idea.

No that IS what it costs when you spec out a Dell server with validated components for say a VCF setup - Dell will guarantee that it will work with VCF and will stand by that. It will talk to all the vendors for all the components on your behalf. And it will sort and solve problems so you don't have to. You're paying for the privilege of getting 100 of these servers, installing VCF on it and seeing it work properly first time and every time

Oh and Linux DOES stop updating because if Dell/Microsoft/IBM etc all stop funding something AND there's no one interested - you're out of luck

I think from your posts it is clear you're waaayy out of your depth

 

[ano]

anyone know of any good current deals? thought I had more than I had :| P4610 for $200 would have been nice now...

 

[Tech Junky]

@hmw

This thread isn't about your wet dream of 100 Dell servers.

 

[hmw]

@hmw

This thread isn't about your wet dream of 100 Dell servers.

You're right - it's about U.2 enterprise drives - which you seem to have huge problems with. Why not just give it up, move on, and let us discuss enterprise grade storage here without the off-topic comments? Putting you on ignore so bye

 

[hmw]

Getting back to the topic - all the U.2 drives I can see on eBay are Gen3 ones, mostly WD and Kioxia. Just got two 7.68TB SN640's, still under warranty for another year. Can't find any PCIe Gen4 drives - so I am guessing this is folks moving off Gen3 or then consolidating storage by buying the bigger sized SSDs.

Speaking of larger sized SSDs - Solidgm claims to have started shipping their D5-P5336 drive in 2.5" U.2 format, the 30.72TB goes for $1800 ~ $1900 which is actually a decent price. The problem is - have not seen a single vendor with stock, they all tell you it's a special order.

Even if it the D5-P5336 is a QLC drive, the native sector size is 16K, which matches nicely with NTFS's sector size for >30TB partitions. If you consider one of these can replace 4 of 7.68TB SSDs, the power savings alone are enormous: 4x5W= 20W idle and 4x25 = 100W at load

 

[ano]

power savings is a thing, I'm just debating nvme size vs power myself.

power mgmt is.. well lacking in most nvme storage devies in enterprice/dc class right now, most edsff is as power hungry as u.2

we have some ceph nodes that use around 200w with 24core, 512GB ram, etc.. and adding just a few more OSD we will probably move to more nvme per ode, from 1, to 2.. thats a lotof more power, thats with 7.68TB.. with smaller drives I would need 4! or more. and they all use a lot in u.2

 

[hmw]

power savings is a thing, I'm just debating nvme size vs power myself.

power mgmt is.. well lacking in most nvme storage devies in enterprice/dc class right now, most edsff is as power hungry as u.2

we have some ceph nodes that use around 200w with 24core, 512GB ram, etc.. and adding just a few more OSD we will probably move to more nvme per ode, from 1, to 2.. thats a lotof more power, thats with 7.68TB.. with smaller drives I would need 4! or more. and they all use a lot in u.2

Eh .. my SN640 is looking like crap in power & idle

ps      0 : mp:12.00W operational enlat:50 exlat:50 rrt:0 rrl:0
            rwt:0 rwl:0 idle_power:- active_power:-
            active_power_workload:-
ps      1 : mp:11.00W operational enlat:50 exlat:50 rrt:1 rrl:1
            rwt:1 rwl:1 idle_power:- active_power:-
            active_power_workload:-
ps      2 : mp:10.00W operational enlat:50 exlat:50 rrt:2 rrl:2
            rwt:2 rwl:2 idle_power:- active_power:-
            active_power_workload:-
ps      3 : mp:9.00W operational enlat:50 exlat:50 rrt:3 rrl:3
            rwt:3 rwl:3 idle_power:- active_power:-
            active_power_workload:-
ps      4 : mp:8.00W operational enlat:50 exlat:50 rrt:4 rrl:4
            rwt:4 rwl:4 idle_power:- active_power:-
            active_power_workload:-

Doesn't this thing have idle ???

 

[Tech Junky]

power savings is a thing

I haven't used a ton of these with only 2 brands so far but, they do seem to just be less efficient when it comes to power draw,

Model Number:                       KIOXIA KCD8XRUG15T3
Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W   25.00W       -    0  0  0  0   500000  500000
 1 +    12.00W   12.00W       -    1  1  1  1   500000  500000

Model Number:                       WDS100T1X0E-00AFY0
Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W    9.00W       -    0  0  0  0        0       0
 1 +     4.10W    4.10W       -    0  0  0  0        0       0
 2 +     3.50W    3.50W       -    0  0  0  0        0       0
 3 -   0.0250W       -        -    3  3  3  3     5000   10000
 4 -   0.0050W       -        -    4  4  4  4     3900   45700

Model Number:                       WD_BLACK SN770 1TB
Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     5.00W    5.00W       -    0  0  0  0        0       0
 1 +     3.30W    3.00W       -    0  0  0  0        0       0
 2 +     2.20W    2.00W       -    0  0  0  0        0       0
 3 -   0.0150W       -        -    3  3  3  3     1500    2500
 4 -   0.0050W       -        -    4  4  4  4    10000    6000
 5 -   0.0033W       -        -    5  5  5  5   176000   25000

I don't think it would be a huge issue for the OEMs to add more power profiles to the disks via firmware updates but, unless companies squeeze them on it I doubt they'll do it. Looking at U vs M2 though it's easy to spot that the consumer stuff is geared more for low power use like laptops but, even the difference between DRAM (SN850) and w/o (770) there's about ~50% more max power available. Still the U drive minimum is ~33% higher than the top profile for the 850. I didn't capture the info o the Micron drives though while I had them to compare since they didn't last longer than a week.

 

[nexox]

Eh .. my SN640 is looking like crap in power & idle

The official specs are "<5W" idle, likely it just uses less than the reported power mode limit when there's no IO.

Actual idle power states present a bunch of opportunities to add bugs, particularly weird incompatibilities with specific systems which are super annoying to reproduce and fix, just not worth it when customers don't intend to idle the hardware.

 

[jei]

Model Number:                       Micron_7450_MTFDKCC15T3TFR
Firmware Version:                   E2MU200
PCI Vendor/Subsystem ID:            0x1344
IEEE OUI Identifier:                0x00a075
Total NVM Capacity:                 15,362,991,415,296 [15.3 TB]

nvmecontrol power -l nvme0

Power States Supported: 8

 #   Max pwr  Enter Lat  Exit Lat RT RL WT WL Idle Pwr  Act Pwr Workloadd
--  --------  --------- --------- -- -- -- -- -------- -------- --
 0: 25.0000W    0.000ms   0.000ms  0  0  0  0  0.0000W  0.0000W 0
 1: 20.0000W    0.000ms   0.000ms  1  1  1  1  0.0000W  0.0000W 0
 2: 18.0000W    0.000ms   0.000ms  2  2  2  2  0.0000W  0.0000W 0
 3: 16.0000W    0.000ms   0.000ms  3  3  3  3  0.0000W  0.0000W 0
 4: 14.0000W    0.000ms   0.000ms  4  4  4  4  0.0000W  0.0000W 0
 5: 12.0000W    0.000ms   0.000ms  5  5  5  5  0.0000W  0.0000W 0
 6: 10.0000W    0.000ms   0.000ms  6  6  6  6  0.0000W  0.0000W 0
 7:  8.2500W    0.000ms   0.000ms  7  7  7  7  0.0000W  0.0000W 0

 

[mattventura]

anyone know of any good current deals? thought I had more than I had :| P4610 for $200 would have been nice now...

There were new Kioxia CD6 drives going for $175 for 3.84TB, but the seller has since raised their prices substantially. The best deal I see now is $205 for 3.84TB. The seller probably isn't getting the best price they could, because they don't have anything related to gen4 in the title, and they're mistakenly advertising it as U.2 when it's actually U.3 (but is backwards compatible).