SSD design question…

[NickKX]

So modular SSDs typically have three components…a controller silicon, dram cache, and the actual nand flash chips, right?

But, why do it this way?

Wouldn’t it be better to have use a software controller, use system DRAM for cache, and expose the raw nand chips directly to the CPU?

So many controller firmware issues have caused data corruption / permanent data loss…and putting dram directly on the module has a lot of downsides as well, usually in the form of there not being enough dram cache to handle certain io patterns and workloads, and also added costs & complexity for SSD vendors.

Having a fully flash aware file system (like combination software SSD controller & file system) would also have a lot of performance benefits too, I would think.

Thoughts…?

 

[BlueFox]

How exactly do you propose to do that? Via PCIe? Then you're back to needing a controller.

If literally directly wired to the CPU, it would take a considerable number of pins that would be effectively wasted on a very slow device, nevermind having to completely redesign CPU architecture. You also need something to keep track of wear leveling, encryption, etc, so, you're back to having a controller.

 

[Chriggel]

HMB already uses system memory to cache data on cacheless SSDs. The controller on the SSD is important though, it has the NAND interface, performs NAND specific operations and talks the protocol of choice to the host (SATA, SAS or NVMe). Routing the connections from the NAND to the CPU would be an engineering nightmare. CPUs needed to be redesigned to include a NAND interface and I think the concept of adding more devices to a system would also just die.

 

[NickKX]

How exactly do you propose to do that? Via PCIe? Then you're back to needing a controller.

If literally directly wired to the CPU, it would take a considerable number of pins that would be effectively wasted on a very slow device, nevermind having to completely redesign CPU architecture. You also need something to keep track of wear leveling, encryption, etc, so, you're back to having a controller.

I think Pure Storage does it like this (Apple too). The modules do have controllers (more like bridges, I think) but they are very dumb and simplified, and all they really do is present the raw nand chips over PCIe using the nvme protocol. The actual data moving, encryption, wear leveling, over provisioning, garbage collection, etc is all done in software.

 

[i386]

Pure Storages uses open channel ssds: they have controllers but no flash transition layer (basically the host cpu & ram must take take care of wear level, gc, logical to physical storage mapping etc.)

 

[twin_savage]

Wouldn’t it be better to have use a software controller, use system DRAM for cache, and expose the raw nand chips directly to the CPU?

Any remotely modern NAND controller needs to have an analog sampling circuit in it that can sample cells voltages, this isn't something a digital system like a CPU has on it.

This is why you never see any modern NAND without a discreet controller attached to it. When you get down into embedded systems with low part counts they tend to have a controller integrated into the NAND itself like with EMMC.

 

[zir_blazer]

What you're describing is Open-Channel SSD. A decade old idea that as far that I know, went nowhere.
You are supposed to need some kind of interface between a Bus like PCIe and a number of NAND chips, but the interface can be dumb since all the processing is done host side so is not a full blown NAND controller, just a PCIe-to-NAND Flash bridge.

Pure Storages uses open channel ssds: they have controllers but no flash transition layer (basically the host cpu & ram must take take care of wear level, gc, logical to physical storage mapping etc.)

I always forgot that this idea had actually a physical implementation.

 

[NickKX]

Probably requires a massive amount of engineering resources and vertical integration, and that’s why we lowly consumers and prosumers / home labbers don’t get to experience this tech. Glad to know there’s a name for it though, thank you!

I am at least happy to see HMB taking off more…that’s a good thing IMO.

 

[Mithril]

Really what it comes down to is this is something a CPU would be really really really bad at from a silicon efficiency standpoint. You're looking at similar reasons and design choices why GPUs happened. Even CPUs with iGPUs are using GPU cores on the same silicon or as chiplets.

"So many controller firmware issues have caused data corruption / permanent data loss" citation needed I guess?

Even if the CPU "could" do everything (and do it as well, which it couldn't) you'd need to pre-determine how many NAND channels it has, then place them all VERY close to the CPU physically to get decent speeds. You'd be limited to a single set of NAND, and maybe the CPU can carve it out to multiple "drives" that the OS sees, but no easy pulling your steam drive out to clone it to a bigger SSD when you upgrade. No easy way to clone your OS drive when you get warnings about failing flash. Strong possibility of the Apple problem where you can't even move the storage to another system at all, or upgrade the size past what the OEM decided on, or upgrade to faster/better storage.

If you look at where datecenters are going (one of the bigger drivers of tech and often a vision into what comes to consumers in some form) its PCIe everywhere. PCIe switches like ethernet switches, rackmounted units that have no CPU just PCIe devices connected with PCIe fabric the way SAS disk shelves expanded storage outside of a single rackmount unit.

Apple's solution isn't *really* controllerless either, its closer to how other SOCs work (the M1 and M2 are basically SOC). And the end goal there is "what is good for Apple".

 

[unwind-protect]

It does not matter whether bad software runs on the SSD controller or on the main CPU.

 

[NickKX]

It does not matter whether bad software runs on the SSD controller or on the main CPU.

Right, but what I was getting at is that drive controller firmware tends to be closed source & proprietary. It’s actually better from a security standpoint too, since you don’t know what the hell is running on that drive controller silicon

. No easy way to clone your OS drive when you get warnings about failing flash. Strong possibility of the Apple problem where you can't even move storage to another system at all, or upgrade the size past what the OEM decided on, or upgrade to faster/better storage.

Apple does this to lock down their devices from competitors so they can charge more for storage upgrades, these are all artificial limitations (not actual). As stated the Apple solution as well as purestorage and other enterprise vendors use nand to PCIe bridges, not full blown controllers, and that’s more or less what I was thinking of, as you would ofc need some way to talk to the raw nand from the cpu.

 

[unwind-protect]

I am not sure that Apple's current SSDs run controllerless.

 

[111alan]

People have already thought that, for example, Fusion IO drives. Then there are a lot of problems such as high latency, extensive traffics between CPU and the drive due to FTL maintainance, dependence on CPU load, and meta data corruptions due to faults in system memory.

There is already a "peripheral" part on the NAND chip, so CPU doesn't have to have an analog driving circuit for it to work. But PCI-e usually don't have more than 16 lanes(x32 is very uncommon, and most SSDs use x4 or less), and the channels don't switch on and off that frequently. So even if NAND manufacturers changes their NAND protocol from ONFI/Toogle to PCI-e, the flexibility will be severely limited.

And yes, Apple SSD has its controller inside the T2 chip.

 

[Whaaat]

If literally directly wired to the CPU, it would take a considerable number of pins that would be effectively wasted on a very slow device, nevermind having to completely redesign CPU architecture. You also need something to keep track of wear leveling, encryption, etc, so, you're back to having a controller.

Intel Optane Persistent Memory is already there

 

[BlueFox]

Intel Optane Persistent Memory is already there

It's not. There is a controller between the NAND and CPU.

 

[i386]

optane is 3dxpoint (a "new" technology), not nand

 

[BoredSysadmin]

and optane has been killed

 

[Whaaat]

There is a controller between the NAND and CPU.

Memory controller is not between, but inside the CPU. Any device either communicate with CPU via controllers and buses or becomes part of it

 

[BlueFox]

Memory controller is not between, but inside the CPU. Any device either communicate with CPU via controllers and buses or becomes part of it

There's a sizeable controller on each DIMM: https://www.servethehome.com/a-close-look-at-intel-optane-dc-persistent-memory-modules/

 

[Whaaat]

There's a sizeable controller on each DIMM

Technically yes, it is required at least for encryption purposes. Anyway it's as close to CPU as you can get with NAND, because any DRAM LRDIMM module contains a sizeable controller on each DIMM called iMB: