Cheap NVMe performance from HP - 9,000MB/s for $900 or so?

[dba]

HP is getting close to releasing something that should appeal to those looking for serious NVMe performance on the cheap. It's the HP Z Turbo Drive Quad Pro.

I just learned about this and wanted to share. You get a full-height PCI x16 card into which you can stuff up to four NVMe x4 boards. They are claiming IOMeter 128kb sequential reads at 9,000MB/s for goodness sake. Four of those in a dual Xeon E5 server and you've got some really nice throughput. Retail pricing is supposed to start at $300 for the card with one 256GB NVMe board. I am thinking that you could buy one, throw out the small NVMe card, and add four larger NVMe boards of your own (maybe Samsung 950 pro?) to maximize throughput/$. Actually if their pricing holds, the retail of $900 for 1TB at 9,000MB/s is plenty good already, though the 1TB version likely has lower throughput than the 2TB version from their test results. We'll see.

HP Reveals New Z Turbo Drive Quad Pro | StorageReview.com - Storage Reviews
HP Z Turbo Drive Quad Pro | HP® Official Site

UPDATE: See below. Chuntzu found that the card is BIOS locked to specific HP workstations.

 

[Patrick]

HP is getting close to releasing something that should appeal to those looking for serious NVMe performance on the cheap. It's the HP Z Turbo Drive Quad Pro.

I just learned about this and wanted to share. You get a full-height PCI x16 card into which you can stuff up to four NVMe x4 boards. They are claiming IOMeter 128kb sequential reads at 9,000MB/s for goodness sake. Four of those in a dual Xeon E5 server and you've got some really nice throughput. Retail pricing is supposed to start at $300 for the card with one 256GB NVMe board. I am thinking that you could buy one, throw out the small NVMe card, and add four larger NVMe boards of your own (maybe Samsung 950 pro?) to maximize throughput/$. Actually if their pricing holds, the retail of $900 for 1TB at 9,000MB/s is plenty good already, though the 1TB version likely has lower throughput than the 2TB version from their test results. We'll see.

HP Reveals New Z Turbo Drive Quad Pro | StorageReview.com - Storage Reviews
HP Z Turbo Drive Quad Pro | HP® Official Site

Looks somewhat like the Dell card I got. Fan + 4x m.2 slots. Much to test, little time.

The big question for me is how are they dealing with PCIe bifurcation.

 

[pyro_]

I was just wondering today if someone made something like this for a server I am putting together. Will be interesting to see ho well it works and if white box machines can use it

 

[tare55]

wow looks nice. I have not seen too many x16 x 4 server boards

 

[Chuntzu]

Tomsitpro article I just read mentions a bios lock maybe on this device. Boy that would sure piss me off if they did. This thing is priced perfectly and could be a fantastic product as long as I am not tied into buying an hp workstation so the thing will function.

 

[Patrick]

Tomsitpro article I just read mentions a bios lock maybe on this device. Boy that would sure piss me off if they did. This thing is priced perfectly and could be a fantastic product as long as I am not tied into buying an hp workstation so the thing will function.

I have been trying to get the Dell version of that card to work:

The lack of a PCIe switch chip makes it more difficult.

Thanks to @Patriot for the heads up on this unit.

 

[Patrick]

I posted pictures of the Dell version on the main site: Dell 4x m.2 PCIe x16 version of the HP Z Turbo Quad Pro

@dba if you want to try it out let me know.

 

[dba]

...
The big question for me is how are they dealing with PCIe bifurcation.

True. It could be BIOS-specific so that it only works in certain HP boards. If PCIe bifurcation is the issue and not BIOS lock, then one of those GPU boards that splits 4 x16 paths into 8 x16 slots by using four PLX chips could do it perhaps? But then again they probably just BIOS locked it to specific machines like Chuntzu said and called it a day.

UPDATE: Looks like they did:
"Thanks to a BIOS lock, the device is supported only on the HP Z440, Z640, and Z840 Workstations"
http://www.tomsitpro.com/articles/hp-reveals-turboz-quad-pro,1-3022.html

 

[dba]

wow looks nice. I have not seen too many x16 x 4 server boards

There are quite few boards will multiple x16 slots. Just look for boards designed for GPUs or BIG server boards designed for high IO.
A dual-CPU board can easily have four x16 slots. My quad CPU Supermicro DB server happens to have seven x16 slots directly connected to the CPUs:

Supermicro | Products | SuperServer | 4U | 4047R-7JRFT

 

[dba]

I posted pictures of the Dell version on the main site: Dell 4x m.2 PCIe x16 version of the HP Z Turbo Quad Pro

@dba if you want to try it out let me know.

I wish, but no time right now. I have 18,000 CPU cores worth of servers showing up in two weeks. eBay of course!

 

[Patrick]

Awesome @dba! 1125 nodes?

 

[alex1002]

@dba do you know if this true to the Dell card?

 

[dba]

Awesome @dba! 1125 nodes?

Now that the Xeon E5 CPUs are reasonably priced (E5-2670) , I'm going crazy. I have C6220 nodes going into one location where I have available traditional rack space and then tons of the OpenCompute V2 "Windmill" nodes going into cages at other data centers. All told I'll have ten separate "Pods" of servers running around the US with this expansion.

BTW, the very first pod has been up and running for ~280 days with no issues. I haven't even had to reboot the two EdgeRouter Pro-8 routers there, each of which has a direct fiber connection (AWS Direct Connect) to Amazon.

 

[dba]

@dba do you know if this true to the Dell card?

My knowledge is limited to what I found in the press releases - and we all know what that means. Exciting stuff, but until someone actually gets one running, it's just theory.

 

[Chuntzu]

I wish, but no time right now. I have 18,000 CPU cores worth of servers showing up in two weeks. eBay of course!

Awesome so you pulled the trigger on those windmill racks, very very cool!

 

[Chuntzu]

I have been trying to get the Dell version of that card to work:
View attachment 1145
The lack of a PCIe switch chip makes it more difficult.

Thanks to @Patriot for the heads up on this unit.

You have been siting on that for a month! Glad that you spent the time testing it though, I am curious what the requirements were for the nonplx version of the supermicro card were? Did they motherboards have to have some sort of on board bifurcation, like on socket 1155 boards which will do a 16x slot or if two cards are I stalled it would end up being 2 8x slots? So the dell board would work on those boards as well, perhaps? have little faith that the hp card will have the on board plex chip based only on the proposed price. Curious though if you have had that dell board fully populated and running yet?

 

[Patrick]

You have been siting on that for a month! Glad that you spent the time testing it though, I am curious what the requirements were for the nonplx version of the supermicro card were? Did they motherboards have to have some sort of on board bifurcation, like on socket 1155 boards which will do a 16x slot or if two cards are I stalled it would end up being 2 8x slots? So the dell board would work on those boards as well, perhaps? have little faith that the hp card will have the on board plex chip based only on the proposed price. Curious though if you have had that dell board fully populated and running yet?

I think it requires a PCIe switch chip onboard the motherboard.

 

[PnoT]

I guess once someone can figure out the hardware part the BIOS lock should be fairly easy to circumvent with the right person getting their hands on it

 

[Paul A. Mitchell]

> "it's just theory"

I believe it's more than just theory:

one NVMe device uses x4 PCIe 3.0 lanes;
four of those on a single AOC equals x16 PCIe 3.0 lanes.

That bandwidth equivalence has engineering elegance:
the raw upstream bandwidth equals the raw downstream bandwidth.

Now, extrapolate that equivalence to PCIe 4.0's 16 GHz clock rate!

It's about time that storage performance started to catch up
with multiple PCIe video cards (SLI and Crossfire):
it's been too long already.


MRFS

 

[Patriot]

> "it's just theory"

I believe it's more than just theory:

one NVMe device uses x4 PCIe 3.0 lanes;
four of those on a single AOC equals x16 PCIe 3.0 lanes.

That bandwidth equivalence has engineering elegance:
the raw upstream bandwidth equals the raw downstream bandwidth.

Now, extrapolate that equivalence to PCIe 4.0's 16 GHz clock rate!

It's about time that storage performance started to catch up
with multiple PCIe video cards (SLI and Crossfire):
it's been too long already.
MRFS

Until someone shows it working... it is a theory...
PCIe bifurcation is an issue... you can't just split an x16 to an x4 without a chip to handle that.
That chip adds latency... scaling will not be perfect, when dealing with multi cpu setups, you might be crossing numa nodes... Raw theoretical is awesome, actual.... we will see.