Cheap NVMe performance from HP - 9,000MB/s for $900 or so?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Jun 24, 2015
140
13
18
75
So then, let's factor in some aggregate controller efficiency e.g. 0.76
which is conservative because it was measured with SATA-III SSDs
and the extra overhead required by the SATA protocol:

x16 lanes @ 8 GHz / 8.125 bits per byte x 0.76 = 11.97 GB/second

Clock rate doubles at PCIe 4.0:

x16 lanes @ 16 GHz / 8.125 bits per byte x 0.76 = 23.95 GB/second

23,950 MB/second / 8 ~= DDR4-3000

Conclusion:
A PCIe 4.0 NVMe RAID controller with four U.2 ports in RAID-0 mode
should approximate the overall throughput of DDR4-3000 DRAM
particularly when advanced memory technologies like Intel's Optane
populate the storage devices. At PCIe 3.0, project DDR3-1500,
which is very close to DDR3-1600 performance.


Here's one source of data on raw SATA overhead:

http://www.thessdreview.com/our-rev...ing-3gbs-recorded-with-8-crucial-c400-ssds/3/

Using a cheap Highpoint 2720SGL and 8 x Crucial C400 SSDs,
best case READ was 3,041 MB/second.

That add-on card is PCIe 2.0 with an x8 edge connector,
and the 8b/10b legacy frame (10 bits per byte):

x8 @ 5 GHz / 10 = 4.0 GB/second raw bandwidth upstream

Measured 3.041 / 4.0 = 0.760

Hopefully, the higher protocol efficiency of NVMe
will improve on that raw overhead quite a lot.


MRFS
 
Last edited by a moderator:

Patriot

Moderator
Apr 18, 2011
1,450
789
113
Just because the pcie bandwidth is there, doesn't mean the nand is.
Samsung has already shown a drive that can do 5500MB/sec for sequential reads and 1800MB/s for writes.

That's cool... that is 300MB/s more than my m.2 sm951.

for reference, ddr4 2133 in a 2p quad channel setup is 99GB/s or 12.4GB/s read and write.
DDR4 3000 is 17GB/s per channel.

I frankly don't think the nand is there... so, if scaling was 100% (which it won't be) and you had enough money for 8 of those ridiculous nvme drives. You could get to 1/10th ddr4 performance for writes and ~half for reads.

At this point, I don't know why anyone would be using raid... it counteracts the point of these drives which is low latency.
 
Jun 24, 2015
140
13
18
75
> I frankly don't think the nand is there

I agree!

How would your analysis above change,
if you factored in the specs claimed for Intel's Optane (3D XPoint)?

The jump from PCIe 3.0's 8G clock to PCIe 4.0's 16G clock
also speaks volumes, imho.

Is it possible that working Optane will need more raw bandwidth
than one PCIe 3.0 lane can provide upstream (8G / 8.125 bits per byte)?

If I had to guess, I suspect that Intel did their own internal analysis
and came to the same conclusion as you did:

THE NAND IS JUST NOT THERE!


MRFS
 
Jun 24, 2015
140
13
18
75
Allow me to take a stab at estimating the
raw upstream bandwidth required by Intel's Optane.

Please understand that this is a very rough estimate.

Here's an apples-to-oranges comparison
(using an IOPS measurement to predict READ speeds):

x4 PCIe 3.0 @ 8G / 8.125 = 3.94 MB/sec raw bandwidth

DC P3700 Sequential READ: 2,800 MB/second (per Intel's specs: x4 edge connector)

Intel® SSD DC P3700 Series Specifications

Controller Efficiency: 2.80 / 3.94 = 0.710

Ratio of Optane / DC P3700 = 7.23X (IOPS performance, per Intel press conference)

2,800 MB/sec x 7.23 = 20.24 GB/second
(note that we are using the IOPS ratio
to predict sequential READ speed, however)

Even though this is an apples-to-oranges comparison,
the preliminary calculation above seems to indicate
that Optane's raw bandwidth exceeds the raw bandwidth
of a PCIe 3.0 x16 edge connector ~= 16GB/sec READs
but NOT the raw bandwidth
of a PCIe 4.0 x16 edge connector ~= 32GB/sec READs

p.s. I'm looking for an Optane sequential READ measurement:
I'll report back if/when I find one, which may be difficult
at this point in time e.g.:

SSDs based on Intel's 3D XPoint technology will arrive next year under the 'Optane' brand

"With a read queue depth of one, the 3D XPoint-based drive
scored 76,600 IOPS compared to the 10,600 IOPS turned in by NAND-based SSD.
Unfortunately, that was the extent of Intel's public demo."



MRFS
 
Jun 24, 2015
140
13
18
75
From here:
HP Z Turbo Drive Quad Pro | HP® Official Site
get sequential (READ) performance up to 9.0GB/s:

"Sequential read performance tested with Iometer 1.1.0 with 12 workers, queue depth of 128, file size of 128K."


from here:
HP Z Turbo Drive Quad Pro (Data sheet/4AA6-1667ENW.pdf)
get PCI Express Gen3 x16 -- but Read Bandwidth n/a


From those numbers, we can derive an estimated raw controller overhead:

x16 lanes @ 8 GHz / 8.125 bits per byte = 15.75 GB/sec MAX HEADROOM

sequential performance = 9.0 GB/second (see above)

raw controller overhead =

9.0 GB per sec MAX READ speed / 15.75 GB per sec MAX HEADROOM = 0.571

Here are more detailed specs:
HP Z Turbo Quad Pro (QuickSpecs/c04798669.pdf)

Note: There are restrictions to using RAID with boot volumes.

One of the M.2 SSD modules on the card can be used as a Boot device, and the other devices can be used as Data storage.
Alternatively, all of the M.2 SSD modules can be used as Data devices.



Sounds like software RAID is the only RAID option:
booting is supported from one JBOD M.2 SSD.

Let's be on the lookout for technical reviews
that should answer several of these questions.


MRFS
 
Jun 24, 2015
140
13
18
75
YES, IT'S THE NAND!!

M.2 PCIe SSD Modules
NAND Type: MLC
Read Bandwidth (128KB): 2,150 MB/s

x4 @ 8G / 8.125 = 3.94 GB/second

raw overhead = 2.150 / 3.94 = 0.546

But, we can't reach 9.0 GB/second with
four of those SSDs in RAID-0,
even with perfect scaling ...

Note that, with perfect scaling in RAID-0 mode,
4 x 2,150 = 8,600 MB/s NOT 9.0 MB/s as claimed
in HP's specs:

HP Z Turbo Drive Quad Pro (Data sheet/4AA6-1667ENW.pdf)

"Drive your workstation workflows at blazing fast speeds 16X faster than a SATA SSD
with a solution that delivers sequential performance up to 9.0GB. fn2"
Footnote 2:
"Sequential read performance Tested with Iometer 1.1.0
with 12 workers, queue depth of 128, file size of 128K."



AGAIN, IT'S THE NAND + a little "fudging" in HP's advertising
(maybe they just "rounded up"?)


MRFS
 
Last edited:

ATS

Member
Mar 9, 2015
96
32
18
48
Looks somewhat like the Dell card I got. Fan + 4x m.2 slots. Much to test, little time.

The big question for me is how are they dealing with PCIe bifurcation.
Assuming we are talking about E5/E7s then the actual CPUs support full 9x4 support. The more pressing question is one of motherboard configuration (clks et al) and bios. So, I would assume they rely on the underlying system to handle the x4 support.
 
Jun 24, 2015
140
13
18
75
So, replace the MLC Nand Flash with
something like Intel's Optane (3D XPoint).

Now, let's project probable effects
from four Optane M.2 SSDs mounted on
an add-on card with x16 edge connector,
and configured in a RAID-0 array for
maximum READ speed:

Assume a range of controller efficiencies
between a minimum of 500/600 and maximum of 550/600
(from lots of empirical experiences with Nand Flash SSDs):

x4 @ 8G / 8.125 = 3.94 GB/s for one M.2

x16 @ 8G / 8.125 = 15.75 GB/s for four M.2 (perfect scaling)

15.75 GB/s x (5/6) = 13.125 GB/s

15.75 GB/s x (55/60) = 14.438 GB/s

Now, calculate comparable DDR3 and DDR4 bandwidths:

13,125 MB/s / 8 ~= DDR3-1640

14,438 MB/s / 8 ~= DDR3-1804

And, finally, double the upstream bandwidth
for PCIe 4.0's 16 GHz clock:

26,250 MB/s / 8 ~= DDR4-3281

28,876 MB/s / 8 ~= DDR4-3609

Thus, with the latest HP and Dell designs,
using four M.2 NVMe SSDs mounted on an
add-on card with x16 edge connector,
Optane should be fast enough to shift the
bottleneck to the upstream x16 edge connector.

Even with PCIe 3.0's 8 GHz clock rate,
a RAID-0 array with four M.2 Optane SSDs mounted on
an add-on card with an x16 edge connector
is predicted to achieve comparable throughput
ranging between DDR3-1600 and DDR3-1800.

With PCIe 4.0's 16 GHz clock rate,
a RAID-0 array with four M.2 Optane SSDs mounted on
an add-on card with an x16 edge connector
is predicted to achieve comparable throughput
ranging between DDR4-3200 and DDR4-3600
(exactly twice the prediction for PCIe 3.0 chipsets).

MRFS
 
Jun 24, 2015
140
13
18
75
Forgive me if this suggestion belabors the obvious ...

This Icy Dock enclosure gave me an idea:

ICY DOCK Black Vortex MB074SP-B 3.5" Black IDE / SAS / SATA HDD 4-in-3 Module Metal Open Frame Cooler Cage w/ 120 mm blue LED fan - Newegg.com

If we leave the back completely "open",
we can then connect up to four U.2 cables
to as many 2.5" NVMe SSDs.

The latter enclosure uses 3 x 5.25" drive bays, however
(roughly 40mm per 5.25" drive bay x 3 => 120mm fan).


This next 5.25" enclosure supports four SSDs each up to a Z height of 15mm
in a single 5.25" drive bay:

ICY DOCK MB994SP-4S Full Metal 4 x 2.5" Hard Drive in 1 x 5.25" bay SAS / SATA 6Gb Hot Swap Backplane Cage - Newegg.com

... but it needs to be modified:

the backplane would need to be removed,
to allow direct-connect cables (as above);
and, up to 3 x 40mm fans should be mounted at the front,
to provide adequate cooling e.g.:

"E-buy World" EverCool EC-HK-3F-BK Hard Disk Drive 5.25 inch Bay 3 Fan Case HDD Cooler Black - Newegg.com

Clearly, these 3 fans would prevent
such a cheap enclosure from supporting
hot-swapped NVMe SSDs.

But, this idea does provide an inexpensive
5.25" enclosure for 4 x 2.5" NVMe SSDs
cabled directly to the host controller(s).


MRFS
 

Wondersausage

New Member
Dec 22, 2015
10
4
3
66
Seems to me that it's very unlikely to be BIOS-locked in the sense that it actually looks for those specific HP workstations. More likely it simply requires PCIe lane allocation BIOS support that doesn't exist in other systems (yet). I know some folks at HP that I can ask.

On the previous single-M.2 card HP offered called the HP Z Turbo Drive they used Samsung XP941. The pics of the new 4-slot unit are too low res to see the exact part numbers (to my eyes) but the labels are white so they aren't 950 Pro, they are surely XP941 or XM951. In any case, there are no other suppliers of M.2 NVMe modules besides Samsung at the moment. The problem is that Samsung uses 512MB volatile cache which in my experience will frequently cause an array to fail on a system crash (I've had that happen half a dozen times with 850 Pro SATA using iRST, whereas the Intel 730 array on the same system never failed under the same circumstances). This even happens with RST disabling the cache, which I presume doesn't actually disable it (we know that LSI auto disables the cache on Samsung SATA devices for this same reason).

It's interesting that only the first M.2 device is bootable. The Intel DC P3608 is actually two x4 SSDs on one x8 card and appears as separate volumes which iRSTe (Intel says only iRSTe version on selected systems) can combine into a single volume which is presumably bootable (at least they don't say it isn't bootable). Why would there be any difference between aggregating 2 and 4 devices in iRSTe?

If only Asus or ASRock would get on the BIOS bandwagon and provide support for this sort of card on their PLX X99 boards! I have an X99-E WS here that's currently running a DC P3500 which would love one of those 4-M.2 solutions (as soon as someone ships an M.2 NVMe device without a volatile cache).
 

amarshonarbangla

New Member
Jan 12, 2016
14
0
1
True. It could be BIOS-specific so that it only works in certain HP boards. If PCIe bifurcation is the issue and not BIOS lock, then one of those GPU boards that splits 4 x16 paths into 8 x16 slots by using four PLX chips could do it perhaps? But then again they probably just BIOS locked it to specific machines like Chuntzu said and called it a day.

UPDATE: Looks like they did:
"Thanks to a BIOS lock, the device is supported only on the HP Z440, Z640, and Z840 Workstations"
HP Updates Z Turbo SSDs, Reveals New Z Turbo Quad Pro (Updated)
That sucks...
 
Jun 24, 2015
140
13
18
75
ROG MAXIMUS VIII HERO ALPHA - Overview

[begin quote]

LIGHTNING-FAST STORAGE
SPEED WITH NATIVE M.2/U.2
NVMe PCIe RAID SUPPORT

Don't want [to] be limited by the read/write speeds of a single
storage device? Then join the fast lane by building multi-
device RAID array with Maximus VIII Hero Alpha's two U.2
and one M.2 onboard connectors -- and enjoy PCIe data
transfers at lightning speed. Our lab guys put both through
their paces, achieving write speeds of 3,386 MB/s and
read operations of an incredible 3,575 MB/s.

[end quote]

Because of those numbers, we infer that the 2 x U.2 ports
are downstream of the DMI 3.0 link.