PCIe NVMe HBA FYI

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Interesting discussions here but to me it's not at all about maximum performance as to why you need 4 x NVMe or 8 or 12,16 etc, it's about being able to start with 2 x 1.6tb drives and then as you need more space add more drives, you may have a working set that is 20tb, well you won't get that in 1 or 2 drives, you need 8 for example.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Paul I feel we've provided solutions to your RAID0 of NVME, and rehash it over and over again. You're looking for a specific piece of hardware that can be accomplished already (because it's made) with that hardware from Intel, HP, etc... you just don't want those branded pieces. You can then get SM NVME HBA and run them in most any workstation and again have 2,4, 6 NVME drives in RAID,1, whatever you make-up via software be it Windows or ZFS or MD. Take it a step further and get a NVME backplane model chassis from SuperMIcro and run 24x NVME drives. There are MANY solutions to get the performance you discuss, but you are focused on this 5.25" device you want to patent and missing everything. If you have patent research you need to do and/or sell to your clients, lawyer friends, etc... then why are you not making these servers and selling them to them? Anyway, like the other thread I'll stop replying to you now ;) since we just go in circles.
 
  • Like
Reactions: pgh5278

edge

Active Member
Apr 22, 2013
203
71
28
Back to the point of just how much CPU it takes to consume the output of NVME drives. The current DL380 TPC-H benchmark configuration has 44-cores and they are able to consume the io of only 20 SAS SSD's on 5 P441 Smart Arrays, full disclosure report with all the gory details:

http://c970058.r58.cf2.rackcdn.com/...pe_proliant_dl380_gen9~fdr~2017-04-17~v02.pdf

Yeah, just 4 SSD's per controller - anymore and you're wasting money because the system is controller bound. Raid 0 because this is a cost sensitive test and that means uses the cheapest (i.e. smallest capacity drives) drives. You could double the drive capacity and use raid 1 and get the same performance reading off both disks - load time would be worse.

That's 44 2.2Ghz cores = less then 1/2 an SSD per core. When you actually have to do work on the data it really changes the balance of how much storage IO you need.

I have been waiting on NVME drives since I learned they were in dev 4 years ago. I can't wait to get a dual ported shelf in the HO to build a vm cluster on.
 
  • Like
Reactions: Evan

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Amazon is killing me on the cards.. At the time of order they showed 17 in stock, prime, get it tomorrow.. Because I ordered qty 8, it still hasn't shipped. Apparently, rather than drop ship the items individually they send them all to a central location and package them together? Odd.. Never run into this with them before.
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
So I made a real bonehead move..



Not sure how I'm going to address that.. I suppose I have 8 new PM953's for sale.

On a more positive note, the following mod transformed my GS7200 from a leer jet into a quiet church mouse:




Not sure it's going to provide the necessary cooling under load yet as I can't get the switch and T580 to talk after loading the T580 driver (they seem to at least recognize each other before the driver is loaded), but right now the switch is maintaining ~31 DegC.
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Can't 'mod it' to fit through ?
I looked at that.. I can, but it would require some serious hacking up of the trays. There are 8x 2.5" PM963's headed this way and I hate to hack up the trays now. For at least two more weeks I can only run 8 NVMe drives in this thing anyway.
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Can't 'mod it' to fit through ?
I think the best solution is going to be either to sell/trade them, or mount them in 3.5" bays in my SC847 with an HBA/direct connection.

But for now I couldn't resist... What do you think, Intel/Samsung approved?





Edit: Setting the fans to performance, and re-running @T12:
4KQ32 Read is @3122 MB/s & 762,246.3 IOPS
4KQ32 Write is @2832 MB/s & 691,477.1 IOPS
 
Last edited:
  • Like
Reactions: T_Minus

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Sure wish I could put the 2696's in there to see if the core count/clock speed/2400MHz mem had any effect. After 1/4 tube of thermal paste, however, I'm pretty convinced that the Intel board flat out isn't going to support the "OEM" processors..

If anyone has a tip/trick/or general knowledge that disputes this, do share.
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Nice numbers! Are those cards they're installed on supported on the BOTTOM of the chassis? If so I'd fab up a simple AL latch and call it good, for home rack at-least... :) I'd also re-run that test with more GB so you're not playing in cache/rAM only on those drives... figure out how much RAM cache they have and then * # of cards and then go 2GB above that and re-test :) That will be interesting to see if it changes much, it may even go up... testing 1GB spread over a # of cards isn't enough I don't think to really put them into 'real' test :D

Did you order your E5-2620 v3s ?
I have a pair I'm going to test replacing with QS CPU and then sell them if the QS work (Ill run QS) in the Intel Motherboard.
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Nice numbers! Are those cards they're installed on supported on the BOTTOM of the chassis? If so I'd fab up a simple AL latch and call it good, for home rack at-least... :) I'd also re-run that test with more GB so you're not playing in cache/rAM only on those drives... figure out how much RAM cache they have and then * # of cards and then go 2GB above that and re-test :) That will be interesting to see if it changes much, it may even go up... testing 1GB spread over a # of cards isn't enough I don't think to really put them into 'real' test :D

Did you order your E5-2620 v3s ?
I have a pair I'm going to test replacing with QS CPU and then sell them if the QS work (Ill run QS) in the Intel Motherboard.
I ran it @4GB earlier without much change.. how can I verify what it takes to out run the ram cache? Currently has 128GB ram installed.

Looking at the settings in the device manager write caching in NOT enabled...

On a different note, I've created a problem with the switch fans.. while converting the Sunon fans to the Noctua's I made the change one module at a time, checked the switch via terminal and the led's on back, and each time it was happy. What I did not do was reboot the switch....

Now having rebooted, I'm locked out due to "fan failure". I replaced 6x 4 wire PWM fans with 6x 3 wire non PWM fans... because it didn't lock me out as each group was swapped, I made the poor assumption that it would let me run like that as long as temps stayed in check.

Apparently it will not.

Does anyone have a technical paper explaining exactly how PMW works? I have to assume that the switch is determining fan function based on the PWM signal... so what's the best way to feed it false info? Is it simply a voltage signal (I'm assuming not.. hence the "pulse" in PWM".

I'd rather go back to AOL dialup than listen to the Sunon fans run 17K RPM.

Also considered:
-water cooling the damn thing (complicated/overkill/cool & silent)
-building/buying a new chassis that allows for larger fans and transplanting the internals (also a pita, current internals are well engineered with ducting/flow control... hate to tear that up).
-building a 2U cover for the existing base chassis... would have to rebuild internal ducting, but not too bad.
-Building a plug an play hot swap adapter for each of the three fan modules allowing for a larger fan. This appeals to my inner engineer and is probably the cleanest/coolest option... would take some fab work to do it right. I can get one of our packagers to cut it out if I put together the drawings, but still not a get it tonight option.
-Cutting a hole in the top cover just in front of the existing fan modules and mounting two PWM Noctua industrial 120mm 3K RPM fans... This is easy and definitely a get it done immediately option... but it feels akin to putting "spinners" on a 911 Turbo... such a nice switch; hate to rice it out. This would also be pulling intake air from the top of the rack which is not ideal... I suppose I could add some ducting to pull from the front of the rack, but now we've got spinners, neon under chassis lighting, and a wing out back :eek:
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Nice numbers! Are those cards they're installed on supported on the BOTTOM of the chassis? If so I'd fab up a simple AL latch and call it good, for home rack at-least... :) I'd also re-run that test with more GB so you're not playing in cache/rAM only on those drives... figure out how much RAM cache they have and then * # of cards and then go 2GB above that and re-test :) That will be interesting to see if it changes much, it may even go up... testing 1GB spread over a # of cards isn't enough I don't think to really put them into 'real' test :D

Did you order your E5-2620 v3s ?
I have a pair I'm going to test replacing with QS CPU and then sell them if the QS work (Ill run QS) in the Intel Motherboard.
Re-reading your post, you were referring to the cache built into the cards.. wasn't paying attention...
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
I was referencing the RAM/Cache on the drive itself not system, I'd check device specs.

You'll likely need to do 10GB+ for 8x of these...
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Yep! You got it, replied at same time.

I'm not sure if you're using ZFS or what for actual FS but that may play a part too depending on your config :)
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
At the moment it has WS16DC loaded since I'm familiar with it.. once I'm comfortable that everything is operational, I was planning to dive off into FreeNAS (never used it before).
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
AOC-SLG3-2E4
Super Micro Computer, Inc. - Products | Accessories | Add-on Cards | AOC-SLG3-2E4
Works great but only includes two connectors

AOC-SLG3-4E2P
Super Micro Computer, Inc. - Products | Accessories | Add-on Cards | AOC-SLG3-4E2P
Haven't actually purchased this one yet, but wiredzone can get them (even though inventory shows out of stock).
Occulink cables are available through:
High Speed Serial Cables, Adapters and Accessories from Serial Cables
Just found a similar one AOC-SLG3-4E4T - it claims to have support for all X11 boards ?

Super Micro Computer, Inc. - Products | Accessories | Add-on Cards | AOC-SLG3-2E4

That would be worth a test if one had occulink cables;)
Available freely it seems and quite cheap (€128)