NVMe: 2.5" SFF drives working in a normal desktop

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
I'd just like a place to get the cables from :) If you wind up with any extra you're not needing @Patrick I could use 2 :D :D
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,905
4,866
113
I need more cables also. Did you get the $250 cards with the PLX chip?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
I got the 2 cheaper cards literally minutes before you updated/posted.

We'll see if they work, if not I'll return and get the higher $$ ones. My motherboards are SM and the part is SM so I'm hoping it works :)

Just need to get a cable to test them, or cash in some amazon points for a 750 w/cable :D
 

Chuntzu

Active Member
Jun 30, 2013
383
98
28
Updated the first post. The more expensive version uses a PLX chip which helps with bifurcation down to x4. Some platforms (many of the ones tested) have this already. The other thing is that with only 1 cable could only connect 1 drive so did not see the issue with the bifurcation on the lower cost card (R).

Hopefully will have some more info to share later today.
Just to clarify, since my brain is failing me, the cheaper card has been working fine with one or two drives in some or most of your systems despite the fact that it does not have the plx chip on it? The plx version is needed if the motherboard does not already support bifurcation? Not sure if this will be easy to answer but is there a way to verify if the motherboard supports the bifurcation? I remember trying to figure this out a very long long time ago and struggled with finding an answer. Thank you for clarifying/walking me through the explanation.
 

neo

Well-Known Member
Mar 18, 2015
672
360
63
I'd just like a place to get the cables from
I am not sure if you saw my post, but I posted a link in #13 - yet it's still expensive.

Just to clarify, since my brain is failing me, the cheaper card has been working fine with one or two drives in some or most of your systems despite the fact that it does not have the plx chip on it? The plx version is needed if the motherboard does not already support bifurcation? Not sure if this will be easy to answer but is there a way to verify if the motherboard supports the bifurcation? I remember trying to figure this out a very long long time ago and struggled with finding an answer. Thank you for clarifying/walking me through the explanation.
You can have a look at your motherboard's manual if there is any mention of a "PCIe switch" or a glance at your motherboard.

Here is a video that breaks it down to layman's terms: Configure PCI-Express* Lanes for Simultaneous Applications
 
  • Like
Reactions: T_Minus

Patrick

Administrator
Staff member
Dec 21, 2010
11,905
4,866
113
Just to clarify, since my brain is failing me, the cheaper card has been working fine with one or two drives in some or most of your systems despite the fact that it does not have the plx chip on it? The plx version is needed if the motherboard does not already support bifurcation? Not sure if this will be easy to answer but is there a way to verify if the motherboard supports the bifurcation? I remember trying to figure this out a very long long time ago and struggled with finding an answer. Thank you for clarifying/walking me through the explanation.
So the Xeon D seemed to work. Other boards with PLX chips onboard seemed to have worked also. Note this is one drive so not 2.

I think the real answer is in many systems you need the PLX card.

I should have some of the differences between the two cards clearer tomorrow.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
So the Xeon D seemed to work. Other boards with PLX chips onboard seemed to have worked also. Note this is one drive so not 2.

I think the real answer is in many systems you need the PLX card.

I should have some of the differences between the two cards clearer tomorrow.
Any idea why we don't have access to a list of SuperMicro boards this will work with?
It's their item so I'd hope we can get definite answers from them.

I'm trying to recall, but I think some of their boards/chassis have this item listed as "OPTIONAL" at the very bottom, I however don't recall 100% or which board it was.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
From SuperMicro

"Bifurcation in this case means to split up a PCIe port into multiple ports (i.e. make 2 PCIe x8 ports from 1 PCIe x16 port). We only support this on our UIO and WIO motherboards. Take a low at our X9SPU, X9DBU, X9SRW and X9DRW series motherboards."

Posted: 03/14/14 By SM on their Q&A site.

Question
Does motherboard X10SLV-Q support bifurcation?
Answer
Motherboard X10SLV-Q does not support bifurcation. You can only use 1 device on that PCIe slot.

Posted: 02/21/14
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
Does that mean we can use the card for 1 drive only though?

@Patrick did you test with 2x 750s or just 1 on the XeonD?
 

neo

Well-Known Member
Mar 18, 2015
672
360
63
I'm not sure how up to date that Q&A is but my X10SLR-F has PCIe switching capabilities.
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
I HAVE MY FINGERS CROSSED THEN!

I really hope it works in my E5 v1/v2 builds, but am more thinking it will work in the E5 v3 builds.

My plan is to use 1 at home in my 2P 14C v3 and compare performance to a E5-1660 v3 then migrate it to home/dev/non-production analysis and then use 1 in production with a E5 v2 10C DB server.

I'm damn excited to test this, it will make my analysis soooo much faster :) Opening new doors here with these drives is a true reality.

I'm also excited to test 2 of them in RAID0 :-X

Now... to order the 2 cables for $70 each or order 1x750 to get me to July, LOL!! The 750 likely will go in ESXI home box.
 

neo

Well-Known Member
Mar 18, 2015
672
360
63
If you do decide to order those cables, keep in mind they do offer them in different lengths. I linked to the shortest ones - 0.5M
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
With $150 in amazon points and $80 for a cable I'm >50% of a 750 so I may just grab 1o f those from TigerDirect (I'm in CA, so, yay).

I just may have to take some points from my starwood hotel and tell the wife we can't stay in hawaii as long as we though... haha... that would go over well :D
 

tigweld0101

Active Member
Apr 18, 2015
107
27
28
53
That ASUS workstation you have in FS is now a prime candidate for this. IIRC those mobos use PLX chips to get all those PCIe slots active right?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,905
4,866
113
Hot off the press: here are the official/ unofficial AOC specs on the two cards.

This is what I think I am going to try next:

Supermicro AOC-SLG3-2E4 Specs - 800.JPG

This is the one I have been using with some mixed results:

Supermicro AOC-SLG3-2E4R Specs - 800.jpg

Will keep digging and will order one of the 2E4's for when I get back.
 
  • Like
Reactions: eva2000 and T_Minus

ATS

Member
Mar 9, 2015
96
32
18
44
Anyone know what the non-R version of the card does that's so great for $100 more?
Non-R has a PLX 1x8 -> 2x4 PCIe 3.0 switch on it. For most platforms this is going to be required to support 2x4 devices in a x8 slot. Its overall a somewhat complicated issue getting into actual chipset/processor support along with bios configuration support as well as what else is already configured in the system. The PCIe complex of the E5/7 processors support a limited number of PCIe masters/initiators. Generally this number is significantly less than #num_pcie_lanes/4.

For the most part, the R version should work if only 1 NVMe drive is connected as this is just like plugging a x4 card in a x8 slot. The complications with the R version will only start to show when you try to connect 2 NVMe drives and the combination of processor and bios aren't configured to support 2 devices at the root PCIe complex. And in general, this configuration isn't exactly dynamic on the CPU/bios side. It generally needs to be statically configured at boot time to allocate the various PCIe masters on the CPU to the various PCIe lanes.

As to why we don't have a qual list from Supermicro, probably because its a bit complicated. When in doubt, get the PLX version (and why of why didn't they use a x8->4x4 version of the PLX chip?????) and it will always work.

Still annoyed they didn't allow for over-subscription in the PLX card. There's very few workloads nor very few NVMe drives that can hit peak performance in linear streams and over-subscription would allow much better performance in real world workloads with less PCIe host lanes. Hopefully someone is coming out with a x8->4x4 or x16->8/12x4.