adding nvme to X9 board via NVMe HBA (supermicro AOC-SLG3-4X4P )

james23

Active Member
Nov 18, 2014
409
75
28
48
so im kind of thinking out loud here, but i would really like to make use of some of the pcie lanes on my X9DR3-LN4F+ rev 1.20 , to run a few NVMe drives (u.2 2.5"). I know this board does not support pcie bifuracation, else this would be pretty easy!
(ive read the threads where some people say that have got pcie bi-fur working on this baord, but the general consensus, and my own tests, is it does not work on this board).

Id like to use one of my true wired x16 slots to run 4 or less NVMe drives-

Thus ill need to use a NVMe card/hba with a pcie switch- Does anyone know if this card would work like that?

AOC-SLG3-4X4P | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

(~300$ on eaby: Supermicro AOC-SLG3-4X4P Low Profile JBOF Gen-3 external NVMe Host Bus Adapter | eBay )

I would obviously need some kind of external NVMe enclosure (if those are even made?), or i could maybe do something sloppy like this cable/setup (but would also need a cable /convert the 4x ports on the AOC card to SFF-8643)
Cable Kit for 2.5″ PCIe NVMe SSD Drives - CS Electronics
vncmeCapture.JPG

Ideally im trying to avoid the 16x pcie cards that allow you to put 2x or 4x nvme m.2 sticks on the card (like the asus card, w a pcie switch), this is bc id like to use u.2 nvme drives. I am also aware that in most cases the "white" nvme ports are a bit different than normal 12g SFF-8643 ports and cant be used inter-changeably (ie like the white nvme ports on my 2u supermicro backplane: BPN-SAS3-826EL1-N4 which has 4x nvme bays).
upload_2019-10-2_16-17-53.png

anyone have any input or ideas on how to get some u.2 NVMe bays connected to a X9 supermicro board that does not support pcie bifurcation?

thanks
 

azev

Active Member
Jan 18, 2013
737
203
43
I've used X9 motherboard on a CSE-216 with BPN-SAS3-216EL1-N4. For controller card I've used AOC-SLG3-4E2P and with proper oculink cable I connected directly to the backplane. I am running 4 U.2 NVME drive in the chassis in this setup and it work just fine.

If anything, the one cosmetic issue that I cant figure out is that the red light (usually mean bad) is always light up on all of the 4 nvme slots.
 
  • Like
Reactions: james23

james23

Active Member
Nov 18, 2014
409
75
28
48
awesome info! thanks alot!
3 questions for my use case:

1- anyone know of a way to get occulink out of the chassis to go to a "nvme diskshelf"? (maybe this is a dead issue tho bc of nvme+cable length, but ie my server is a 4u with 24bay SAS BP, right atop this server, on my rack, i do have a *empty/no MB* 2u chasis with this backplane (4x nvme): BPN-SAS3-826EL1-N4 , is there anyway i can get nvme cableing from my 4u up to that 2u's nvme backplane?)

2- it looks like the AOC-SLG3-4X4P i cited, is very similar (maybe same pcie sw chip) as your AOC-SLG3-4E2P , so maybe that would work for my external nvme disk shelf concept- can anyone comment on cabling for an external nvme disk shelf (ie going from the AOC-SLG3-4X4P up to my 2u chasis with its BPN-SAS3-826EL1-N4) ?

(manual for the AOC-SLG3-4X4P https://www.supermicro.com/manuals/other/AOC-SLG3-4X4P.pdf )

3- how has performance been with use of a pcie SW + 4x nvme drives? (ie how do you have disks configured / raid? and what OS?)

thanks, your info is really helpful!
 

azev

Active Member
Jan 18, 2013
737
203
43
If you are hard set to using external nvme I would suggest maybe getting something like this. I personally never tried to connect nvme drive externally so I dont have much information to share. As for performance using AOC-SLG3-4E2P as controller and 4x nvme drive in ZFS configured as mirrored vdev, performance is great. I used optane aic in one of the PCI-E slot as ZIL and configured sync = always and get very good performance.
 
  • Like
Reactions: james23

james23

Active Member
Nov 18, 2014
409
75
28
48
thanks for the reply / info- that unit is a great find!
Does anyone know if that (highpoint-tech SSD6540 4x external nvme enclosure) would work with the AOC-SLG3-4X4P nvme hba?
NVMe RAID 4-Bay Enclosures Supports 4x M.2 and 4x U.2 NVMe SSDs Category, Thunderbolt™ 3 to NVMe SSD Adapter Support 4x M.2 NVMe SSDs Category - HighPoint Global Website

only reason i ask is im not big on using their raid card (non hba) as i will be using this with ZFS / Freenas (so want an hba). (and 2nd issue is their included raid card does not have drivers for freeBSD it appears).

to add some tadbit related info, i thought i could run 2x enterprise NVMe drives in a zfs mirror and NOT need a slog/zil for vmware NFS sharing (sync writes) - WRONG, you still need a zil to get stable/solid performance. anyone interested in more info on this specific point see my post here :
nvme Mirror (or separate) = 100% busy / lag io? (bug maybe?)

thanks

edit; here is a review on that highpoint-tech unit:
HighPoint SSD6540 Kit Review - Full Speed External 4x NVMe
 

ullbeking

Active Member
Jul 28, 2017
395
33
28
41
London
"Racks on Racks on Racks..."
Favorite Mfgs: Hitachi / HGST, Mikrotik, Ruckus, SuperMicro, Adaptec
(should also mention Intel, for CPUs and Expanders... the Intel p3700 - the best nvme drive in the history of the world :cool:)
Hi, why did you put this is in, leave SuperMicro in there, and then add a strikethrough? It's very curious and I'm interested in the motivation.
 

klui

Active Member
Feb 3, 2019
132
50
28
The SLG3-4X4P is designed to be used with Supermicro's JBOF chassis like the SSG-136R-N32JBF or SSG-136R-NEL32JBF. It also is meant to be installed on X11 and not older generations.
 

Spartacus

Active Member
May 27, 2019
600
219
43
Austin, TX
The SLG3-4X4P is designed to be used with Supermicro's JBOF chassis like the SSG-136R-N32JBF or SSG-136R-NEL32JBF. It also is meant to be installed on X11 and not older generations.
Maybe so, but shouldn't it be fine as long as its installed in a pcie 3.0 16x slot?