adding nvme to X9 board via NVMe HBA (supermicro AOC-SLG3-4X4P )

Discussion in 'RAID Controllers and Host Bus Adapters' started by james23, Oct 2, 2019.

  1. james23

    james23 Active Member

    Joined:
    Nov 18, 2014
    Messages:
    401
    Likes Received:
    67
    so im kind of thinking out loud here, but i would really like to make use of some of the pcie lanes on my X9DR3-LN4F+ rev 1.20 , to run a few NVMe drives (u.2 2.5"). I know this board does not support pcie bifuracation, else this would be pretty easy!
    (ive read the threads where some people say that have got pcie bi-fur working on this baord, but the general consensus, and my own tests, is it does not work on this board).

    Id like to use one of my true wired x16 slots to run 4 or less NVMe drives-

    Thus ill need to use a NVMe card/hba with a pcie switch- Does anyone know if this card would work like that?

    AOC-SLG3-4X4P | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

    (~300$ on eaby: Supermicro AOC-SLG3-4X4P Low Profile JBOF Gen-3 external NVMe Host Bus Adapter | eBay )

    I would obviously need some kind of external NVMe enclosure (if those are even made?), or i could maybe do something sloppy like this cable/setup (but would also need a cable /convert the 4x ports on the AOC card to SFF-8643)
    Cable Kit for 2.5″ PCIe NVMe SSD Drives - CS Electronics
    vncmeCapture.JPG

    Ideally im trying to avoid the 16x pcie cards that allow you to put 2x or 4x nvme m.2 sticks on the card (like the asus card, w a pcie switch), this is bc id like to use u.2 nvme drives. I am also aware that in most cases the "white" nvme ports are a bit different than normal 12g SFF-8643 ports and cant be used inter-changeably (ie like the white nvme ports on my 2u supermicro backplane: BPN-SAS3-826EL1-N4 which has 4x nvme bays).
    upload_2019-10-2_16-17-53.png

    anyone have any input or ideas on how to get some u.2 NVMe bays connected to a X9 supermicro board that does not support pcie bifurcation?

    thanks
     
    #1
  2. azev

    azev Active Member

    Joined:
    Jan 18, 2013
    Messages:
    621
    Likes Received:
    158
    I've used X9 motherboard on a CSE-216 with BPN-SAS3-216EL1-N4. For controller card I've used AOC-SLG3-4E2P and with proper oculink cable I connected directly to the backplane. I am running 4 U.2 NVME drive in the chassis in this setup and it work just fine.

    If anything, the one cosmetic issue that I cant figure out is that the red light (usually mean bad) is always light up on all of the 4 nvme slots.
     
    #2
    james23 likes this.
  3. james23

    james23 Active Member

    Joined:
    Nov 18, 2014
    Messages:
    401
    Likes Received:
    67
    awesome info! thanks alot!
    3 questions for my use case:

    1- anyone know of a way to get occulink out of the chassis to go to a "nvme diskshelf"? (maybe this is a dead issue tho bc of nvme+cable length, but ie my server is a 4u with 24bay SAS BP, right atop this server, on my rack, i do have a *empty/no MB* 2u chasis with this backplane (4x nvme): BPN-SAS3-826EL1-N4 , is there anyway i can get nvme cableing from my 4u up to that 2u's nvme backplane?)

    2- it looks like the AOC-SLG3-4X4P i cited, is very similar (maybe same pcie sw chip) as your AOC-SLG3-4E2P , so maybe that would work for my external nvme disk shelf concept- can anyone comment on cabling for an external nvme disk shelf (ie going from the AOC-SLG3-4X4P up to my 2u chasis with its BPN-SAS3-826EL1-N4) ?

    (manual for the AOC-SLG3-4X4P https://www.supermicro.com/manuals/other/AOC-SLG3-4X4P.pdf )

    3- how has performance been with use of a pcie SW + 4x nvme drives? (ie how do you have disks configured / raid? and what OS?)

    thanks, your info is really helpful!
     
    #3
  4. azev

    azev Active Member

    Joined:
    Jan 18, 2013
    Messages:
    621
    Likes Received:
    158
    If you are hard set to using external nvme I would suggest maybe getting something like this. I personally never tried to connect nvme drive externally so I dont have much information to share. As for performance using AOC-SLG3-4E2P as controller and 4x nvme drive in ZFS configured as mirrored vdev, performance is great. I used optane aic in one of the PCI-E slot as ZIL and configured sync = always and get very good performance.
     
    #4
    james23 likes this.
  5. james23

    james23 Active Member

    Joined:
    Nov 18, 2014
    Messages:
    401
    Likes Received:
    67
    thanks for the reply / info- that unit is a great find!
    Does anyone know if that (highpoint-tech SSD6540 4x external nvme enclosure) would work with the AOC-SLG3-4X4P nvme hba?
    NVMe RAID 4-Bay Enclosures Supports 4x M.2 and 4x U.2 NVMe SSDs Category, Thunderboltâ„¢ 3 to NVMe SSD Adapter Support 4x M.2 NVMe SSDs Category - HighPoint Global Website

    only reason i ask is im not big on using their raid card (non hba) as i will be using this with ZFS / Freenas (so want an hba). (and 2nd issue is their included raid card does not have drivers for freeBSD it appears).

    to add some tadbit related info, i thought i could run 2x enterprise NVMe drives in a zfs mirror and NOT need a slog/zil for vmware NFS sharing (sync writes) - WRONG, you still need a zil to get stable/solid performance. anyone interested in more info on this specific point see my post here :
    nvme Mirror (or separate) = 100% busy / lag io? (bug maybe?)

    thanks

    edit; here is a review on that highpoint-tech unit:
    HighPoint SSD6540 Kit Review - Full Speed External 4x NVMe
     
    #5
  6. ullbeking

    ullbeking Active Member

    Joined:
    Jul 28, 2017
    Messages:
    368
    Likes Received:
    29
    Hi, why did you put this is in, leave SuperMicro in there, and then add a strikethrough? It's very curious and I'm interested in the motivation.
     
    #6
  7. klui

    klui Member

    Joined:
    Feb 3, 2019
    Messages:
    38
    Likes Received:
    16
    The SLG3-4X4P is designed to be used with Supermicro's JBOF chassis like the SSG-136R-N32JBF or SSG-136R-NEL32JBF. It also is meant to be installed on X11 and not older generations.
     
    #7
  8. Spartacus

    Spartacus Active Member

    Joined:
    May 27, 2019
    Messages:
    291
    Likes Received:
    83
    Maybe so, but shouldn't it be fine as long as its installed in a pcie 3.0 16x slot?
     
    #8
Similar Threads: adding nvme
Forum Title Date
RAID Controllers and Host Bus Adapters Adding 24th drive to my SC846 makes it throw errors, card or backplane? Yesterday at 7:14 AM
RAID Controllers and Host Bus Adapters Upgrading HBA card to sas 6Gbit and adding more storage Sep 25, 2019
RAID Controllers and Host Bus Adapters HGST 4u60 + Dell SC8000 adding drive issue Nov 12, 2017
RAID Controllers and Host Bus Adapters LSI 9266-8i problems after adding aditional drive Nov 17, 2013
RAID Controllers and Host Bus Adapters Adding SAS Expander in the future Feb 23, 2011

Share This Page