Intel NVMe/SAS3 2U Disk Enclosure w/ Oculink interface + SuperMicro AOC-SLG3-4E2P

Discussion in 'Chassis and Enclosures' started by larryexchange, Jul 10, 2018.

  1. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    Cable or NVMe -> U.2 adapter issue. But I have 8 adapters and the odds that every.single.one is bad seems low. Easier to believe a wiring issue.

    Here's the link to the the 8643 cables I picked up. -> https://www.amazon.com/gp/product/B0773R3FB6
    Here's the link to the version of the adapter IIRC is linked in this thread somewhere. -> https://www.amazon.com/gp/product/B01HR0PKJQ
    Here's the other variant. -> https://www.amazon.com/gp/product/B07FMQPPRF

    While there's no mention of the wiring spec those employ _there_, I found them w/some search effort that got me to this -> https://www.amazon.com/Micro-SATA-Cables-SFF-8643-Connector/dp/B0773R3L5V Which if you look at the description you see mention of sff9402 and refclk culmination at the very bottom with:
    'For use with our High Speed Cable PN # RHS36-2583 Adapter Micro SATA Cables PN # SFF-1332-U2X4'

    A search for that on pn linked back to the cable.

    Correct; Intel plx card -> u.2 adapter.

    Sure. That would be grand! Not at all cheap though. I ordered only 2 of these -> https://www.amazon.com/gp/product/B07CVMX4GV just to try since they aren't cheap.

    In an attempt to eliminate some other variables, I ordered some more variants of adapters and cables. e.g.:
    nvme -> u.2 adapter : https://www.amazon.com/gp/product/B01HOWXAPI
    'shotgun franken-kit': https://www.amazon.com/gp/product/B01LZHUJ8W
    pcie x4 -> 8643: https://www.amazon.com/gp/product/B01LY9HRDB
    nvme -> 8643 adapter: https://www.amazon.com/gp/product/B01LZREOF1

    Those were _just_ delivered. I'll see if I can get _anything_ to work. The idea is to, at least, see if the 960/970 pros even work in the u.2 adapters using 'regular' pcie adapter-fu. If that works, then the sleds are validated and it confirms cabling.

    I ended up having to email my question/request for clarity from icydoc. Their response thus far is:
    "Thank you for your inquiry regarding the MB699VP-B and the card you are using. I have contacted engineering regarding your request and they should be able to provide additional information on the pinout we use on the MB699VP-B and the SFF-9402 specification.
    I will let you know when they get back on this request."

    Thanks for your assistance!
     
    #41
  2. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    Some tests w/the new stuff appear to validate that the icydoc uses the legacy wiring. So there's that. However, I'm not certain why the oculink (8611) -> 8643 cables don't work w/the pci-e switch. The cables are 8611 (host) and 8643 (target) which should be correct. They are these -> https://www.amazon.com/gp/product/B07CJYNYQK
     
    #42
    T_Minus likes this.
  3. Christopher Lightfoot

    Joined:
    Jun 14, 2019
    Messages:
    2
    Likes Received:
    0
    Hi;
    So I also took the plunge and ordered an AXXP3SWX08080 from scan:
    https://www.scan.co.uk/products/8-port-intel-pcie-gen3-x8-switch-aic-connects-8x-nvme-drives

    And these 8611 to 8643 adapters:
    https://www.amazon.co.uk/gp/product/B07PSDDGNY/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

    In the description of the cables they also say:
    • This OCuLink cable can be used to connect internal devices, e.g. a backplane with SFF-8643 connector to a controller with OCuLink SFF-8611 connector.
    But I'm having what sounds like the same issue as NaCl where all I see is a memory controller showing up and nothing else -- so i'm Interested in finding out what you got working in the end!

    My intention is to run 4 4x PCIe cards from a single slot (with over subscription) so I was initially trying to use these PCIe x4 adapters:
    https://www.amazon.co.uk/gp/product/B0733G42MF/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1

    But when that failed; reverted to these M.2 adapters (still using 8643) that I had to at least prove I could use this card with NVME drives using a Samsung PRO NVME:
    https://www.amazon.co.uk/gp/product/B071FJLRMJ/ref=ppx_yo_dt_b_asin_title_o08_s01?ie=UTF8&psc=1

    (I haven't tried a 8611 to 8639 on this board yet since I don't have those... £30 a cable is getting painful to test this stuff out... )

    And still nothing.

    I've proven already that these adaptors work fine on the motherboard directly (or through other adaptors); but no luck with the intel card.

    I'm using an ASUS Z10PH:
    Z10PH-D16 | Server & Workstation | ASUS United Kingdom

    With a pair of E5-2687W (which have enough lanes etc)

    Some things I've found which may help others:

    The PCIe listing shows "PMC-Sierra Memory Controller"

    PMC-Sierra do their own PCIe switch AICs; which are different to PLX/PEX;
    There's a page here: https://pcisig.com/developers/integrators-list that lists _some_ of the products
    The only switch by PMC listed there is the:

    Company: PMC-Sierra
    Product Name: Switchtec™ PSX 96xG3 Storage Switch
    Identifier: PM8546A-FEI-P
    Revision: PCIe 3.0 at 8GT/s
    Max lane width tested: x16
    Function: 96-Lane, 48-Port Storage Switch
    Date added: Jan 15, 2016

    Which is different to the x8 card intel make.

    Has anyone taken the heatsink off of the Intel card yet to figure out which AIC they're using? I'm confident it's not PLX just from the PCIE liseting I see in ESXi but maybe that's "something else" and the PLX just isn't displaying anything...

    Also reading the PDF for the supermicro card:
    https://www.supermicro.com/manuals/other/AOC-SLG3-4E2P.pdf

    They clearly state:
    1. Using the above chart, verify that the cable provided is appropriate for the expansion card connectors and the backplane connectors. Note that the MiniSAS HD to OCuLink cables and the OCuLink to MiniSAS HD cables may physically look the same but are wired differently and may not work if the wrong cable is used. Verify that the appropriate cable was included. For example, if your expansion card has OCuLink connectors and is plugging into a MiniSAS HD backplane, a cable with a green marking at the OCuLink end must be used.
    So those 8643 cables have to be the right way around.
     
    #43
    Last edited: Jun 14, 2019
  4. Christopher Lightfoot

    Joined:
    Jun 14, 2019
    Messages:
    2
    Likes Received:
    0
    I got an RMA on the AXXP3SWX08080 so sending that back early next week;

    Found a supplier in the UK who can sell me an AOC-SLG3-4E2P:
    AOC-SLG3-4E2P | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

    So going to try that as a last attempt before I give up on this idea.

    I know there isn't exactly a "market" for what I want but it's technically what some motherboards do already (Asus z10pe, Supermicro x10d...) just split into cable-form and distributed across a few PCBs rather than all being on the motherboard;

    I really wish this wasn't so hard or so expensive. Who pays £1k for an add in card (Serial cables, One stop, etc) when you can get an alternate motherboard that does the same instead for half the price... they're truly gauging the server consumers in these prices. That or the limited market makes their own costs so high.
     
    #44
  5. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    Greetings,

    In the end the only thing I was able to get working was:
    0) The Intel plx card.
    1) The associated cable kit (Google Express)
    2) The associated enclosure (https://www.amazon.com/Intel-Corp-A2u8x25s3phs-Hot-swap-8x2-5inch/dp/B076PMCX8W)

    w/2 they claim to be 8611 straight-through type so _maybe_ (https://www.amazon.com/Dilinker-OCulink-SFF-8611-OCuLink-Cable/dp/B07H5GB231) would work just as well. Would _certainly_ be cheaper. All of the above is meant for replacement parts in one of their servers so lengths and whatnot are all 'just so'. That and its Intel. They never had a customer they wouldn't fleece/treat poorly.

    And wrt to the Broadcom 9405W-16i tri-mode SAS HBA, the _ONLY_ way I was able to get NVME joy out of that thing was using _THEIR_ special ($$$$) 'enabler' cables. (1m 2 SFF-8643 to 2 SFF-8643 NVMe Cable - 05-50061-00)

    I think there's too much 'wild west' wrt to NVME and 8643 wiring 'standards' atm.

    I have been looking into doing something similar wrt trying to get pci-e fanout from a plx card
    in an attempt at creating a 'portable' nas/plex server solution. Base would be a Latte Panda Alpha which has 1 m.2 pcie connector. Using an m.2 -> pci-e x4 plus flexible extension/'riser'. I grabbed 2 different x16 variants because I couldn't find an x4 -> x4 that didn't look like complete garbage wrt to shielding/soldering quality. Female extender would hold the pcie plx oculink card. Oculink -> SAS -> NVME tray -> m.2 to pci-e x4 -> another pcie plx oculink card. Rinse, lather, and repeat. Althought bw, especially from that 2nd plx card is going to be pretty anemic. At least by NVME standards anyway. (< 100-120MB/s after overhead and everything 'talking') But that should still be workable for travel plx 'box'. If 2TB NVME sticks were cheaper, with some quality trade-off, I could _just_ get my entire collection to fit.

    Various other parts per above:

    # Oculink -> SAS (w/NVME pins) (8611 -> 8639) cables
    https://www.amazon.com/DiLinKer-OCuLink-SFF-8611-SFF-8639-Power/dp/B07NY4S5CS

    # SAS/NVME -> m.2 NVME tray
    https://www.amazon.com/gp/product/B01HR0PKJQ

    # m.2 to pcie x4 adapter (2 pack)
    https://www.amazon.com/gp/product/B074Z5YKXJ

    # pci-e x16 - x16 extender/'riser' cables. (haven't tried either yet)
    0) https://www.amazon.com/gp/product/B07PJ8JSTJ
    1) https://www.amazon.com/gp/product/B07K6TXRK6

    _If_ there's no 'funny business' wrt to the Intel plx card w/some nasty fw mandating not only a backplane but an _Intel_ backplane, then the cabling/adapter chain should be sound.
     
    #45
  6. ullbeking

    ullbeking Active Member

    Joined:
    Jul 28, 2017
    Messages:
    368
    Likes Received:
    29
    Thank you for assembling this guide, @larryexchange ! I'm doing some similar things on an SC747 workstation using a AOC-SLG3-4E2P too, which I'm very excited about. I'm going to study your guides first because they're the best I've seen. I've researched the hell out of this, but still I wouldn't be surprised if I have missed something.

    I also have a couple of 1U servers that I'm configuring for colo and here I am using the quad-port M.2 carrier (AOC-SHG3-4M2P) for mounting 2x or 4x M.2 NVMe SSD's in a PCI-e slot, effectively turning these M.2 drives into a single AIC in each server. All the other storage, 4x LFF 3.5" front bays, is taken with high capacity disks. This also uses a PLX switch.

    BTW, how is the Z840 to use? Is it quiet? I want one because I am realizing the important of workstations now. And I looked for the 5.25" bays in the specs but couldn't find them. What are the Z840 chassis like; are they good at acoustic dampening while allowing good airflow?
     
    #46
    Last edited: Jun 21, 2019
  7. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    You definitely want these:
    Either of those x16 extender/'riser' cables work.

    Confirmed that this works. I wanted to upload a file but the forum 'upload' feature just keeps telling me an error occurred but doesn't state the nature of the error.
     
    #47
  8. ullbeking

    ullbeking Active Member

    Joined:
    Jul 28, 2017
    Messages:
    368
    Likes Received:
    29
    I would love to see what you were expecting going to upload, e.g., photos, dmesg output, &c.

    Maybe try uploading to imgur? A lot of people here do that.

    Also, please report the problem report to the administrators re: the upload problem. It would be good to keep the site working well.
     
    #48
  9. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    I made an imgur account. Here're the images -> The 'Franken_Beast'

    Simple bonnie++ run so I could see it do _something_:
    Code:
    # bonnie++ -d /mnt/franken_pool -s 16G -n 0 -m TEST -f -b -u root
    1.98,1.98,TEST,1,1561338786,16G,,8192,5,,,674066,73,405369,80,,,1049829,99,3744,242,,,,,,,,,,,,,,,,,,,31663us,103ms,,10638us,64800us,,,,,,
     
    #49
  10. Ouroboros

    Ouroboros New Member

    Joined:
    Jul 26, 2012
    Messages:
    20
    Likes Received:
    0
    Is this only 2 PCIe switches deep then?

    Also, have you ever found right angle OCulink to U.2 cables to fit to the intel PCIe switch card?

    Supermicro does now have a 8 OCulink card in x16 half height flavor (AOC-SLG3-8E2P) but getting one seems difficult...
     
    #50
  11. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    Greetings! Pardon my tardiness/dereliction in a timely response.

    Yes. I only have the one sff8611 -> sff8639 cable so couldn't load up the first pcie switch.

    I have not. I've periodically checked, but all I've been able to find are sff8611 -> sff8611 w/various combinations of 'right angle'-ness; perpendicular or parallel to the board. IIRC microsatacables.com had the best selection. It should be noted that I have not tried any of their cables yet.

    Shiny! I can't even see an option to _try_ to order one. Seems 'difficult', in this instance, approximates 'impossible'. :(

    What I'd truly _love_ to get is an sff8643 or sff8612 (the x8 80p variant) on a backplane in the same form factor as this intel 2.5" x8 enclosure w/a plx pcie switch and the sff8639 ports subbed out for m.2 nvme sockets. Something like CP067_CONCEPT PRODUCTS_ICY DOCK manufacturer Removable enclosure, Screwless hard drive enclosure, SAS SATA Mobile Rack, DVR Surveillance Recording, Video Audio Editing, SATA portable hard drive enclosure but w/the pcie switch functionality.

    But that's 'crazy talk'. :(
     
    #51
  12. Ouroboros

    Ouroboros New Member

    Joined:
    Jul 26, 2012
    Messages:
    20
    Likes Received:
    0
    I just want to confirm here, that you were unable to get the Intel AIC AXXP3SWX08080 to work with a SFF-8211 OCulink to SFF-8643 MiniSAS HD cable, particularly to connect to the IcyDock MB699VP-B? Did IcyDock support get back to you regarding why (legacy signaling?)

    The cable you mentioned
    https://www.amazon.com/gp/product/B07CJYNYQK
    may have been problematic, but are DiLinker cables more likely to succeed in connecting the Intel card to the IcyDock, such as this cable?
    https://www.amazon.com/dp/B07VT3YC9P
     
    #52
  13. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    Correct.

    Support got back to me that the IceDock uses legacy wiring, but I don't know that that is the reason the AIC AXXP3SWX08080 did not work w/it. I was originally trying to get the IcyDock to work w/a 9405W-16 and, after a bit of research found sff-9402 to be the requisite spec. Their support confirmed that the unit did not use the newer spec. I only tried going down the Intel AIC AXXP3SWX08080 oculink 'rabit hole' because, in my research trying to figure out why the 9405W wasn't working, I stumbled up on this forum thread and decided to acquire/try the Intel hba in lieu of the 9405W. As it happened, that didn't ever work. So I ultimately ended up also getting the 'mated' Intel u.2 enclosure. I've since, acquire another set.

    Here are the responses IcyDock sent me:
    "Thank you for your inquiry regarding the MB699VP-B and the card you are using. I have contacted engineering regarding your request and they should be able to provide additional information on the pinout we use on the MB699VP-B and the SFF-9402 specification.
    I will let you know when they get back on this request."

    and the follow-up:
    "Engineering has followed up with me regarding your inquiry on the MB699VP-B and SFF-9402. MB699VP-B miniSAS HD connectors are under legacy wiring."


    I'm not sure. Given that these others didn't and they claimed they should, I could only suggest 'rolling the dice' and ordering a set and trying it out if you already have the rest of the stuff. I ended up returning the IcyDock and those cables because I was still w/in my return window.

    For completeness here's the rest of the email chain I had w/IcyDock:

    ME: "Thank you very much! I think I confirmed this last night w/a series of 'franken-adapters' (pci-e x4 -> nvme -> u.2). On another note, could you provide me w/the proper oculink -> 8643 type cable that should work w/the icydoc? I have these -> https://www.amazon.com/gp/product/B07CJYNYQK w/an Intel AXXP3SWX08080 8PORT PCI Express GEN3 X8 Switch. The 8 ports are Oculink. Those cables are 8611 (male oculink) (host side) to 8643 (target side). However, the green LED in the icydoc is constantly illuminated and no NVMe devices therein are enumerated."

    ID: "Regarding cabling between the SFF-8643 connector in the rear of the MB699VP-B and a SFF-8611 OCUlink connector, the cable you are using should work."

    ID: "We would like to follow up with you if you were able to get our MB699VP-B to work with your system setup? If so, let us know what cabling was used to connect our product to which specific hardware. This information would be helpful to our engineers on compatible cabling and in providing information on how to connect our product with various hardware."

    ME:"Never w/Oculink cabling. I returned both of them because I could not get it to work. Was sad as I thought the units were very nice. But it wouldn't work so I had to make a command decision to go a different direction.
    The only way I was able to get the icydoc to work was w/the 'franken-adapter' I mentioned a few messages ago. That data was comprised of:
    0) https://www.amazon.com/Asus-Hyper-Expansion-Supports-Threadripper/dp/B07NQBQB6Z/ w/bifurcation AND/OR https://www.amazon.com/gp/product/B01MUB54TD w/out.
    1) https://www.amazon.com/U-2-M-2-Adapter-Interface-Drive/dp/B073WGN61Y in the m2. slot(s)
    2) https://www.amazon.com/CableCreation-Internal-Cable-SFF-8643-3-3FT/dp/B013G4FGSC

    To get the 9405W-16i to work, I had to purchase the 'liquid gold McSpecial' cables from Broadcom. That data was comprised of:
    0) Broadcom 9405W-16i
    1) Broadcom 'enabler' cable -> 1m 2 SFF-8643 to 2 SFF-8643 NVMe Cable - 05-50061-00
    2) https://www.amazon.com/gp/product/B076PMCX8W (because of the icydoc's 'legacy' wiring)

    ....
    PS I still have a number of the 'pieces parts' but if you wanted to send me a unit (happy to return when done) and whatever cabling you come up with for testing purposes, I'd be happy to help out. A _cheaper_ solution would be quite swell. While the Intel nvme cage works, it's meant for their rack mount case so it doesn't work in any 'normal' space. Just an option.

    PSS On another note, the m.2 nvme formfactor 2280 anyway ends up with a lot of wasted space in a u.2/nvme adapter. It'd be _super_..._SUPER_ swell if you could design an enclosure like your nvme unit, but specifically for m.2 format sticks. If you set them up w/the m2. connect on the backplane in a vertical fashion w/plastic guide rails such that the orientation/operation is akin to a normal thin/flat sticks of chewing gum are arranged. Even w/some generous spacing for airflow purposes, I'd expect that you could get some serious density in the same form factor as the current 4 drive nvme unit. (Standard 5.25" drive bay). At 2.3mm thick and an easy 5.25" horizontal space to work with, you could give each stick a bit over 11mm and get 12 sticks per enclosure. Then add occulink and/or minisas hd (sff-9402 wiring) on the back and you'd be golden. Even better would be to add a variant w/a plx chip capable of bandwidth management to all of the slots based on how many of the upstream links were configured. Then people that don't require each stick to have it's own x4 connection can do what they can and 'get by' w/the level of bandwidth their platform _can_ provide. Any chance of something like this? _I'd_ buy one!"

    ID:"Thank you on the follow up on our questions regarding connecting the units you had with your system. We are currently developing NVMe cages that would be using slimmer trays for the NVMe m.2 drives. One concept product we have in development that fits with the use of slimmer trays is the CP082 (CP082_CONCEPT PRODUCTS_ICY DOCK manufacturer Removable enclosure, Screwless hard drive enclosure, SAS SATA Mobile Rack, DVR Surveillance Recording, Video Audio Editing, SATA portable hard drive enclosure). For rear connections in our concept NVMe products, we can consider the usage of Oculink or miniSAS SFF-9402 wiring standard.

    As for helping to test our products, we appreciate the offer. For some of our concept products, there might be beta tests to check the usability of our products. If you have the hardware for testing along with the time for testing, we could consider you. We currently do not have one as of now, but if there is a request for beta testers for a concept product, you can be informed by signing up for our email newsletter. The newsletter is the best way to learn about new products, promotions, and news updates from ICY DOCK.

    If you haven't signed up and you are interested in signing up for our newsletter, simply reply to this email with a message that you want to sign up and I will sign you up. We promise not to spam you and we limit our emails as much as possible to give you only the most important information. In the event that you do not want to receive the newsletter, there is a 1-click unsubscribe on each newsletter. Thank you for your consideration!"

    ------
    As it turns out the CP081 is very close to what I was envisioning. If they shaved a few mm off the spacing I think they could get 12 sticks in a single 5.25" enclosure.

    Anyway, I hope that helps/provides the clarity you are seeking.
     
    #53
  14. Ouroboros

    Ouroboros New Member

    Joined:
    Jul 26, 2012
    Messages:
    20
    Likes Received:
    0
    The IcyDock cage being problematic may indicate it may need one of those "blessed" trimode cables in OCulink format, perhaps such as the Supermicro one

    https://www.amazon.com/OCULINK-V-1-0-SOURCE-TO/dp/B077BVPJGG

    Supermicro's 8x OCulink AOC card manual specifically says to make sure you get the OCulink to SFF-8643 cable with the OCulink end that is green wrapped (as a differentiation marker, though also the SFF-8643 end is usually white for NVMe use as well). That miiiiight be wired right.

    IcyDock at least has a trimode warning in the their FAQ as a PDF

    https://www.icydock.com/PDF/MB699VP-B_Tri-mode_compatability_announcement.pdf

    but that deals with RAID cards, not PCIe switch HBA's.
     
    #54
  15. NaCl

    NaCl New Member

    Joined:
    Dec 15, 2018
    Messages:
    18
    Likes Received:
    2
    The compatibility announcement they have there is per what I was just talking about. That _is_ the sff-9402 spec. I'd dare to assert that my inquiry probably helped push them into making that. The Broadcom 9405W-16 is a tri-mode hba; not raid.

    Whether or not the Intel board is expecting the same adherence, I haven't checked.
     
    #55
Similar Threads: Intel NVMe/SAS3
Forum Title Date
Chassis and Enclosures Intel P4000 chassis UPS question Oct 31, 2019
Chassis and Enclosures Intel P4308 + S2600CP4 barebones New-In-Box Jun 17, 2019
Chassis and Enclosures Intel P4000M series chassis divider Jan 14, 2019
Chassis and Enclosures Intel P4000 Chassis - 2.5 drive adapter air flow issues Sep 13, 2018
Chassis and Enclosures Blade chassis with blades for Intel C246 chipset Sep 7, 2018

Share This Page