Intel NVMe/SAS3 2U Disk Enclosure w/ Oculink interface + SuperMicro AOC-SLG3-4E2P

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

NaCl

New Member
Dec 15, 2018
25
2
3
This sounds like a cable problem to me, but I'd expect the oculink->8639
cable to work fine. I've seen some suggestions that oculink->8643 cables are directional, but that wouldn't make any sense for a 8639 cable.
Cable or NVMe -> U.2 adapter issue. But I have 8 adapters and the odds that every.single.one is bad seems low. Easier to believe a wiring issue.

Here's the link to the the 8643 cables I picked up. -> https://www.amazon.com/gp/product/B0773R3FB6
Here's the link to the version of the adapter IIRC is linked in this thread somewhere. -> https://www.amazon.com/gp/product/B01HR0PKJQ
Here's the other variant. -> https://www.amazon.com/gp/product/B07FMQPPRF

While there's no mention of the wiring spec those employ _there_, I found them w/some search effort that got me to this -> https://www.amazon.com/Micro-SATA-Cables-SFF-8643-Connector/dp/B0773R3L5V Which if you look at the description you see mention of sff9402 and refclk culmination at the very bottom with:
'For use with our High Speed Cable PN # RHS36-2583 Adapter Micro SATA Cables PN # SFF-1332-U2X4'

A search for that on pn linked back to the cable.

I mean, it's not like you're going to have the other end of 8639 on a card and try to wire it to oculink on an enclosure, right? Also, there's no other use for an oculink to 8639 cable that I can think of, so I'd expect them all be wired right for NVMe use.
Correct; Intel plx card -> u.2 adapter.

Do you have a link to 8639 cable that you're using? Maybe I could give it a try here and see if it works with any of my devices.
Sure. That would be grand! Not at all cheap though. I ordered only 2 of these -> https://www.amazon.com/gp/product/B07CVMX4GV just to try since they aren't cheap.

In an attempt to eliminate some other variables, I ordered some more variants of adapters and cables. e.g.:
nvme -> u.2 adapter : https://www.amazon.com/gp/product/B01HOWXAPI
'shotgun franken-kit': https://www.amazon.com/gp/product/B01LZHUJ8W
pcie x4 -> 8643: https://www.amazon.com/gp/product/B01LY9HRDB
nvme -> 8643 adapter: https://www.amazon.com/gp/product/B01LZREOF1

Those were _just_ delivered. I'll see if I can get _anything_ to work. The idea is to, at least, see if the 960/970 pros even work in the u.2 adapters using 'regular' pcie adapter-fu. If that works, then the sleds are validated and it confirms cabling.

I ended up having to email my question/request for clarity from icydoc. Their response thus far is:
"Thank you for your inquiry regarding the MB699VP-B and the card you are using. I have contacted engineering regarding your request and they should be able to provide additional information on the pinout we use on the MB699VP-B and the SFF-9402 specification.
I will let you know when they get back on this request."

Thanks for your assistance!
 

NaCl

New Member
Dec 15, 2018
25
2
3
Some tests w/the new stuff appear to validate that the icydoc uses the legacy wiring. So there's that. However, I'm not certain why the oculink (8611) -> 8643 cables don't work w/the pci-e switch. The cables are 8611 (host) and 8643 (target) which should be correct. They are these -> https://www.amazon.com/gp/product/B07CJYNYQK
 
  • Like
Reactions: T_Minus
Jun 14, 2019
2
0
1
Hi;
So I also took the plunge and ordered an AXXP3SWX08080 from scan:
https://www.scan.co.uk/products/8-port-intel-pcie-gen3-x8-switch-aic-connects-8x-nvme-drives

And these 8611 to 8643 adapters:
https://www.amazon.co.uk/gp/product/B07PSDDGNY/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

In the description of the cables they also say:
  • This OCuLink cable can be used to connect internal devices, e.g. a backplane with SFF-8643 connector to a controller with OCuLink SFF-8611 connector.
But I'm having what sounds like the same issue as NaCl where all I see is a memory controller showing up and nothing else -- so i'm Interested in finding out what you got working in the end!

My intention is to run 4 4x PCIe cards from a single slot (with over subscription) so I was initially trying to use these PCIe x4 adapters:
https://www.amazon.co.uk/gp/product/B0733G42MF/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1

But when that failed; reverted to these M.2 adapters (still using 8643) that I had to at least prove I could use this card with NVME drives using a Samsung PRO NVME:
https://www.amazon.co.uk/gp/product/B071FJLRMJ/ref=ppx_yo_dt_b_asin_title_o08_s01?ie=UTF8&psc=1

(I haven't tried a 8611 to 8639 on this board yet since I don't have those... £30 a cable is getting painful to test this stuff out... )

And still nothing.

I've proven already that these adaptors work fine on the motherboard directly (or through other adaptors); but no luck with the intel card.

I'm using an ASUS Z10PH:
Z10PH-D16 | Server & Workstation | ASUS United Kingdom

With a pair of E5-2687W (which have enough lanes etc)

Some things I've found which may help others:

The PCIe listing shows "PMC-Sierra Memory Controller"

PMC-Sierra do their own PCIe switch AICs; which are different to PLX/PEX;
There's a page here: https://pcisig.com/developers/integrators-list that lists _some_ of the products
The only switch by PMC listed there is the:

Company: PMC-Sierra
Product Name: Switchtec™ PSX 96xG3 Storage Switch
Identifier: PM8546A-FEI-P
Revision: PCIe 3.0 at 8GT/s
Max lane width tested: x16
Function: 96-Lane, 48-Port Storage Switch
Date added: Jan 15, 2016

Which is different to the x8 card intel make.

Has anyone taken the heatsink off of the Intel card yet to figure out which AIC they're using? I'm confident it's not PLX just from the PCIE liseting I see in ESXi but maybe that's "something else" and the PLX just isn't displaying anything...

Also reading the PDF for the supermicro card:
https://www.supermicro.com/manuals/other/AOC-SLG3-4E2P.pdf

They clearly state:
  1. Using the above chart, verify that the cable provided is appropriate for the expansion card connectors and the backplane connectors. Note that the MiniSAS HD to OCuLink cables and the OCuLink to MiniSAS HD cables may physically look the same but are wired differently and may not work if the wrong cable is used. Verify that the appropriate cable was included. For example, if your expansion card has OCuLink connectors and is plugging into a MiniSAS HD backplane, a cable with a green marking at the OCuLink end must be used.
So those 8643 cables have to be the right way around.
 
Last edited:
Jun 14, 2019
2
0
1
I got an RMA on the AXXP3SWX08080 so sending that back early next week;

Found a supplier in the UK who can sell me an AOC-SLG3-4E2P:
AOC-SLG3-4E2P | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

So going to try that as a last attempt before I give up on this idea.

I know there isn't exactly a "market" for what I want but it's technically what some motherboards do already (Asus z10pe, Supermicro x10d...) just split into cable-form and distributed across a few PCBs rather than all being on the motherboard;

I really wish this wasn't so hard or so expensive. Who pays £1k for an add in card (Serial cables, One stop, etc) when you can get an alternate motherboard that does the same instead for half the price... they're truly gauging the server consumers in these prices. That or the limited market makes their own costs so high.
 

NaCl

New Member
Dec 15, 2018
25
2
3
Hi;
So I also took the plunge and ordered an AXXP3SWX08080 from scan:
https://www.scan.co.uk/products/8-port-intel-pcie-gen3-x8-switch-aic-connects-8x-nvme-drives

And these 8611 to 8643 adapters:
https://www.amazon.co.uk/gp/product/B07PSDDGNY/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

In the description of the cables they also say:
  • This OCuLink cable can be used to connect internal devices, e.g. a backplane with SFF-8643 connector to a controller with OCuLink SFF-8611 connector.
But I'm having what sounds like the same issue as NaCl where all I see is a memory controller showing up and nothing else -- so i'm Interested in finding out what you got working in the end!
Greetings,

In the end the only thing I was able to get working was:
0) The Intel plx card.
1) The associated cable kit (Google Express)
2) The associated enclosure (https://www.amazon.com/Intel-Corp-A2u8x25s3phs-Hot-swap-8x2-5inch/dp/B076PMCX8W)

w/2 they claim to be 8611 straight-through type so _maybe_ (https://www.amazon.com/Dilinker-OCulink-SFF-8611-OCuLink-Cable/dp/B07H5GB231) would work just as well. Would _certainly_ be cheaper. All of the above is meant for replacement parts in one of their servers so lengths and whatnot are all 'just so'. That and its Intel. They never had a customer they wouldn't fleece/treat poorly.

And wrt to the Broadcom 9405W-16i tri-mode SAS HBA, the _ONLY_ way I was able to get NVME joy out of that thing was using _THEIR_ special ($$$$) 'enabler' cables. (1m 2 SFF-8643 to 2 SFF-8643 NVMe Cable - 05-50061-00)

I think there's too much 'wild west' wrt to NVME and 8643 wiring 'standards' atm.

My intention is to run 4 4x PCIe cards from a single slot (with over subscription) so I was initially trying to use these PCIe x4 adapters:
https://www.amazon.co.uk/gp/product/B0733G42MF/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1

But when that failed; reverted to these M.2 adapters (still using 8643) that I had to at least prove I could use this card with NVME drives using a Samsung PRO NVME:
https://www.amazon.co.uk/gp/product/B071FJLRMJ/ref=ppx_yo_dt_b_asin_title_o08_s01?ie=UTF8&psc=1

(I haven't tried a 8611 to 8639 on this board yet since I don't have those... £30 a cable is getting painful to test this stuff out... )

And still nothing.

I've proven already that these adaptors work fine on the motherboard directly (or through other adaptors); but no luck with the intel card.

I'm using an ASUS Z10PH:
Z10PH-D16 | Server & Workstation | ASUS United Kingdom

With a pair of E5-2687W (which have enough lanes etc)

Some things I've found which may help others:

The PCIe listing shows "PMC-Sierra Memory Controller"

PMC-Sierra do their own PCIe switch AICs; which are different to PLX/PEX;
There's a page here: https://pcisig.com/developers/integrators-list that lists _some_ of the products
The only switch by PMC listed there is the:

Company: PMC-Sierra
Product Name: Switchtec™ PSX 96xG3 Storage Switch
Identifier: PM8546A-FEI-P
Revision: PCIe 3.0 at 8GT/s
Max lane width tested: x16
Function: 96-Lane, 48-Port Storage Switch
Date added: Jan 15, 2016

Which is different to the x8 card intel make.

Has anyone taken the heatsink off of the Intel card yet to figure out which AIC they're using? I'm confident it's not PLX just from the PCIE liseting I see in ESXi but maybe that's "something else" and the PLX just isn't displaying anything...

Also reading the PDF for the supermicro card:
https://www.supermicro.com/manuals/other/AOC-SLG3-4E2P.pdf

They clearly state:
  1. Using the above chart, verify that the cable provided is appropriate for the expansion card connectors and the backplane connectors. Note that the MiniSAS HD to OCuLink cables and the OCuLink to MiniSAS HD cables may physically look the same but are wired differently and may not work if the wrong cable is used. Verify that the appropriate cable was included. For example, if your expansion card has OCuLink connectors and is plugging into a MiniSAS HD backplane, a cable with a green marking at the OCuLink end must be used.
So those 8643 cables have to be the right way around.
I got an RMA on the AXXP3SWX08080 so sending that back early next week;

Found a supplier in the UK who can sell me an AOC-SLG3-4E2P:
AOC-SLG3-4E2P | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

So going to try that as a last attempt before I give up on this idea.

I know there isn't exactly a "market" for what I want but it's technically what some motherboards do already (Asus z10pe, Supermicro x10d...) just split into cable-form and distributed across a few PCBs rather than all being on the motherboard;

I really wish this wasn't so hard or so expensive. Who pays £1k for an add in card (Serial cables, One stop, etc) when you can get an alternate motherboard that does the same instead for half the price... they're truly gauging the server consumers in these prices. That or the limited market makes their own costs so high.
I have been looking into doing something similar wrt trying to get pci-e fanout from a plx card
in an attempt at creating a 'portable' nas/plex server solution. Base would be a Latte Panda Alpha which has 1 m.2 pcie connector. Using an m.2 -> pci-e x4 plus flexible extension/'riser'. I grabbed 2 different x16 variants because I couldn't find an x4 -> x4 that didn't look like complete garbage wrt to shielding/soldering quality. Female extender would hold the pcie plx oculink card. Oculink -> SAS -> NVME tray -> m.2 to pci-e x4 -> another pcie plx oculink card. Rinse, lather, and repeat. Althought bw, especially from that 2nd plx card is going to be pretty anemic. At least by NVME standards anyway. (< 100-120MB/s after overhead and everything 'talking') But that should still be workable for travel plx 'box'. If 2TB NVME sticks were cheaper, with some quality trade-off, I could _just_ get my entire collection to fit.

Various other parts per above:

# Oculink -> SAS (w/NVME pins) (8611 -> 8639) cables
https://www.amazon.com/DiLinKer-OCuLink-SFF-8611-SFF-8639-Power/dp/B07NY4S5CS

# SAS/NVME -> m.2 NVME tray
https://www.amazon.com/gp/product/B01HR0PKJQ

# m.2 to pcie x4 adapter (2 pack)
https://www.amazon.com/gp/product/B074Z5YKXJ

# pci-e x16 - x16 extender/'riser' cables. (haven't tried either yet)
0) https://www.amazon.com/gp/product/B07PJ8JSTJ
1) https://www.amazon.com/gp/product/B07K6TXRK6

_If_ there's no 'funny business' wrt to the Intel plx card w/some nasty fw mandating not only a backplane but an _Intel_ backplane, then the cabling/adapter chain should be sound.
 

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
Thank you for assembling this guide, @larryexchange ! I'm doing some similar things on an SC747 workstation using a AOC-SLG3-4E2P too, which I'm very excited about. I'm going to study your guides first because they're the best I've seen. I've researched the hell out of this, but still I wouldn't be surprised if I have missed something.

I also have a couple of 1U servers that I'm configuring for colo and here I am using the quad-port M.2 carrier (AOC-SHG3-4M2P) for mounting 2x or 4x M.2 NVMe SSD's in a PCI-e slot, effectively turning these M.2 drives into a single AIC in each server. All the other storage, 4x LFF 3.5" front bays, is taken with high capacity disks. This also uses a PLX switch.

BTW, how is the Z840 to use? Is it quiet? I want one because I am realizing the important of workstations now. And I looked for the 5.25" bays in the specs but couldn't find them. What are the Z840 chassis like; are they good at acoustic dampening while allowing good airflow?
 
Last edited:

NaCl

New Member
Dec 15, 2018
25
2
3
My intention is to run 4 4x PCIe cards from a single slot (with over subscription)...
...
(I haven't tried a 8611 to 8639 on this board yet since I don't have those... £30 a cable is getting painful to test this stuff out... )
You definitely want these:
# Oculink -> SAS (w/NVME pins) (8611 -> 8639) cables
https://www.amazon.com/DiLinKer-OCuLink-SFF-8611-SFF-8639-Power/dp/B07NY4S5CS

# SAS/NVME -> m.2 NVME tray
https://www.amazon.com/gp/product/B01HR0PKJQ

# m.2 to pcie x4 adapter (2 pack)
https://www.amazon.com/gp/product/B074Z5YKXJ

# pci-e x16 - x16 extender/'riser' cables. (haven't tried either yet)
0) https://www.amazon.com/gp/product/B07PJ8JSTJ
1) https://www.amazon.com/gp/product/B07K6TXRK6
Either of those x16 extender/'riser' cables work.

Base would be a Latte Panda Alpha which has 1 m.2 pcie connector. Using an m.2 -> pci-e x4 plus flexible extension/'riser'. I grabbed 2 different x16 variants because I couldn't find an x4 -> x4 that didn't look like complete garbage wrt to shielding/soldering quality. Female extender would hold the pcie plx oculink card. Oculink -> SAS -> NVME tray -> m.2 to pci-e x4 -> another pcie plx oculink card. Rinse, lather, and repeat.
Confirmed that this works. I wanted to upload a file but the forum 'upload' feature just keeps telling me an error occurred but doesn't state the nature of the error.
 

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
Confirmed that this works. I wanted to upload a file but the forum 'upload' feature just keeps telling me an error occurred but doesn't state the nature of the error.
I would love to see what you were expecting going to upload, e.g., photos, dmesg output, &c.

Maybe try uploading to imgur? A lot of people here do that.

Also, please report the problem report to the administrators re: the upload problem. It would be good to keep the site working well.
 

NaCl

New Member
Dec 15, 2018
25
2
3
I would love to see what you were expecting going to upload, e.g., photos, dmesg output, &c.

Maybe try uploading to imgur? A lot of people here do that.

Also, please report the problem report to the administrators re: the upload problem. It would be good to keep the site working well.
I made an imgur account. Here're the images -> The 'Franken_Beast'

Simple bonnie++ run so I could see it do _something_:
Code:
# bonnie++ -d /mnt/franken_pool -s 16G -n 0 -m TEST -f -b -u root
1.98,1.98,TEST,1,1561338786,16G,,8192,5,,,674066,73,405369,80,,,1049829,99,3744,242,,,,,,,,,,,,,,,,,,,31663us,103ms,,10638us,64800us,,,,,,
 

Ouroboros

New Member
Jul 26, 2012
27
5
3
I made an imgur account. Here're the images -> The 'Franken_Beast'

Simple bonnie++ run so I could see it do _something_:
Code:
# bonnie++ -d /mnt/franken_pool -s 16G -n 0 -m TEST -f -b -u root
1.98,1.98,TEST,1,1561338786,16G,,8192,5,,,674066,73,405369,80,,,1049829,99,3744,242,,,,,,,,,,,,,,,,,,,31663us,103ms,,10638us,64800us,,,,,,
Is this only 2 PCIe switches deep then?

Also, have you ever found right angle OCulink to U.2 cables to fit to the intel PCIe switch card?

Supermicro does now have a 8 OCulink card in x16 half height flavor (AOC-SLG3-8E2P) but getting one seems difficult...
 

NaCl

New Member
Dec 15, 2018
25
2
3
Greetings! Pardon my tardiness/dereliction in a timely response.

Is this only 2 PCIe switches deep then?
Yes. I only have the one sff8611 -> sff8639 cable so couldn't load up the first pcie switch.

Also, have you ever found right angle OCulink to U.2 cables to fit to the intel PCIe switch card?
I have not. I've periodically checked, but all I've been able to find are sff8611 -> sff8611 w/various combinations of 'right angle'-ness; perpendicular or parallel to the board. IIRC microsatacables.com had the best selection. It should be noted that I have not tried any of their cables yet.

Supermicro does now have a 8 OCulink card in x16 half height flavor (AOC-SLG3-8E2P) but getting one seems difficult...
Shiny! I can't even see an option to _try_ to order one. Seems 'difficult', in this instance, approximates 'impossible'. :(

What I'd truly _love_ to get is an sff8643 or sff8612 (the x8 80p variant) on a backplane in the same form factor as this intel 2.5" x8 enclosure w/a plx pcie switch and the sff8639 ports subbed out for m.2 nvme sockets. Something like CP067_CONCEPT PRODUCTS_ICY DOCK manufacturer Removable enclosure, Screwless hard drive enclosure, SAS SATA Mobile Rack, DVR Surveillance Recording, Video Audio Editing, SATA portable hard drive enclosure but w/the pcie switch functionality.

But that's 'crazy talk'. :(
 

Ouroboros

New Member
Jul 26, 2012
27
5
3
Greetings! Pardon my tardiness/dereliction in a timely response.


Yes. I only have the one sff8611 -> sff8639 cable so couldn't load up the first pcie switch.


I have not. I've periodically checked, but all I've been able to find are sff8611 -> sff8611 w/various combinations of 'right angle'-ness; perpendicular or parallel to the board. IIRC microsatacables.com had the best selection. It should be noted that I have not tried any of their cables yet.


Shiny! I can't even see an option to _try_ to order one. Seems 'difficult', in this instance, approximates 'impossible'. :(

What I'd truly _love_ to get is an sff8643 or sff8612 (the x8 80p variant) on a backplane in the same form factor as this intel 2.5" x8 enclosure w/a plx pcie switch and the sff8639 ports subbed out for m.2 nvme sockets. Something like CP067_CONCEPT PRODUCTS_ICY DOCK manufacturer Removable enclosure, Screwless hard drive enclosure, SAS SATA Mobile Rack, DVR Surveillance Recording, Video Audio Editing, SATA portable hard drive enclosure but w/the pcie switch functionality.

But that's 'crazy talk'. :(
I just want to confirm here, that you were unable to get the Intel AIC AXXP3SWX08080 to work with a SFF-8211 OCulink to SFF-8643 MiniSAS HD cable, particularly to connect to the IcyDock MB699VP-B? Did IcyDock support get back to you regarding why (legacy signaling?)

The cable you mentioned
https://www.amazon.com/gp/product/B07CJYNYQK
may have been problematic, but are DiLinker cables more likely to succeed in connecting the Intel card to the IcyDock, such as this cable?
https://www.amazon.com/dp/B07VT3YC9P
 

NaCl

New Member
Dec 15, 2018
25
2
3
I just want to confirm here, that you were unable to get the Intel AIC AXXP3SWX08080 to work with a SFF-8211 OCulink to SFF-8643 MiniSAS HD cable, particularly to connect to the IcyDock MB699VP-B?
Correct.

Did IcyDock support get back to you regarding why (legacy signaling?)
Support got back to me that the IceDock uses legacy wiring, but I don't know that that is the reason the AIC AXXP3SWX08080 did not work w/it. I was originally trying to get the IcyDock to work w/a 9405W-16 and, after a bit of research found sff-9402 to be the requisite spec. Their support confirmed that the unit did not use the newer spec. I only tried going down the Intel AIC AXXP3SWX08080 oculink 'rabit hole' because, in my research trying to figure out why the 9405W wasn't working, I stumbled up on this forum thread and decided to acquire/try the Intel hba in lieu of the 9405W. As it happened, that didn't ever work. So I ultimately ended up also getting the 'mated' Intel u.2 enclosure. I've since, acquire another set.

Here are the responses IcyDock sent me:
"Thank you for your inquiry regarding the MB699VP-B and the card you are using. I have contacted engineering regarding your request and they should be able to provide additional information on the pinout we use on the MB699VP-B and the SFF-9402 specification.
I will let you know when they get back on this request."

and the follow-up:
"Engineering has followed up with me regarding your inquiry on the MB699VP-B and SFF-9402. MB699VP-B miniSAS HD connectors are under legacy wiring."


The cable you mentioned
https://www.amazon.com/gp/product/B07CJYNYQK
may have been problematic, but are DiLinker cables more likely to succeed in connecting the Intel card to the IcyDock, such as this cable?
https://www.amazon.com/dp/B07VT3YC9P
I'm not sure. Given that these others didn't and they claimed they should, I could only suggest 'rolling the dice' and ordering a set and trying it out if you already have the rest of the stuff. I ended up returning the IcyDock and those cables because I was still w/in my return window.

For completeness here's the rest of the email chain I had w/IcyDock:

ME: "Thank you very much! I think I confirmed this last night w/a series of 'franken-adapters' (pci-e x4 -> nvme -> u.2). On another note, could you provide me w/the proper oculink -> 8643 type cable that should work w/the icydoc? I have these -> https://www.amazon.com/gp/product/B07CJYNYQK w/an Intel AXXP3SWX08080 8PORT PCI Express GEN3 X8 Switch. The 8 ports are Oculink. Those cables are 8611 (male oculink) (host side) to 8643 (target side). However, the green LED in the icydoc is constantly illuminated and no NVMe devices therein are enumerated."

ID: "Regarding cabling between the SFF-8643 connector in the rear of the MB699VP-B and a SFF-8611 OCUlink connector, the cable you are using should work."

ID: "We would like to follow up with you if you were able to get our MB699VP-B to work with your system setup? If so, let us know what cabling was used to connect our product to which specific hardware. This information would be helpful to our engineers on compatible cabling and in providing information on how to connect our product with various hardware."

ME:"Never w/Oculink cabling. I returned both of them because I could not get it to work. Was sad as I thought the units were very nice. But it wouldn't work so I had to make a command decision to go a different direction.
The only way I was able to get the icydoc to work was w/the 'franken-adapter' I mentioned a few messages ago. That data was comprised of:
0) https://www.amazon.com/Asus-Hyper-Expansion-Supports-Threadripper/dp/B07NQBQB6Z/ w/bifurcation AND/OR https://www.amazon.com/gp/product/B01MUB54TD w/out.
1) https://www.amazon.com/U-2-M-2-Adapter-Interface-Drive/dp/B073WGN61Y in the m2. slot(s)
2) https://www.amazon.com/CableCreation-Internal-Cable-SFF-8643-3-3FT/dp/B013G4FGSC

To get the 9405W-16i to work, I had to purchase the 'liquid gold McSpecial' cables from Broadcom. That data was comprised of:
0) Broadcom 9405W-16i
1) Broadcom 'enabler' cable -> 1m 2 SFF-8643 to 2 SFF-8643 NVMe Cable - 05-50061-00
2) https://www.amazon.com/gp/product/B076PMCX8W (because of the icydoc's 'legacy' wiring)

....
PS I still have a number of the 'pieces parts' but if you wanted to send me a unit (happy to return when done) and whatever cabling you come up with for testing purposes, I'd be happy to help out. A _cheaper_ solution would be quite swell. While the Intel nvme cage works, it's meant for their rack mount case so it doesn't work in any 'normal' space. Just an option.

PSS On another note, the m.2 nvme formfactor 2280 anyway ends up with a lot of wasted space in a u.2/nvme adapter. It'd be _super_..._SUPER_ swell if you could design an enclosure like your nvme unit, but specifically for m.2 format sticks. If you set them up w/the m2. connect on the backplane in a vertical fashion w/plastic guide rails such that the orientation/operation is akin to a normal thin/flat sticks of chewing gum are arranged. Even w/some generous spacing for airflow purposes, I'd expect that you could get some serious density in the same form factor as the current 4 drive nvme unit. (Standard 5.25" drive bay). At 2.3mm thick and an easy 5.25" horizontal space to work with, you could give each stick a bit over 11mm and get 12 sticks per enclosure. Then add occulink and/or minisas hd (sff-9402 wiring) on the back and you'd be golden. Even better would be to add a variant w/a plx chip capable of bandwidth management to all of the slots based on how many of the upstream links were configured. Then people that don't require each stick to have it's own x4 connection can do what they can and 'get by' w/the level of bandwidth their platform _can_ provide. Any chance of something like this? _I'd_ buy one!"

ID:"Thank you on the follow up on our questions regarding connecting the units you had with your system. We are currently developing NVMe cages that would be using slimmer trays for the NVMe m.2 drives. One concept product we have in development that fits with the use of slimmer trays is the CP082 (CP082_CONCEPT PRODUCTS_ICY DOCK manufacturer Removable enclosure, Screwless hard drive enclosure, SAS SATA Mobile Rack, DVR Surveillance Recording, Video Audio Editing, SATA portable hard drive enclosure). For rear connections in our concept NVMe products, we can consider the usage of Oculink or miniSAS SFF-9402 wiring standard.

As for helping to test our products, we appreciate the offer. For some of our concept products, there might be beta tests to check the usability of our products. If you have the hardware for testing along with the time for testing, we could consider you. We currently do not have one as of now, but if there is a request for beta testers for a concept product, you can be informed by signing up for our email newsletter. The newsletter is the best way to learn about new products, promotions, and news updates from ICY DOCK.

If you haven't signed up and you are interested in signing up for our newsletter, simply reply to this email with a message that you want to sign up and I will sign you up. We promise not to spam you and we limit our emails as much as possible to give you only the most important information. In the event that you do not want to receive the newsletter, there is a 1-click unsubscribe on each newsletter. Thank you for your consideration!"

------
As it turns out the CP081 is very close to what I was envisioning. If they shaved a few mm off the spacing I think they could get 12 sticks in a single 5.25" enclosure.

Anyway, I hope that helps/provides the clarity you are seeking.
 

Ouroboros

New Member
Jul 26, 2012
27
5
3
The IcyDock cage being problematic may indicate it may need one of those "blessed" trimode cables in OCulink format, perhaps such as the Supermicro one

https://www.amazon.com/OCULINK-V-1-0-SOURCE-TO/dp/B077BVPJGG

Supermicro's 8x OCulink AOC card manual specifically says to make sure you get the OCulink to SFF-8643 cable with the OCulink end that is green wrapped (as a differentiation marker, though also the SFF-8643 end is usually white for NVMe use as well). That miiiiight be wired right.

IcyDock at least has a trimode warning in the their FAQ as a PDF

https://www.icydock.com/PDF/MB699VP-B_Tri-mode_compatability_announcement.pdf

but that deals with RAID cards, not PCIe switch HBA's.
 

NaCl

New Member
Dec 15, 2018
25
2
3
The IcyDock cage being problematic may indicate it may need one of those "blessed" trimode cables in OCulink format, perhaps such as the Supermicro one

https://www.amazon.com/OCULINK-V-1-0-SOURCE-TO/dp/B077BVPJGG

Supermicro's 8x OCulink AOC card manual specifically says to make sure you get the OCulink to SFF-8643 cable with the OCulink end that is green wrapped (as a differentiation marker, though also the SFF-8643 end is usually white for NVMe use as well). That miiiiight be wired right.

IcyDock at least has a trimode warning in the their FAQ as a PDF

https://www.icydock.com/PDF/MB699VP-B_Tri-mode_compatability_announcement.pdf

but that deals with RAID cards, not PCIe switch HBA's.
The compatibility announcement they have there is per what I was just talking about. That _is_ the sff-9402 spec. I'd dare to assert that my inquiry probably helped push them into making that. The Broadcom 9405W-16 is a tri-mode hba; not raid.

Whether or not the Intel board is expecting the same adherence, I haven't checked.
 

Ouroboros

New Member
Jul 26, 2012
27
5
3
A bit of an update, I finally got the IcyDock MB699VP-B and some cables and a u.2 drive installed on my test rig.

Directly connecting the U.2 drive to the Intel AXXP3SWX08080 card with a DiLinker SFF-8611 to SFF-8639 works as expected, so the card and the U.2 drive are known good.

Unfortunately, connecting the Intel AXXP3SWX08080 card to the IcyDock MB699VP-B with one of the green marked Supermicro CBL-SAST-0929 does not work.

This should not be this hard. I am highly inclined at this point either give up, or get custom cables made. I'd eventually need some right angle to straight oculink extender adapters anyways since 4 of the intel card's oculink connectors are right angled, and most cables use straight connectors.

EDIT
Minor update, IcyDock apparently won't/can't recommend a cable, but helpfully did provide their "legacy" SFF-8643 pinouts in a PDF, which should make ordering a custom cable easier for anyone making an attempt. IcyDock seems to have doubts regarding the pinouts of the OCulink connectors on the cards, but considering other types of cables (U.2 particularly) seem to work fine, that's open to debate...

At this point, I suspect that both the Intel AXXP3SWX08080 and Supermicro AOC-SLG3-8E2P are both based on PMC Sierra Switchtec PFX 48xG3 chips, based on the exposed management endpoints. This would imply they may be based on the same reference design, though visually the cards are quite different, which would suggest the OCulink connectors would be wired the same...
 

Attachments

Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
@William has one of those IcyDock for NVME, I got it from him.
Maybe he can help hook it up for you?

I haven't hooked it up, but my plan was supermicro PCIE card.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Sorry kind of distracted here on Christmas eve so I haven't gone too far back in the thread to read.

This is the cable I use.
https://www.amazon.com/gp/product/B01J3MYD8K/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1

That connected straight to my ASUS C621E SAGE MB then to any of the Icy Dock cages or even direct connect to any NVMe U.2 drive.

On Threadripper boards I simply got one of these.
https://www.amazon.com/gp/product/B01LY9HRDB/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

And used the same cable as above.

This again works with any Icy Dock cage I have used.

I don't know about the Oculink cables as I haven't tried any of those.
 
  • Like
Reactions: T_Minus

Ouroboros

New Member
Jul 26, 2012
27
5
3
A bit of an update, I finally got the IcyDock MB699VP-B and some cables and a u.2 drive installed on my test rig.

Directly connecting the U.2 drive to the Intel AXXP3SWX08080 card with a DiLinker SFF-8611 to SFF-8639 works as expected, so the card and the U.2 drive are known good.

Unfortunately, connecting the Intel AXXP3SWX08080 card to the IcyDock MB699VP-B with one of the green marked Supermicro CBL-SAST-0929 does not work.

This should not be this hard. I am highly inclined at this point either give up, or get custom cables made. I'd eventually need some right angle to straight oculink extender adapters anyways since 4 of the intel card's oculink connectors are right angled, and most cables use straight connectors.

EDIT
Minor update, IcyDock apparently won't/can't recommend a cable, but helpfully did provide their "legacy" SFF-8643 pinouts in a PDF, which should make ordering a custom cable easier for anyone making an attempt. IcyDock seems to have doubts regarding the pinouts of the OCulink connectors on the cards, but considering other types of cables (U.2 particularly) seem to work fine, that's open to debate...

At this point, I suspect that both the Intel AXXP3SWX08080 and Supermicro AOC-SLG3-8E2P are both based on PMC Sierra Switchtec PFX 48xG3 chips, based on the exposed management endpoints. This would imply they may be based on the same reference design, though visually the cards are quite different, which would suggest the OCulink connectors would be wired the same...

A minor update for folks who may be interested.

I just received word that my custom cable order to connect an Intel AXXP3SWX08080 card to an IcyDock MB699VP-B has been changed to a normal catalog cable type order, so the cables can now be purchased by anyone.


First is a convenience adapter cable, to change those right angle connectors on the card into more common straight connectors.

Right angle male OcuLink to straight female Oculink

The second cable is supposed to be correctly wired for "legacy" pinouts to the IcyDock cage

male straight OcuLink to "legacy" SFF-8643

I'm ordering these right now now and will report back on the results.
 
  • Like
Reactions: Jason Hirsch