Inaccessible Drives Behind NVMe Switch (AOC-SLG3-2E4)

[Deviant0ne]

Hello - long time reader, first time poster.

I've been looking for a way to add 2x NVMe drives to my Supermicro-based (X10SLL-F) system. This system does not support bifurcation, so after extensive research, I purchased a Supermicro AOC-SLG3-2E4. In addition, I purchased an external NVMe bay (Athena Power BP-M22NVMEO350, BP-M22NVMEO350 | athenapower.com) and some generic SFF-8643 to SFF-8611 (4i) cables to make all of the necessary connections. I don't need to boot from these NVMe drives and I don't need the hot-swap functionality.

After connecting all components and booting the machine, I found that the operating system (FreeBSD 14.1) is able to detect the AOC-SLG3-2E4, but neither of the drives I have connected are accessible. I can see the PLX/PEX bridge listed 3x times in the output from pciconf, (I assume 1x instance for the card itself and then 2x additional instances for each of the ports). I've tried swapping PCIe ports and disabling the ability to boot from the device in the BIOS but none of the changes I made are allowing me to access the drives.

I've seen a number of posts indicating this card should work on almost any motherboard with little/no modifications. I've also seen similar postings where users have reported needing Supermicro-branded cables. I am hoping someone can help point me in the right direction, thank you in advance for any assistance.

 

[frankharv]

All the parts jibe so I do wonder if you can give a link to the cables.

Also I did have one of these cards that quit working. The large tall power regulator chip on it was scalding hot.

I added a little heatsink to all mine.

See what yours feels like. The chip runs hot but on my bad card it would burn you.

 

[Deviant0ne]

All the parts jibe so I do wonder if you can give a link to the cables.
...

Thank you for the reply - here is a link to the cables I purchased: https://www.amazon.com/dp/B0BZJLSZ7L?psc=1&ref=ppx_yo2ov_dt_b_product_details

I will boot the machine back up and check the temperature of the power regulator.

 

[T_Minus]

All the parts jibe so I do wonder if you can give a link to the cables.

Also I did have one of these cards that quit working. The large tall power regulator chip on it was scalding hot.

I added a little heatsink to all mine.

See what yours feels like. The chip runs hot but on my bad card it would burn you.

Can you share pics of what you did? I always put a fan near mine in desktop\homelab setup.

 

[T_Minus]

Hello - long time reader, first time poster.

I've been looking for a way to add 2x NVMe drives to my Supermicro-based (X10SLL-F) system. This system does not support bifurcation, so after extensive research, I purchased a Supermicro AOC-SLG3-2E4. In addition, I purchased an external NVMe bay (Athena Power BP-M22NVMEO350, BP-M22NVMEO350 | athenapower.com) and some generic SFF-8643 to SFF-8611 (4i) cables to make all of the necessary connections. I don't need to boot from these NVMe drives and I don't need the hot-swap functionality.

After connecting all components and booting the machine, I found that the operating system (FreeBSD 14.1) is able to detect the AOC-SLG3-2E4, but neither of the drives I have connected are accessible. I can see the PLX/PEX bridge listed 3x times in the output from pciconf, (I assume 1x instance for the card itself and then 2x additional instances for each of the ports). I've tried swapping PCIe ports and disabling the ability to boot from the device in the BIOS but none of the changes I made are allowing me to access the drives.

I've seen a number of posts indicating this card should work on almost any motherboard with little/no modifications. I've also seen similar postings where users have reported needing Supermicro-branded cables. I am hoping someone can help point me in the right direction, thank you in advance for any assistance.

Can you get a cable for direct attach tot hat AOC? It will connect to 2.5" NVME with SATA power, then you can rule out cable or issue with your cage.

 

[frankharv]

OK lets use this pix:

Supermicro AOC-SLG3-2E4 2-Port 6.4GB/s NVME PCIE HBA Host Bus Adapter | eBay

Find many great new & used options and get the best deals for Supermicro AOC-SLG3-2E4 2-Port 6.4GB/s NVME PCIE HBA Host Bus Adapter at the best online prices at eBay! Free shipping for many products!

www.ebay.com

The big VITEC chip right in front of heatsink. It is very tall and right next to a capacitor.

I used my heatsink kit for Arm boards and stuck one on the VITEC chip.

High Frequency Power Inductor

https://www.viteccorp.com/data/af4383.pdf

 

[Deviant0ne]

I've attached some images of the current configuration. The Supermicro card is connected to the 16x slot on the motherboard.

Can you get a cable for direct attach tot hat AOC? It will connect to 2.5" NVME with SATA power, then you can rule out cable or issue with your cage.

Thank you for the suggestion, that was something I considered. I would need to purchase an M.2 to U.2 enclosure and the mini SAS to U.2 cable, I do not have one on hand for testing.

 

[T_Minus]

I've attached some images of the current configuration. The Supermicro card is connected to the 16x slot on the motherboard.



Thank you for the suggestion, that was something I considered. I would need to purchase an M.2 to U.2 enclosure and the mini SAS to U.2 cable, I do not have one on hand for testing.

Oh I didn't realize that was an m2 adapter\enclosure my bad!

I've never gone that direction, but I have gone from m2 port on motherboard to u2\2.5" NVME and the cheap adapter\cables did not work or randomly worked, I went with Intel which are amphenol iirc brand, and highest quality if you can buy them on ebay\direct etc.

 

[Deviant0ne]

Thank you for your input, it sounds like using a U.2 enclosure and associated cable(s) might be a viable course of action.

 

[frankharv]

Maybe a $15 dollar single port NVMe PCIe adapter with sff8643 to test with. Bifurcation not needed.
Proving individual cables and cage.

Amazon.com

 

[nexox]

U.2 drives are designed to operate on the end of a cable, M.2 drives may not be as tolerant of the signal distortion in such a setup. If you go that way, I have had good results with these cables (made with Amphenol connectors): https://www.amazon.com/gp/product/B07JH92TD9/

Unless you really need hot swap (which isn't necessarily going to work without some effort, though starting with this card gets you pretty close) an enclosure is optional for U.2 drives, you can just screw them into internal trays or whatever, but airflow is going to be important, some drives need a lot, but pretty much every U.2 needs some.

 

[T_Minus]

U.2 drives are designed to operate on the end of a cable, M.2 drives may not be as tolerant of the signal distortion in such a setup. If you go that way, I have had good results with these cables (made with Amphenol connectors): https://www.amazon.com/gp/product/B07JH92TD9/

Unless you really need hot swap (which isn't necessarily going to work without some effort, though starting with this card gets you pretty close) an enclosure is optional for U.2 drives, you can just screw them into internal trays or whatever, but airflow is going to be important, some drives need a lot, but pretty much every U.2 needs some.

Yes those are the cables I was talking about, not sure about that brand but the Intel variant of those and the intel variant m2->adapter plugs into those cables too, they came with a lot of Intel Optane drives and what I've used since they were released with no issue. This as after I experienced random issues with generic cables too, it seems NVME in general and some motherboards are picky on the cable.

 

[nexox]

not sure about that brand

I agree, the brand is clearly just another made up name to sell stuff on Amazon, but in this case they appear to have just ordered a bunch of legit cables made by Amphenol. I opened up the SATA power connector on one and it's not the fire-hazard molded kind, so that's good enough for me.

 

[Deviant0ne]

U.2 drives are designed to operate on the end of a cable, M.2 drives may not be as tolerant of the signal distortion in such a setup. If you go that way, I have had good results with these cables (made with Amphenol connectors): https://www.amazon.com/gp/product/B07JH92TD9/

Unless you really need hot swap (which isn't necessarily going to work without some effort, though starting with this card gets you pretty close) an enclosure is optional for U.2 drives, you can just screw them into internal trays or whatever, but airflow is going to be important, some drives need a lot, but pretty much every U.2 needs some.

I appreciate the input, I will likely order the cables you referenced above along with an M.2 to U.2 adapter/tray (https://www.amazon.com/gp/product/B08JHHNWLM/ref=ox_sc_saved_image_1?smid=ATVPDKIKX0DER&psc=1).

 

[mattventura]

Maybe a $15 dollar single port NVMe PCIe adapter with sff8643 to test with. Bifurcation not needed.
Proving individual cables and cage.

Amazon.com

Agree, do this first.

Also check your kernel log for any suspicious messages. PCIe switches can potentially run into resource issues, though I would expect that motherboard to not have such problems. Doubly so if the drives are connected at boot.

 

[Deviant0ne]

Agree, do this first.

Also check your kernel log for any suspicious messages. PCIe switches can potentially run into resource issues, though I would expect that motherboard to not have such problems. Doubly so if the drives are connected at boot.

Thank you, that's my plan for now. I will post an update once I order/receive the replacement parts for testing.

I did closely monitor/review dmesg but there wasn't anything related to the switch.

 

[nexox]

I appreciate the input, I will likely order the cables you referenced above along with an M.2 to U.2 adapter/tray (https://www.amazon.com/gp/product/B08JHHNWLM/ref=ox_sc_saved_image_1?smid=ATVPDKIKX0DER&psc=1).

I'm not sure that will get you very far, since that adapter is passive, you're still going to have the M.2 controller that may just not be compatible with cables. Try the other troubleshooting first, you'll either get the M.2 drives to work on a cable or you'll have to either choose between different drives or a different PCIe switch / adapter that can mount M.2 drives directly.

 

[Deviant0ne]

I'm not sure that will get you very far, since that adapter is passive, you're still going to have the M.2 controller that may just not be compatible with cables. Try the other troubleshooting first, you'll either get the M.2 drives to work on a cable or you'll have to either choose between different drives or a different PCIe switch / adapter that can mount M.2 drives directly.

Just so I understand correctly, you're suggesting I purchase the 10Gtek single adapter and test with the existing enclosure/cables vs. purchasing/testing with the Vantec adapter, the cables you linked above and the Supermicro switch?

 

[nexox]

Just so I understand correctly, you're suggesting I purchase the 10Gtek single adapter and test with the existing enclosure/cables vs. purchasing/testing with the Vantec adapter, the cables you linked above and the Supermicro switch?

Yeah, test without the PCIe switch, then see how to proceed based on how that goes.

Maybe I missed it, but does the switch itself show up in lspci? I forget how they're labeled, bridge device or something.

 

[Deviant0ne]

Yeah, test without the PCIe switch, then see how to proceed based on how that goes.

Maybe I missed it, but does the switch itself show up in lspci? I forget how they're labeled, bridge device or something.

Thank you for the clarification. I think I will modify my plan slightly and purchase the single adapter, along with the enclosure and U.2 cable so that I can test all scenarios at once.

The switch does indeed show in pciconf (FreeBSD), it is listed 3x times as a bridge, presumably 1x listing for the device itself and the other 2x for the ports (just a guess).

UPDATE 1: I am waiting on the parts mentioned above, which I should receive today. I will update this post again when I have additional information.

UPDATE 2: The AOC-SLG3-2E4 along with the U.2 enclosure and the U.2 to MiniSAS cable allows access to the NVMe drive, early tests indicate the card seems to be fine. The single port NVMe PCIe adapter was not recognized by my motherboard at all, so I am still not able to access the NVMe drive installed in the enclosure. I ordered a Supermicro MiniSAS to OCuLink cable (CBL-SAST-0929) to determine if the issue resides in the original cables or the enclosure. I will post a final update once I receive the Supermicro cable next week.

UPDATE 3: The Supermicro cable had no effect on the ability to access the NVMe drives installed in the enclosure. I will be sending back the enclosure and the Supermicro cable in exchange for another U.2 enclosure and U.2 to MiniSAS cable. Thank you for the assistance.