Help with SerialCables.com PCI4-AD-x16HE-BG4

[mattventura]

I picked up one of these for $65 on the bay. Seems like a great deal for a 4x4 external PCIe switch good functionality. It uses a Broadcom Atlas switch chip, along with a management processor to let you reconfigure the chip.

The problem is, the one I got came pre-programmed in 16x1 mode. I'd like it to be 4x4. I've connected to the debug USB serial port, and updated the firmware to the latest on the website (HC_DPR_43.fw). But there are a lot of problems:

1. The manual shows a different set of commands. It says you should be able to use the 'setmode' command - I don't have that. I'm also missing some other commands like the fan control.
2. I don't have the 'fdl sw' command, but I do have 'fdl cfg', 'fdl sbr', 'fdl fw', and 'fdl mfg'. I used 'fdl fw' to upgrade the firmware, but I can't seem to get any of the other files (like the XML file included in the firmware which I edited to run in x4x4x4x4 mode), nor the .BIN files included in the "base mode" download to do anything. Either the XMODEM transfer just stops partway through, or it accepts the file but it does absolutely nothing.
3. If I try to write the .BIN file as the firmware (i.e. fdl fw), it accepts it, but the switch don't function. It displays the red error LED and the blue heartbeat LED is solid. I have to reflash the real firmware file to get it working again.

Any ideas what I'm doing wrong? I notice it also exposes a /dev/mpt3ctl device (and an SES device) device which can supposedly be used for some management tasks, but "change the lanes" does not seem to be one of them. Strangely, I seem to have a newer hardware revision (1.2) compared to the manual (1.1), so it's probably not an issue of outdated firmware.

 

[mattventura]

Updates:

• I bought another one, version 1.1 this time.
• The 1.1's management commands seem to actually match what's in the manual. I could change it to 4x4 and actually see it reflected in the PCIe topology after a reboot.
• The 1.1 switch didn't work for me either - I looked at dmesg and saw that it was running out of PCIe BAR space. Got the same result on a few different systems.
  

Seems I'm kind of out of luck on getting any of these working unless I can figure out a way around the BAR space limitations. I probably need a newer board.

I've noticed that even on the older card, the PCI topology is a bit strange. It seemed to expose a few extra downstream busses, which is contributing to the issue. A more "barebones" switch doesn't seem to have these extra busses, so they tend to work in a larger variety of hardware. Definitely don't buy these expecting them to work in a typical system.

I'll try throwing them in my main desktop at some point. Anything that can support Thunderbolt probably has a BIOS equipped to handle a larger number of PCIe devices in the topology.

 

[meklort]

@mattventura I also ordered a couple of the devices, and have the 1.2 version.
I was able to flash the 4x4 configuration (B0 HOST_X4_SRIS_V02.bin) using fdl sbr from the Serial Cable's website to get things working.

My setup is a mac min w/ TB5 to the card, then the card to an Icydock.

Note, from the Gen5 manuals:
fdl sbr=update the SBR file into flash of Atlas2 switch. (Applicable in base switch mode)
fdl fw=program or upgrade FW into flash of Atlas2 switch (Applicable in Synthetic mode)
fdl mcu=on-board MCU FW upgrading


Also note that these use a slightly custom pinout - the SFF-8644 to SFF-8611 cables that you can find on ebay don't work with the device. I did, however have a few custom cables made and have been able to get these to work. You can also buy the cables from the serial cables website, but... I really didn't want to pay $220 + shipping for each cable.

 

[mattventura]

@mattventura I also ordered a couple of the devices, and have the 1.2 version.
I was able to flash the 4x4 configuration (B0 HOST_X4_SRIS_V02.bin) using fdl sbr from the Serial Cable's website to get things working.

My setup is a mac min w/ TB5 to the card, then the card to an Icydock.

Note, from the Gen5 manuals:
fdl sbr=update the SBR file into flash of Atlas2 switch. (Applicable in base switch mode)
fdl fw=program or upgrade FW into flash of Atlas2 switch (Applicable in Synthetic mode)
fdl mcu=on-board MCU FW upgrading


Also note that these use a slightly custom pinout - the SFF-8644 to SFF-8611 cables that you can find on ebay don't work with the device. I did, however have a few custom cables made and have been able to get these to work. You can also buy the cables from the serial cables website, but... I really didn't want to pay $220 + shipping for each cable.

Wow, thank you!

I'll try flashing the base switch mode again to see if that works. I may have been confused by the management commands not working correctly with it, but I guess that's to be expected.

Do you know what the custom pinout is, or where you got the custom cables?

 

[meklort]

I'm just about to place an order for some more cables (0.5m), but could increase the quantity if you need some (note that my price ends up being ~$40 per cable.

Cable website: Gen4 x4 OCUlink (SFF-8611) 42P to Ext. HD (SFF-8644) 36P Cable | Gen4 PCIe | Serial Cables
Pinout is available here: https://serialcables.com/vendor-med...32351723134997.pdf?title=Drawings&optionId=56

 

[madscience]

I also recently picked up three of these adapters, and have been experimenting with trying to use them as a target. I'll dump some observations below.

All three of the ones I got were version 1.2 with firmware version 0.1.9 installed. I had the same experience as mattventura where the fdl commands did not do much of anything, despite manually editing the mfg file.
I did try the USB firmware upgrade mode (bridge J78) and had no luck. It seems that mode wants a different type of file to the ones available on SerialCables.com

Also to add more confusion, I found another manual for the same card that seems to be newer. It does have a section for using the card as a target using the three jumpers, but in my experiments it did not work. Perhaps it requires a specific basemode to be set?


The comments for the XML file states that you can use a tool called xutil and xflash to read and write firmware on these devices. This seems to be a broadcom OEM tool, of which I only found one link to. I was unable to get it to work with a TTL to USB adapter through its UART headers (CN2).

Surprisingly when changing the jumper to "PCI-SIG" I did get it to link between cards at gen 4 speeds with a cable from Aliexpress. However I have not tested it with any actual host targets.

 

[slidermike]

Would you guys mind sharing a brief summary of the use of these sorts of cards? I did a quick google and not really seeing it. Thanks!

 

[madscience]

In practise you can use it like an external NVME HBA. So if you have a disk shelf, or in my case are building an external gpu chassis, you can hook it up to the host without bifurcation. It's different to a HBA in the fact that it switches the PCIE signal, kind of like a network switch. On the other hand tri-mode HBAs act as PCIE host for the drive and exposes them to the host as a SAS drive. Even the ones that can pass through are much less performant than PCIE switch based solutions.

Switches can go a bit further with possibility for building a PCIE fabric where GPUs and storage are shared between multiple hosts (aka Liqid). Of course us mortals buying ebay hardware can't do that yet because the tools are restricted only to OEMs.

Now, why not use a retimer? At least for storage it can be support for dual porting. Or if your workload involves any peer to peer access between targets (multi-GPU or GPU to NIC) you can potentially skip the host's PCIE root complex for lower P2P latency (but higher Host to GPU latency). I also like not relying on bifurcation on the host side.

So, the primary use-case that you'd see for this card is connecting to an external flash storage chassis. However the possibilities are limited to whatever you can cobble together with PCIE risers, cards and cables.

 

[mattventura]

In my case, I mostly just want something that lets me easily hot-(un)plug various cards so I can test things without a reboot.

I was also looking at the AOC-SLG3-4X4P but:

1. It's only gen3
2. It seems to only support 2x8 and 1x16 downstream, no 4x4. Maybe it can do 2x4 if you just don't use half the ports?
3. No confirmation that it uses a standard pinout

 

[DRW]

Dolphin interconnect cards/cables also seem to use the same (nonstandard?) pinout (A:0+3, B:1+2).

https://dolphinics.com/download/CABLES/PCIe4L_Data_Sheet.pdf

 

[mattventura]

So far, I've been trying to plug the card into a minisforum DEG1, which doesn't seem to work. Is there a breakout board that uses the same SFF-8644 connector that is known to use the same non-standard pinout so that I can use a normal cable with this?

Do SerialCables.com JBOFs and stuff use the non-standard pinout as well? I'd imagine they probably do.

Though now I wonder - if the issue is that the lanes are just wired up a bit oddly, would it not work to set the card into 16x1 downstream mode, and then plug a x1 card into the dock?

 

[luckylinux]

I also recently picked up three of these adapters, and have been experimenting with trying to use them as a target. I'll dump some observations below.

All three of the ones I got were version 1.2 with firmware version 0.1.9 installed. I had the same experience as mattventura where the fdl commands did not do much of anything, despite manually editing the mfg file.
I did try the USB firmware upgrade mode (bridge J78) and had no luck. It seems that mode wants a different type of file to the ones available on SerialCables.com

Also to add more confusion, I found another manual for the same card that seems to be newer. It does have a section for using the card as a target using the three jumpers, but in my experiments it did not work. Perhaps it requires a specific basemode to be set?
View attachment 44041

The comments for the XML file states that you can use a tool called xutil and xflash to read and write firmware on these devices. This seems to be a broadcom OEM tool, of which I only found one link to. I was unable to get it to work with a TTL to USB adapter through its UART headers (CN2).

Surprisingly when changing the jumper to "PCI-SIG" I did get it to link between cards at gen 4 speeds with a cable from Aliexpress. However I have not tested it with any actual host targets.
View attachment 44040

I don't see any explanation of that on the Dolphinics MXH932 that I bought unfortunately .

Any Luck that used SFF-8644 to SFF-8644 Cables from e.g. Dlel/Fujitsu/etc (which can be bought relatively cheap from several Recyclers) would work in the same weird Pinout ?

 

[bkvaluemeal]

I'm just about to place an order for some more cables (0.5m), but could increase the quantity if you need some (note that my price ends up being ~$40 per cable.

Cable website: Gen4 x4 OCUlink (SFF-8611) 42P to Ext. HD (SFF-8644) 36P Cable | Gen4 PCIe | Serial Cables
Pinout is available here: https://serialcables.com/vendor-med...32351723134997.pdf?title=Drawings&optionId=56

I converted the pinout from the datasheet into a wiring diagram using WireViz and along with CONN_0 from the manual.

 

[luckylinux]

I'm just about to place an order for some more cables (0.5m), but could increase the quantity if you need some (note that my price ends up being ~$40 per cable.

Cable website: Gen4 x4 OCUlink (SFF-8611) 42P to Ext. HD (SFF-8644) 36P Cable | Gen4 PCIe | Serial Cables
Pinout is available here: https://serialcables.com/vendor-med...32351723134997.pdf?title=Drawings&optionId=56
View attachment 43825

Where did you order those custom Cables from ?

 

[bkvaluemeal]

I have one of these cards, and like @madscience I also want to build an external GPU enclosure. I'd prefer to use SFF-8644 cables outside of the enclosures because they are designed for that whereas SFF-8611 (OCuLink) male connectors feel too flimsy. I'm using an SFF-8644 to SFF-8643 adapter board at the edge of the enclosure, and then SFF-8643 to PCIe. However, after painstakingly probing pins with a multimeter I've learned that the SFF-8644 cables don't wire the PCIe clock pins (ie: pin A1) -- those go to EEPROM chips inside the connector housing. Everything else is wired as expected, and this would be the reason the port's LED is red.

 

[luckylinux]

I'm wondering if it would almost be easier to make a custom PCB with 4xSFF-8644 Connectors and do the rewiring there, then use it to "plug" into a x16 Slot, which will then distribute in 4 x PCIE x16 Slots (x4 Electrical each) .

Because otherwise between custom SFF-8644 to SFF-8644 Cables, SFF-8644 to SFF-8643 Adapters / Brackets, SFF-8643 to SFF-8643 Cables or SFF-8643 to SFF-8654 Cables (these Days the x16 Slot risers originating from SFF-8654 Cables are much cheaper, but it's true that most Cables are SFF-8654 Host and SFF-8643 Target, NOT the other Way around like we would need here unfortunately ).

I would need 4x - 8x of these Cables per each Server (total 20x Cables plus possibly a few Spares) for a few GPU LLMs Appliances (with AMD Radeon Mi50 16GB GPUs, not ideal compared to the 32GB Version, but still managed to grab these for a decent Price).

At 40 USD + Shipping + Import that would be 800 EUR just for THOSE custom Cables. I would still need all of the SFF-8644 to SFF-8643 Brackets/Adapters, SFF-8643 to SFF-8643 Cable (25cm if possible) and SFF-8643 to x16. Plus probably some Molex 6-pin Power Splitters as well.

A custom PCB (on a PCI Bracket Interface) on the same Form Factor as the LSI 9305-16e / LSI 9300-16e with 4xSFF-8644 Connectors plugging directly into a x16 Slot (on the PCB Board, where you would typically install the Motherboard) would work much better and be much cheaper at the same Time. It would mean a lot let Connections and signal Lengths. It could also save on some Power Supply Splitters since the 3.3V 5V etc can be made only once-twice instead of 8 Times !

EDIT 1: And of course on the custom PCB Interface we could simply rearrange the signals so that these cheap SFF-8644 Cables work with the Dolphinics / SerialCables Adapter .

 

[madscience]

Did you manage to get it working with "normal" (cheap) Cables in the End ?

Hello. It's been a while since I looked back at this thread. Unfortunately I never got either card working for external GPU purposes with the cables I had. I haven't had the chance to dive deep due to changes in life circumstances leading to the lab being parted off.

Apologies if it's a bit disappointing. Honestly I'm thinking that this ex enterprise hardware is probably not the best way forward, particularly with some of the hardware that I've seen on Aliexpress and Taobao over the last few months.

ADT-Link are out selling PEX88096 boards with up to 10 8654 ports for a total of five! internal X16 GPUs. For external use there's been a tidal wave of boards also based on the PEX88096. You should find some results if you search for that. There's a promising one from ServerHive that has several reviews.

 

[mattventura]

For external use there's been a tidal wave of boards also based on the PEX88096. You should find some results if you search for that. There's a promising one from ServerHive that has several reviews.

Interesting, would you happen to know of any good switches that sit inside the case but have external ports? My main use is that I'd like to be able to hotplug a device into the host from outside, but without needing to have anything attached externally if I don't need it at the moment.

 

[luckylinux]

I also recently picked up three of these adapters, and have been experimenting with trying to use them as a target. I'll dump some observations below.

All three of the ones I got were version 1.2 with firmware version 0.1.9 installed. I had the same experience as mattventura where the fdl commands did not do much of anything, despite manually editing the mfg file.
I did try the USB firmware upgrade mode (bridge J78) and had no luck. It seems that mode wants a different type of file to the ones available on SerialCables.com

Also to add more confusion, I found another manual for the same card that seems to be newer. It does have a section for using the card as a target using the three jumpers, but in my experiments it did not work. Perhaps it requires a specific basemode to be set?
View attachment 44041

The comments for the XML file states that you can use a tool called xutil and xflash to read and write firmware on these devices. This seems to be a broadcom OEM tool, of which I only found one link to. I was unable to get it to work with a TTL to USB adapter through its UART headers (CN2).

Surprisingly when changing the jumper to "PCI-SIG" I did get it to link between cards at gen 4 speeds with a cable from Aliexpress. However I have not tested it with any actual host targets.
View attachment 44040

Did you manage to get it working with "normal" (cheap) Cables in the End ?

 

[luckylinux]

Hello. It's been a while since I looked back at this thread. Unfortunately I never got either card working for external GPU purposes with the cables I had. I haven't had the chance to dive deep due to changes in life circumstances leading to the lab being parted off.

Apologies if it's a bit disappointing. Honestly I'm thinking that this ex enterprise hardware is probably not the best way forward, particularly with some of the hardware that I've seen on Aliexpress and Taobao over the last few months.

ADT-Link are out selling PEX88096 boards with up to 10 8654 ports for a total of five! internal X16 GPUs. For external use there's been a tidal wave of boards also based on the PEX88096. You should find some results if you search for that. There's a promising one from ServerHive that has several reviews.

Yeah but now I got 3 Cards of one Type and 6 (?) of these Serial Cables now .