Sun Oracle 7096186 PCIE Switch Adapter... plx?

[Dreece]

Anyone have one of these and/or able to share what asic it's using (ie PLX PEX8xxx etc)...

SUN 7096186 NVME 8-Port PCI Express Switch Card

NVME 8-Port PCI Express Switch Card

www.vibrant.com

Trying to find any info on Oracle hardware is akin to searching for water in a desert.

 

[BeTeP]

My educated guess based on the picture and the fact that the card came out of a SPARC server is that the card is not using a switch made by PLX Technology (now owned by Broadcom) but rather by PMC-Sierra (now owned by Microchip).

 

[Dreece]

PMC-Sierra

as in the same manufacturer who does the asics for Intel's nvme expander card? which then tells me that the card may not play nicely outside of a SPARC server, in which case I will keep my distance from it.

So it's probably one of these PSX chips under the heatsink:

 

[UhClem]

as in the same manufacturer who does the asics for Intel's nvme expander card? which then tells me that the card may not play nicely outside of a SPARC server, in which case I will keep my distance from it.

So it's probably one of these PSX chips under the heatsink: ...

SuperMicro's AOC-SLG3-8E2P card (8 x NVMe via x16_PCIe3) appears to use one of those PSX switch chips (link). Maybe SM did it right? ["Don't ... baby ... bath-water."] (or, does the layout/connectors on the SM card not suit you?)

 

[Dreece]

I have a x8 slot to play with in this particular server, have been eyeing up the x8 supermicro card, just thought I'd look at what's floating around out there and understand the underlying technology a little better across the board to know limits/advantages before purchasing, I like things that grow with you as you grow and not hit a brick wall. I do already have a HP one, but it's HP lol

Also we have to keep in mind many of these cards have compatibility issues with whatever SSDs, often just a firmware update, but some drive manufacturers simply don't do that, and then the choice of nvme 2.5 drives becomes rather limited. So far the supermicro looks the best best as HGST appears to support them as well as Intel, though not sure on Samsung, but outside of their AICs I simply run a mile from their enterprise U.2 drives.

I'll update once I've chosen and explain why, would help others in the same boat soon enough as more jump onto the nvme-bay bandwagon.

 

[BeTeP]

For NVMe cards of that generation one can guess which switch was used by looking at the number of PCIe lanes.
PLX chips were simpler design generic PCIe switches which just basically did bifurcation so they were splitting x16 connector into 4 ports x4 each. PMC added some NVME specific "secret sauce" which allowed them to use half as many lanes per port without sacrificing performance (too much). So this Oracle branded card is only x8 but it has 4 NVMe ports and AOC-SLG3-8E2P is x16 but has 8 NVME ports.

 

[Dreece]

The PLX 8724 does do PCIe-lane sharing rather than simply bific, it unfortunately doesn't support modern tech, which makes it rather limited if one wants to passthrough an nvme namespace out of however many on an individual drive to a particular vm. The big problem with the more newer chipsets is that outside of RHEL, ie Debian based distros, hotplug doesn't work right on a few it appears. It is more about finding that perfect balance where what is needed is served on the platform I intend to use it on. A few weeks of tinkering buying/selling I guess, and I'll know what's right for my use-case. Like most enterprise gear, usually designed and supported to work one way with compatibility only one way too, and this kind of information tends to be lacking greatly out there in the open until someone's tried it and talked about it, but even then my 'google-fu' as people like to call it is absolutely horrendous.

 

[CA_Tallguy]

Did anyone ever try this card out?

 

[mattventura]

So... I bought one of these to try it out, since they were on sale for $35.

I can't get it to work. Tried it on a few different machines (X9, X10, and a Gigabyte board). The card itself shows up fine in lspci, and I can see it enumerated as several devices, as well as the fact that it blocked off the next four bus numbers which is what I'd expect. But drives do not show up, even with SFF-8643 -> U.2 adapters and directly attached drives. Rebooting, hotplugging, rescanning, toggling slot power, nothing works.

Some other posts on the internet reference an oprom where you're supposed to be able to configure it. I tried both legacy and UEFI, but no oprom. I've got a cheapo AliExpress switch on the way, hoping that works better. Maybe I just got a bad one, but I kind of figured that $35 for a PCIe switch card was too good to be true.

 

[Canuckfire]

By chance did you find out what type the actual switch chip was from the lspci output?
I am watching some of these and am tempted to try them with solaris just to see what they show up as.

 

[BeTeP]

The card uses IDT 89H32H8G3

 

[mattventura]

I'd poke around in the PLX device editor (part of the PLX SDK), but the editor requires a license, and I'm pretty sure you can only get the license by buying the chips.

 

[shiin03]

so, this card didnt do passthrough?. i'd like to have them since i'm running out of PCIe lanes but maybe i should pay more and get aliexpress PEX8747 card instead?

 

[Canuckfire]

The card uses IDT 89H32H8G3

They're under Renesas now and the product page has Windows software listed that i am assuming is an SDK or an interface.

If it is actually available then i will see if I can reach out through work and see if I can get it running.

Guess I will be ordering one from ebay too.

 

[Canuckfire]

Well, I was able to download the software and it looks like it is just a PCIE device browser with some management options over SMBUS.
(super simple software but needs to be installed in Win7 mode (32 bit) and run as an Administrator)
I am not sure if I can 'share' software but there was not any disclaimer on the Renesas site once I made an account.

I will order a card and a pair of cheap intel NVME drives and see if the software just needs to write to the card and put it in a different mode; there looks like there is an onboard eeprom or flash device that should hold configuration.

I am waiting for a review to see if I can download the full manuals and development kit documentation.

*EDIT* I think that I will probably just try this under Solaris or Illumos/Openindiana to see if their software already has the required components.

 

[trixie]

There is an upstream linux driver for the management interface of Switchtec chips over so-called MRPC. More: here GitHub - Microsemi/switchtec-user: Userspace code for the Microsemi PCIe switch is a nifty tool for monitoring, management, and benchmarking that relies on said kernel driver.

Unfortunately, for my "oracle" board this management PCI endpoint is not enumerated, and, as a result, I cannot bind the driver to it. The card is only visible as a bridge from one to four ports behind it.

This management interface is likely disabled somehow within the EEPROM configuration.