Customized PCI Express Gen 3 Carrier Board for 4 M.2 SSD modules

[larryexchange]

bout one year ago, I bought an Amfeltec PCI Express Gen 3 Carrier Board for 4 M.2 SSD modules for my SYS-E300-8D. But later on I found my X10SDV MB can support PCIe Bifurcation. So I decide to replace Amfeltec with a carrier board w/o PLX.

But the problem is at that time I could not find any board meet the following requirements in the market.
1. HHHL size, otherwise it could not fit into my SYS-E300-8D
2. PCIe 3.0 x16 interface
3. Support 4 M.2 NVMe SSD up to 22110 size.
4. Low cost w/o PLX

So I convinced the vendor who help me design Cheap 1U PCIe M.2 Riser design a brand-new board for me. Here is their drafted design.


Different from Amfeltec's design, in order to be more cost effective, vendor put all 4 SSD on one side of the board. Due to HHHL size limitation, vendor uses two layers, 2 SSD each layer.

Today I got the chance to play with the new board. Here is what real produce looks like.


Now I am ready to plug in my 3 SM963 22110 size SSDs into the board.


Here is what it looks like after plug in M.2 SSD.






After plug the board into my SYS-E300-8D and power it on. I can see all the 3 SM953 SSDs from the device manager.


CDI also shows each of the SSD running at PCIe 3.0 x4. The temperature is also good.


The vendor is still optimizing the design. But from my point of view, the bottom line the board already meets all of my requirements the above. Super!

 

[i386]

The temperature is also good.

Can you run "stress tests" on these ssds and then look what temperatures you get?

 

[alex_stief]

I would also expect this layout to be prone to thermal throttling issues under sustained load.

 

[larryexchange]

I would also expect this layout to be prone to thermal throttling issues under sustained load.

I don't worry about that coz I added the 3rd 13Krpm fan into SYS-E300-8D, which is dedicate for the PCIe device.
mini Azure Stack

 

[alex_stief]

Ok that's a different story then

 

[Morphers]

Anyway you can do this for u.2?

 

[Terry Kennedy]

Different from Amfeltec's design, in order to be more cost effective, vendor put all 4 SSD on one side of the board. Due to HHHL size limitation, vendor uses two layers, 2 SSD each layer.

Today I got the chance to play with the new board. Here is what real produce looks like.
View attachment 7337

The vendor is still optimizing the design. But from my point of view, the bottom line the board already meets all of my requirements the above. Super!

I would be very interested in a version of this that had a perforated (vent holes) FH bracket. Ideally the M.2 sockets would be:

Upper (furthest from PCIe connector) bottom (closest to PC board) - PCIe lanes 1-4
Lower (closest to PCIe connector) bottom (closest to PC board) - PCIe lanes 5-8
Upper top - PCIe lanes 9-12
Lower top - PCIe lanes 13-16

That would mean that a single M.2 drive goes in the position with the best airflow and should work even on a system that doesn't support PCIe bifurcation, since the host will only see a single M.2 on PCIe lanes 1-4.

Bonus points for individual activity LEDs on the board as well as jumpers for 4 activity LEDs plus one "combined" activity LED.

While I'm wishing, route the PCIe lanes on the board so that as much material as possible can be milled out (holes) from the board to allow more airflow across the underside of the bottom M.2 boards.

I'm not crazy about the "dog bone" stacking adapter, but as I'd only be using these with 1 or 2 drives, I can live with it.

 

[larryexchange]

I would be very interested in a version of this that had a perforated (vent holes) FH bracket. Ideally the M.2 sockets would be:

Upper (furthest from PCIe connector) bottom (closest to PC board) - PCIe lanes 1-4
Lower (closest to PCIe connector) bottom (closest to PC board) - PCIe lanes 5-8
Upper top - PCIe lanes 9-12
Lower top - PCIe lanes 13-16

That would mean that a single M.2 drive goes in the position with the best airflow and should work even on a system that doesn't support PCIe bifurcation, since the host will only see a single M.2 on PCIe lanes 1-4.

Bonus points for individual activity LEDs on the board as well as jumpers for 4 activity LEDs plus one "combined" activity LED.

While I'm wishing, route the PCIe lanes on the board so that as much material as possible can be milled out (holes) from the board to allow more airflow across the underside of the bottom M.2 boards.

I'm not crazy about the "dog bone" stacking adapter, but as I'd only be using these with 1 or 2 drives, I can live with it.

Great Feedbacks! I definitely will share them with the vendor.

Btw, if you only want to plug in up to 2 M.2 drives, you could find more options in the market. AFIAK, both DELL and SuperMicro have the similar products. Here is an example.

 

[gigatexal]

Benchmarks or it didn’t happen ;-)p

 

[Terry Kennedy]

Great Feedbacks! I definitely will share them with the vendor.

Thanks!

Btw, if you only want to plug in up to 2 M.2 drives, you could find more options in the market. AFIAK, both DELL and SuperMicro have the similar products. Here is an example.
View attachment 7359 View attachment 7360

I'm not sure that is a genuine Dell card, for a couple of reasons:

• The Dell part number on the sticker is bogus - 02NGFF/2NGFF.
• The Dell copyright says 2009, but the PCB fab date on the silk screen is 2015. NVMe wasn't even started until late 2009, with the NVMe 1.0 spec not released until March 2011.
• PCIe 3.0 wasn't released until November 2010, making the idea of a PCIe 3 / NVMe 1.1 Dell board in 2009 pretty implausible.
• I don't see Tripod in Wuxi City, Jiangsu listed as a Dell supplier.
• There are a lot of active components on that board, which I can't make out in detail on that picture. Given the obvious crystal, though, I bet there is a PLX PCIe switch or something similar on there.
• Neither "zoom pcie ssd card 2 device" nor "zoom-pcie-ssd-card-2-device" return any Google matches when quoted to force a literal search.
• The "ventilation" holes on that card are pretty laughable, given the small amount of airflow you get over the bottom of an M.2 stick. Both Dell and Supermicro use brackets with larger vent holes on their cards.
• I'm a little confused by what look like 4 un-populated SMA antenna connector pads with "swoopy" PCB traces, all 4 of which would impinge on any M.2 sticks except for the very shortest. I'm not sure why RF antennas would be needed on a a storage card.

I think what we have here (until proven otherwise) is a "counterfeit" of something that doesn't actually exist in a "real" version.

 

[larryexchange]

I am not 100% sure but personally I think that card is authenticated. I could answer some of questions you have.

• The Dell part number on the sticker is bogus - 02NGFF/2NGFF.
• The Dell copyright says 2009, but the PCB fab date on the silk screen is 2015. NVMe wasn't even started until late 2009, with the NVMe 1.0 spec not released until March 2011.
• PCIe 3.0 wasn't released until November 2010, making the idea of a PCIe 3 / NVMe 1.1 Dell board in 2009 pretty implausible.
• I don't see Tripod in Wuxi City, Jiangsu listed as a Dell supplier.

@Patrick Kennedy reviewed the similar card before. The Dell 4x m.2 PCIe x16 version of the HP Z Turbo Quad Pro . If you looked that quad card, copyright is also 2009. The difference is one is Duo and the other is Quad.

• There are a lot of active components on that board, which I can't make out in detail on that picture. Given the obvious crystal, though, I bet there is a PLX PCIe switch or something similar on there.

​

I can confirm, same as Quad adapter, the Duo adapter doesn't have PLX.

• Neither "zoom pcie ssd card 2 device" nor "zoom-pcie-ssd-card-2-device" return any Google matches when quoted to force a literal search.

​

In the official page below, DELL mentioned:
High-performance storage: Choose the actively cooled Dell Precision Ultra-Speed Drive Duo and Quad PCIe NVMe SSD storage solutions for the fastest and most reliable way to store your content, up to 14xfaster than traditional SATA SSD storage. So you could see DELL have both Duo and Quad version of Zoom PCIe SSD Card.
Dell Precision Tower 7810 Workstation | Dell India

• The "ventilation" holes on that card are pretty laughable, given the small amount of airflow you get over the bottom of an M.2 stick. Both Dell and Supermicro use brackets with larger vent holes on their cards.

​

I managed to replace the bracket w/ one from LSI HBA adapter (9207/9211). Sorry for misleading. ​

 

[Terry Kennedy]

I am not 100% sure but personally I think that card is authenticated. I could answer some of questions you have.

@Patrick Kennedy reviewed the similar card before. The Dell 4x m.2 PCIe x16 version of the HP Z Turbo Quad Pro . If you looked that quad card, copyright is also 2009. The difference is one is Duo and the other is Quad.

The card in Patrick's article is part 80G5N and is from one of Dell's regular suppliers, Foxconn in Tucheng, New Taipei. The PCB fab was subcontracted to Broad Technology in Guangzhou, Guangdong.

I can confirm, same as Quad adapter, the Duo adapter doesn't have PLX.

There are a fair amount of active components on there - none of the file photos are high enough resolution to see what they are.

The board does have the same oddball SMA antenna pads and loopy traces. The red board you show may be a prototype or from a vendor that was not selected to build it.

I managed to replace the bracket w/ one from LSI HBA adapter (9207/9211). Sorry for misleading.

​

You switched to a HH bracket? The 805GN file photos show a normal-for-Dell 2 rows of square holes, like they use on their filler brackets.

 

[larryexchange]

The card in Patrick's article is part 80G5N and is from one of Dell's regular suppliers, Foxconn in Tucheng, New Taipei. The PCB fab was subcontracted to Broad Technology in Guangzhou, Guangdong.

There are a fair amount of active components on there - none of the file photos are high enough resolution to see what they are.

The board does have the same oddball SMA antenna pads and loopy traces. The red board you show may be a prototype or from a vendor that was not selected to build it.

​

You switched to a HH bracket? The 805GN file photos show a normal-for-Dell 2 rows of square holes, like they use on their filler brackets.

Yes, when I was plugging it into my SYS-E300-8D, I had to use a HH bracket. Here are more pictures. I am sorry these are largest resolution pictures I could upload.

 

[Rand__]

Today I got the chance to play with the new board. Here is what real produce looks like.
...
The vendor is still optimizing the design. But from my point of view, the bottom line the board already meets all of my requirements the above. Super!

Very nice work
For the general public some cooling optimization will be important as most won't run this in a very well cooled system to start with.
Regardless, are you planning to make those available like your other adapters? Whats the cost associated with it?

 

[larryexchange]

Very nice work
For the general public some cooling optimization will be important as most won't run this in a very well cooled system to start with.
Regardless, are you planning to make those available like your other adapters? Whats the cost associated with it?

It's still kind of prototype. When it's ready, I believe the vendor would love to sale it in US market. They didn't tell me the price coz the design has not been finalized yet. Personally I don't think there are many people need such card, especially considering most of MBs don't have PLX or support PCIe Bifurcation.

 

[Rand__]

From what I gathered many of the Supermicro X11 (e5) boards have bifurcation support built-in. Also it has been added to some x11(e3) and x10(e5) boards

 

[Evan]

PCIe Bifurcation Is not that uncommon at all, as mentioned most modern CPU’s or rather their boards tend to support it.
Much more efficient way to use PCIe slots if you can put 2 or 4 x NVMe m.2 in a single slot.
In 1u servers or small boxes this is especially important.

Slot7 on x10sdv boards also splits x16 to x8/x8

 

[larryexchange]

I would be very interested in a version of this that had a perforated (vent holes) FH bracket. Ideally the M.2 sockets would be:

Upper (furthest from PCIe connector) bottom (closest to PC board) - PCIe lanes 1-4
Lower (closest to PCIe connector) bottom (closest to PC board) - PCIe lanes 5-8
Upper top - PCIe lanes 9-12
Lower top - PCIe lanes 13-16

That would mean that a single M.2 drive goes in the position with the best airflow and should work even on a system that doesn't support PCIe bifurcation, since the host will only see a single M.2 on PCIe lanes 1-4.

Bonus points for individual activity LEDs on the board as well as jumpers for 4 activity LEDs plus one "combined" activity LED.

While I'm wishing, route the PCIe lanes on the board so that as much material as possible can be milled out (holes) from the board to allow more airflow across the underside of the bottom M.2 boards.

I'm not crazy about the "dog bone" stacking adapter, but as I'd only be using these with 1 or 2 drives, I can live with it.

I discussed your feedback with the vendor. Basically they took all of them.

1. FH/HH bracket with vent holes. They will provide the brackets with vent holes. In addition, I suggest them use the standard bracket. In that way, end users could replace it with the brackets they favorite.

2. They redesign the board as below.


3. They will use a 7-pin connector to show drive activities on the chassis. Meanwhile, they will add 4 LEDs on the board. So user could read the status of the drives from the vent hole of bracket.

4. They will add as many holes as possible on the board to improve cooling.

 

[larryexchange]

Very nice work
For the general public some cooling optimization will be important as most won't run this in a very well cooled system to start with.
Regardless, are you planning to make those available like your other adapters? Whats the cost associated with it?

The current design includes two holes on the board which could be used to install fan. That fan can cover all 4 controllers.

 

[larryexchange]

I just got the v2.