Dual Xeon-D

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
more image detective work - so the mac address mfg id is asrock.
There's two (one per node?) inphi wpcs4227c a2 which if I read the spec sheet each can handle dual 15Gbps (802.3ap) .
SFF-8643 has the label SATA by each. so guessing that's 4 more SATA ports, but why do you need 10 SATA ports for something like this?
 
  • Like
Reactions: bob_dvb and Markess

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
more image detective work - so the mac address mfg id is asrock.
There's two (one per node?) inphi wpcs4227c a2 which if I read the spec sheet each can handle dual 15Gbps (802.3ap) .
SFF-8643 has the label SATA by each. so guessing that's 4 more SATA ports, but why do you need 10 SATA ports for something like this?
That's interesting. Are there many ASRock boards with mac address IDs that aren't ASRock? I wouldn't have even thought to check that. So much people are able to figure out by pictures of components and component placement.
 

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
That's interesting. Are there many ASRock boards with mac address IDs that aren't ASRock? I wouldn't have even thought to check that. So much people are able to figure out by pictures of components and component placement.
well since these may have been custom manufactured for OVH (earlier in the thread) I was wondering who'd show up for the vendor ID. I actually misread it the first time and it was a partial match for Iskratel which is a Solvenian company...

I'm still thinking about the 10 sata ports per node ... and that 6 are normal sata connectors and the other 4 are in an 8643... T you can get 10x hotswap 2.5" in a 1U and up to 25 in a 2U space so maybe one board design to support multiple chassis types for your deployment?

I'm also wondering whether this board takes DDR4 or DDR3? the 1541 can handle either. The ASROCK single cpu board says DDR4 but again custom build special run - YMMV. Just thinking out loud while the parcel moves through the system.
 

bob_dvb

Active Member
Sep 7, 2018
214
116
43
Not quite London
www.orbit.me.uk
To convert the m.2 to pcie...
Also some cheap single port ConnectX3 cards on eBay at the moment, could be a nice addition.

Still wondering, may wait until someone gets one booted, still too risky for my blood.

Note that the D-1541 has 6 SATA connections, so the SFF connectors must Ben attached to a controller on the board.
 

andreathing

New Member
May 5, 2020
23
11
3
well since these may have been custom manufactured for OVH (earlier in the thread) I was wondering who'd show up for the vendor ID. I actually misread it the first time and it was a partial match for Iskratel which is a Solvenian company...

I'm still thinking about the 10 sata ports per node ... and that 6 are normal sata connectors and the other 4 are in an 8643... T you can get 10x hotswap 2.5" in a 1U and up to 25 in a 2U space so maybe one board design to support multiple chassis types for your deployment?

I'm also wondering whether this board takes DDR4 or DDR3? the 1541 can handle either. The ASROCK single cpu board says DDR4 but again custom build special run - YMMV. Just thinking out loud while the parcel moves through the system.
You will be our first motherboard detective extraordinaire
 
  • Like
Reactions: itronin

rachet06

New Member
May 27, 2020
15
12
3
Miami, FL
I think that if you assume product engineering teams don't want to verify any more changes from a known good thing as possible you can make some pretty educated guesses about these.

Guess 1(very confident) - Each node is a slight re-spin of the ASRock Rack D1541D4U-2O8R
The pics of the 10G PHY show that it's an Inphi/Cortina CS4227. Of the few commercial options ASRock Rack has for the D1541 only the D1541D4U-2O8R uses that part. The commercial board exposes all 6 SATA ports, 2 M.2, and adds an LSI 3008.
The commercial M.2 ports are adjacent to the PCIe slots and the lower one is set back a bit. On the mystery board the M.2 is near what appear to be OCP slots and on the bottom node there appears to be an unpopulated second M.2 header below and set back from the populated one. My assumption is that whatever engineer was tasked with building the mystery board started by just slapping 2 of their existing designs together, changing the PCIe slot out for an OCP, and removing whatever unneeded parts didn't fit anymore.
That doesn't explain the 8643 plugs, the LSI3008 is a pretty big chip and the mystery board doesn't have any big heatsinks for it.

Guess 2 (very confident) - Each node is totally independent
Building a custom board for a datacenter based around a low-power SoC is kinda weird except for one sorta niche usecase. They let you offer totally discrete dedicated servers with reasonable power for a pretty modest power usage. OVH lists dedicated servers in their locations based on the D1541, so business-wise I can't see any other reason to go through the cost of a custom system except that. Also, in the ebay photos you can kinda make out 2 headers labeled PORT80_A1 and PORT80_B1, which I'm assuming are for connecting LPC motherboard debuggers. If these were a shared system I don't think they would have two low-level management busses?

Guess 3 (very confident) - RAM is DDR4
I'm guessing these were sold at least in part under the OVH ADV-STOR-2 SKU, which lists DDR4 RAM. The commercial board from before is also DDR4 only.

Guess 4 (pretty confident) - The M.2 are PCIe
I'm guessing these were sold at least in part under the OVH ADV-STOR-2 SKU, which has an option for a 500GB NVMe SSD.

Guess 5 (less confident) - The FPGA is the BMC
If I squint enough, it says its an Artix-7. The Artix-7 is the cheap version of a Kintex-7. The OpenBMC has an existing port to the a Kintex-7 eval board, so maybe if you cut out unneeded features you can make a basic controller for 2 nodes fit?
If it is an OpenBMC-based controller it will at the very least have IPMI support and might go all the way up to KVM and a web interface, which would be neat.
 

andreathing

New Member
May 5, 2020
23
11
3
I think that if you assume product engineering teams don't want to verify any more changes from a known good thing as possible you can make some pretty educated guesses about these.

Guess 1(very confident) - Each node is a slight re-spin of the ASRock Rack D1541D4U-2O8R
The pics of the 10G PHY show that it's an Inphi/Cortina CS4227. Of the few commercial options ASRock Rack has for the D1541 only the D1541D4U-2O8R uses that part. The commercial board exposes all 6 SATA ports, 2 M.2, and adds an LSI 3008.
The commercial M.2 ports are adjacent to the PCIe slots and the lower one is set back a bit. On the mystery board the M.2 is near what appear to be OCP slots and on the bottom node there appears to be an unpopulated second M.2 header below and set back from the populated one. My assumption is that whatever engineer was tasked with building the mystery board started by just slapping 2 of their existing designs together, changing the PCIe slot out for an OCP, and removing whatever unneeded parts didn't fit anymore.
That doesn't explain the 8643 plugs, the LSI3008 is a pretty big chip and the mystery board doesn't have any big heatsinks for it.

Guess 2 (very confident) - Each node is totally independent
Building a custom board for a datacenter based around a low-power SoC is kinda weird except for one sorta niche usecase. They let you offer totally discrete dedicated servers with reasonable power for a pretty modest power usage. OVH lists dedicated servers in their locations based on the D1541, so business-wise I can't see any other reason to go through the cost of a custom system except that. Also, in the ebay photos you can kinda make out 2 headers labeled PORT80_A1 and PORT80_B1, which I'm assuming are for connecting LPC motherboard debuggers. If these were a shared system I don't think they would have two low-level management busses?

Guess 3 (very confident) - RAM is DDR4
I'm guessing these were sold at least in part under the OVH ADV-STOR-2 SKU, which lists DDR4 RAM. The commercial board from before is also DDR4 only.

Guess 4 (pretty confident) - The M.2 are PCIe
I'm guessing these were sold at least in part under the OVH ADV-STOR-2 SKU, which has an option for a 500GB NVMe SSD.

Guess 5 (less confident) - The FPGA is the BMC
If I squint enough, it says its an Artix-7. The Artix-7 is the cheap version of a Kintex-7. The OpenBMC has an existing port to the a Kintex-7 eval board, so maybe if you cut out unneeded features you can make a basic controller for 2 nodes fit?
If it is an OpenBMC-based controller it will at the very least have IPMI support and might go all the way up to KVM and a web interface, which would be neat.
Great findings! I think the 8643 can be a U.2 connector, not a sas/sata one (CPU has 32 PCIe lanes, only 2x4 used for the M.2 slots, assuming those are M.2). This would explain the missing controller since the only disk connection are those directly coming from the CPU.

I totally agree on the two nodes on the same board and on the fact they would totally try to reuse as much of an existing and know working design. I was too lazy to check similarities with other commercially available motherboards though :|

I also didn't squint hard enough and you are totally right: the FPGA is an artix 7, not a kintex 7. Same family but lower tier.

Now I'm so curious about this that i think i will ask the seller for a shipping quote for this side of the Atlantic.
 
  • Like
Reactions: itronin

fake-name

Active Member
Feb 28, 2017
180
144
43
73
Using a FPGA for the BMC is really the one thing I'm dubious about. You can get a Zynq for about the same price with MUCH more performance and still have enough fabric to get weird stuff done. It makes zero sense to use a soft core for much at this point, unless you're doing *super* bizarre things.

You don't need the something as expensive as the Xilinx FPGA to do that. If you look closely, you'll see that the ST3243EC ICs are RS232 transceivers (routed to the 8P8C connectors) and the SIL2104 ICs are USB-UART bridges (routed to the USB type-B ports).
The issue with that is the 8P8C connector is labelled "LAN_1_2". Maybe the didn't update the designator for the 8P8C connector when changing what they're using it for, but that'd be super sloppy, and I don't see a good reason they'd have both RS232 AND USB/Serial interfaces on the same board.
On the other hand, it does look like the 8P8C connectors are routed to the ST3243EC ICs, so ¯\_(ツ)_/¯.

Anyways, I threw in a BO too.
 

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
this item has gone from 1 sold to 10 in the last 24 hours with a lot of OBO hanging out there. 2 left in inventory.

I'm really impressed with all the thought and detective work + knowledge coming out before anyone has one in their hands too!

I'm still thinking the 8643 is SATA - in one of the bay photos you can see all the normal sata ports are labeled and then you can make out SATA_ next to one of the 8643's (bottom right of the photo).

totally concur with reusing the board design and I did see that OVH was offering dedicated servers based on the 1541 - but missed the part about the nvme storage so this is all starting to make sense.

for BMC wouldn't you need two IPMI partial implementations for node specific portions but then back to a single for overall board specific functions? Am wondering if there's nothing OTS that would allow that hence a custom implementation?

On the PCIE lanes:

concur this is likely a respin sooo per node

x8 for the dual 10gb
x4 for the populated m.2
x4 for the unpopulated m.2 header (though didn't see it in the other node I'd not be surprised if it was still allocated)
x16 for the OCP conenctor?

that adds up to 32, but maybe that 8643 is u.2 so there'd be x12 for the ocp?

just thinking out loud here.
 

andreathing

New Member
May 5, 2020
23
11
3
this item has gone from 1 sold to 10 in the last 24 hours with a lot of OBO hanging out there. 2 left in inventory.

I'm really impressed with all the thought and detective work + knowledge coming out before anyone has one in their hands too!

I'm still thinking the 8643 is SATA - in one of the bay photos you can see all the normal sata ports are labeled and then you can make out SATA_ next to one of the 8643's (bottom right of the photo).

totally concur with reusing the board design and I did see that OVH was offering dedicated servers based on the 1541 - but missed the part about the nvme storage so this is all starting to make sense.

for BMC wouldn't you need two IPMI partial implementations for node specific portions but then back to a single for overall board specific functions? Am wondering if there's nothing OTS that would allow that hence a custom implementation?

On the PCIE lanes:

concur this is likely a respin sooo per node

x8 for the dual 10gb
x4 for the populated m.2
x4 for the unpopulated m.2 header (though didn't see it in the other node I'd not be surprised if it was still allocated)
x16 for the OCP conenctor?

that adds up to 32, but maybe that 8643 is u.2 so there'd be x12 for the ocp?

just thinking out loud here.
It doesn't add up unless there's a SATA controller somewhere. The CPU has only 6 directly controllable SATA ports (CPU product brief). I also read OCP specification for mezzanine card: my impression is that x16 cards would require two connectors (A and B, see page 24 of the specifications and a mellanox 1/10/25/40/50/100 Gb/s Ethernet card for an actual implementation)

x8 for the dual 10gb
x8 for the OCP conenctor
x4 for the populated m.2
x4 for the unpopulated m.2 (?)
x4 for the 8643

leaving x4 usable for other (FPGA?) connections.

I'm really hoping the seller accept my offer, otherwise it would be too expensive (shiiping considered) to acquire one just to play with it.
 
  • Like
Reactions: itronin

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,063
1,482
113
I'm still thinking the 8643 is SATA - in one of the bay photos you can see all the normal sata ports are labeled and then you can make out SATA_ next to one of the 8643's (bottom right of the photo).
It's not. They just don't print things consistently on the board due to component placement and text size. There is one label to the left of the SATA ports for port 0, four below for ports 1-4, and finally one above for port 5.
 
  • Like
Reactions: itronin

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
It's not. They just don't print things consistently on the board due to component placement and text size. There is one label to the left of the SATA ports for port 0, four below for ports 1-4, and finally one above for port 5.
@BlueFox and @andreathing my eyes are horrible. You're right. I also see what I think is the other unused / abandoned pad for an m.2 on the top node just above the bottom node's dimm slots., so yeah makes sense the SFF-8643 may be a u.2 to preserve/keep that pcie functionality per node.

Be nice to get this in and on the bench so I can take a nice hires photo and be able to see what's what.

on another note:

Not really seeing whats an obivous pwm 4pin fan header anywhere but there may be a control cable going to whatever chassis this was supposed to be in with fans (hot swap maybe?) plugged into the chassis. spent a little time trying to find an OVH image that might have a suspect system in it no luck.

having to add in fan control though is not really that big a deal.
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,063
1,482
113
@BlueFox and @andreathing my eyes are horrible. You're right. I also see what I think is the other unused / abandoned pad for an m.2 on the top node just above the bottom node's dimm slots., so yeah makes sense the SFF-8643 may be a u.2 to preserve/keep that pcie functionality per node.
That one isn't M.2. The pad is for a MIPI 60 pin connector (second one is above the lower battery). They're generally used for JTAG (amongst other things) and usually only populated on pre-production motherboards for testing/debugging.
 
  • Like
Reactions: itronin

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
That one isn't M.2. The pad is for a MIPI 60 pin connector (second one is above the lower battery). They're generally used for JTAG (amongst other things) and usually only populated on pre-production motherboards for testing/debugging.
LOL - okay I'll stop guessing now. seems mine are mostly wrong anyway. as I've said I'm not a hardware person but I'm glad that there are who are actively participating.
 

Markess

Well-Known Member
May 19, 2018
1,146
761
113
Northern California
So, I'm pretty certain that in no time at all, somebody will crack how to get these running. How do I know? My usual luck, or lack of it, it about to kick in.

I've held off on buying one of these because the smart folks here (and you know who you are) haven't figured it out yet, and I know that if they haven't figured it out yet, then I certainly have no chance! ATM there's only 2 left and 7 pending offers. Once they're sold out, and I can no longer buy one, the solution will be discovered and revealed (and it will probably be something easy to boot!).... because that's just my luck! For those of you who have one/have one on the way....you're welcome ;)
 
  • Like
Reactions: bob_dvb and Fritz