Dual Xeon-D

[itronin]

Looking at the traces, I don't see any clear connections between the FPGA and anything else. Any chance we can get a pictures of the back of the board too?

------

The mezzanine connectors are weird too. Just following a few of their pins makes me think they're pinned out for the standard OCP 2.0 mezzanine interface, but the headers are *backwards*. The connectors would be pointed towards the CPUs if you install mezz card that fits the OCP standard.

Resin blobs:
BU8_ probably is BU8_2, the other blob is probably BU8 just covered by the resin. Two nodes, two blobs. BIOS Chip is labled BU7.
That follows the label standard on the board. Node one is just the label. Node two has _2 appended.


Picture of motherboard back:
Let me see if I can get a similar clear picture of the back as I did of the front. Maybe later today as I still have the lights and camera stand on my bench.

OCP:
I don't know a whole lot about OCP - Is this just a mezz PCIe connector standard?
Are there ribbon cable adapters to present an OCP as a PCIe, esp. open ended PCIe slot? (thinking GPU if you wanted?)
Assuming physical space is not an issue can you take any OCP mezz adapter and use it? For example a Quanta 3008 or an OCP NIC?

 

[andreathing]

The thing that really interests me is the two epoxy blobsright above the Artix FPGA.

One of them seems to be labeled "BU8_<something>", the other either isn't labelled or the epoxy is covering the label.

I can't think of any reason there'd be a directly bonded IC on a motherboard like this. I wonder if it actually covers a normal IC and is a security thing. That would also maybe indicate the FPGA is doing some sort of operation which requires protected key material (encryption?).

Looking at the traces, I don't see any clear connections between the FPGA and anything else. Any chance we can get a pictures of the back of the board too?



------

The mezzanine connectors are weird too. Just following a few of their pins makes me think they're pinned out for the standard OCP 2.0 mezzanine interface, but the headers are *backwards*. The connectors would be pointed towards the CPUs if you install mezz card that fits the OCP standard.

I do not think you would be able to follow traces: that board is probably 8 layers PCB...

Resin blobs:
BU8_ probably is BU8_2, the other blob is probably BU8 just covered by the resin. Two nodes, two blobs. BIOS Chip is labled BU7.
That follows the label standard on the board. Node one is just the label. Node two has _2 appended.


Picture of motherboard back:
Let me see if I can get a similar clear picture of the back as I did of the front. Maybe later today as I still have the lights and camera stand on my bench.

OCP:
I don't know a whole lot about OCP - Is this just a mezz PCIe connector standard?
Are there ribbon cable adapters to present an OCP as a PCIe, esp. open ended PCIe slot? (thinking GPU if you wanted?)
Assuming physical space is not an issue can you take any OCP mezz adapter and use it? For example a Quanta 3008 or an OCP NIC?

Based on my understanding OCP is a physical specification, so what you get is basically just PCIe. The only breakout adapter i could find is this, maybe it can be used in conjuction to a PCIe ribbon cable.

 

[zjr8]

I am wondering if the bios versions are different. WHen I use AMIBCP and check the version it says Major =1 and minor =03
There was another dump (not @rachet06 ) which shows Major =1 and minor =2 and the one you worked on shows 00.18

00.16 are the sticked version slapped on both mine. Did I get stuck with the old bios lol I guess I’ll have to try and ask one of you guys nicely to dump a newer one off your boards lol

 

[itronin]

00.16 are the sticked version slapped on both mine. Did I get stuck with the old bios lol I guess I’ll have to try and ask one of you guys nicely to dump a newer one off your boards lol

same here for the sticker version.

 

[tp1]

thats what I have as well.

same here for the sticker version.

On another note, has anyone tried sticking a M.2 and seeing if its recognized? I have and its not recognizing it though I have a 180gb HP which might be really old.

 

[zjr8]

thats what I have as well.


On another note, has anyone tried sticking a M.2 and seeing if its recognized? I have and its not recognizing it though I have a 180gb HP which might be really old.

I have a nvme m2 drive in and it works

 

[rachet06]

In case anyone is looking to further pimp their board(s), I've seen a fair number of OCP NICs floating around, but a (presumably compatible) drive controller recently popped up.

ASRock Rack M3008 PCI-e x8 2x HD Mini SAS SFF-8643 12Gbps Mezzanine Card | eBay

Find many great new & used options and get the best deals for ASRock Rack M3008 PCI-e x8 2x HD Mini SAS SFF-8643 12Gbps Mezzanine Card at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I've got one on the way and can report back once it gets here.

 

[itronin]

In case anyone is looking to further pimp their board(s), I've seen a fair number of OCP NICs floating around, but a (presumably compatible) drive controller recently popped up.

ASRock Rack M3008 PCI-e x8 2x HD Mini SAS SFF-8643 12Gbps Mezzanine Card | eBay

Find many great new & used options and get the best deals for ASRock Rack M3008 PCI-e x8 2x HD Mini SAS SFF-8643 12Gbps Mezzanine Card at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I've got one on the way and can report back once it gets here.

Looking at the mezz picture and the motherboard picture it looks like the mezz standoff holes line up with appropriate holes on the motherboard too.

 

[zjr8]

I have a Quanta 3008 that I incorrectly order for a different OCP board that was listen in a similar style -> just a “mezzanine card” but once it arrived it turned out to be a plain PCIe mezz card (QCT) and not an OCP mezz -> this auction looks to actually be an OCP mezz from the photo price is worth a gamble I think

 

[tp1]

There is another flash chip right next the FPGA and in enclosed in a covered socket. It seems to be a MX25L25635FMI-10G. Curious to see what the dump of that chip consists of, might be to bootstrap the FPGA? Does anyone have a reader that can read that chip and do a dump?

 

[zjr8]

There is another flash chip right next the FPGA and in enclosed in a covered socket. It seems to be a MX25L25635FMI-10G. Curious to see what the dump of that chip consists of, might be to bootstrap the FPGA? Does anyone have a reader that can read that chip and do a dump?

My board doesn’t boot w/ it removed but my programmer claims it’s empty .... will try again tomorrow with flashrom instead of my USB programmers crummy windows app...

 

[fake-name]

Resin blobs:
BU8_ probably is BU8_2, the other blob is probably BU8 just covered by the resin. Two nodes, two blobs. BIOS Chip is labled BU7.
That follows the label standard on the board. Node one is just the label. Node two has _2 appended.

Hmm, interesting that there's two of them, that would suggest they're for the CPUs, not the FPGA, since there's just one FPGA.

It also indicates they're probably some sort of storage. It's still interesting that they bothered to try to epoxy-blob them.

I do not think you would be able to follow traces: that board is probably 8 layers PCB...

Only if they routes all the signals on internal layers, which they don't seem to be doing for other stuff (the OCP mezz connector has at least 1/2 the pairs on the top layer.

Even if we can just see part of the routed connections, seeing where they go could be interesting.

There is another flash chip right next the FPGA and in enclosed in a covered socket. It seems to be a MX25L25635FMI-10G. Curious to see what the dump of that chip consists of, might be to bootstrap the FPGA? Does anyone have a reader that can read that chip and do a dump?

Even if you can dump it, you can't do anything with the dump. FPGA bitstreams are undocumented, and potentially encrypted. There's been some work reverse engineering a few lattice bitstream formats, but nothing for modern xilinx parts.

And that's assuming the flash encryption isn't used. The FPGAs have OTP for an encryption key that can be programmed when the flash is encrypted, making them damn near impossible to reverse (you'd have to decap the IC to get the key, I think, unless someone finds a side-channel).

My board doesn’t boot w/ it removed but my programmer claims it’s empty .... will try again tomorrow with flashrom instead of my USB programmers crummy windows app...

It's probably the FPGA boot prom. The fact that the FPGA needs to be programmed to start the board is interesting. That means it at least has power sequencing control of the Xeons.

 

[rachet06]

Hmm, interesting that there's two of them, that would suggest they're for the CPUs, not the FPGA, since there's just one FPGA.

It also indicates they're probably some sort of storage. It's still interesting that they bothered to try to epoxy-blob them.



Only if they routes all the signals on internal layers, which they don't seem to be doing for other stuff (the OCP mezz connector has at least 1/2 the pairs on the top layer.

Even if we can just see part of the routed connections, seeing where they go could be interesting.



Even if you can dump it, you can't do anything with the dump. FPGA bitstreams are undocumented, and potentially encrypted. There's been some work reverse engineering a few lattice bitstream formats, but nothing for modern xilinx parts.

And that's assuming the flash encryption isn't used. The FPGAs have OTP for an encryption key that can be programmed when the flash is encrypted, making them damn near impossible to reverse (you'd have to decap the IC to get the key, I think, unless someone finds a side-channel).



It's probably the FPGA boot prom. The fact that the FPGA needs to be programmed to start the board is interesting. That means it at least has power sequencing control of the Xeons.

The bottoms of the boards aren't super interesting, I put some photos up here.

 

[fake-name]

The bottoms of the boards aren't super interesting, I put some photos up here.

I dunno if I'd agree. I'm seeing length-matched traces from the FPGA to both the CPUs. I'm not sure what the interface is, since it doesn't look like differential pairs.

I still think there's a possibility the FPGA is doing some sort of crypto thing, but I suppose it could also be clock distribution or platform initialization.I'm not sure what the bringup processes for modern CPUs looks like.

 

[rachet06]

The M3008 OCP card came in today, I had to rig together some standoffs to feel good about the mounting but it dropped right in and was detected in Ubuntu without any drama.

 

[craig5571]

got my board today, and its 12x13 and it won't fit in any of the cases I have. so I was looking for what cases people are using? ( i really dont want to spend alot of the case if possible) alot of cases claim to fit a motherboard of this size.. but some of the reviews are not so confidence enducing.
would like to hear from someone, who actually has the board installed somewhere.

also does the memory need to be populated in pairs per node? ( eg.. 2 sticks for each node, for a total of 4)

also what cable is being used to access the serial console? and what is the minimum power supply i could get away with here?.
thanks for all the help in advance.

 

[itronin]

got my board today, and its 12x13 and it won't fit in any of the cases I have. so I was looking for what cases people are using?
also does the memory need to be populated in pairs per node? ( eg.. 2 sticks for each node, for a total of 4)

also what cable is being used to access the serial console? and what is the minimum power supply i could get away with here?.
thanks for all the help in advance.

case:
Not installed yet so IDK - I have a CSE-826 with the split nvme /A type backplane in a box I'm going to try when I get time.

memory:
I have fired it up with one dimm and it worked. Dual channel so optimal will be 2 stick min per node.

console cables:
embedded serial uart to usb bridge, so an A to B usb 2.0 cable will work fine (did on my macbook and a windows box). BIOS showed up here for me.

The stacked RJ45's are serial as previously reported by @tp1 ttys0 in Linux so makes me think that's com1 on the node.

per @tp1 "3. Got Unraid booted up on one of the nodes, works well. The RJ45's are serial cables, any standard Cisco RJ45 to USB should work. Was able to see unraid console output on these."

power supply:
I used an old seasonic 330W with some 4-8 pin splitter/adapters and powered each cpu using only a 4pin connection, plus the mainboard ATX. Powered each node with: 4x32GB PC4-2400T memory, 120m fan for the cpu, a SATA SSD, and an pm961 plugged into each node.

idle thought:
wondering if we need to build a FAQ and/or start a thread in the CPU/Motherboard Forum with a couple of reserved posts at the top for START HERE.

 

[craig5571]

thanks for the fast response, for the power supply is this what you used (or something similar)

20cm CPU 8-Pin to Dual CPU 8 Pin (4+4) PSU Power Supply Extension Cable | eBay

Find many great new & used options and get the best deals for 20cm CPU 8-Pin to Dual CPU 8 Pin (4+4) PSU Power Supply Extension Cable at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I like the idea of a FAQ and/or start a thread in the CPU/Motherboard Forum with a couple of reserved posts at the top for START HERE.

 

[Scott Laird]

I'm also putting mine in a CSE-826, although I'm not intending to use any of the drives. I'd use something smaller, but I don't have a 1U 12x13 case handy. Unfortunately, the CPU power cables are too short on mine, so I'm stuck waiting for Amazon to deliver extension cables. Although I guess I could splice longer wires into the existing harness, but I don't really think it's worth the work.

I'd love to see a cheap, simple 1U chassis that these would fit in. Bonus points if I could dremel out access to the OCP slots.

 

[tp1]

I use this, though it is a little unreliable. Occasionally needs the computer rebooted for access.

Amazon.com: Cisco Compatible Console Cable, 6ft, FTDI USB to RJ45 Console Cable/Windows 7, 8 / Vista/MAC/Linux / RS232 Switch Router: Computers & Accessories

Amazon.com: Cisco Compatible Console Cable, 6ft, FTDI USB to RJ45 Console Cable/Windows 7, 8 / Vista/MAC/Linux / RS232 Switch Router: Computers & Accessories

www.amazon.com

also what cable is being used to access the serial console?
thanks for all the help in advance.