Dual Xeon-D

craig5571

Member
May 31, 2020
60
6
8
i just moved the cmos pins on both nodes to clear the config..
this message came up from the usb console..on node 0

ERROR: Class:3000000; Subclass:50000; Operation: A

then the message went away and it went to the normal press f2 to get into the bios..


i still can get nothing from the rj45 console ( im a cisco ccnp and a wireless engineer..) i used secure crt at 9600, 19200, 38400, 57600 , 115200 and absolutely nothing.. the cable is a trendnet su-9 prolific usb to serial cable connected to a standard cisco blue flat console cable. this works fine in my hp switch..

its probably .. something im doing or not doing.. or dont know how to do..

but can someone confirm the rj45s do send output , i dont mind working out the details.. but if wont work.. that will just make me crazy..

thanks

oh and i tried glc-t sfp on the board that is a cisco sftp to copper 1 gig transciever.. same result . lights on the switch but nothing else..
also tried a 10g sfp+ to coppper to a cat 6a cable . to switch. switch shows 10 gig link light.

everything is back to working after clearing the cmos.. but this is with the batteries removed , with the batteries installed ( they are brand new) i just get "3" on the led...

so im sorta good at the moment..

are the usb ports 3.0 or 2.0?

hope this info will help someone else.
 
Last edited:

itronin

Well-Known Member
Nov 24, 2018
644
371
63
Denver, Colorado
i still can get nothing from the rj45 console ( im a cisco ccnp and a wireless engineer..) i used secure crt at 9600, 19200, 38400, 57600 , 115200 and absolutely nothing.. the cable is a trendnet su-9 prolific usb to serial cable connected to a standard cisco blue flat console cable. this works fine in my hp switch..


its probably .. something im doing or not doing.. or dont know how to do..
You said you are using ESXI - did you install ESXI by booting off the BIOS and running the installer or did you install on another server and bring the disk over? I'm pretty sure you have to explicitly configure ESXI to use a serial port for the console.

With these boards I think there's a chicken and egg process. You might consider pulling your dual node esxi boot disk install it in a system with a head on it and then follow the process to configure it for serial console and then move the boot media back.

I've been researching setting up proxmox to boot and run diskless (pxe boot and then NFS). Finding "fun time" to do this has been challenging with everything else I have going on. too many work and personal projects mid-flight at the moment.
 

bob_dvb

Active Member
Sep 7, 2018
147
67
28
Not quite London
www.orbit.me.uk
i just moved the cmos pins on both nodes to clear the config..
this message came up from the usb console..on node 0

ERROR: Class:3000000; Subclass:50000; Operation: A

then the message went away and it went to the normal press f2 to get into the bios..


i still can get nothing from the rj45 console ( im a cisco ccnp and a wireless engineer..) i used secure crt at 9600, 19200, 38400, 57600 , 115200 and absolutely nothing.. the cable is a trendnet su-9 prolific usb to serial cable connected to a standard cisco blue flat console cable. this works fine in my hp switch..

its probably .. something im doing or not doing.. or dont know how to do..

but can someone confirm the rj45s do send output , i dont mind working out the details.. but if wont work.. that will just make me crazy..

thanks

oh and i tried glc-t sfp on the board that is a cisco sftp to copper 1 gig transciever.. same result . lights on the switch but nothing else..
also tried a 10g sfp+ to coppper to a cat 6a cable . to switch. switch shows 10 gig link light.

everything is back to working after clearing the cmos.. but this is with the batteries removed , with the batteries installed ( they are brand new) i just get "3" on the led...

so im sorta good at the moment..

are the usb ports 3.0 or 2.0?

hope this info will help someone else.
They are USB 2.0 ports as far as I can tell.

The Rj45 ports are only serial ports for the machine, as itronin says, if the OS isn't using them then they are not doing anything.

The easiest way I found to install an OS was to use a M.2 to PCIe adapter and plug in a cheap low-profile GPU to that. This gave me complete console in the VGA/DP sense.
 

craig5571

Member
May 31, 2020
60
6
8
You said you are using ESXI - did you install ESXI by booting off the BIOS and running the installer or did you install on another server and bring the disk over? I'm pretty sure you have to explicitly configure ESXI to use a serial port for the console.

With these boards I think there's a chicken and egg process. You might consider pulling your dual node esxi boot disk install it in a system with a head on it and then follow the process to configure it for serial console and then move the boot media back.

I've been researching setting up proxmox to boot and run diskless (pxe boot and then NFS). Finding "fun time" to do this has been challenging with everything else I have going on. too many work and personal projects mid-flight at the moment.
i did a scripted install of esxi, its very easy.. i just modified the ks.cfg file i used this for a reference


with the scripted install, you just insert the usb, and wait.. it works automagically..

thank you so much for the help..

I do have another question , would one of these nodes make a good TRUEnas 12 NAS?
the horsepower is there , just wondering if there was any gotchas?

and is anyone utilizing the m.2 slot in addition to a SAS OCP card ... i have an asrock rack m3008x8 sas card and it installs just above the m.2 slot. my only concern with using both of them is the heat from the SAS card.. but with proper cooling it should be ok i think..

and i actually use a box fan on low setting on top of the case.. this makes the case ultra quiet and provides great cooling..
 

itronin

Well-Known Member
Nov 24, 2018
644
371
63
Denver, Colorado
and is anyone utilizing the m.2 slot in addition to a SAS OCP card ... i have an asrock rack m3008x8 sas card and it installs just above the m.2 slot. my only concern with using both of them is the heat from the SAS card.. but with proper cooling it should be ok i think..
I'm hoping to take advantage of both the OCP (asrock 3008) and the underlying m.2 slot by using something like this.
 
  • Like
Reactions: bob_dvb

dswartz

Active Member
Jul 14, 2011
494
49
28
Stratus used to or maybe still do make some special systems that all code is executed in 2 sets it cpu and memory and compared for consistent result and high availability. But Xeon-D I have never heard of anything like this. Interesting fine for sure !
yeah, that was called 'pair and a spare'. comparators and such on each board. any miscompare, board goes offline. there were 2 of each such board, and they had to run in lockstep too. some pretty complicated juju was involved.
 

bob_dvb

Active Member
Sep 7, 2018
147
67
28
Not quite London
www.orbit.me.uk
Okay, that's three sold of the two I had spare because someone else got there before you @craig5571, but as I bought four and I am not sure I need two myself, I will sell you two and cut it there.

If I get more feedback, I could order more, but that will take a few weeks to procure from China, not intending to make a living on this!
 
  • Like
Reactions: craig5571

craig5571

Member
May 31, 2020
60
6
8
Challenge accepted... At least for the OCP part. Not going to insert a photo of my crimes against GPUs. View attachment 15682
i found this nic on ebay, it looks like it would work in the ocp slots on the board.. they appear to be reversed as compared to the cx341a, but the cost is too much... Intel Ethernet Network Connection OCP X557-T2 (x557t2ocpg1p5)
 

craig5571

Member
May 31, 2020
60
6
8
I managed to fit a MCX341A-XCGN into Node 0, but I haven't found something to fit Node 1 yet.

Bob
Great idea, I just tried that , with the same card and it works.. doesn't look pretty, but fully functional. i was using usb to ethernet adapters on node 0. which totally work. my board the sfp ports dont wont on either node 0 or node 1. they dont even show up in the bios.

so this trick with the cx341 is very helpful , although it does block the other ocp port from being used.. wish there was an ocp card that fit properly.

i have heard the onboard 8643 connnectors on the motherboard are nvme... is that true?
 

bob_dvb

Active Member
Sep 7, 2018
147
67
28
Not quite London
www.orbit.me.uk
Great idea, I just tried that , with the same card and it works.. doesn't look pretty, but fully functional. i was using usb to ethernet adapters on node 0. which totally work. my board the sfp ports dont wont on either node 0 or node 1. they dont even show up in the bios.

so this trick with the cx341 is very helpful , although it does block the other ocp port from being used.. wish there was an ocp card that fit properly.

i have heard the onboard 8643 connnectors on the motherboard are nvme... is that true?
Glad the CX341 worked for you.

I believe they are, I have bought a miniSAS to U.2 cable but haven't had the time to test it yet and a U.2 to PCIe adapter which is weird but should work.j

I think I've mentioned this before, but I got my GPU working by using an M.2 to PCIe riser with a flexible extender. I suppose the sameness could be achieved with a network card on Node 1 if you could live without the M.2 or use the U.2 instead.

Look for ADT-Link risers for something similar a right angled one from the slot works best., I think I got mine from AliExpress, can look at the model tomorrow.
 
Last edited:
  • Like
Reactions: craig5571

craig5571

Member
May 31, 2020
60
6
8
what i would like to do, is somehow connect the four 8tb SAS drives I have on the system, but if i am using the OCP card of node 1 for the cx341.. is there another way to do it. can I somehow convert the u.2 ( nvme port ) on the board to work with SAS? I'm a bit of a fish out of water here.
 

bob_dvb

Active Member
Sep 7, 2018
147
67
28
Not quite London
www.orbit.me.uk
what i would like to do, is somehow connect the four 8tb SAS drives I have on the system, but if i am using the OCP card of node 1 for the cx341.. is there another way to do it. can I somehow convert the u.2 ( nvme port ) on the board to work with SAS? I'm a bit of a fish out of water here.
My plan was to convert the U.2 to PCIe and then use that with an HBA controller for SATA.
 

bob_dvb

Active Member
Sep 7, 2018
147
67
28
Not quite London
www.orbit.me.uk
Update, I have installed a GPU on the NVMe port using a SFF to U.2 and U.2 to PCIe converter.

I am now able to play with my Node 0 which I have left idle to date. I am also finding that Node 0 is much slower and when I was in the BIOS menus I disabled host bios flash protection, maybe that will help me do more investigation.

Bob
 

craig5571

Member
May 31, 2020
60
6
8
Update, I have installed a GPU on the NVMe port using a SFF to U.2 and U.2 to PCIe converter.

I am now able to play with my Node 0 which I have left idle to date. I am also finding that Node 0 is much slower and when I was in the BIOS menus I disabled host bios flash protection, maybe that will help me do more investigation.

Bob
Bob,
what parts did you use, for "SFF to U.2 and U.2 to PCIe converter"?

Thanks
 

bob_dvb

Active Member
Sep 7, 2018
147
67
28
Not quite London
www.orbit.me.uk
I bought one of these, which wasn't cheap and they delivered it not as SFF but U.2, luckily I hadn't ordered the cable yet.

Then I ordered a cheap SSF cable from Aliexpress.

€ 12,72 34%OFF | 0.5M/1.5Ft Mini SAS SFF 8643 to U.2 SFF-8639 Cable with 15 Pin Female SATA Connector SSD Power Cable Wire 12Gb/S
 

whbeers

Member
Jul 11, 2020
31
33
18
started reverse engineering the fpga a bit after stumbling across an attack (PDF) that looked promising to decrypt the fpga bitstream.

the JTAG pinout seems to be a pretty standard xilinx one (confirmed by removing the fpga on one of my DOA boards with a hot-air gun and tracing connections back to the jtag header):
1 VRef
2 TMS *
3 TCK *
4 TDO *
5 TDI *
6 NC
7 Halt (unneeded)
8 ?
9 Ground
10 Ground
11 Ground
12 Ground
13 Ground
14 Pseudo Ground


* indicates signals necessary for jtag debugging

Pin 14 needs to be pulled low in order to pass the TCK signal through a mux (U6790) between the jtag header and the fpga - a jumper across to the ground next to it did the trick.

To confirm that the fpga was indeed loading an encrypted bitstream, I manually decoded the unencrypted portion of the bitstream read off the BU9 ROM:
Code:
00000120: 0000 00bb                                BUS WIDTH DETECT
          1122 0044                                BUS WIDTH DETECT
          ffff ffff
          ffff ffff
00000130: aa99 5566                                SYNC WORD
          2000 0000                                NOOP
          3003 e001                                WRITE BSPI reg (1 word)
          0000 000c                                BSPI = 0xc (undocumented)
00000140: 3000 8001                                WRITE CMD reg (1 word)
          0000 0012                                CND = 0x12 (BPI/SPI re-initiate bitstream read)
          2000 0000                                NOOP
          3000 c001                                WRITE MASK reg (1 word)
00000150: 8000 0040                                bit mask for write to CTL0
          3000 a001                                WRITE CTL0 reg (1 word)
          8000 0040                                CTL0 = 0x8000040 (use key from eFuse, decryptor enabled)
          3001 c001                                WRITE COR1 reg (1 word)
00000160: 0000 0000                                COR1 = 0x0
          2000 0000 2000 0000 2000 0000            NOOPs
00000170: 2000 0000 2000 0000 2000 0000 2000 0000
00000180: 2000 0000 2000 0000 2000 0000 2000 0000
00000190: 2000 0000 2000 0000 2000 0000
          3001 6004                                WRITE CBC reg, 4 words (128bit)
000001a0: 6554 23cf beac 9e34 9f0a fdb7 f8c6 7e29  CBC IV (encoded)
000001b0: 3003 4001                                WRITE DWC reg, 1 word (decrypt word count)
          0008 5b98                                0x85b98 / 547,736 words
After trying to execute the WBSTAR attack linked above for far too long using bunnie's "jtag-trace" code on an rpi400, I poked around more randomly and was able to read some of the efuse registers. At first I assumed I was getting masked output, given that the key data was all zeros / fuse control register indicated it was possible to read the key + control data.
Code:
$ ./jtag_gpio.py -f jtag.jtg -d
Executing .jtg command file: jtag.jtg
DEBUG:root:start: [<JtagLeg.RS: 2>, '0', '0'] () / 0
DEBUG:root:tms reset
DEBUG:root:start: [<JtagLeg.IR: 1>, '001001', ' id'] (IDCODE) /  id
DEBUG:root:result: 0x35
DEBUG:root:start: [<JtagLeg.DR: 0>, '0000000000000000000000000000000000000000000000000000000000000000', ' '] () /
DEBUG:root:result: 0x362d093
DEBUG:root:start: [<JtagLeg.IR: 1>, '110010', ' dna'] (FUSE_DNA) /  dna
DEBUG:root:result: 0x35
DEBUG:root:start: [<JtagLeg.DR: 0>, '0000000000000000000000000000000000000000000000000000000000000000', ' '] () /
DEBUG:root:result: 0x2a11a69201233e93
DEBUG:root:start: [<JtagLeg.IR: 1>, '110011', ' user'] (FUSE_USER) /  user
DEBUG:root:result: 0x35
DEBUG:root:start: [<JtagLeg.DR: 0>, '00000000000000000000000000000000', ' '] () /
DEBUG:root:result: 0x0
DEBUG:root:start: [<JtagLeg.IR: 1>, '110001', ' key'] (FUSE_KEY) /  key
DEBUG:root:result: 0x35
DEBUG:root:start: [<JtagLeg.DR: 0>, '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', ' '] () /
DEBUG:root:result: 0x0
DEBUG:root:start: [<JtagLeg.IR: 1>, '110100', ' cntl'] (FUSE_CNTL) /  cntl
DEBUG:root:result: 0x35
DEBUG:root:start: [<JtagLeg.DR: 0>, '00000000000000', ' '] () /
DEBUG:root:result: 0xc0
After getting stuck for awhile, I decided to try decrypting it with a null key on a hunch, which entropy analysis and somewhat uninformed manual inspection seems to indicate has done the trick:
1606111066946.png

To decrypt I used another of bunnie's contributions after failing to get the byte order mangling working with my own more compact attempt. Populate a NKY file with a null key and IV (it'll get the proper IV from the encrypted bitstream regardless) and use -d for debugging output - it'll produce a "debug.clr" file with the plaintext bitstream.


So now I have a (probably?) decrypted bitstream and a bit less curiosity after searching for tools to reverse it into a netlist... anyone have tricks up their sleeve?

As a reminder - the two things I'm most curious about for the FPGA are (1) whether it contains any remote management functionality via a shared nic / non-obvious functionality on the serial console or (2) whether it bridges between the two nodes in any interesting way. Otherwise I'm just curious how it works (and mostly I'm just learning the techniques I used to complete everything above :)).
 

Attachments

Last edited: