Dual Xeon-D

itronin

Well-Known Member
Nov 24, 2018
567
323
63
Denver, Colorado
They're definitely seeing the value in these... I paid $165 shipped for mine which I gave to a friend to repair when it showed up with a torn off M.2 connector - and others on here got working and complete ones for around $180. I am willing to go around $180-200 but not needing any repairs at all.

That said - I am having a hard time not throwing in an offer for another...
You can always try lower and not use the code.

I'm a big believer in not trying to squeeze the absolute last penny out of someone and being fair.
My calculus is if the board is $230 shipped, you get two nodes, so 1 cpu, motherboard bits, and dual 10Gbe is actually $115.00 dunno about you but to me that seems like a good deal. Assuming it all works and there no broken bits of course.

I definitely got the impression that the seller sees value in these boards now..

To be clear I have no interest in or relationship with the ebay seller other than being a customer and buying 4 boards total now.
 
  • Like
Reactions: Navy_BOFH

int0x2e

Member
Dec 9, 2015
86
52
18
41
2 @ 200 was rejected.
2 @ 275 CAD + S&H got accepted very quickly using the code (not sure if auto accepted or seller was just responsive).
275 isn't a steal, but I guess it's still alright if you want to experiment with clustering and don't mind the fuss of a weird board...

Thanks for sharing the deal, @itronin !
 

craig5571

Member
May 31, 2020
59
6
8
the cpu heatsinks on these boards are kinda not the best , they measure 60x60x25mm . I took both of them off and cleaned off the old thermal paste. it wasn't in good shape. going to reapply with artic silver 5.

if i wanted to get a better cpu heatsink that was all copper (i have heard that works better) does anyone have any reccomendations?
and should i stay passive? the case is a 1u server case. so space is at a premium. trying to keep this as quiet as possible.

found this thread , that talks about 1u CPU heatsinks post #242 is really detailed...

thanks
also, is there anywhere in the bios of the motherboard to control fan speed?
 
Last edited:
  • Like
Reactions: bob_dvb

fake-name

Active Member
Feb 28, 2017
153
115
43
70
So mine showed up.

There's a bunch of weirdness going on. I think someone has been monkeying with these. The mobo I got had two different BIOS revision ICs on it (one is L0.16 and the other is L0.20), and one of them was installed backwards!

Flipping the backwards BIOS IC around, and both nodes at least POST, but that was super weird.

Interestingly, the node with the backwards BIOS IC still had the status LED showing "3", so it's possible that's what's going on with yours @bob_dvb.


-----


Further notes: BIOS L0.20 boots substantially faster then L0.16, but seems otherwise similar. I swapped the bios flash ICs, and both nodes still came up fine, so I've now reflashed the 0.16 one to 0.20.

I'm attaching both BIOS variants. You'll have to remove the trailing .zip to open the archive (the forum requires one of a whitelisted set of file names, and has size limits).

The RJ-45 port is apparently ttyS0, while the USB serial port is ttyS1. You can skip needing to have a rj-45 -> serial adapter by specifying console=ttyS1 when doing installation.
 

Attachments

Last edited:

bob_dvb

Active Member
Sep 7, 2018
145
65
28
Not quite London
www.orbit.me.uk
Good news @fake-name, both my nodes boot but one has broken/non-working 10GbE. When I get back from my break I might see what bios versions I have.

My "3" problem was a power cabling issue in the end. Although weirdly my cheap 600W PSU struggles to start the board, I think the peak current at boot drains the PSU caps too far. It "bounces" three times before starting properly and works fine after that. Wondering if I need a better PSU or just accept the funny start.
 

zjr8

New Member
Jun 3, 2020
23
12
3
Hope everyone is doing well - been swamped at work so havent been around much - I actually also have a board bob with non-working 10Gbe and its running .20 will have to look into that a little bit more..... never noticed the different versions as nobody updated the stickers lol but it does show on the serial console during boot derp.... Anyone done much research into BIOS image diffing ? Im guessing UEFI tool the images / make a list of drivers and versions etc and compare differences? There is probably a better way but not my area of expertise


Trailing back a few weeks to NVME issues - all my boards seem to kernel panic on writing an NVME m2 drive unless bios settings are change from default.

Extended Tag -> Auto , Max Read Request - Leave as hw , Latency Tollerance off

Those were the three I narrowed it down to.... if anyone else wants to go further let me know if they figure out which one of them it is specifically lol I stopped after I got nvme working again
 
Last edited:
  • Like
Reactions: bob_dvb

fake-name

Active Member
Feb 28, 2017
153
115
43
70
I'm running off a Kingston A2000 NVMe m2 drive without any BIOS changes.

I'm having issues with the fact that it seems the x552 network interface doesn't work with any 1gb copper transcievers.

Looking at the ixgbe source, this seems intentional: torvalds/linux

Annoying.
 

ske4za

Member
Feb 4, 2019
77
37
18
I'm running off a Kingston A2000 NVMe m2 drive without any BIOS changes.

I'm having issues with the fact that it seems the x552 network interface doesn't work with any 1gb copper transcievers.

Looking at the ixgbe source, this seems intentional: torvalds/linux

Annoying.
Intel makes that clear on the drivers they provide too, unfortunately.

NOTE: Devices based on the Intel(R) Ethernet Connection X552 and Intel(R)
Ethernet Connection X553 do not support the following features:
* Energy Efficient Ethernet (EEE)
* Intel PROSet for Windows Device Manager
* Intel ANS teams or VLANs (LBFO is supported)
* Fibre Channel over Ethernet (FCoE)
* Data Center Bridging (DCB)
* IPSec Offloading
* MACSec Offloading

In addition, SFP+ devices based on the Intel(R) Ethernet Connection X552 and
Intel(R) Ethernet Connection X553 do not support the following features:
* Speed and duplex auto-negotiation.
* Wake on LAN
* 1000BASE-T SFP Modules
 

zjr8

New Member
Jun 3, 2020
23
12
3
Soooo I may have a sickness and grabbed another one of these..... but I think I must have the worst luck alive because I got one without bu7 / bu7_2 and bu9 present lol.... I had a spare BU9 (FPGA ROM) from the last one but no spare bios roms...

Cant seem to find an exact match for the flash thats currently on my other boards.... Looking at the BIOS dumps does the VSCC table imply that I can use any rom that matches the table?

Code:
Flash chips in VSCC table:
1F4700h
EF3016h
BF2541h
1F4700h
1F4501h
1F4600h
C22014h
C22015h
C22016h
C22017h
C22018h
9D7F7Ah
BF258Eh
BF2541h
208015h
EF3014h
EF3016h
EF3017h
EF4017h
EF4018h
EF6017h
EF6018h
C22018h

Alternatively if I think I dislike myself enough it looks like I can get the same flash I have currently but only in an SOIC8 package and solder it to a dip8 adapter like this one..



But I feel like thats probably the stupid way to solve my issue
 
Last edited:
  • Like
Reactions: craig5571

zjr8

New Member
Jun 3, 2020
23
12
3
Tried a few things to revive my weirdly acting probably broken network ports..... so far not much progress but intels bootutil for firmware update detects the adapters and will write the latest firmware... will try on one of my working adapters to confirm it doesnt break any functionality (update found here) the adapters just seem to drop off the face of the earth as soon as you plug an SFP+ DAC into them... DAC cable works on the other AK-D1541's I got... Interestingly the firmware update utility continues to see the nic once they drop out of bios / os when a cable is plugged in.
 

zjr8

New Member
Jun 3, 2020
23
12
3
In the real world can you tell a difference?
LOL it makes my epeen feel smaller on synthetic benchmarks whats this real world you speak of

Unrelated side note is anyone aware of a way to make ethtool -E take a bin file written out by ethtool -e ? Can only seem to use it to change offsets and not restore an entire dump... diffed the dumps off my working and non working SFP+ ports and found a few differences besides MAC addresses... I guess I could try writing each change a value at a time worse case? Can post dumps from working / non working adapters if anyone else wants to get their eyes on them.
 
Last edited:

int0x2e

Member
Dec 9, 2015
86
52
18
41
So, I got one board that's in good condition and one that won't boot. Both nodes on the bad board hang are stuck showing "3" with all NIC LEDs turned on. Fans do spin.

Here's what I tried so far:
  1. Both boards are hw revision 1.03 and came with "L0.16" bios stickers.
  2. The good board seems to work fine with the exact same 4 rdimms and power wiring that fails on the bad board.
  3. Checked all jumpers - they seem to all be set the same way across the two boards.
  4. Checked bios chip installation orientation - seemed to be fine on both boards.
  5. I tried swapping bios ICs (BU7 + BU7_2) between the boards - the ones from the bad board work fine on the good board the ones from the good board don't help the bad one boot. I suspect it's not a bad flash.
  6. I then tried swapping the FPGA rom (BU9) between the two boards, and again - the board from the bad chip boots fine on the good board, and the one from the good board doesn't make the bad board behave any differently. Again - I assume this means the flash image is not the issue.
  7. I'm pretty sure the issue is linked to the FPGA, but I'm not sure what exactly is wrong.
Next steps:
  1. Dump all the bios and FPGA rom chips I have to see if anything seems off
  2. Try the "020" bios revision linked above,
  3. Having said that - I'm not very hopeful these would help. Does anyone have any advice on anything else I should try? Or should I just write this one off?
 
Last edited:

Fritz

Well-Known Member
Apr 6, 2015
2,251
548
113
66
So, I got one board that's in good condition and one that won't boot. Both nodes on the bad board hang are stuck showing "3" with all NIC LEDs turned on. Fans do spin.

Here's what I tried so far:
  1. Both boards are hw revision 1.03 and came with "L0.16" bios stickers.
  2. The good board seems to work fine with the exact same 4 rdimms and power wiring that fails on the bad board.
  3. Checked all jumpers - they seem to all be set the same way across the two boards.
  4. Checked bios chip installation orientation - seemed to be fine on both boards.
  5. I tried swapping bios ICs (BU7 + BU7_2) between the boards - the ones from the bad board work fine on the good board the ones from the good board don't help the bad one boot. I suspect it's not a bad flash.
  6. I then tried swapping the FPGA rom (BU9) between the two boards, and again - the board from the bad chip boots fine on the good board, and the one from the good board doesn't make the bad board behave any differently. Again - I assume this means the flash image is not the issue.
  7. I'm pretty sure the issue is linked to the FPGA, but I'm not sure what exactly is wrong.
Next steps:
  1. Dump all the bios and FPGA rom chips I have to see if anything seems off
  2. Try the "020" bios revision linked above,
  3. Having said that - I'm not very hopeful these would help. Does anyone have any advice on anything else I should try? Or should I just write this one off?
Time is money.
 

rachet06

New Member
May 27, 2020
15
12
3
Miami, FL
WD SN520 NVMe Gen3 X2 - I see them in a linux distro I *believe* it was also visible in bios but I can get the board back out and check currently messing around with fixing one that had the dead ROM.... SFP ports are working after cloning the FPGA rom of another board... just have to re-attach battery connector to see if it fixes the serial console giving a usb error on the second node where its detached.... board status lightly weirdly indicate its booting thought
Hope everyone is doing well - been swamped at work so havent been around much - I actually also have a board bob with non-working 10Gbe and its running .20 will have to look into that a little bit more..... never noticed the different versions as nobody updated the stickers lol but it does show on the serial console during boot derp.... Anyone done much research into BIOS image diffing ? Im guessing UEFI tool the images / make a list of drivers and versions etc and compare differences? There is probably a better way but not my area of expertise


Trailing back a few weeks to NVME issues - all my boards seem to kernel panic on writing an NVME m2 drive unless bios settings are change from default.

Extended Tag -> Auto , Max Read Request - Leave as hw , Latency Tollerance off

Those were the three I narrowed it down to.... if anyone else wants to go further let me know if they figure out which one of them it is specifically lol I stopped after I got nvme working again
@zjr8, its your SSDs causing that. I have SN520s running on some of these and other motherboards that need latency tweaks due to them going to a lower power state than most systems expect. You can tweak the latency allowances like you did in the bios or pass a boot parameter to linux to allow longer latencies while the drive transitions back up as seen here
 

bob_dvb

Active Member
Sep 7, 2018
145
65
28
Not quite London
www.orbit.me.uk
I have been away and so I haven't managed to revisit my board in August.

I did order an OCP Mezz 10G MCX341A-XCGN, hopefully that will get it connectivity and it was only £13.99 with free postage from HK so no matter what happens I won't easily regret that purchase. Unless you all tell me the MCX341A is garbage!

Also an NVMe to PCIe adapter has arrived, so I can play with that as well. I might use that to get Proxmox working without bothering with a custom install.

I am otherwise stumped as to what to do with my failed NICs.