X10QBI and v3/v4 cpus (e.g. supermicro sys-4048b-trft)

[willo]

An Ebay seller accepted my offer of $30 for four more e7-8880v4s, so I guess I'm building out the bent chassis. (Yeah, that's $.034 per core.) The most expensive thing will be the NVME drive. Free server chassis, $30 procs, 1.5TB of free ram, $20 power cords, $10 pci card, and like $120 for a 2Tb NVME. So it's a $60 server before I add storage.

Power:
The BMC reports around 825 watts (but that box also has a bunch of SSDs and spinning disks. Max ever was 1234 watts.
I'll check the power suck on the bent build when I've got the bits together.

 

[willo]

Learned something else about the old bios:
It appears that NVME support isn't in the older bios either. I cannot even see the pci/nvme card on this box. (It works on my other one with updated bios).
I'll confirm this once I get the thing to boot something I can run ipmitool on.

Currently doing test boots with two cards with memory and the v3 cpu I had on hand from the other box.

 

[zorg]

You can even run multiple NVME SSDs in one slot if you enable bifurcation in BIOS image. Unfortnately it is disabled by default.

 

[willo]

OMG. I forgot how loud these things are.

So I installed Debian on a SATA SDD hanging off my favorite USB-SATA adapter. Booted that, installed ipmitool, reset the user on the BMC, then I successfully upgraded the BIOS via the BMC web ui.

Ram test boot worked - no errors, but I'm only rocking the one v3 cpu right now. So the BIOS is happy with the 1.5TB of ram but I can only see 386GB of ram once the box is booted on the single physical 8890v3. I still have like 10 sticks of 32GB ram in the bin, I'm plotting to just keep them as spares.

After playing with the new bios, I'm still not seeing my nvme drive. After some searching it seems that this is probably due to the v3 cpu I had to use to be able to boot the bios. I suppose this project is sidelined until my E7-8880v4 CPUs show up.

 

[Sean Ho]

you should be able to see the nvme from a booted OS, no? Seeing it in BIOS or being able to boot off it is a function of the BIOS, which on most boards of this vintage did not support NVMe boot. You can mod the BIOS to add NVMe support, or boot off a SATA SSD. Or PXE boot, which is what I do.

Also without CPUs in the other sockets, the BIOS can't see or test the RAM associated with the other CPU sockets. So while it's unlikely there's an issue with the DIMM slots or CPU socket pins, that can't be ruled out yet until you get your set of four matching CPUs.

 

[zorg33]

Have you paid attention to the slot connections?

OMG. I forgot how loud these things are.

So I installed Debian on a SATA SDD hanging off my favorite USB-SATA adapter. Booted that, installed ipmitool, reset the user on the BMC, then I successfully upgraded the BIOS via the BMC web ui.

Ram test boot worked - no errors, but I'm only rocking the one v3 cpu right now. So the BIOS is happy with the 1.5TB of ram but I can only see 386GB of ram once the box is booted on the single physical 8890v3. I still have like 10 sticks of 32GB ram in the bin, I'm plotting to just keep them as spares.

After playing with the new bios, I'm still not seeing my nvme drive. After some searching it seems that this is probably due to the v3 cpu I had to use to be able to boot the bios. I suppose this project is sidelined until my E7-8880v4 CPUs show up.

 

[willo]

That's a good point and it is an error that I almost made. I did remember to make sure it was in CPU1 Slot 1 with just the one CPU.
No mods required to boot NVME on my other x10qbi system with the E7-8880v4 cpus.
That said the SSD I have on hand is an older nvme. My other system has a fresh WD Black in it. Firmware could be a thing there.

Parts should show up tomorrow, I'm not gonna mess with it much more until those arrive.

Have you paid attention to the slot connections?
View attachment 39685

 

[zorg33]

Anyone ever got one of these to run DDR3 at 1600?
I just threw in some Samsung 32GB 14900 4Rx4 LRDIMMs and they still run at 1333.
I have Mem1 1.01 mem boards and only put 1 dimm per channel. I only left 1 cpu (8880V4) in the server with 2 mem boards for testing.

 

[willo]

So Yarp - Once I swapped in a new NVME drive, it now appears under the NVME section of the bios. I didn't test it with a v3 CPU but I don't really care. I'm pretty sure it's just old firmware (or a deadish NVME that I pulled out of my parts bin.)

Debian Bookworm installer now shows the nvme drive as an install option.

Successful NVMEs that I've used with the X10QBI and Bios 6.2a and a $10 pcie to single nvme adapter card:
Western Digital Black SN850X 2TB
Team Group MP44 2TB

Both drive have very good speed and IOPS numbers.
I chose the 2TB sizes over 1TB because it's generally enough for my VMs and test cases.
The bigger size means a longer lifetime as well - bigger drives spread the writes out so wear takes longer. (4tb would be better but I'm being cheap here)

On the mem speed question, I'm seeing the same thing.
# dmidecode --type 17 | grep Speed
Speed: 1600 MT/s
Configured Memory Speed: 1333 MT/s
Speed: 1600 MT/s
Configured Memory Speed: 1333 MT/s
Speed: 1600 MT/s
.
.
.

I'll poke around more, but I don't know if there's even a config option for that. (Using Mem1 1.01 boards. Each with 6x 32GB LRDIMMS)

 

[willo]

Found memory speed under north bridge configuration. I'll see what's up once it finishes booting in like 10 minutes. (Still running a single v4 cpu, need to pop in the rest and breakout my thermal paste here in a bit.)


After reboot, same thing - 1333 MT/s. Probably another option that needs to be set? The manual isn't helpful on this one.

 

[NablaSquaredG]

Anyone ever got one of these to run DDR3 at 1600?
I just threw in some Samsung 32GB 14900 4Rx4 LRDIMMs and they still run at 1333.
I have Mem1 1.01 mem boards and only put 1 dimm per channel. I only left 1 cpu (8880V4) in the server with 2 mem boards for testing.

The X10QBi manual is wrong (the 8048B-TRFT is correct), I argued with Supermicro a while back.

With X10QBi and the Supermicro memory boards, there is only ONE configuration in which the memory will run at 1600.

Only with QRx4 LRDIMMs, running at 1.5V, Maximum 8 per memory board and Lockstep Mode (1:1) will your DIMMs run at 1600.

 

[willo]

Thanks for that, I don't have to dork with settings now. Considering what I'm getting out of these older boxes - I am very OK with that limitation!

 

[willo]

I figure I should note this. I have seen one odd behavior with one of my NVME builds - If I boot it from power off to up, no problems. But if I reboot the system without a power cycle, the drive tosses an error and goes into read only.

 

[zorg]

Found memory speed under north bridge configuration. I'll see what's up once it finishes booting in like 10 minutes. (Still running a single v4 cpu, need to pop in the rest and breakout my thermal paste here in a bit.)

View attachment 39758
After reboot, same thing - 1333 MT/s. Probably another option that needs to be set? The manual isn't helpful on this one.

This only applies to DDR4. I learned this from a Chinese board that had both DDR3/4 slots.

 

[zorg]

The X10QBi manual is wrong (the 8048B-TRFT is correct), I argued with Supermicro a while back.

With X10QBi and the Supermicro memory boards, there is only ONE configuration in which the memory will run at 1600.

Only with QRx4 LRDIMMs, running at 1.5V, Maximum 8 per memory board and Lockstep Mode (1:1) will your DIMMs run at 1600.

Yes, I just had them at 1600 in 1:1 mode. But as it's been discussed here it's much slower than 2:1 @1333