X10QBI and v3/v4 cpus (e.g. supermicro sys-4048b-trft)
- Thread starter gmaxwheel
- Start date
-
- Tags
- 2011-1 quad socket x10qbi
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Good to know, at least. I would hope that a previous-gen CPU would work if you had success with a v3.
- 32GB 4Rx4 1333 LRDIMMs (Samsung M386B4G70DM0)
- 8GB 2Rx4 1333 LRDIMMs (Samsung M393B1K70QB0)
- 4GB 2Rx8 1600 LRDIMMs (Samsung M393B5273DH0).
here's he post codes file you need:
I was looking through that last night, and 0x79 seems to correspond to CSM initialization. I’m not sure how I can do anything about that with 0 peripherals attached and no ability to get into BIOS, though.
No PCIe cards, nothing attached to USB (not even a keyboard).
Keyboard attached to USB1, mashing the Del key as soon as I power it up all the way through memory training and subsequent POST sections, no dice. Still hangs on 79 and reboots.
Well, it sounds like the memory is good.
If you have time, I'd get a v3 cpu. Mine was magical and got things going. After the bios was updated, I was able to swap in other CPUs.
The supermicro page is odd and calls out v3 and v4, but then referenced v2 as well lower down.
What I do know is that these things are picky. If you look at the INTEL arc pages for the v2 and v3 there are some significant differences and they could be the problem. The x10qbi really pushes the boundaries of those architectures, the price for that is highlighted by the compatibility issues.
Both of mine were bootable with the v3, then I was able to reset the BMC password using ipmitool, then update the bios, then run v4 cpus.
If you have time, I'd get a v3 cpu. Mine was magical and got things going. After the bios was updated, I was able to swap in other CPUs.
The supermicro page is odd and calls out v3 and v4, but then referenced v2 as well lower down.
What I do know is that these things are picky. If you look at the INTEL arc pages for the v2 and v3 there are some significant differences and they could be the problem. The x10qbi really pushes the boundaries of those architectures, the price for that is highlighted by the compatibility issues.
Both of mine were bootable with the v3, then I was able to reset the BMC password using ipmitool, then update the bios, then run v4 cpus.
I’ve ordered a set of e7-8890 v3s that should hopefully be here soon enough to give me ~2 weeks before the return window closes. Fingers’ crossed, I would really like to just get this system working haha.
Quick update here in case anyone comes across the same symptoms. Ended up being some kind of issue with either the motherboard (‘s hardware) or BIOS/UEFI - 4 sets of CPUs (E7-4680 v2, E7-8890 v3, E7-8890 v4, and ES QFTG) all failed to boot past the CSM init step, but all succeeded on a different motherboard with the same RAM and memory risers.
I may see about flashing the BIOS manually (hopefully with my CH341A) to determine if it was some sort of BIOS corruption. If that is the case I’ll report back here.
I may see about flashing the BIOS manually (hopefully with my CH341A) to determine if it was some sort of BIOS corruption. If that is the case I’ll report back here.
This seems to be the most authoritative thread on this system, so I'll leave this here for others to find. I recently bought a used SYS-8048-TR4FT, which appears to be the newer variant with the MEM2 memory boards. When I received it, it came with 4x E7-4850v4 CPUs, and BIOS v3.1 (2018/06/12). I had no issues booting straight away, and was quickly able to configure remote access via IPMI.
I currently have it configured without any 3.5" drives at all, and instead got a simple PCIe 3.0 x4 NVMe adaptor, and connected a Samsung 990 Pro with Heatsink. I've installed an OS and haven't had any issues booting from the NVMe drive. I also purchased 2x PWS-1K28P-SQ 1280W PSUs, and replaced the 4x PWS-1K62P-1R 1600W PSUs that it came with. I have 4x 32GB RDIMMS installed; one in each of the 4x MEM boards; the other 4 MEM boards have been removed to improve startup speeds.
The noise, however, is a problem. At stock configuration, it is 90dB at a distance of 2ft. There are three fan profiles that can be selected via the IPMI interface: "Standard", "Full" and "Heavy IO". The 90dB measurement was using the "Full" profile. Changing the profile to "Standard" improves things substantially, reducing the noise to ~60dB.
I've done a bit of research and confirmed what others have here, that the
1. GitHub - brightshine/X10QBI: Debug PWM Fan control for X10QBI
2. Supermicro X8 - Fan control (smx8fancontrol)
3. GitHub - alexx427/smx8fancontrol: Supermicro X8 - Fan control
4. https://forums.servethehome.com/ind...9-x10-x11-fan-speed-control.10059/post-315632
(4) Applies as it turns out that the X10QBi appears to use the same nct7904 fan controller, which (1) helped me identify.
What I found in reference (1) is the guts of these findings, but (2) and (3) helped me make some sense of what it was actually doing. Essentially it is possible to modify the fan speed duty cycle as follows:
To illustrate, to set the fans to 15% PWM:
This worked! This has reduced the 4x 92mm midplane fans to 2000rpm, and the 3x exhaust fans in the rear to 2300-2500rpm. The noise is now reliably 50dB - a significant improvement, and didn't require any hardware modifications. This is still a little louder than what I would like it to be when sitting next to it in my home office, but it is tolerable.
CPU temps at idle are hovering at approx 30C, and power consumption at ~275W with an ambient temp of ~17C.
I haven't applied any serious workloads to this yet, so I don't know how sustainable this will be, but I thought it would be useful to dispel the findings that fan speeds on this board weren't modifiable. Happy days! I'm probably going to take inspiration from some of the existing fan control scripts out there and write a PID controller that monitors all of the interesting temperatures and scales this duty cycle up and down to keep things in happy ranges.
Edit: a 5% duty cycle drops the midplane rpm's to 1500-1700 and the rear exhaust rpm's to 1600-1700. This has a noise of ~43-45dB. CPU temps at idle are up to ~35C. The MB_10G Temp is up to ~80C as well though (from 50C in the "Standard" profile, ~68C at 15% PWM). The threshold is 95C, so it's getting to be more than I'm comfortable with.
I currently have it configured without any 3.5" drives at all, and instead got a simple PCIe 3.0 x4 NVMe adaptor, and connected a Samsung 990 Pro with Heatsink. I've installed an OS and haven't had any issues booting from the NVMe drive. I also purchased 2x PWS-1K28P-SQ 1280W PSUs, and replaced the 4x PWS-1K62P-1R 1600W PSUs that it came with. I have 4x 32GB RDIMMS installed; one in each of the 4x MEM boards; the other 4 MEM boards have been removed to improve startup speeds.
The noise, however, is a problem. At stock configuration, it is 90dB at a distance of 2ft. There are three fan profiles that can be selected via the IPMI interface: "Standard", "Full" and "Heavy IO". The 90dB measurement was using the "Full" profile. Changing the profile to "Standard" improves things substantially, reducing the noise to ~60dB.
I've done a bit of research and confirmed what others have here, that the
ipmitool raw commands that we see for many other supermicro boards don't work in the same way. Having said that, I found a few obscure references that provided some good guidance on how we might proceed:1. GitHub - brightshine/X10QBI: Debug PWM Fan control for X10QBI
2. Supermicro X8 - Fan control (smx8fancontrol)
3. GitHub - alexx427/smx8fancontrol: Supermicro X8 - Fan control
4. https://forums.servethehome.com/ind...9-x10-x11-fan-speed-control.10059/post-315632
(4) Applies as it turns out that the X10QBi appears to use the same nct7904 fan controller, which (1) helped me identify.
What I found in reference (1) is the guts of these findings, but (2) and (3) helped me make some sense of what it was actually doing. Essentially it is possible to modify the fan speed duty cycle as follows:
Code:
# Put fan in manual mode
## Set T1FMR (Temp1 Fan Mapping Relationships) ~ T4FMR to 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x03 0x00 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x03 0x01 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x03 0x02 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x03 0x03 0x00
## Set T5FMR ~ T10FMR to 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x04 0x00 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x04 0x01 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x04 0x02 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x04 0x03 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x04 0x04 0x00
$ ipmitool raw 0x30 0x91 0x5c 0x04 0x05 0x00
## Set F4OMC (FANCTL4 output mode control) to PWM output.
$ ipmitool raw 0x30 0x91 0x5c 0x03 0x07 0x00
# Query the duty cycle
## 0x10 is the controller that I found worked on my machine
$ ipmitool raw 0x30 0x90 0x5c 0x03 0x10 0x01
# Set the duty cycle
$ ipmitool 0x30 0x91 0x5c 0x03 0x10 0x${duty_cycle}
Code:
$ ipmitool 0x30 0x91 0x5c 0x03 0x10 0x0FCPU temps at idle are hovering at approx 30C, and power consumption at ~275W with an ambient temp of ~17C.
I haven't applied any serious workloads to this yet, so I don't know how sustainable this will be, but I thought it would be useful to dispel the findings that fan speeds on this board weren't modifiable. Happy days! I'm probably going to take inspiration from some of the existing fan control scripts out there and write a PID controller that monitors all of the interesting temperatures and scales this duty cycle up and down to keep things in happy ranges.
Edit: a 5% duty cycle drops the midplane rpm's to 1500-1700 and the rear exhaust rpm's to 1600-1700. This has a noise of ~43-45dB. CPU temps at idle are up to ~35C. The MB_10G Temp is up to ~80C as well though (from 50C in the "Standard" profile, ~68C at 15% PWM). The threshold is 95C, so it's getting to be more than I'm comfortable with.
Last edited:
In my quest to make this thing quiet, I have replaced all of the fans. I have removed the 3x 80mm exhaust fans, and replaced the stock fan wall (4x 92mm fans) with Supermicro CSE-848 Fan Array by taggartalec, coupled with 3x Phanteks T30 120mm fans (they have a better CFM/Noise ratio than the equivalent Noctuas). I'm using the OEM fan shroud which makes a massive difference to cooling capability for CPU1 and CPU4 (approx 7-9C). Preliminary findings (no compute load, at idle):
One other thing worth reporting is that the BMC ipmi device has been a bit flakey. I've had to reset CMOS twice to get BIOS to register the IPMI device/drivers properly. Note that this is more than just removal of the CMOS battery; you also need to short the CMOS pads. I couldn't get it working again until doing this (
- Standard fan profile
- Noise: 38dB (1000 RPM) - practically silent - my N100 board is audible over this
- CPU Temps: 30-38C
- System Temp: 25C
- MB_10G Temp: 57C
- DIMM Temps: ~40C average
- Full fan profile
- Noise: 60dB (3000 RPM) - reasonably loud, wouldn't want this all the time but not uncomfortable
- CPU Temps: 23-27C
- System Temp: 21C
- MB_10G Temp: 44C
- DIMM Temps: ~30C average
One other thing worth reporting is that the BMC ipmi device has been a bit flakey. I've had to reset CMOS twice to get BIOS to register the IPMI device/drivers properly. Note that this is more than just removal of the CMOS battery; you also need to short the CMOS pads. I couldn't get it working again until doing this (
ipmitool was reporting Could not open device at /dev/ipmi: No such file or directory)
Last edited:
hi, im trying to do the same ipmitool fan control commands but they're not working for me. What's your bios and bmc firmware version?
Firmware: 03.65.00
BIOS: 3.1
IPMI: 2.0
Redfish: 1.0.1
I found that it was pretty sensitive to the configuration state - the duty cycle commands wouldn't work unless I executed all of the initial fan mapping relationship commands first, so make sure as a baseline you execute every command above in order.
ah ok, many thanks for that
i spent a couple hours testing yesterday and still no dice with BIOS 3.2
It's been well known that only earlier versions of the BIOS can do fan control, but support was removed in later versions. so it's likely that 3.1 is the last version to still work
I've spent some more time digging into the above commands and understanding the
For the purpose of fan control on the X10QBi, I've learned the following:
This probably still doesn't look perfectly clear. Why do we need to execute all of these? It turns out that commands 1-10 are required to be executed only once to put the controller into manual mode. From the documentation, setting the Temperature Fan Mapping Relationships has two purposes:
Commands 12-15 are where the magic happens. each of FANCTL1-4 controllers are selectable with the values
NOTE: Be careful not to set the fan controller value to
My reflection on this exercise is that I feel like I'm a lot more comfortable with the
It does raise the question about why this (reportedly) no longer works with newer BIOS versions. Maybe they changed the address space, and so
In terms of references that I found useful in this investigation:
ipmitool raw interface generally a bit better.For the purpose of fan control on the X10QBi, I've learned the following:
- The
netfnargument is always0x30 - The
cmdargument is0x91to write data to the interface, and0x90to read data from the interface. This is a command however, so maybe this is better phrased as "write to device" and "read from device" - The remaining arguments are for data:
data[0]=0x5c: I assume this identifies which device to read/write from. I'm not sure where to discover these values, but this appears to refer to the onboard Nuvoton NCT7904D HW monitordata[1]=0x01-0x04: These are the Bank Registers on the device, each of which controls different functionsdata[2]= ??: This can be any bit/byte referring to a specific function in the bank register (refer to docs for functions in each bank register)data[3]= the value to set for the function
netfn=0x30 and data[0]=0x5c are constant, I'll leave those out of the table below but the commands below generally takes the form
Code:
ipmitool raw <netfn> <cmd> <data[0]> <data[1]> <data[2]> <data[3]>| Index | cmd | data[1] | data[2] | data[3] | Description |
|---|---|---|---|---|---|
| 1 | 0x91 | 0x03 | 0x00 | 0x00 | Set T1FMR (Temp 1 Fan Mapping Relationship)=0 (manual mode) |
| 2 | 0x91 | 0x03 | 0x01 | 0x00 | Set T2FMR=0 (manual mode) |
| 3 | 0x91 | 0x03 | 0x02 | 0x00 | Set T3FMR=0 (manual mode) |
| 4 | 0x91 | 0x03 | 0x03 | 0x00 | Set T4FMR=0 (manual mode) |
| 5 | 0x91 | 0x04 | 0x00 | 0x00 | Set T5FMR=0 (manual mode) |
| 6 | 0x91 | 0x04 | 0x01 | 0x00 | Set T6FMR=0 (manual mode) |
| 7 | 0x91 | 0x04 | 0x02 | 0x00 | Set T7FMR=0 (manual mode) |
| 8 | 0x91 | 0x04 | 0x03 | 0x00 | Set T8FMR=0 (manual mode) |
| 9 | 0x91 | 0x04 | 0x04 | 0x00 | Set T9FMR=0 (manual mode) |
| 10 | 0x91 | 0x04 | 0x05 | 0x00 | Set T10FMR=0 (manual mode) |
| 11 | 0x91 | 0x03 | 0x07 | 0x00 | Set FOMC (FANCTL Output Mode Control)=0 (PWM) |
| 12 | 0x91 | 0x03 | 0x10 | 0xFF | Set F1OV (FANCTL1 Output Value) PWM = 100% This command is RW in manual mode only. This is RO in Smart Fan mode The value to provide here is 0x00 (0%) - 0xFF (100%) |
| 13 | 0x91 | 0x03 | 0x11 | 0x7F | Set F2OV PWM = 50% This command is RW in manual mode only. This is RO in Smart Fan mode The value to provide here is 0x00 (0%) - 0xFF (100%) |
| 14 | 0x91 | 0x03 | 0x12 | 0x7F | Set F3OV PWM = 50% This command is RW in manual mode only. This is RO in Smart Fan mode The value to provide here is 0x00 (0%) - 0xFF (100%) |
| 15 | 0x91 | 0x03 | 0x13 | 0x7F | Set F4OV PWM = 50% This command is RW in manual mode only. This is RO in Smart Fan mode The value to provide here is 0x00 (0%) - 0xFF (100%) |
| 16 | 0x90 | 0x03 | 0x10 | 0x01 | Read F1OV |
| 17 | 0x90 | 0x03 | 0x11 | 0x01 | Read F2OV |
| 18 | 0x90 | 0x03 | 0x12 | 0x01 | Read F3OV |
| 19 | 0x90 | 0x03 | 0x13 | 0x01 | Read F4OV |
This probably still doesn't look perfectly clear. Why do we need to execute all of these? It turns out that commands 1-10 are required to be executed only once to put the controller into manual mode. From the documentation, setting the Temperature Fan Mapping Relationships has two purposes:
- Build the relationship between temperature sources 1-10 and FANCTL1-4
- Fan control mode assignment
0x00:Command 11 sets the output mode (FANCTL Output Mode Control) to PWM. The other option is DC, but we want to set a duty cycle, so PWM is what we're after. Each of the fan controllers can be set individually by setting different bits in the byte, but since we want all of them set to PWM, we can use
0x00 here.Commands 12-15 are where the magic happens. each of FANCTL1-4 controllers are selectable with the values
0x10 - 0x13, and the value to pass is the duty cycle represented as an 8bit hex value. What this means is that contrary to the advice I provided above, 100% is represented as 0xFF (integer 255) and **not** 0x64 (integer 100) like it is on some other supermicro platforms. These require the fan controllers to be in manual mode to work, otherwise they are read only values, which is why setting TMFR and FOMC values were required. I suspect having to execute this command so many times is just an artifact of this individual fan controller - I note that similar requests for other systems only require a single command.NOTE: Be careful not to set the fan controller value to
0x0F in any of commands 12-15. I did this whilst experimenting with the interface, and it locked all of my fans at 100%. This survived through BMC warm and cold resets, server reboots, server power off, and even factory resetting the BMC. I resolved this by resetting CMOS and shorting the CMOS pads, but this was a bit of a heart attack moment. After looking at the documentation, I can see that this sets PROCHOT Fan Select, which places the specified FANCTL into "respond to thermal alert event" mode, so in hindsight it makes sense that this ramped up all of my fans to 100%.My reflection on this exercise is that I feel like I'm a lot more comfortable with the
ipmitool raw interface now; it certainly makes a lot more sense from the perspective of identifying the right values to pass through. There are still a few uncertainties (what are the other netfn, cmd or data[0] values?) but it seems like there's enough here to work through fan control, which is probably the most I'll need to engage with ipmitool, since other control is more available through the native interface. It does make me wonder what other capabilities are available through the ipmitool raw commands now that we know we can link into any other bit of hardware on the board given the right addresses and API description - drivers not even required, which I found interesting.It does raise the question about why this (reportedly) no longer works with newer BIOS versions. Maybe they changed the address space, and so
0x5c no longer refers to the Nuvoton NCT7904D HW monitor device (if that's actually what it means)? I suspect that it's something like this, so there may be value in some experimentation by someone able to replicate the results. This is the value I've been able to find the least information on though, so this is entirely speculation.In terms of references that I found useful in this investigation:
- The nct7904 Linux kernel driver code, which confirmed my suspicions around bank register values and also FANCTL1 controller value: linux/drivers/hwmon/nct7904.c at 6146a0f1dfae5d37442a9ddcba012add260bceb0 · torvalds/linux
- The NCT7904D data sheet, which contains a pretty detailed description of the functional API for this bit of hardware: https://www.nuvoton.com/export/resource-files/en-us--Nuvoton_NCT7904D_Datasheet_V17.pdf