X10QBI and v3/v4 cpus (e.g. supermicro sys-4048b-trft)
- Thread starter gmaxwheel
- Start date
-
- Tags
- 2011-1 quad socket x10qbi
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Well here's my take on this build. I found a deal on a 4048B-TR4FT with the MEM2 risers from TheServerStore and loaded it with E7-8890 v4s. There's 256GB of DDR4 (32x8GB), and I rehomed an Nvidia 3090 from another machine. Sadly, the GPU has an enormous backplate that won't quite fit, so it's going to live in an external enclosure thanks to a few PCIe extension cables - not pretty, but it does the job.
I wanted to keep it quiet, but I really didn't want to go with water cooling since I plan to run it remotely frequently. Dynatron make the R27, which just barely fits in this machine. After a bit of cajoling the final fit of those towers is satisfying! All of the stock fans are swapped with Noctuas and I replaced the stock PSUs with PWS-1K28P-SQs. They are indeed silent - thanks @nickblack! The entire machine is dead quiet, and I was able to -j96 build without overheating.
I'll post a bit more once I get to the OS and other tuning. At present, here are some questions I have about the setup:
I wanted to keep it quiet, but I really didn't want to go with water cooling since I plan to run it remotely frequently. Dynatron make the R27, which just barely fits in this machine. After a bit of cajoling the final fit of those towers is satisfying! All of the stock fans are swapped with Noctuas and I replaced the stock PSUs with PWS-1K28P-SQs. They are indeed silent - thanks @nickblack! The entire machine is dead quiet, and I was able to -j96 build without overheating.
I'll post a bit more once I get to the OS and other tuning. At present, here are some questions I have about the setup:
- I tried configuring two SSDs in RAID0 in the BIOS, but I'm only getting 500MBps when I was expecting them to double up for 1GBps. Am I doing something silly?
- At idle and low load the CPUs run at 3.4GHz as expected, but under full load, they were cut back to 2.8GHz. The temps were only 70C, so I don't think it's thermal throttling. Is the lower core speed expected? Can I override it?
- PWS-1K28P-SQs which each put out 1000W. I have them plugged into smart outlets, and they are each drawing about 500W under load for a total of 1000W that settles down to about 800W on a longer build. That shows the PSUs are balancing the power draw which is great, but I was expecting it to consume even more. The PSUs and cores seem to be capable of more. Any ideas?
Attachments
Does anyone know what the correct cpu socket cover/cap is for the X10QBI. I've tried a bunch of 2011-0 to 2011-3 covers, none seem to fit?
2011-1
LGA2011-1 LGA 2011-1 CPU SOCKET COVER, LGA 2011 CPU SOCKET COVER Protector | eBay
ETA 3 to 6 Weeks.
www.ebay.com
Nice quiet build! I really wish I could do the same with an R930. Did you have to do any soldering to install any of those fans? Also, what is your idle power draw?
Nice posting, a lot of info too. Nice to know there are other heatsink that will fit. Are these dynatrons much more quite than the original SM heatsinks? Did you replace the external fans (the one at the back of the casing, by the power supply/PCIE) too?
I also want to know about the GPU. How did you connect the RTX3090 to the PSU? Are the power cables existing power cables? If so, where are these cables? How do I get to them? Finally, how many 3090 were you able to get in the casing?
Hey @farid and @chrgrose, sorry for the late reply. No, I didn't need to do any soldering. The noctua fans match up well in width, but they are thinner so they sit loosely in the original fan mounts. The Idle power draw is about 500W, at full CPU load it draws about 1000W, and it will top out at 1400W when I spin up the GPU for modeling or graphics on top of that. Conveniently, the max load is just under the limit of my breakers, so I'm able to run the machine on regular wall power. I plug the entire thing into a smart plug so I can monitor the power draw in real time and power on/off remotely.
Yes, @farid, the Dynatrons are much beefier than the original SM heatsinks. They fill up the entire vertical space and allow for mounting 80mm fans. The rear mount fans are also replaced with noctua. I used this configuration so I could swap the very noisy Dynatron fans with noctua ones. One of my goals with the build was to avoid water cooling.
The GPU power cord was the hardest part of the build. The PSU has lots of spare 8pin connectors, but they are positioned deep under the motherboard. I had to remove the entire riser, move a ton of cables out of the way, and then use chopsticks with my phone camera to nudge the cable extension into a port. You can buy something similar to this.
The consumer grade RTX3090 DOES NOT fit inside the machine due to enormous back plane heat sink. I didn't want to hack at my GPU, so I just bought a PCIe extension cable and a cheap Mining rig mount to place on top of the server. The server has four full size PCIe slots, which you could fill up if you use the extension cables. If you're looking for a quiet build, the external solution is nice because you can improve the airflow inside the server and let the GPUs cool off in their own enclosure.
Yes, @farid, the Dynatrons are much beefier than the original SM heatsinks. They fill up the entire vertical space and allow for mounting 80mm fans. The rear mount fans are also replaced with noctua. I used this configuration so I could swap the very noisy Dynatron fans with noctua ones. One of my goals with the build was to avoid water cooling.
The GPU power cord was the hardest part of the build. The PSU has lots of spare 8pin connectors, but they are positioned deep under the motherboard. I had to remove the entire riser, move a ton of cables out of the way, and then use chopsticks with my phone camera to nudge the cable extension into a port. You can buy something similar to this.
The consumer grade RTX3090 DOES NOT fit inside the machine due to enormous back plane heat sink. I didn't want to hack at my GPU, so I just bought a PCIe extension cable and a cheap Mining rig mount to place on top of the server. The server has four full size PCIe slots, which you could fill up if you use the extension cables. If you're looking for a quiet build, the external solution is nice because you can improve the airflow inside the server and let the GPUs cool off in their own enclosure.
Just realized I never posted a picture of the final form with the external GPU. Here it is 
I made a custom glass cover to pass the cables though without cutting the original case. The green lighting is stock, and I really enjoy the glow when it's on.
Btw, @farid and @chrgrose, this build puts out an enormous amount of heat. I live in a high rise, and the server will easily raise the room temperature by 10-15F. I have to open windows, set the AC to max, or just leave the room altogether when it's running. The heat output is something I did not anticipate when I planned the server.
I made a custom glass cover to pass the cables though without cutting the original case. The green lighting is stock, and I really enjoy the glow when it's on.Btw, @farid and @chrgrose, this build puts out an enormous amount of heat. I live in a high rise, and the server will easily raise the room temperature by 10-15F. I have to open windows, set the AC to max, or just leave the room altogether when it's running. The heat output is something I did not anticipate when I planned the server.
Attachments
I finally got one of these X10QBI systems! I almost got one before the pandemic, and then prices went crazy, but better late than never.
I put a single DIMM it in to get it to boot because I heard they are picky about memory, and it did boot but the fans seem to be running way too fast and I'm not sure how to tell why.
If I'm reading this correctly, the midplane fans are running over 9000 RPM and the 3 exhaust fans are over 10,000 RPMs. It sure is blasting a lot of cold air out the back!
Is it because it didn't like the DIMM I used, or do I need to populate more memory boards since it has 4 CPUs? It has no disks and no PCIe cards other than the special I/O module.
I almost got one before the pandemic, and then prices went crazy, but better late than never.I put a single DIMM it in to get it to boot because I heard they are picky about memory, and it did boot but the fans seem to be running way too fast and I'm not sure how to tell why.
If I'm reading this correctly, the midplane fans are running over 9000 RPM and the 3 exhaust fans are over 10,000 RPMs. It sure is blasting a lot of cold air out the back!
Is it because it didn't like the DIMM I used, or do I need to populate more memory boards since it has 4 CPUs? It has no disks and no PCIe cards other than the special I/O module.
Code:
# ipmitool sensor
CPU1 Temp | 21.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 82.000 | 87.000 | 87.000
CPU2 Temp | 21.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 82.000 | 87.000 | 87.000
CPU3 Temp | 23.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 82.000 | 87.000 | 87.000
CPU4 Temp | 24.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 82.000 | 87.000 | 87.000
System Temp | 19.000 | degrees C | ok | -10.000 | -5.000 | 0.000 | 75.000 | 77.000 | 79.000
Peripheral Temp | 21.000 | degrees C | ok | -10.000 | -5.000 | 0.000 | 75.000 | 77.000 | 79.000
PCH Temp | 43.000 | degrees C | ok | 0.000 | 5.000 | 16.000 | 90.000 | 95.000 | 100.000
MB_10G Temp | 47.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 90.000 | 95.000 | 100.000
P1M1 DIMMAB Tmp | 21.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000
P1M1 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P1M2 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P1M2 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P2M1 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P2M1 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P2M2 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P2M2 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P3M1 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P3M1 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P3M2 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P3M2 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P4M1 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P4M1 DIMMCD Tmp | na | | na | na | na | na | na | na | na
P4M2 DIMMAB Tmp | na | | na | na | na | na | na | na | na
P4M2 DIMMCD Tmp | na | | na | na | na | na | na | na | na
FAN1 | na | | na | na | na | na | na | na | na
FAN2 | 9100.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN3 | 9100.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN4 | 9000.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN5 | 9100.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN6 | na | | na | na | na | na | na | na | na
FAN7 | 10800.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN8 | 10300.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN9 | 10700.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN10 | na | | na | na | na | na | na | na | na
Vcpu1 | 1.776 | Volts | ok | 0.544 | 0.576 | 0.608 | 1.952 | 2.000 | 2.096
Vcpu2 | 1.792 | Volts | ok | 0.544 | 0.576 | 0.608 | 1.952 | 2.000 | 2.096
Vcpu3 | 1.792 | Volts | ok | 0.544 | 0.576 | 0.608 | 1.952 | 2.000 | 2.096
Vcpu4 | 1.792 | Volts | ok | 0.544 | 0.576 | 0.608 | 1.952 | 2.000 | 2.096
VMSE_CPU12 | 1.350 | Volts | ok | 1.174 | 1.190 | 1.238 | 1.494 | 1.526 | 1.542
VMSE_CPU34 | 1.350 | Volts | ok | 1.174 | 1.190 | 1.238 | 1.494 | 1.526 | 1.542
1.5VSSB | 1.500 | Volts | ok | 1.324 | 1.340 | 1.388 | 1.644 | 1.660 | 1.692
VTT | 0.936 | Volts | ok | 0.824 | 0.840 | 0.888 | 1.144 | 1.160 | 1.192
3.3V | 3.347 | Volts | ok | 2.771 | 2.819 | 2.963 | 3.539 | 3.635 | 3.683
3.3VSB | 3.299 | Volts | ok | 2.782 | 2.829 | 2.970 | 3.534 | 3.628 | 3.675
12V | 12.000 | Volts | ok | 10.198 | 10.304 | 10.728 | 12.954 | 13.272 | 13.378
VBAT | 3.018 | Volts | ok | 2.634 | 2.730 | 2.826 | 3.834 | 3.930 | 4.026
Chassis Intru | 0x1 | discrete | 0x0100| na | na | na | na | na | na
PS1 Status | 0x1 | discrete | 0x0100| na | na | na | na | na | na
PS2 Status | 0x1 | discrete | 0x0100| na | na | na | na | na | na
PS3 Status | 0x1 | discrete | 0x0100| na | na | na | na | na | na
PS4 Status | 0x1 | discrete | 0x0100| na | na | na | na | na | naHi,
I've got a weird problem recently. I've been running X10QBI servers for some time now, but haven't seen anything like this before.
I had samsung/hynix 10600R DDR3 in my servers that were working fine.
Then I got access to some Micron 14900R DDR3 16GB 2Rx4 modules and they just go through memory test fine, but the server always freezes during OS startup. The symptomes are exactly like HW error interrupts, like the win10 circle stutters more and more until it freezes.
In BIOS there is no issue at all. RAM configuration does not have any effect, I tried every setting possible.
What could this be? And if Micron is incompatible with this board, then how does it pass the memory test?
And I have only modules in the blue slots but they only run at 1333MHz, while they should run at 1600. I tried 2 and 4 modules per board also.
CPUs are 8880v4 QS
Update: win10 boots up fine with only 1 module per CPU.
Currently trying 1:1 mode...still freezes with 2 modules per cpu (channels A & C)
I've got a weird problem recently. I've been running X10QBI servers for some time now, but haven't seen anything like this before.
I had samsung/hynix 10600R DDR3 in my servers that were working fine.
Then I got access to some Micron 14900R DDR3 16GB 2Rx4 modules and they just go through memory test fine, but the server always freezes during OS startup. The symptomes are exactly like HW error interrupts, like the win10 circle stutters more and more until it freezes.
In BIOS there is no issue at all. RAM configuration does not have any effect, I tried every setting possible.
What could this be? And if Micron is incompatible with this board, then how does it pass the memory test?
And I have only modules in the blue slots but they only run at 1333MHz, while they should run at 1600. I tried 2 and 4 modules per board also.
CPUs are 8880v4 QS
Update: win10 boots up fine with only 1 module per CPU.
Currently trying 1:1 mode...still freezes with 2 modules per cpu (channels A & C)
Last edited:
RDIMMs cannot ever run faster than 1333 on the Rev 1.01 DDR3 memboards, as it's a 3 SPC (Slots per Channel) Setup. Only LRDIMMs can run at 1600