This is so much fun.
For a beginner like me, can you please provide a minimal parts list to get started? So of course the X10QBi motherboard. But how many memory boards do you need? Can you start with one and still operate with multiple CPUs? Do you need the official SuperMicro power supply or is any other one working?
I had to look up what a BMC board is and found this nice article.
Explaining the Baseboard Management Controller or BMC in Servers
But what I don't get if you can run a setup without a BMC?
I feel like a little kid again figuring out all kind of new stuff
If I have to create a new thread for these questions, just let me know.
im happy to clarify any questions that might not be immediately obvious, though i think your questions about the cpus and memboards has been already addressed in the previous pages. there is alot of valuable information there. i really suggest having a thorough read of its entirety before you buy any parts.
im pretty sure you can have as many memboards as you want? but there are 2 memboards per cpu and the cpus without memboard will not be active (someone refute this if im wrong). the x10qbi manual from supermicro has a comprehensive map-out of which memboard slots belong to which cpu. so you need at least 4 memboards if you want all 4 cpus to be running.
@jpk has noted before that you probably can't start with only one cpu; you need 2, 3 or 4. and it could be because the board is technically 2 dual sockets combined
I wanted to post an update here.
I did get things working. It was similar to
@gmaxwheel 's experience - I got a controller card from the seller that had had the IPMI password reset and I was able to flash the bios from in that, and now it boots!
Some of my observations about the system:
CPU/memory stuff
- After from the long initialization before you get to the bios, it behaves a lot like a regular server.
- That long initialization time is actually the memory cards - if you take out the memory cards (or possibly just don't populate them) the lines for those empty cpus stay at 00:00 instead of going through the BA:0F etc.
- It does recognize the CPUs in the system that don't have any memory cards - they are shown like regular to the OS and are present in the bios.
- It will boot with only 2 CPUs, as long as the second CPU is in either socket 3 or 4 - just having them in sockets 1 & 2 doesn't work. It also works with 3 processors (I only tested with sockets 1,2,3, but I don't see why it wouldn't work with 1,3,4 or 1,2,4) But it does require that socket 1 is filled. It kind of behaves like this might be two dual socket boards put together, and each set of sockets needs to have a CPU to boot fully.
pci-e stuff
- it happily boots off an NVME M2 stick on a pci-e card off a pci-e slot attached to CPU3/4
- I don't know why they put pci-e slots in between the center memory boards, because even with the extremely low profile card I got (only slightly bigger than the NVME M2 stick) the heatsinks on the memory cards nearly touch the memory card next to it, and so the card won't fit.
Noise/power
- noise level isn't bad, but it is definitely louder than my other 846 (4U) machine. The four 92mm fans in the middle won't run any slower than just over 6k rpm (and the three 80mm fans that are supposed to pull air over the middle memory cards run at about the same speed). I'm pretty susceptible to that kind of noise, and there's no way I could have it in the same room as me for a long period of time.
- you can run it off of a single power supply (assuming you don't use more power than a single one can supply)
- It runs at about 320W idle booted into linux with 4 CPUs, 8 memory cards and 16x 16GB dimms.
- Each set of two memory cards seems to take about 40W at idle - running only 2 CPUs + a pair of memory cards it idled just under 200W, and with 2 pairs of memory cards, it's idling just under 240W
Weird stuff
- It does not think my E7-8857 CPUs have hyper-threading (the line for HT is simply not present in the bios) which seems really weird. Has anyone else seen this before?
- Does anyone have any experience putting different CPUs in different sockets in these? I'm wondering if it might be possible to have something different in sockets 3/4... I know from past experience with older dual-socket systems (x56xx), you could put mismatched CPUs in as long as they had the same amount of L2 cache (and I think same wattage needs)
Does anyone have some spare E7 v3/4 CPUs they would be interested in selling or lending me? I'd like to make sure that my system works well with v3/4 CPUs before I spend a lot of money on some larger ones.
Next things I want to look at are memory performance on only one card versus two - that might open up a couple of slots in the middle a little bit.
as for power supply, the motherboard has standard ATX connectors that you typically find on consumer PSUs, with the exception of special power connectors such as SATA-DOM. i would not recommend getting a consumer PSU instead of the 1620W redundant power supply that is designed for it, because
to operate 4 sockets, you need 8 EPS (8-pin) power connections. This is too much for many typical consumer power supplies, as the system routinely draws 1200W+ power. that being said, you can probably get away with a 1600W+ power supply. but then the question is, where are you going to put it??? the supermicro case for the x10qbi (CSE-848X) does not have room for a consumer ATX power supply. if you're thinking of putting it into another case other than the CSE-848X, there are no other cases that will fit the x10qbi. it's a chungus motherboard that measures 19" x 17" (48.26cm x 43.18cm). I recommend being sensible and just using the supermicro case intended for the x10qbi, unless you have easy access to metal modding... well in that case I would like your contact details because I have some mod jobs to be done
and no, the system will not boot without the BMC card. it should come with all prebuilt and pretested systems. the only time when it doesnt is when you're buying a barebones system.
**MINIMAL COMPONENTS**
- the mobo
- cpu heatsinks
- enough memboards and memory for the number of CPUs you want to use. as usual, make sure the DIMMs are the same type for each channel
- sufficient power supply
- if you use a supermicro power supply, make sure the case contains a power distribution board on the ground floor.
- if you plan to use discrete graphics or other devices that require power supply, the power distribution board has 8 modular connectors at your disposal.
- if you plan to use the 24 drive bays, you need a SAS backplane + SAS/raid controller. you can also use the onboard SATA powers, although cable management will be difficult because you also need to route SATA power.
- BMC card - this has onboard LAN and single-output VGA graphics
- fans for the midplane (92mm)
- strongly recommend rear fans to help suck air out (80mm)
exerpt from official supermicro website (
4048B-TRFT | 4U | SuperServer | Products | Super Micro Computer, Inc.):
