X10QBI and v3/v4 cpus (e.g. supermicro sys-4048b-trft)

[jpk]

I purchased one of these from a well known ebay reseller in the southwest, and I figured I'd share my experiences thus far.
First was shipping. It was ground Fedex and Slow. (Fedex has been really bad about saying stuff is going to arrive a day earlier than their initial estimation, and then just having it sit idle at the last distribution center for a full day before delivering it a day late. (to say nothing of the trip around the whole west coast that one particular package took)). The big box finally got here and I got it set up on the workbench, got E7-8857 v2 CPUs installed and 16x 16gb dimms I had in another machine and went to plug in the VGA and realized the reseller had pulled the card out D-: I contacted them and offered to pay for expedited shipping, and they were gracious about it and shipped it out via Fedex ground 9_9
It finally got here a day late and I put it in and the machine powered up, pretty as you please!

The initializing process is kinda cool - you can watch each CPU going through it's testing stuff, but when I tried to get into the bios, it was password protected. D-: Luckily, a friend passed on the SoftLayer bios password that has been going around, and I figured it was worth a shot, and it worked! So I got into the bios, removed the password, did an restore optimal settings, and rebooted, it went through the initializing routine, and reset, and started the initializing routine again, and again, and again.

And that's where I'm stuck right now. I've disabled the BMC, LAN, Watchdog, cleared the CMOS and pretty much everything else I can think of, I've pulled memory boards out of everything except CPU1 (and even pulled the memory boards out of CPU1 when they were still in for CPU2 - it was cool that that worked, but ultimately it didn't help). The IPMI is of course password protected, and I don't have a way to reset that.
I'm pretty certain I got the CPUs nicely seated (this isn't my first complex server rodeo), and all the memory boards seem to be working normally.

So, yeah, any advice for me? It seemed to be working before I did the reset bios settings, but I can't see just that causing it to go into an initializing loop...

 

[PabloChakon]

Hey guys I am putting this system together and im looking into well.... not wasting money if not needed. I was wondering what is the difference between the lets say CSE-848X X10QBI and any other X10QBI server? The more I look the more I dont see a reason sellers are putting v1/2 or v3/4 xeon listings, I might be wrong but even this thread reading thru it gave me that feeling.

If I buy one,since I only have any knowledge in X10DAI-O boards and that is limited to the 6 systems I have (I havethe QBIs as well but they are put together by a 3rd party guy and he moved due to corona and financial opportunity elswhere) I am in trouble maintaining the systems we have and upgrading to more. They seem to be all working fine for now but im trying to learn as much as possible down the road since I dont have any experts to call anymore.

We have V3s and v4 systems built, but im not sure what the real differences in setting it up are.

If anyone would be kind enough to help out on this thing i would be willing to pay for some help in building it and starting it up over skype or something if anyone is interested?

Hope you bros have a great day, btw I love these systems since the day we got our first one I just dont know a lot about them

 

[jpk]

I wanted to post an update here.
I did get things working. It was similar to @gmaxwheel 's experience - I got a controller card from the seller that had had the IPMI password reset and I was able to flash the bios from in that, and now it boots!

Some of my observations about the system:

CPU/memory stuff

• After from the long initialization before you get to the bios, it behaves a lot like a regular server.
• That long initialization time is actually the memory cards - if you take out the memory cards (or possibly just don't populate them) the lines for those empty cpus stay at 00:00 instead of going through the BA:0F etc.
• It does recognize the CPUs in the system that don't have any memory cards - they are shown like regular to the OS and are present in the bios.
• It will boot with only 2 CPUs, as long as the second CPU is in either socket 3 or 4 - just having them in sockets 1 & 2 doesn't work. It also works with 3 processors (I only tested with sockets 1,2,3, but I don't see why it wouldn't work with 1,3,4 or 1,2,4) But it does require that socket 1 is filled. It kind of behaves like this might be two dual socket boards put together, and each set of sockets needs to have a CPU to boot fully.

pci-e stuff

• it happily boots off an NVME M2 stick on a pci-e card off a pci-e slot attached to CPU3/4
• I don't know why they put pci-e slots in between the center memory boards, because even with the extremely low profile card I got (only slightly bigger than the NVME M2 stick) the heatsinks on the memory cards nearly touch the memory card next to it, and so the card won't fit.

Noise/power

• noise level isn't bad, but it is definitely louder than my other 846 (4U) machine. The four 92mm fans in the middle won't run any slower than just over 6k rpm (and the three 80mm fans that are supposed to pull air over the middle memory cards run at about the same speed). I'm pretty susceptible to that kind of noise, and there's no way I could have it in the same room as me for a long period of time.
• you can run it off of a single power supply (assuming you don't use more power than a single one can supply)
• It runs at about 320W idle booted into linux with 4 CPUs, 8 memory cards and 16x 16GB dimms.
• Each set of two memory cards seems to take about 40W at idle - running only 2 CPUs + a pair of memory cards it idled just under 200W, and with 2 pairs of memory cards, it's idling just under 240W

Weird stuff

• It does not think my E7-8857 CPUs have hyper-threading (the line for HT is simply not present in the bios) which seems really weird. Has anyone else seen this before?
• Does anyone have any experience putting different CPUs in different sockets in these? I'm wondering if it might be possible to have something different in sockets 3/4... I know from past experience with older dual-socket systems (x56xx), you could put mismatched CPUs in as long as they had the same amount of L2 cache (and I think same wattage needs)

Does anyone have some spare E7 v3/4 CPUs they would be interested in selling or lending me? I'd like to make sure that my system works well with v3/4 CPUs before I spend a lot of money on some larger ones.

Next things I want to look at are memory performance on only one card versus two - that might open up a couple of slots in the middle a little bit.

 

[Raziel]

I don't know why they put pci-e slots in between the center memory boards, because even with the extremely low profile card I got (only slightly bigger than the NVME M2 stick) the heatsinks on the memory cards nearly touch the memory card next to it, and so the card won't fit.

Because there's a setup in which you only use one memory card per CPU and that leaves all PCI-E slots usable (although not for dual-slot cards or such), sacrificing some memory for those. Also, maybe with some very compact and flexible risers you might be able to fit SSDs or something hanging around in there even with memory cards?

As for not detecting HT, that's weird. Mine does see the HT fine on E7-4880v2. Unfortunately I don't have any other CPUs to play around with, so can't test that further, or mismatched CPUs (although I wouldn't really recommend that to begin with)

I was wondering what is the difference between the lets say CSE-848X X10QBI and any other X10QBI server?

Generally the only relevant difference would be the version of the memory cards. Servers that come with v3 or v4 CPUs should be running DDR4 and therefore -mem2 cards. Servers running v1 or v2 should be running DDR3 and -mem1 cards. In my market sleuthing I haven't really seen any DDR4 X10QBis being sold, they seem to be a lot less common. Based on testing in this thread it's been shown that v3/v4 CPUs have been running even with DDR3 and -mem1 cards on X10QBi though.
Also, some servers might have a different riser backplane too - the two you can find seem to be the older SAS2 direct attach (BPN-SAS-846A), and the SAS3 expander (BPN-SAS3-846EL1). I'd recommend the SAS3 expander, and it's not that cheap to get separately, going for over 300$ as far as I'm seeing.

 

[jpk]

Because there's a setup in which you only use one memory card per CPU and that leaves all PCI-E slots usable (although not for dual-slot cards or such), sacrificing some memory for those. Also, maybe with some very compact and flexible risers you might be able to fit SSDs or something hanging around in there even with memory cards?

Right, I did know of that setup, somehow it was separate from the slots in my head X-D
I like the idea of a flexible riser, but even then, I don't think getting all eight memory cards in there will work. I might look into that if I do see a big difference in memory performance with all 8 cards over 4.

As for not detecting HT, that's weird. Mine does see the HT fine on E7-4880v2.

Hmm. I've tried reflashing the bios a couple times, but might try once more with feeling (and from DOS) and see if that changes anything.
FWIW, Linux does show HT as a flag in the CPU, but only shows 12 threads.
Because you can run these with only 2 CPUs, I might just go buy a pair of difference CPUs and see if they have HT, that way I can see if its the motherboard or the CPUs

Unfortunately I don't have any other CPUs to play around with, so can't test that further, or mismatched CPUs (although I wouldn't really recommend that to begin with)

I know mixing CPUs isn't a good idea, but I used to have an SGI Challenge with different speed CPUs, and I've always wanted to have that again. X-D I'll probably give it a try, and if it blows up, I'll edit this post and remove this bit. I mean, let you know :-D

...the difference between the lets say CSE-848X X10QBI and any other X10QBI server?

It looks like there are four different models that use the X10QBI motherboard:
SYS-8048B-TRFT
SYS-8048B-TR4FT
SYS-4048B-TRFT
SYS-4048B-TR4FT
the SYS-8* servers have 24x 3.5" drives, and the SYS-4* have (up to) 48x 2.5" drives
The -TR4FT servers have the MEM2 cards , and the -TRFT ones have the MEM1 cards

What @Raziel said about the backplanes is true too - the one I got came with the SAS3 expander (even though it wasn't advertised as such)

 

[jpk]

I just wanted to report that I picked up 2x E7-8891 v4 CPUs and they are working perfectly in the system - hyperthreading and everything! Yay!
I also ordered a pair E7-4870 v2 to see if I can mix-n-match CPUs.
What fun!

 

[DieBlub]

Hey everybody,
I've been interested in these kind of systems for a while. Especially, for the high core counts and DDR3 shenanigans. I got one question though - how much GPU can you actually cram inside this thing? Mostly concerned with number of slots/depth/width.

That would make it a great all-in-one box. For storage/work/ and LAN parties. So has anyone tried to get like 4 decently powerful GPUs in there? I think somebody mentioned using it as a central editor workstation box? How is that working out?

 

[tconrado]

@gmaxwheel do you think that 4x e7-8890 v3 + 32 x 16GB DDR3 would work? @jpk did u have any lucky? any particular reason for picking E7-8891v4?

 

[jpk]

@gmaxwheel do you think that 4x e7-8890 v3 + 32 x 16GB DDR3 would work? @jpk did u have any lucky? any particular reason for picking E7-8891v4?

My two 8891v4 work fine, I went with them because I got them for cheap, and I don't really need all that many cores, but they tend to be rather expensive, so getting two more for not a lot of money might be difficult.
I am thinking of getting 4x e7-8890 v3 cpus too, I don't see a problem with it. The 8890v3 cpus seem to be a pretty decent price right now.
As for the memory, I've only used the Samsung 16gb dimms that @gmaxwheel mentioned, but they work perfectly for me.

 

[tconrado]

@jpk tks for such fast reply! yeah, we got 4 x e7-8890 v3 for $100 ea on ebay, but all grade B! To be sincere, I just felt more secure with something that looks old and genuine (as it is old for sure), but I saw some high reputable sellers for $125 ea (grade A+), almost to good to be true... anyway 72 cores for $400 is an amazing deal... with DDR3 that is cheaper... we plan to do a 24 x SAS3 software raid 10 + 4 HHHL NvME, I'll keep u guys posted! Stay safe!

 

[jpk]

@tconrado that's a pretty nice deal for the 8890s! Do you mind if I ask which auction that was for?
I have a 24x raid60 array that I've been using in my current (e5v2) machine, and I'm hoping transferring that over won't be a big deal.

@DieBlub I don't think you'll be able to get 4x big cards in there - the memory cards will get in the way of 2 of them. Take a look at the photos:

Spoiler: Photos of 16x pci-e slots

 

[tconrado]

@jpk I did direct offers to sellers (Best offer button)... (purchase 2 from one, and 2 from another one)
this one still have 2... check this item on ebay: 2133947462008 (just post in the search), make him a direct offer... he did accept mine (before finding this one, I did purchased 2 from the one bellow and offered him 2 x $100)



I was planning to use 4 NVME like the ebay item: 274377402385 that is half height hope it does fit bellow the memory

btw, I just want to say thanks for u guys reply... I did order the thing... lets see if the can setup (Unix Surplus from ebay too)

 

[jpk]

@tconrado If you're planning on using all 8 of the memory boards, there's no space between them. I went for a really low profile adapter with an nvme m2 stick in it, and no luck - there's a big heatsink on the bottom of the memory board, and the P3 board 1 covers both slots.

I'm thinking of trying either some flexible pci-e extenders, or possibly see if I can just use CPU-3's memory board 2.

Here's a photo of my card and just how it doesn't fit in there with CPU3 memory board1

Spoiler: Photos of memory card heatsinks and pci-e slot clearance

super thin!
View attachment 15119
^^^ I like this installation method


really no space :-(


ka-thwunk (the memory board above isn't plugged in - it is resting on top of the pci-e card :-( )

If you're using CPU 2 and not the 2nd memory board for it, you can fit one behind the CPU 3 memory board 1.

Spoiler: card does fit behind CPU 3 memory board 1

Are you planning on booting off of those NVME cards?

 

[tconrado]

I may be missing something... but would it not fit in the slots 2,4, 8 and 9?

 

[jpk]

They totally will!
Sorry, I totally conflated your project and the one with the 4 graphics cards X-D
I want to mirror two of these nvme, and was thinking I want to make sure the mirror is on the same CPU to avoid some crosstalk

It's just a little silly that they didn't plan for the 2x CPU3 slots in the middle to work with the CPU3 Memory Board 1. I do wonder if there is any penalty with using memory board 2 instead of 1.

 

[tconrado]

how much clearance are in these lines A, B and C for slots 11, 9 and 8


the Intel DC P3608 cards are small, but plenty of heatpreader... maybe we will have to bend some heatspreader pins from the memory controler lol


tks again for the pictures! those slots in the middle are indeed a weird thing...

 

[tconrado]

They totally will!
Sorry, I totally conflated your project and the one with the 4 graphics cards X-D
I want to mirror two of these nvme, and was thinking I want to make sure the mirror is on the same CPU to avoid some crosstalk

It's just a little silly that they didn't plan for the 2x CPU3 slots in the middle to work with the CPU3 Memory Board 1. I do wonder if there is any penalty with using memory board 2 instead of 1.

yeah.. at least they should swap the 2nd mem board for cpu 4 by 1st for cpu 3, so, you can guarantee the middle clear for 2 gigantic GPU lol, so maybe 4 large GPU...

 

[Prof_G]

I’m wondering to those of you that have this board would you buy it over or would you move to Epyc ?

 

[angel_bee]

hi! just wana say that im glad i've found this thread. there seem to be alot of things to consider!

im planning to build a 4x xeon e7 8870 v3 system by swapping the parts from a prebuilt SYS-8048B-TRFT from ebay (it already has 4x v2 processors installed). but things that are holding me back are
1. fears that (as mentioned previously) v3 processors don't work with it. What would I do if that happens? (BMC cards were mentioned previously, but I am still unclear as to how they could help one navigate this issue)
2. I also want at least 512 GB of ram in my machine, but in case I fork out all that money just to find out that the RAM doesn't work, how do I make absolute sure that the RAM will be compatible??

 

[tconrado]

hi! just wana say that im glad i've found this thread. there seem to be alot of things to consider!

im planning to build a 4x xeon e7 8870 v3 system by swapping the parts from a prebuilt SYS-8048B-TRFT from ebay (it already has 4x v2 processors installed). but things that are holding me back are
1. fears that (as mentioned previously) v3 processors don't work with it. What would I do if that happens? (BMC cards were mentioned previously, but I am still unclear as to how they could help one navigate this issue)
2. I also want at least 512 GB of ram in my machine, but in case I fork out all that money just to find out that the RAM doesn't work, how do I make absolute sure that the RAM will be compatible??

while doing this things, we pay an extra fee so the send with our processor... I do recommend LinuxSurplus...
also, you can pay $30 that they do install OS and do a fair stability testing