2x Builds... Upgrading and a server for second house

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Build 1:

Build’s Name:
Main server upgrade​
Operating System/ Storage Platform:
Server 2019 w/ Hyper-V​
CPU:
2x Xeon Platinum 8260 ES 8276L QS​
Motherboard:
Intel S2600STB Supermicro X11DPH-T​
Chassis:
SuperMicro SC836​
Drives:
(new)
Intel Optane 800p 118GB NVMe 256GB PM981
2x Samsung PM981 2TB NVMe​
(migrating from old server)
13x Seagate ST12000NM0027 12TB SAS (in RAID 60 with hot spare)
2x Ultrastar SSD800MM 400GB SAS (in RAID1 for LSI CacheCade)​
RAM:
1TB - 16x Hynix HMAA8GL7MMR4N-UH​
Add-in Cards:
(migrating)
Mellanox ConnectX-3 dual 56GbE
LSI 9270CV-8i​
Power Supply:
2x SuperMicro 1200w​
Other Bits:

Usage Profile:

Virtualization host, primary storage, router, 30+ VMs doing all kinds of stuff.

Build 2:

Build’s Name:
Lake house server​
Operating System/ Storage Platform:
Server 2019 w/ Hyper-V (most likely, might try something else)​
CPU:
4x Xeon E5-4650v2​
Motherboard:
SuperMicro X9QRi+​
Chassis:
SuperMicro SC848​
Drives:
+Intel Optane 800p 118GB NVMe​
(migrating)
2x Intel SSD DC S3700 400GB
Intel SSD 750 PCIe 1TB​
RAM:
(migrating)
256GB - 16x Hynix HMT42GR7BFR4C-PB
might add another 256GB later to fill out the slots.​
Add-in Cards:
ConnectX-3 56GbE​
Power Supply:
3x SuperMicro 1200w​
Other Bits:
Supermicro AOC-SLG3-2M2
NVMe BIOS mod​

Usage Profile:

Virtualization host.
Don't even know what I'm putting on it yet. At least a local domain controller and virtualizing the router here so I stop bottlenecking IPSec. Failover mail and DNS servers. Case has room for 24 drives, probably going to put at least 40TB in for off-site backups between the 2 sites.

Retiring:
Current main server, has served me well for 6 years or so.
2x Xeon E5-2665 ES
SuperMicro X9DAi
Unknown 4U chassis I've had since 2007
Other parts pulled for the 2 above builds. ^^

I have a spare SC836 and some RAM that I've pulled upgrading. Will probably slap this board in there and stick it under my desk as a workstation.
 
Last edited:
  • Like
Reactions: Patrick

Boris

Member
May 16, 2015
76
11
8
Drives:
13x Seagate ST12000NM0027 12TB SAS (in RAID 60 with hot spare)
2x Ultrastar SSD800MM 400GB SAS (in RAID1 for LSI CacheCade)​
Add-in Cards:
LSI 9270CV-8i​
Have you used this array without CacheCade?
What kind of sequential read and write speeds you got if you did it?
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Have you used this array without CacheCade?
What kind of sequential read and write speeds you got if you did it?
I have not. I can disable the cache any time, but a benchmark wouldn't be very accurate with the number of VMs running on it. Once the build is complete, I can give it a try, since I plan on moving most of the high activity VMs to the NVMe SSDs.
 

Boris

Member
May 16, 2015
76
11
8
I have not. I can disable the cache any time, but a benchmark wouldn't be very accurate with the number of VMs running on it. Once the build is complete, I can give it a try, since I plan on moving most of the high activity VMs to the NVMe SSDs.
Thank you in advance.
It will be informative in any case. With or without CacheCade.
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Headaches and headaches (and $$$$)... Seems like my board is not compatible with the 8260 ES. Sent it back and got up-sold to a 8276L QS that'll hopefully work.

I'm seriously spoiled by Amazon shipping.. still waiting on Supermicro bits and pieces from various places.
 

Geran

Active Member
Oct 25, 2016
316
55
28
36
Headaches and headaches (and $$$$)... Seems like my board is not compatible with the 8260 ES. Sent it back and got up-sold to a 8276L QS that'll hopefully work.

I'm seriously spoiled by Amazon shipping.. still waiting on Supermicro bits and pieces from various places.
I'll honestly be surprised if any of the Platinum QS chips will be compatible with the Intel board you have. Most of the ones I found were only compatible with Supermicro board and only ones that were prior to revision 3.0.
 
Last edited:

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
I'll honestly be surprised if any of the QS chips will be compatible with the Intel board you have. Most of the ones I found were only compatible with Supermicro board and only ones that were prior to revision 3.0.
It worked with some 5115 QS CPUs, but unfortunately those were listed as a 20-core chip when I bought them, so I ended up selling them after it was too late to get a refund. If the 8276 QS don't work, I'll probably end up getting another board.
 

Geran

Active Member
Oct 25, 2016
316
55
28
36
It worked with some 5115 QS CPUs, but unfortunately those were listed as a 20-core chip when I bought them, so I ended up selling them after it was too late to get a refund. If the 8276 QS don't work, I'll probably end up getting another board.
I meant the Platinum chips. Most of the Gold QS chips work with the Intel boards that I've seen on eBay.
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
I meant the Platinum chips. Most of the Gold QS chips work with the Intel boards that I've seen on eBay.
I guess we'll find out in a week or so. Hopefully it works and I don't have to wait another week for a new board.
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
QQBE not working either. Guess I'm getting a new MB. I wonder what the difference between the Gold/Platinum series that causes the incompatibility. At least X11DPH can be picked up with Amazon next day.
 

Geran

Active Member
Oct 25, 2016
316
55
28
36
SM boards might have been what Intel was using to test the samples before providing it out to other suppliers to get boards ready.

I would check eBay for your CPUs to see if you can find a listing that states which board and revision it is potentially compatible with.
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Not much to go on on for the Cascade Lake chips. The seller I bought from had the Intel board listed as compatible until I told them it wasn't and they updated their listings. They took their screenshots using an ASRock board, but I can't find one for sale. The Supermicro ones seem to have the best all around compatibilty from the listings I've seen so that's what I'm trying now, Amazon prime so I can send it back easily if it doesn't work. Just hoping I can sell the Intel board for a decent price.
 

gigatexal

I'm here to learn
Nov 25, 2012
2,753
526
113
Portland, Oregon
alexandarnarayan.com
Build 1:

Build’s Name:
Main server upgrade​
Operating System/ Storage Platform:
Server 2019 w/ Hyper-V​
CPU:
2x Xeon Platinum 8260 ES 8276L QS​
Motherboard:
Intel S2600STB​
Chassis:
SuperMicro SC836​
Drives:
(new)
Intel Optane 800p 118GB NVMe
2x Samsung PM981 2TB NVMe​
(migrating from old server)
13x Seagate ST12000NM0027 12TB SAS (in RAID 60 with hot spare)
2x Ultrastar SSD800MM 400GB SAS (in RAID1 for LSI CacheCade)​
RAM:
1TB - 16x Hynix HMAA8GL7MMR4N-UH​
Add-in Cards:
(migrating)
Mellanox ConnectX-3 dual 56GbE
LSI 9270CV-8i​
Power Supply:
2x SuperMicro 1200w​
Other Bits:

Usage Profile:

Virtualization host, primary storage, router, 30+ VMs doing all kinds of stuff.

Build 2:

Build’s Name:
Lake house server​
Operating System/ Storage Platform:
Server 2019 w/ Hyper-V (most likely, might try something else)​
CPU:
4x Xeon E5-4650v2​
Motherboard:
SuperMicro X9QRi+​
Chassis:
SuperMicro SC848​
Drives:
(migrating)
2x Intel SSD DC S3700 400GB
Intel SSD 750 PCIe 1TB​
RAM:
(migrating)
256GB - 16x Hynix HMT42GR7BFR4C-PB
might add another 256GB later to fill out the slots.​
Add-in Cards:
ConnectX-3 56GbE​
Power Supply:
3x SuperMicro 1200w​
Other Bits:

Usage Profile:
Virtualization host.
Don't even know what I'm putting on it yet. At least a local domain controller and virtualizing the router here so I stop bottlenecking IPSec. Failover mail and DNS servers. Case has room for 24 drives, probably going to put at least 40TB in for off-site backups between the 2 sites.

Retiring:
Current main server, has served me well for 6 years or so.
2x Xeon E5-2665 ES
SuperMicro X9DAi
Unknown 4U chassis I've had since 2007
Other parts pulled for the 2 above builds. ^^

I have a spare SC836 and some RAM that I've pulled upgrading. Will probably slap this board in there and stick it under my desk as a workstation.
My favorite part is this line:

“Add another 256GB to fill out the slots”
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Finally up and running :D X11DPH-T works. Had to order 2 heatsinks as well, didn't realize there were 2 different styles for LGA3647... Need some more adapter cards and bits since I was using the OcuLink ports on the Intel board that don't exist on the SM.




Will take a few hours to decommission the old server replace it with this one when I can tolerate some down time. Hoping I can live migrate some VMs once I have a way to connect storage to minimize it.
 

Geran

Active Member
Oct 25, 2016
316
55
28
36
Glad you were able to get it up and running. If you don't mind me asking, what did you pay for your 8276L QS?
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Almost ready to pull the old server. Having some issues with my Optane 800P - it causes POST to freeze up at "DXE--AHCI Initialization" unless I reset CMOS, doesn't matter if I make any changes, it'll freeze again at the next reboot. Ordered a 256GB PM981 to replace it with, and going to try to run it in Build 2 with a NVMe BIOS mod.

New server almost ready to go in:


Old server:


Think I'm going to run the new one on the 5 gal bucket it's sitting on top of for a while, and rearrange/rewire the whole rack.
 

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
Finally up and running! Got everything migrated over and the old server has been P2V'd in case there's anything I forgot. Ended up having to copy 22TB, delete all my backups, and reconstruct my whole RAID array because I couldn't get the card to import a foreign configuration. Putting everything on RAID0 on the hot spares was a bit scary!



I'm going to try and 3D print an air dam to keep hot air from CPU1 out of CPU2. Definitely not a fan of the 15deg temp differential.

Too tired of dealing with issues to rewire the rack for now - it's gonna stay on the bucket for a week or so.

Build 2 should come together pretty quickly now that I can strip the old server for the needed components.
 
Last edited:

mb300sd

Active Member
Aug 1, 2016
197
74
28
30
And... Build 2 up and running. Nothing on it yet.

NVMe BIOS mod worked great, it boots off my Optane 800p. I only put half the RAM in so far, since I want to mess with the old board a little and don't know where I put the RAM I pulled out of it a few years ago.