2x Builds... Upgrading and a server for second house

Discussion in 'DIY Server and Workstation Builds' started by mb300sd, Aug 9, 2019.

  1. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Build 1:

    Build’s Name:
    Main server upgrade​
    Operating System/ Storage Platform:
    Server 2019 w/ Hyper-V​
    CPU:
    2x Xeon Platinum 8260 ES 8276L QS​
    Motherboard:
    Intel S2600STB Supermicro X11DPH-T​
    Chassis:
    SuperMicro SC836​
    Drives:
    (new)
    Intel Optane 800p 118GB NVMe 256GB PM981
    2x Samsung PM981 2TB NVMe​
    (migrating from old server)
    13x Seagate ST12000NM0027 12TB SAS (in RAID 60 with hot spare)
    2x Ultrastar SSD800MM 400GB SAS (in RAID1 for LSI CacheCade)​
    RAM:
    1TB - 16x Hynix HMAA8GL7MMR4N-UH​
    Add-in Cards:
    (migrating)
    Mellanox ConnectX-3 dual 56GbE
    LSI 9270CV-8i​
    Power Supply:
    2x SuperMicro 1200w​
    Other Bits:

    Usage Profile:

    Virtualization host, primary storage, router, 30+ VMs doing all kinds of stuff.

    Build 2:

    Build’s Name:
    Lake house server​
    Operating System/ Storage Platform:
    Server 2019 w/ Hyper-V (most likely, might try something else)​
    CPU:
    4x Xeon E5-4650v2​
    Motherboard:
    SuperMicro X9QRi+​
    Chassis:
    SuperMicro SC848​
    Drives:
    +Intel Optane 800p 118GB NVMe​
    (migrating)
    2x Intel SSD DC S3700 400GB
    Intel SSD 750 PCIe 1TB​
    RAM:
    (migrating)
    256GB - 16x Hynix HMT42GR7BFR4C-PB
    might add another 256GB later to fill out the slots.​
    Add-in Cards:
    ConnectX-3 56GbE​
    Power Supply:
    3x SuperMicro 1200w​
    Other Bits:
    Supermicro AOC-SLG3-2M2
    NVMe BIOS mod​

    Usage Profile:

    Virtualization host.
    Don't even know what I'm putting on it yet. At least a local domain controller and virtualizing the router here so I stop bottlenecking IPSec. Failover mail and DNS servers. Case has room for 24 drives, probably going to put at least 40TB in for off-site backups between the 2 sites.

    Retiring:
    Current main server, has served me well for 6 years or so.
    2x Xeon E5-2665 ES
    SuperMicro X9DAi
    Unknown 4U chassis I've had since 2007
    Other parts pulled for the 2 above builds. ^^

    I have a spare SC836 and some RAM that I've pulled upgrading. Will probably slap this board in there and stick it under my desk as a workstation.
     
    #1
    Last edited: Sep 17, 2019
    Patrick likes this.
  2. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,332
    Likes Received:
    205
    Build one is a beastly server. Looking forward to pictures of the builds, etc.
     
    #2
  3. Boris

    Boris Member

    Joined:
    May 16, 2015
    Messages:
    75
    Likes Received:
    11
    Have you used this array without CacheCade?
    What kind of sequential read and write speeds you got if you did it?
     
    #3
  4. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    I have not. I can disable the cache any time, but a benchmark wouldn't be very accurate with the number of VMs running on it. Once the build is complete, I can give it a try, since I plan on moving most of the high activity VMs to the NVMe SSDs.
     
    #4
  5. Boris

    Boris Member

    Joined:
    May 16, 2015
    Messages:
    75
    Likes Received:
    11
    Thank you in advance.
    It will be informative in any case. With or without CacheCade.
     
    #5
  6. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Headaches and headaches (and $$$$)... Seems like my board is not compatible with the 8260 ES. Sent it back and got up-sold to a 8276L QS that'll hopefully work.

    I'm seriously spoiled by Amazon shipping.. still waiting on Supermicro bits and pieces from various places.
     
    #6
  7. Geran

    Geran Active Member

    Joined:
    Oct 25, 2016
    Messages:
    247
    Likes Received:
    32
    I'll honestly be surprised if any of the Platinum QS chips will be compatible with the Intel board you have. Most of the ones I found were only compatible with Supermicro board and only ones that were prior to revision 3.0.
     
    #7
    Last edited: Aug 14, 2019
  8. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    It worked with some 5115 QS CPUs, but unfortunately those were listed as a 20-core chip when I bought them, so I ended up selling them after it was too late to get a refund. If the 8276 QS don't work, I'll probably end up getting another board.
     
    #8
  9. Geran

    Geran Active Member

    Joined:
    Oct 25, 2016
    Messages:
    247
    Likes Received:
    32
    I meant the Platinum chips. Most of the Gold QS chips work with the Intel boards that I've seen on eBay.
     
    #9
  10. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    I guess we'll find out in a week or so. Hopefully it works and I don't have to wait another week for a new board.
     
    #10
  11. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    QQBE not working either. Guess I'm getting a new MB. I wonder what the difference between the Gold/Platinum series that causes the incompatibility. At least X11DPH can be picked up with Amazon next day.
     
    #11
  12. Geran

    Geran Active Member

    Joined:
    Oct 25, 2016
    Messages:
    247
    Likes Received:
    32
    SM boards might have been what Intel was using to test the samples before providing it out to other suppliers to get boards ready.

    I would check eBay for your CPUs to see if you can find a listing that states which board and revision it is potentially compatible with.
     
    #12
  13. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Not much to go on on for the Cascade Lake chips. The seller I bought from had the Intel board listed as compatible until I told them it wasn't and they updated their listings. They took their screenshots using an ASRock board, but I can't find one for sale. The Supermicro ones seem to have the best all around compatibilty from the listings I've seen so that's what I'm trying now, Amazon prime so I can send it back easily if it doesn't work. Just hoping I can sell the Intel board for a decent price.
     
    #13
  14. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,683
    Likes Received:
    498
    My favorite part is this line:

    “Add another 256GB to fill out the slots”
     
    #14
  15. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Finally up and running :D X11DPH-T works. Had to order 2 heatsinks as well, didn't realize there were 2 different styles for LGA3647... Need some more adapter cards and bits since I was using the OcuLink ports on the Intel board that don't exist on the SM.

    [​IMG]
    [​IMG]

    Will take a few hours to decommission the old server replace it with this one when I can tolerate some down time. Hoping I can live migrate some VMs once I have a way to connect storage to minimize it.
     
    #15
  16. Geran

    Geran Active Member

    Joined:
    Oct 25, 2016
    Messages:
    247
    Likes Received:
    32
    Glad you were able to get it up and running. If you don't mind me asking, what did you pay for your 8276L QS?
     
    #16
  17. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Around $1200 a piece, not bad for a $8k chip :)
     
    #17
  18. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Almost ready to pull the old server. Having some issues with my Optane 800P - it causes POST to freeze up at "DXE--AHCI Initialization" unless I reset CMOS, doesn't matter if I make any changes, it'll freeze again at the next reboot. Ordered a 256GB PM981 to replace it with, and going to try to run it in Build 2 with a NVMe BIOS mod.

    New server almost ready to go in:
    [​IMG]

    Old server:
    [​IMG]

    Think I'm going to run the new one on the 5 gal bucket it's sitting on top of for a while, and rearrange/rewire the whole rack.
     
    #18
  19. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Finally up and running! Got everything migrated over and the old server has been P2V'd in case there's anything I forgot. Ended up having to copy 22TB, delete all my backups, and reconstruct my whole RAID array because I couldn't get the card to import a foreign configuration. Putting everything on RAID0 on the hot spares was a bit scary!

    [​IMG]

    I'm going to try and 3D print an air dam to keep hot air from CPU1 out of CPU2. Definitely not a fan of the 15deg temp differential.

    Too tired of dealing with issues to rewire the rack for now - it's gonna stay on the bucket for a week or so.

    Build 2 should come together pretty quickly now that I can strip the old server for the needed components.
     
    #19
    Last edited: Sep 16, 2019
  20. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    And... Build 2 up and running. Nothing on it yet.

    NVMe BIOS mod worked great, it boots off my Optane 800p. I only put half the RAM in so far, since I want to mess with the old board a little and don't know where I put the RAM I pulled out of it a few years ago.

    [​IMG]
     
    #20
Similar Threads: Builds Upgrading
Forum Title Date
DIY Server and Workstation Builds Some information about HP T620 Plus Flexible Thin Client machines for network appliance builds... Jul 2, 2018
DIY Server and Workstation Builds sharing an idea for hotswap boot drives on older Supermicro 846 builds Jan 11, 2018
DIY Server and Workstation Builds Small business ESXi and NAS builds/new network from scratch - Xeon D vs E5 v4 and more May 31, 2016
DIY Server and Workstation Builds When server builds go wrong! Jun 20, 2015
DIY Server and Workstation Builds My ZFS NAS build/benchmark history so far (4+ builds) - long time lurker now a member May 23, 2015

Share This Page