lga3647 esxi build to host my Oracle Apps/Databases

Discussion in 'DIY Server and Workstation Builds' started by BennyT, Dec 14, 2018.

  1. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    to enable six channel memory do all 6 DIMMs for CPU1 need to be identical spec and density? Meaning same RANK for all 6 DIMMs and same density?

    Answer: after quick YouTube viewing I believe the answer is that all six DIMMs should be same rank and density to enable 6 channel.

    And taking that one step further, does CPU2 need to have same memory topology as that of CPU1?

    Edit: I'm still unsure of that last question. Could I for example use 2 quantity quad rank 64 GB sticks on CPU2 if i have 6qty dual rank 32GB sticks on cpu1.



    thanks

    *EDIT: (march 29 2019)
    I found this nice document describing (with diagram pictures!) what I needed to know.
    https://lenovopress.com/lp0742.pdf
     
    #81
    Last edited: Mar 29, 2019
  2. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    2,535
    Likes Received:
    372
    Quad rank 64gb are usually LRDIMM and dual rank 32gb are more often than not RDIMM with some being LRDIMM and you certainly can’t mix R and LR DIMM’s. So be careful to know and check this.

    Anyway I would not do what your proposing.
     
    #82
    BennyT likes this.
  3. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    Thank you Evan. I currently have two qty 32GB 2 RANK RDIMM on cpu1, so I'll stick with those throughout.
     
    #83
  4. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    This new ESXi server is overkill right now, I'm loving it. Long way to go yet before I get all the Oracle apps and databases cloned from the baremetal boxes to the ESXi server.

    I added a VM for oracle linux 7u6 minimal install with openssl, httpd, subversion to act as SVN repository. My idea is to have it take over duties from a small HP box that stores my source code and documentation revisions. I'm setting the hp box to sync nightly to the new VM. Eventually I'll cutover to using just the new vm and shutdown the HP box, or maybe I'll keep the HP box as a backup server or as a redundant SVN respository. so many ideas.

    I can't cutover from the baremetal boxes to the new ESXi server until I can prove I have a good backup recovery solution. I've been experimenting with veeam backup and recovery community edition. It's full featured except limited to backing up 10VMs. It also doesn't have the enterprise features such as Oracle database backup. But I use Oracle's own RMAN instead anyway. If I can't get veeam to work I may try to use clonezilla (simplest solution) to create images of shutdown VMs.
     
    #84
    Last edited: Feb 28, 2019
  5. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    question: does VMWare have any native backup recovery tools for ESXi guests?
     
    #85
  6. jcl333

    jcl333 Member

    Joined:
    May 28, 2011
    Messages:
    72
    Likes Received:
    4
    Hey, this is a great thread I just started reading it after I just posted my build thread.

    - Good choice of MB, I am going with the single-socket version
    - That Norco is pretty neat, they have come a long way since they started, I might consider it
    - Lots of good info on this thread that I can use, already answered some of my questions

    -JCL
     
    #86
    BennyT likes this.
  7. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    Hi @jcl333

    Thanks. I'm glad you found this thread helpful. I'm really happy with the system and I've minimal complaints about the Norco chassis (was going to say zero complaints but nothing is perfect) and SM motherboard.

    --

    Here is my progress...

    SVN - My Subversion (SVN) repositories from the old bare metal servers have been synchronized to a VM on the new server. I've cutover to using it instead of the repository on my old server and it is working great. I've set up the linux vm hosting the subversion repositories to synchronize (using svnsync which utilizes rsync i think) nightly back to the old servers for redundancy just incase the ESXi host goes down or if I lose the VM, etc. Working copies sync to my dropbox cloud.

    Plexmediaserver - I've setup my plex server on a small VM and I'm pretty happy with it. It is quite sparse compared to some of the hardcore plex servers I see on the forum. Nothing much to share except one thing I noticed, if a movie stream requires transcoding (such as if someone is watching a movie remotely and at a low bandwidth or lower resolution) a single virtual core goes up to 100% temporarily and at 3.5GHz according to monitor in vSphere Client. It eventually settles back down to almost idle after a few minutes. Other cores remained at idle during transcoding. I think that is normal as transcoding is pretty much single threaded per stream. I just hadn't noticed it before.

    Oracle Apps and Databases - I'm still in process of planning this migration. I will use Oracle Rapid Clone to facilitate the move to the new guest VMs on the ESXi host. I've tar'ed the filesystems. I'm waiting before proceeding too far because I want to have some backup methodology in place first and to get my ducks and standards all in a row.

    Backup and Recovery (for the ESXi guest VMs) - I'm still hopeful to use Veeam Backup Community Edition but I'm having trouble wrapping my head around how to configure it.
    - veeam server - This drives the backup process. It requires a Windows OS but it can be a VM. So I plan to use a guest VM running on my laptop using VMware Workstation Pro for this.

    - veeam proxy server - I'm not a network person by any stretch. Not sure if this veeam proxy server is required or if I can just point the veeam server directly to the backup repository server without a proxy server.

    - veeam repository server (target location for the backups) - I would like to use one of my old bare metal linux servers as the backup repository. Not sure if I can simply setup an SMB or NFS on it or what.​

    Else, instead of veeam for backups, I could use a clonezilla iso file I've saved to a datatore that I can use when booting up any guest VM. I'd attach a "toaster" HDD caddy via USB3, make that HDD accessible to the guest VM, bootup the guest using the clonezilla iso, and make image of the /dev/sda1-sdd1 (for example) and send that backup image to the HDD in the toaster caddy. Pretty crude but it would work.

    My other backup options might be to write custom shell scripts using dd or incremental backups using rsync and tar and then sync the images or tarballs to another storage such as an external drive, NAS or dropbox cloud. Not sure yet. Still would like to hear of other solutions from folks in the forum.

    Have a great week.
     
    #87
    Last edited: Mar 20, 2019
  8. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    I was able to configure Veeam Backup And Replication (Community Edition) to schedule backups of my ESXi guests VMs and it is cool.

    - installed veeam into a small Windows 10 guest VM on my laptop.
    - created a backup repository by telling veeam console to use my little HP box with gigabit NIC and a dual core pentium E5200 @ 2.50GHz
    - plugged in three HDDs into the HP's hotswap drive sleds.
    - each of the three drive sleds are mounted as standalone filesystems on the little hp box
    /backup_drivebay_2/
    /backup_drivebay_3/
    /backup_drivebay_4/​
    upload_2019-3-20_14-35-53.png

    upload_2019-3-20_14-30-48.png
    upload_2019-3-20_14-31-49.png

    upload_2019-3-20_14-32-37.png

    - created a backup job which points to the vCenter server ip with credentials specified, Datacenter, ESXi Host, then finally pointing to the individual guest vm.
    upload_2019-3-20_14-35-7.png

    Now I just need to decide which type of incremental ("forward incremental" or "forever forward" or "forward with occasional full backups", etc) and what kind of compression to use. It will be interesting to see how this performs. I have one linux vm with 600GB allocated but with only 327GB currently being used. I'm curious to find out if it will try to backup all 600GB (even the unused portion) or just the 327GB. It is thick lazy provisioned so seems it will try to backup all unless it were thin provisioned.
    edit: regarding this I later learned that veeam enables "deduplication"...
    "Data deduplication decreases the size of backup files. You can enable data deduplication if you add to backup or replication jobs several VMs that have a great amount of free space on their logical disks or VMs that have similar data blocks — for example, VMs that were created from the same template. With data deduplication enabled, Veeam Backup & Replication does not store to the resulting backup file identical data blocks and space that has been pre-allocated but not used."

    I'll post again later after I actually perform a few backups and recovery tests. I'm just really excited I have it configured to identify the source and targets and that it can all be scheduled.

    *edit: Discovered that I'll want to move this Veeam vm I installed on my laptop into a vm on the esxi host. I don't want to keep my laptop on and connected to ethernet each night to schedule nightly backups. I'll migrate the veeam guest into a new vm on the esxi server tomorrow and retest.
     
    #88
    Last edited: Mar 22, 2019
  9. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    I've added the veeam backup VM to ESXi and works great so far. Full back up of a 600GB VM took just 32 minutes across my gigabit network using simple spinners and no RAID array striping. The resulting backup was 125GB. The incrementals of course will be much faster/smaller backups.


    backing up using veeam (32 minutes):
    upload_2019-3-21_13-21-30.png

    restoring using veeam:
    upload_2019-3-21_18-21-45.png

    A full restore of that 600GB VM with three drives took about 2hr 15 minutes.
    Restore completed:
    upload_2019-3-21_18-22-52.png

    Verifying the filesystems and logical volumes are okay on the restored system:
    upload_2019-3-21_18-24-58.png

    So far I'm very pleased with performance by running veeam in a vm on the ESXi server. By the way, veeam community edition "only" allows up to 10 VMs to be backed up (scheduled with incremental backups). But I found out I can do unlimited VM backups using veeamzip utility if I don't mind making them full backups (not incremental).

    Note though that veeam community edition will not work with free version of ESXi. You must have a licensed ESXi. I got my vSphere license keys really cheap using VMUG Advantage.

    Edit: here is the veeam report of the oracle linux vm 600GB backup process (of the 600gb only about 400GB used)
    upload_2019-3-22_9-25-9.png

    and here is the veeam report for a 900GB plexmediaserver vm (of the 900GB only about 600GB used)
    upload_2019-3-22_9-34-19.png

    Note that the plex backup took much longer. Partly because it had more stuff on it. But mostly because the movies on it were already heavily compressed and veeam was not able to compress them further. This meant veeam had to pass much more data through the network to the target backup repository and therefore took 1.5 hrs compared to the other backup that only took 32 minutes.

    Edit2:
    Also, when running veeam from a vm on the ESXi host, it can attach itself directly to the VM that is being backed up... it dynamically attaches the virtual disks from the vm being backed up to the veeam vm. It couldn't do this when I had veeam running from my laptop. End result is that it is much much faster for me when running veeam in a vm on the esxi host that I'm backing up. After the backup completes, the virtual disks are automatically removed from the veeam vm. Pretty cool
    upload_2019-3-22_10-7-21.png
    after the backup the virtual disks from the vm being backed up are removed from the veeam vm:
    upload_2019-3-22_11-8-25.png
     

    Attached Files:

    #89
    Last edited: Mar 22, 2019
  10. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    Success!.. successfully moved one of my bare metal Oracle APPS/EBS Database environments to the new ESXi server! Note, this is not simply installing an Oracle appliance with pre-configured EBS. These EBS and database nodes were built from scratch using Oracle Rapid Install process. Migrated using Oracle's migration tool "Rapid Clone" from the standalone boxes to guest VMs running on my new ESXi server. So far so good. Now I just have about 5 more to do. Then a few non APPS databases.

    I'm feeling very proud of the new setup especially now that I also have scheduled backups. I'm very excited to try upgrading to latest EBS R12.2.8 or to try the new Oracle 18c database. So many possibilities with my new ESXi server that I didn't have available before on bare metal boxes.

    I'm going to want to buy some more RAM soon. I have 64GB now but I think I'm going to want 128GB. Nice to see memory prices beginning to fall.

    upload_2019-3-24_21-4-13.png
     
    #90
    Last edited: Mar 24, 2019
    Gadgetguru and itronin like this.
  11. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    Yesterday while adding wedding photos to my plexmediaserver VM's photo library I made a bad mistake AS root...

    I was logged in as root user to the linux VM. Thinking I was in the folder for plex library content I ran "chmod -R 755 *" to correctly set few permissions on couple of files. Except I wasn't in my .../plexdrive/ subfolder, I was in "/". Took me a moment to figure out why it was taking so long to run a simple command. I couldn't figure out how to reverse this. It affected the system in such a way that I was unable to ssh into it from another shell window, and who knows what else that affected or how far that chmod command propagated, screwing up file and folder permissions.

    Fortunately, my plex VM is not super large. It is just under 1TB and I have incremental backups weekly using veeam. I began a full recovery last night and it restored perfectly. I have three backup repositories for my various VMs. This plex VM was backed up to one of those on a single HDD repository located on another local physical server via gigabit. It took 5 hrs and 10 minutes to fully restore. Not bad. It would have perhaps restored faster if I told it to check blocks and only restore blocks that had changed since the last backup, not sure. But I felt better doing a full restore.

    That was my first real recovery using Veeam (it wasn't just a test recovery) and very pleased how it went. To avoid this happening again I will change the owner of the plex library content to a new user, something like "plexuser" perhaps (currently owner is root for that library). Then I wouldn't be logging in as root when adding content or changing file permissions.

    Are you guys with HUGE plex library backing up your content?
     
    #91
  12. Marsh

    Marsh Moderator

    Joined:
    May 12, 2013
    Messages:
    1,998
    Likes Received:
    915
    My media files are stored in a NAS , easier to play and restore a VM with just media server software.
    Most of the time, if I am working on the VM, take a snapshoot first, it something goes wrong, revert to previous snapshoot.

    If you do not have a real physical NAS , created a virtual NAS fileserver.
     
    #92
  13. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    Hi @Marsh Thanks for the reply.

    I'll research your suggesting regarding setting up a NAS (physical or otherwise) to house a few VMs. I see many here use whitebox NAS, out of box NAS, or virtual NAS. I need to understand better why a "NAS" would be advantageous. For example, I'd like to understand the "easier to play or restore the VM". I understand the snapshot part though. But I think I read in VMware best practice document not to rely on snapshot as sole means for backups.

    Regarding snapshots, I use them as a "just in case" measure prior to a major, questionable or experiment action of some type. I remove snapshot days later after I confirm whatever I was doing that necessitated the snapshot is working properly. I don't snapshots often and I don't use them for backups. In this instance it didn't even occur to me to make a snapshot. I forget that feature sometimes now that I've virtualized my servers but it would've saved me 5+ hrs if I had made one.

    Not sure if best approach but here is high level summary of how I backup my ESXi guest VMs (all currently reside in datastores on the hosts backplane):
    - Scheduled incremental backups using veeam with occasional full backups to repository drives onto another physical server. The repository drives of this other server are hot swappable.
    - Monthly I remove the hotswapp repository drives and I lock them in fireproof safe and I put in a newly formatted drive in it's place. I perform this repository drive rotation each month using a pile of hotswap drives reserved for this purpose that sit in the fireproof safe.
    - snapshots taken if I'm about to do something I think could backfire on me. If things go bad then I can quickly restore from snapshot and I won't have to restore from backup.

    It's a work in progress.

    I've also been experimenting using veeam to backup my physical servers. I've not used clonezilla for about a month now which was my previous means of making backups.

    Edit: just thought of something I'm missing - Now that I have automated scheduled backups of the guest VMs I now need to research how to backup the ESXi Host, the vCenter Server Appliance (perhaps I'll back this up just like any other vm using veeam) and also the Veeam Backup server (which is also a guest VM).
     
    #93
    Last edited: Apr 24, 2019 at 11:45 AM
  14. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    896
    Likes Received:
    303
    As someone who's also nuked permissions on the wrong tree before, it's a good idea to be able to back up the permissions separately from the files - it probably wouldn't have helped you much in this particular case if the system became unusable (although I dare say you could have done the steps below in single user mode), but you can cut down your restore time considerably by just restoring the permissions.

    Use a command line something like this in cron/root crontab to back up all permissions at a set interval (assuming you're using an ACL-enabled filesystem and have the ACL utils installed; the backslashes in the date sub are there to escape the % symbols inside the crontab, you don't need them in a regular shell):
    Code:
    getfacl -R /somedir > /var/backups/acl/somedir_permissions_$(/bin/date '+\%a_\%H\%M'|tr [A-Z] [a-z])
    You can of course just use the root path to do the whole system at once but I prefer a bit more granularity, then you can simply restore permissions with a command like the following:
    Code:
    setfacl --restore=/var/backups/acl/somedir_permissions_wed_2105
    Tadaa! Hopefully a more-or-less instant (and correct!) restore of all your permissions, including ACLs if you're using them.
     
    #94
    Evan and BennyT like this.
  15. BennyT

    BennyT Member

    Joined:
    Dec 1, 2018
    Messages:
    73
    Likes Received:
    16
    Very cool @EffrafaxOfWug

    Good idea. I'll start setting something like that up on a cron schedule.
     
    #95
Similar Threads: lga3647 esxi
Forum Title Date
DIY Server and Workstation Builds LGA3647platinum MY PC#1 Jan 31, 2019
DIY Server and Workstation Builds Migrating aging HP workstation to ESXi 6.5U2 for Plex/other-tasks - WWYD? Apr 5, 2019
DIY Server and Workstation Builds New ESXi/Proxmox/FreeNAS server build advice Feb 19, 2019
DIY Server and Workstation Builds ESXi/PLEX/Storage server upgrade... WWYD Jan 3, 2019
DIY Server and Workstation Builds My new low wattage homelab esxi build Nov 17, 2018

Share This Page