lga3647 esxi build to host my Oracle Apps/Databases

BennyT

Active Member
Dec 1, 2018
146
39
28
to enable six channel memory do all 6 DIMMs for CPU1 need to be identical spec and density? Meaning same RANK for all 6 DIMMs and same density?

Answer: after quick YouTube viewing I believe the answer is that all six DIMMs should be same rank and density to enable 6 channel.

And taking that one step further, does CPU2 need to have same memory topology as that of CPU1?

Edit: I'm still unsure of that last question. Could I for example use 2 quantity quad rank 64 GB sticks on CPU2 if i have 6qty dual rank 32GB sticks on cpu1.



thanks

*EDIT: (march 29 2019)
I found this nice document describing (with diagram pictures!) what I needed to know.
https://lenovopress.com/lp0742.pdf
 
Last edited:

Evan

Well-Known Member
Jan 6, 2016
3,346
593
113
to enable six channel memory do all 6 DIMMs for CPU1 need to be identical spec and density? Meaning same RANK for all 6 DIMMs and same density?

Answer: after quick YouTube viewing I believe the answer is that all six DIMMs should be same rank and density to enable 6 channel.

And taking that one step further, does CPU2 need to have same memory topology as that of CPU1?

Edit: I'm still unsure of that last question. Could I for example use 2 quantity quad rank 64 GB sticks on CPU2 if i have 6qty dual rank 32GB sticks on cpu1.



thanks
Quad rank 64gb are usually LRDIMM and dual rank 32gb are more often than not RDIMM with some being LRDIMM and you certainly can’t mix R and LR DIMM’s. So be careful to know and check this.

Anyway I would not do what your proposing.
 
  • Like
Reactions: BennyT

BennyT

Active Member
Dec 1, 2018
146
39
28
Thank you Evan. I currently have two qty 32GB 2 RANK RDIMM on cpu1, so I'll stick with those throughout.
 

BennyT

Active Member
Dec 1, 2018
146
39
28
This new ESXi server is overkill right now, I'm loving it. Long way to go yet before I get all the Oracle apps and databases cloned from the baremetal boxes to the ESXi server.

I added a VM for oracle linux 7u6 minimal install with openssl, httpd, subversion to act as SVN repository. My idea is to have it take over duties from a small HP box that stores my source code and documentation revisions. I'm setting the hp box to sync nightly to the new VM. Eventually I'll cutover to using just the new vm and shutdown the HP box, or maybe I'll keep the HP box as a backup server or as a redundant SVN respository. so many ideas.

I can't cutover from the baremetal boxes to the new ESXi server until I can prove I have a good backup recovery solution. I've been experimenting with veeam backup and recovery community edition. It's full featured except limited to backing up 10VMs. It also doesn't have the enterprise features such as Oracle database backup. But I use Oracle's own RMAN instead anyway. If I can't get veeam to work I may try to use clonezilla (simplest solution) to create images of shutdown VMs.
 
Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
question: does VMWare have any native backup recovery tools for ESXi guests?
 

jcl333

Active Member
May 28, 2011
225
52
28
Hey, this is a great thread I just started reading it after I just posted my build thread.

- Good choice of MB, I am going with the single-socket version
- That Norco is pretty neat, they have come a long way since they started, I might consider it
- Lots of good info on this thread that I can use, already answered some of my questions

-JCL
 
  • Like
Reactions: BennyT

BennyT

Active Member
Dec 1, 2018
146
39
28
Hi @jcl333

Thanks. I'm glad you found this thread helpful. I'm really happy with the system and I've minimal complaints about the Norco chassis (was going to say zero complaints but nothing is perfect) and SM motherboard.

--

Here is my progress...

SVN - My Subversion (SVN) repositories from the old bare metal servers have been synchronized to a VM on the new server. I've cutover to using it instead of the repository on my old server and it is working great. I've set up the linux vm hosting the subversion repositories to synchronize (using svnsync which utilizes rsync i think) nightly back to the old servers for redundancy just incase the ESXi host goes down or if I lose the VM, etc. Working copies sync to my dropbox cloud.

Plexmediaserver - I've setup my plex server on a small VM and I'm pretty happy with it. It is quite sparse compared to some of the hardcore plex servers I see on the forum. Nothing much to share except one thing I noticed, if a movie stream requires transcoding (such as if someone is watching a movie remotely and at a low bandwidth or lower resolution) a single virtual core goes up to 100% temporarily and at 3.5GHz according to monitor in vSphere Client. It eventually settles back down to almost idle after a few minutes. Other cores remained at idle during transcoding. I think that is normal as transcoding is pretty much single threaded per stream. I just hadn't noticed it before.

Oracle Apps and Databases - I'm still in process of planning this migration. I will use Oracle Rapid Clone to facilitate the move to the new guest VMs on the ESXi host. I've tar'ed the filesystems. I'm waiting before proceeding too far because I want to have some backup methodology in place first and to get my ducks and standards all in a row.

Backup and Recovery (for the ESXi guest VMs) - I'm still hopeful to use Veeam Backup Community Edition but I'm having trouble wrapping my head around how to configure it.
- veeam server - This drives the backup process. It requires a Windows OS but it can be a VM. So I plan to use a guest VM running on my laptop using VMware Workstation Pro for this.

- veeam proxy server - I'm not a network person by any stretch. Not sure if this veeam proxy server is required or if I can just point the veeam server directly to the backup repository server without a proxy server.

- veeam repository server (target location for the backups) - I would like to use one of my old bare metal linux servers as the backup repository. Not sure if I can simply setup an SMB or NFS on it or what.​

Else, instead of veeam for backups, I could use a clonezilla iso file I've saved to a datatore that I can use when booting up any guest VM. I'd attach a "toaster" HDD caddy via USB3, make that HDD accessible to the guest VM, bootup the guest using the clonezilla iso, and make image of the /dev/sda1-sdd1 (for example) and send that backup image to the HDD in the toaster caddy. Pretty crude but it would work.

My other backup options might be to write custom shell scripts using dd or incremental backups using rsync and tar and then sync the images or tarballs to another storage such as an external drive, NAS or dropbox cloud. Not sure yet. Still would like to hear of other solutions from folks in the forum.

Have a great week.
 
Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
I was able to configure Veeam Backup And Replication (Community Edition) to schedule backups of my ESXi guests VMs and it is cool.

- installed veeam into a small Windows 10 guest VM on my laptop.
- created a backup repository by telling veeam console to use my little HP box with gigabit NIC and a dual core pentium E5200 @ 2.50GHz
- plugged in three HDDs into the HP's hotswap drive sleds.
- each of the three drive sleds are mounted as standalone filesystems on the little hp box
/backup_drivebay_2/
/backup_drivebay_3/
/backup_drivebay_4/​
upload_2019-3-20_14-35-53.png

upload_2019-3-20_14-30-48.png
upload_2019-3-20_14-31-49.png

upload_2019-3-20_14-32-37.png
- created a backup job which points to the vCenter server ip with credentials specified, Datacenter, ESXi Host, then finally pointing to the individual guest vm.
upload_2019-3-20_14-35-7.png

Now I just need to decide which type of incremental ("forward incremental" or "forever forward" or "forward with occasional full backups", etc) and what kind of compression to use. It will be interesting to see how this performs. I have one linux vm with 600GB allocated but with only 327GB currently being used. I'm curious to find out if it will try to backup all 600GB (even the unused portion) or just the 327GB. It is thick lazy provisioned so seems it will try to backup all unless it were thin provisioned.
edit: regarding this I later learned that veeam enables "deduplication"...
"Data deduplication decreases the size of backup files. You can enable data deduplication if you add to backup or replication jobs several VMs that have a great amount of free space on their logical disks or VMs that have similar data blocks — for example, VMs that were created from the same template. With data deduplication enabled, Veeam Backup & Replication does not store to the resulting backup file identical data blocks and space that has been pre-allocated but not used."

I'll post again later after I actually perform a few backups and recovery tests. I'm just really excited I have it configured to identify the source and targets and that it can all be scheduled.

*edit: Discovered that I'll want to move this Veeam vm I installed on my laptop into a vm on the esxi host. I don't want to keep my laptop on and connected to ethernet each night to schedule nightly backups. I'll migrate the veeam guest into a new vm on the esxi server tomorrow and retest.
 
Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
I've added the veeam backup VM to ESXi and works great so far. Full back up of a 600GB VM took just 32 minutes across my gigabit network using simple spinners and no RAID array striping. The resulting backup was 125GB. The incrementals of course will be much faster/smaller backups.


backing up using veeam (32 minutes):
upload_2019-3-21_13-21-30.png

restoring using veeam:
upload_2019-3-21_18-21-45.png

A full restore of that 600GB VM with three drives took about 2hr 15 minutes.
Restore completed:
upload_2019-3-21_18-22-52.png

Verifying the filesystems and logical volumes are okay on the restored system:
upload_2019-3-21_18-24-58.png

So far I'm very pleased with performance by running veeam in a vm on the ESXi server. By the way, veeam community edition "only" allows up to 10 VMs to be backed up (scheduled with incremental backups). But I found out I can do unlimited VM backups using veeamzip utility if I don't mind making them full backups (not incremental).

Note though that veeam community edition will not work with free version of ESXi. You must have a licensed ESXi. I got my vSphere license keys really cheap using VMUG Advantage.

Edit: here is the veeam report of the oracle linux vm 600GB backup process (of the 600gb only about 400GB used)
upload_2019-3-22_9-25-9.png

and here is the veeam report for a 900GB plexmediaserver vm (of the 900GB only about 600GB used)
upload_2019-3-22_9-34-19.png

Note that the plex backup took much longer. Partly because it had more stuff on it. But mostly because the movies on it were already heavily compressed and veeam was not able to compress them further. This meant veeam had to pass much more data through the network to the target backup repository and therefore took 1.5 hrs compared to the other backup that only took 32 minutes.

Edit2:
Also, when running veeam from a vm on the ESXi host, it can attach itself directly to the VM that is being backed up... it dynamically attaches the virtual disks from the vm being backed up to the veeam vm. It couldn't do this when I had veeam running from my laptop. End result is that it is much much faster for me when running veeam in a vm on the esxi host that I'm backing up. After the backup completes, the virtual disks are automatically removed from the veeam vm. Pretty cool
upload_2019-3-22_10-7-21.png
after the backup the virtual disks from the vm being backed up are removed from the veeam vm:
upload_2019-3-22_11-8-25.png
 

Attachments

Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
Success!.. successfully moved one of my bare metal Oracle APPS/EBS Database environments to the new ESXi server! Note, this is not simply installing an Oracle appliance with pre-configured EBS. These EBS and database nodes were built from scratch using Oracle Rapid Install process. Migrated using Oracle's migration tool "Rapid Clone" from the standalone boxes to guest VMs running on my new ESXi server. So far so good. Now I just have about 5 more to do. Then a few non APPS databases.

I'm feeling very proud of the new setup especially now that I also have scheduled backups. I'm very excited to try upgrading to latest EBS R12.2.8 or to try the new Oracle 18c database. So many possibilities with my new ESXi server that I didn't have available before on bare metal boxes.

I'm going to want to buy some more RAM soon. I have 64GB now but I think I'm going to want 128GB. Nice to see memory prices beginning to fall.

upload_2019-3-24_21-4-13.png
 
Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
Yesterday while adding wedding photos to my plexmediaserver VM's photo library I made a bad mistake AS root...

I was logged in as root user to the linux VM. Thinking I was in the folder for plex library content I ran "chmod -R 755 *" to correctly set few permissions on couple of files. Except I wasn't in my .../plexdrive/ subfolder, I was in "/". Took me a moment to figure out why it was taking so long to run a simple command. I couldn't figure out how to reverse this. It affected the system in such a way that I was unable to ssh into it from another shell window, and who knows what else that affected or how far that chmod command propagated, screwing up file and folder permissions.

Fortunately, my plex VM is not super large. It is just under 1TB and I have incremental backups weekly using veeam. I began a full recovery last night and it restored perfectly. I have three backup repositories for my various VMs. This plex VM was backed up to one of those on a single HDD repository located on another local physical server via gigabit. It took 5 hrs and 10 minutes to fully restore. Not bad. It would have perhaps restored faster if I told it to check blocks and only restore blocks that had changed since the last backup, not sure. But I felt better doing a full restore.

That was my first real recovery using Veeam (it wasn't just a test recovery) and very pleased how it went. To avoid this happening again I will change the owner of the plex library content to a new user, something like "plexuser" perhaps (currently owner is root for that library). Then I wouldn't be logging in as root when adding content or changing file permissions.

Are you guys with HUGE plex library backing up your content?
 

Marsh

Moderator
May 12, 2013
2,549
1,399
113
Are you guys with HUGE plex library backing up your content?
My media files are stored in a NAS , easier to play and restore a VM with just media server software.
Most of the time, if I am working on the VM, take a snapshoot first, it something goes wrong, revert to previous snapshoot.

If you do not have a real physical NAS , created a virtual NAS fileserver.
 

BennyT

Active Member
Dec 1, 2018
146
39
28
Hi @Marsh Thanks for the reply.

I'll research your suggesting regarding setting up a NAS (physical or otherwise) to house a few VMs.

Edit: removed my wall of text. Basically I will continue to use direct attached storage on my ESXi host chassis. Eventually I will experiment with a well built NAS to handle datastore(s).
 
Last edited:

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
506
113
Yesterday while adding wedding photos to my plexmediaserver VM's photo library I made a bad mistake AS root...

Are you guys with HUGE plex library backing up your content?
As someone who's also nuked permissions on the wrong tree before, it's a good idea to be able to back up the permissions separately from the files - it probably wouldn't have helped you much in this particular case if the system became unusable (although I dare say you could have done the steps below in single user mode), but you can cut down your restore time considerably by just restoring the permissions.

Use a command line something like this in cron/root crontab to back up all permissions at a set interval (assuming you're using an ACL-enabled filesystem and have the ACL utils installed; the backslashes in the date sub are there to escape the % symbols inside the crontab, you don't need them in a regular shell):
Code:
getfacl -R /somedir > /var/backups/acl/somedir_permissions_$(/bin/date '+\%a_\%H\%M'|tr [A-Z] [a-z])
You can of course just use the root path to do the whole system at once but I prefer a bit more granularity, then you can simply restore permissions with a command like the following:
Code:
setfacl --restore=/var/backups/acl/somedir_permissions_wed_2105
Tadaa! Hopefully a more-or-less instant (and correct!) restore of all your permissions, including ACLs if you're using them.
 
  • Like
Reactions: Evan and BennyT

BennyT

Active Member
Dec 1, 2018
146
39
28
Edit: removed my wall of text which was mostly erroneous. I was having serious lag which I thought was storage related but it turned out to be from my network setup. As mentioned in following post the issue was that I had my two physical NICs on the ESXi host setup as teaming in ESXi but the physical switch they were connected to was not capable of handling it which caused weird network looping/lag.

I did have hdd latency issues, but was those were not what was causing my much more serious lag from network.



Benny
 

Attachments

Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
Another issue I've been having with laggy response is when I'd connect through my router to a VM via Remote Console (workstation pro) or ssh into ESXi linux command line. I'd get serious lag where if I type keypresses I'd have to wait about 10 seconds for it to appear.

Turned out to be because I had my two physical NICs setup as teaming, but I have an unmanaged switch. This caused lag if going through my router -> then to my TP-Link unmanaged switch which had the two NICs from the ESXi host connected.

I changed this so that the 2nd NIC from my ESXi host acted as failover instead of teaming. That fixed the lag issues described above. I originally thought this lagg was from HDD latency(see my posts above), and it may have been HDD latency in part, but mostly from this issue trying to team NICs using an unmanaged switch.
 
Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
Applying a few patches to my Oracle Applications tiers running EBS 12.2.7. The patches fix having to use java plugin from the client side web browsers to instead use JWS (Java Web Start kit) when bringing up Oracle EBS/APPS. The point of me mentioning this is that I've allocated 12 vCPU cores to my guest linux systems while I'm applying the patches. I'm so pleased with how this setup is working by having my oracle database and application tiers in VMs... I have 12 workers (12 cores) while applying the patches to my patch filesystem while I still have my "run" filesystem up and running in the same guest VM (online patching in Oracle EBS R12.2+ is cool).

The patch cycles can run for hours and hours while the run filesystem continues to be up. Then there is fast cutover so that outage is minimal to the users. I did this in another VM where I gave it 6 vCPUs and it ran for 2.5 hrs followed by another 2 hrs to clone filesystems (cutover takes under 10 minuetes but fs clone takes a few hrs). I'm hoping that by throwing twice as many cores at this Guest VM I'll be able to measure an improvement in how long it takes to apply my oracle patches.

*edit: patch just finished and took only 50 minutes compared to same patches being applied to another Guest that had half as many cores, and that one took 2+ hrs. Yes!

I've also gone back to using RAID0 striping in this Linux VM (spread accross 4 vhdds, each on seperate physical disks), while I have the root,boot and swap filesystems and partitions on a seperate 100GB disk (linear not RAID). I'm still running mixture of various sized HDD directly attached to the ESXi host. These are NOT fast drives, but in RAID0 accross 4 disks for this VM I get about 260MBps read which is good enough for me until I can get SSDs.

I just wanted to report that I'm very happy since virtualizing my Oracle environments.
vCPU.png


*plus I still have my daily, weekly and monthly incremental backups using Veeam. I love this setup
veeam.png
 
Last edited:

BennyT

Active Member
Dec 1, 2018
146
39
28
So it's been almost a year since I began this build. I've been using VMUG Advantage subscription and prepaid for 3 yrs.

I learned however that although I have three yr subscription the VMware product keys still expire each year. I just started getting this warning in vCenter Server about the license keys expiring soon.

I contacted VMUG Advantage Team at advantage@vmug.com and they replied back following day explaining that I'll have about 30 day from when updated keys become available to me (12/31/2019 based on when I first downloaded keys) and when my current license keys stop working in vCenter (01/30/2020). Nice to have that overlap, I was initially worried I'd have forced downtime until new keys became available to me.
2019-12-03_11-46-38.png

in vCenter Client it shows my keys expire on January 30 2020:
2019-12-03_11-56-05.png

and according to VMUG Advantage Team, they say I should get new keys beginning January 1 2020 after these Dec 31 2019 expiry dates are reached:
2019-12-03_11-59-07.png

This was my first time using VMUG so it was a little confusing to me how to renew keys each year and if I'd have any downtime. Hopefully I'll login to VMUG store January 1st and have new keys available to me. :)

-Benny
 

itronin

Well-Known Member
Nov 24, 2018
931
601
93
Denver, Colorado
@BennyT, I *just* went through this. It is a bit confusing.

edit: Oh - one thing I'll note. I pay year to year so maybe that makes a difference for the process I used?

I renewed my membership before it expired and I waited and waited and my keys never showed up. there's actually some bit of a grace period after the original VMUG order date before they will expire out as you are seeing in your vsphere client: 30 days.

I believe their description of the process is poor.

From what I figured out - reading posts etc you have to "reorder" your VMUG products each year (that is explicitly said somewhere in the VMUG process). The update keys automatically appearing - I dunno but the reorder generates the keys and then they do appear and you simply load the new keys in your installation reassign and all is well. They keys showed up within an hour of my reorder order (IIRC).

Maybe I'm doing it wrong? but it worked for me.