lga3647 esxi build *Nov 2024 --> Began Switching from vSphere to Proxmox

[BennyT]

I went with 6x 64GB LRDIMMs for 384GB total. I'm happy I went with six. MEM-DR464L-HL03-LR26. *edit: corrected part number to HL03


Gotta have a photos after spending $:

the labels on the plastic were smudged because I sprayed them with lysol. LOL.



I now have two used 32GB 2666MHz DDR4 RDIMMS back in a box waiting to be used again some future day:



about 9 months ago I enabled Transparent Page Sharing (TPS) for deduplicating memory pages and also disabled large memory pages (smaller memory pages increased chances of being able to deduplicate those smaller pages between VMs). This article explains it well: VMware memory optimization

mem.ShareForceSalting = 0 *enables TPS on all VMs of the host
Mem.AllocGuestLargePage = 0 * disables large memory pages

The above settings would keep my consumed host memory below 64GB while running my necessary guest VMs. Without TPS the VMs that were running were consuming too much RAM and balooning would occur followed by swap files and dog slow performance.


But now that I have plenty of RAM at the moment (for now) I've reset those values to the following:
mem.ShareForceSalting = 2 *disables TPS on all VMs of the host
Mem.AllocGuestLargePage = 1 * enables large memory pages (advantage is better performance relating to MMU but at cost more RAM being consumed by each VM because larger pages means less chance of deduplication if using TPS)

If you have enough RAM do you still enable TPS (VM page sharing) and small memory pages?... or do you prefer large memory pages and therefore no page sharing but having greater MMU performance?

Thanks

 

[Rand__]

Ah cant help you with that, my lab is on steroids, so never looked into conserving memory options

 

[EffrafaxOfWug]

Do you that use vSphere and have enough RAM do you still enable TPS (VM page sharing) and small memory pages?... or do you prefer large memory pages and therefore lesser to none page sharing but having greater MMU performance?

I've not used ESX at home since the 5.5 days, but both in the homelab and at work the saving from page-sharing were basically minimal in my environments; you'd only reclaim a few hundred MB tops from each VM from shared common files, and only then if you had the same OS and versions for everything. Since most of RAM is usually full of one application or another rather than system libraries, the vast majority of pages were unique to that VM.

Page-sharing is/was a much bigger benefit to the people that run huge swathes of largely homogenous VMs at a very high consolidation ratio but for most workloads I don't think it's worth it.

(Anecdotal, as I didn't come across a way of measuring it objectively back in the day and at the last two companies page-sharing was kept disabled for security reasons - cheaper to buy more RAM than to spend time dealing with troublesome auditors)

 

[BennyT]

Most of my VMs up to this point have similar environments, same kernal and release distro, same applications (different releases). The only differences being different variations of Oracle EBS Applications or Databases and my proof of concepts and development etc. but the OS are mostly the same.

With small memory pages enabled (4k) rather than large memory pages (2-4MB?) I could see pretty good memory savings by them sharing.

Using small page (4k) sharing:


But with Large page enabled and with TPS turned off there is basically no sharing:


I have enough RAM now and I do not need to mess with memory Optimizations. But if they don't hurt having those optimizations enabled I may turn them back on. Sounds like there is some benefit though in using large memory pages if not needing to have memory sharing.


On completely different subject, in the next month I want to figure out how to change my homelab subnet from 192.168.1.* to be somthing like Class C 255.255.255.0/24 so range: 10.10.77.1 – 10.10.77.254 for 254 devices.
or a Class B 255.255.252.0/22 MASK 10.10.76.1 – 10.10.79.254 for 1022 devices

Not sure what my consumer grade netgear R7800 wifi router is capable of, or if I need to get more involved and build my own router PC like I see some kids do with pfsense etc. Or if I need enterprise network gear. I'm not a network admin obviously.

if anyone has suggestions or recommendations on this subject of changing subnets feel free to comment below. Thanks! I have idea in my head what I may need to do and it sounds like pretty big project, but doable.

 

[edge]

I have run several different model netgear routers on the 10.0.0.0 255.0.0.0 and 172.16.0.0 255.240.0.0 private networks. Advanced/setup/lan setup: set router ip and configure starting and ending ip address for dhcp, apply, and then dhcp release/renew all your devices, then deal with static and reserved ips.

 

[BennyT]

I'm currently at ESXi v6.7u1 and vCenter Server Appliance v6.7 which I installed back around Feb 2019 from scratch by using the VMUG Advantage products that I downloaded from their online store.

vCenter v7 and ESXi v7 have been out since April of this year, so I'm thinking about upgrading. Problem is that I've not performed an upgrade before and I'm unsure exactly of the steps. I'm a little timid that I might bork my ESXi host.

VMUG Advantage has the ESXI v7 and vCenter Server v7 products available for download. Not sure if I am to use those VMUG products for upgrading or if those are used only for fresh installations. Or if I should seek the necessary upgrade files for ESXi v7 and vCenter Server v7 from VMWare website or what?

Those of you who have done an ESXi and vCenter Server upgrade and were VMUG Advantage subscribers, what were your basic steps and did you use the VMUG v7 products for upgrading?



Ive been delaying this upgrade mostly because v7 is farely new still, but mostly because I don't want to break my working ESXi v6.7 host by messing up the upgrade.

Thanks!

Benny

 

[williedee]

Cool thread, just finished reading. I personally haven't seen many customers upgrade to version v7 yet, mostly because it is so new. If 6.7 is doing everything you need it to then you may want to wait a bit for the first or second service update.

 

[BennyT]

Thank you Willie. Yeah, I think that is wise decisision and I'm thinking to stay on 6.7. I don't see needing anything really on v7 except for experimenting with kubernetes or dockers.

 

[BennyT]

It is that time of year again for me when I need to download the license keys again from VMUG store. I tried to re-order the products to get the updated yearly keys from VMUG Advantage. I still have my 3 yr VMUG with 1 yr remaining on it. But each year I login to the store and it thinks my Advantage has expired.


They are usually very quick to respond when I email them. They give a 30day buffer window to download the new keys before the products stop working.

I'll post again with my results after emailing them.

Thanks,

Benny

Happy New Year.

Regarding multi-year VMUG subscriptions (such as a prepaid 3 yr subscription), and the reason for my posting... once 12 months have elapsed and you are now ready to re-download new 1 YR product license keys, you still have to email your VMUG rep. The reason is because the VMUG Webstore isn't aware of multi-year Advantage memberships. It requires a manual process where a VMUG rep will re-activate your membership again and then you can download your new keys.

I first subscribed to VMUG Advantage on Dec 31, 2018 with a 3 yr. That is also the date I downloaded license keys. So I'm ready to get new keys... I logged into VMUG webstore to download new license keys early this morning, but it said my advantage membership had expired. I emailed VMUG this morning and they replied by 8:30AM (on New Years day no less) and said they've just updated my webstore account to "active status". I was then able to download the new license keys.

Basically the keys last for 13 months. You can re-download new keys again after 12 months elapse, but not before. That gives you a 30 day window to acquire new keys. If you have multi-year Advantage, contact VMUG at advantage@vmug.com to remind them to re-activate your store account each year before you download another set of new keys..

One thing I noticed, the vSphere license keys are good for 12 CPU sockets now instead of 6 CPUs. Nice! Now I just need to buy more hardware and CPUs.

 

[BennyT]

wooo-eeee they are fast. They replied back to my email within 5 minutes and re-activated advantage for another year.

Time to download my license keys and see if I want to try any other products or upgrade. I think I'll stick at vSphere 6.7. But I'll upgrade my workstation Pro from 15.5 to 16.x and also check out VMWare Site Recovery. The recovery tool sounds a little bit like veeam, not sure yet. I'm very happy using Veeam community edition though, although I'm limited to only 10 scheduled backup jobs.

 

[BennyT]

One thing I just realized, it looks like vSphere 6.x (including 6.7?) will be retired from VMUG Advantage store soon. That may force me to upgrade to vCenter Server and vSphere (ESXi) to v7+ this year.

Even though I was happy on vSphere 6.7 I may have to upgrade to v7 because v6.x license keys may not be extendable next time mine expire.

Or maybe the vSphere v7 license keys will work with my v6.7 vsphere products?

 

[Rand__]

Good to know , thanks, had cancelled my VMUG Advantage for the time being. Prolly gonna renew when my Horizon View licenses expire.
Could you have a look whether Horizon v7 is set to expire too anytime soon? Thanks. Theoretically there is a Horizon 7 long time service version, but thats not sure to be on VMUGAd.

Regarding your question - I dont think keys for 7 will work for 6.
I am sure you know that 7 has (partially) dropped support for a bunch of devices, so check compatibility early

 

[BennyT]

Good to know , thanks, had cancelled my VMUG Advantage for the time being. Prolly gonna renew when my Horizon View licenses expire.
Could you have a look whether Horizon v7 is set to expire too anytime soon? Thanks. Theoretically there is a Horizon 7 long time service version, but thats not sure to be on VMUGAd.

Regarding your question - I dont think keys for 7 will work for 6.
I am sure you know that 7 has (partially) dropped support for a bunch of devices, so check compatibility early

the VMUG store has v8 and v7 of Horizon. There was a strange note on v7 that didn't make sense, like they accidentally have notes pertaining to vSan instead of Horizon (they have the same note under the vSan 6.x product).



I've emailed advantage@vmug.ccom to ask them if Horizon v7 will be retired from the store soon. Maybe they will reply as quickly as before.

 

[Rand__]

Thank you

 

[BennyT]

@Rand__
* received this email from VMUG today regarding Horizon v7 and how long it will remain in the Advantage Store:

Hi Benny,

Thank you for contacting VMUG. Unfortunately, we are unable to comment on how long products will be available but they are active on the storefront for the time being!

Hope this helps.

Please let me know if you have any additional questions. Thank you!

Molly Smith
VMUG Advantage Team

 

[BennyT]

Has anyone tried using JNVME1 (Oculink connectors) on the Supermicro boards to connect SATA drives?

I have two oculink connections each with PCIeX4 gen3. And supermicro sells this breakout cable...

CBL-SAST-0933

Supermicro 50cm OCuLink to 4 SATA Cable (CBL-SAST-0933)

Copper Cable Assembly OCuLink SFF-8611 (x4) to 4 SATA OCuLink Storage Expansion Supermicro Certified

store.supermicro.com

Do you think this cable would work for connecting 4 SATA drives to the JNVME1 oculink connector? I'd basically be using this oculink and cable instead of buying an LSI SAS2308 (9207-8i).

Thanks.




*edit: Aug 17, 2021 - Decided instead on 2nd CPU for 2nd socket, that opens up all of the PCIE x16 expansion slots for use in this mobo. Then we'll invest in an ASUS Hyper M.2 x16 Card V2 and high speed NVMe PCIe Gen3 SSDs, only down side is no powerloss prevention in those, but I think I'm okay with that.

 

[BennyT]

had a power outage at home and the vCenter Server Appliance v6.7 (vcsa) postgres DB was corrupted. Being a small vCenter Server and not having backed it up, I simply am re-installing from scratch the VCSA, but this time I'm installing v7u2.
*edit: the vcsa v7u2 install went super smooth and easy. had to reschedule backup jobs though because Veeam sees all the VMs as being different guests from what they were when under vcsa v6.7.
Now I'm one step closer to being able to upgrade the ESXi host from v6.7 to ESXi v7

 

[BennyT]

New parts should begin arriving today in mail:
-2nd XEON 6130 SRB39 (not an ES)
- LSI9300-8i card
- SFF-8643 to SFF-8087 cables
- Four 4TB SATA SSDs
-ASUS Hyper x16 M.2 card (PCIe gen4 version because the gen3 version i couldn't find in stock )
-Two 2TB NVMe M.2 SSDs

Explanation of my purchase decisions:

I'd like to have bought U.2 NVMe but i don't have a backplane for U.2 connections, and although i could've tossed out one of the six SAS backplanes and just straight up direct connected U.2 drives to a tri-mode NVMe controller, i just didn't feel like doing that.

And Im still unsure if the PCIe M.2 card will work with ESXi so i didn't want to go hog wild with a bunch of M.2 devices, so i ordered just two M.2 SSDs and the Hyper card to experiment with ESXi.

Which is why i mostly went with SATA SSDs. And the reason for LSI9300-8i instead of a 9207-8i is again because of ESXi compatibility. LSI9207 is not supported in ESXi v7.

I'm excited though. Feels good to upgrade and there's still allot of room in this chassis for future upgrades.

I really really need to research a good uninterruptible power backup, especially with these non Enterprise SSDs. And especially since last power outage corrupted the VCSA db which required mer einstalling and upgrading to vcenter 7.

* oh, also renewed to another 3yr VMUG advantage for only 180 total! You'll have to ask me how that sweet deal happened, if interested.

 

[BennyT]

2nd cpu installed successfully.

My original CPU has noctua heatsink with two 92mm fans.

The new CPU will be using supermicro's standard lga3647 cooler with one 80mm fan which was left over and unused from earlier experiment three years ago.

Initially i installed the new CPU with the supermicro heatsink in socket 2, near the back of the chassis.

The newly purchased CPU in socket 2 with supermicro cooler would run between 54c and 58c. Meanwhile, the original CPU with noctua cooler and two 92mm fans would be at 38c - 41c in socket 1.

Diagnosing, I removed both CPUs with their respective heatsinks from their sockets. I removed the supermicro heatsink from newly purchased CPU and discovered i put way too much thermal paste on it. Paste had oozed from between the heatsink and CPU. I cleaned that mess up and reapplied paste and heatsink. This time i put the newer CPU with supermicro cooler into socket#1, closest to the fan wall with the 120mm fans.

The original CPU with the better noctua cooler went in socket 2.

Now temps are 44c in socket 1 with the supermicro cooler nearest the 120mm fan wall.

Socket 2 temps with the noctua cooler are 41c.

Good!

Conclusions:

- I had too much paste.

- Noctua with 2x92mm fans is superior. Also runs 1000 RPM lower than single 80mm fan that's on the supermicro cooler

- socket nearest the 120mm fan wall benefits from their cooling

Both sockets now run within 3-4 degrees of each other.

 

[Rand__]

Nice
How did it go with bifurcating the Asus Card?