Ubuntu and vsphere deploy from template hell

wildchild · Jan 4, 2017

Hi guys,

Just wondering if any of you have experienced this and hopefully have quick and easy fix.
A client has the need for an autoscaling webserver farm.
They are running vsphere 6 and their preffered os is ubuntu 16.4
What i did is create a golden template minimal ubuntu 16.4 that gets its finally destination and latest updates/grades as either nginx proxy or webserver after the golden image has been cloned.
Here in lies the problem.

The template customization in all cases misforms the /etc/network/interfaces file.
The nic is always identified as ether, while the proper name would be Ens192
If i delete the interfaces file and rename the interfaces.BeforeVMwareCUstomization to interfaces, give a reboot, all goes as expected and i have network again,hence it can pickup the step 2 script.

While i of course could put the remove and rename in stage 2 script and bake it in the golden image, this is not really nice, and would make making change to stage 2 irritating difficult and would also mean breaking up the golden image each time.

Anybody ran in to this and have a work around ?

Thanx !

TuxDude · Jan 4, 2017

My experience is with CentOS instead of Ubuntu, though still hosted on vSphere 6 - I've stopped using VMware templates / golden-masters for Linux VMs. It is faster and easier to do fresh/unattended installs into blank VMs, and where the unattended-install script ends with configuring the puppet client which then takes over after the first reboot to configure the VM into whatever role is required of it.

At work, this all works using TheForeman to automate generating kickstart files, and the VMs PXE-boot from a SpaceWalk server, which is also where they pull all required files during the unattended-install (installing directly from update repo's, there are no updates to apply after install), then the puppet client again talks to TheForeman after first reboot as it is the puppet-master. At home in my lab I've got the next-gen working almost to the point where I'd considering using it at work, with Katello single-handedly serving the roles of TheForeman and SpaceWalk (technically, Katello is a plugin to TheForeman which add's the SpaceWalk patch-management functionality).

Comparing times to get a new VM running at work - whether it comes from a cloned template, or individual RPM files from the SpaceWalk server, in either case it's reading from the same Compellent FiberChannel SAN, over the same 8GB FC SAN fabric, and any/all network traffic over 10GbE - so having the install come in already fully updated means less data to move around and no templates to maintain/update, hence faster and easier.

wildchild · Jan 4, 2017

Well.. i tried it with centos,suse,oracle and red hat , it works just fine.
The reason for using images is a bit security wise: these are webservers running in a dmz and except ssh its a pretty closed env.
I use ansible to do the stage 2 things i need, but otherwise it's a pretty closed off part of the network.
Also , on the normal part of the network i tested pxe vs golden image and golden image still has the upperhand in their env.
Thing is , clusters are being dynamicly expanded based on monitoring load metrics, so speed, next to closed off network parts are things to keep in mind.

Thing is, client is pretty strict in what os they want to use and they are pretty heavy set on ubuntu

Just cant believe vmware simply wont work with debian/ubuntu and mint ( these are the oses i have tested)

nitrobass24 · Jan 4, 2017

I recall having this same issues previously, but it was supposedly fixed back in Ubuntu 10 I believe. Is their 16.04 image a fresh install or the bastard of several upgrades? That said all my enterprise customers use Cent/RHEL

EDIT: Bug #440179 “service fails to start/stop/restart networking daem...” : Bugs : sysvinit package : Ubuntu

wildchild · Jan 4, 2017

Fresh install... for a dvd.

It really seems like a vsphere crap up here, somehow not using the right nic name and not putting its data in resolve.conf but jamming it into the interfaces file.
Renaming the original file ( interfaces.BeforeVMwareCustomization) back to interface fixes it all.
Was hoping it might be fixable by fiddling with the customization xml

nitrobass24 · Jan 4, 2017

So I just cloned my Unifi Controller running on Ubuntu 16.04.1 min install and the interfaces came up just fine.

wildchild · Jan 4, 2017

nitrobass24 said:
So I just cloned my Unifi Controller running on Ubuntu 16.04.1 min install and the interfaces came up just fine.

The problem isnt the cloning part of it.
Its the customization after by the customization template.
Network specifcally. All others seem to do what they are supposed to do

nitrobass24 · Jan 4, 2017

Here is my Customization XML, I just used DHCP for the interfaces. Happy to try setting it as static or anything else if you want.

Code:

<ConfigRoot>
  <_type>vim.CustomizationSpecItem</_type>
  <info>
    <_type>vim.CustomizationSpecInfo</_type>
    <changeVersion>1483562147</changeVersion>
    <description/>
    <lastUpdateTime>2017-01-04T20:35:47Z</lastUpdateTime>
    <name>Ubuntu</name>
    <type>Linux</type>
  </info>
  <spec>
    <_type>vim.vm.customization.Specification</_type>
    <globalIPSettings>
      <_type>vim.vm.customization.GlobalIPSettings</_type>
    </globalIPSettings>
    <identity>
      <_type>vim.vm.customization.LinuxPrep</_type>
      <domain>davis.local</domain>
      <hostName>
        <_type>vim.vm.customization.PrefixNameGenerator</_type>
        <base>ubuntu</base>
      </hostName>
      <hwClockUTC>true</hwClockUTC>
      <timeZone>America/Chicago</timeZone>
    </identity>
    <nicSettingMap>
      <_length>1</_length>
      <_type>vim.vm.customization.AdapterMapping[]</_type>
      <e id="0">
        <_type>vim.vm.customization.AdapterMapping</_type>
        <adapter>
          <_type>vim.vm.customization.IPSettings</_type>
          <ip>
            <_type>vim.vm.customization.DhcpIpGenerator</_type>
          </ip>
        </adapter>
      </e>
    </nicSettingMap>
    <options>
      <_type>vim.vm.customization.LinuxOptions</_type>
    </options>
  </spec>
</ConfigRoot>

wildchild · Jan 4, 2017

Let me see if yours will run.
Strange thing is, i am missing dns servers search domains in your file, and i am seeing those in mine.
When creating a template customization, it will not allow me not to add them, except when i edit the xml and import

nitrobass24 · Jan 4, 2017

wildchild said:
The problem isnt the cloning part of it.
Its the customization after by the customization template.
Network specifcally. All others seem to do what they are supposed to do

Right, you are defining in the XML custom network settings and its not working. Mind posting that section of your XML?

nitrobass24 · Jan 4, 2017

wildchild said:
When creating a template customization, it will not allow me not to add them, except when i edit the xml and import

That is super odd. You mean you dont see these steps in the Vsphere UI, during customization? What version of VSphere are you on?

Re-ran again but with 1 NIC set to static IP, 1 NIC on DHCP and custom DNS servers and search domains.

Code:

<ConfigRoot>
  <_type>vim.CustomizationSpecItem</_type>
  <info>
    <_type>vim.CustomizationSpecInfo</_type>
    <changeVersion>1483562936</changeVersion>
    <description/>
    <lastUpdateTime>2017-01-04T20:48:56Z</lastUpdateTime>
    <name>Clone of Ubuntu</name>
    <type>Linux</type>
  </info>
  <spec>
    <_type>vim.vm.customization.Specification</_type>
    <globalIPSettings>
      <_type>vim.vm.customization.GlobalIPSettings</_type>
      <dnsServerList>
        <_length>1</_length>
        <_type>string[]</_type>
        <e id="0">192.168.10.215</e>
      </dnsServerList>
      <dnsSuffixList>
        <_length>1</_length>
        <_type>string[]</_type>
        <e id="0">davis.local</e>
      </dnsSuffixList>
    </globalIPSettings>
    <identity>
      <_type>vim.vm.customization.LinuxPrep</_type>
      <domain>davis.local</domain>
      <hostName>
        <_type>vim.vm.customization.PrefixNameGenerator</_type>
        <base>ubuntu</base>
      </hostName>
      <hwClockUTC>true</hwClockUTC>
      <timeZone>America/Chicago</timeZone>
    </identity>
    <nicSettingMap>
      <_length>2</_length>
      <_type>vim.vm.customization.AdapterMapping[]</_type>
      <e id="0">
        <_type>vim.vm.customization.AdapterMapping</_type>
        <adapter>
          <_type>vim.vm.customization.IPSettings</_type>
          <gateway>
            <_length>1</_length>
            <_type>string[]</_type>
            <e id="0">192.168.10.1</e>
          </gateway>
          <ip>
            <_type>vim.vm.customization.FixedIp</_type>
            <ipAddress>192.168.10.190</ipAddress>
          </ip>
          <subnetMask>255.255.255.0</subnetMask>
        </adapter>
      </e>
      <e id="1">
        <_type>vim.vm.customization.AdapterMapping</_type>
        <adapter>
          <_type>vim.vm.customization.IPSettings</_type>
          <ip>
            <_type>vim.vm.customization.DhcpIpGenerator</_type>
          </ip>
        </adapter>
      </e>
    </nicSettingMap>
    <options>
      <_type>vim.vm.customization.LinuxOptions</_type>
    </options>
  </spec>
</ConfigRoot>

wildchild · Jan 4, 2017

Vsphere 6 u2..
And my xml looks just like the second one you posted..
Now this is odd..
This would imply i am missing packages in my golden image that are preventing something.
Could you check your machine and see if there is a /etc/network/interfaces.BeforeVMwareCustomization ?

5mall5nail5 · Jan 4, 2017

Make sure you have perl installed in the original image or customization doesn't work

nitrobass24 · Jan 4, 2017

wildchild said:
Vsphere 6 u2..
And my xml looks just like the second one you posted..
Now this is odd..
This would imply i am missing packages in my golden image that are preventing something.
Could you check your machine and see if there is a /etc/network/interfaces.BeforeVMwareCustomization ?

Yep its here.

Code:

unifi@ubuntu-0:~$ cat /etc/network/interfaces.BeforeVMwareCustomization
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens160
iface ens160 inet dhcp
gateway 192.168.10.1

#UBNT Discover interface
auto ens192
iface ens192 inet dhcp


unifi@ubuntu-0:~$

Code:

unifi@ubuntu-0:~$ cat /etc/network/interfaces
iface lo inet loopback
auto lo

auto ens160
iface ens160 inet static
address 192.168.10.190
netmask 255.255.255.0
up route add default gw 192.168.10.1
auto ens192
iface ens192 inet dhcp
dns-search      davis.local
dns-nameservers 192.168.10.215
unifi@ubuntu-0:~$

wildchild · Jan 4, 2017

5mall5nail5 said:
Make sure you have perl installed in the original image or customization doesn't work

Did install perl, but surely worth double checking.

Thanks guys, will start with that first thing tomorrow morning.

nephri · Jan 4, 2017

just to say, make sure also to regnrate ssh keys identifier after using a clone
it's a way to have multiples server with same keys...

Like said, i use CentOS, and prefer a fresh install from the minimal iso.
After that,i install required packages.
today, with package manager (yum,apt,...) it's so simple and quick...

But i will give look to TheForeman, i didn't know.

Search

Ubuntu and vsphere deploy from template hell

wildchild

Active Member

TuxDude

Well-Known Member

wildchild

Active Member

nitrobass24

Moderator

wildchild

Active Member

nitrobass24

Moderator

wildchild

Active Member

nitrobass24

Moderator

wildchild

Active Member

nitrobass24

Moderator

nitrobass24

Moderator

wildchild

Active Member

5mall5nail5

Active Member

nitrobass24

Moderator

wildchild

Active Member

nephri

Active Member