Need thoughts/ advice on the STH lab provisioning tool

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,803
113
After a bunch of reading and trial, I think I need opinions on this. The goal is to have a default provisioning tool for the STH lab. When we had 5-10 nodes in there, many of them running fairly static tasks, it was not a big deal. At this point I think we have upwards of 30 nodes so the time and need for a better solution is here.

Bare metal provisioning is an absolute must and is really the primary need here. I also think being able to install Ubuntu, Debian, RHEL, CentOS and Windows Server should be on the list as a requirement. If something had the ability to create IPMI users (Operator level maybe?) and give those login credentials out that would be even better.

My eventual goal is to take all this really cool hardware we get an have an easy system to get sets of hardware ready for STH reviews, STH guides and eventually do a service where we let folks log on some really cool machines and run their workloads on them. I do also want to be able to provision nodes then have them be used for folks to do how-to guides on (e.g. here is how to do a Kubernetes cluster, Hyper-V server 2016 cluster and etc.)

A harder point is that the STH budget is extremely tight right now and I only get 1-2 hours to work on this at a time with those work sessions sometimes separated by days. I also do want to make whatever gets done here into a how-to series.

Options I have been thinking about:
  • MAAS - I think I have to re-do (e.g. wipe) everything to start using this in the STH lab. It seems like the OpenStack installer always picks a node that fails, causing a compile re-do every time it runs.
  • Foreman - it seems like there is a good community behind this one
  • Puppet - This is a possibility I am looking at fairly strongly as it is well supported
  • RedHat Satellite - This is a strong possibility but it does move me back to learning RedHat v. sticking with Ubuntu/ Debian.
  • Razor - seems like support for this one is waning but maybe in conjunction with Puppet
  • CloneZilla + PXE boot - simple yet not really the level of provisioning I want
  • RedHat SpaceWalk - I looked at this, and am just less excited about it although it does support most of the OSes we need.

MAAS in the STH lab:
  • I do like the fact that it is Ubuntu based. I do a lot with Ubuntu
  • I also like the fact you can tell MAAS to erase nodes upon release. This should be a requirement of mine.
  • Does not support the Xeon D X552/ X557 onboard NIC using the out of box 14.04 commissioning OS. For nodes that have 1GbE, this is not really an issue. For nodes with 10GbE only, this can be challenging.
  • I like having pfSense manage DHCP. That solution works well since I am using pfSense for the VPN gateway. Assume there will always be some amount of hardware that I cannot auto-provision.
  • Launchpad is only free for 10 machines and starts to get too costly too quickly
  • I am a little bit annoyed at how fragile the Ubuntu OpenStack install is, and how much it assumes 100% fresh/ new hardware. E.g. this killed my last OpenStack install:
Code:
BLKRRPART: Device or resource busy
failed to partition /dev/sda [
Disk /dev/sda: 24321 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
/dev/sda: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sda1   *      2048 390721934  390719887  83  Linux
/dev/sda2             0         -          0   0  Empty
/dev/sda3             0         -          0   0  Empty
/dev/sda4             0         -          0   0  Empty
Successfully wrote the new partition table
The fact that Xeon D node had a non-empty disk meant that the Ubuntu OpenStack installer went into a loop of unhappiness:
Code:
patrick@maas01:~$ sudo openstack-install -u
Warning:

This will uninstall OpenStack and make a best effort to return the system back to its original state.
Proceed? [y/N] y
Restoring system to last known state.
Ubuntu Openstack Installer Uninstalling ...Could not determine install type, was /home/sthroot/.cloud-install removed prior to running the uninstallation?
patrick@maas01:~$ sudo openstack-install

Error:

Previous installation detected. Did you mean to run openstack-status instead?
If attempting to re-install please run     $ sudo openstack-install -u


patrick@maas01:~$ sudo openstack-install -u
Warning:

This will uninstall OpenStack and make a best effort to return the system back to its original state.
Proceed? [y/N] y
Restoring system to last known state.
Ubuntu Openstack Installer Uninstalling ...Could not determine install type, was /home/sthroot/.cloud-install removed prior to running the uninstallation?
patrick@maas01:~$
Any thoughts/ advice would be greatly appreciated as I am starting to feel the pain and do not have a ton of time (I am aware that no time, low budget, and no clue is a bad combination.)
 
  • Like
Reactions: Marsh

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
This may not be the most helpful comment you are going to get on this thread, but here's my take.

You try to learn too much. You need someone who has deployed something like this before to set it up for you. There I said it. Does the STH lab run STH production? If not, maybe someone who aspires to do this or has done it before on STH can help if you can get them remote access?

I see you go into inane microarchitecture details, you admin WP+XF front ends + back ends, Proxmox, the underlying site hardware, networks between hosting sites, write articles like crazy about components, servers, storage, networking, shucking hard drives, you know way too much about used hardware and what's a deal. I read a post yesterday where you were trying to reset a cumulus Linux password. I think its safe to venture that you also have other non-technical STH stuff going on. Doesn't that seem like too much for one person to know?

If I had the automation skill, I'd offer to do this just as a learning experience. Maybe there's someone out there that can do in 4 hours what it would take you 40 hours do to.
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
After a bunch of reading and trial, I think I need opinions on this. The goal is to have a default provisioning tool for the STH lab. When we had 5-10 nodes in there, many of them running fairly static tasks, it was not a big deal. At this point I think we have upwards of 30 nodes so the time and need for a better solution is here.

Bare metal provisioning is an absolute must and is really the primary need here. I also think being able to install Ubuntu, Debian, RHEL, CentOS and Windows Server should be on the list as a requirement. If something had the ability to create IPMI users (Operator level maybe?) and give those login credentials out that would be even better.

My eventual goal is to take all this really cool hardware we get an have an easy system to get sets of hardware ready for STH reviews, STH guides and eventually do a service where we let folks log on some really cool machines and run their workloads on them. I do also want to be able to provision nodes then have them be used for folks to do how-to guides on (e.g. here is how to do a Kubernetes cluster, Hyper-V server 2016 cluster and etc.)

A harder point is that the STH budget is extremely tight right now and I only get 1-2 hours to work on this at a time with those work sessions sometimes separated by days. I also do want to make whatever gets done here into a how-to series.

Options I have been thinking about:
  • MAAS - I think I have to re-do (e.g. wipe) everything to start using this in the STH lab. It seems like the OpenStack installer always picks a node that fails, causing a compile re-do every time it runs.
  • Foreman - it seems like there is a good community behind this one
  • Puppet - This is a possibility I am looking at fairly strongly as it is well supported
  • RedHat Satellite - This is a strong possibility but it does move me back to learning RedHat v. sticking with Ubuntu/ Debian.
  • Razor - seems like support for this one is waning but maybe in conjunction with Puppet
  • CloneZilla + PXE boot - simple yet not really the level of provisioning I want
  • RedHat SpaceWalk - I looked at this, and am just less excited about it although it does support most of the OSes we need.
I tried Razor but failed to run it successfully - my fault I'm sure. MAAS was fairly easy to get to production mode but eventually had some limitations (can't remember the details) that we could not live with. Currently running Foreman, which wasn't that bad to get running. It's too dependent on Puppet whereas we use Chef, but other than that it's OK. No experience with the others.

All of these bare-metal tools seems pretty immature. I wish that there was something mature and solid.

Conceptually only:
Spacewalk seems too monolithic
Satellite is an improvement to Spacewalk, but far too RHEL focused
Clonezilla seems focused on imaging for a lab, not bare-metal provisioning in a data center
 
  • Like
Reactions: Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,803
113
@MiniKnight - point taken

@dba - I think that is what I am finding out. The other issue I have is high churn/ low stable base. Actually, the public cloud like networking is too complex for what I need as well.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Options I have been thinking about:
  • MAAS - I think I have to re-do (e.g. wipe) everything to start using this in the STH lab. It seems like the OpenStack installer always picks a node that fails, causing a compile re-do every time it runs.
  • Foreman - it seems like there is a good community behind this one
  • Puppet - This is a possibility I am looking at fairly strongly as it is well supported
  • RedHat Satellite - This is a strong possibility but it does move me back to learning RedHat v. sticking with Ubuntu/ Debian.
  • Razor - seems like support for this one is waning but maybe in conjunction with Puppet
  • CloneZilla + PXE boot - simple yet not really the level of provisioning I want
  • RedHat SpaceWalk - I looked at this, and am just less excited about it although it does support most of the OSes we need.
Just to make some of the relationships between the above options obvious....

RedHat Satellite is the RedHat product that gives you a local management server (local mirror of packages to distribute, etc.) instead of using their "cloud-based" subscription-management/etc. software - RedHat Network. Like virtually all of RedHat's products, the commercial 'Satellite' product has an upstream free/open-source equivalent - Satellite versions up to and including 5.x are based on the SpaceWalk project, while Satellite 6.x is based on Katello/Foreman. Katello is a plugin to Foreman that adds the functionality for mirroring patch repositories, pushing packages to nodes, etc. (gives the combo mostly feature-parity with SpaceWalk from a patch-management point-of-view), and Foreman is a front-end to Puppet (the config-management in the older SpaceWalk is .... not good), though it can work with other config-management software as well (eg. salt, etc.) the puppet integration runs pretty deep.

My thoughts... With the above in mind, and no need for managing official RedHat subscriptions, support contracts, etc., the options are reduced to just MAAS vs Foreman (or maybe Razor, I know nothing about it). Forget about claims that SpaceWalk supports debian, you don't wanna go there - better chance that Pulp (the content management back-end behind Katello) will get Debian-style repo support than to get the hacked-together SpaceWalk support for it working. MAAS has better integration with the rest of the Ubuntu stack - personally I hate Ubuntu though so whether you consider that integration an advantage or not I leave to you. Foreman on the other hand can optionally have a lot of integration with the RedHat stack (especially with Katello), or you can just ignore those parts of it and use it as a web-GUI for Puppet, which supports pretty much everything. And so my vote would be for Foreman - as just a Puppet front-end that also brings a bit of automation to the initial deployment process (everything needed to get to the point where Puppet can do the rest) it should be able to work with just about anything that crosses through the lab.
 
  • Like
Reactions: Patrick and Marsh

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,803
113
Just an update to this, I finally had a little "me" time and put TheForeman in a Ubuntu 14.04.4 LTS VM and setup pfSense for it.

It did take a bit more time than I had wanted to get this far, but fairly excited:
upload_2016-6-24_15-30-11.png

For anyone who is wondering what that is, it is a Pentium D1508 system that had Ubuntu 14.04.4 LTS installed automatically. Logged in and networking is working.
 
  • Like
Reactions: RandyC and Chuntzu

Chuntzu

Active Member
Jun 30, 2013
383
98
28
I really like all these deployment t tools, curious how you like this vs maas or any other tools you have used. I am using ipxe chain loading pxelinux and then some pressed files I have cobbled together to iscsi boot. So I can use wds and automate some Linux installs. I really liked the look and feel of maas each time I have used it. But very little I formation on customizing the curtin Installer for maas. Forman has always peaked my interest but since I haven't spent time working with puppet I'm not sure how useful it is until I master puppet.

Sent from my SM-N920T using Tapatalk
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,803
113
I actually like Foreman documentation a bit better than MAAS. Not too hard to setup. I put it in a Hyper-V VM on my desktop and it is now provisioning servers. I do feel like there is a big learning curve left.
 
  • Like
Reactions: Chuntzu

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Whew, just installed foreman as well. Gotta say, SLICK...and this is coming from a guy who has administered LARGE Spacewalk, Satellite 5.x and Satellite 6.x env's. Got the foreman-discovery plugin magic happening and also toying w/ foreman-discovery-image for PXE/DHCP/TFTP-less environments but that is where all the fun is really for cloud svcs.

Got Ubuntu 14.04 LTS and CentOS 7 auto-provisioning rockin' so far, Windows up next!

Good stuff!
 
  • Like
Reactions: RandyC and Chuntzu

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Hmm, hung up on CentOS7 provisioning, dunno what the hell is wrong but here is CentOS6 and Ubuntu 14.04 LTS provisioning from Foreman.

foreman-provisioning.png

EDIT: Had to untick the 'Minor Version' under Hosts -> Operating Systems -> CentOS7 entry I made. Dunno why under CentOS6 I selected Major 6/Minor 8 (since 6.8 released a month or so ago), maybe CentOS7 doesn't go by RHEL 7.2 definitions but like the 1511 release/etc. Who knows, it resolved gracefully the minor though w/out anything in there, only major has a * (required) field so that's good to know.

Took off like a rocket after that, rock on!

foreman-provisioning-centos7-success.png
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,803
113
@whitey - getting stuck here. Totally bummed:
upload_2016-6-28_7-0-22.png

upload_2016-6-28_7-0-42.png

The host is nowhere in the foreman UI. Is that was you had an issue with?
 

Chuntzu

Active Member
Jun 30, 2013
383
98
28
Whew, just installed foreman as well. Gotta say, SLICK...and this is coming from a guy who has administered LARGE Spacewalk, Satellite 5.x and Satellite 6.x env's. Got the foreman-discovery plugin magic happening and also toying w/ foreman-discovery-image for PXE/DHCP/TFTP-less environments but that is where all the fun is really for cloud svcs.

Got Ubuntu 14.04 LTS and CentOS 7 auto-provisioning rockin' so far, Windows up next!

Good stuff!
I am pumped about foreman now after spending some time reading about it. My big push for it is to replace the need to wds and then maas, maas is to much of a pain customizing windows images and the complete lack of curtin Installer information sucks too. I just want to deploy any os from the tool. It would be great if deploying to iscsi can be integrated, which may be possible.

Sent from my SM-N920T using Tapatalk
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
@whitey - getting stuck here. Totally bummed:
View attachment 2902

View attachment 2903

The host is nowhere in the foreman UI. Is that was you had an issue with?
I have not yet tried Foreman Discovery Image .iso w/ my home lab foreman setup, use it everyday at work for Linux provisioning across routed layer 3 subnets w/out DHCP/PXE/TFTP protocols/capabilities. To get the auto-provisioning to work you definitely have to configure a host group, subnet, domain (unless you used the provisioning setup wizard and have these already). Then you MUST associate 'Hosts -> Provisioning Templates' and then associate 'Hosts -> Operating Systems' -> Templates' to align w/ provisioning template changes in previous step.

Then I just boot up a VM on the same network segment as foreman, it PXE/TFTP boots to foreman JEOS net installer, the discover option is not working for me but I bet FDI would if I used that. Hit tab from there to see mac addy or go to vSphere (whatever hypervisor you are using and get MAC addy of VM/host). Next select 'Hosts -> New Host' and fill out relevant details (I enter MAC addy), check other tabs for proper info, on the 'Operating Systems' tab ensure you resolve 'Provisioning Templates' and all looks well there. Finally when you are happy/confident that everything is good hit 'Submit', you may have to reboot box of use Foreman to 'Build Host' or if you click the host there will be a 'Build' button. I have had to reboot/loop VM's for it to kick off. Not quite a slick as Satellite 6 and kexec process but close.

I followed these links/guides. It's a bit of a trick to get provisioning going 'properly'

Foreman
Foreman (add these two apt sources to /etc/apt/sources.list)

Code:
deb http://deb.theforeman.org/ trusty 1.9
deb http://deb.theforeman.org/ plugins 1.9
wget -q http://deb.theforeman.org/pubkey.gpg -O- | apt-key add -
apt-get update
Then run the installer again if you didnt enable provisioning the first time arnd.

'foreman-installer --enable-foreman-plugin-discovery'

Let me know if you could use screenshots of the above described process, I snapped them but didn't wanna shit up forums unless it is of value.
 
Last edited:
  • Like
Reactions: Chuntzu

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,803
113
Thanks @whitey - I can get the manual provisioning to work, just not the auto discovery one. It would be nice not to key in information for a host every time.

I will give that a try this afternoon.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
This is what I get when I do a FDI provision. Kexec doesn't seem to be gracefully rebooting the boxes and kicking off installer, I now have to go and provision in foreman, then shutdown VM, detach FDI iso, boot back up and it takes off!

Need to sort this out for sure but FWIW this does work silky smooth and as expected w/ RH Satellite 6.x...go figure pay 10K for an enterprise product subscription/entitlement and all the anomalies are squashed.

foreman-fdi-discovery.png

'May' have to engage 'Discovery Rules' but if the kexec reboot/installer (at least for RH/CentOS) is not kicking off on manual FDI provision and w/in Foreman through 'New Host' methodology we probably have to get that sorted out first before Discovery Rules will play nice.

You may want to check that under 'Administer -> Settings -> Discovered tab that 'discovery_auto' is set to true as well, gonna need that although I have it set and it STILL does not work so far. Pretty sure it's needed though when we reach our intended utopia.
 
Last edited:
  • Like
Reactions: wsuff and Chuntzu

iLya

Member
Jun 22, 2016
48
5
8
I know it might sound silly but has anyone tried System Center Virtual Machine Manager?
I am starting to learn about it but it does do bare metal provisioning as well as it gives you the ability to manage SDN(Software Defined Networking) and 2016 is coming with SDS (Software Defined Storage).
It gives you the ability to create profiles for physical and virtual deployments and even application profiles.
You can build application templates that automate provisioning of infrastructure and automatically deploy software on top of it.
I think you can then integrate it with System Center Orchestrator to perform even more complicated tasks and you can layer on top DSC (Desired State Configuration) which works very similar to Puppet for Windows and Linux to perform many other configuration/deployment activities for provisioned systems.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Had some time to test foreman a bit more, made this vid showing VMware integrated provisioning. You need to do the following to enable it w/in foreman interface:

Code:
apt-get install foreman-vmware
foreman-available-providers.png

Super slick as it enables 'hooks' to vSphere, no more running FDI iso image (attaching/detaching) or messing w/ 'Discovered Hosts' -> 'Provision' -> All those other screens/settings.

PXE booting of CentOS/Ubuntu/Fedora working well, may try Windows next but that make my stomach churn honestly and may pass it off to my buddy who is a WDS guru.

https://owncloud3.undercovernerds.com/index.php/s/bw6P18nkwQwbl0k


Bare metal provisioning of a sc113 host using IPMI working great as well using 'Hosts' -> 'New Host' methodology. All you need to do is plop in MAC addy under 'New Host' -> 'Interfaces' tab and pwr cycle via IPMI...takes off like a rocket!

Next steps...deeper dive of puppet integration :-D
 
Last edited:
  • Like
Reactions: Patrick