Lets build a PXE Server for Our Mining Rigs

[marcoi]

I figure I start a thread on getting a PXE Server running and serving OS images to host of miners.
I'm just starting the research for this so any help appreciated. As we figure this out we can post guide here as well.

I found this guide so far that uses clonezilla
Clonezilla PXE Boot Server. – Michael Gallant's corner of the world.

I have yet to try it out.

My goals for the project:
1. setup the PXE server on the network as a VM (for me running on ESXI)
2. Have it assign a different range of dhcp IP (different from home network) (dont want to use my home block up)
3. Create a miner image that has various mining tools already built on it with configurations.
4. Boot whatever host from PXE image and have all mining ready to go.

I'm thinking of the following images
1. ubuntu server with dockers and miner builds
2. Ubuntu server with dockers, nvidia drivers and nvidia docker
3. IS windows possible?

 

[K D]

I'm interested. Not for mining though. Just want to setup a pxe boot environment. I've gone from 2 windows boxes several physical boxes with multiple flavors of linux/unix/windows being installed and re-installed. I had a WDS server for windows installs but That got hosed a while ago.

 

[dwright1542]

I figure I start a thread on getting a PXE Server running and serving OS images to host of miners.
I'm just starting the research for this so any help appreciated. As we figure this out we can post guide here as well.

I found this guide so far that uses clonezilla
Clonezilla PXE Boot Server. – Michael Gallant's corner of the world.

I have yet to try it out.

My goals for the project:
1. setup the PXE server on the network as a VM (for me running on ESXI)
2. Have it assign a different range of dhcp IP (different from home network) (dont want to use my home block up)
3. Create a miner image that has various mining tools already built on it with configurations.
4. Boot whatever host from PXE image and have all mining ready to go.

I'm thinking of the following images
1. ubuntu server with dockers and miner builds
2. Ubuntu server with dockers, nvidia drivers and nvidia docker
3. IS windows possible?

The big issue I'd have is the single point of failure. USB sticks are cheap.

Windows is definitely doable.

-D

 

[audio catalyst]

The big issue I'd have is the single point of failure. USB sticks are cheap.

Windows is definitely doable.

-D

agreed windows installation services, coupled with chocolatey should be very doable, not real sure any mining stuff is av. through chocolatey though

send from a mobile device, so typo's are to be expected

 

[marcoi]

First thing I did was clone one of my ubuntu server templates on ESXI. I also decided to try this on a bare metal server that is with my mining boxes. I wanted to compare speed of PXE load of local vs one that needs to connect to mining rigs via WIFI bridge. I assigned static ip address to both servers.

command so far:
sudo wget -q http://drbl.org/GPG-KEY-DRBL -O- | sudo apt-key add -
sudo vi /etc/apt/sources.list
add the following line (ubuntu):
deb Index of /drbl-core drbl stable
sudo apt-get update
sudo mkdir /clonezilla_mining_images
sudo apt-get install drbl
sudo /usr/sbin/drblpush -i

At this point I'm having issues with ubuntu 17.10, seems like drbl isnt updated for the newest version.

ill update once i get something working

 

[Joel]

So I have a script for install already mostly done. The startup script is still a TODO, but this should help.

GitHub - joelmusicman/ubuntu-miner

The main thing to look at is install.sh.

Sources are this guide as well as a reddit thread:

Mining ZenCash with Ubuntu linux and GTX 1070 #Tutorial — Steemit

Finally got headless Ubuntu with NVIDIA going with a mining watchdog. • r/EtherMining

 

[marcoi]

@ Joel, thanks for the script. when i get to the point where im building the images, i will def try it out.

So an update my progress. I was able to get DRBL live cd running using a vm on my esxi host. I download the live cd iso and uploaded it to a datastore then mounted it to cdrom of the test vm.
I was able to get it running and got it to run in PXE mode for clonezilla to clone a host. I wasnt able to figure out if or how I can have that image become boot able to other hosts. It might not be an out of the box option for DRBL.

So i continue my search.

 

[Patrick]

When you guys get this done, happy to feature on the main site.

 

[Joel]

My plan with the scripts are to have a second script that reads all nvidia GPUs in the system and asks for desired power, clock, and fan settings for each.

Just finished bios flashing all my 6027s (until I start buying more!). Each node will house a 1050 or 1050 Ti. Note with the ASUS models that look like this:



They look like they'll fit but I had to remove the plastic shroud on the GPU so it would clear the rear CPU heatsink. It was dorky looking IMO so no big loss, but may hurt eventual resale.

EVGA models fit just fine, no finagling required.

 

[Algeroth]

wow...
i'm using cobbler to do exaclty that, so every spare cpu cycles for vm's and not used currently bare metal
cobbler serves via tftp and dhcp kernel image and initrd, and than from that initrd image of bootable iso with miner software inside is deployed to vm/bare,
so in fact everything is only in memory when minig.
dhcp configuration forbids answering dhcpoffers for not known machines.

rock solid and reliable.

so in short... use cobler... or ironic, it's made for such things.

regads
A

 

[yuzmemak]

Hi @marcoi ,

I am trying to understand your project with my lack of knowledge. Do you mind to share your hardware list in this project for better understanding and reading.

Best Regards

 

[marcoi]

@Algeroth Thanks for the suggestion, i will check it out as an option.

@yuzmemak - My plan for the PXE server is to have it running as a Virtual Machine, so the backend hardward wouldn't matter as much. For the servers the PXE will provide boot images for, they range but most of my mining rigs are dual e5-2600 v1/v2 based servers some with video cards such as 1050 ti 4gb. Other boxes I have are either server boards or desktops running graphic cards such as 1080 and vega 64.

Goal is to have a PXE server with multi images provide various booting options to my miners. IE If i wanted to change mining of my server with 2 1050 ti cards, I would reboot the server and instead of running ubuntu OS with dockers, i might want to have it run windows OS with nicehash,

 

[marcoi]

made some progress.
I got ThinStation by Donald A. Cupp Jr. running in a vm.
I can PXE boot off of it and run the thinstation build and even ubuntu install disk.
What I dont know how to do is build a new OS image with mining tools to have passed via PXE to load disk-less on mining hardware.

Any tips?

In the meantime Ill check out cobbler.

 

[Joel]

I had this same idea, but since I "only" have 6 nodes, I gave up on it. I'm still interested in Ansible or something where I can make updates once and have the changes propagate to all nodes, but I'm ready to get these things working ASAP!

Aside: my bedroom sounds like a datacenter for now until I get colo set up. I switch them off when I go to bed.

 

[Marsh]

I been running diskless Linux , Windows servers and workstations over 10 years ,
therefore I have a set of scripts to build systems, deploy systems, diskless boot infrastructure already.
The diskless miner machines need to fit into my existing infrastructure , because I do not want to re-invent the wheel again.

All my miner are dedicated mining machine, no extra hardware or software wanted.
Example, most of the miner only have 1GB ram , except DDR4 machines have 4GB ( smallest ram that is available to me).
So it is impractical to run the OS within RAM, I rely on iSCSI disk for both Microsoft and Linux world.

My work flow to install Ubuntu on a diskless machine.

Record diskless machine network MAC into PXE dhcp server ,
assign hostname to MAC address in /etc/dhcp/dhcpd.conf

PXE boot, hit enter to allow Ubuntu installation ( for failsafe purpose, do not want to automatically install Linux , could eliminate "hitting return" step )
Then select iSCSI disk ( for safety reason , do not want to overwrite other iscsi disk , could program it into preseed file )
At the end of preseed file, call my own bootstrap.sh to config system , then reboot , call my minersetupsystem.sh to setup OS to mine , copy miners software tree and config files, alter /etc/rc.local for auto-startup mining.

All done.

 

[Joel]

To revive this a bit, I figured out how to update the IPMI firmware and set the fans to Optimized mode. The CPUs run hotter now (about 63C vs 55) but it's much more tolerable to be near noise wise. Ambient temp in the room is currently 80F.

If there's interest I can have the install script also download the newest IPMI firmware, install it, and set fans to optimized mode.

I also figured out some problems I was having with my script not setting fan speeds. The following lines need to be added to /etc/X11/xorg.conf under the "Device" heading:

BusID "PCI:4:0:0" (use "lspci | grep NVIDIA" to make sure this matches your config)
Option "Coolbits" "28"

 

[marcoi]

i been busy with work upgrades, so havent had much time to play with this. I did get by setting up my 4 node server by using the clonezilla boot and 32gb usb keys. i just copied the one that i had setup to the other three nodes. side note I need to get another 20A line put in before i run the 4 node server and cpu mining been down a bit so i havent been pushing this project.

 

[hifijames]

i been busy with work upgrades, so havent had much time to play with this. I did get by setting up my 4 node server by using the clonezilla boot and 32gb usb keys. i just copied the one that i had setup to the other three nodes. side note I need to get another 20A line put in before i run the 4 node server and cpu mining been down a bit so i havent been pushing this project.

How did you get that work? I tried the same thing, installing Ubuntu on a USB stick, and Clonezilla another one, but the cloned USB won't boot, complaining about missing root fs.

 

[Joel]

How did you get that work? I tried the same thing, installing Ubuntu on a USB stick, and Clonezilla another one, but the cloned USB won't boot, complaining about missing root fs.

I've looked into that; basically after cloning you need to chroot into the new usb stick and update the grub boot loader.

 

[marcoi]

I used the clonzilla live CD iso. I setup a VM with disk space to store the backups. I have the VM launch off of the live CD iso. Once booted you run drbl which configures network stuff. Then I ran clonzilla server and selected VM disk as storage and backup option to store whole disk. Boot server one with good install on pxe it loads clonzilla and copies the disk.

Next re run clone zilla and tell it you want to restore, do beginner option and select you cloned disk image. I think I select broadcast pxe mode. Don't recall exactly which one worked.

Boot server 2, load pxe clonzilla and follow restore prompts.

I had to use the same USB drives size and make for it to work.

Once done reboot server 2 and it should load os.

This is going from memory so might be missing something.