Lets build a PXE Server for Our Mining Rigs

Discussion in 'Cryptocurrency Mining and Markets' started by marcoi, Jan 18, 2018.

  1. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    I figure I start a thread on getting a PXE Server running and serving OS images to host of miners.
    I'm just starting the research for this so any help appreciated. As we figure this out we can post guide here as well.

    I found this guide so far that uses clonezilla
    Clonezilla PXE Boot Server. – Michael Gallant's corner of the world.

    I have yet to try it out.

    My goals for the project:
    1. setup the PXE server on the network as a VM (for me running on ESXI)
    2. Have it assign a different range of dhcp IP (different from home network) (dont want to use my home block up)
    3. Create a miner image that has various mining tools already built on it with configurations.
    4. Boot whatever host from PXE image and have all mining ready to go.

    I'm thinking of the following images
    1. ubuntu server with dockers and miner builds
    2. Ubuntu server with dockers, nvidia drivers and nvidia docker
    3. IS windows possible?
     
    #1
    gigatexal, yuzmemak, nkw and 2 others like this.
  2. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,411
    Likes Received:
    300
    I'm interested. Not for mining though. Just want to setup a pxe boot environment. I've gone from 2 windows boxes several physical boxes with multiple flavors of linux/unix/windows being installed and re-installed. I had a WDS server for windows installs but That got hosed a while ago.
     
    #2
  3. dwright1542

    dwright1542 Active Member

    Joined:
    Dec 26, 2015
    Messages:
    356
    Likes Received:
    68
    The big issue I'd have is the single point of failure. USB sticks are cheap.

    Windows is definitely doable.

    -D
     
    #3
  4. audio catalyst

    Joined:
    Jan 4, 2014
    Messages:
    88
    Likes Received:
    12
    agreed windows installation services, coupled with chocolatey should be very doable, not real sure any mining stuff is av. through chocolatey though

    send from a mobile device, so typo's are to be expected
     
    #4
  5. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    First thing I did was clone one of my ubuntu server templates on ESXI. I also decided to try this on a bare metal server that is with my mining boxes. I wanted to compare speed of PXE load of local vs one that needs to connect to mining rigs via WIFI bridge. I assigned static ip address to both servers.

    command so far:
    sudo wget -q http://drbl.org/GPG-KEY-DRBL -O- | sudo apt-key add -
    sudo vi /etc/apt/sources.list
    add the following line (ubuntu):
    deb Index of /drbl-core drbl stable
    sudo apt-get update
    sudo mkdir /clonezilla_mining_images
    sudo apt-get install drbl
    sudo /usr/sbin/drblpush -i

    At this point I'm having issues with ubuntu 17.10, seems like drbl isnt updated for the newest version.

    ill update once i get something working
     
    #5
  6. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    807
    Likes Received:
    155
    #6
  7. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    @ Joel, thanks for the script. when i get to the point where im building the images, i will def try it out.

    So an update my progress. I was able to get DRBL live cd running using a vm on my esxi host. I download the live cd iso and uploaded it to a datastore then mounted it to cdrom of the test vm.
    I was able to get it running and got it to run in PXE mode for clonezilla to clone a host. I wasnt able to figure out if or how I can have that image become boot able to other hosts. It might not be an out of the box option for DRBL.

    So i continue my search.
     
    #7
  8. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,569
    Likes Received:
    4,507
    When you guys get this done, happy to feature on the main site.
     
    #8
    gigatexal and Joel like this.
  9. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    807
    Likes Received:
    155
    My plan with the scripts are to have a second script that reads all nvidia GPUs in the system and asks for desired power, clock, and fan settings for each.

    Just finished bios flashing all my 6027s (until I start buying more!). Each node will house a 1050 or 1050 Ti. Note with the ASUS models that look like this:

    14-126-169-V01.jpg

    They look like they'll fit but I had to remove the plastic shroud on the GPU so it would clear the rear CPU heatsink. It was dorky looking IMO so no big loss, but may hurt eventual resale.

    EVGA models fit just fine, no finagling required.
     
    #9
  10. Algeroth

    Algeroth New Member

    Joined:
    Sep 29, 2017
    Messages:
    15
    Likes Received:
    3
    wow...
    i'm using cobbler to do exaclty that, so every spare cpu cycles for vm's and not used currently bare metal
    cobbler serves via tftp and dhcp kernel image and initrd, and than from that initrd image of bootable iso with miner software inside is deployed to vm/bare,
    so in fact everything is only in memory when minig.
    dhcp configuration forbids answering dhcpoffers for not known machines.

    rock solid and reliable.

    so in short... use cobler... or ironic, it's made for such things.

    regads
    A
     
    #10
    Blinky 42 likes this.
  11. yuzmemak

    yuzmemak New Member

    Joined:
    Aug 31, 2017
    Messages:
    27
    Likes Received:
    1
    Hi @marcoi ,

    I am trying to understand your project with my lack of knowledge. Do you mind to share your hardware list in this project for better understanding and reading.

    Best Regards
     
    #11
  12. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    @Algeroth Thanks for the suggestion, i will check it out as an option.

    @yuzmemak - My plan for the PXE server is to have it running as a Virtual Machine, so the backend hardward wouldn't matter as much. For the servers the PXE will provide boot images for, they range but most of my mining rigs are dual e5-2600 v1/v2 based servers some with video cards such as 1050 ti 4gb. Other boxes I have are either server boards or desktops running graphic cards such as 1080 and vega 64.

    Goal is to have a PXE server with multi images provide various booting options to my miners. IE If i wanted to change mining of my server with 2 1050 ti cards, I would reboot the server and instead of running ubuntu OS with dockers, i might want to have it run windows OS with nicehash,
     
    #12
    MortenB likes this.
  13. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    made some progress.
    I got ThinStation by Donald A. Cupp Jr. running in a vm.
    I can PXE boot off of it and run the thinstation build and even ubuntu install disk.
    What I dont know how to do is build a new OS image with mining tools to have passed via PXE to load disk-less on mining hardware.

    Any tips?

    In the meantime Ill check out cobbler.
     
    #13
  14. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    807
    Likes Received:
    155
    I had this same idea, but since I "only" have 6 nodes, I gave up on it. I'm still interested in Ansible or something where I can make updates once and have the changes propagate to all nodes, but I'm ready to get these things working ASAP!

    Aside: my bedroom sounds like a datacenter for now until I get colo set up. I switch them off when I go to bed. :)
     
    #14
    Marsh likes this.
  15. Marsh

    Marsh Moderator

    Joined:
    May 12, 2013
    Messages:
    2,114
    Likes Received:
    964
    I been running diskless Linux , Windows servers and workstations over 10 years ,
    therefore I have a set of scripts to build systems, deploy systems, diskless boot infrastructure already.
    The diskless miner machines need to fit into my existing infrastructure , because I do not want to re-invent the wheel again.

    All my miner are dedicated mining machine, no extra hardware or software wanted.
    Example, most of the miner only have 1GB ram , except DDR4 machines have 4GB ( smallest ram that is available to me).
    So it is impractical to run the OS within RAM, I rely on iSCSI disk for both Microsoft and Linux world.

    My work flow to install Ubuntu on a diskless machine.

    Record diskless machine network MAC into PXE dhcp server ,
    assign hostname to MAC address in /etc/dhcp/dhcpd.conf

    PXE boot, hit enter to allow Ubuntu installation ( for failsafe purpose, do not want to automatically install Linux , could eliminate "hitting return" step )
    Then select iSCSI disk ( for safety reason , do not want to overwrite other iscsi disk , could program it into preseed file )
    At the end of preseed file, call my own bootstrap.sh to config system , then reboot , call my minersetupsystem.sh to setup OS to mine , copy miners software tree and config files, alter /etc/rc.local for auto-startup mining.

    All done.
     
    #15
    pgh5278, poutnik, K D and 2 others like this.
  16. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    807
    Likes Received:
    155
    To revive this a bit, I figured out how to update the IPMI firmware and set the fans to Optimized mode. The CPUs run hotter now (about 63C vs 55) but it's much more tolerable to be near noise wise. Ambient temp in the room is currently 80F.

    If there's interest I can have the install script also download the newest IPMI firmware, install it, and set fans to optimized mode.

    I also figured out some problems I was having with my script not setting fan speeds. The following lines need to be added to /etc/X11/xorg.conf under the "Device" heading:

    BusID "PCI:4:0:0" (use "lspci | grep NVIDIA" to make sure this matches your config)
    Option "Coolbits" "28"
     
    #16
    Last edited: Feb 8, 2018
    Patrick likes this.
  17. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    i been busy with work upgrades, so havent had much time to play with this. I did get by setting up my 4 node server by using the clonezilla boot and 32gb usb keys. i just copied the one that i had setup to the other three nodes. side note I need to get another 20A line put in before i run the 4 node server and cpu mining been down a bit so i havent been pushing this project.
     
    #17
  18. hifijames

    hifijames Member

    Joined:
    Dec 26, 2017
    Messages:
    32
    Likes Received:
    7
    How did you get that work? I tried the same thing, installing Ubuntu on a USB stick, and Clonezilla another one, but the cloned USB won't boot, complaining about missing root fs.
     
    #18
  19. Joel

    Joel Active Member

    Joined:
    Jan 30, 2015
    Messages:
    807
    Likes Received:
    155
    I've looked into that; basically after cloning you need to chroot into the new usb stick and update the grub boot loader.
     
    #19
  20. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    I used the clonzilla live CD iso. I setup a VM with disk space to store the backups. I have the VM launch off of the live CD iso. Once booted you run drbl which configures network stuff. Then I ran clonzilla server and selected VM disk as storage and backup option to store whole disk. Boot server one with good install on pxe it loads clonzilla and copies the disk.

    Next re run clone zilla and tell it you want to restore, do beginner option and select you cloned disk image. I think I select broadcast pxe mode. Don't recall exactly which one worked.

    Boot server 2, load pxe clonzilla and follow restore prompts.

    I had to use the same USB drives size and make for it to work.

    Once done reboot server 2 and it should load os.

    This is going from memory so might be missing something.
     
    #20
    hifijames likes this.
Similar Threads: Lets build
Forum Title Date
Cryptocurrency Mining and Markets Dedicated Open Compute miner build- Part 2.The RandomX Edition. Aug 17, 2019
Cryptocurrency Mining and Markets Building Coin/pool switching software for Cryptonight Coins - what coins/pools would you pick? Feb 21, 2018
Cryptocurrency Mining and Markets Building the next-generation STH Aeon Docker Image Jan 9, 2018
Cryptocurrency Mining and Markets Thoughts on build Oct 30, 2017
Cryptocurrency Mining and Markets Dedicated Crypto Mining Open Compute server build. May 23, 2017

Share This Page