Dedicated Open Compute miner build- Part 2.The RandomX Edition.

Discussion in 'Cryptocurrency Mining and Markets' started by Klee, Aug 17, 2019.

  1. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    A couple of years ago Monero and other cryptonight based coins were really not worthwhile to mine on cpu's anylonger because of the creation of GPU miner programs then the creation of ASIC's, With the RandomX and RandomX based algo's cpu's will once again become worthwhile to use cpu's to mine.

    Some of you know about and have played around some mining one of the few current crypto coins that are out that use RandomX or a version of RandomX, Loki seems to be the most stable, so this is really not geared to you but to those that are unfamiliar with RandomX and have some older Open Compute servers or other servers that are underutilized and collecting dust.

    RandomX is the next algo that Monero will use with the next network update due in October. It is VERY ASIC and GPU unfreindly but works extremely well on cpu's especially the new Ryzen 3XXX and EPYC Rome cpu's.

    But as long as the cpu has the following specs it will mine Monero/RandomX MUCH faster than the old cryptonight based algos:

    64-bit architecture.

    16 KiB of L1 cache, 256 KiB of L2 cache and 2 MiB of L3 cache per mining thread.

    IEEE 754 compliant floating point unit.

    Hardware AES support (AES-NI extension for x86, Cryptography extensions for ARMv8).

    Support for large memory pages.

    Plus you will need at least 2.5 GiB of free RAM per NUMA node, multiple channels and the faster* the ram the better.

    *DDR3 memory is limited to about 1500-2000 H/s per channel (depending on frequency and timings)
    *DDR4 memory is limited to about 4000-6000 H/s per channel (depending on frequency and timings)

    Github of RandomX's creator for more info:

    tevador/RandomX

    This guide is about using and cheaply upgrading older hardware that alot of people still have for home use since they are new enough to be really usefull to use for many purposes and old enough to be cheap to run or upgrade.

    E5-26xx Xeons are good enough to mine, not as good as the new AMD cpu's but alot of people have these readily available.

    (Also old AMD Opteron four cpu servers seem to be very usable mining RandomX tho very power hungry.)

    I do have a couple of Ryzen 3XXX cpu pc's but I also have these old Open Compute servers that I bought about three years ago.

    Each was built mainly to a price/low tdp as possible as a priority.

    Specs on them:

    Wiwynn Windmill socket 2011 Open Compute Server with two dual socket 2011 nodes with 16 ram slots per node that can run DDR3 ECC 800, 1066, 1333 and 1600 speeds.

    Will only run E5-26xx V1 cpu's so if you want to run V2 cpu's you need to use the Quanta version, yes I had trials and tribulations trying to mod the bios to accept Ivy Bridge microcode but gave up.

    (But I was able to mod the bios with much newer V1 cpu microcode and newer code for the two network chips.)

    Currently upgraded 2x E5-2650L v1 cpu's, 16x DDR3 PC8500R ram and WD 250 GB hard drives on each node.

    *NOTE: Go ahead and install new CR2032 motherboard batterys, the old ones that still show around 3.0v instead of 3.3v caused the servers to have issues.

    They all were cheap!! Cpu's $18.00 each, ram was $.32 per GB, and the hard drives were $4.00 each.

    The cpu choice was because I wanted the most cores and lowest TDP as cheap as possible, the ram is way more than needed at 64gb per node but it was so cheap and if needed I can remove some if the power use is unacceptable.

    The hard drives I already had as well as the three OC servers.

    I admit this setup will not be nearly as efficient as the newer cpu's but it's about using what I have and getting the best bang for the buck out of them.

    To update them I spent $56.83 per node and each node should hash about 4000 H/s with dual 2650L cpu's. So six nodes cost me $340 to update and should get about 24,000 H/s mining randomX total on the three OC servers with the 2650L's.

    To put that in perspective my AMD Ryzen 3700X was $329.00, X570 MSI motherboard was $169.00 and the rest I already had and it mines RandomX at ~7700 H/s.

    Yes I could have went for E5-2670's for more performance but they just put out too much heat and use too much power and the 2650L's should be a good price/performance/power/lower heat compromise. I also could have went with faster ram but I tested PC3-14900 ram and it was not a whole lot faster and 14900 is quite a bit more in price.


    So that said lets get started.

    I chose Ubuntu 19.04 simply because it runs so nice and is so easy to use on these servers and I had an issue with Ubuntu 18.04.3 with a 2 minute delay on booting waiting on Ubuntu deciding that it cant configure the second network interface that is unused.

    My install procedure with Ubuntu 19.04 server:

    1. Temporarily install an old PCI-e video card , keyboard and usb dvd drive or flash drive.

    2. Boot up and go into bios and select the bootable install media drive.

    3. Save and then reset and then start the install with the Ubuntu 19.04 installer.

    4. I picked ALL the default install options including the use the whole disk without LVM and also the "install ssd server" option and nothing else, you can add additional roles if you would like but for now it is not needed by me.

    5. Finish the install and reboot, and made sure it actually rebooted and is at the login prompt on the monitor. You may have to hit the reset or more likely the power off due to the reboot hang issue that all the newer versions of Ubuntu have with Open Compute servers that will get fixed in the steps below.

    6. Log in via ssh from another pc with these instructions open.

    7.To fix reboot hang in newer versions of Ubuntu type this command :

    echo "options mei-me disable_msi=1" | sudo tee /etc/modprobe.d/mei-me.conf

    Next enter sudo password when prompted.

    8. To mine using hugepages with Xmrig 3.0 type in this command:

    sudo nano /etc/sysctl.conf

    Add this to the end:

    vm.nr_hugepages = 2560

    (Enabling hugepages is a must with RandomX as you will take a huge hit if it is not enabled. Also Xmrig mining RandomX needs alot more than the old algo, some have been able to set it to "vm.nr_hugepages =1280" and run fine with that.)

    9.Next edit:

    sudo nano /etc/security/limits.conf

    Add these two lines at the end:

    * soft memlock unlimited
    * hard memlock unlimited

    NOTE: Probably not a good idea to use "memlock unlimited" on a normal install but this server is mining only and will not have any other user except for me and I will only run one miner program and so far it seems to work fine.

    10. Reboot

    11. If it fails to reboot, power off then power back on.

    I've had to enter this command again on about half the Ubuntu installs as it does not always "stick" the first time for some reason.

    echo "options mei-me disable_msi=1" | sudo tee /etc/modprobe.d/mei-me.conf

    12. Reboot again if you did the above.

    13. Log in via ssh from your desktop.

    14. Update Ubuntu then reboot again.

    15. Install and configure lm-sensors to be able to monitor the cpu temps.

    16. Download and compile what ever miner you are using, currently only Xmrig 3.0 for RandomX as of now.

    Latest version of Xmrig:
    xmrig/xmrig

    17. Configure the miner config files and then start mining. ( I'm currently mining Loki for now just to test mining a RandomX algo.)

    This like the original Dedicated Crypto Mining Open Compute server build from three years ago this is a work in progress.



    Also I am in the middle of installing Ubuntu and configuring the servers as I was typing this so I only tested by using the randomx benchmark not actually mining on one node on one server with the 2650L cpu's and several ram combinations and also a couple of E5-2603 cpu's.

    Using RandomX benchmark the binary version on a Open Compute node running Ubuntu 18.04.3:
    Dual E5-2650L cpu's ~4000 H/s w/PC3-8500R ram in quad channel and ~4005 PC3-14900 in dual channel
    Dua1 E5-2603 cpu's ~1900 H/s w/ PC3-8500 quad channel

    Mining Loki with xmrig 3.0:
    Ryzen 3700x ~7700 H/s Ubuntu 18.04.3 ~7400 H/s Windows 10
    Dual E5-2667 V3 ES cpu's Ubuntu 18.04.3 ~ 6600 H/s
     
    #1
    Last edited: Aug 21, 2019
    ari2asem likes this.
  2. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Just a note, there is a RandomX command line benchmark you can run to see what your cpu/cpu's will run on the Randomx github link above.

    But there is a handy list of benchmarks already to give you an idea if your hardware is worthwhile to use.

    RANDOMX BENCHMARKS FOR MONERO MINING!

    I use xmrig and actually mine Loki to have a real world test.
     
    #2
  3. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Some Ram speed thoughts, With DDR3 the ram speed is not super critical since it does not seem to make much of a difference but with DDR4 it can make a large difference.

    Of course faster ram is better in all cases but with DDR3 it really does not seem to be worthwhile trying to get fast ram if you already have the slower speed ram.

    I could have gotten by with the ram and cpu's I already had and spend not another dime and still run them to mine using the 2603 and 2620 cpu's but I was able to get such a good deal on 400 GB's of ram and 12 low TDP 8 core cpu's. And it lets me do other stuff in the future with them.
     
    #3
    Last edited: Aug 23, 2019
  4. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    185
    Likes Received:
    59
    October is perfect for me :) New server builds should be complete by then, so I'll have 96 cores, probably 90 mostly idle, and it'll be cooling off enough to not need to run AC.
     
    #4
  5. jims2321

    jims2321 Active Member

    Joined:
    Jul 7, 2013
    Messages:
    178
    Likes Received:
    42
    Looking forward to actually heating my basement this fall/winter with my server farming. Not sure what it is, but RandomX seems to favor the AMD cpus (both consumer and server). I just ran the randomx benchmark tool on my r815 again.

    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1310.51 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1309.81 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1308.74 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1308.53 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1308.03 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1307.81 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1306.91 hashes per second
    Calculated result: a9fbc0a016b77c6b9993aa593f0bd9b1bd02ab609ceb1af5b21dd2728955a373
    Performance: 1302.99 hashes per second

    = 10463.33 @ 722W or 14.49h/w

    Running it on my 3900x with tuned memory at 4Ghz yields 13105.5 @ 280w or 46.8h/w.

    I also did some testing with u/sech1 (one of the developers of Randomx algorithm) to see if using 1G hugepages would improve the performance.

    RandomX, Ryzen 7 3700X and memory speed dependency : MoneroMining

    But right now, barring any major code changes I would expect the best bang for the buck will be 3600 with 8G of ram, bootable usb stick and cheap B450 board. Its probably in the 40 to 50h/w range, and will sip power at 150w or less if undervolted.
     
    #5
  6. jims2321

    jims2321 Active Member

    Joined:
    Jul 7, 2013
    Messages:
    178
    Likes Received:
    42
    Just rereading some reddit threads about real world RandomX testing. And as u/sech1 points out, the randomx/loki results are at least 7% lower than the monero randomx due to the number of programs/commands different between them.

    sech1
    5 points ยท 14 days ago
    Loki runs 7 programs, 320 instructions each. Monero runs 8 programs, 256 instructions each, so Loki runs 9.375% more generated code. This alone makes RandomXL slower.

    So based upon that you should expect (assuming Monero and loki are similar number of miners/difficult) 9.375% more hashs/s
     
    #6
  7. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    First test mining Loki on one node of my OC servers.

    Dual E5-2650L, 64 GB PC-8500R ram.

    Default setting in bios on everything.

    Whats funny about that is among the tons of settings for the ram there are three settings listed as "Power Saving" "Default" and "Max Performance"

    The "Default" is not acually the default, the default setting is "Power Saving". LOL

    Running Xmrig 3.1.0.

    It seems the best performance is letting Xmrig pick 16 threads, I was expecting 19 or 20 with the 40MB of total L3.

    I ran the temps in the room the servers are in at worst case, no A/C and its probably 85 F.

    So so temps are ok.

    Also the cheap ram feels room temp unlike the hot ass samsung PC-8500 that was in this node.

    This is mining Loki so I expect about %10 more mining Monero RX.

    Capture4.PNG Capture5.PNG
    Capture6.PNG
     
    #7
  8. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Next test :

    Turn hyperthreading off since its seems to mine slower if I add more than 16 threads.

    Temp went up a little bit, fans are a little louder.

    Sure wish it would use more than 16 cores as max performance, maybe after xmrig is tweaked more.

    Capture7.PNG Capture8.PNG Capture9.PNG
     
    #8
    Last edited: Aug 20, 2019
  9. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Next test:

    Turn hyperthreading back on and set memory for "max performance"

    Mines about 25H/s more, temps are up, fan speed took longer to ramp up.

    Capture10.PNG Capture11.PNG Capture12.PNG
     
    #9
  10. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    I am trying to get max hash rate first then i'll throw a kilowatt meter on it then start trying for the best efficiency.

    The temps are noticeably lower than with the 2660's and especially the 2665's.

    Will find out the power use in a few days.
     
    #10
  11. ari2asem

    ari2asem Member

    Joined:
    Dec 26, 2018
    Messages:
    156
    Likes Received:
    13
    how many cpu's do you have in totall in your windmill node? totall of 4 cpu in the whole case, divided over 2 nodes?
     
    #11
  12. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Two dual E5-26xx v1 cpu's per node, teo nodes per server so 4 cpu's per server.
     
    #12
    Last edited: Aug 22, 2019
  13. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Next test:
    ( EDIT: I forgot to set the other node to the bios settings, so I reran all the test and edited the results)

    Both nodes, Mem set to Max Performance and Hyperthreading is on.

    Both nodes powered on and idling, ~118.5 w to ~135w.

    Both nodes mining, 455.5 watts.

    Node05 is 4281.3 H/s.

    Node06 is 4282.2H/s

    Total of 8563.5H/s at 455.5 watts, so 18.80 hashes per watt.

    Temps mining are 60c and 65c on node05.
    Temps mining on node06 are 63c and 55c.

    Edit:
    From the original oc mining build: Two nodes after 10 minutes mining Monero with 40 threads :
    401w @ 1736 H/s".
    So 4.329 hashes per watt mining the original cryptonight algo.
     
    #13
    Last edited: Aug 22, 2019
  14. modder man

    modder man Active Member

    Joined:
    Jan 19, 2015
    Messages:
    628
    Likes Received:
    64
    Either I cant math or you are doing this just for the fun? Running some quick calculators I ended up with profit margin less than $10 per month.
     
    #14
  15. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,676
    Likes Received:
    493
    Is this a new coin?
     
    #15
  16. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    What calculators?

    There are no calculators for Monero running RandomX , it wont be updated with the new algo until October so anything until then is a guess.

    Comes down to this, tens of thousands if not hundreds thousands of Gpu's become massively unprofitable in October.

    What will be the best option then?
     
    #16
  17. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353

    No, just getting ready for when Monero changes to an new algo that is cpu friendly but gpu and asic unfriendly.

    Loki is as of right now the best RandomX algo coin to use as a test.
     
    #17
  18. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    Been having an issue with corrupted bios from all the shutting off, unplugging, replugging, powering on I have been doing.

    These OC servers have never really liked that, they like to be powered on and in use 24/7.

    So I decided to go back to the un-modded factory bios and go with defaults with everything in the bios except pick "Max Performance" in the memory setting page then call it done.

    Tired of reflashing and the power saving is not a whole lot when the servers are running full tilt anyway.
     
    #18
  19. Klee

    Klee Well-Known Member

    Joined:
    Jun 2, 2016
    Messages:
    1,187
    Likes Received:
    353
    I understand now why the best hash rate is with 16 threads instead of 20.

    The key is this: " 16 KiB of L1 cache, 256 KiB of L2 cache and 2 MiB of L3 cache per mining thread."

    Each E5-2650L V1 has:

    Level 1 cache size, 8 x 32 KB

    Level 2 cache size, 8 x 256 KB

    Level 3 cache size, 20 MB shared.

    So with level 1 cache you can mine 16 threads if you have enough L2 and L3, With level 3 you can mine 10 threads if you have enough L1 and L2.

    But there is only enough L2 cache to mine on 8 threads per cpu.

    So that explains it.
     
    #19
    Last edited: Aug 23, 2019
    T_Minus likes this.
  20. jims2321

    jims2321 Active Member

    Joined:
    Jul 7, 2013
    Messages:
    178
    Likes Received:
    42
    did some tweaking on my Dell R815. disable HT assist to free up 2MB of ram per core. And installed TurionPowerControl (that was a bitch). reran the randomX benchmark and now I am up to 11253 h/s on the box. but the power draw jumps to 815w

    11253 h/s / 815w = 13.807 h/w

    At that h/w level, not sure it will be worth it mining on my Dell except during cold weather so I can head my basement!
     
    #20
Similar Threads: Dedicated Open
Forum Title Date
Cryptocurrency Mining and Markets Dedicated Crypto Mining Open Compute server build. May 23, 2017
Cryptocurrency Mining and Markets My ASUS 4nodes dedicated Monero miner May 28, 2017
Cryptocurrency Mining and Markets Open source cuda / nvidia zcash miner? Oct 23, 2017

Share This Page