Monero Mining Performance

[kfriis]

... forgot to say - - I realize I will not "get rich" or anything like that with this type of setup. I am just trying to about break even and finance my hobby. But I might as well get what is the best bang for the buck - that is what I am looking for input for.

Thanks!

 

[Marsh]

You are in a good place to be able to pay $0.1 KWh.
With your power rate , you should be able to turn a profit. Just watch your hardware cost.

Are you planing to use the hardware rig exclusive for mining?
or at some point in the future, turn your mining rig for other purpose.

My original plan is to have the hardware pay for itself partially, then deploy the hardware to serve other purpose.
I picked hardware platform that would support E5 v3 and v4 CPU.
See thread My ASUS 4nodes dedicated Monero miner

I am also buying cheap X99 or dual C612 LGA2011-v3 boards, then wait for cheap E5 v3 CPU to show up on Ebay.
I do BIOS mod of X99 board to reduce power cost ( I pay $0.28 to $0.4 KWh ), and increase hashrate.
See E5 v3 CPU systemboard BIOS mod to increase hashrate

 

[kfriis]

@Marsh

Thanks you for input and the links. I would say this initial rig will be used 90% for mining - the rest doing light work as a VM host. I think my goal is similar to yours - mine to pay for the HW cost so I can have a "free" server after some time.

My challenge is to stay close to USD1000 for an initial setup. At the same time, I really like your plan to by something that can accept V3/V4 CPUs down the line.

I like the ASUS 4 node server (2 node 1U or 4 node 2U seems to be the way to go), but I am not sure where to buy the ASUS at a reasonably low cost (seems like you ran into a really good deal!).

I can get a 2 node setup with 4x E5-2660 v2 CPUs all-in for about $900 so about $450/node. Is there anything out there better than that?

 

[funkywizard]

Hi -

New here. This is a good thread.

I am interested in the Open Compute servers for XMR mining. Trying to decide what CPUs are the best bang for the buck.

In post #1 I see:
2x Intel Xeon E5-2670 V1 = 829H/s
2x Intel Xeon E5-2650 V1 = 750H/s

However, I do not see ANY E5-26xx v2 CPUs. Several v1, v3, v4, but no v2. I am wondering why. Is that a really bad CPU for mining?

For example, I am looking at the E5-2660 v2 - would that not be quite a bit better than the v1? It has 25MB L3 cache so should be able to run 12 threads. Anyone care to guess the H/s given the v1 numbers above?

Appreciate any advice on the best Open Compute server CPU to start out with (probably in one of the Quanta v2 systems).

Thanks!

The v2's are like the v1's only better. More performance at a lower power draw.

 

[funkywizard]

Thank you for the very quick replies guys!

Any other CPUs I should be looking at instead of the E5-2660 v2 for a low cost setup? I would like to start at a relative low cost so that rules out EPYC and later gen Intels I think and makes me look at the Open Computer servers which as far as I know only supports up to E5-26xx V2 (max. 115W TDP?).

2650v2 and 2660v2 are reasonable options.

2680v2 costs more $ per hash, similar watts / hash, but you do get more hash per server.

 

[funkywizard]

@Marsh
I can get a 2 node setup with 4x E5-2660 v2 CPUs all-in for about $900 so about $450/node. Is there anything out there better than that?

Probably about the best you can do with CPU mining

 

[Marsh]

The key is to start slow, do not spend $1000 at once. Find a old box, install Linux and or Docker, learn and tweak the miner software.

You are in a good position to make profit or at least a decent ROI. Take your time.

E5 v1 and v2 platform is a good since your power cost is low.

If you approach it as a fun hobby, I like E5 v3 and v4 more.
Here is why.
Take advantage of Ebay $15 off $75 coupon, have your family sign up for multiple accounts of Ebay/Payal

X99 single CPU board could be had for $100-$110 after Ebay Coupon.
I purchased multiple X99 board below $100 without coupon.

E5 v3 CPU ( I like v3 more than v4 today ) because allcore turbo feature.
I had purchased 2 x E5-2683 v3 production CPU for $125 each ( my most expensive CPU )
Most of my E5 v3 production and QS CPU are below $110 ( some are $80 each ), I brought over 30 x E5 v3 CPU.
If you could use Ebay Coupon , and find a cheap E5 v3 CPU .

For mining, you only need one 4gb ddr4 ram stick, I am not aware any smaller than 4 gb DDR4 ram stick.
I paid around $15 to $20 per stick before Ebay coupon.

Power supply, the entire single E5 v3 CPU draws less than 100watt ( more like 70-80watt ).
Ebay have Seasonic 350w PS for $12 , Newegg / Amazon / BHphoto have EVGA 450w PS for $10 AR
I just purchased 6 EVGA 450w PS.

With some X99 board, you do not need video card to boot Linux.

When you done mining , these hardware make a good future homelab or desktop by adding more memory, and SSD.

 

[Klee]

Cheapest most bang for the buck:

Need 208v or more, I lucked out and have a 20amp 250 v a/c outlet thats unused that I am using.

Quanta Open Compute servers used on ebay: $174.00, x4 E5-2650 v1 cpu's ~$210.00, used ram 1 gig per cpu is enough ~ $15.00, used sata hard hard drive $5.00, used pci-e video card $7.50, Ubuntu 16.04 64 bit $00.00.

That puts you at $411.50 per server.

One Open Comute server with two nodes with 4 E5-2650 v1's cpu's will crank out ~1500- ~1600 h/s.

Thats assuming you have a monitor and keyboard to install.

The open compute servers have sketchy V2 support plus the V1's are way cheaper.

The 2660 v1's used to be the sweet spot for price/performance but the price went up.

The 2665 v1's are not worth the price over the 2660's as they hash maybe another 50H/s more and run 10c hotter and use more power that what the hash rate gain gives you.

 

[jims2321]

Klee,

What is your power usage for the open servers?

 

[Klee]

Klee,

What is your power usage for the open servers?

From my open compute thread.

"No cores turned off.
One server plugged in and both nodes powered off: 15w
One node powered on, Ubuntu 17.04 server booted up and idling : 56.5w
One node after 10 minutes mining Monero with 20 threads : 208w @ 868 H/s
Two nodes powered on, Ubuntu 17.04 server booted up and idling : 117w
Two nodes after 10 minutes mining Monero with 40 threads : 401w @ 1736 H/s

So two Open Compute servers running eight E5-2660 V1 cpu's are mining Monero at 3472 H/s using 802 watts."

 

[Joel]

Dayum. My 2x2667v2 idles at 120w. But that's with 6x8tb Reds inside as well.

Monero is 970H/s @ 310w all in. I'm still new to this game and running the minergate image. Trying to figure out the xmrig thing...

 

[mcpain]

Hi guys. How do you achieve such high hashrates? I have a few rigs mining on different systems (different hardware, mining software and OS) and always get bad hashrate. Latest example is 2x Xeon E5670 v3 machine using nicehash cpuminer-multi and it barely gets over 600 Hashs/s. This is not right, right?

Thanks for answer!

 

[kfriis]

Without optimizing anything , I am seeing around 1040 H/s for 2 x E5-2660 V2 (xmr-stak on Windows).

 

[Marsh]

@mcpain

Could you tells us more about your setup?
Which OS? Which miner software? Which pool are you using?
You should have more than 1K H/s or close to 1K

 

[mcpain]

I'm using cpuminer-multi by nicehash (GitHub - nicehash/cpuminer-multi: crypto cpuminer (linux + windows)), OS is redhat 6. If I run "--benchmark" it does the same speed. I expected at least 1KH/s too.

 

[Marsh]

Most of us use xmr-stak-cpu or xmrig .

Few ideas comes to mind.

Is it possible to do a quick test run with xmr-stak-cpu or xmrig for comparison ?

Is it possible Nicehash doing dual mining ?

Is the host dedicated to mining?

have you enable large pages and set ulimits?
vm.nr_hugepages = 60
* soft memlock 626688
* hard memlock 626688

Most likely is NUMA issue , and number of mining threads

 

[mcpain]

Host does very very little besides mining right now and I also tested while it was doing nothing. It's not doing dual mining. I've set vm.nr_hugepages to 128 on one machine and left 0 on the other to see if there would be a difference - there wasn't.
I tought issue is probably NUMA related at first too, but all threads produce about the same H/s, I wouldn't expect that if that would be the case.

I might try xmr-stak-cpu if there won't be too much trouble compiling it on such old OS (I want to keep the machines relatively clean).

Thanks for suggestions.

 

[mcpain]

Update: I compiled xmr-stak-cpu and now I get only a bit over 500 H/s - so performance is even worse

Edit: what would be good affinity settings? I tried some combinations without improvement.

 

[Marsh]

Please post your lscpu print out, and partial xmr-stak-cpu config related to cpu section.
Here is my lscpu and xmr-stak-cpu config

Dual E5-2683 v3 cpu , 1162.5 H/s

Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
Stepping:              2
CPU MHz:               2500.156
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              3991.99
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              35840K
NUMA node0 CPU(s):     0-8,18-26
NUMA node1 CPU(s):     9-17,27-35

"cpu_thread_num" : 34,

"cpu_threads_conf" : [
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 0 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 1 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 2 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 3 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 4 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 5 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 6 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 7 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 8 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 9 },
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 10},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 11},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 12},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 13},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 14},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 15},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 16},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 17},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 18},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 19},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 20},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 21},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 22},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 23},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 24},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 25},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 27},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 28},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 29},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 30},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 31},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 32},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 33},
        { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 34},
],

 

[mcpain]

lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
Stepping:              2
CPU MHz:               2300.042
BogoMIPS:              4599.35
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0-11,24-35
NUMA node1 CPU(s):     12-23,36-47

config

"cpu_threads_conf" :
[
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 0 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 1 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 2 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 3 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 4 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 5 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 6 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 7 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 8 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 9 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 10 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 11 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 12 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 13 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 14 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 15 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 16 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 17 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 18 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 19 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 20 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 21 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 22 },
   { "low_power_mode" : false, "no_prefetch" : true, "affine_to_cpu" : 23 },
],

Hashrate

Totals:   545.9 (na) (na) H/s
Highest:  546.2 H/s

I use 24 threads because I tried using only physical cores