Dedicated Crypto Mining Open Compute server build.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Marsh

Moderator
May 12, 2013
2,651
1,500
113
@arron

For E5-2670 v1, I also see 975H/s

For E5-2660 v1 = 866H/s

You may want to double check which CPU in the machine?
 

Titan

New Member
Jan 14, 2018
12
2
3
34
I see. Holding XMR is good. is there any tool that switch pools easily?
This is my current hash on E5630 x2
upload_2018-1-15_9-10-18.png
 

Klee

Well-Known Member
Jun 2, 2016
1,289
397
83
@Klee Why dont you change to mining other coin instead of XMR? because as I have checked, sumokoin has better profit than XMR coin atm.
And I decide to buy a Wiwynn system in CN because it's cheaper and easier to ship to me.
Sorry for bad English

I have not mined Monero in about 2 1/2 months.

Mined Aeon until about a month ago.

There are more profitable cryptonight coins out there.
 
  • Like
Reactions: Joel

Indecided

Active Member
Sep 5, 2015
163
83
28
Are you running your nodes headless? Three of my nodes run headless and get the 945 number, the fourth node doubles as my desktop (I VNC into it, so it has an X server running, etc) and it trends closer to 850-900 h/s. It's a weird node though, since it's also a proxmox node and the deskto/miner are running in one of the LXC containers, which probably doesn't help.

View attachment 7609
@arron

For E5-2670 v1, I also see 975H/s

For E5-2660 v1 = 866H/s

You may want to double check which CPU in the machine?

@polyfractal - Interesting. When I run with your config I do get approximately 930H/s, still a little short of 945H/s but not very close to 975H/s that Marsh is getting, but certainly better then before.

The difference with your config is that i had my CPU affinity on even numbers instead of sequential.

I.E. 0,2,4,6,8,10 vs yours as in 0,1,2,3,4,5 so on and so forth. Not sure what the logic is behind that, but I had set up even numbers after reading one of the sample config files way back several months ago. Something to do with picking the physical cores vs the hyperthreads or so. Also, not sure why your config chooses to use threads 24 and 25 specifically with a gap in between.

I noticed with your config it pushes CPU #1 all the way and just a little of CPU #2. Wonder if that's intended behavior....

Now I guess it's time to tweak the whole lot.... Thanks!
 

polyfractal

Member
Apr 6, 2016
33
27
18
Yeah, not sure either. I also tried fiddling with the config to do every-other and similar arrangements... could never beat the auto-configuration that does it (mostly) sequential. Really not sure why it likes this, or why it seems to prefer loading up one socket more than the other. Perhaps to minimize inter-socket communication?

It's been on my todo list to write a quick simulated annealing solver to see if the config can be optimized some more, but I haven't found time yet. I want to dig into the xmr-stak code a bit more first to see how it's using hwlock to determine these configs.

Glad it helped, goodluck!
 

Joel

Active Member
Jan 30, 2015
857
199
43
42
@polyfractal - Interesting. When I run with your config I do get approximately 930H/s, still a little short of 945H/s but not very close to 975H/s that Marsh is getting, but certainly better then before.

The difference with your config is that i had my CPU affinity on even numbers instead of sequential.

I.E. 0,2,4,6,8,10 vs yours as in 0,1,2,3,4,5 so on and so forth. Not sure what the logic is behind that, but I had set up even numbers after reading one of the sample config files way back several months ago. Something to do with picking the physical cores vs the hyperthreads or so. Also, not sure why your config chooses to use threads 24 and 25 specifically with a gap in between.

I noticed with your config it pushes CPU #1 all the way and just a little of CPU #2. Wonder if that's intended behavior....

Now I guess it's time to tweak the whole lot.... Thanks!
Note that Windows assigns cores differently than linux on HT machines so this may be what's causing the confusion.

Windows: Even numbered cores are physical cores (0,2,4...) and odd are HT (1,3,5...).

Linux: Physical cores are first, then HT cores. So for a 2x 2670 it would be: 0-7 (CPU0), 8-15 (CPU1), 16-23 (CPU0 HT cores), 24-31 (CPU1 HT cores)
 

Joel

Active Member
Jan 30, 2015
857
199
43
42
Yeah, not sure either. I also tried fiddling with the config to do every-other and similar arrangements... could never beat the auto-configuration that does it (mostly) sequential. Really not sure why it likes this, or why it seems to prefer loading up one socket more than the other. Perhaps to minimize inter-socket communication?

It's been on my todo list to write a quick simulated annealing solver to see if the config can be optimized some more, but I haven't found time yet. I want to dig into the xmr-stak code a bit more first to see how it's using hwlock to determine these configs.

Glad it helped, goodluck!
The other thing that's helpful is to run two separate instances of your miner, affined to the cores of each NUMA node.
 

polyfractal

Member
Apr 6, 2016
33
27
18
Ah, well that'd explain what's going on. It's using 8 physical cores + 2 HT for each socket.

The other thing that's helpful is to run two separate instances of your miner, affined to the cores of each NUMA node.
Oh, interesting, even with xmr-stak? I knew you had to do that with xmrig because it didn't handle NUMA well, but I thought xmr-stak did NUMA correctly with it's affine settings.? Should I instead be running two xmr-stak instances each assigned to a socket's worth of threads (in my case, the relevant 8 physical + 2 HT threads)
 

Indecided

Active Member
Sep 5, 2015
163
83
28
Note that Windows assigns cores differently than linux on HT machines so this may be what's causing the confusion.

Windows: Even numbered cores are physical cores (0,2,4...) and odd are HT (1,3,5...).

Linux: Physical cores are first, then HT cores. So for a 2x 2670 it would be: 0-7 (CPU0), 8-15 (CPU1), 16-23 (CPU0 HT cores), 24-31 (CPU1 HT cores)
Wow, I feel pretty stupid now. Thanks Joel for clearing that up!

I think I've been able to increase hashrate another 15% or so at least, especially on the larger multicore procs like the E5-2651v2. from 750 to 885. Pretty significant.
 

Titan

New Member
Jan 14, 2018
12
2
3
34
Wow, I feel pretty stupid now. Thanks Joel for clearing that up!

I think I've been able to increase hashrate another 15% or so at least, especially on the larger multicore procs like the E5-2651v2. from 750 to 885. Pretty significant.
How is temp on your E5-2651 v2?
 
Last edited:

yuzmemak

New Member
Aug 31, 2017
27
1
3
52
Based on the $199 Quanta thread seems that was a result of Video output, I ended up ordering two of these Wiwynn i'll give it a go once I have them setup as headless, maybe try using PCI-e risers I have laying around.

I ended up going the transformer route, but even then I fear I will be maxing out my dinky 20A 120v outlet that is already powering my Home Server rack that has a pfSense box, an R710, and a GPU mining rig.

That magical outlet you found, is it a simple 12-2 single phase? Or is that a fancier 12-3 setup that was used for a Washer / Furnace?
Hi

Which model did you order?

Sent from my SM-G900F using Tapatalk
 

yuzmemak

New Member
Aug 31, 2017
27
1
3
52
@Klee Why dont you change to mining other coin instead of XMR? because as I have checked, sumokoin has better profit than XMR coin atm.
And I decide to buy a Wiwynn system in CN because it's cheaper and easier to ship to me.
Sorry for bad English
Hi

Which model did you order?

Sent from my SM-G900F using Tapatalk
 

polyfractal

Member
Apr 6, 2016
33
27
18
A note for anyone thinking about mining Sumo: the only real exchange that I'm aware of for sumo is Cryptopia... which appears to be flaky at best. I deposited a few SUMO there as a test, and it still hasn't showed up after a few days. Seems to be a reoccurring issue with them, according to reddit posts.

So... eh. It may look more profitable on paper, but I'm leery of wasting much time on a coin which is only usable at one flaky and illiquid exchange.
 

Titan

New Member
Jan 14, 2018
12
2
3
34
A note for anyone thinking about mining Sumo: the only real exchange that I'm aware of for sumo is Cryptopia... which appears to be flaky at best. I deposited a few SUMO there as a test, and it still hasn't showed up after a few days. Seems to be a reoccurring issue with them, according to reddit posts.

So... eh. It may look more profitable on paper, but I'm leery of wasting much time on a coin which is only usable at one flaky and illiquid exchange.
Pools are gone too
 

Joel

Active Member
Jan 30, 2015
857
199
43
42
A note for anyone thinking about mining Sumo: the only real exchange that I'm aware of for sumo is Cryptopia... which appears to be flaky at best. I deposited a few SUMO there as a test, and it still hasn't showed up after a few days. Seems to be a reoccurring issue with them, according to reddit posts.

So... eh. It may look more profitable on paper, but I'm leery of wasting much time on a coin which is only usable at one flaky and illiquid exchange.
I just recently found out about LiveCoin after having similar issues with Cryptopia (for some reason I'm not getting 2FA codes any more).

One exchange with Sumo, and while a bit slow, it worked.
 

Klee

Well-Known Member
Jun 2, 2016
1,289
397
83
I have not been running my open compute servers for about a week, so today I replaced the four E5-2660 cpu's in the first server with 4 E5-2603 four core cpu's to see what they do.

OS is Ubuntu 17.10 server upgraded from 17.04 server.

Powered on both servers and just idling: 104 watts.

Fired up xmrig 2.5.0 on node 1: 130 watts with all 8 cores running.

Fired up xmrig 2.5.0 on node 2: Both nodes mining 8 threads each is 556 H/s and using 163 watts.

First cpu is 47c and second (the rear one) is 48c. The air coming out of the rear fans seems cool on my hand vs warm with the 2660's and hot with the 2665's.

@node1:~/monero-xmrig-2.5.0$ ./xmrig
* VERSIONS: XMRig/2.5.0 libuv/1.19.2 gcc/5.4.0
* HUGE PAGES: available, enabled
* CPU: Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz (2) x64 AES-NI
* CPU L2/L3: 4.0 MB/20.0 MB
* THREADS: 8, cryptonight, av=1, donate=1%
* POOL #1: xmr-usa.dwarfpool.com:8005
* COMMANDS: hashrate, pause, resume
[2018-03-24 15:53:18] use pool xmr-usa.dwarfpool.com:8005 144.217.117.111
[2018-03-24 15:53:18] new job from xmr-usa.dwarfpool.com:8005 diff 20000
[2018-03-24 15:53:36] speed 2.5s/60s/15m 278.2 n/a n/a H/s max: 278.3 H/s
[2018-03-24 15:53:50] accepted (1/0) diff 20000 (112 ms)
[2018-03-24 15:53:51] speed 2.5s/60s/15m 278.3 n/a n/a H/s max: 278.3 H/s
[2018-03-24 15:54:06] speed 2.5s/60s/15m 278.2 n/a n/a H/s max: 278.3 H/s
[2018-03-24 15:54:21] speed 2.5s/60s/15m 278.3 278.2 n/a H/s max: 278.3 H/s
[2018-03-24 15:54:35] accepted (2/0) diff 20000 (113 ms)
[2018-03-24 15:54:36] speed 2.5s/60s/15m 278.1 278.2 n/a H/s max: 278.3 H/s
[2018-03-24 15:54:51] speed 2.5s/60s/15m 278.3 278.3 n/a H/s max: 278.3 H/s
[2018-03-24 15:55:06] speed 2.5s/60s/15m 278.3 278.3 n/a H/s max: 278.3 H/s
[2018-03-24 15:55:21] speed 2.5s/60s/15m 278.3 278.3 n/a H/s max: 278.3 H/s

EDIT: The results from earlier in the tread of the same server running E5-2660's just to compare.

"No cores turned off.
One server plugged in and both nodes powered off: 15w
One node powered on, Ubuntu 17.04 server booted up and idling : 56.5w
One node after 10 minutes mining Monero with 20 threads : 208w @ 868 H/s
Two nodes powered on, Ubuntu 17.04 server booted up and idling : 117w
Two nodes after 10 minutes mining Monero with 40 threads : 401w @ 1736 H/s"

I might run the 2603's during the summer just because the power use AND heat is alot less.
 
Last edited:

Klee

Well-Known Member
Jun 2, 2016
1,289
397
83
Downloading the daily build of Ubuntu Server 18.04 to give that a spin, been needing to replace the two WD black 1 TB hard drives with two of the 12 WD 250 gig drives that bought to put in all the oc server nodes. I already installed them in the other servers so now all the drives will be identical.
 

Klee

Well-Known Member
Jun 2, 2016
1,289
397
83
Well, rebooted and ran htop since it was already installed and it shows memory use at 655 mb.


Edit: Still does the hang on reboot just like Ubuntu 16.04, 17.04, 17.10.

So have to still do this: echo "options mei-me disable_msi=1" | sudo tee /etc/modprobe.d/mei-me.conf
 
Last edited:

Klee

Well-Known Member
Jun 2, 2016
1,289
397
83
Compiled xmrig 2.5.0 and running it now.

The fans are so quiet, I can only hear them when I put my head about 6 inches away from them. LOL

The power supply fan is about two or three times as loud.

The cpu's are running so cool, 45c and 47c, that the fans are barely spinning.

So three open compute servers running E5-2603 cpu's would be about 1675 H/s at about 490 watts, about the same as one oc server running 2660's but without the heat and noise.

Not real impressive in hash rate but real nice as far as being quiet and more importantly the lack of heat being put out compared to the 2660 and especially the hot 2665's.

Defiantly thinking about running them in the summer to cut down on the a/c usage which would lower my power bill a lot.
 
  • Like
Reactions: pgh5278