Mining Burst coins?

[funkywizard]

Wow, I have never seen anyone use such a large stagger size, interesting. Curious to see how it goes for you....Looking to get 100 drives plotted so any methods of speeding that up are great.

= D

I like to go big *and* go home.

So far the large stagger size hasn't been as easy as I anticipated. At first, did nonces per plot = 4x stagger (even with stagger > 100GB. In direct mode, this caused quite an i/o performance bottleneck even on the NVME SSD.

Direct mode with stagger = nonces seems to work better. Sequential i/o still takes at least as long, if not longer, tban calculating the nonces, but I'm working on that.

I'm hoping that telling windows to run the "move" command in the background, after saving a plot file to the SSD, it should be able to start plotting the next file right away.

Logically, it goes something like this (syntax is very different):

batch file 1:

generate c:/plotfile1
async move c:/plotfile1 d:/plots/

generate c;/plotfile2
async move c:/plotfile2 e:/plots/

generate c:/plotfile3
move c:/plotfile3 f:/plots/

(repeat above with g:, h:/, i:/)


batch file 2:

generate c:/plotfile7
move c:/plotfile1 j:/plots/

generate c;/plotfile8
async move c:/plotfile2 k:/plots/

generate c:/plotfile9
async move c:/plotfile3 l:/plots/

(repeat above with m:, n:/, o:/)


So, on the first batch file, with any luck it will start the second plot immediately after the first, while the first one moves in the background. Same with the second plot file. The third plot file (and 6th, 9th, etc) will do a sync move to help make sure the SSD doesn't get full or bogged down.

On the second batch file, similar idea, except the first, fourth, seventh, etc plot will be moved syncronously, while the 2nd, 3rd, 5th, 6th, etc will move async. Hopefully this lets the nonce processing and the most intense portions of the ssd i/o be staggered a bit between the two batch files.

If it works as expected, I should be writing to up to 6 hard drives at one time.

 

[funkywizard]

= D

I like to go big *and* go home.

So far the large stagger size hasn't been as easy as I anticipated. At first, did nonces per plot = 4x stagger (even with stagger > 100GB. In direct mode, this caused quite an i/o performance bottleneck even on the NVME SSD.

Direct mode with stagger = nonces seems to work better. Sequential i/o still takes at least as long, if not longer, tban calculating the nonces, but I'm working on that.

I'm hoping that telling windows to run the "move" command in the background, after saving a plot file to the SSD, it should be able to start plotting the next file right away.

Logically, it goes something like this (syntax is very different):

batch file 1:

generate c:/plotfile1
async move c:/plotfile1 d:/plots/

generate c;/plotfile2
async move c:/plotfile2 e:/plots/

generate c:/plotfile3
move c:/plotfile3 f:/plots/

(repeat above with g:, h:/, i:/)


batch file 2:

generate c:/plotfile7
move c:/plotfile1 j:/plots/

generate c;/plotfile8
async move c:/plotfile2 k:/plots/

generate c:/plotfile9
async move c:/plotfile3 l:/plots/

(repeat above with m:, n:/, o:/)


So, on the first batch file, with any luck it will start the second plot immediately after the first, while the first one moves in the background. Same with the second plot file. The third plot file (and 6th, 9th, etc) will do a sync move to help make sure the SSD doesn't get full or bogged down.

On the second batch file, similar idea, except the first, fourth, seventh, etc plot will be moved syncronously, while the 2nd, 3rd, 5th, 6th, etc will move async. Hopefully this lets the nonce processing and the most intense portions of the ssd i/o be staggered a bit between the two batch files.

If it works as expected, I should be writing to up to 6 hard drives at one time.

With the 175GB stagger size, here's what I see thus far:

First plot, both copies running at once. Both finish in 15 minutes, each doing 45k nonce / minute. Spends the next 5 minutes writing to SSD. One of them starts a background write to drive 1 -- likely to take 20 minutes -- and starts making the next plot right away. The second batch file starts moving the plot file to drive 2, but won't start the next plot until that has finished -- eta 20 minutes. Meanwhile, batch file 1 is doing its second plot a bit faster -- 62k nonce / min -- because it has the gpus all to itself. Eta 12 minutes.

At 174gb per plot, I need 16 plots to fill one drive. In theory this method allows for writing up to 6 drives at once at full sequential speed. The drive writing process is almost certainly the bottleneck here. A drive that can finish a full "dd" write in "X" time should take no more than 1.5X here. Being generous / optimistic, let's say that's 150MB/s. At "X" = 1.5, that gives you an effective speed of 100MB/s / drive, or 600MB/s for the whole server.

I've currently got 11x 3tb drives in here, 33TB, 31,471,252 MiB. If everything goes off flawlessly then, can expect around 52,450 seconds, or 874 minutes, or 14.5 hours to process 33TB.

To go any faster I would need more of these SSDs, for both the speed and the space.

 

[modder man]

So you are just using the 2 GPU's to plot correct? Not using CPU as well

 

[funkywizard]

With the 175GB stagger size, here's what I see thus far:

First plot, both copies running at once. Both finish in 15 minutes, each doing 45k nonce / minute. Spends the next 5 minutes writing to SSD. One of them starts a background write to drive 1 -- likely to take 20 minutes -- and starts making the next plot right away. The second batch file starts moving the plot file to drive 2, but won't start the next plot until that has finished -- eta 20 minutes. Meanwhile, batch file 1 is doing its second plot a bit faster -- 62k nonce / min -- because it has the gpus all to itself. Eta 12 minutes.

At 174gb per plot, I need 16 plots to fill one drive. In theory this method allows for writing up to 6 drives at once at full sequential speed. The drive writing process is almost certainly the bottleneck here. A drive that can finish a full "dd" write in "X" time should take no more than 1.5X here. Being generous / optimistic, let's say that's 150MB/s. At "X" = 1.5, that gives you an effective speed of 100MB/s / drive, or 600MB/s for the whole server.

I've currently got 11x 3tb drives in here, 33TB, 31,471,252 MiB. If everything goes off flawlessly then, can expect around 52,450 seconds, or 874 minutes, or 14.5 hours to process 33TB.

To go any faster I would need more of these SSDs, for both the speed and the space.

After watching a couple more of these execute, it seems that the hard drives are not the bottleneck. The first "background write" finishes around the time the second one is ready to start. So realistically I can expect to have 2-3 hard drives writing at any one time, not 6. 30 hours for 11x 3tb drives seems more realistic on this setup. Will let you know.

 

[funkywizard]

So you are just using the 2 GPU's to plot correct? Not using CPU as well

Dual E5-2660v1 could plot about 8k / min. The two gpus around 90k / min together. Not worth the effort. Also, I suspect we're single-thread-limited on CPU when gpu plotting. Running multiple instances increases the total cpu usage as well as performance (up to a point). Don't want to max out the cpus and then have the gpus plot slower as a consequence.

 

[modder man]

Dual E5-2660v1 could plot about 8k / min. The two gpus around 90k / min together. Not worth the effort. Also, I suspect we're single-thread-limited on CPU when gpu plotting. Running multiple instances increases the total cpu usage as well as performance (up to a point). Don't want to max out the cpus and then have the gpus plot slower as a consequence.

CPU's should do better than that.....my pair of 2670v1's do about ~30K

 

[funkywizard]

CPU's should do better than that.....my pair of 2670v1's do about ~30K

Interesting. I just used the gpu plotter but set it to the device showing up as cpus. Didn't put much effort into it. At 30k / minute without using GPUs, I could definitely see that being a viable option.

It's kind of a tough one though. Optimized plots are supposed to be better, but, the disk i/o needed to optimize them after the fact isn't very appealing. Meanwhile, if you want to optimize them "as you go" with huge stagger sizes, you're tying up a large amount of expensive ram. I have this ram on hand for the oddball 256gb server request, so I could set up one or two more 384gb servers, but that's it. I feel like the faster nonce rate combined with the pcie ssd for buffering lets me make good use of that ram.

If I were to cut my nonce rate from a peak of 90k / min down to 30 or 40k / min, harder to justify the extra ram and pcie ssd as the nonce rate would be a strong bottleneck. That said, I could drop to 192gb ram as that uses cheaper 8gb dimms I have on hand. Then running only one copy of the plotter at once, I could still keep the same stagger size I'm using now. Then so that the plotter can release its ram and start on new work, I do want somewhere fast to dump the files before writing them to disk. Could probably raid 0 some sata ssds, or I could still use a pcie ssd in a handful of servers. I guess until someone orders them, the SSDs aren't making me any money, but the CPUs are, so that's a fair tradeoff -- would keep the cpus working more of the time if flushing ram to disk goes faster.

Could probably do 8GB/min easily enough. 175gb stagger makes that 22 minutes. Something that can write 800MB/s sequential would take 4 minutes to flush the 175GB -- not bad. 2-4 sata SSDs in raid 0 or 1 nvme ssd should do that pretty easily. Then flushing to disk at 150MB/s, you'd spend 20 minutes dumping off that data -- just in time for the CPUs to be ready with another batch.

If it took end-to-end 30 minutes per 175GB, probably looking at 8 hours per 3tb drive per server. If you've got 12 drives and a pcie SSD, set it up and come back in 4 days for them all to be done. So tying up 8 servers for less than a week you'd be able to process 100 drives. About $250 in lost Aeon revenue, not bad = P

 

[modder man]

If it took end-to-end 30 minutes per 175GB, probably looking at 8 hours per 3tb drive per server. If you've got 12 drives and a pcie SSD, set it up and come back in 4 days for them all to be done. So tying up 8 servers for less than a week you'd be able to process 100 drives. About $250 in lost Aeon revenue, not bad = P

Where do you see the huge benefit to using the SSD to cache? are the disks not the bottleneck anyway? if the disks are writing all the time then it wouldn't make a difference no?

 

[funkywizard]

Where do you see the huge benefit to using the SSD to cache? are the disks not the bottleneck anyway? if the disks are writing all the time then it wouldn't make a difference no?

Using an SSD makes it easier to be writing to disk and calculate nonces at the same time. If you want your stagger size to be as large as possible, this is important. You can have the largest stagger size if you only run 1 plotter and use all your ram for it. When a stagger finishes being computed, the plotter pauses to write it to disk before continuing. As well, while plotting, the process won't be writing to disk. If you only run one or two plotter processes, this can really slow things down.

If you use a smaller stagger size and run multiple plotters at once, the SSD cache is less important during plotting. At any given time, there would be some plotters plotting and some writing to disk, so you'd be effectively using both resources. However, if you use a small stagger size, optimizing your plots becomes more important, which is very i/o intensive. At which point, you would also benefit from an SSD.

Personally, I would rather write optimized plots in one go, since either way you slice it, you'll be slowed down by i/o, even if your bigger bottleneck is plotting speed.

 

[Benjamin Vogel]

Would just like to share that I have been getting about 30 burstcoin a day using 4x 3TB hard drives. The computer I have them in is also mining monero with 2x E5-2660, and I don't see much of a drop in hashrate from burstcoin. It only takes about 15 seconds for the hard drives to read on each block.

 

[modder man]

Using an SSD makes it easier to be writing to disk and calculate nonces at the same time. If you want your stagger size to be as large as possible, this is important. You can have the largest stagger size if you only run 1 plotter and use all your ram for it. When a stagger finishes being computed, the plotter pauses to write it to disk before continuing. As well, while plotting, the process won't be writing to disk. If you only run one or two plotter processes, this can really slow things down.

If you use a smaller stagger size and run multiple plotters at once, the SSD cache is less important during plotting. At any given time, there would be some plotters plotting and some writing to disk, so you'd be effectively using both resources. However, if you use a small stagger size, optimizing your plots becomes more important, which is very i/o intensive. At which point, you would also benefit from an SSD.

Personally, I would rather write optimized plots in one go, since either way you slice it, you'll be slowed down by i/o, even if your bigger bottleneck is plotting speed.

well by now if you plots went as planned they should have finished. how is it looking for you?

 

[Mishka]

Been looking into this but I run plotter for 500GB (small test of it) and it basically instantly creates the file something seems a bit off? am using Qbundle

 

[funkywizard]

well by now if you plots went as planned they should have finished. how is it looking for you?

Last I looked at it, it was on track to complete within 43 hours. There are some opportunities to speed things up.

I had it wait every third file move (on each process) for the file to finish moving before continuing the processing. This would have made sense if computing nonces was dramatically faster than moving files from ssd to hdd, to avoid running out of disk space. However, computing nonces is one of the more important bottlenecks, and i never came close to running out of ssd space.

Going to try another batch later today. I think I can shave 10 hours off by having it always do the file moves in the background, instead of doing 2/3 of them in the background. Also going to start the second batch file 5 minutes after the first one. Hopefully the different processing stages for the two won't overlap as much, so I can get more use out of both the gpus and the ssd i/o.

I still need to see if I can get the cpu plotting to go a reasonable speed, and see if there is any opportunity for the gpu plotting to go faster as well. Any tips there would be appreciated.

 

[pyro_]

Question can you create the plot in one machine and then move it to another one?

 

[Patrick]

I think we need a best known methods post for Windows / Linux so we can get more people iterating

 

[modder man]

I think we need a best known methods post for Windows / Linux so we can get more people iterating

I agree. I have been using xplotter on windows with decent results. It is just not all that fast. I have ~120 drives on hand that currently need plotted. Very time consuming with this method.

 

[funkywizard]

Question can you create the plot in one machine and then move it to another one?

Yes.

The plot is tied to the wallet id# and / or the pool details. Not tied to the machine.

 

[funkywizard]

I think we need a best known methods post for Windows / Linux so we can get more people iterating

I'd be happy to run some tests if people want me to try out specific plotting software and/or settings. I have no idea what would be considered "good" for nonces / minute for cpus or gpus. I was under the impression that plotting was windows-only, so have only tried that thus far, but I do see that gpuplotter has a linux version.

So far have tried gpu plotter with 1 and 2 gtx 1070's, as well as with 1x 1080ti. I can give optimized gpuplotter "devices.txt" settings for the 1070 at this point, did plenty of testing on that. 1080ti was not faster enough to justify tying it up plotting. I could also try a 1060, 1080, and vega, if there is interest in that.

Have tried direct and buffer with small stagger sizes for small test plots, but I've found that things behave differently with larger production plots. I find that performance plotting degrades quite a bit using "direct" and using nonces>stagger, even when using an nvme ssd. If you must use nonces>stagger, I suspect that using buffer mode and optimizing afterwards will be a better strategy.

I should really try that out -- use buffer mode to write to a fast disk with a reasonable stagger size, and use a plot optimizer once the plot is written to the fast disk. The plot optimizer can write the finished plots to the slower hard drive, reading the source data from the nvme ssd.

This might let me keep the GPUs busy more of the time, plotting with 4 processes instead of 2. Using a stagger size of, say, 16gb instead of 174gb -- uses 64gb ram plotting instead of 370gb -- far more manageable. I'll probably keep the plots at 174gb apiece. That lets me fit 8 plots on the SSD (4 completed and 4 in-progress), and fits 16 plots to a 3TB hard drive.

 

[funkywizard]

Anyone have a recommendation for CPU plotting? Still want to try that out, but my initial results for nonce / minute were very bad.

Also, anyone know a windows plot optimizer that can be run command line?

 

[modder man]

Anyone have a recommendation for CPU plotting? Still want to try that out, but my initial results for nonce / minute were very bad.

Also, anyone know a windows plot optimizer that can be run command line?

Xplotter works quite well on windows for CPU. quick and easy