THE CHIA FARM

funkywizard

mmm.... bandwidth.
Jan 15, 2017
848
402
63
USA
ioflood.com
HGST is actually owned by WD now, and the drives are WDs under the hood. The other manufacturer besides WD and Seagate who makes drives themselves instead of just relabeling is Toshiba.

And yes, all of them have put out some terrible products over the years, as well as some excellent ones. Spinning rust is a fundamentally unreliable technology, so failures are to expected from any brand -- that's why we have RAID. It's valid to compare failure rates between the different manufacturers, but you really need to do that for a particular capacity point or year for it to mean much, because when a new technology is introduced (such as HAMR/MAMR), that can change the whole picture. Backblaze data is pretty useful for making those kinds of comparisons: Backblaze Hard Drive Stats
If you look at physically the more recent WD drives versus the equivalent / older HGST models, I would argue the newer WD, especially enterprise / helium filled models, are really HGST under the hood. The shape of the drives are clearly the same, and it's not as if WD wanted to ditch the manufacturing plants and technology when buying HGST. WD HC530 Ultrastar for example, pretty sure that's just an HGST drive for all intents and purposes.

There was also a long period after the acquisition where China held up the merger because they wanted to ensure HGST drives continued to be manufactured in China, so even after the acquisition for quite some time there were two separate lines where a true technology / product line merger was not allowed to move forward.

I think for a while you'll see some WD product lines that clearly reflect WD engineering, production, and design, and some product lines that clearly reflect HGST engineering, production, and design.
 

Rand__

Well-Known Member
Mar 6, 2014
6,401
1,632
113
Hm... not sure which is the most fitting reaction

"Yay, we can move the bottleneck from CPU back to storage"
"Yay, Netspace is gonna grow even faster now"
"Yay, now I can plot even faster and buy more drives soon"

But o/c its nice to see increased efficiency :)
 

Bert

Active Member
Mar 31, 2018
562
233
43
44
Well this makes plotting much easier. IIUC this doesn't reduce cpu usage just increase parallelism.

I guess network explosion will happen much faster now. That may be a good thing as it will discourage new comers and we can start to get cheaper hard drives.
 
Last edited:

Rock

Member
Jan 28, 2020
76
47
18
Northern California
If this plotter is real (and not a tool that steals your private keys) then the maker should have submitted it since there is/was a $50,000 reward for finding a faster or more efficient plotting solution.
 

NateS

Active Member
Apr 19, 2021
159
88
28
Sacramento, CA, US
If you look at physically the more recent WD drives versus the equivalent / older HGST models, I would argue the newer WD, especially enterprise / helium filled models, are really HGST under the hood. The shape of the drives are clearly the same, and it's not as if WD wanted to ditch the manufacturing plants and technology when buying HGST. WD HC530 Ultrastar for example, pretty sure that's just an HGST drive for all intents and purposes.

There was also a long period after the acquisition where China held up the merger because they wanted to ensure HGST drives continued to be manufactured in China, so even after the acquisition for quite some time there were two separate lines where a true technology / product line merger was not allowed to move forward.

I think for a while you'll see some WD product lines that clearly reflect WD engineering, production, and design, and some product lines that clearly reflect HGST engineering, production, and design.
Yes, good point. I didn't mean that they just shut down the HGST manufacturing line and started using the WD line only for both labels. Rather, today drives built on either production line can get labeled WD or HGST depending on what market they're being sold into.
 
  • Like
Reactions: funkywizard

iotapi322

Member
Sep 8, 2017
66
14
8
46
Are you using a ramdisk for temp2? Or is 3 hours just using nvme's?
straight up NVMe's and on the old e5 just an old samsung 840 Pro SSD... Working on getting my raid0 disk array with 15 disks up again.
This has been a fun learning lesson in reliability and redundancy.
 

iotapi322

Member
Sep 8, 2017
66
14
8
46
If this plotter is real (and not a tool that steals your private keys) then the maker should have submitted it since there is/was a $50,000 reward for finding a faster or more efficient plotting solution.
If you look at the code it's not doing anything interesting. Also I've checked my ports on the machine that's doing this work and it's not opening anything.
 
  • Like
Reactions: NateS

NateS

Active Member
Apr 19, 2021
159
88
28
Sacramento, CA, US
If this plotter is real (and not a tool that steals your private keys) then the maker should have submitted it since there is/was a $50,000 reward for finding a faster or more efficient plotting solution.
Who was offering a reward? And has no one won it yet? At this point several people have released plotters (either from scratch or forks of the original) that improved on the original plotting code, which the Chia devs admitted was never very optimized. There's almost certainly several more which are not publicly released, such as Hpool's plotter, and I would not be surprised if some whales are hiring devs to optimize custom plotters for them only.
 

Bert

Active Member
Mar 31, 2018
562
233
43
44
If you look at the code it's not doing anything interesting. Also I've checked my ports on the machine that's doing this work and it's not opening anything.
Can you tell more about your set up?

1. How many plots are you running in parallel when you generate 3 hours per plot?
2. How many total cores and how much memory do you have?
3. What is your nvme make/model? How many do you have in the system?
4. Why do you need to RAID 0 15 disks?
5. What are you using for temp1?


Thanks a lot!
 

RimBlock

Active Member
Sep 18, 2011
838
28
28
Singapore
Will have to see how much better it is against parallel and staggered plotting.
  • 4x 4t/4gb ram (16c/ 16GB Ram) = 9 hours = 4 plots (parallel). More ram but one more plot for the same given time.
  • 16t/6GB ram = 3 hours = 1 plot (single) = 9 hours = 3 plots.
If staggered it will add maybe 3 hours to the parallel runs (based on my setup) it lines up but still no improvements in plot count.
  • 8x 4t/4gb ram (32c/ 32GB Ram) = 12 hours = 8 plots (parallel staggered).
  • 2x 16t/6GB ram (32c/12GB Ram) = 3 hours = 2 plot (Parallel) = 12 hours = 8 plots.

The advantage of staggered currently is that stages 2-4 only use very little in the way of c/t so other plots can then take up those resources. This potentially removes that.

There is surely some tuning that can be done with parallel, staggered, threads / ram settings with the new plotter so I guess we will see how this goes.

One concern is the creators future plans to integrate OpenCL and if that allows a shift away from high CPU load, how will this change the landscape... Bram mentioned that one of the core requirements was to make it so people cannot plot too fast and react to the network, gaining advantage by replotting. Will be interesting how the team work to manage the inevitable march of progress.
 
  • Like
Reactions: Bert

RimBlock

Active Member
Sep 18, 2011
838
28
28
Singapore
Had an issue with Syncing yesterday with a new Windows machine.

I have built a new fullnode for farming, around an Intel Nuc which has USB storage attached.

After installing Win 10 Pro and the Chia software it would not sync (copied the DB over from another machine but would not move forwards). Nat confirmed working, going to an older version of the Chia software had no effect etc.

Solution ended up being that the Chia software only opened incoming firewall ports and so traffic could get to me but I could not send anything out.

Added the rules and now Chia has synced.

Updated an issue on the bug tracker here in case anyone else hits this issue (seems to be a fair few with the same symptoms).
 
  • Like
Reactions: Bert

Bert

Active Member
Mar 31, 2018
562
233
43
44
One concern is the creators future plans to integrate OpenCL and if that allows a shift away from high CPU load, how will this change the landscape... Bram mentioned that one of the core requirements was to make it so people cannot plot too fast and react to the network, gaining advantage by replotting. Will be interesting how the team work to manage the inevitable march of progress.
When it comes to plotting, I think community got mislead. There was no need for NVMEs, threadripper or high end CPU. I bought a supermicro server with 24 2.5" bays. I came with an H8DGU motherboard so I just bough cheap 16 core AMD opertons, total less than $40. Put 24 15K spindles on them, now I am almost memory bound, with 48GB memory. I didn't think CPU would handle so much load but it did. Less than $350 with memory, I can plot 32+ plots a day. With more memory, I will probably get


Overall, plotting is cheap, all we need to do parallelism hence disk and memory. Did anyone use the spindles they have for farming also for plotting?
 

bash

Active Member
Dec 14, 2015
131
61
28
40
scottsdale
I was already plotting 250/day without this new plotter. Your available disk space will always be the bottleneck
 

Bert

Active Member
Mar 31, 2018
562
233
43
44
I was already plotting 250/day without this new plotter. Your available disk space will always be the bottleneck
Yes, I am out of disk space weeks ago. All I have left is SMR drives from Costco and bottleneck over there is the write speed. It is really questionable to add drives given how low the return is.

I am running the plotter for a while now, and even without ramdisk, it is super fast. Phase 1 took 2658 second with 4 spindles as temp drive.


Perhaps this will push memory prices higher now :)
 

Bert

Active Member
Mar 31, 2018
562
233
43
44
Was anyone able to try? it worked with regular drives but when I switched to ramdisk (tmpfs), plotting hangs after Table3 in phase 1. It actually stops all plotting activity on my server so there must be an I/O lock somewhere.

I have 160GB Memory on my system. I am not sure if system ran out of memory
 

msg7086

Active Member
May 2, 2017
407
148
43
35
Can you make sure you are not running out of memory? I finished a plot without problems.