Not here for the $20, just here to offer some advice. Spent MANY hours working out how to plot fast and figured I'd see if I could help.
As a side note, Not sure what you did with the ramdisk to make it NTFS but simply do as the README.md says. Delete your current ramdisk and do:
Code:
sudo mount -t tmpfs -o size=110G tmpfs /mnt/ram/
Don't format it. Just point yourTMP2 to it.
Also, don't use the RAID0 modes on the p410i, they're crap and halved the perf of my 4x Intel S4510 SSD RAID0. Switch to HBA mode (you'll need to google how) and create an MDADM Raid0.
On Windows I used a program called ImDisk, and the default file system was NTFS when you configure it. I don't remember what the other options were, but I never touched it because I had no idea what that would do.
On Ubuntu I'm doing it exactly as it says in the plotter wiki, exactly that line.
The p410 doesn't have HBA mode, so I'm stuck with raid 0 on each disk. I'll try different cache settings and turning it off altogether. It's on right now and set to 50% read/50% write, and that was the best option for Windows, but it might be different on ubuntu. I'm only doing this with 1 plot right now, so I'm expecting it to be much slower with a few plots in parallel.
btrfs seems to be even better (
btr, ha-ha) than ext4. I can't give you the exact numbers because I didn't do all of phase one, but it's noticeable.
I have p410 (non i), and I've read about how to flash it into IT mode, that's a bit too advanced for me right now, I'm quite happy with the performance boost so far, I'm just going to test different file systems/cache settings/threads/buckets for now.
he did say he tried with nvme in the original post, what was your times with nvme disk? even my non enterprise nvmes can do sub 2000s plottimes.
dont trust HWmon for power usage, get a wattometer at 1.05 full load you are def using 2-400w. and i would check temps in IPMI where they probabaly are labeled correctly so you know which one is VRM/memory etc. how are you cooling this? seems like you are pushing it near thermal throttling limits
My ipmi readings are pretty scarce, but the temps are about the same as the CCD temps (in the high 50s). This is how little info there is (not under load)
ID | Name | Type | Reading | Units | Event
1 | CPU1 Temperature | Temperature | 44.00 | C | 'OK'
2 | +VCORE1 | Voltage | 1.02 | V | 'OK'
3 | +VSOC1 | Voltage | N/A | V | N/A
4 | +VDDIO_ABCD_CPU1 | Voltage | N/A | V | N/A
5 | +VDDIO_EFGH_CPU1 | Voltage | N/A | V | N/A
6 | +12V | Voltage | 12.12 | V | 'OK'
7 | +5V | Voltage | 5.14 | V | 'OK'
8 | +5VSB | Voltage | 4.97 | V | 'OK'
9 | +3.3V | Voltage | 3.34 | V | 'OK'
10 | +3.3VSB | Voltage | 3.31 | V | 'OK'
11 | VBAT | Voltage | 3.10 | V | 'OK'
12 | FRNT_FAN1 | Fan | N/A | RPM | N/A
13 | FRNT_FAN2 | Fan | 800.00 | RPM | 'OK'
14 | FRNT_FAN3 | Fan | 800.00 | RPM | 'OK'
15 | FRNT_FAN4 | Fan | N/A | RPM | N/A
16 | FRNT_FAN5 | Fan | N/A | RPM | N/A
17 | FRNT_FAN6 | Fan | N/A | RPM | N/A
18 | REAR_FAN1 | Fan | N/A | RPM | N/A
19 | REAR_FAN2 | Fan | N/A | RPM | N/A
20 | PMBPower1 | Power Supply | N/A | W | N/A
21 | PSU1 Over Temp | Temperature | N/A | N/A | N/A
22 | PSU1 AC Lost | Power Supply | N/A | N/A | N/A
23 | PSU1 Slow FAN1 | Fan | N/A | N/A | N/A
24 | PSU1 PWR Detect | Power Supply | N/A | N/A | 'Power Supply Failure detected'
25 | CPU1_ECC1 | Memory | N/A | N/A | 'Presence detected'
26 | Memory_Train_ERR | OEM Reserved | N/A | N/A | 'OK'
27 | DIMMA1_Temp | Temperature | N/A | C | N/A
28 | DIMMA2_Temp | Temperature | N/A | C | N/A
29 | DIMMB1_Temp | Temperature | N/A | C | N/A
30 | DIMMB2_Temp | Temperature | N/A | C | N/A
31 | DIMMC1_Temp | Temperature | N/A | C | N/A
32 | DIMMC2_Temp | Temperature | N/A | C | N/A
33 | DIMMD1_Temp | Temperature | N/A | C | N/A
34 | DIMMD2_Temp | Temperature | N/A | C | N/A
35 | DIMME1_Temp | Temperature | N/A | C | N/A
36 | DIMME2_Temp | Temperature | N/A | C | N/A
37 | DIMMF1_Temp | Temperature | N/A | C | N/A
38 | DIMMF2_Temp | Temperature | N/A | C | N/A
39 | DIMMG1_Temp | Temperature | N/A | C | N/A
40 | DIMMG2_Temp | Temperature | N/A | C | N/A
41 | DIMMH1_Temp | Temperature | N/A | C | N/A
42 | DIMMH2_Temp | Temperature | N/A | C | N/A
44 | Watchdog2 | Watchdog 2 | N/A | N/A | 'OK'
This is how I'm cooling it. Please, don't laugh
. Ikea
dish rack +
Immeln shower basket work like a charm for the 2.5 inch drives. This
plate holder is a good fit for the 3.5 inch drives. I thought that this was pretty ingenious,
ha-ha. r/RedneckChiaFarmer/ for the win!
So I don't think I'm thermally throttling XD
Happy you figured it out, sounds like you should just stick to ramdisk plotting, maybe get a few higher capacity sticks if you are low
If I had the cash, I would just get 512 Gigs of ram, because the RAMdisk seems to be the next bottleneck. With 256 gigs you can still only do 1 parallel plot in a RAMdisk, if you do 2, then you're really limited on threads (like I was. That 27.5 min time is with only 16 threads because I was afraid of running out of RAM)... But I don't have the money for that, so I'll have to optimize a bunch of SAS drives for now. Thank you for your help!
Edit: XFS actually turned out to be much faster in phase 3, and that makes up for the drawbacks of phases 1 and 2. Phase 4 is also faster than btrfs. So it does seem to be the better choice.
This is btrfs↓
oem@KRPA-U16-Series:~/Desktop/chia-plotter/build$ sudo ./chia_plot -n 2 -r 32 -u 256 -t /mnt/sas10kbtrfs1/ -2 /mnt/sas10kbtrfs2/ -d /mnt/hdd1/chia/ -p [redacted] -f [redacted]
[sudo] password for oem:
Multi-threaded pipelined Chia k32 plotter - 9e649ae
Final Directory: /mnt/hdd1/chia/
Number of Plots: 2
Crafting plot 1 out of 2
Process ID: 100975
Number of Threads: 32
Number of Buckets P1: 2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Public Key: [redacted]
Farmer Public Key: [redacted]
Working Directory: /mnt/sas10kbtrfs1/
Working Directory 2: /mnt/sas10kbtrfs2/
Plot Name: plot-k32-2021-06-25-01-42-05a014c560fc3a0897b8043d7528ee8b4fa161240a90fa614130ae01057b42de
[P1] Table 1 took 194.272 sec
[P1] Table 2 took 687.062 sec, found 4295039141 matches
[P1] Table 3 took 1358.74 sec, found 4295079574 matches
[P1] Table 4 took 1723.49 sec, found 4294933818 matches
[P1] Table 5 took 1620.12 sec, found 4295014589 matches
[P1] Table 6 took 1288.18 sec, found 4294973874 matches
[P1] Table 7 took 808.838 sec, found 4294916267 matches
Phase 1 took 7680.77 sec
[P2] max_table_size = 4295079574
[P2] Table 7 scan took 146.502 sec
[P2] Table 7 rewrite took 192.834 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 159.919 sec
[P2] Table 6 rewrite took 123.088 sec, dropped 581282465 entries (13.534 %)
[P2] Table 5 scan took 334 sec
[P2] Table 5 rewrite took 146.915 sec, dropped 762006171 entries (17.7416 %)
[P2] Table 4 scan took 277.278 sec
[P2] Table 4 rewrite took 177.166 sec, dropped 828841876 entries (19.2981 %)
[P2] Table 3 scan took 233.01 sec
[P2] Table 3 rewrite took 145.591 sec, dropped 855149411 entries (19.91 %)
[P2] Table 2 scan took 255.942 sec
[P2] Table 2 rewrite took 88.4865 sec, dropped 865621825 entries (20.154 %)
Phase 2 took 2323.16 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 293.145 sec, wrote 3429417316 right entries
[P3-2] Table 2 took 167.358 sec, wrote 3429417316 left entries, 3429417316 final
[P3-1] Table 3 took 355.395 sec, wrote 3439930163 right entries
[P3-2] Table 3 took 152.394 sec, wrote 3439930163 left entries, 3439930163 final
[P3-1] Table 4 took 729.451 sec, wrote 3466091942 right entries
[P3-2] Table 4 took 221.746 sec, wrote 3466091942 left entries, 3466091942 final
[P3-1] Table 5 took 422.651 sec, wrote 3533008418 right entries
[P3-2] Table 5 took 514.36 sec, wrote 3533008418 left entries, 3533008418 final
[P3-1] Table 6 took 493.065 sec, wrote 3713691409 right entries
[P3-2] Table 6 took 783.113 sec, wrote 3713691409 left entries, 3713691409 final
[P3-1] Table 7 took 1619.62 sec, wrote 4294916267 right entries
[P3-2] Table 7 took 1908.04 sec, wrote 4294916267 left entries, 4294916267 final
Phase 3 took 7672.57 sec, wrote 21877055515 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 646.671 sec, final plot size is 108834923212 bytes
Total plot creation time was 18323.3 sec (305.388 min)
And this is XFS↓
oem@KRPA-U16-Series:~/Desktop/chia-plotter/build$ sudo ./chia_plot -n 2 -r 32 -u 256 -t /mnt/sas10kxfs1/ -2 /mnt/sas10kxfs2/ -d /mnt/sdg2/chia/ -p [redacted] -f [redacted]
[sudo] password for oem:
Multi-threaded pipelined Chia k32 plotter - 9e649ae
Final Directory: /mnt/sdg2/chia/
Number of Plots: 2
Crafting plot 1 out of 2
Process ID: 101048
Number of Threads: 32
Number of Buckets P1: 2^8 (256)
Number of Buckets P3+P4: 2^8 (256)
Pool Public Key: [redacted]
Farmer Public Key: [redacted]
Working Directory: /mnt/sas10kxfs1/
Working Directory 2: /mnt/sas10kxfs2/
Plot Name: plot-k32-2021-06-25-01-42-bebcca109c74eb7b75c039292aaffbd5a0d01adef9beca52f05f5fc253a83f20
[P1] Table 1 took 193.733 sec
[P1] Table 2 took 737.067 sec, found 4294898626 matches
[P1] Table 3 took 1569.28 sec, found 4294892720 matches
[P1] Table 4 took 1695.52 sec, found 4294775625 matches
[P1] Table 5 took 1593.61 sec, found 4294615134 matches
[P1] Table 6 took 1352.24 sec, found 4294089837 matches
[P1] Table 7 took 1000.04 sec, found 4293192544 matches
Phase 1 took 8145.56 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 290.979 sec
[P2] Table 7 rewrite took 354.516 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 228.482 sec
[P2] Table 6 rewrite took 168.975 sec, dropped 581374327 entries (13.5389 %)
[P2] Table 5 scan took 347.717 sec
[P2] Table 5 rewrite took 166.808 sec, dropped 762125691 entries (17.7461 %)
[P2] Table 4 scan took 317.125 sec
[P2] Table 4 rewrite took 336.656 sec, dropped 828926885 entries (19.3008 %)
[P2] Table 3 scan took 227.433 sec
[P2] Table 3 rewrite took 184.671 sec, dropped 855147880 entries (19.9108 %)
[P2] Table 2 scan took 289.571 sec
[P2] Table 2 rewrite took 127.379 sec, dropped 865606147 entries (20.1543 %)
Phase 2 took 3069.45 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 261.695 sec, wrote 3429292479 right entries
[P3-2] Table 2 took 140.268 sec, wrote 3429292479 left entries, 3429292479 final
[P3-1] Table 3 took 340.78 sec, wrote 3439744840 right entries
[P3-2] Table 3 took 152.976 sec, wrote 3439744840 left entries, 3439744840 final
[P3-1] Table 4 took 315.486 sec, wrote 3465848740 right entries
[P3-2] Table 4 took 171.027 sec, wrote 3465848740 left entries, 3465848740 final
[P3-1] Table 5 took 300.838 sec, wrote 3532489443 right entries
[P3-2] Table 5 took 174.087 sec, wrote 3532489443 left entries, 3532489443 final
[P3-1] Table 6 took 317.332 sec, wrote 3712715510 right entries
[P3-2] Table 6 took 174.976 sec, wrote 3712715510 left entries, 3712715510 final
[P3-1] Table 7 took 727.86 sec, wrote 4293192544 right entries
[P3-2] Table 7 took 327.776 sec, wrote 4293192544 left entries, 4293192544 final
Phase 3 took 3415.87 sec, wrote 21873283556 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 92.7084 sec, final plot size is 108811925901 bytes
Total plot creation time was 14723.7 sec (245.395 min)
Edit2: I'm getting ready to scale up my plotting, and decided to test without the 1 GB cache module on the P410. XFS is even faster (14%) than with the cache, but BTRFS is much, much slower. Decided to add this in case anyone is in the same boat.