Enterprise SSD "small deals"

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

ano

Well-Known Member
Nov 7, 2022
743
324
63
I've had a 4 x vdev x 6 disk rz2 pool running the same config for the last decade +. Every 24mo or so I refresh a whole vdev one disk at a time with a size bump. This has been working for me with used sas disks the whole time. (optane l2arc, 6 disk ssd 3 way mirror for meta)
did you use scripts to shuffle data around? or just have an unoptimized layout? with sas ssd its probably fine still
 

ca3y6

Well-Known Member
Apr 3, 2021
685
664
93
I can likely vouch for that. I was testing 8-wide RAID-Z2 vdevs with HGST 8TB SAS drives (~200MB/s rated), and you'd think that I'd at least get N-2*200 = ~1.2GB/s...but no. Got ~700-800MBps best case.
I think RAIDZ2 is more computationally expensive than RAIDZ1 which should be a simple XOR. What sort of hardware (CPU/RAM speed/HBA) were you running this on?
 

kapone

Well-Known Member
May 23, 2015
1,821
1,210
113
I think RAIDZ2 is more computationally expensive than RAIDZ1 which should be a simple XOR. What sort of hardware (CPU/RAM speed/HBA) were you running this on?
16 cores with 256gb RAM. I’ve run these tests on different platforms, all the way from Ivy Bridge to Epyc. The results are the same, with minor variations.
 
  • Like
Reactions: abq

thulle

Member
Apr 11, 2019
84
35
18
I think RAIDZ2 is more computationally expensive than RAIDZ1 which should be a simple XOR. What sort of hardware (CPU/RAM speed/HBA) were you running this on?
That would only be an issue for writes though, the parity calculation isn't necessary on reads, just the checksum.

And this fits with my observations too, 2x 10-wide Z2 VDEVs give about 1.5GB/s if i dd a file to null for example.
64 cores & ~128GB 3200MT/s ddr4 passed to the VM along with a 9305 24i HBA.

Interestingly a scrub can do well above 3.2GB/s with peaks up to 4.5-5GB/s
 
Last edited:

ca3y6

Well-Known Member
Apr 3, 2021
685
664
93
That would only be an issue for writes though, the parity calculation isn't necessary on reads, just the checksum.

And this fits with my observations too, 2x 10-wide Z2 VDEVs give about 1.5GB/s if i dd a file to null for example.
64 threads, ~64cores passed to the VM along with a 9305 24i HBA.

Interestingly a scrub can do well above 3.2GB/s with peaks up to 4.5-5GB/s
Doesn’t ZFS check the parity on reads as well? I thought that was a key data reliability feature of ZFS. Though I presume it can do that in a non blocking way.
 

thulle

Member
Apr 11, 2019
84
35
18
Doesn’t ZFS check the parity on reads as well? I thought that was a key data reliability feature of ZFS. Though I presume it can do that in a non blocking way.
It's read on scrubs, but otherwise not afaik. Unless the checksum fails. Better to spend the disk bandwidth on actual data reads?
 

ca3y6

Well-Known Member
Apr 3, 2021
685
664
93
It's read on scrubs, but otherwise not afaik. Unless the checksum fails. Better to spend the disk bandwidth on actual data reads?
From a performance point of view it makes sense. But from a “self-healing” point of view, if you check the checksum on the data but don’t check the parity, then you are exposed to rot in the parity data which will bite you the day you lose a disk.
 

thulle

Member
Apr 11, 2019
84
35
18
From a performance point of view it makes sense. But from a “self-healing” point of view, if you check the checksum on the data but don’t check the parity, then you are exposed to rot in the parity data which will bite you the day you lose a disk.
Yeah, that's what scrubs are for, making sure you check all the data with the performance benefit of not having to read hotter (parity) data over and over and over again during normal operation.
 
Last edited:

RandellH

New Member
Oct 18, 2020
28
13
3
This leads me to believe the checksum is calculated again on read and compared to what was calculated when it was written.

OpenE Link

Maybe I misunderstood what was said and this already know
 

Fritz

Well-Known Member
Apr 6, 2015
3,640
1,593
113
71
So what exactly does “Working Datacenter Pulls” tell me about the health of the drives?
 

MSameer

Member
May 8, 2025
85
70
18
Oh crap. I have not noticed that. Cannot even read it properly.

I will report back but shipping to the EU will take a long time. Might be gone by the time it arrives.
I got the drives at last. Well, shipping to EU is not fast.

I surprisingly did not pay extra VAT. No idea how that happened. post claimed VAT has been payed even though I only payed the original price without VAT on top.

The drives were sold out long ago but I am attaching smartctl -a output in case anyone is interested. Seems from my unexperienced eyes the drives are fine with not much TBW.
 

Attachments

potrebitel

New Member
May 8, 2020
29
9
3
Cool :)

VAT up to specific value should be added automatically by eBay, so we (as in EU) do not pay for separately.

Its much more convenient to be handled by eBay/Amazon/etc (even if Aliexpress claiming they do), instead of us and makes buying easier for PC parts (as they should be excluded from duty in some countries, so we pay VAT only).