I'm in process of converting/expanding 8x4tb raid 5 array to 14x4tb raid 6 array, and thought it may be helpful to others to know times and impact on performance.
Equipment:
LSI 9271-8i raid controller
HGST 724040AL 4tb sata hdd (qty14)
RES2CV360 sas expander
Intel 520 4x240gb ssd cachecade
Workload:
medium/light - home/lab nas, streaming media
current array has 21tb data, 18% free space
Progress:
4 days 15 hrs 80%
5 days 18 hrs 20 min 100%
ETA:
1 day 5 hrs
expansion complete
(changes to reconstruction priority rate in MSM did not seem to have any effect)
Impact:
onboard read/write cache are disabled, as well as cachecade, during reconstruction
I'm too cautious to hammer on the array with random i/o, but sequential read/write are 15MB/s and 90MB/s, respectively.
Why?
I have always wiped arrays, created new & reloaded from backups on previous expansions, and was curious to see how long it would take. The new HDD were stressed for 48hrs and backups are available, so why not.
Conclusions/remaining questions:
Impact too high for production, OK for home lab
total array rebuild time - 5 days 18hr 20min
(controller kicked off a backgound init after expansion, which seems redundant after a complete rebuild, I aborted & will wait for scheduled patrol read) apparently the controller is not to be denied - init restarted
time required to expand NTFS partition - n/a immediate
formatted space increased from 25.4tb to 43.6tb
Good to know it can be done if needed, but as Chuckleb said, only if you have to.
benchmarks:
HD Tune ave read - 742MB/s
Anvil - 3,704
sequential network copy - read - 950MB/s, write - 650MB/s
(probably limited by older/slower HDD on target array, will update with ssd target pending time and ambition)
-----------------------------------------------------------------------
CrystalDiskMark 4.0.3 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 738.775 MB/s
Sequential Write (Q= 32,T= 1) : 409.090 MB/s
Random Read 4KiB (Q= 32,T= 1) : 247.399 MB/s [ 60400.1 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 82.007 MB/s [ 20021.2 IOPS]
Sequential Read (T= 1) : 770.921 MB/s
Sequential Write (T= 1) : 394.301 MB/s
Random Read 4KiB (Q= 1,T= 1) : 22.250 MB/s [ 5432.1 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 49.727 MB/s [ 12140.4 IOPS]
Test : 4096 MiB [S: 59.3% (26492.8/44705.9 GiB)] (x2)
Date : 2015/08/06 8:18:24
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)
Equipment:
LSI 9271-8i raid controller
HGST 724040AL 4tb sata hdd (qty14)
RES2CV360 sas expander
Intel 520 4x240gb ssd cachecade
Workload:
medium/light - home/lab nas, streaming media
current array has 21tb data, 18% free space
Progress:
4 days 15 hrs 80%
5 days 18 hrs 20 min 100%
ETA:
1 day 5 hrs
expansion complete
(changes to reconstruction priority rate in MSM did not seem to have any effect)
Impact:
onboard read/write cache are disabled, as well as cachecade, during reconstruction
I'm too cautious to hammer on the array with random i/o, but sequential read/write are 15MB/s and 90MB/s, respectively.
Why?
I have always wiped arrays, created new & reloaded from backups on previous expansions, and was curious to see how long it would take. The new HDD were stressed for 48hrs and backups are available, so why not.
Conclusions/remaining questions:
Impact too high for production, OK for home lab
total array rebuild time - 5 days 18hr 20min
(controller kicked off a backgound init after expansion, which seems redundant after a complete rebuild, I aborted & will wait for scheduled patrol read) apparently the controller is not to be denied - init restarted
time required to expand NTFS partition - n/a immediate
formatted space increased from 25.4tb to 43.6tb
Good to know it can be done if needed, but as Chuckleb said, only if you have to.
benchmarks:
HD Tune ave read - 742MB/s
Anvil - 3,704
sequential network copy - read - 950MB/s, write - 650MB/s
(probably limited by older/slower HDD on target array, will update with ssd target pending time and ambition)
-----------------------------------------------------------------------
CrystalDiskMark 4.0.3 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 738.775 MB/s
Sequential Write (Q= 32,T= 1) : 409.090 MB/s
Random Read 4KiB (Q= 32,T= 1) : 247.399 MB/s [ 60400.1 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 82.007 MB/s [ 20021.2 IOPS]
Sequential Read (T= 1) : 770.921 MB/s
Sequential Write (T= 1) : 394.301 MB/s
Random Read 4KiB (Q= 1,T= 1) : 22.250 MB/s [ 5432.1 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 49.727 MB/s [ 12140.4 IOPS]
Test : 4096 MiB [S: 59.3% (26492.8/44705.9 GiB)] (x2)
Date : 2015/08/06 8:18:24
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)
Last edited: