Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Hi,
I am hoping someone (*cough Acquacow) might be able to help me figure out some very strange and inconsistent behavior that I am seeing when using my Fusion IO Drive 2 1.2 TB MLC 1.2TB in Windows Storage Spaces,
System configuration is as follows:
EPYC 7601
64GB RAM 8x8GB - DIE (UMA) interleaving
SuperMicro H11SSL-NC
3x IBM IODrive2 1.2 TB MLC (latest firmware)
I use this machine for 4k video editing/rendering, CAD/CAM, and various engineering applications. I am attempting to performance optimize a (simple) RAID 0 stripe set across all 3 of my IODrive2 cards, counting on their internal redundancy and frequent backups to keep data safe. This is a working space for editing complex video timelines with multiple overlaid effects and so needs to be configured for maximum sequential read/write performance. I have been attempting (unsuccessfully) to optimize the performance of this configuration for around 3 weeks and am being stymied by inconsistent measurements in CrystalDiskMark 6.0.2 even when conducting runs using the same settings. It is driving me crazy because without reproduceable results I am shooting in the dark with regards to which (fairly convoluted/complex) set of configurations provides me with the best performance between the options available via fio-config, storage pool setup, and virtual disk configuration. Some runs I will get north of 4GB/s for reads and 2.7GB/s for writes and then other runs I will get 2.7GB/s for reads and 4GB/s for sequential writes (!), and other times both will absolutely suck and be around 2.5/2.5. With the same settings. Needless to say this is maddening behavior.
I want to stick with storage spaces primarily because I have found that if I just make a plain software RAID 0 stripe set via Disk Manager I lose the ability to issue a TRIM command to the drives, whereas this is retained when creating a storage spaces simple volume.
Anyhow currently I have the drives configured as follows:
PCI-E power override 50W
4KB low level reformat - standard capacity
Latest FW/Driver
Paging Support Off
Storage Pool Configured:
Simple (stripe set)
3 Columns
8GB Write Cache
Virtual Hard Drive Configured:
Default 256KB stripe width (have also tried 64KB and it is substantially slower)
4KB allocation unit
64KB NTFS Formatting
Some odd behavior that I have noticed is that if I optimize the storage pool it consistently causes performance to drop in subsequent benchmarking runs until I destroy and recreate it. The same behavior seems to be appearing if I use TRIM support in Windows.
It is a head scratcher. One possibility is that I am experiencing some strange thread management behavior with this manycore CPU and performance is varying by what core the VSL driver is parked on, its proximity to memory, and its proximity to the PCIE lanes connected to the IO Drives. I am wondering if switching out of UMA (die interleaving) and using NUMA affinity commands might help?
This is driving me nuts I think I have tried probably 50 combinations of storage spaces, storage pool, and IODrive settings and am getting all sorts of crazy inconsistent results that aren't helping me to narrow down the best (and stable) combination of settings.
Raw math says that CrystalDiskMark should be reporting something close to 4.5GB/s reads and 4GB/s writes with three drives in an array. I have gotten close to that number on both sides but always at the cost of the other and not consistently.
Any ideas? I am happy to nuke the array a dozen times until I get it right, mostly because it is annoying me and also because it does cause some strange variability when using Davinic Resolve that I would like to go away.
Thanks!
I am hoping someone (*cough Acquacow) might be able to help me figure out some very strange and inconsistent behavior that I am seeing when using my Fusion IO Drive 2 1.2 TB MLC 1.2TB in Windows Storage Spaces,
System configuration is as follows:
EPYC 7601
64GB RAM 8x8GB - DIE (UMA) interleaving
SuperMicro H11SSL-NC
3x IBM IODrive2 1.2 TB MLC (latest firmware)
I use this machine for 4k video editing/rendering, CAD/CAM, and various engineering applications. I am attempting to performance optimize a (simple) RAID 0 stripe set across all 3 of my IODrive2 cards, counting on their internal redundancy and frequent backups to keep data safe. This is a working space for editing complex video timelines with multiple overlaid effects and so needs to be configured for maximum sequential read/write performance. I have been attempting (unsuccessfully) to optimize the performance of this configuration for around 3 weeks and am being stymied by inconsistent measurements in CrystalDiskMark 6.0.2 even when conducting runs using the same settings. It is driving me crazy because without reproduceable results I am shooting in the dark with regards to which (fairly convoluted/complex) set of configurations provides me with the best performance between the options available via fio-config, storage pool setup, and virtual disk configuration. Some runs I will get north of 4GB/s for reads and 2.7GB/s for writes and then other runs I will get 2.7GB/s for reads and 4GB/s for sequential writes (!), and other times both will absolutely suck and be around 2.5/2.5. With the same settings. Needless to say this is maddening behavior.
I want to stick with storage spaces primarily because I have found that if I just make a plain software RAID 0 stripe set via Disk Manager I lose the ability to issue a TRIM command to the drives, whereas this is retained when creating a storage spaces simple volume.
Anyhow currently I have the drives configured as follows:
PCI-E power override 50W
4KB low level reformat - standard capacity
Latest FW/Driver
Paging Support Off
Storage Pool Configured:
Simple (stripe set)
3 Columns
8GB Write Cache
Virtual Hard Drive Configured:
Default 256KB stripe width (have also tried 64KB and it is substantially slower)
4KB allocation unit
64KB NTFS Formatting
Some odd behavior that I have noticed is that if I optimize the storage pool it consistently causes performance to drop in subsequent benchmarking runs until I destroy and recreate it. The same behavior seems to be appearing if I use TRIM support in Windows.
It is a head scratcher. One possibility is that I am experiencing some strange thread management behavior with this manycore CPU and performance is varying by what core the VSL driver is parked on, its proximity to memory, and its proximity to the PCIE lanes connected to the IO Drives. I am wondering if switching out of UMA (die interleaving) and using NUMA affinity commands might help?
This is driving me nuts I think I have tried probably 50 combinations of storage spaces, storage pool, and IODrive settings and am getting all sorts of crazy inconsistent results that aren't helping me to narrow down the best (and stable) combination of settings.
Raw math says that CrystalDiskMark should be reporting something close to 4.5GB/s reads and 4GB/s writes with three drives in an array. I have gotten close to that number on both sides but always at the cost of the other and not consistently.
Any ideas? I am happy to nuke the array a dozen times until I get it right, mostly because it is annoying me and also because it does cause some strange variability when using Davinic Resolve that I would like to go away.
Thanks!
Last I looked at storage spaces, there's no SSD caching in windows desktop OS's.
There's an SSD "tier", but it's only populated via manually setting which files are supposed to be on that tier in the config. StorageSpaces will do a nightly scrub and move any blocks on the HDD level up to the SSDs.
When benching that setup, there's no write-first to SSD, and then spill over to HDD. Writes will hit all disks and your performance will be variable.
It's the main reason I gave up on using storage spaces at home with any flash media.
I was certainly able to get decent all-iodrive performance out of storage spaces, but not mixed with HDD:
It was ~4x slower when I tried to use the ioDrives as cache...showing me that there's actually no cache...
And just for fun, scaling to 4 drives with a 2-way mirror:
There's an SSD "tier", but it's only populated via manually setting which files are supposed to be on that tier in the config. StorageSpaces will do a nightly scrub and move any blocks on the HDD level up to the SSDs.
When benching that setup, there's no write-first to SSD, and then spill over to HDD. Writes will hit all disks and your performance will be variable.
It's the main reason I gave up on using storage spaces at home with any flash media.
I was certainly able to get decent all-iodrive performance out of storage spaces, but not mixed with HDD:
It was ~4x slower when I tried to use the ioDrives as cache...showing me that there's actually no cache...
And just for fun, scaling to 4 drives with a 2-way mirror:
I thought that it is better when DBBlockSize == FileSystemBlockSize == DiskBlockSize. Can you tell why 512b DiskBlockSize is better for 8kb DBBlockSize?
As far as ioDrive performance, it doesn't really matter, there are no block sizes/etc on the ioDrive itself, only in the virtual block device at you OS level, which is all in DRAM.
I'm just saying from the standpoint of putting databases or other apps that require 512b low-level sectors, you'd want to use that.
What databases require 512B sectors though? I don't recall seeing any DBs complain about it. I'm an infrastructure guy, not a DBA, but TTBOMK we've never had anyone spec a system that has to use 512B sector discs.
The only time I've seen MSSQL complain about it was when a DB system with 512B sectors was log shipping to a server with 4kN sectors; the page also mentions SQL will also assume a log drive is 512B if you add a 4kN drive to an already existing 512B system. IIRC a colleague at work also ran into a problem with lync and 4kN drives but as far as I'm aware MSSQL's supported 4kN drives for a while now.
I use them pretty exclusively for editing 4k and 5k video and the 4k formatting significantly increases throughput for my workflow.
I think I have narrowed it down to issues with the VSL driver support for TRIM. I get about 4GB/s read if I run disk benchmarks immediately after running TRIM, however the second run will drop down to 3GB/s and will not budge higher until I manually run TRIM again.
Oddly write speeds (which should be more sensitive to an un-TRIMed disk) are stable at about 3.3GB/s and TRIM has no impact. Any idea what could be causing me to lose 25% of my read throughput in the above scenario? It really doesn't impact my workflow as 3GB/s is enough throughput for even very complex 4K projects with multiple parallel streams being edited, however I am OCD about problems like this and am now really stuck on figuring out what is going on. This doesn't seem to be an issue related to CrystalDiskMark as I see the same behavior in IOSphere while running the benchmark
I am using IBM branded cards that shipped with the latest firmware revision so I haven't bothered to cross-flash them to the Sandisk version. I am using the Sandisk drivers.
V/r,
Tom
Well, I don't recommend mixing/matching driver/firmware versions, that was never a supported config for various inconsistency issues.
As for speeds/etc, I'd need to know your device/stripe/filesystem layout to understand your theoretical max perf.
At ~3GB/sec you have to have at least two ioDrive IIs. Irregardless of low-level formatting, 4k writes shouldn't have a speed difference between 512b low-level and 4k low-level. Write performance with/without TRIM also shouldn't be impacted unless you are writing more than 20% of the device size at a time.
So I need to know what size drives, how many, how they are striped, and what your normal output filesize is/etc.
As for speeds/etc, I'd need to know your device/stripe/filesystem layout to understand your theoretical max perf.
At ~3GB/sec you have to have at least two ioDrive IIs. Irregardless of low-level formatting, 4k writes shouldn't have a speed difference between 512b low-level and 4k low-level. Write performance with/without TRIM also shouldn't be impacted unless you are writing more than 20% of the device size at a time.
So I need to know what size drives, how many, how they are striped, and what your normal output filesize is/etc.
3x IO_Drive 2 MLC 1.2TB in a striped storage space with column count of 3 and a stripe size of 64KB, formatted at 64KB. Using a 1GB file in CrystalDiskMark, typical file size for video runs around 400GB.
I turned on power override for all of them with the wildcard option. Should I try it individually?
Ok so here is some new weirdness. I can set the power override for a single card. I have tried comma separating them but it is only sticking for a single card. As soon as I set it for one card it changes back to the default on the other two.
I figured out how to set all 3 at once with comma separation. All cards now reflect a 74.5W power limit, however it hasn't helped as I am still getting exactly the same performance issue.
Also THANK YOU for your help keeping these cards relevant for broke ass cheapskates like myself!