I am trying to get the most 4k iops I can get out of a raid array for and experimenting with different configurations to try to get a sense of where the sweet spot is in terms of number of drives, performance, and raid config. I was hoping to see more linear performance gains with number of drives on raid 10 but this is not the case as you'll see below.
Initially this post was about trying to optimize the array but I think it is clear that I'll need something other than mdadm to get the most IOPs in raid 10 configurations so to cut to the question first, what should I be looking at to get more IOPs out of raid 10 configurations? Was looking at:
- SW raid like xinnor / raidix - Don't know pricing / if I could even afford them
- HW raid card like Highpoint - ssd7580B
- Graid - Not considering for this setup but maybe in the future
Server has a 64 core 7702P, tons of ram, and up to 16 samsung pm9a3 pcie gen4 nvme drives. I am configuring it with ansible with this script that runs mdadm to create the array, runs a bunch of fio tests, summarizes those tests in a csv and exports it. I have to manually run it for each raid config but at least it will sweep across a number of test settings. Applications are generally based on rocksdb / postgres which is heavy on 4k r/w so only benching that block size.
So I ran bunch of times with different raid configs, consolidated the data, and then put that in gpt et voila,
So it is pretty clear that software raid 10 doesn't scale well.
## Questions
- Is it normal for SW raid to be a bottleneck in these high IOPs situations? Is it normal to need to get huge CPUs to drive this many drives? In the past this is not something I have observed but I was working with much smaller arrays
- Any suggestions to the FIO command to squeeze out more performance / consistently max out drives so that I can better gauge the performance impact on different array setups.
- Any config values I should be setting to maximize performance? I am running tuned right now but thinking other settings I am sure I am missing.
- Why does raid10 not scale? I was expecting much better results
Thank you very much for any feedback. I'll be updating this thread with more data and iterations with any suggestions.
Initially this post was about trying to optimize the array but I think it is clear that I'll need something other than mdadm to get the most IOPs in raid 10 configurations so to cut to the question first, what should I be looking at to get more IOPs out of raid 10 configurations? Was looking at:
- SW raid like xinnor / raidix - Don't know pricing / if I could even afford them
- HW raid card like Highpoint - ssd7580B
- Graid - Not considering for this setup but maybe in the future
Server has a 64 core 7702P, tons of ram, and up to 16 samsung pm9a3 pcie gen4 nvme drives. I am configuring it with ansible with this script that runs mdadm to create the array, runs a bunch of fio tests, summarizes those tests in a csv and exports it. I have to manually run it for each raid config but at least it will sweep across a number of test settings. Applications are generally based on rocksdb / postgres which is heavy on 4k r/w so only benching that block size.
So I ran bunch of times with different raid configs, consolidated the data, and then put that in gpt et voila,
So it is pretty clear that software raid 10 doesn't scale well.
## Questions
- Is it normal for SW raid to be a bottleneck in these high IOPs situations? Is it normal to need to get huge CPUs to drive this many drives? In the past this is not something I have observed but I was working with much smaller arrays
- Any suggestions to the FIO command to squeeze out more performance / consistently max out drives so that I can better gauge the performance impact on different array setups.
- Any config values I should be setting to maximize performance? I am running tuned right now but thinking other settings I am sure I am missing.
- Why does raid10 not scale? I was expecting much better results
Thank you very much for any feedback. I'll be updating this thread with more data and iterations with any suggestions.
Attachments
-
98.1 KB Views: 0
Last edited: