I recently got 10x HP 800GB HSSC0800S5xnNMLC SSDs, as I understand these are the same as HGST HUSMM8080ASS204, but perhaps with HP firmware? I'm pairing them with 2 Adaptec 1000-8i HBAs in a Supermicro 216 chassis with 216A backplane (direct attached). The Adaptec card is showing these SSDs connecting at 12Gbps speeds. They are in a X8DT6-F machine, so not PCI-E 3.0, but 2.0 rather, even so, with PCI-E 2.0 x8, the theoretical bandwidth is about 4GB/sec, right? That's the hardware setup; OS is CentOS7 Linux; just a test setup.
If i pick any single SSD and just create a file system on it, a simple 'dd' test for sequential write/read is showing me about 800MB/sec write, and 500MB/sec read? So, first thing, I'm immediately surprised the reads are slower than the writes? And this seems to be far below the HGST specs? At least the 800MB/sec confirms I am in the 12Gbps zone.
So, next I use ZFS and put the 10x SSDs in raidz2 just to see what i get... 1.2GB/sec writes, 1.1GB/sec reads. 2 things still bother me here... why are reads still slower than writes? And with 10x SSDs that each can do at least 800MB/sec writes, why is the raidz2 aggregate only 1.2GB/sec? I'm not expect 8x 800MB/sec writes, since I know the PCI-E 2.0 x8 limits me to 4GB/sec, but i'm not even close? Earlier, before the test in this paragraph, i think i was bottle-necking at the CPUs with a pair of E5506 Xeons, as I was getting 650MB/sec writes, and 550MB/sec reads in the same raidz2 set; then I swapped in a pair of X5670 CPUs and that got me the 1.2GB/sec writes, 1.1GB/sec reads. obviously huge difference between the pair of E5506 and X5670; is the raidz2 parity intense enough to require more compute i/o than the E5506 can handle for 10x 12Gbps SSDs?
so, thinking "let me take the parity thing out of the equation here..", I put all 10x SSDs in a ZFS vdev stripe pool. Same 'dd' test got me 1.8GB/sec writes and 1.8GB/sec reads. So, about 50% faster than raidz2 on the same set of 10x SSDs. that huge of a difference makes me wonder if the raidz2 parity is really that compute or i/o intense? and yet, even without parity, I'm not seeing even half of the 4GB/sec PCI-E 2.0 x8 limit?
and before anyone asks, these SSDs report 4K sectors, and ZoL correctly used ashift=12.
so, question is, why am I not seeing bigger numbers? any hints or suggestions?
If i pick any single SSD and just create a file system on it, a simple 'dd' test for sequential write/read is showing me about 800MB/sec write, and 500MB/sec read? So, first thing, I'm immediately surprised the reads are slower than the writes? And this seems to be far below the HGST specs? At least the 800MB/sec confirms I am in the 12Gbps zone.
So, next I use ZFS and put the 10x SSDs in raidz2 just to see what i get... 1.2GB/sec writes, 1.1GB/sec reads. 2 things still bother me here... why are reads still slower than writes? And with 10x SSDs that each can do at least 800MB/sec writes, why is the raidz2 aggregate only 1.2GB/sec? I'm not expect 8x 800MB/sec writes, since I know the PCI-E 2.0 x8 limits me to 4GB/sec, but i'm not even close? Earlier, before the test in this paragraph, i think i was bottle-necking at the CPUs with a pair of E5506 Xeons, as I was getting 650MB/sec writes, and 550MB/sec reads in the same raidz2 set; then I swapped in a pair of X5670 CPUs and that got me the 1.2GB/sec writes, 1.1GB/sec reads. obviously huge difference between the pair of E5506 and X5670; is the raidz2 parity intense enough to require more compute i/o than the E5506 can handle for 10x 12Gbps SSDs?
so, thinking "let me take the parity thing out of the equation here..", I put all 10x SSDs in a ZFS vdev stripe pool. Same 'dd' test got me 1.8GB/sec writes and 1.8GB/sec reads. So, about 50% faster than raidz2 on the same set of 10x SSDs. that huge of a difference makes me wonder if the raidz2 parity is really that compute or i/o intense? and yet, even without parity, I'm not seeing even half of the 4GB/sec PCI-E 2.0 x8 limit?
and before anyone asks, these SSDs report 4K sectors, and ZoL correctly used ashift=12.
so, question is, why am I not seeing bigger numbers? any hints or suggestions?