whitey's FreeNAS ZFS ZIL testing

[Benten93]

how long did the vmotion say it took? per vecenter task start and end times

Right smack dab in the middle of the sVMOtion would be better/ideal or a quick sampling on average but that could just be a 'spot-check' every 5-10 mins as the sVMotion progresses, if the numbers align close enough then I say you have one helluva SLOG device. Good testing methodology btw, happy to see some serious stress @ 800+ GB of data moving between src/dest stg/platforms.

EDIT: 6Gbps of sVMotion traffic is no joke if that really is where it runs 'steady-state' through the duration of the operation.

@marcoi
Begin was: 18:26:03
End was: 18:53:42
Size of the VM is 832GB

@whitey

I will rerun the test tomorrow if i find the time for it!
Then i will make a timer for myself to get some checks on the go.

 

[T_Minus]

I'm curious if you're maxing out that RaidZ2 pool with that P4800x as SLOG?

 

[Benten93]

I'm curious if you're maxing out that RaidZ2 pool with that P4800x as SLOG?

From the reports of FreeNAS Activity of the HDDs had an average of about 80%.

EDIT: Some had some longer peaks at 100%. So i guess the WD RED 5TB are mostly maxed out
Much better as with the P3700 before!

EDIT2: @whitey the ESXi Host and the FreeNAS Box where directly connected without switch in between. Just for your info as i saw your signature. And as far as my tests resulted in the hint to disable jumbo frames.

 

[whitey]

Would be VERY happy to see stats for sVMotion in the other direction (SLOG P4800X backed pool back to the P3700) and if those numbers align or just a tad lower?

 

[whitey]

So roughly 30-31GB of data transfer per minute...(832GB divided by 27 mins)

1.25GB/s ='s 10Gbps (max theoretical right 10 divided by 8)...1.25 x's 60 secs ='s 75GB per min (again max 10GbE).

If I were a betting man I'd say you saw averages right abt 550-600MBps on 'sustained' throughput/pool I/O stats if my math doesn't suck...which it very well 'may' :-D

(550 MBps) .55 x's 60 secs = 33GB per min x's 27 mins ='s 891GB transfered
(600MBps) .60 x's 60 secs = 36GB per min x's 27 mins ='s 972GB transfered

EDIT: Even so if you just look at SLOG performance the P4800X trounces and is KING w/out a shadow of a doubt...for now :-D

Just for quick reference had my breakouts for devices as follows:
hussl - 125MBps
ZeusRAM - 150MBps
husmm - 200MBps
P3700 - 300MBps

You're P3700 is native to vSphere right (VMFS formatted), not vt-D passthru to FreeNAS, wonder what you would see there if you had the time (not expecting just casually inquiring) :-D

 

[T_Minus]

So maxing out the WD RED's and the Network per @whitey post.

Curious to see what that thing can do with SSD POOL & 40GbE network

Going to be real interesting to see how the new 900P compares too.

 

[whitey]

Eh hem...yeah send it my way LOL

EDIT: I'm REALLY starting to wonder if overprovisioning has more of a positive effect than I want to believe, I didn't do that...maybe that or there is more overhead w/ vt-D/virtualization/FreeNAS/ZFS sucking in my scenaro than I 'want' to believe.

Care to share the overprovisioning method used/cmds? I know I have seen one from thomas-krenn. Same one may be? I WOULD be willing to re-test w/ over-prov in the mix.

 

[DaveBC]

Hi @whitey, did you ever test the ZeusRAM with faster backing pool? I'm curious if you'd get better performance with striped mirrors or an SSD pool. Or was there some reason you thought the ZeusRAM was the bottleneck?

 

[whitey]

Nope never did test that but I can, only have 4 slots free to test and limited disks unless I get creative.

 

[azev]

Just did my own testing and I am getting absolute crap write performance with 6x 800GB sandisk ascend ssd configured in raid0 with 1x 400GB lightning II (12gb slc sas) drive as slog connected to lsi-9300 controller.
Testing was done from a vm running on an ESXI host with dual 10Gb iscsi uplinks configured with multipath.

When the pool is set with sync=standard none of my write hitting the slog, but performance was damn good.
Once I enable sync=always it slows to unbearable crawl. I was waiting for diskmark bench with 8gb dataset for a while now and it still not finished. Using iometer sequential 32Mb data I am only getting 1.5Mbps.

I was expecting to get at least a few hundred megs throughput but I am barely getting 10Mb/s, its super weird.
I am running the latest freenas11 stable.

 

[Rand__]

Did you check whether the lightning2 was tested as slog anywhere? SLC is not the only aspect when it comes to making a good slog...

 

[MiniKnight]

@azev you need a slog that's faster than your pool. You've got the opposite so your slog will slow you down.

 

[T_Minus]

@azev you need a slog that's faster than your pool. You've got the opposite so your slog will slow you down.

I would agree with this. 6+ SSD pool of mirrors is going to require a very fast SSD or NVME to outperform on-pool ZIL.

Try with sync=always and no SLOG device.

 

[azev]

I would agree with this. 6+ SSD pool of mirrors is going to require a very fast SSD or NVME to outperform on-pool ZIL.

Try with sync=always and no SLOG device.

The lightning II ultra SSD is the fastest device I have in my arsenal; and according to the spec sheet it should be pretty dang fast.

I understand the argument of needing super fast device for slog, however, I am curious why it slowed down my pool to a complete crawl ?? I was expecting that my pool would perform as fast as the SLOG ssd can handle the writes in the few hundred megs.

I remember testing the SLOG ssd and it was really fast (I tested this on a test machine running windows)

Anyway tonight I will test enabling sync=always without a slog and see.

 

[T_Minus]

The lightning II ultra SSD is the fastest device I have in my arsenal; and according to the spec sheet it should be pretty dang fast.

I understand the argument of needing super fast device for slog, however, I am curious why it slowed down my pool to a complete crawl ?? I was expecting that my pool would perform as fast as the SLOG ssd can handle the writes in the few hundred megs.

I remember testing the SLOG ssd and it was really fast (I tested this on a test machine running windows)

Anyway tonight I will test enabling sync=always without a slog and see.

I don't know why it drops to <100MB/s

Maybe it's a latency problem with acknowledging the writes?

I would try with no SLOG device and sync=always to see what you get, then re-add the Ultra II and try again.

@gea might have encountered this issue / drive specific problem ?

Maybe the drive doesn't have a write cache at all? (Seems doubtful based on perf. rating, but maybe it uses SLC only???)

 

[Rand__]

Slog speed is not about pure transfer speed, its about latency.
Thats why nvme shines and S3700's are the next best thing (next to Zeus)

If it was pure speed (ignoring plp etc) then a PM951 would outrun most ssds, but thats actually a crappy slog device (quite slow, and o/c no plp)


Edit:
Is your drive the SANDISK LIGHTNING ULTRA GEN II ? I have some of those (200gb) and could run a comparison if you'd like to verify your values.Not sure how soon I find the time but if it helps...

 

[T_Minus]

512? 4k? Is it properly aligned or whatever the sg tool does?

ashift=?? in ZFS for your configuration?

 

[whitey]

@azev n' crew.

I just spun up a very small sanity check on my spare sc113 box, vSphere 6.5, FreeNAS11U4 AIO, 2vcpu, 12GB memory, only had a 100gb hussl and 240gb s3500 laying arnd to play w/, this is w/ s3500 as capacity, hussl as SLOG, iSCSI zvol/VMFS6, 'zfs set sync=always', w/ a 40GB VM sVMotioning to it from my src 4 disk raidz husmm w/ an additional husmm SLOG device.

Nothing fancy here so something must be awry w/ your results. Ohh even worse, I am using the onboard LSI 1068 sas1 HBA built onto the X8DTL-3F ancient board in that test bench box in vt-D/passthru to FN11U4 and still getting 'ok-ish' performance.

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            48.7G   171G      0  1.99K  44.0K   200M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  48.7G   171G      0    585  44.0K  52.3M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   153M  92.9G      0  1.42K      0   148M
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            48.7G   171G      1  1.94K  46.0K   195M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  48.7G   171G      1    630  46.0K  58.5M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   306M  92.7G      0  1.32K      0   137M
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            48.7G   171G      1  2.18K  72.0K   215M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  48.7G   171G      1    781  72.0K  68.2M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   155M  92.8G      0  1.42K      0   147M
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            48.7G   171G      0  1.94K  43.3K   191M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  48.7G   171G      0    538  43.3K  44.0M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   152M  92.9G      0  1.42K      0   147M
--------------------------------------  -----  -----  -----  -----  -----  -----

For prosperity here's a list of iSCSI VAAI support on FreeNAS 11U4 for the zvol iSCSI block device.

[root@esxi6a:~] esxcli storage core device vaai status get
naa.6589cfc0000004f1a875e2fb95f1c82c
VAAI Plugin Name:
ATS Status: supported
Clone Status: supported
Zero Status: supported
Delete Status: supported

And here is NFS performance to that same setup:

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            51.5G   168G      0  1.95K      0   162M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  51.5G   168G      0    672      0  74.4M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   252M  92.8G      0  1.29K      0  87.8M
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            51.7G   168G      0  1.99K      0   173M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  51.7G   168G      0    746      0  87.6M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   252M  92.8G      0  1.26K      0  85.9M
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            51.9G   168G      0  1.96K      0   166M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  51.9G   168G      0    691      0  79.0M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   252M  92.8G      0  1.29K      0  87.4M
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
s3500-hybrid                            52.0G   168G      0  2.04K      0   176M
  gptid/253f7fc2-c100-11e7-9923-000c295487c1  52.0G   168G      0    775      0  88.6M
logs                                        -      -      -      -      -      -
  gptid/258dc7ef-c100-11e7-9923-000c295487c1   252M  92.8G      0  1.29K      0  87.6M
--------------------------------------  -----  -----  -----  -----  -----  -----

 

[azev]

Got a late night tonight, I will do some test tomorrow morning and post back the result.
@whitey that test result is pretty amazing compared to what I got on mine with 6x ssd, it does not make any sense at all.
You're probably right something is completely wrong with my current setup.

 

[azev]

Is your drive the SANDISK LIGHTNING ULTRA GEN II ? I have some of those (200gb) and could run a comparison if you'd like to verify your values.Not sure how soon I find the time but if it helps...

you are correct, that is the drive I am using as slog. Mine is 400Gb and partitioned to only 16Gb for slog purpose.