Slow write sync performance on Intel S3700

Discussion in 'Hard Drives and Solid State Drives' started by lni, Jan 25, 2019.

  1. lni

    lni Member

    Joined:
    Aug 20, 2017
    Messages:
    34
    Likes Received:
    10
    I grabbed an Intel S3700 to use for SLOG kind of workload. I understand that write sync performance (fsync() latency) is the most important factor here. Did a quick pg_test_fsync run and found that fsync() latency is almost 8ms each write sync.


    This is very different from what I heard previously on this forum, e.g. in the thread below, BackupProphet had 50x better latency on his S3710.

    https://forums.servethehome.com/index.php?threads/inexpensive-ssd-for-slog-device.13889/

    Any idea what might went wrong? Thanks

    I am running on Ubuntu 16.04 with 4.15 kernel, my mount options are

    smartctl -a output attached below

     
    #1
    Last edited: Jan 25, 2019
  2. BackupProphet

    BackupProphet Well-Known Member

    Joined:
    Jul 2, 2014
    Messages:
    785
    Likes Received:
    278
    Could you try without the max_batch_time=0,data=ordered options

    The S3700 has similar sync performance to the S3710, though I fast single-threaded cpu also helps.

    But in your case, the sync performance is ridiculously low. I could also be a firmware issue.
     
    #2
  3. lni

    lni Member

    Joined:
    Aug 20, 2017
    Messages:
    34
    Likes Received:
    10
    @BackupProphet

    Thanks a lot for your reply! I think I've found the issue - it was a bug in Linux Kernel 4.15.

    On Ubuntu 16.04 with its 4.4 kernel, everything seems to be fine:

    uname -a output:

    Once on Ubuntu 16.04 with its hwe kernel 4.15, fsync becomes stupidly slow:


    uname -a here:

    This kernel issue has actually been reported, discussed and addressed:
    Bug #1785020 “fsync is slow on later kernels with ext4 filesystm...” : Bugs : linux package : Ubuntu

    4.15 is not something new to me, I've been running it for ages, but all with NVME disks (Optane and P3700), they seem to be unaffected. Optane 900P results look normal on the 4.15 Kernel -

    Anyway, this is the whole drama I got tonight, hope it will be useful for other people who might get the same issue.
     
    #3
Similar Threads: Slow write
Forum Title Date
Hard Drives and Solid State Drives Ceph very slow writes Apr 25, 2017
Hard Drives and Solid State Drives Samsung 850 Pro, software RAID-10, 400 MB/s dd slow sequential writes. Mar 11, 2016
Hard Drives and Solid State Drives Slow Write Speeds Crucial MX100 512GB Raid 0 Jul 19, 2014
Hard Drives and Solid State Drives SSD fast in ESXi and Windows slow in Linux Mar 22, 2019
Hard Drives and Solid State Drives 8 TB drives slow to a crawl and then show up as 1.2 TB? Feb 16, 2018

Share This Page