Seq write speeds not as expected

[EpicFailGuy]

Hello everyone,

Long time lurker first time poster. I have a small cluster of Proxmox Mini PCs runing a test lab and some simple self hosted services. I built a OMV out of an old optiplex 3070 and an Adaptec ASR-71605 HBA in IT mode. The Motherboard only has one PCI slot that I'm using for the HBA, and I connected a couple of external 2.5 Gbe adapters on the USB 5.0 to use my with 2.5G Switch in LCA.

I have 4 SAS Seagate EXO (7E8) in raid 10 and Also 4x Samsung 465 GB 870 Evo SSDs that I configured with BCache in front of the HDDs

It all seems to be working, but when I run a large backup with Proxmox I can see that the write speed is what I would expect the HDDs to perform individually (Around 125 on a good day)

I've torn down the setup and redone it a few time. If I setup the array with just the SSDs I see what I would expect (Speed capped at a single 2.5 Gigabit)

But when I throw the Bcache in front of the Z pool I see no significant performance advantage, in spite of the SSDs reporting usage.

Any troubleshooting tips? Could I be saturating the HBA?

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 5.5T 0 disk
└─bcache1 251:128 0 5.5T 0 disk
sdb 8:16 1 5.5T 0 disk
└─bcache2 251:256 0 5.5T 0 disk
sdc 8:32 1 5.5T 0 disk
└─bcache3 251:384 0 5.5T 0 disk
sdd 8:48 1 5.5T 0 disk
└─bcache0 251:0 0 5.5T 0 disk
sde 8:64 1 465.8G 0 disk
sdf 8:80 1 465.8G 0 disk
sdg 8:96 1 465.8G 0 disk
└─bcache2 251:256 0 5.5T 0 disk
sdh 8:112 1 465.8G 0 disk
└─bcache0 251:0 0 5.5T 0 disk
sdi 8:128 1 465.8G 0 disk
└─bcache1 251:128 0 5.5T 0 disk
sdj 8:144 1 476.9G 0 disk
└─bcache3 251:384 0 5.5T 0 disk
sdk 8:160 0 238.5G 0 disk
├─sdk1 8:161 0 976M 0 part /boot/efi
├─sdk2 8:162 0 225.2G 0 part /var/lib/containers/storage/overlay
│ /
└─sdk3 8:163 0 12.3G 0 part

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 5.5T 0 disk
└─bcache1 251:128 0 5.5T 0 disk
sdb 8:16 1 5.5T 0 disk
└─bcache2 251:256 0 5.5T 0 disk
sdc 8:32 1 5.5T 0 disk
└─bcache3 251:384 0 5.5T 0 disk
sdd 8:48 1 5.5T 0 disk
└─bcache0 251:0 0 5.5T 0 disk
sde 8:64 1 465.8G 0 disk
sdf 8:80 1 465.8G 0 disk
sdg 8:96 1 465.8G 0 disk
└─bcache2 251:256 0 5.5T 0 disk
sdh 8:112 1 465.8G 0 disk
└─bcache0 251:0 0 5.5T 0 disk
sdi 8:128 1 465.8G 0 disk
└─bcache1 251:128 0 5.5T 0 disk
sdj 8:144 1 476.9G 0 disk
└─bcache3 251:384 0 5.5T 0 disk
sdk 8:160 0 238.5G 0 disk
├─sdk1 8:161 0 976M 0 part /boot/efi
├─sdk2 8:162 0 225.2G 0 part /var/lib/containers/storage/overlay
│ /
└─sdk3 8:163 0 12.3G 0 part

root@olympus:~# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 08)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 08)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630]
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 RAID bus controller: Intel Corporation SATA Controller [RAID mode] (rev 10)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #5 (rev f0)
00:1f.0 ISA bridge: Intel Corporation H370 Chipset LPC/eSPI Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 RAID bus controller: Adaptec Series 7 6G SAS/PCIe 3 (rev 01)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15)

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
Bus 002 Device 003: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

 

[EpicFailGuy]

Pool status (zpool status -v):



  pool: DATA

 state: ONLINE

config:



    NAME         STATE     READ WRITE CKSUM

    DATA         ONLINE       0     0     0

      mirror-0   ONLINE       0     0     0

        bcache1  ONLINE       0     0     0

        bcache2  ONLINE       0     0     0

      mirror-1   ONLINE       0     0     0

        bcache3  ONLINE       0     0     0

        bcache0  ONLINE       0     0     0



errors: No known data errors

History for 'DATA':

2026-05-24.13:01:42 zpool create DATA mirror bcache0 bcache1 mirror bcache2 bcache3

2026-05-24.13:03:39 zfs create -p DATA/Backup

2026-05-24.13:03:39 zfs set omvzfsplugin:uuid=4b160c2c-a10b-43cb-96d2-816a64b8a958 DATA

2026-05-24.13:03:39 zfs set omvzfsplugin:uuid=d94b6a31-e87c-45b7-8383-9eff423b7076 DATA/Backup

2026-05-24.13:03:44 zfs set omvzfsplugin:uuid=4b160c2c-a10b-43cb-96d2-816a64b8a958 DATA

2026-05-24.13:03:44 zfs set omvzfsplugin:uuid=d94b6a31-e87c-45b7-8383-9eff423b7076 DATA/Backup

2026-05-24.13:03:46 zfs set omvzfsplugin:uuid=4b160c2c-a10b-43cb-96d2-816a64b8a958 DATA

2026-05-24.13:03:46 zfs set omvzfsplugin:uuid=d94b6a31-e87c-45b7-8383-9eff423b7076 DATA/Backup

2026-05-24.13:04:01 zfs create -p DATA/Nextcloud

2026-05-24.13:04:04 zfs set omvzfsplugin:uuid=4b160c2c-a10b-43cb-96d2-816a64b8a958 DATA

2026-05-24.13:04:04 zfs set omvzfsplugin:uuid=d94b6a31-e87c-45b7-8383-9eff423b7076 DATA/Backup

2026-05-24.13:04:04 zfs set omvzfsplugin:uuid=f36649f2-a1b7-49f2-9bde-a33248c8d8ca DATA/Nextcloud

 

[i386]

Any reason why you use bcache in combination with zfs?

Any troubleshooting tips?

Did you try a longer write benchmark with a ssd only volume? Older samsung consumer ssd performance used to slow down extremely under sustained reads/writes

Could I be saturating the HBA?

No. It has 16x (or 4x multilane) sas ports at 6GBIt/s (total bandwidth = 96GBit/s), which is faster than the pcie 3.0 x8 connection can provide ~ 7.8GByte/s).
In a raid 10/stripe+mirros that would still be ~3.9GByte/s net bandwidth.

 

[EpicFailGuy]

Any reason why you use bcache in combination with zfs?

I chose this as opposed to an slod vdeb because it would have to be in some form of fault tolerant raid, and I understand that's not the best idea with SSDs because of write amplification with ZFS. BCache doesn't seem to have the same problem since is not scrubbing or performing any checks, it simply writes whatever to the SSD first and then it slowly drains it into the HDD (or that's my understanding) also because the BCache devices themselves are in RAID10 it should help improve write speeds, no?

Did you try a longer write benchmark with a ssd only volume? Older samsung consumer ssd performance used to slow down extremely under sustained reads/writes

I have a couple of spare SSDs in there now so I threw up a test share and did 4 passes of a 4GB test in "real world performance" profile on crystal mark. It's about what I expect (2.5GBE minus some overhead)

 

[nexox]

It's been a long time since I messed with bcache, but it has several modes, I don't see your config in this thread, is it possible you have it set up in one of the read cache modes where it immediately writes to backing storage?

 

[EpicFailGuy]

It's been a long time since I messed with bcache, but it has several modes, I don't see your config in this thread, is it possible you have it set up in one of the read cache modes where it immediately writes to backing storage?

Good point, I set it up on the GUI in OMV so I'm not exactly sure.

These are all the options I can find:

sb.magic        ok
sb.first_sector        8 [match]
sb.csum            50DADB411A1EFBB8 [match]
sb.version        3 [cache device]

dev.label        (empty)
dev.uuid        629bdf69-ba2b-4f08-8d9a-ea7d1672855c
dev.sectors_per_block    1
dev.sectors_per_bucket    1024
dev.cache.first_sector    1024
dev.cache.cache_sectors    976772096
dev.cache.total_sectors    976773120
dev.cache.ordered    yes
dev.cache.discard    no
dev.cache.pos        0
dev.cache.replacement    0 [lru]

cset.uuid        d7a5f119-e635-4a42-8902-155b4a0ea357

 

[nexox]

These are all the options I can find:

Again it's been a long time but that looks like maybe the info for a single disk in the cache, but not the cache overall, I expect to see something like "write-through" or "write-back".

 

[EpicFailGuy]

Again it's been a long time but that looks like maybe the info for a single disk in the cache, but not the cache overall, I expect to see something like "write-through" or "write-back".

Bingo



I have now changed it to write back, will re-test and report.

Thank you much!

 

[nexox]

Those SSDs don't have power loss protection, so think real hard about whether you want to use them as a write cache, a single power outage or bumping the switch on the power strip could corrupt the entire filesystem. You may be able to find some SAS SSDs in that capacity range with PLP fairly cheaply to make write caching safer.

 

[EpicFailGuy]

Those SSDs don't have power loss protection, so think real hard about whether you want to use them as a write cache, a single power outage or bumping the switch on the power strip could corrupt the entire filesystem. You may be able to find some SAS SSDs in that capacity range with PLP fairly cheaply to make write caching safer.

Thanks for the concern ... the whole setup is on a decent UPS and the HBA itself has a battery. More than anything this is a proof of concept using whatever I have on hand before I start a more .... permanent build.

I tried first with Unraid expecting that to be more performant, then learnt about the issues with single disk redundancy and the risks of rebuilding on a live array, plus found their model of parity less than performant for what I need. Then went to Truenas Core ... found some compatibility issues due to the BSD base, tried TrueNAS scale and found the lack of modularity and plugins inadequate for my purposes.

Now testing the "ideal build" with OMV, if this fails I will try RockStor. Also heard good things about just using a base Debian system with a cockpit GUI and a combination of mergeFS + SNAP raid to achieve the same benefits as Unraid without any of the downsides.

 

[nexox]

whole setup is on a decent UPS and the HBU itself has a battery

UPSes, even decent ones, tend to cause a fair number of outages, particularly if you don't stay ahead on battery replacements. The HBA battery would only keep its own cache powered, the SSDs would still lose data. Small SAS SSDs really are pretty cheap, and they'll probably get you better sustained write performance than 870 Evos, plus SAS supports concurrent read and write operations which may be a nice bonus.

I don't know if any of the GUI platforms support it, and there are difficulties since it got kicked out of the mainline kernel, but bcachefs might get close to doing what you want. I keep meaning to try it, but I'm constantly distracted by other projects and never get around to finishing that server build. They got parity RAID implemented recently and it should be reasonably fast because you can set your write cache SSDs to a mirror and then it converts to parity as it writes out to the backing devices (which also eliminates the write hole issue in the btrfs parity implementation.)

 

[EpicFailGuy]

I'll add BcacheFS to the list, thank you for the rec. that sounds like exactly what I'm looking for!

I'm even more puzzled now. When I run a local test by mounting a CIFS share on my Windows desktop I get slightly better thought puts, but again capped at the bandwidth of a single interface .... but when I run the same test with fio from within one of the lxds that has the NFS mounted ... I get 35 Mb/s average .....

I'm losing my mind. I'm going to tear the whole thing apart and re-test with too separate pools, one in all flash and one in all spinning rust.

Thanks again for all the help

 

[nexox]

I'm even more puzzled now. When I run a local test by mounting a CIFS share on my Windows desktop I get slightly better thought puts, but again capped at the bandwidth of a single interface .... but when I run the same test with fio from within one of the lxds that has the NFS mounted ... I get 35 Mb/s average .....

That could be a sync vs async issue, copying files from Windows is likely async, benchmarks and possibly NFS mounts often default to sync. SSDs without PLP lean heavily on their volatile write caches to boost performance, sync forces them not to do that. I have seen a drive go from writing 3MB/s to maxing out my 10Gb link just switching NFS to async.