Quick summary of my setup, I have an ESXi (currently 5.5) all-in-one on a Dell R510 w/ H200 and 8x WD REDs using pcie passthrough and 32GB RAM to FreeNAS. I understand vmware causes sync writes over NFS and it's really slow.
I have a Fusion IO Duo (which is really 2 x 320GB on a single PCIe card) that I'm using as a local datastore for my "critical" VMs to bootstrap the looped back NFS mount where initially I intended most everything else to be. I had experimented with a 4GB vmdk as a SLOG to improve writes and I go from ~9MB/s to 80MB/s. I've been on a quest to figure out why it's still so slow when I can get something like 300MB/s disabling sync. (I'm not at home where I took notes on the numbers, so may have to update but it's in the ballpark).
I was really hesitant to even try to do this at first after reading and understanding about direct disk access, however, I'm seeing more or this used for testing and thinking it's actually sort-of 'ok' for this purpose, if I can get some benefit. I enabled the vm setting so that it passes through the unique id/serial for tracking. I had somewhat given up on my quest until I got back into doing better analytics and more testing. In particular I was also messing with a purely virtualized omv and trying out the I/O Analyzer vm to benchmark my Fusion IO directly compared to a new Samsung PM953 NVMe. I got a second PCIe based storage so that I could configure another backup R510 similarly. I don't really have room for 2 cards in one machine, nor would I be able to RAID them, and I know the Fusion IO doesn't have drivers in FreeNAS if I passed it, or one side through.
Anyways, let me skip to my suspicions. I had noticed that the vmdk shows up in the boot messages as a 150MB/s interface, and ironically I seem to be limited to about what I'd get with a single HDD. Then I realized I'm using the LSI (Parallel) SCSI controller. The paravirtual interface (which I noticed when comparing to the omv and I/O analyzer vms) doesn't seem to be supported for FreeBSD, but I tried switching to the LSI SAS controller, and it shows up as 300MB/s interface during boot, but still not really any faster. I've also tried adding an additional 4GB vmdk as slog to slice between the 2, and I can get a whopping 90MB/s (10MB/s more).
I thought if I switched to omv I could use the paravirtual controller and get more performance, but I see other tests (like gea's napp-it optane performance tests) showing this works. Is there something else here that I'm missing?
I'm also considering something like an HGST SAS SSD in one of the internal bays directly on the passed through backplane. I'm just struggling with the need because I already have fast storage and/or could probably just live with sync disabled for my purposes.
Based on my latest I/O analyzer I know I can get great performance out of either PCIe storage directly. I even tried doubling up to stress it further, i.e. see how the two sides of the Fusion IO Duo would work if I striped across them and in general how much contention there would be if other VMs run on it directly while it's also being used for all writes as a SLOG.
I'm really trying to get this resolved so I can move on, but it's really fundamental to (re)architecting/optimizing my setup. Now I'm thinking about just using the PCIe based storage directly (I think I'd switch my primary to the NVMe because it's faster, bigger continuous space, and lower power) for VMs and the HD array for large bulk data. Beyond that I'm struggling with how/where to store the "persistant data" in my VMs and containers, like Plex metadata, influxdb for metrics/home automation, centralized logging, etc. because backing up the VMs (ghettovcb) are now full copies not just incremental so as it gets bigger will become more painful. I was kind of hoping to use ZFS snapshots and send/receive to optimally backup on and off site. Tell me for home use, this is fine and the single (high quality) SSDs are generally reliable enough themselves not to over complicate because I'm worried about redundancy.
I have a Fusion IO Duo (which is really 2 x 320GB on a single PCIe card) that I'm using as a local datastore for my "critical" VMs to bootstrap the looped back NFS mount where initially I intended most everything else to be. I had experimented with a 4GB vmdk as a SLOG to improve writes and I go from ~9MB/s to 80MB/s. I've been on a quest to figure out why it's still so slow when I can get something like 300MB/s disabling sync. (I'm not at home where I took notes on the numbers, so may have to update but it's in the ballpark).
I was really hesitant to even try to do this at first after reading and understanding about direct disk access, however, I'm seeing more or this used for testing and thinking it's actually sort-of 'ok' for this purpose, if I can get some benefit. I enabled the vm setting so that it passes through the unique id/serial for tracking. I had somewhat given up on my quest until I got back into doing better analytics and more testing. In particular I was also messing with a purely virtualized omv and trying out the I/O Analyzer vm to benchmark my Fusion IO directly compared to a new Samsung PM953 NVMe. I got a second PCIe based storage so that I could configure another backup R510 similarly. I don't really have room for 2 cards in one machine, nor would I be able to RAID them, and I know the Fusion IO doesn't have drivers in FreeNAS if I passed it, or one side through.
Anyways, let me skip to my suspicions. I had noticed that the vmdk shows up in the boot messages as a 150MB/s interface, and ironically I seem to be limited to about what I'd get with a single HDD. Then I realized I'm using the LSI (Parallel) SCSI controller. The paravirtual interface (which I noticed when comparing to the omv and I/O analyzer vms) doesn't seem to be supported for FreeBSD, but I tried switching to the LSI SAS controller, and it shows up as 300MB/s interface during boot, but still not really any faster. I've also tried adding an additional 4GB vmdk as slog to slice between the 2, and I can get a whopping 90MB/s (10MB/s more).
I thought if I switched to omv I could use the paravirtual controller and get more performance, but I see other tests (like gea's napp-it optane performance tests) showing this works. Is there something else here that I'm missing?
I'm also considering something like an HGST SAS SSD in one of the internal bays directly on the passed through backplane. I'm just struggling with the need because I already have fast storage and/or could probably just live with sync disabled for my purposes.
Based on my latest I/O analyzer I know I can get great performance out of either PCIe storage directly. I even tried doubling up to stress it further, i.e. see how the two sides of the Fusion IO Duo would work if I striped across them and in general how much contention there would be if other VMs run on it directly while it's also being used for all writes as a SLOG.
I'm really trying to get this resolved so I can move on, but it's really fundamental to (re)architecting/optimizing my setup. Now I'm thinking about just using the PCIe based storage directly (I think I'd switch my primary to the NVMe because it's faster, bigger continuous space, and lower power) for VMs and the HD array for large bulk data. Beyond that I'm struggling with how/where to store the "persistant data" in my VMs and containers, like Plex metadata, influxdb for metrics/home automation, centralized logging, etc. because backing up the VMs (ghettovcb) are now full copies not just incremental so as it gets bigger will become more painful. I was kind of hoping to use ZFS snapshots and send/receive to optimally backup on and off site. Tell me for home use, this is fine and the single (high quality) SSDs are generally reliable enough themselves not to over complicate because I'm worried about redundancy.