InfiniBand for ZFS replication in Proxmox HA


New Member
Nov 9, 2020
Hi Everyone,

I just wanted someone to have a look at my proposal and make sure it makes sense. This is for a homelab environment, nothing production.

My current setup is 1 Quanta D51B and 1 Fujitsu RX300 S7, the Quanta running ESXI and RX300 running Proxmox. The ESXI is so suboptimal, as I used a TrueNAS VM to pass iSCSI back to the host for ZFS redundancy, but then run another TrueNAS to actually act as my nas from inside that iSCSI.

I have a 2(.5) node Proxmox HA setup in mind. 1 Quanta D51B, 1 Fujitsu RX300 S7 and a Raspberry Pi acting as a qdevice for HA. Both servers will have a ZFS pool in RAIDZ2, which replicate between the two servers for key services every 15 minutes. I would like to connect the two servers directly via InfiniBand, seen as how much cheaper they are for higher throughput. This will carry replication traffic and HA traffic. The builtin NICs would be used to for all other traffic.

Some notes about my experience when replying:
  • I have no professional experience, google is my friend.
  • I have never used InfiniBand before, so any tips would be advised.
  • I have only started using Proxmox a few days ago, still new to it.
  • I've used ZFS for abit, but never replication.

So I have some questions
  1. From my understanding I should be able to up IPoIB to make infiniband work. Very little experience and only google-fu to go by. Will this work?
  2. How exactly will ZFS replication work in this scenario, will it sync (i.e the newest version of data takes precedent). I'm thinking of a scenario where there's a replication task from Server 1 (S1) to S2. S1 goes down and S2 takes over operation of VMs. When S1 comes back up, will the changes from S2 be synced back?
Now you may ask, why don't I just add a third node and go down the ceph route. Well I live in a small apartment and really don't have the space, or the need for another server drawing power from the wall. Can't blame me for trying to save a penny or two.

Any advice or input would be greatly appreciated.