Okay, I tried first on /r/storage and they kinda turned their noses up at me...this crowd seems a little more adventurous.
Long story short I had a project that due to budget constraints needed a home-grown, dual-node(for redundancy) FC storage "SAN". Performance didn't have to be stellar. Tying it into our existing Ceph cluster would be a plus.
I checked out all the storage systems, ESOS, OpenFiler, OpenAttic, FreeNAS, and probably a bunch of others that I can't think of, and none of them seemed to do quite what I needed.
I needed:
-Software raid(hardware was already purchased, no raid controllers), or equivalent redundancy
-At least two hosts, with the ability to survive a host failure
-Ability to be a FibreChannel target with support for standard scsi features, persistent reservations, etc...
-Ability to present >16 LUNs via FC
After a ton of RTFM, trial and error, I finally settled on plain Ubuntu 16.04, tied into our existing Ceph cluster, with LIO serving up the FC target, with a fileio backstore that lives on CephFS.
Performance sucks with the fileio on CephFS, but to go with RBD I had to use nbd-rbd(because the RBD built into the kernel was old and didn't support half of what I needed). nbd-rbd was limited to 16 block devices /dev/nbd0-15 and I had concerns over it's ability to work well, given that pretty much everyone has forgotten about FC with the LIO development.
Which brings me to the current state - everything is working well, except I can't find any info on how to get LIO to tell the initiators(currently testing with ESXi, but the final solution will be talking to some Sun servers) that the luns on both nodes are the same -- that they're two paths to 1 device(the fileio backstores, both pointing to the same file on CephFS), not two paths to two devices.
My RTFM all tells me that ALUA is the magic that I need. ALUA is well documented for the older versions of LIO, using tcm_node.py and I think if I could use this, I could configure what I needed. But, tcm_node.py is gone, and targetcli is what I have. I can't find anything about ALUA at all in targetcli, other than it's 'implicit'.
I don't see how it can be 'implicit' when I have no way of informing LIO that another node exists that is exporting the same storage device, and to tell the initiators about it.
Everything I read says that if I was doing this with iSCSI instead of FC, that I would be done already. Why is it so much harder with FC?
Any suggestions? Is there some magic buried somewhere in the documentation that I haven't found yet? Maybe something I can do editing the config file directly?
Using the stock versions of targetcli and LIO with Ubuntu Server 16.04
Long story short I had a project that due to budget constraints needed a home-grown, dual-node(for redundancy) FC storage "SAN". Performance didn't have to be stellar. Tying it into our existing Ceph cluster would be a plus.
I checked out all the storage systems, ESOS, OpenFiler, OpenAttic, FreeNAS, and probably a bunch of others that I can't think of, and none of them seemed to do quite what I needed.
I needed:
-Software raid(hardware was already purchased, no raid controllers), or equivalent redundancy
-At least two hosts, with the ability to survive a host failure
-Ability to be a FibreChannel target with support for standard scsi features, persistent reservations, etc...
-Ability to present >16 LUNs via FC
After a ton of RTFM, trial and error, I finally settled on plain Ubuntu 16.04, tied into our existing Ceph cluster, with LIO serving up the FC target, with a fileio backstore that lives on CephFS.
Performance sucks with the fileio on CephFS, but to go with RBD I had to use nbd-rbd(because the RBD built into the kernel was old and didn't support half of what I needed). nbd-rbd was limited to 16 block devices /dev/nbd0-15 and I had concerns over it's ability to work well, given that pretty much everyone has forgotten about FC with the LIO development.
Which brings me to the current state - everything is working well, except I can't find any info on how to get LIO to tell the initiators(currently testing with ESXi, but the final solution will be talking to some Sun servers) that the luns on both nodes are the same -- that they're two paths to 1 device(the fileio backstores, both pointing to the same file on CephFS), not two paths to two devices.
My RTFM all tells me that ALUA is the magic that I need. ALUA is well documented for the older versions of LIO, using tcm_node.py and I think if I could use this, I could configure what I needed. But, tcm_node.py is gone, and targetcli is what I have. I can't find anything about ALUA at all in targetcli, other than it's 'implicit'.
I don't see how it can be 'implicit' when I have no way of informing LIO that another node exists that is exporting the same storage device, and to tell the initiators about it.
Everything I read says that if I was doing this with iSCSI instead of FC, that I would be done already. Why is it so much harder with FC?
Any suggestions? Is there some magic buried somewhere in the documentation that I haven't found yet? Maybe something I can do editing the config file directly?
Using the stock versions of targetcli and LIO with Ubuntu Server 16.04