SAN target on linux

[tinfoil3d]

Hi.
Anyone can get me oriented in iscsi a little bit.
I've read there are two common ways to deploy a target on linux: scst and kernel-builtin lio with targetcli.
Aside from that there isn't really much info to judge which is better or faster.
And I'm totally new to the topic. I have this dell server here with an it-crossflashed perc that i can put a whole bunch of 3.5 disks into. And I have at least two, but potentially more small servers which are supposed to be initiators.
Now I can see that both scst and lio support a qlogic 16g hbas.
The question is basically, which one is faster, and is it better/faster to use ethernet or fc hbas to serve the thing? I mean, ethernet is simplier(but no switching would be involved as only two nodes currently planned, direct connect would suffice), but fc is faster (16g vs 10g). Ethernet can be faster if I'd jump to 25 or 40 or 50gig which are a little bit more expensive but then I'm not sure speed would ever go above 16gig for 1 port as no nvme ssd is planned and applications generally would not need that much bandwidth but above 10g is preferred so I think fc?
Besides, I'm gonna get som experience with SANs on my own. When it comes to SAN i'm only watching from afar, never touched the thing myself and only have vague idea. Am I even heading the right way, I mean, I can serve my individual disks to the individual nodes this way right? And I can choose if I use ethernet, fc or something else as a fabric?
The info on enterprise SAN is either too technical with lots of reading just to get the idea is involved or just scarce. But I figured I can probably use the skill. Many companies in Japan just love SANs. Although, I bet they also never heard of a regular linux being a target.

 

[EffrafaxOfWug]

Possible I'm behind the times since it's been a while since I was at that particular coalface, but iSCSI and Fibre Channel are different protocols; one is basically SCSI over ethernet, the other is SCSI over fibre. The waters were muddied somewhat with FCoE (Fibre Channel over Ethernet), but that's still using the FC protocol as opposed to iSCSI.

Classically, Fibre Channel was always faster; it used dedicated adapters that were almost always faster than the ethernet available, and used a dedicated fabric without any TCP/IP overhead and other associated networking niggles. But because every server came with ethernet already, iSCSI started "winning" on a cost/economy of scale basis.

For a home lab, I doubt there's any point in picking FC HBAs unless you've got a very nice deal on them and the storage switches. If you're just starting out, setting up your box with a big NIC or two and configuring it as an iSCSI target is a nice way to start learning (it was actually part of my RHCE exam back in the day). You certainly don't need 25Gb adapters to do it, my first lab was just using a dual-port 1Gb adapter although obviously that limits maximum throughput to ~100MB/s. There are Converged Network Adapters (CNAs) that'll support both regular ethernet and FCoE if you want to give either a whirl, although FCoE isn't something I've ever used myself (it's possible it might need a better class of switch than your average home lab, can't remember all the details though).

I recall using SCST at some point, but to keep it simple start with the LIO kernel builtin as I suspect that's more thoroughly tested/documented.

 

[tinfoil3d]

Thanks for input.

Possible I'm behind the times since it's been a while since I was at that particular coalface, but iSCSI and Fibre Channel are different protocols; one is basically SCSI over ethernet, the other is SCSI over fibre. The waters were muddied somewhat with FCoE (Fibre Channel over Ethernet), but that's still using the FC protocol as opposed to iSCSI.

I'm using this as a reference point: Linux SCSI Target - Main Page , also Linux SCSI Target - Fibre Channel
From which I conclude I can use this kernel server over FC or I can use plain ethernet iscsi. Or anything else. Am I right?

I recall using SCST at some point, but to keep it simple start with the LIO kernel builtin as I suspect that's more thoroughly tested/documented.

Well there seems to be some kind of holy war there SCST vs LIO/TCM but that aside I really have no idea which one is more mature and stable. While kernel modules probably are faster it's still a speculation... i didn't come across any real testing there.

 

[EffrafaxOfWug]

They've both been around for at least a decade so I'd consider them both quite mature certainly. My usual linux flavour, debian, doesn't seem to package SCST any more for whatever reason.

Yes, the FIO stuff should allow you to present a target over iSCSI, FCoE or FC depending on what storage fabric you're using. If you're just looking to experiment with a home lab, I'd recommend sticking to iSCSI at least to begin with as there's no exotic hardware needed beyond a switch and some NICs.

 

[tinfoil3d]

My usual linux flavour, debian, doesn't seem to package SCST any more for whatever reason.

Startled me too...
I guess I'll just stick with targetcli for testing. Thanks

 

[tinfoil3d]

@EffrafaxOfWug do you know how cache works in iscsi in linux? Say I have 128 or more gb of RAM, can I use that to accelerate writes? I know of filesystems cache of course but this host os would not see anything but raw block data. Is there any particular way of allocating cache or it's done transparently by default just like fs cache?

 

[EffrafaxOfWug]

Assuming you mean server-side, I'm not aware of any special caching methodologies used by FIO but it's not something I've studied in any detail. At a guess I'd say the target would follow the usual caching semantics for the underlying block devices; there'll be some read caching but cached writes would be kept to a minimum to mitigate potential data loss.

There's several ways of creating the backing stores though and the way the kernel handles these will vary the way caching is used; I've only used block-based myself (pseudo block devices created through LVM; this was before the days of ZFS et al) but IIRC there's also file-based backing stores as well. At a guess I'd say the file-based method would allow for a very large read caches (and potentially write caches as well) in the same way linux behaves with almost every file.

Edit: Had a wee search around and found this post which goes in to a bit more detail about the caching methodology and puts it quite succintly.

iscsitarget-devel - newbie question: difference between fileio and blockio in IET

newbie question: difference between fileio and blockio in IET. Hi all, I have a newbie question about the difference between fileio and blockio when it comes to IET. To my own understanding, in a...

iscsi-enterprise-target.996254.n3.nabble.com

Fileio uses the VFS layer to access it's backing store so it can
utilize regular files on file systems as well as block devices and
IO is cached in the page cache.

Blockio goes direct to the device, bypassing the VFS layer, so it
can only work with block devices directly.

Fileio provides good performance on systems with lots of RAM
but because it is memory mapped all IO is broken into 4k chunks
when going to disk, the system does a good job of combining them
for asynchronous IO, but not synchronous IO. This can lead to
performance issues in situations where either IO sizes are large
enough and fast enough to overwhelm the system in IOPS or the IOs
are >4k and synchronous.

Blockio talks directly to the storage drivers bypassing page cache
which means that the only caching taking place is what happens on
the storage controllers and drives. This will allow IOs of
arbitrary size to be passed to the storage controllers in a zero
copy method. Think of it as directly tieing your network interface
to your disk controllers. While no IO is cached it provides good
performance for large IO sizes and excellent latency for random
and synchronous IO.