NFS / iSCSI security on a compromised Initiator (client)

[james23]

I posted something similar over on the freenas forums, as im using a freenas box in this (new) setup im doing at my home office. I figured i would rewrite most of the post here as i know there are more enterprise admins here vs at FN forums.

The meat of my question is, how do admins (including enterprise), deal with SAN/NAS security in terms of a exploited client device? (ie a client device, connected to SAN storage either via iSCSI or NFS, is exploited, how do you prevent attacker from pivoting into SAN/NAS storage spaces that the client box should not be allowed to access).
Are things like file level permissions and use of "nfsnobody" group enough? (maybe not, see example exploit:
How to exploit weak NFS permissions through Privilege Escalation?)

Assume that only iSCSI / NFS services are available to the dedicated NIC(s)/vlan/subnet connected to the exploited client box (ie no SSH nor web gui on dedicated nic).

I ask this as in my new build, i will be running a publicly accessible web server , and this site needs access to 4-6tb of storage, so as opposed to direct attaching 3 or 4x HDDs to this baremetal server, i was hoping to save power/heat and serve the 4-6TB from my FreeNAS box (via nfs or iscsi). But im concerned about unauth access to other parts of my NAS. (keep in mind, i most likely will just end up using a isolated, direct disk attached baremetal server for this public httpd server, bc of security nfs/iscsi sec. concerns - but wanted to ask as i know this has to be a issue for enterprises)

Another option:
As opposed to a baremetal web server (connected to SAN via NFS), Is it a better idea to run ESXi standalone on the baremetal box, and then have a single guest/VM for the public webserver (and provide a NFS backed 4-6tb datastore TO the guest, such that the guest VM does not even use/see NFS back to the SAN). ie the attacker would have to gain access to ESXi first, and then pivot from esxi into the SAN via NFS.

here is a rough sketch of how i would lay this out, if i did go the SAN/nas (vs direct attached HDDs) route:


thanks for any input.

EDIT some good links (i dont think theyw ill apply to me as FN does not fully support nfs v4 or v4.1 i think):
Red Hat Enterprise Linux 7 8.8. Securing NFS - Red Hat Customer Portal

 

[mstone]

The meat of my question is, how do admins (including enterprise), deal with SAN/NAS security in terms of a exploited client device? (ie a client device, connected to SAN storage either via iSCSI or NFS, is exploited, how do you prevent attacker from pivoting into SAN/NAS storage spaces that the client box should not be allowed to access).
Are things like file level permissions and use of "nfsnobody" group enough?

In classic NFS the permissions are enforced client side. (In fact, if on an OS that doesn't enforce privileged ports, a regular user can connect to the NFS server directly and access anything!) If you use kerberos authentication, then the file access is controlled server-side via the rights of the appropriate kerberos user. (But, on a compromised client, any file accessible by any currently-authenticated user is at risk. So to secure things across compromised clients, each client would need its own distinct kerberos user which has sole access to the file(s) in question.)

iSCSI typically has minimal security. In some cases it's possible to set up per-client CHAP authentication is fairly meh, but the best available mechanism to prevent automatic exposure of all LUNs to a compromised client.

Most "enterprises" IME rely on the "assume nothing ever gets compromised" security primitive.

 

[Evan]

One reason to use fiber channel !

If you must use NAS type things you need to have a NAS for each security layer, and should still be restricted and zoned as much as possible. Would normally not allow any NFS or iscsi between a regular zone and an internet facing zone. Asking for big trouble I say.

 

[mstone]

One reason to use fiber channel !

How does that help? In most cases I've seen the FC security is based on restricting access to specific WWNs...which are spoofable. It is possible to lock things to particular switch ports, but the management overhead involved is significant and (again, IME) rarely implemented. It's also possible to implement CHAP authentication over FC, but you'd end up roughly in the same place as iSCSI security wise. Even with that, I'd still be leery of exposing this stuff to a truely hostile client because it is increasingly complex security-critical code which tends to not get updated very often and isn't (IMO) well-reviewed and battle hardened.

 

[EffrafaxOfWug]

If you're talking about exposing an internal LUN/filesystem to a DMZ/extranet, I'm not sure there's a good way of securing NFS in such an environment. Kerberos auth and encryption gets you decent security from rogue agents exposed to the same export, but it does nothing in terms of a compromised client - if your NFS is exported read/write then a compromised client can do anything with it.

iSCSI you'd generally a) have a dedicated storage network separate from the regular data network and b) zone/ACL/CHAP it to restrict access to rogue clients (in a similar way to FC as mstone mentions above). I'm not aware of any exploits where a compromised iSCSI client could gain access to different LUNs in that scenario (but I'm happy to learn).

As you've surmised already, the safest way to handle this is to provision the storage to a hypervisor or container system, and then have that handle the storage abstraction as that gets you the greatest amount of protection from a compromised guest. But ultimately any data exposed read/write to a compromised client is also potentially compromised itself - there's no way around that fundamental truth. The only mitigating factor in this scenario is backups, backups; airgapped backups; backups, egg, backups, backups, bacon and backups.

 

[mstone]

If you're talking about exposing an internal LUN/filesystem to a DMZ/extranet, I'm not sure there's a good way of securing NFS in such an environment. Kerberos auth and encryption gets you decent security from rogue agents exposed to the same export, but it does nothing in terms of a compromised client - if your NFS is exported read/write then a compromised client can do anything with it.

No, the machine needs to be authorized to mount the NFS+kerberos export, but access to specific files is controlled on a per-user basis, and a user ticket with appropriate authorization is required for each file access; compromising a host in this scenario only compromises files that are accessible to the system itself (e.g., world read/write) or to users whose active tickets are on the compromised host. This is unlike traditional NFS, where access is controlled client-side.

 

[EffrafaxOfWug]

Yes, but if a client is compromised then so is any user running on that client (esp. if we're including privilege escalation as per the OP). AIUI it's perfectly possible to steal user tickets if you've got root.

 

[mstone]

Yes, but if a client is compromised then so is any user running on that client (esp. if we're including privilege escalation as per the OP). AIUI it's perfectly possible to steal user tickets if you've got root.

I'm pretty sure I specifically said that a compromised client could access files for which valid user tokens exist on the client--but that's entirely different from "if your NFS is exported read/write then a compromised client can do anything with it". If the permissions are based on per-machine service accounts then the compromise of one machine does not imply compromise of files restricted to accounts on other machines. Obviously, the specifics of how permissions are structured is extremely relevant, and the use of highly-privileged global accounts or unrestricted files will negate the benefits of per-user access control (but the security model isn't there to keep you from shooting yourself in the foot).

 

[james23]

wow did this thread get alot of action. Thanks for that guys, i have read all the replies a few times now. I feel a little better in the sense that im not missing something big (ie some way to secure shared storage).
(i should have been a bit more clear; i am and will always use a isolated storage network, so in terms of a threat source, it would be OS or Guest level access (ie how the OSes (or vm guests) access that shared storage). not a malicious user scanning the network and finding a iSCSI )

Its pretty clear your security comes from the hypervisor "hiding" the shared storage from the Guest/attacker (ie it just looks like a disk).

So my next thought is- are SANs / NAS used for anything *other* than VM infrastructures? ie if you dont have a hypervisior protecting your users / oses from the mostly insecure protocols (ie nfs / iscsi) , how do you make use of a SAN?

its as if you need to hide from your attacker the fact that your making use of nfs or iscsi , else the attacker knows there is probably a nice "prize" if they increment or explore up that storage chain.

(in the real work / admin's experience) how common are vm guest escape incidents (or threats)? I do keep up with vSphere patches sec. news sources, but i just dont have the experience (or quantity of installs) to really have an answer to this.
(ie i know recently there was a "svga based" VMware guest escape exploit, that was patched pretty quickly, but did provide full access to the HV)

thanks

 

[EffrafaxOfWug]

I'm pretty sure I specifically said that a compromised client could access files for which valid user tokens exist on the client--but that's entirely different from "if your NFS is exported read/write then a compromised client can do anything with it". If the permissions are based on per-machine service accounts then the compromise of one machine does not imply compromise of files restricted to accounts on other machines. Obviously, the specifics of how permissions are structured is extremely relevant, and the use of highly-privileged global accounts or unrestricted files will negate the benefits of per-user access control (but the security model isn't there to keep you from shooting yourself in the foot).

Not sure that's what I meant but I didn't mean to imply that "just anything" could be done. The scenario the OP seemed to be describing was that of a single LUN/volume exported via either iSCSI or NFS - which to me implies a single host with more-or-less unfettered access to the entirety of the blocks/files contained therein, so if you're assuming the possibility of privilege escalation and the rest of it the assumption is that any of those blocks/files can be modified by a sufficiently compromised client.

An NFS export of a dir tree to multiple hosts, each secured with different KRB principals, is a different kettle of fish and like you say a compromise of one client shouldn't give it any access to areas readable only by other clients (assuming NFS exploits aren't in effect).

Using privilege escalation or other exploits over NFS is of course a whole 'nother ball game, but in those sorts of situations all bets are usually off anyway.

So my next thought is- are SANs / NAS used for anything *other* than VM infrastructures? ie if you dont have a hypervisior protecting your users / oses from the mostly insecure protocols (ie nfs / iscsi) , how do you make use of a SAN?

There was (and still is) plenty of usage of SAN fabrics before the advent of modern virtualisation to just provide storage to physicals. Segregation, as mentioned above, is normally just done on the concept of only provisioning the storage to certain WWNs and assuming nothing on the same storage fabric will try to spoof them (in our FC environment at least it was brought up but ultimately wasn't considered a credible threat that could realistically be prevented). As mstone mentions there are ACL mechanisms to restrict WWNs to particular physical ports but I've never seen them utilised in practice as no-one wants to deal with the overhead of setting them up and keeping changing them as blades are moved around, etc. I dare say they're used in ultra-secure environments.

(in the real work / admin's experience) how common are vm guest escape incidents (or threats)? I do keep up with vSphere patches sec. news sources, but i just dont have the experience (or quantity of installs) to really have an answer to this.
(ie i know recently there was a "svga based" VMware guest escape exploit, that was patched pretty quickly, but did provide full access to the HV)

They certainly occur, but it's quite easy to overblow the risk. Remember if you read through many CVEs a lot of them will require exotic things like guest root access to exploit, at which point the VM would have been considered compromised beyond redemption anyway (similarly, spoofing/changing a WWN typically requires admin access, with CHAP the password is still stored somewhere on the client, etc etc).

You can quickly go mad trying to prevent every possible attack vector and there's no way of making something 100% secure, so I worry a little you might be overthinking things somewhat. The best place to start is the most vulnerable part of the whole equation which'd be a) the interface between the outside world and your internal servers and b) the interaction between the application and the OS. Of course don't neglect the storage side of things but concentrate on the largest attack surfaces first.

If you had a description of your intended setup (probably needs another thread) that'd give people a better idea of what you were trying to achieve, as well as if things like a VPN into a DMZ is applicable (since that'll almost eliminate the risk of $internet_script_kiddie running headlong into your setup).

 

[Evan]

To answer @mstone about FC risks, was not trying to say that spoofing a wwn was not possible (anything but, it’s the basis of how NPIV works), what I was trying to convey is that assuming the physical network is secure if I give you a host that’s FC connected unlike iSCSI over Ethernet you can really use FC to attack other hosts and jump from machine to machine through the storage.

Happy to be corrected but in general FC as far as I am concerned as a storage network is safe to provision across security zones.