Shortish version:
On Fedora devel@, a concern has been raised regarding binaries with vulnerablities being persistently available via Btrfs snapshots in the normal file system hierarchy. This is a request for assessing the significance of this concern, and how to mitigate it. Therefore the context is rootfs on Btrfs.
The first email bringing up the concern is here: https://lists.fedoraproject.org/pipermail/devel/2014-January/194558.html
And a possible work around proposed here: https://lists.fedoraproject.org/pipermail/devel/2014-January/194620.html
How significant is the risk of stale binaries being persistently available in the normal file system hierarchy? Should something be done to either make sure they aren't persistently available (make sure they aren't available in the mounted file system hierarchy), and if they're mounted should noexec or nosuid be used?
Slightly longer version:
Other distros install to Btrfs the way they do any other file system. This means any snapshots are available via the mounted top level of the file system.
Fedora on the other hand has a different layout, with a subvolume named root in the top level of the file system. Inside this subvolume is the tree you'd normally find as rootfs. Instead of the top level file system being mounted, the root subvolume is mounted at /. This means it's possible to "hide" snapshots in the unmounted top level of the file system; or nested within a subvolume called snapshots, for example.
Presently, the optional yum-plugin-fs-snapshot creates snapshots of subvolumes prior to updates, and places the snapshots in the parent subvolume. (Since snapshotting subvolumes isn't recursive, the snapshots themselves aren't snapshot.) For example:
#btrfs subvolume list / ID 256 gen 352 top level 5 path root ID 257 gen 352 top level 5 path home ID 266 gen 95 top level 256 path yum_20140212183241 ID 267 gen 97 top level 5 path home/yum_20140212183242
Because the snapshot is located within a mounted subvolume, it makes the snapshot's stale binaries persistently available. So restating the earlier question, is this a security risk, how significant is it, and is it worth changing the behavior? Instead, these yum snapshots could be placed in a subvolume at the top level of the file system, which isn't mounted by default. The other question is whether there's a meaningful distinction between persistently mounting this snapshots subvolume, or only mounting it on demand when snapshots are about to be taken? And then when it's mounted, should the mount option be noexec or nosuid.
For what it's worth, the questions use this yum plug in as an example, but I'm uncertain if a future snapshot/rollback strategy will actually use it, or snapper, or some other utility entirely.
Rollbacks and logs:
Another thing that comes to mind, is the possibility of doing a rollback on the system. The yum snapshot plugin doesn't automatically do, or aid the user in doing rollbacks. Optional package snapper can.
Setting aside, for the moment, the granularity needed to properly do such rollbacks with file system snapshots, I'm wondering about logging behavior from a security perspective. It seems pretty clear to me we'd want a single persistent audit log kept, regardless of which snapshot is booted.
Some of the questions I have are, what logs shouldn't be rolled back? Maybe none should be? And is there a use case for snapshotting, for example /var/log, but simply not rolling it back. For what it's worth, Btrfs supports read only snapshots. They're not writable even by root, and even if they're mounted with a rw option. The snapshot also contains a UUID, as well as its parent's UUID. So, I don't know, maybe there's some benefit to the audit daemon tracking this information in a persistent log that we simply never rollback, even if it's being snapshot.
A part of the dialog I hope this generates is not only what we should be doing, but also how important it is to do it, as it might affect how anaconda creates its layouts for different Fedora.next products. Do any of the questions or concerns (and unasked questions of course) have different answers depending on whether the context is workstation, server, or cloud?
Thanks,
Chris Murphy
Chris Murphy
----- Original Message -----
From: "Chris Murphy" lists@colorremedies.com To: security@lists.fedoraproject.org Sent: Thursday, 13 February, 2014 3:16:34 AM Subject: btrfs snapshots, rollbacks
Shortish version:
On Fedora devel@, a concern has been raised regarding binaries with vulnerablities being persistently available via Btrfs snapshots in the normal file system hierarchy. This is a request for assessing the significance of this concern, and how to mitigate it. Therefore the context is rootfs on Btrfs.
The first email bringing up the concern is here: https://lists.fedoraproject.org/pipermail/devel/2014-January/194558.html
And a possible work around proposed here: https://lists.fedoraproject.org/pipermail/devel/2014-January/194620.html
How significant is the risk of stale binaries being persistently available in the normal file system hierarchy? Should something be done to either make sure they aren't persistently available (make sure they aren't available in the mounted file system hierarchy), and if they're mounted should noexec or nosuid be used?
As long as the old /bin and /usr/bin are not part of PATH, I'd say we've done our job. We can't protect the user from shooting himself in the foot in all cases.
The logs are a different matter, we should aim to preserve them. Dunno where journald is in this picture (binary log forward and backward compatibility).
On Feb 13, 2014, at 5:11 AM, Hubert Kario hkario@redhat.com wrote:
As long as the old /bin and /usr/bin are not part of PATH, I'd say we've done our job. We can't protect the user from shooting himself in the foot in all cases.
The snapshots aren't in PATH. However, the yum plugin would put them at
/yum_<datetime>/bin /yum_<datetime>/usr/bin
Snapper puts them in
/.snapshots/<#>/snapshot/bin /.snapshots/<#>/snapshot/usr/bin
I'm not sure what you mean by the user shooting himself - these locations aren't up to the user with these tools. And installer behavior can limit user choice as to where the snapshots can be placed.
So, is the ability to hide snapshots in an unmounted portion of the (on-disk) file system valuable from a security perspective? Or it it trivial?
The logs are a different matter, we should aim to preserve them. Dunno where journald is in this picture (binary log forward and backward compatibility).
If by preserve you mean a single contiguous log location, then that implies needing a subvolume for logs. For example:
http://lists.freedesktop.org/archives/systemd-devel/2014-January/016253.html
I have implemented this and it appears to work, although probably it should be a log subvolume mounted at /var/log so that all logs can be kept contiguous, not just the journal.
Chris Murphy
----- Original Message -----
From: "Chris Murphy" lists@colorremedies.com To: "Hubert Kario" hkario@redhat.com Cc: security@lists.fedoraproject.org Sent: Thursday, 13 February, 2014 5:02:57 PM Subject: Re: btrfs snapshots, rollbacks
On Feb 13, 2014, at 5:11 AM, Hubert Kario hkario@redhat.com wrote:
As long as the old /bin and /usr/bin are not part of PATH, I'd say we've done our job. We can't protect the user from shooting himself in the foot in all cases.
The snapshots aren't in PATH. However, the yum plugin would put them at
/yum_<datetime>/bin /yum_<datetime>/usr/bin
Snapper puts them in
/.snapshots/<#>/snapshot/bin /.snapshots/<#>/snapshot/usr/bin
I'm not sure what you mean by the user shooting himself - these locations aren't up to the user with these tools. And installer behavior can limit user choice as to where the snapshots can be placed.
So, is the ability to hide snapshots in an unmounted portion of the (on-disk) file system valuable from a security perspective? Or it it trivial?
I would consider it trivial.
The logs are a different matter, we should aim to preserve them. Dunno where journald is in this picture (binary log forward and backward compatibility).
If by preserve you mean a single contiguous log location, then that implies needing a subvolume for logs. For example:
http://lists.freedesktop.org/archives/systemd-devel/2014-January/016253.html
I have implemented this and it appears to work, although probably it should be a log subvolume mounted at /var/log so that all logs can be kept contiguous, not just the journal.
Yes, that's what I was thinking about.
If we're going to support update rollback through snapshots I think that /var/log should be kept separate in default install.
On Feb 13, 2014, at 9:27 AM, Hubert Kario hkario@redhat.com wrote:
----- Original Message -----
From: "Chris Murphy" lists@colorremedies.com To: "Hubert Kario" hkario@redhat.com Cc: security@lists.fedoraproject.org Sent: Thursday, 13 February, 2014 5:02:57 PM Subject: Re: btrfs snapshots, rollbacks
On Feb 13, 2014, at 5:11 AM, Hubert Kario hkario@redhat.com wrote:
As long as the old /bin and /usr/bin are not part of PATH, I'd say we've done our job. We can't protect the user from shooting himself in the foot in all cases.
The snapshots aren't in PATH. However, the yum plugin would put them at
/yum_<datetime>/bin /yum_<datetime>/usr/bin
Snapper puts them in
/.snapshots/<#>/snapshot/bin /.snapshots/<#>/snapshot/usr/bin
I'm not sure what you mean by the user shooting himself - these locations aren't up to the user with these tools. And installer behavior can limit user choice as to where the snapshots can be placed.
So, is the ability to hide snapshots in an unmounted portion of the (on-disk) file system valuable from a security perspective? Or it it trivial?
I would consider it trivial.
OK good. (There are other reasons why hidden snapshots may be preferred, such as keeping snapshot contents out of find results, and better parity with LVM snapshot behavior.)
The logs are a different matter, we should aim to preserve them. Dunno where journald is in this picture (binary log forward and backward compatibility).
If by preserve you mean a single contiguous log location, then that implies needing a subvolume for logs. For example:
http://lists.freedesktop.org/archives/systemd-devel/2014-January/016253.html
I have implemented this and it appears to work, although probably it should be a log subvolume mounted at /var/log so that all logs can be kept contiguous, not just the journal.
Yes, that's what I was thinking about.
If we're going to support update rollback through snapshots I think that /var/log should be kept separate in default install.
I don't know where Fedora is at with Btrfs by default, let alone how the Workstation PRD envisions implementing "Better upgrade/rollback control" or how any other Fedora product considers such functionality should work. There is this:
http://fedoraproject.org/wiki/Changes/Rollback
But that only mentions LVM thin provisioning. Not Btrfs.
Yet the same concern with logs applies to LVM snapshotting and rollback. So if /var/log should be kept separate in a default install then it sounds like you'd support an RFE for anaconda that calls for automatic partitioning creating a log subvolume/LV to be mounted at /var/log. True?
I think the rule would be something like "if rootfs is on Btrfs or a virtual LV, then automatically create a "log" subvolume/LV and mount it at /var/log". Ack/nack/patch?
Chris Murphy
----- Original Message -----
From: "Chris Murphy" lists@colorremedies.com To: "Hubert Kario" hkario@redhat.com Cc: security@lists.fedoraproject.org Sent: Thursday, 13 February, 2014 5:58:45 PM Subject: Re: btrfs snapshots, rollbacks
On Feb 13, 2014, at 9:27 AM, Hubert Kario hkario@redhat.com wrote:
The logs are a different matter, we should aim to preserve them. Dunno where journald is in this picture (binary log forward and backward compatibility).
If by preserve you mean a single contiguous log location, then that implies needing a subvolume for logs. For example:
http://lists.freedesktop.org/archives/systemd-devel/2014-January/016253.html
I have implemented this and it appears to work, although probably it should be a log subvolume mounted at /var/log so that all logs can be kept contiguous, not just the journal.
Yes, that's what I was thinking about.
If we're going to support update rollback through snapshots I think that /var/log should be kept separate in default install.
I don't know where Fedora is at with Btrfs by default, let alone how the Workstation PRD envisions implementing "Better upgrade/rollback control" or how any other Fedora product considers such functionality should work. There is this:
http://fedoraproject.org/wiki/Changes/Rollback
But that only mentions LVM thin provisioning. Not Btrfs.
Yet the same concern with logs applies to LVM snapshotting and rollback. So if /var/log should be kept separate in a default install then it sounds like you'd support an RFE for anaconda that calls for automatic partitioning creating a log subvolume/LV to be mounted at /var/log. True?
yes
I think the rule would be something like "if rootfs is on Btrfs or a virtual LV, then automatically create a "log" subvolume/LV and mount it at /var/log". Ack/nack/patch?
Sounds reasonable
On 02/13/2014 03:16 AM, Chris Murphy wrote:
How significant is the risk of stale binaries being persistently available in the normal file system hierarchy? Should something be done to either make sure they aren't persistently available (make sure they aren't available in the mounted file system hierarchy), and if they're mounted should noexec or nosuid be used?
This is similar to security measurements (version status and malware scanning) on suspend virtual machines or their snapshots. I think a considerable amount of cycles has been spent on trying to address it there. The libvirt folks might already have something.
On Thu, Feb 13, 2014 at 3:16 AM, Chris Murphy lists@colorremedies.comwrote:
How significant is the risk of stale binaries being persistently available in the normal file system hierarchy?
Depending on the vulnerability being fixed, up to "critical".
Should something be done to either make sure they aren't persistently
available (make sure they aren't available in the mounted file system hierarchy)
Yes.
and if they're mounted should noexec or nosuid be used?
Probably; A possible alternative would be to mount them into /$snapshots, where $snaphosts is 0700 root-only (and protected by SELinux to only an unconfined administrator) - but without SELinux a root-owned process with no capabilities would still be able to access the files.
Because the snapshot is located within a mounted subvolume, it makes the snapshot's stale binaries persistently available. So restating the earlier question, is this a security risk, how significant is it, and is it worth changing the behavior?
Yes, very, and yes. (The risk is that there is a security vulnerability in a setuid-root program, and making the vulnerable application runnable by unprivileged users keeps the vulnerability exploitable. It doesn't matter whether the file is in PATH, or where exactly it is located, as long any process that doesn't have the privileges of an unconfined system administrator can access it.) Mirek
On Feb 13, 2014, at 11:05 AM, Miloslav Trmač mitr@volny.cz wrote:
On Thu, Feb 13, 2014 at 3:16 AM, Chris Murphy lists@colorremedies.com wrote: How significant is the risk of stale binaries being persistently available in the normal file system hierarchy? Depending on the vulnerability being fixed, up to "critical".
Should something be done to either make sure they aren't persistently available (make sure they aren't available in the mounted file system hierarchy) Yes.
and if they're mounted should noexec or nosuid be used? Probably; A possible alternative would be to mount them into /$snapshots, where $snaphosts is 0700 root-only (and protected by SELinux to only an unconfined administrator) - but without SELinux a root-owned process with no capabilities would still be able to access the files.
Because the snapshot is located within a mounted subvolume, it makes the snapshot's stale binaries persistently available. So restating the earlier question, is this a security risk, how significant is it, and is it worth changing the behavior?
Yes, very, and yes. (The risk is that there is a security vulnerability in a setuid-root program, and making the vulnerable application runnable by unprivileged users keeps the vulnerability exploitable. It doesn't matter whether the file is in PATH, or where exactly it is located, as long any process that doesn't have the privileges of an unconfined system administrator can access it.)
Interesting :-) two opposite opinions on this.
Fedora is already in a good position to go either way, because rootfs is in a subvolume called "root". The name can be different, but the fact it will be in a separate subvolume and not at the top level of the Btrfs file system is immutable. Conversely, other distros are putting rootfs at the top level of the Btrfs file system as if it were any other file system, so there's no flexibility.
Now if we can get /boot on Btrfs supported…
Chris Murphy
----- Original Message -----
From: "Chris Murphy" lists@colorremedies.com To: security@lists.fedoraproject.org Sent: Thursday, 13 February, 2014 7:13:22 PM Subject: Re: btrfs snapshots, rollbacks
On Feb 13, 2014, at 11:05 AM, Miloslav Trmač mitr@volny.cz wrote:
On Thu, Feb 13, 2014 at 3:16 AM, Chris Murphy lists@colorremedies.com wrote: How significant is the risk of stale binaries being persistently available in the normal file system hierarchy? Depending on the vulnerability being fixed, up to "critical".
Should something be done to either make sure they aren't persistently available (make sure they aren't available in the mounted file system hierarchy) Yes.
and if they're mounted should noexec or nosuid be used? Probably; A possible alternative would be to mount them into /$snapshots, where $snaphosts is 0700 root-only (and protected by SELinux to only an unconfined administrator) - but without SELinux a root-owned process with no capabilities would still be able to access the files.
Because the snapshot is located within a mounted subvolume, it makes the snapshot's stale binaries persistently available. So restating the earlier question, is this a security risk, how significant is it, and is it worth changing the behavior?
Yes, very, and yes. (The risk is that there is a security vulnerability in a setuid-root program, and making the vulnerable application runnable by unprivileged users keeps the vulnerability exploitable. It doesn't matter whether the file is in PATH, or where exactly it is located, as long any process that doesn't have the privileges of an unconfined system administrator can access it.)
Interesting :-) two opposite opinions on this.
Not really, I was thinking from a desktop system perspective.
It's completely different matter on a server.
On Wed, Feb 12, 2014 at 07:16:34PM -0700, Chris Murphy wrote:
isn't mounted by default. The other question is whether there's a meaningful distinction between persistently mounting this snapshots subvolume, or only mounting it on demand when snapshots are about to be taken? And then when it's mounted, should the mount option be noexec or nosuid.
If old snapshots are mounted, there are several possible security implications:
An old snapshot might contain
- A world-readable confidential files thats permissions were fixed after creating the snapshot, e.g. /etc/pki/tls/private/foo.key
- A confidential file with too many ACLs that were fixed after creating the snapshot
- A confidential file with the bad selinux context allowing to be read by an exploited daemon
- A vulnerable suid binary
- A vulnerable binary with capabilities
- An executable with the wrong selinux context allowing an exploited daemon to execute a binary that is not executable on the current system
- A device file with bad permissions/ACL/selinux context
Therefore I guess it needs to be made sure that no unprivileged process can access the contents of a mounted snapshot. Maybe the root directory can be protected with strict permissions/ACLs and a selinux context that does not allow anything else to access the contents.
Regards Till
----- Original Message -----
From: "Till Maas" opensource@till.name To: security@lists.fedoraproject.org Sent: Monday, 17 February, 2014 11:40:42 PM Subject: Re: btrfs snapshots, rollbacks
On Wed, Feb 12, 2014 at 07:16:34PM -0700, Chris Murphy wrote:
isn't mounted by default. The other question is whether there's a meaningful distinction between persistently mounting this snapshots subvolume, or only mounting it on demand when snapshots are about to be taken? And then when it's mounted, should the mount option be noexec or nosuid.
If old snapshots are mounted, there are several possible security implications:
An old snapshot might contain
A world-readable confidential files thats permissions were fixed after creating the snapshot, e.g. /etc/pki/tls/private/foo.key
A confidential file with too many ACLs that were fixed after creating the snapshot
A confidential file with the bad selinux context allowing to be read by an exploited daemon
A vulnerable suid binary
A vulnerable binary with capabilities
An executable with the wrong selinux context allowing an exploited daemon to execute a binary that is not executable on the current system
A device file with bad permissions/ACL/selinux context
Therefore I guess it needs to be made sure that no unprivileged process can access the contents of a mounted snapshot. Maybe the root directory can be protected with strict permissions/ACLs and a selinux context that does not allow anything else to access the contents.
I agree. That is needed for /, especially on multi user systems.
What about /home? Some of those issues apply to it too, problem is that making snapshots readable only for root user will make snapshots on /home nearly useless (if the admin needs to be involved to restore old version of a file he may just as well restore them from backup).
I'm afraid we will have to relax the permissions of /home snapshots.
On Feb 17, 2014, at 3:40 PM, Till Maas opensource@till.name wrote:
On Wed, Feb 12, 2014 at 07:16:34PM -0700, Chris Murphy wrote:
isn't mounted by default. The other question is whether there's a meaningful distinction between persistently mounting this snapshots subvolume, or only mounting it on demand when snapshots are about to be taken? And then when it's mounted, should the mount option be noexec or nosuid.
If old snapshots are mounted, there are several possible security implications:
Just to be clear, the old snapshot doesn't necessarily need to be mounted for it to be available. If the snapshot is located below a mounted subvolume, then it's available as if it were a normal directory. That's the case here, where yum-plugin-fs-snapshot takes a snapshot of /, and places it in the root of the mounted file system (which on Fedora is a mounted Btrfs subvolume named "root").
If instead they were placed either at the top level of the Btrfs file system (in subvolume id 5, along with the root and home subvolumes) they'd effectively be hidden since the top level isn't normally mounted.
So a more thorough look at this would be whether a DE can be tricked into mounting that top level. And whether a compromised root user can mount subvolumes/snapshots over existing rootfs directories such as /lib.
An old snapshot might contain
- A world-readable confidential files thats permissions were fixed after
creating the snapshot, e.g. /etc/pki/tls/private/foo.key
- A confidential file with too many ACLs that were fixed after creating
the snapshot
- A confidential file with the bad selinux context allowing to be read
by an exploited daemon
A vulnerable suid binary
A vulnerable binary with capabilities
An executable with the wrong selinux context allowing an exploited
daemon to execute a binary that is not executable on the current system
- A device file with bad permissions/ACL/selinux context
Therefore I guess it needs to be made sure that no unprivileged process can access the contents of a mounted snapshot.
RFE: btrfs rootfs snapshots should be placed outside the mounted path https://bugzilla.redhat.com/show_bug.cgi?id=1060241
Depending on the concern and how strict the security policy needs to be:
a. Only place such snapshots outside of the the mounted file system; i.e. don't create the snapshot of a subvolume inside that same subvolume. For example, not this: btrfs subvolume snapshot /var.1 /var But rather: mount -o subvol=snapshots <device> /mnt/snapshots btrfs subvolume snapshot /mnt/snapshots/var.1 /var umount /mnt/snapshots
b. Per above, but then use btrfs send to put the data onto a completely separate file system, possibly with -f switch so that it's a file, and then we can set some stringent selinux label to how it's accessed.
c. Don't use snapshots. Maybe we need an selinux policy that inhibits the creation of snapshots for certain subvolumes? Even by root?
Maybe the root directory can be protected with strict permissions/ACLs and a selinux context that does not allow anything else to access the contents.
Some way to make rootfs snapshot contents non-executable unless they're mounted by root? Or unless they're mounted at boot time (i.e. a rollback). It sounds like a layered, or conditional context, that would apply if a containing "snapshots" subvolume is mounted; but if an actual rootfs snapshot itself where mounted, such as at boot time, its usual selinux labeling applies. Is this possible?
Chris Murphy
security@lists.fedoraproject.org
-
Chris Murphy -
Florian Weimer -
Hubert Kario -
Miloslav Trmač -
Till Maas