https://fedoraproject.org/wiki/Changes/FsVerityRPM
== Summary ==
Enable the use of fsverity for installed RPM files validation.
== Owners ==
* Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris Burkov]], [[User:Filbranden|Filipe Brandenburger]], [[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew Almond]] * Email: dcavalca@fb.com, borisb@fb.com, filbranden@fb.com, michel@fb.com, malmond@fb.com
== Detailed description ==
fs-verity is a [https://www.kernel.org/doc/html/latest/filesystems/fsverity.html Linux kernel feature] that does transparent on-demand integrity/authenticity verification of the contents of read-only files, using a hidden Merkle tree (hash tree) associated with the file. The mechanism is similar to dm-verity, but implemented at the file level rather than at the block device level.
When fsverity is enabled for a file, the kernel reads every block and generates a hash tree for it, which is stored within the filesystem. On subsequent reads, the kernel computes the block hash and compares it with the one stored in the tree, protecting against alterations and corruption. Because this happens at the filesystem data block read layer, it encompasses all file operations (<code>open</code>, <code>mmap</code>,<code>exec</code>, etc.).
In the context of rpm, there are two parts to this: * at build time, we compute the Merkle tree for the files within a package, then sign it and ship it as part of the rpm metadata; * at run time, if the fsverity rpm plugin is enabled, rpm will install the fsverity signature key and enable fsverity on files that are installed.
This proposal is primarily concerned with the first part, which will make it possible for users to leverage fs-verity for RPM if they so desire. Specifically, installing and enabling the fs-verity rpm plugin by default is explicitly considered out of scope here.
=== Caveats ===
==== Merkle tree cost ====
The Merkle tree used by fsverity needs to be generated (once at build time, once when the package is installed) and stored on disk. The generation process involves reading all blocks and computing the hash, which has a non-trivial cost; however, it does not appear to meaningfully slow down package installs during empirical testing. Once generated, the Merkle tree will use up some disk space for its storage (about 1/127th of the original file size). Note that the Merkle tree is ''not'' shipped with the RPM itself (only its signature is) and is only generated and stored at install time if the fsverity rpm plugin is enabled. Hence, there is no cost (neither in generation time nor in disk space usage) if the plugin is disabled.
==== Signature overhead cost ====
To leverage fsverity every rpm needs to include the hash signature as part of its metadata, which will increase its size. The signature size is roughly proportional to the number of files in the package. From empirical testing, in the vast majority of cases we expect to see minimal to no size increase thanks to RPM header packing.
=== Relationship with IMA ===
[https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another technology meant to provide detection of file alterations. IMA and fsverity operate very differently, and are somewhat complementary.
fs-verity works by using a Merkle tree to generate a checksum for every data block in the system, and reads will fail if a single data block read fails it’s checksum. The signature of the the file is validated against a public key loaded into the kernel keyring. Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.
IMA works by measuring a file as a whole and comparing its signature whenever it’s read of executed. It has a higher runtime cost than fsverity (as it needs to verify the whole file at once) and it cannot detect subsequent alternations. IMA provides a much more rich and complex policy system, allowing one to define system-wide policies around trusted files that tie into LSMs such as SELinux.
IMA and fsverity could potentially be integrated (meaning, an fsverity backend for IMA could be implemented to leverage its policy controls), but this is not currently planned or being worked on.
=== Relationship with native checksums ===
By default, btrfs already checksums each file extent, which could potentially be leveraged to implement a HMAC solution. This currently exists as a [https://lore.kernel.org/linux-btrfs/SN4PR0401MB3598DDEF9BF9BACA71A1041D9BA70... patch series] but it hasn’t been merged yet. Similarly to IMA, we see this approach as complementary to fs-verity. The [https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-bt... blog post] goes into more details of the tradeoffs involved.
== Feedback ==
Pending the devel discussion
== Benefit to Fedora ==
The main benefit is the ability to do block-level verification of RPM-installed files. In turn, this can be used to implement usecase-specific validation and verification policies depending on the environment requirements.
== Scope ==
* Proposal owners ** btrfs kernel enablement work ([https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... landed in 5.15]); see this [https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-bt... blog post] for more details ** koji integration: koji will need to add the fs-verity metadata to packages when signing them * Other developers: ** deploy the koji integration changes to production * Release engineering: tbd * Policies and guidelines: N/A * Trademark approval: N/A
== Upgrade/compatibility impact ==
None
== How to test ==
Install the fs-verity RPM plugin to validate package contents:
<pre>$ sudo dnf install rpm-plugin-fsverity</pre> Note that this will only be useful if the packages being installed contain the appropriate fs-verity metadata (which, for Fedora upstream packages, requires Koji integration that is part of this Change). However, you should still be able to test this if you locally sign a package with <code>rpmsign --addverity</code>.
== User experience ==
This Change is fully transparent and there is no user impact by default. If the user chooses to enable the fs-verity RPM plugin, they can then leverage the additional verification features provided by fs-verity.
== Dependencies ==
* fs-verity support is available in RPM as of 4.17, which is available as of Fedora 35 and is already enabled in rpm-4.17.0-0.rc1.0.fc36 * CONFIG_FS_VERITY in the kernel config; this is already enabled * fs-verity requires filesystem support; currently support for ext4 and f2fs is already available; support for btrfs landed in 5.15 * there is no filesystem dependency on the builders, only at runtime (and only if the rpm fsverity plugin is installed and one wishes to use it)
== Contingency plan ==
Revert the changes to koji.
== Documentation ==
* https://www.kernel.org/doc/html/latest/filesystems/fsverity.html * https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-bt... * The proposal owners plan to document the fsverity plugin and integration in RPM (https://github.com/rpm-software-management/rpm/issues/1849)
== Release Notes ==
The RPM package manager now supports validation of file contents using fs-verity.
=== Relationship with IMA ===
[https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another technology meant to provide detection of file alterations. IMA and fsverity operate very differently, and are somewhat complementary. [...]
Do these two systems use the same per-file signature metadata in the RPMs?
- FChE
On Thu, 2021-12-02 at 15:08 -0500, Frank Ch. Eigler wrote:
=== Relationship with IMA ===
[https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another technology meant to provide detection of file alterations. IMA and fsverity operate very differently, and are somewhat complementary. [...]
Do these two systems use the same per-file signature metadata in the RPMs?
Both fs-verity and IMA use file signatures, but they each have their own dedicated flags and signing flows in RPM (e.g. see https://github.com/rpm-software-management/rpm/blob/4afe2d14d33db82ccb41c0a8... for the signing implementation). The signatures themselves are not interchangeable -- fs-verity's signature is based off the Merkle tree (which itself is block-based), while IMA measures the file as a whole.
Cheers Davide
On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote: ...snip...
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
This is some kind of seperate signing that happens at build time?
Or it's added to the rpm metadata and covered by the normal package signing if/when the package is signed?
- at run time, if the fsverity rpm plugin is enabled, rpm will install
the fsverity signature key and enable fsverity on files that are installed.
Is this signature key the fedora rpm package signing key? Or some seperate fsverity key?
...snip...
fs-verity works by using a Merkle tree to generate a checksum for every data block in the system, and reads will fail if a single data
every data block? or every packaged in rpms data block?
block read fails it’s checksum. The signature of the the file is validated against a public key loaded into the kernel keyring. Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.
Is this via dm_verity kernel module? Or thats a completely seperate thing?
...snip...
== Scope ==
- Proposal owners
** btrfs kernel enablement work ([https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... landed in 5.15]); see this [https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-bt... blog post] for more details
What does this mean for all the other filesystems? They would just be slower since btrfs is computing the trees as part of it's normal checksumming?
** koji integration: koji will need to add the fs-verity metadata to packages when signing them
Well, koji doesn't sign packages. robosignatory listens for messages on the message bus for koji tagging and based on it's config, it asks sigul to sign rpms and import the signatures into koji.
There's support in robosignatory to ask to sign files (used for the short lived IMA stuff), but I suspect it would need a new ability for this.
Finally who is going to write this? Change owners? Or do you expect robosignatory maintainers to do so?
- Other developers:
** deploy the koji integration changes to production
- Release engineering: tbd
- Policies and guidelines: N/A
- Trademark approval: N/A
== Upgrade/compatibility impact ==
None
== How to test ==
Install the fs-verity RPM plugin to validate package contents:
<pre>$ sudo dnf install rpm-plugin-fsverity</pre>
Note that this will only be useful if the packages being installed contain the appropriate fs-verity metadata (which, for Fedora upstream packages, requires Koji integration that is part of this Change). However, you should still be able to test this if you locally sign a package with <code>rpmsign --addverity</code>.
== User experience ==
This Change is fully transparent and there is no user impact by default. If the user chooses to enable the fs-verity RPM plugin, they can then leverage the additional verification features provided by fs-verity.
== Dependencies ==
- fs-verity support is available in RPM as of 4.17, which is available
as of Fedora 35 and is already enabled in rpm-4.17.0-0.rc1.0.fc36
- CONFIG_FS_VERITY in the kernel config; this is already enabled
- fs-verity requires filesystem support; currently support for ext4
and f2fs is already available; support for btrfs landed in 5.15
No xfs support?
Thanks in advance for answers.
kevin
On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote: ...snip...
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
This is some kind of seperate signing that happens at build time?
Or it's added to the rpm metadata and covered by the normal package signing if/when the package is signed?
As part of the signing flow (e.g. via rpmsign), the Merkle tree is generated and a signature is computed from it, which is then added to the rpm metadata.
- at run time, if the fsverity rpm plugin is enabled, rpm will
install the fsverity signature key and enable fsverity on files that are installed.
Is this signature key the fedora rpm package signing key? Or some seperate fsverity key?
fs-verity needs a RSA key/cert pair for file signing at package signature time. At package install time, the cert needs to be loaded in the appropriate kernel keyring. We've always used a dedicated keypair during testing -- I'm not actually sure if the package signing key could be reused for this, as it's a GPG key, but this is something we should follow up on.
fs-verity works by using a Merkle tree to generate a checksum for every data block in the system, and reads will fail if a single data
every data block? or every packaged in rpms data block?
fs-verity only operates on files where it has been enabled via its ioctl (which, if you install the RPM plugin, is taken care of by RPM on your behalf). For those, fs-verity will checksum every data block whenever it's accessed and validate it still matches.
block read fails it’s checksum. The signature of the the file is validated against a public key loaded into the kernel keyring. Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.
Is this via dm_verity kernel module? Or thats a completely seperate thing?
It's somewhat inspired by dm-verity, but it's a separate implementation, the only shared logic is the hash computation code in the kernel.
== Scope ==
- Proposal owners
** btrfs kernel enablement work ([ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... landed in 5.15]); see this [ https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-bt... blog post] for more details
What does this mean for all the other filesystems? They would just be slower since btrfs is computing the trees as part of it's normal checksumming?
fs-verity requires support in the underlying filesystem. If you're using a filesystem that doesn't support it and attempt to enable fs- verity on a file, the ioctl will fail. Note that this is only a concern at runtime, not at build time.
** koji integration: koji will need to add the fs-verity metadata to packages when signing them
Well, koji doesn't sign packages. robosignatory listens for messages on the message bus for koji tagging and based on it's config, it asks sigul to sign rpms and import the signatures into koji.
There's support in robosignatory to ask to sign files (used for the short lived IMA stuff), but I suspect it would need a new ability for this.
Finally who is going to write this? Change owners? Or do you expect robosignatory maintainers to do so?
Thanks for clarifying! Yes, I think robosignatory is likely what we want here. We (the Change owners) expect to do the work, though we'll likely need some advice/help around code review, testing and deployment.
- fs-verity requires filesystem support; currently support for ext4
and f2fs is already available; support for btrfs landed in 5.15
No xfs support?
Correct, XFS doesn't support fs-verity at the moment (though it could be implemented if one wanted to).
Cheers Davide
On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel < devel@lists.fedoraproject.org> wrote:
On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote: ...snip...
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
This is some kind of seperate signing that happens at build time?
Or it's added to the rpm metadata and covered by the normal package signing if/when the package is signed?
As part of the signing flow (e.g. via rpmsign), the Merkle tree is generated and a signature is computed from it, which is then added to the rpm metadata.
IMA received significant pushback on the impact to RPMs and signing implications in Fedora. How does fs-verity compare here? Alternatively/additionally, why is fs-verity worth the hit for Fedora where IMA was not. I'm not looking to create a false equivalence or compare apples to oranges, but I want to better understand why this is a more acceptable approach for Fedora.
As an aside, a few people have asked why RHEL didn't get IMA into Fedora first. Frankly, we tried and got feedback that it didn't apply to Fedora's needs and use cases. As of right now, we have no plans to force something into Fedora that the community doesn't want. If fs-verity is more acceptable or is described in a way that is better understood with enough similarities to IMA, we may try again.
- at run time, if the fsverity rpm plugin is enabled, rpm will
install the fsverity signature key and enable fsverity on files that are installed.
Is this signature key the fedora rpm package signing key? Or some seperate fsverity key?
fs-verity needs a RSA key/cert pair for file signing at package signature time. At package install time, the cert needs to be loaded in the appropriate kernel keyring. We've always used a dedicated keypair during testing -- I'm not actually sure if the package signing key could be reused for this, as it's a GPG key, but this is something we should follow up on.
fs-verity works by using a Merkle tree to generate a checksum for every data block in the system, and reads will fail if a single data
every data block? or every packaged in rpms data block?
fs-verity only operates on files where it has been enabled via its ioctl (which, if you install the RPM plugin, is taken care of by RPM on your behalf). For those, fs-verity will checksum every data block whenever it's accessed and validate it still matches.
block read fails it’s checksum. The signature of the the file is validated against a public key loaded into the kernel keyring. Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.
Is this via dm_verity kernel module? Or thats a completely seperate thing?
It's somewhat inspired by dm-verity, but it's a separate implementation, the only shared logic is the hash computation code in the kernel.
== Scope ==
- Proposal owners
** btrfs kernel enablement work ([
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
landed in 5.15]); see this [
https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-bt...
blog post] for more details
What does this mean for all the other filesystems? They would just be slower since btrfs is computing the trees as part of it's normal checksumming?
fs-verity requires support in the underlying filesystem. If you're using a filesystem that doesn't support it and attempt to enable fs- verity on a file, the ioctl will fail. Note that this is only a concern at runtime, not at build time.
** koji integration: koji will need to add the fs-verity metadata to packages when signing them
Well, koji doesn't sign packages. robosignatory listens for messages on the message bus for koji tagging and based on it's config, it asks sigul to sign rpms and import the signatures into koji.
There's support in robosignatory to ask to sign files (used for the short lived IMA stuff), but I suspect it would need a new ability for this.
Finally who is going to write this? Change owners? Or do you expect robosignatory maintainers to do so?
Thanks for clarifying! Yes, I think robosignatory is likely what we want here. We (the Change owners) expect to do the work, though we'll likely need some advice/help around code review, testing and deployment.
- fs-verity requires filesystem support; currently support for ext4
and f2fs is already available; support for btrfs landed in 5.15
No xfs support?
Correct, XFS doesn't support fs-verity at the moment (though it could be implemented if one wanted to).
That means it would exclude Fedora Server and ELN as they are XFS based.
josh
Hello,
On Thu, Dec 02, 2021 at 07:10:32PM -0500, Josh Boyer wrote:
On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel < devel@lists.fedoraproject.org> wrote:
On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote: ...snip...
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
This is some kind of seperate signing that happens at build time?
Or it's added to the rpm metadata and covered by the normal package signing if/when the package is signed?
As part of the signing flow (e.g. via rpmsign), the Merkle tree is generated and a signature is computed from it, which is then added to the rpm metadata.
IMA received significant pushback on the impact to RPMs and signing implications in Fedora. How does fs-verity compare here?
We wrote up a comparison here: https://fedoraproject.org/wiki/Changes/FsVerityRPM#Relationship_with_IMA
Alternatively/additionally, why is fs-verity worth the hit for Fedora where IMA was not.
The hit is much smaller; per https://fedoraproject.org/wiki/Changes/FsVerityRPM#Merkle_tree_cost - if the plugin is installed, the Merkle tree is stored but it's 1/127th the original file size - the RPM only ships the signature, not the tree itself; per https://fedoraproject.org/wiki/Changes/FsVerityRPM#Signature_overhead_cost in practice we see a minimal to no increase in the size of the RPM
So as proposed in this Change, users can opt in by installing the plugin, otherwise they will be mostly unimpacted.
As discussed in the write-up - IMA does have a richer policy system, and could potentially be integrated (so we use IMA but with a fsverity backend) to get the best of both worlds.
Best regards,
On Thu, Dec 2, 2021 at 7:27 PM Michel Alexandre Salim salimma@fedoraproject.org wrote:
Hello,
On Thu, Dec 02, 2021 at 07:10:32PM -0500, Josh Boyer wrote:
On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel < devel@lists.fedoraproject.org> wrote:
On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote: ...snip...
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
This is some kind of seperate signing that happens at build time?
Or it's added to the rpm metadata and covered by the normal package signing if/when the package is signed?
As part of the signing flow (e.g. via rpmsign), the Merkle tree is generated and a signature is computed from it, which is then added to the rpm metadata.
IMA received significant pushback on the impact to RPMs and signing implications in Fedora. How does fs-verity compare here?
We wrote up a comparison here: https://fedoraproject.org/wiki/Changes/FsVerityRPM#Relationship_with_IMA
Yes, I saw that and I appreciate it. That's a comparison between the two implementations. I am asking about what benefits and use cases fs-verity solves in Fedora. Right now, the change simply says:
"The main benefit is the ability to do block-level verification of RPM-installed files. In turn, this can be used to implement usecase-specific validation and verification policies depending on the environment requirements."
which is also largely true of IMA. The IMA change went into more detailed use cases, which perhaps may have been it's downfall. So can you describe what most Fedora users would use this for or benefit from it? Or if "most users" is not an applicable qualifier, can you at least give some more detailed use cases that you would expect people to use it for?
Alternatively/additionally, why is fs-verity worth the hit for Fedora where IMA was not.
The hit is much smaller; per https://fedoraproject.org/wiki/Changes/FsVerityRPM#Merkle_tree_cost
- if the plugin is installed, the Merkle tree is stored but it's 1/127th the original file size
- the RPM only ships the signature, not the tree itself; per
https://fedoraproject.org/wiki/Changes/FsVerityRPM#Signature_overhead_cost in practice we see a minimal to no increase in the size of the RPM
So as proposed in this Change, users can opt in by installing the plugin, otherwise they will be mostly unimpacted.
OK. I guess I was looking for some side-by-side data comparisons in the overhead between IMA metadata and fs-verity. "1/127th of the original Merkel tree size" doesn't tell me much.
Are there some test runs with numbers to show before/after data for both the RPM size and installed FS usage? Perhaps with an example install. The IMA change attempted to document this and seeing a 1.1% average increase in RPM size was easier to understand.
josh
As discussed in the write-up - IMA does have a richer policy system, and could potentially be integrated (so we use IMA but with a fsverity backend) to get the best of both worlds.
Best regards,
-- Michel Alexandre Salim profile: https://keyoxide.org/michel@michel-slm.name
On Thu, 2021-12-02 at 20:05 -0500, Josh Boyer wrote:
Yes, I saw that and I appreciate it. That's a comparison between the two implementations. I am asking about what benefits and use cases fs-verity solves in Fedora. Right now, the change simply says:
"The main benefit is the ability to do block-level verification of RPM-installed files. In turn, this can be used to implement usecase-specific validation and verification policies depending on the environment requirements."
which is also largely true of IMA. The IMA change went into more detailed use cases, which perhaps may have been it's downfall. So can you describe what most Fedora users would use this for or benefit from it? Or if "most users" is not an applicable qualifier, can you at least give some more detailed use cases that you would expect people to use it for?
Broadly speaking, fs-verity makes it possible to ensure that files that were installed via an RPM have not been modified. It is useful in environments where an attacker might be able to modify system files (say, replace /bin/ls with a compromised version) and you want to protect against that. For example, consider an appliance-like system placed in an untrusted location where you may not be able to control who has physical access (this could be a server, but it could also be a kiosk in an internet point or a school). In this scenario, fs-verity can be one of the building blocks to ensure and maintain system trust.
This Change is mostly about putting in place the necessary plumbing for this to be at all possible. We'll try to expand the Change proposal and flesh out potential usecases a bit more.
OK. I guess I was looking for some side-by-side data comparisons in the overhead between IMA metadata and fs-verity. "1/127th of the original Merkel tree size" doesn't tell me much.
Are there some test runs with numbers to show before/after data for both the RPM size and installed FS usage? Perhaps with an example install. The IMA change attempted to document this and seeing a 1.1% average increase in RPM size was easier to understand.
We've done some empirical testing on this (showing neglibible increases), but still need to gather more formal data. We'll try to prioritize that and add it to the Change once it's available.
Cheers Davide
On Fri, Dec 03, 2021 at 06:08:49PM +0000, Davide Cavalca via devel wrote:
Broadly speaking, fs-verity makes it possible to ensure that files that were installed via an RPM have not been modified. It is useful in environments where an attacker might be able to modify system files (say, replace /bin/ls with a compromised version) and you want to protect against that. For example, consider an appliance-like system placed in an untrusted location where you may not be able to control who has physical access (this could be a server, but it could also be a kiosk in an internet point or a school). In this scenario, fs-verity can be one of the building blocks to ensure and maintain system trust.
I'm unclear about the threat model - this is an attacker who is someone able to overwrite single files (eg. /bin/ls) but cannot turn off the fs-verity system as a whole?
Also if RPM can update /bin/ls then surely an attacker who can widely compromise system files must also be able to update /bin/ls in the same way?
Rich.
On Fri, 3 Dec 2021 at 17:09, Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Dec 03, 2021 at 06:08:49PM +0000, Davide Cavalca via devel wrote:
Broadly speaking, fs-verity makes it possible to ensure that files that were installed via an RPM have not been modified. It is useful in environments where an attacker might be able to modify system files (say, replace /bin/ls with a compromised version) and you want to protect against that. For example, consider an appliance-like system placed in an untrusted location where you may not be able to control who has physical access (this could be a server, but it could also be a kiosk in an internet point or a school). In this scenario, fs-verity can be one of the building blocks to ensure and maintain system trust.
I'm unclear about the threat model - this is an attacker who is someone able to overwrite single files (eg. /bin/ls) but cannot turn off the fs-verity system as a whole?
Also if RPM can update /bin/ls then surely an attacker who can widely compromise system files must also be able to update /bin/ls in the same way?
Or just pad /usr/bin/rpm with some null characters at the end to break its signature and also stop updates from happening. [Or the fs-verity daemon which will report that these problems are occuring. ]
On Sat, 2021-12-04 at 09:37 -0500, Stephen John Smoogen wrote:
Or just pad /usr/bin/rpm with some null characters at the end to break its signature and also stop updates from happening. [Or the fs-verity daemon which will report that these problems are occuring. ]
If the attacker has filesystem access, this wouldn't work, as fs-verity makes the file immutable. If they have block level access to the underlying storage, they can alter the data blocks of the rpm binary, and that would indeed result in failure on the next exec as the signature wouldn't match.
Cheers Davide
On Fri, 2021-12-03 at 22:08 +0000, Richard W.M. Jones wrote:
I'm unclear about the threat model - this is an attacker who is someone able to overwrite single files (eg. /bin/ls) but cannot turn off the fs-verity system as a whole?
Also if RPM can update /bin/ls then surely an attacker who can widely compromise system files must also be able to update /bin/ls in the same way?
Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered.
Where is this useful? For example, fs-verity can help in the scenario where an attacker has out-of-band access to the storage device (say, they pull a hard drive from a colo'd server or a sdcard from an embedded device, or they boot into a liveusb, or they access a VM image directly from a host).
Let's say that happens, and the attacker changes a few blocks of /bin/ls on the device to make it run nefarious code. When you boot your system again, it would fail at exec() time because the Merkle tree wouldn't match.
Let's say that instead the attacker mounts (or gains access to) your filesystem, unlinks /bin/ls and replaces it wholesale with a new copy (hence creating a new inode). The attacker doesn't have your signing key, so they can't resign the file and enable fs-verity on it (they could resign it with their own key, but unless they can then find a way to load its cert into the kernel keyring it won't do much good). To protect against this, you now have a few options:
- you could use a LSM to enforce that exec() can only happen on files with valid fs-verity signatures; this would protect any binary - you could use a launcher booted from secure storage (say, a dm-verity volume, which could even be the initrd), and have this launcher perform the verification; this of course only protects against binaries executed from the launcher, but depending on your threat model it might be enough
Like most security solutions, this isn’t a silver bullet and it’s not something that in and of itself would necessarily prevent all possible attacks. However, fs-verity can be a useful building block in a defense-in-depth approach against specific attacks, depending on your threat model.
Cheers Davide
Dne 10. 12. 21 v 0:08 Davide Cavalca via devel napsal(a):
On Fri, 2021-12-03 at 22:08 +0000, Richard W.M. Jones wrote:
I'm unclear about the threat model - this is an attacker who is someone able to overwrite single files (eg. /bin/ls) but cannot turn off the fs-verity system as a whole?
Also if RPM can update /bin/ls then surely an attacker who can widely compromise system files must also be able to update /bin/ls in the same way?
Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered.
Trying to debug some issue in shell/ruby/python script, will it be possible to modify such file?
Vít
On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
Dne 10. 12. 21 v 0:08 Davide Cavalca via devel napsal(a):
On Fri, 2021-12-03 at 22:08 +0000, Richard W.M. Jones wrote:
I'm unclear about the threat model - this is an attacker who is someone able to overwrite single files (eg. /bin/ls) but cannot turn off the fs-verity system as a whole?
Also if RPM can update /bin/ls then surely an attacker who can widely compromise system files must also be able to update /bin/ls in the same way?
Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered.
Trying to debug some issue in shell/ruby/python script, will it be possible to modify such file?
Any file covered by fs-verity is immutable after installation. So you cannot modify the contents, the kernel refuses. But you can just replace the file (like during an upgrade), and of course copy and edit in a different location. If replaced, no fs-verity checking is done any more by the kernel. There was some talk about high-level solution to prevent such files from being executed, e.g. an LSM module, but no details... (Thinking about this, it would be pretty hard, because the LSM would need to be smart enough to know which files are installed through rpm, and which files are not. I would love to hear more details about what is planned here.)
Zbyszek
Dne 12. 12. 21 v 12:33 Zbigniew Jędrzejewski-Szmek napsal(a):
On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
Dne 10. 12. 21 v 0:08 Davide Cavalca via devel napsal(a):
On Fri, 2021-12-03 at 22:08 +0000, Richard W.M. Jones wrote:
I'm unclear about the threat model - this is an attacker who is someone able to overwrite single files (eg. /bin/ls) but cannot turn off the fs-verity system as a whole?
Also if RPM can update /bin/ls then surely an attacker who can widely compromise system files must also be able to update /bin/ls in the same way?
Once fs-verity is enabled for a given file (which, in the RPM case, happens at package installation time), it cannot be disabled, and the file becomes immutable. One can still rename() or unlink() it (and this is indeed how rpm is able to replace files when upgrading packages), but the actual contents cannot be altered.
Trying to debug some issue in shell/ruby/python script, will it be possible to modify such file?
Any file covered by fs-verity is immutable after installation. So you cannot modify the contents, the kernel refuses. But you can just replace the file (like during an upgrade), and of course copy and edit in a different location. If replaced, no fs-verity checking is done any more by the kernel. There was some talk about high-level solution to prevent such files from being executed, e.g. an LSM module, but no details... (Thinking about this, it would be pretty hard, because the LSM would need to be smart enough to know which files are installed through rpm, and which files are not. I would love to hear more details about what is planned here.)
Thx for explanation.
Would it be possible to document the editing of protected file in the change proposal, probably including example of the best way to do it (is it possible to replace the file by symlink?) Or is there a way to temporary enable the editing with some overlay? Is there any other way to restore the original file except "dnf reinstall"?
I am asking because I am torn between the security and OTOH convenient debugging if something goes wrong. Don't take me wrong I am far from editing system file on daily basis, but I have certainly did that.
Vít
On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote:
Would it be possible to document the editing of protected file in the change proposal, probably including example of the best way to do it (is it possible to replace the file by symlink?) Or is there a way to temporary enable the editing with some overlay? Is there any other way to restore the original file except "dnf reinstall"?
I've added this to the wiki: https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_fil...
You could restore the original file via "dnf reinstall", or by moving it back into place (rename() and unlink() are allowed on fs-verity enabled files).
Cheers Davide
On Tue, Dec 14, 2021 at 1:45 AM Davide Cavalca via devel devel@lists.fedoraproject.org wrote:
On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote:
Would it be possible to document the editing of protected file in the change proposal, probably including example of the best way to do it (is it possible to replace the file by symlink?) Or is there a way to temporary enable the editing with some overlay? Is there any other way to restore the original file except "dnf reinstall"?
I've added this to the wiki: https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_fil...
You could restore the original file via "dnf reinstall", or by moving it back into place (rename() and unlink() are allowed on fs-verity enabled files).
I thought fsverity was about determining at runtime that the system has not been tampered with? But if somebody who has (physical) access to the device can just ... move verified files out of the way and put their own (unverified) files there (which then apparently does not trigger red warning signs?) - doesn't that defeat the whole purpose of enabling fsverity?
Fabio
On Tue, Dec 14, 2021 at 08:08:19PM +0100, Fabio Valentini wrote:
On Tue, Dec 14, 2021 at 1:45 AM Davide Cavalca via devel devel@lists.fedoraproject.org wrote:
On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote:
Would it be possible to document the editing of protected file in the change proposal, probably including example of the best way to do it (is it possible to replace the file by symlink?) Or is there a way to temporary enable the editing with some overlay? Is there any other way to restore the original file except "dnf reinstall"?
I've added this to the wiki: https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_fil...
You could restore the original file via "dnf reinstall", or by moving it back into place (rename() and unlink() are allowed on fs-verity enabled files).
I thought fsverity was about determining at runtime that the system has not been tampered with? But if somebody who has (physical) access to the device can just ... move verified files out of the way and put their own (unverified) files there (which then apparently does not trigger red warning signs?) - doesn't that defeat the whole purpose of enabling fsverity?
That's a good question. fs-verity and dm-verity share the same underlying concept (merkle trees and signature verification by the kernel). So this raises the question that you asked… which can also be phrased as "why would you even use fs-verity, if you can do dm-verity"?
My understanding it the following: fs-verity originated in the Android world where you can have an unprivileged process downloading a file, e.g. a jar. This unprivileged process manages the download, but the file is only trusted and executed when it has a matching signature from some central authority. The file contains the whole app, including all resources, so there is no question of other unverified files being used by the app. And the file can be large enough that it's practical to do chunked verification, since checksumming the whole file on first use would be slow.
We don't really have the same considerations: the download process has full privileges, and the download is exploded into individual files, and more importantly, unpackaged files are also used.
Zbyszek
On Tue, Dec 14, 2021 at 09:18:20PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
My understanding it the following: fs-verity originated in the Android world where you can have an unprivileged process downloading a file, e.g. a jar. This unprivileged process manages the download, but the file is only trusted and executed when it has a matching signature from some central authority. The file contains the whole app, including all resources, so there is no question of other unverified files being used by the app. And the file can be large enough that it's practical to do chunked verification, since checksumming the whole file on first use would be slow.
This does seem rather reminiscent of our LiveCD squashfs situation....
On Tue, Dec 14, 2021 at 4:20 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Tue, Dec 14, 2021 at 08:08:19PM +0100, Fabio Valentini wrote:
I thought fsverity was about determining at runtime that the system has not been tampered with? But if somebody who has (physical) access to the device can just ... move verified files out of the way and put their own (unverified) files there (which then apparently does not trigger red warning signs?) - doesn't that defeat the whole purpose of enabling fsverity?
That's a good question. fs-verity and dm-verity share the same underlying concept (merkle trees and signature verification by the kernel). So this raises the question that you asked… which can also be phrased as "why would you even use fs-verity, if you can do dm-verity"?
dm-verity presents a read-only block device. It can't be modified once created.
fs-verity works at the VFS layer. The block device and file system remain read-write capable, only the fs-verity enabled file contents are read-only. Also, fs-verity only really provides integrity checking in an efficient manner, with an optional mechanism for user space to enforce authentication - the kernel doesn't do this. A further option for user space is auditing.
There is this dance in computer security where you're constantly plugging holes, looking for holes, but also assuming certain holes are definitely filled. It might seem like a contradiction. But of course we say physical access means the system is compromised, so you have to assume that hole is filled or all bets are off. Here you have to assume enough of user space is not compromised enough that you can trust that the fs-verity mechanisms can be used (create and verify), and the reason why you're using this regime is to make certain modifications impossible without detection so that the system doesn't become compromised, or at least compromised in silent fashion. fs-verity does not completely provide a secure and closed system, it's just one aspect of creating a more secure system.
I'd go so far as to say it takes some esoteric knowledge to understand that this is only a "probably important small thing", and not a "critical big thing", to do. This is a few years old, 2018, but it discusses some of the confusing points when fs-verity was merged. https://lwn.net/Articles/763729/
Chris Murphy
My understanding it the following: fs-verity originated in the Android world where you can have an unprivileged process downloading a file, e.g. a jar. This unprivileged process manages the download, but the file is only trusted and executed when it has a matching signature from some central authority. The file contains the whole app, including all resources, so there is no question of other unverified files being used by the app. And the file can be large enough that it's practical to do chunked verification, since checksumming the whole file on first use would be slow.
We don't really have the same considerations: the download process has full privileges, and the download is exploded into individual files, and more importantly, unpackaged files are also used.
-- Chris Murphy
On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
Any file covered by fs-verity is immutable after installation. So you cannot modify the contents, the kernel refuses. But you can just replace the file (like during an upgrade), and of course copy and edit in a different location. If replaced, no fs-verity checking is done any more by the kernel. There was some talk about high-level solution to prevent such files from being executed, e.g. an LSM module, but no details... (Thinking about this, it would be pretty hard, because the LSM would need to be smart enough to know which files are installed through rpm, and which files are not. I would love to hear more details about what is planned here.)
Zbyszek
There is such an LSM that supports fs-verity (and dm-verity), being reviewed right now: IPE
https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd36d3@linux.micr... https://microsoft.github.io/ipe/
On Wed, Dec 15, 2021, at 1:45 PM, Luca Boccassi wrote:
On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
Any file covered by fs-verity is immutable after installation. So you cannot modify the contents, the kernel refuses. But you can just replace the file (like during an upgrade), and of course copy and edit in a different location. If replaced, no fs-verity checking is done any more by the kernel. There was some talk about high-level solution to prevent such files from being executed, e.g. an LSM module, but no details... (Thinking about this, it would be pretty hard, because the LSM would need to be smart enough to know which files are installed through rpm, and which files are not. I would love to hear more details about what is planned here.)
Zbyszek
There is such an LSM that supports fs-verity (and dm-verity), being reviewed right now: IPE
https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd36d3@linux.micr... https://microsoft.github.io/ipe/
Hmm. Some interesting stuff going on there but I would have started with a new SELinux access vector. That'd allow fine-grained constraints, e.g. disallowing `init_t` but allowing `unconfined_service_t`. Possibly also landlock should be able to hook into this. IOW it's not clear to me that a new LSM is the best thing for the ecosystem here.
But bigger picture I'd agree that fs-verity is significantly stronger when coupled with such a policy - strong enough to block exploits like the runc one: https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-cve-...
(Of course that one was *also* stopped by ostree's default read-only bind mount, which I keep saying is not really primarily about security, but hey it worked there)
I said this about the IMA-for-Fedora proposal too - to me the approach of "sign all the RPMs" is not a good idea. Instead the focus should be on ensuring the capability works, and is usable from tools to build custom systems. And this of course runs into a bigger picture question of whether we should emphasize the entry point into Fedora being Anaconda/FCOS like (provision and configure a "pre-built" system) or more like Image Builder and tools like that. I think the answer has to be "we do both" really - it's just hard.
On Wed, Dec 15, 2021, at 1:45 PM, Luca Boccassi wrote:
Hmm. Some interesting stuff going on there but I would have started with a new SELinux access vector. That'd allow fine-grained constraints, e.g. disallowing `init_t` but allowing `unconfined_service_t`. Possibly also landlock should be able to hook into this. IOW it's not clear to me that a new LSM is the best thing for the ecosystem here.
But bigger picture I'd agree that fs-verity is significantly stronger when coupled with such a policy - strong enough to block exploits like the runc one: https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-c...
We use this in production, and those kind of constraints are just not enough, because there's too many obvious ways around them. What we needed is for the kernel to enforce that only signed code shall run, and thus IPE was born, so that when coupled with dm-verity/fs-verity it allows to enforce a policy like that.
Some more questions about how the verification happens… IIUC, I need to load some keys to the kernel keyring, and then set fs.verity.require_signatures.
Where do the keys come from? How are the keys themselves verified? At what time are they loaded and by whom?
(Let's say that I'm an attacker with access to the file system. If the keys are loaded from the file system, can I just drop in a rogue key, similarly to what happens when new keys are distributed as part of distro upgrades?)
If fs-verity verification prevents me from successfully modifying or replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is there anything which hinders just adding /etc/systemd/system/foo.service that does whatever I want?
On Thu, Dec 09, 2021 at 11:08:53PM +0000, Davide Cavalca via devel wrote:
- you could use a LSM to enforce that exec() can only happen on files
with valid fs-verity signatures; this would protect any binary
- you could use a launcher booted from secure storage (say, a dm-verity
Is there some LSM module like this ready for use?
Zbyszek
* Zbigniew Jędrzejewski-Szmek:
Some more questions about how the verification happens… IIUC, I need to load some keys to the kernel keyring, and then set fs.verity.require_signatures.
Where do the keys come from? How are the keys themselves verified? At what time are they loaded and by whom?
(Let's say that I'm an attacker with access to the file system. If the keys are loaded from the file system, can I just drop in a rogue key, similarly to what happens when new keys are distributed as part of distro upgrades?)
If fs-verity verification prevents me from successfully modifying or replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is there anything which hinders just adding /etc/systemd/system/foo.service that does whatever I want?
Even if we do that, we can't rule out two scenarios:
* The attacker builds a (perhaps unrelated) Fedora package and lets Fedora sign it with the official key. The file signature is as good as any other produced by Fedora.
(Alternatively, the attacker reuses file contents signed elsewhere in Fedora because it has an unintended side effect when used in a different context. This attack requires finding a suitable gadget in some Fedora package, but a dodgy Fedora package build is not needed.)
* The attacker (who could be the legitimate user) changes the system configuration so that the feature is disabled during boot.
Neither we or hardware suppliers installing Fedora on their devices are permitted to plug the second hole because it's required for software freedom (or Fedora would have to share its private signing keys).
The combination of these two unsolvable issues suggests to me that anyone who wants to deploy this is better off with their own trust root, and that approach will include their particular version of key management as well. But this also means that pre-computed file signatures are not particular useful to them. They would have to discard them anyway before deployment. Their own signing process might as well check the RPM header signature instead.
Thanks, Florian
On 12/12/21 8:26 AM, Florian Weimer wrote:
- Zbigniew Jędrzejewski-Szmek:
Some more questions about how the verification happens… IIUC, I need to load some keys to the kernel keyring, and then set fs.verity.require_signatures.
Where do the keys come from? How are the keys themselves verified? At what time are they loaded and by whom?
(Let's say that I'm an attacker with access to the file system. If the keys are loaded from the file system, can I just drop in a rogue key, similarly to what happens when new keys are distributed as part of distro upgrades?)
If fs-verity verification prevents me from successfully modifying or replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is there anything which hinders just adding /etc/systemd/system/foo.service that does whatever I want?
There is not. dm-verity is the only solution to that that I am aware of in Linux. dm-verity also solves the problem that someone could tamper with the block device and exploit a vulnerability in the kernel's filesystem drivers.
Even if we do that, we can't rule out two scenarios:
The attacker builds a (perhaps unrelated) Fedora package and lets Fedora sign it with the official key. The file signature is as good as any other produced by Fedora.
(Alternatively, the attacker reuses file contents signed elsewhere in Fedora because it has an unintended side effect when used in a different context. This attack requires finding a suitable gadget in some Fedora package, but a dodgy Fedora package build is not needed.)
NetBSD’s Veriexec has a neat solution to this problem, but it is significantly more complicated. It uses additional metadata to validate a file.
- The attacker (who could be the legitimate user) changes the system configuration so that the feature is disabled during boot.
Neither we or hardware suppliers installing Fedora on their devices are permitted to plug the second hole because it's required for software freedom (or Fedora would have to share its private signing keys).
The obvious solution to the second problem is some form of measured boot: if the feature is disabled, it shows up in the system measurements, and access to various secrets may be denied by the TPM. Android uses a similar solution: at least on Google Pixel devices, one can unlock the bootloader, but doing so erases all data on the device.
The combination of these two unsolvable issues suggests to me that anyone who wants to deploy this is better off with their own trust root, and that approach will include their particular version of key management as well. But this also means that pre-computed file signatures are not particular useful to them. They would have to discard them anyway before deployment. Their own signing process might as well check the RPM header signature instead.
+1 on this. There have also been bugs in RPM's handling of IMA signatures, and fs-verity signature handling could have similar issues. Since IMA and fs-verity signatures are currently stored in the signature header, they can be modified without changing the package signature, so they increase the attack surface of RPM.
We don't have a proof of concept for the LSM module. I agree with you that in practice it would probably need to implement some kind of "list of files we care about", but I do not have an intelligent opinion about that.
Based on Roberto's comment in a different sub-thread, there could be some ongoing work integrating with IMA that might be past the hypothetical stage?
We don't have a proof of concept for the LSM module. I agree with you that in practice it would probably need to implement some kind of "list of files we care about", but I do not have an intelligent opinion about that.
Based on Roberto's comment in a different sub-thread, there could be some ongoing work integrating with IMA that might be past the hypothetical stage?
IPE supports fs-verity and dm-verity, and allows to write a policy such as "only allow execution of binaries/libraries from fs-verity verified files or dm-verity verified volumes):
https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd36d3@linux.micr... https://microsoft.github.io/ipe/
On Thu, 2021-12-02 at 19:10 -0500, Josh Boyer wrote:
On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel devel@lists.fedoraproject.org wrote:
Correct, XFS doesn't support fs-verity at the moment (though it could be implemented if one wanted to).
That means it would exclude Fedora Server and ELN as they are XFS based.
There isn't any impact on XFS-based editions by default. Building and signing RPMs with fs-verity works fine, as there's no filesystem dependency for that. Installing RPMs also works fine, with one caveat: if you're running a system with an unsupported filesystem and have rpm- plugin-fsverity (which is not installed by default), fs-verity will not be enabled for the installed files so there will be no verification (but the RPM installation itself should succeed).
Cheers Davide
On Fri, Dec 3, 2021 at 1:15 PM Davide Cavalca dcavalca@fb.com wrote:
On Thu, 2021-12-02 at 19:10 -0500, Josh Boyer wrote:
On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel devel@lists.fedoraproject.org wrote:
Correct, XFS doesn't support fs-verity at the moment (though it could be implemented if one wanted to).
That means it would exclude Fedora Server and ELN as they are XFS based.
There isn't any impact on XFS-based editions by default. Building and signing RPMs with fs-verity works fine, as there's no filesystem dependency for that. Installing RPMs also works fine, with one caveat: if you're running a system with an unsupported filesystem and have rpm- plugin-fsverity (which is not installed by default), fs-verity will not be enabled for the installed files so there will be no verification (but the RPM installation itself should succeed).
Yes, thank you. I meant it would render those Editions/projects unable to use this Change. It's not the end of the world, just an observation.
josh
On 12/2/21 4:32 PM, Davide Cavalca via devel wrote:
There's support in robosignatory to ask to sign files (used for the short lived IMA stuff), but I suspect it would need a new ability for this.
Finally who is going to write this? Change owners? Or do you expect robosignatory maintainers to do so?
Thanks for clarifying! Yes, I think robosignatory is likely what we want here. We (the Change owners) expect to do the work, though we'll likely need some advice/help around code review, testing and deployment.
Could this be directly added to rpm instead of an external tool set?
rpm --addsign ./$RPM to rpm --addsign --addfsverity ./$RPM
I'm thinking about those who might also want to use it... end-users and third-party repos such as RPMFusion.
The signature size is roughly proportional to the number of files in the package.
Can you explain how the signature is performed? I assume that the verity top-level hash is calculated for each file and then … ? And if you have all the per-file hashes, why not do one more level of Merkle and calculate a single hash for all the files in the rpm and sign that (so that the signature overhead is of fixed size)?
Zbyszek
The top-level hash is calculated for each file, then that hash is signed with the inputted rsa key pair and the signed hash is appended to the array of signed hashes in the rpm metadata. I am guessing the way we worded the proposal is a little unclear because we call it "the" signature when it's one rpm metadata item that's an array of the signatures.
fs-verity the kernel feature operates on a per-file basis, and since the ultimate goal is to deliver fs-verity enabled files on the installer's system, we need each file's signature in the rpm. At install, we call the fs-verity enable ioctl for each file, passing in its signature to make use of the kernel authentication functionality.
I'm happy to explain the exact flow at signing and at install in more detail if that would be helpful.
Boris
On Fri, Dec 03, 2021 at 05:31:21PM -0000, Boris Burkov via devel wrote:
The top-level hash is calculated for each file, then that hash is signed with the inputted rsa key pair and the signed hash is appended to the array of signed hashes in the rpm metadata. I am guessing the way we worded the proposal is a little unclear because we call it "the" signature when it's one rpm metadata item that's an array of the signatures.
fs-verity the kernel feature operates on a per-file basis, and since the ultimate goal is to deliver fs-verity enabled files on the installer's system, we need each file's signature in the rpm. At install, we call the fs-verity enable ioctl for each file, passing in its signature to make use of the kernel authentication functionality.
What exactly is appended to the rpm: a) the merkle tree b) the top-level hash from the merkle tree c) the signature for b d) some combination of a, b, c?
Above you seem to say "b+c" ("signed hash").
In the Change page:
at build time, we compute the Merkle tree for the files within a package, then sign it and ship it as part of the rpm metadata;
…which is "a".
"c" would make sense to me.
Please clarify.
Zbyszek
Sorry this wasn't clear.
The rpm carries just 'c' (as well as some small, fixed-size metadata for interpreting it, like hash algorithm)
Just to explain that comment which suggested 'a': we have to compute the Merkle tree at build time in order to get a root hash to sign. The Merkle tree is then fully re-computed by the fs-verity code in the kernel for storage by the local filesystem when the rpm is ultimately installed.
On Mon, Dec 13, 2021 at 07:32:34PM -0000, Boris Burkov via devel wrote:
Sorry this wasn't clear.
The rpm carries just 'c' (as well as some small, fixed-size metadata for interpreting it, like hash algorithm)
Just to explain that comment which suggested 'a': we have to compute the Merkle tree at build time in order to get a root hash to sign. The Merkle tree is then fully re-computed by the fs-verity code in the kernel for storage by the local filesystem when the rpm is ultimately installed.
OK, so please adjust the change page not to say "it", but "the signature" or something like that.
Zbyszek
Perhaps I glossed over it in the description, but what is the expected user experience in the event of a RPM fs-verity mismatch/error?
I think there are two cases of interest:
1) a file or signature in the rpm is corrupted, the signature doesn't have a matching cert installed, etc... in this case, if the plugin is present, when you attempt to install the rpm the verity enable ioctl will explicitly fail, and presumably so will the rpm install. You can see most of the details for this sort of case here in the rpm plugin code: https://github.com/rpm-software-management/rpm/blob/master/plugins/fsverity.....
2) after installation, a file from an fs-verity enabled rpm gets one or more blocks corrupted The first read of a corrupted block from disk (the good uncorrupted page might survive in page cache for a while) will result in EIO for read-like system calls and SIGBUS if the file is mapped (executables, mmap).
I omitted one more interesting case.
If the verity metadata (signature, root hash) is corrupted after installation but before the file is opened, then opening/exec-ing the file can fail. Also, if pages from a binary read in during the exec itself are corrupted, the system call itself could fail (rather than the process getting sigbus like for a random page during execution)
I think there are two cases of interest:
- a file or signature in the rpm is corrupted, the signature doesn't have a matching
cert installed, etc... in this case, if the plugin is present, when you attempt to install the rpm the verity enable ioctl will explicitly fail, and presumably so will the rpm install. You can see most of the details for this sort of case here in the rpm plugin code: https://github.com/rpm-software-management/rpm/blob/master/plugins/fsveri....
- after installation, a file from an fs-verity enabled rpm gets one or more blocks
corrupted The first read of a corrupted block from disk (the good uncorrupted page might survive in page cache for a while) will result in EIO for read-like system calls and SIGBUS if the file is mapped (executables, mmap).
Thanks for the information. So in the event of an error, it's expected that the user would be informed that the error was due to a rpm-fsverity check and they could potentially reinstall the rpm from a trusted source to resolve the problem? Of course there's the underlying issue of _why_ it happened, but from the standpoint of wanting an immediate path forward.
Errors at installation time should be fully diagnosable, and even if the output today doesn't make it totally obvious what happened, it would be easy to fix in rpm.
The errors post-install are a bit trickier. Imagine you install your rpm, and kick off some long running daemon from it. A month later, a block gets corrupted in a way fs checksums don't catch (e.g. ext4, btrfs nodatasum, evil maid), and suddenly that daemon receives a SIGBUS and crashes. You would be able to see clearly that it was a verity issue in dmesg, but I don't think the binary could reasonably know what happened or write a meaningful log. In that sense, I think it's actually pretty similar to the experience if you have corruption in your disk and start getting btrfs checksum errors on a file--you'd have to look in dmesg to know why your file is broken.
The middle ground is when opening/exec-ing the file fails. In that case, you might get a sufficiently specific error code you could figure out it's verity, and the full error would be in dmesg as well.
On Fri, 2021-12-03 at 12:21 +0100, Vitaly Zaitsev via devel wrote:
On 02/12/2021 20:36, Ben Cotton wrote:
Enable the use of fsverity for installed RPM files validation.
-1. RPM already supports files validation and this feature will waste file system space.
To clarify: RPM does support files validation, but fs-verity is more than just that. With RPM, the validation only happens on install time, and when one runs rpm -V manually. With fs-verity, the validation happens on-demand whenever a block of a file that originated from an RPM is accessed. This means, for example, that if an attacker replaces /bin/ls on disk with a compromised one, the next time it's read from disk (e.g. because you ran it) you will see a validation failure and the syscall will be blocked, preventing the compromised code from being executed.
About filesystem usage: unless you install rpm-plugin-fsverity (which is not and will not be installed by default), there is no disk space increase for verity-signed RPM packages. If you do install rpm-plugin- fsverity, some disk space will be used for the Merkle tree as described in the Change.
Cheers Davide
On Friday, 03 December 2021 at 18:49, Davide Cavalca via devel wrote: [...]
About filesystem usage: unless you install rpm-plugin-fsverity (which is not and will not be installed by default), there is no disk space increase for verity-signed RPM packages. If you do install rpm-plugin- fsverity, some disk space will be used for the Merkle tree as described in the Change.
How do I reclaim that space after removing rpm-plugin-fsverity?
Regards, Dominik
You could reinstall the rpm, the fs will reclaim the verity metadata along with the rest of the old file.
Davide Cavalca via devel wrote:
To clarify: RPM does support files validation, but fs-verity is more than just that. With RPM, the validation only happens on install time, and when one runs rpm -V manually. With fs-verity, the validation happens on-demand whenever a block of a file that originated from an RPM is accessed. This means, for example, that if an attacker replaces /bin/ls on disk with a compromised one, the next time it's read from disk (e.g. because you ran it) you will see a validation failure and the syscall will be blocked, preventing the compromised code from being executed.
This means that there is a performance cost in addition to the disk space cost (because something has to compute those checksums each time the file is acessed). It also means that it is harder for users to exercise their right to modify the Free Software (because replacing or patching RPM-installed binaries will lead to them failing to execute).
About filesystem usage: unless you install rpm-plugin-fsverity (which is not and will not be installed by default), there is no disk space increase for verity-signed RPM packages. If you do install rpm-plugin- fsverity, some disk space will be used for the Merkle tree as described in the Change.
Since the change also adds to the metadata in the RPM, that means that it also increases the size of the RPMs. With keepcache=1, this also translates to increased disk space use. But even if the user does not keep cached RPMs, the download sizes will increase, which can cost time and for some users even money.
Kevin Kofler
On Sat, 2021-12-04 at 23:46 +0100, Kevin Kofler via devel wrote:
Davide Cavalca via devel wrote:
To clarify: RPM does support files validation, but fs-verity is more than just that. With RPM, the validation only happens on install time, and when one runs rpm -V manually. With fs-verity, the validation happens on-demand whenever a block of a file that originated from an RPM is accessed. This means, for example, that if an attacker replaces /bin/ls on disk with a compromised one, the next time it's read from disk (e.g. because you ran it) you will see a validation failure and the syscall will be blocked, preventing the compromised code from being executed.
This means that there is a performance cost in addition to the disk space cost (because something has to compute those checksums each time the file is acessed).
There's only a performance cost if fs-verity is actually enabled (which is not in scope for this proposal). The checksums are computed on- demand on a per-block basis, so you only end up checksumming the pages you're actually accessing.
It also means that it is harder for users to exercise their right to modify the Free Software (because replacing or patching RPM- installed binaries will lead to them failing to execute).
There's nothing stopping the user from loading their own key in the kernel keyring and then installing their own locally-build RPM that has been verity-signed with their own key. And again, this only becomes a concern if one is actually enabling fs-verity in the first place.
Since the change also adds to the metadata in the RPM, that means that it also increases the size of the RPMs. With keepcache=1, this also translates to increased disk space use. But even if the user does not keep cached RPMs, the download sizes will increase, which can cost time and for some users even money.
That's correct, there will be (modest) increase in the size of the RPMs. We're going to collect some more data to quantify this more concretely.
Cheers Davide
Vitaly Zaitsev via devel wrote:
-1. RPM already supports files validation and this feature will waste file system space.
I agree with you.
This is yet another "feature" that increases both the size of RPMs and, if enabled by default, the size of default installations. We need to stop tolerating creeping bloat!
So -1 to the Change proposal.
Kevin Kofler
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
[...]
Note that the Merkle tree is ''not'' shipped with the RPM itself (only its signature is)
In that case, "ship it" above should be changed to "ship the signature", unless this is some distinction between "the RPM metadata" and "the RPM itself".
If I enable FS-verity and later find that I need to patch a file to fix some problem, how do I as the sysadmin tell Linux that this change is authorized? Do I disable FS-verity for that specific file? Disable FS-verity globally? Add my own key to the kernel's keyring? Build and sign my own RPM package?
What prevents an attacker from doing the same?
Will files under /etc be covered, or will local configuration still be possible?
Björn Persson
If I enable FS-verity and later find that I need to patch a file to fix some problem, how do I as the sysadmin tell Linux that this change is authorized? Do I disable FS-verity for that specific file? Disable FS-verity globally? Add my own key to the kernel's keyring? Build and sign my own RPM package?
What prevents an attacker from doing the same?
I think this is a good, fair point and is a serious tradeoff in authenticating distributed files. However, I believe it should be possible for the user to securely configure a keypair and load the certificate in the fs-verity keyring s.t. they can sign the files they craft themselves, without allowing an attacker to, just like they would to normally sign things. So you could copy the file, modify it to your liking, (or just rebuild the rpm locally) then enable verity with your own signature.
On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
Enable the use of fsverity for installed RPM files validation.
Can we use this to validate the install media at runtime rather than as a separate boot step?
On Do, 02.12.21 14:36, Ben Cotton (bcotton@redhat.com) wrote:
Hmm, so what I am really missing on the feature page: what's the attack scenario here? Usually security features come with an attack scenario they are supposed to address. But there's no discussion about that.
This protects file contents, not the metada, right? So what about the metadata? if I see a fs-verity enabled inode with libssl.so data in it today, and it's a vulnerably version, and I make a hardlink to it, and then it gets replaced by a fixed version (with a slightly updated name) — how you intend to make sure, that i can't fool you into loading my copy of the old file but under the new name?
What about signing anyway? What is this precisely signed with? The keys for that when are they rolled over? Who manages those keys? The tea for that, who's doing that in Fedora? Is there any protection against downgrades between RPM package versions? Does this in any way protect combinations of binaries/libraries? I mean, pretty much all programs we ship consist of a large number of ELF objects, and you probably need to sign the combination of them, but this model doesn't look like it offers that at all? If a security bug is found in library X, how is it taken out of equation? What would the workflow for that look like on the Fedora side of things?
The security model seems really strange to me I must say: so much infrastructure, and it doesn't protect at all how our programs are usually combined?
Puzzled,
Lennart
On Do, 02.12.21 14:36, Ben Cotton (bcotton(a)redhat.com) wrote:
Hmm, so what I am really missing on the feature page: what's the attack scenario here? Usually security features come with an attack scenario they are supposed to address. But there's no discussion about that.
Good point, we were skimpy on that. In case you missed it, Davide just laid it out in his reply to the same question from Richard Jones higher in the thread. The short answer is that just this change as-is has no practical threat model, we viewed it as an enablement step that we believe has a realistic path to broadly authenticated rpm contents. Alternatively, it does give a sophisticated user an opportunity to build something like that for themselves as well.
This protects file contents, not the metada, right? So what about the metadata? if I see a fs-verity enabled inode with libssl.so data in it today, and it's a vulnerably version, and I make a hardlink to it, and then it gets replaced by a fixed version (with a slightly updated name) — how you intend to make sure, that i can't fool you into loading my copy of the old file but under the new name?
If the file is still properly signed with a trusted matching certificate in the kernel keyring we can't/won't do anything about it.
What about signing anyway? What is this precisely signed with? The keys for that when are they rolled over? Who manages those keys? The tea for that, who's doing that in Fedora? Is there any protection against downgrades between RPM package versions? Does this in any way protect combinations of binaries/libraries? I mean, pretty much all programs we ship consist of a large number of ELF objects, and you probably need to sign the combination of them, but this model doesn't look like it offers that at all? If a security bug is found in library X, how is it taken out of equation? What would the workflow for that look like on the Fedora side of things?
I don't have a particularly specific answer to all of these questions, unfortunately. Perhaps someone with more knowledge about Fedora's existing key management can weigh in. But imagining we have the LSM hooks setup so that we never load anything executable without it being signed by a trusted key, then in my mind, banishing a bad shared library would require revoking the certificate.
I think one could draw a reasonable parallel between package signing and file signing here, though. Users assemble meaningful system functionality out of a collection of packages, which are each individually signed. One of those packages could then be found to be compromised, and need to be removed, otherwise a user could download and install a perfectly valid signed package and have a bad time, which undermines the value of the signing. Is there something about composing executables out of files that is meaningfully different in this sense from composing systems out of packages?
The security model seems really strange to me I must say: so much infrastructure, and it doesn't protect at all how our programs are usually combined?
Puzzled,
Lennart
On Do, 09.12.21 23:55, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
On Do, 02.12.21 14:36, Ben Cotton (bcotton(a)redhat.com) wrote:
Hmm, so what I am really missing on the feature page: what's the attack scenario here? Usually security features come with an attack scenario they are supposed to address. But there's no discussion about that.
Good point, we were skimpy on that. In case you missed it, Davide just laid it out in his reply to the same question from Richard Jones higher in the thread. The short answer is that just this change as-is has no practical threat model, we viewed it as an enablement step that we believe has a realistic path to broadly authenticated rpm contents. Alternatively, it does give a sophisticated user an opportunity to build something like that for themselves as well.
This sounds strange to me. I mean, it's fine if new features initially only implement a subset of what they eventually are supposed to accomplish, but it appears to me you add only infrastructure without *any* use? Infrastructure that has no consuming feature typically doesn't work. Experience tells us that. Hence: I think it's fine if you don't go from 0 to 100 right away but there should be at least *some* benefit if you add such complex infra to the distro.
This protects file contents, not the metada, right? So what about the metadata? if I see a fs-verity enabled inode with libssl.so data in it today, and it's a vulnerably version, and I make a hardlink to it, and then it gets replaced by a fixed version (with a slightly updated name) — how you intend to make sure, that i can't fool you into loading my copy of the old file but under the new name?
If the file is still properly signed with a trusted matching certificate in the kernel keyring we can't/won't do anything about it.
But that's an extremely weak property then? If I can have "ping" and "poweroff" binaries both signed by Fedora, but then swap them without the signing being able to detect that, why do this at all?
What about signing anyway? What is this precisely signed with? The keys for that when are they rolled over? Who manages those keys? The tea for that, who's doing that in Fedora? Is there any protection against downgrades between RPM package versions? Does this in any way protect combinations of binaries/libraries? I mean, pretty much all programs we ship consist of a large number of ELF objects, and you probably need to sign the combination of them, but this model doesn't look like it offers that at all? If a security bug is found in library X, how is it taken out of equation? What would the workflow for that look like on the Fedora side of things?
I don't have a particularly specific answer to all of these questions, unfortunately. Perhaps someone with more knowledge about Fedora's existing key management can weigh in. But imagining we have the LSM hooks setup so that we never load anything executable without it being signed by a trusted key, then in my mind, banishing a bad shared library would require revoking the certificate.
Uff. That might work in focussed deployments, but across a distribution like Fedora? Everytime there's an exploitable bug in *any* of the software Fedora ships you want to invalidate basically the *whole* distro? That sounds absolutely unmanagable and unrealistic to me.
I think one could draw a reasonable parallel between package signing and file signing here, though. Users assemble meaningful system functionality out of a collection of packages, which are each individually signed. One of those packages could then be found to be compromised, and need to be removed, otherwise a user could download and install a perfectly valid signed package and have a bad time, which undermines the value of the signing. Is there something about composing executables out of files that is meaningfully different in this sense from composing systems out of packages?
RPMs sign metadata. You can't trick RPM that easily into accepting "ping" as "poweroff" binary or vice versa. I mean, RPM's security model is from the 1990's, and it only does validation on request, not on access. So it's not great, but it *did* get right to authenticate a whole package in combination with all its metadata. And since versioned deps are part of that metadata you can express relationships.
Still, this feature appears totally not throught to the end. I don't get the usecase?
Lennart
-- Lennart Poettering, Berlin
On Thu, 2 Dec 2021 at 14:37, Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/FsVerityRPM
== Summary ==
Enable the use of fsverity for installed RPM files validation.
== Owners ==
- Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris
Burkov]], [[User:Filbranden|Filipe Brandenburger]], [[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew Almond]]
- Email: dcavalca@fb.com, borisb@fb.com, filbranden@fb.com,
michel@fb.com, malmond@fb.com
....
** koji integration: koji will need to add the fs-verity metadata to packages when signing them
I am going to say that this is a bigger sticking point than people are realising and this is not a F36 but a F37 change.
The changes to koji, signing infrastructure and testing have to be put in place before January 19 2022 (2022-01-19) for the mass rebuild[1]. They need to have been in staging and testing before then so that bugs could be worked out and various groups could work things out. Due to various end of year holidays and 'use your PTO before its lost' schedules, many of the developers who could sign off on the changes to koji/bodhi/whatever-testing/sigul/etc depending on the changes needed are not going to be available. Even if the changes are trivial ones, this is a very short window to land things.
At best for F36, I could see the changes getting put into Fedora before the beta and some packages signed.
If you want this for F37, then all that work needs to be ready and done by (2022-07-20)[2].
[1] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html [2] https://fedorapeople.org/groups/schedule/f-37/f-37-key-tasks.html
- at run time, if the fsverity rpm plugin is enabled, rpm will install
the fsverity signature key and enable fsverity on files that are installed.
This requires CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y. Currently our kernels are built without that. It seems like a simple addition (the amount of code guarded by this is very small), but this should be added to the Scope.
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
There was already some chatter about signatures and verification in the thread, but I didn't see a definite answer, and I think this is the part that is currently the most underspecified.
In Fedora, we use a new package signing key for each Fedora release. What key would be used for the fs-verity signatures: the same key, a separate key? Edit: I see that the Change page says a dedicated key is used.
OK, so is this key rotated one per release or some different schedule? How would the public part of the key be delivered to users?
How does one actually enable signature verification? Let's say I have a kernel with CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y and rpm-plugin-fsverity installed, and I installed some rpms, and the files from those rpms now have now had the FS_IOC_ENABLE_VERITY ioctl done.
IIUC, I need to do some steps at each boot: 1. add all the keys to the keyring 2. set sysctl fs.verity.require_signatures=1
So… in 1., do I always load all the keys that Fedora has used for this purpose, in case there are still some files on disk? Or is there some mechanism to say that e.g. keys older than F(N-2) are now not necessary?
Who does 2.?
I think it'd be hard to enable this during a system upgrade: one would need to reinstall all rpms (with new rpms carrying the fsverity metadata) or some files become unreadable once 2. is done. This brings a question: is there some rpm virtual Provides/Requires to specify that the fs-verity stuff is present? I assume the user would want to triple-check that they don't have any rpms without the metadata before enabling verification.
Zbyszek
P.S. Those questions are about the mechanism, not policy or the threat model. I have some questions about that, but I'll ask separately, since this mail is lengthy already.
In Fedora, we use a new package signing key for each Fedora release. What key would be used for the fs-verity signatures: the same key, a separate key? Edit: I see that the Change page says a dedicated key is used.
Hi all
I'm doing related work in this area. I'll provide some additional thoughts.
Probably, it could be possible to use the package signing key for the fsverity signatures. However, this would require the kernel to be able to load the PGP keys in the builtin keyring and support PGP signatures.
David Howells did that some time ago, and I adapted his patches for the latest kernel. Without going into too much detail, I've modified the kernel build to take the Fedora keys. They are available to verify PGP signatures (I use this feature to verify RPM headers).
If there is interest, I could propose the patch set to the kernel mailing list.
IIUC, I need to do some steps at each boot:
- add all the keys to the keyring
- set sysctl fs.verity.require_signatures=1
So… in 1., do I always load all the keys that Fedora has used for this purpose, in case there are still some files on disk? Or is there some mechanism to say that e.g. keys older than F(N-2) are now not necessary?
Who does 2.?
I think it'd be hard to enable this during a system upgrade: one would need to reinstall all rpms (with new rpms carrying the fsverity metadata) or some files become unreadable once 2. is done. This brings a question: is there some rpm virtual Provides/Requires to specify that the fs-verity stuff is present? I assume the user would want to triple-check that they don't have any rpms without the metadata before enabling verification.
The kernel does not enforce signature verification unless fsverity is explicitly enabled on a file. I guess the rpm plugin can be configured to enable fsverity only if it finds a signature in the RPM header. That would allow a mixed configuration where some files are protected, other not.
This does not seem ideal for mandatory protection, where you want to be sure that integrity is checked, even if only on a subset of files (e.g. executables and shared libraries), regardless of whether fsverity was enabled or not on a file.
It could be task of the security subsystem to do this type of enforcement. At the moment, IMA and IPE (Integrity Protection Enforcement) are planning to support fsverity and do the enforcement based on a policy.
Roberto
HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063 Managing Director: Li Peng, Zhong Ronghua
On Fri, Dec 17, 2021 at 5:14 AM Roberto Sassu via devel devel@lists.fedoraproject.org wrote:
In Fedora, we use a new package signing key for each Fedora release. What key would be used for the fs-verity signatures: the same key, a separate key? Edit: I see that the Change page says a dedicated key is used.
Hi all
I'm doing related work in this area. I'll provide some additional thoughts.
Probably, it could be possible to use the package signing key for the fsverity signatures. However, this would require the kernel to be able to load the PGP keys in the builtin keyring and support PGP signatures.
David Howells did that some time ago, and I adapted his patches for the latest kernel. Without going into too much detail, I've modified the kernel build to take the Fedora keys. They are available to verify PGP signatures (I use this feature to verify RPM headers).
If there is interest, I could propose the patch set to the kernel mailing list.
That would be phenomenal. That would allow us to tie things to our existing process of key rotation for each Fedora release and leverage the existing infrastructure for package signing for fsverity signatures too.
IIUC, I need to do some steps at each boot:
- add all the keys to the keyring
- set sysctl fs.verity.require_signatures=1
So… in 1., do I always load all the keys that Fedora has used for this purpose, in case there are still some files on disk? Or is there some mechanism to say that e.g. keys older than F(N-2) are now not necessary?
Who does 2.?
I think it'd be hard to enable this during a system upgrade: one would need to reinstall all rpms (with new rpms carrying the fsverity metadata) or some files become unreadable once 2. is done. This brings a question: is there some rpm virtual Provides/Requires to specify that the fs-verity stuff is present? I assume the user would want to triple-check that they don't have any rpms without the metadata before enabling verification.
The kernel does not enforce signature verification unless fsverity is explicitly enabled on a file. I guess the rpm plugin can be configured to enable fsverity only if it finds a signature in the RPM header. That would allow a mixed configuration where some files are protected, other not.
This does not seem ideal for mandatory protection, where you want to be sure that integrity is checked, even if only on a subset of files (e.g. executables and shared libraries), regardless of whether fsverity was enabled or not on a file.
It could be task of the security subsystem to do this type of enforcement. At the moment, IMA and IPE (Integrity Protection Enforcement) are planning to support fsverity and do the enforcement based on a policy.
So that means there *will* be a policy control mechanism for this enforced by the kernel, cool!
-- 真実はいつも一つ!/ Always, there's only one truth!
From: Neal Gompa [mailto:ngompa13@gmail.com] Sent: Friday, December 17, 2021 11:17 AM On Fri, Dec 17, 2021 at 5:14 AM Roberto Sassu via devel devel@lists.fedoraproject.org wrote:
In Fedora, we use a new package signing key for each Fedora release. What key would be used for the fs-verity signatures: the same key, a separate key? Edit: I see that the Change page says a dedicated key is used.
Hi all
I'm doing related work in this area. I'll provide some additional thoughts.
Probably, it could be possible to use the package signing key for the fsverity signatures. However, this would require the kernel to be able to load the PGP keys in the builtin keyring and support PGP signatures.
David Howells did that some time ago, and I adapted his patches for the latest kernel. Without going into too much detail, I've modified the kernel build to take the Fedora keys. They are available to verify PGP signatures (I use this feature to verify RPM headers).
If there is interest, I could propose the patch set to the kernel mailing list.
That would be phenomenal. That would allow us to tie things to our existing process of key rotation for each Fedora release and leverage the existing infrastructure for package signing for fsverity signatures too.
Great. I will propose the patch set again. For another Fedora feature I'm proposing (related to this one):
https://fedoraproject.org/wiki/Changes/DIGLIM
I went as far as to be able to appraise with IMA existing RPM headers (I added support for PGP appended signatures).
I'm using this information in a different way: loading the file digests from the RPM header to the kernel and querying them with IMA during appraisal.
That would work with fsverity digests too. If the concern is the increased package size, adding fsverity digests to the RPM header instead of signatures might be more acceptable.
I made a small demo:
https://lore.kernel.org/linux-integrity/a16a628b9e21433198c490500a987121@hua...
where loading of kernel modules is granted by IPE if their fsverity digest is found in the DIGLIM hash table.
Roberto
HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063 Managing Director: Li Peng, Zhong Ronghua
IIUC, I need to do some steps at each boot:
- add all the keys to the keyring
- set sysctl fs.verity.require_signatures=1
So… in 1., do I always load all the keys that Fedora has used for this purpose, in case there are still some files on disk? Or is there some mechanism to say that e.g. keys older than F(N-2) are now not necessary?
Who does 2.?
I think it'd be hard to enable this during a system upgrade: one would need to reinstall all rpms (with new rpms carrying the fsverity metadata) or some files become unreadable once 2. is done. This brings a question: is there some rpm virtual Provides/Requires to specify that the fs-verity stuff is present? I assume the user would want to triple-check that they don't have any rpms without the metadata before enabling verification.
The kernel does not enforce signature verification unless fsverity is explicitly enabled on a file. I guess the rpm plugin can be configured to enable fsverity only if it finds a signature in the RPM header. That would allow a mixed configuration where some files are protected, other not.
This does not seem ideal for mandatory protection, where you want to be sure that integrity is checked, even if only on a subset of files (e.g. executables and shared libraries), regardless of whether fsverity was enabled or not on a file.
It could be task of the security subsystem to do this type of enforcement. At the moment, IMA and IPE (Integrity Protection Enforcement) are planning to support fsverity and do the enforcement based on a policy.
So that means there *will* be a policy control mechanism for this enforced by the kernel, cool!
-- 真実はいつも一つ!/ Always, there's only one truth!
On 12/2/21 11:36, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/FsVerityRPM
== Summary ==
Enable the use of fsverity for installed RPM files validation.
== Owners ==
- Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris
Burkov]], [[User:Filbranden|Filipe Brandenburger]], [[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew Almond]]
- Email: dcavalca@fb.com, borisb@fb.com, filbranden@fb.com,
michel@fb.com, malmond@fb.com
== Detailed description ==
fs-verity is a [https://www.kernel.org/doc/html/latest/filesystems/fsverity.html Linux kernel feature] that does transparent on-demand integrity/authenticity verification of the contents of read-only files, using a hidden Merkle tree (hash tree) associated with the file. The mechanism is similar to dm-verity, but implemented at the file level rather than at the block device level.
When fsverity is enabled for a file, the kernel reads every block and generates a hash tree for it, which is stored within the filesystem. On subsequent reads, the kernel computes the block hash and compares it with the one stored in the tree, protecting against alterations and corruption. Because this happens at the filesystem data block read layer, it encompasses all file operations (<code>open</code>, <code>mmap</code>,<code>exec</code>, etc.).
In the context of rpm, there are two parts to this:
- at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
- at run time, if the fsverity rpm plugin is enabled, rpm will install
the fsverity signature key and enable fsverity on files that are installed.
This proposal is primarily concerned with the first part, which will make it possible for users to leverage fs-verity for RPM if they so desire. Specifically, installing and enabling the fs-verity rpm plugin by default is explicitly considered out of scope here.
=== Caveats ===
==== Merkle tree cost ====
The Merkle tree used by fsverity needs to be generated (once at build time, once when the package is installed) and stored on disk. The generation process involves reading all blocks and computing the hash, which has a non-trivial cost; however, it does not appear to meaningfully slow down package installs during empirical testing. Once
Did you test the impact this has on package build times? Particularly packages like llvm, clang, webkit2gtk3, etc. that have very large debuginfo files?
-Tom
I don't believe we systematically tested this. We will collect that along with the detailed size increase data.
On Mon, Dec 13, 2021, at 5:21 PM, Tom Stellard wrote:
Did you test the impact this has on package build times? Particularly packages like llvm, clang, webkit2gtk3, etc. that have very large debuginfo files?
I think far too often the culture here is "make $change for all RPMs". But this "everything is an RPM" mindset can lead to outcomes and methodology that is at best weird.
For e.g. "let's try building with newer gcc", it would seem far better to me to e.g. start with the things that are in Fedora CoreOS or Workstation or whatever. (And, optionally their build dependencies)
For *this* particular change, the value of pre-signing the -debug RPMs seems...weak. Or even the `-devel` RPMs. Now, obviously choosing *which* binaries to sign would require some thought. But I think that's worth doing instead of blindly doing everything.
On Fri, Dec 17, 2021 at 4:59 PM Colin Walters walters@verbum.org wrote:
On Mon, Dec 13, 2021, at 5:21 PM, Tom Stellard wrote:
Did you test the impact this has on package build times? Particularly packages like llvm, clang, webkit2gtk3, etc. that have very large debuginfo files?
I think far too often the culture here is "make $change for all RPMs". But this "everything is an RPM" mindset can lead to outcomes and methodology that is at best weird.
For e.g. "let's try building with newer gcc", it would seem far better to me to e.g. start with the things that are in Fedora CoreOS or Workstation or whatever. (And, optionally their build dependencies)
Subset validation is very useful, yes, but fundamentally the Rawhide corpus is used to shake out GCC in the first place. It's a big part of how GCC new stable releases become so good. If we don't do it, basically nobody will.
For *this* particular change, the value of pre-signing the -debug RPMs seems...weak. Or even the `-devel` RPMs. Now, obviously choosing *which* binaries to sign would require some thought. But I think that's worth doing instead of blindly doing everything.
I'm not sure I agree. Outputs become inputs for other processes, and I think people generally prefer that the integrity of those inputs could be assured. If we'd want the integrity of the filesystem to be assured for runtime components, I see no reason that we wouldn't want it for development components and build-time components too. Moreover, as infrastructure for granular, network-based access for those things becomes a reality, having signed blobs for those things becomes more valuable so that privileged processes can be reasonably assured that they aren't tampered with.
Hi everyone
(note for the infrastructure mailing list: please check if the changes I'm proposing could be tested in the Fedora infrastructure, like Copr)
I made the first version of the rpm extension to sign fsverity digests with a GPG key. The patch set (with some bug fixes) is available here:
https://github.com/robertosassu/rpm/commits/fsverity-gpg-v1
I tested it locally with my own GPG key. I took an existing Fedora 34 package and signed it with rpmsign:
$ usr/bin/rpmsign --define "%_gpg_name testhost testhost@test.test" \ --define "%_file_signing_key _GPG_" \ --define "%_file_signing_cert _GPG_" \ --addsign --signverity tmux-3.1c-2.fc34.x86_64.rpm
I then checked that the package has now fsverity signatures:
$ usr/bin/rpm -qp tmux-3.1c-2.fc34.x86_64.rpm \ --queryformat '[%{RPMTAG_FILENAMES} %{RPMTAG_VERITYSIGNATURES}\n]' [...] /usr/bin/tmux iQHHBAABCgAxFiEEEiFa0dGZVYzTrIN+rxtXRMfK0McFAmHq0+4THHRlc3Rob3N0 QHRlc3QudGVzdAAKCRCvG1dEx8rQx81nC/42NW9xJx3rcTiR6/5oL55GPkan+OIq t2dW1clJUOrxOGVy/5JQTQf0MQXA7gzH1yPgcrskkahjSfWlp4pt7oOw3rukUyaO zVZxue4XE6XESYtolczK4VEhpc8lbm4hj0e4NCg/dKri/+L5wIdJvmqWNeCfl7uZ [...]
In a VM I tried to install the modified package. The root filesystem is ext4 and has the fsverity feature enabled.
The fsverity rpm plugin is also enabled and hasn't been modified to work with the new PGP signatures.
The kernel includes the patch set I recently sent to the kernel mailing lists to add support for PGP keys and signatures:
https://lore.kernel.org/linux-integrity/20220111180318.591029-1-roberto.sass...
and another patch that calls verify_pgp_signature() in fs/verity/signature.c.
The first installation attempt fails, due to the missing key in the .fs-verity keyring:
# usr/bin/rpm -Uhvi ../tmux-3.1c-2.fc34.x86_64.rpm --debug [...] D: Plugin: calling hook fsm_file_prepare in fsverity plugin D: applying signature: [...] D: failed to enable verity (errno 126) for /usr/bin/tmux;61ead62d
Then, I added the required GPG key to the .fs-verity keyring:
# cat /mnt/repos/linux/certs/pubring.gpg | keyctl padd asymmetric test %keyring:.fs-verity 76292211
The key is now loaded:
# keyctl show %keyring:.fs-verity Keyring 66741466 --a-swrv 0 0 keyring: .fs-verity 76292211 --als--v 0 0 _ asymmetric: test
I retried the tmux installation:
# usr/bin/rpm -Uhvi ../tmux-3.1c-2.fc34.x86_64.rpm --debug [...] D: Plugin: calling hook fsm_file_prepare in fsverity plugin D: applying signature: [...] D: fsverity enabled signature for: path /usr/bin/tmux;61ead713 dest /usr/bin/tmux
This time the installation is successful, which means that the PGP signature has been successfully verified.
Roberto
HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063 Managing Director: Li Peng, Zhong Ronghua
On Fri, Jan 21, 2022 at 04:08:04PM +0000, Roberto Sassu via devel wrote:
Hi everyone
(note for the infrastructure mailing list: please check if the changes I'm proposing could be tested in the Fedora infrastructure, like Copr)
copr uses a different signing setup... so probibly won't work there.
We could perhaps work with you to try and get it in our staging env to test, but I am not sure how involved that might be. I guess the first step would be filing a infratructure ticket and explain what you would need to change/test.
So, question for the change owners:
This is not enabled by default, I assume you wish to enable it for something, can you expand on what you will use it for and what that helps you with?
Also, is there any plan to enable it by default? Is there some way this could be useful to all our users? or more of our users?
This change and the DIGLIM change both feel to me like we are spending time and effort enabling something that only helps a tiny fraction of our users. Is there some way this could be more generally useful to our users?
kevin
From: Kevin Fenzi [mailto:kevin@scrye.com] Sent: Tuesday, January 25, 2022 7:30 PM On Fri, Jan 21, 2022 at 04:08:04PM +0000, Roberto Sassu via devel wrote:
Hi everyone
(note for the infrastructure mailing list: please check if the changes I'm proposing could be tested in the Fedora infrastructure, like Copr)
copr uses a different signing setup... so probibly won't work there.
We could perhaps work with you to try and get it in our staging env to test, but I am not sure how involved that might be. I guess the first step would be filing a infratructure ticket and explain what you would need to change/test.
Thanks, Kevin. Will do.
So, question for the change owners:
This is not enabled by default, I assume you wish to enable it for something, can you expand on what you will use it for and what that helps you with?
Also, is there any plan to enable it by default? Is there some way this could be useful to all our users? or more of our users?
This change and the DIGLIM change both feel to me like we are spending time and effort enabling something that only helps a tiny fraction of our users. Is there some way this could be more generally useful to our users?
I guess not many would use the remote attestation capability offered by DIGLIM (sealing a TPM key to the software running in the machine). The two scenarios in which that could be used are:
- web servers or other kind of servers where you, as client, would like the guarantee that your data is processed only if the software running in the server is not compromised
- closed networks where you, as network administrator, would like the guarantee that your clients can access those networks only if their software is not compromised
Probably, the secure boot functionality at application level is much more appealing, and less invasive. It should have almost zero impact if the users use their system for web browsing or writing. Although it is unlikely to happen, this feature prevents arbitrary execution of software, even by root. The trust anchor will be the kernel, and one would not have to rely on the correct execution of the package manager (and on its verification of the package signature).
If the users often make changes on their system, but confined to their environment, DIGLIM could still be helpful, as the damage of running arbitrary software will be limited to the users environment (no arbitrary software execution as root).
If the users often make changes on their system, with high privileges, I agree that DIGLIM would simply cause too much overhead for the configuration (every time the users make a change, they have to whitelist their software).
Roberto
HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063 Managing Director: Li Peng, Zhong Ronghua
On Wed, 2022-01-26 at 09:25 +0000, Roberto Sassu via devel wrote:
...snip
If the users often make changes on their system, with high privileges, I agree that DIGLIM would simply cause too much overhead for the configuration (every time the users make a change, they have to whitelist their software).
Yeah that will definitely be a problem in my use cases. You know, if I wanted to have something interfere with my work, like Windows, I would just install Windows.
Stephen
Roberto
HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063 Managing Director: Li Peng, Zhong Ronghua _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 1/26/22 3:25 AM, Roberto Sassu via devel wrote:
[Snip]
- web servers or other kind of servers where you, as client, would like the guarantee that your data is processed only if the software running in the server is not compromised
For what it's worth, I, and several people I work with, have been hoping for something to enable this kind of functionality for quite some time now. Hope to see more about it soon!
From: Brandon Nielsen [mailto:nielsenb@jetfuse.net] Sent: Wednesday, January 26, 2022 5:14 PM On 1/26/22 3:25 AM, Roberto Sassu via devel wrote:
[Snip]
- web servers or other kind of servers where you, as client, would like the guarantee that your data is processed only if the software running in the server is not compromised
For what it's worth, I, and several people I work with, have been hoping for something to enable this kind of functionality for quite some time now. Hope to see more about it soon!
Thanks Brandon! That is very encouraging.
All the pieces have been already developed and demonstrated at the final review meeting of the FutureTPM European project.
Here, at the bottom of the email, you can find some links:
https://www.spinics.net/lists/fedora-devel/msg295543.html
I planned to create a dedicated Fedora Change for the remote attestation part, which would depend on:
https://fedoraproject.org/wiki/Changes/DIGLIM
I'm still looking for some support from people interested like you, to comment in the kernel mailing list and help for the acceptance of the kernel patches in the upstream kernel.
Thanks
Roberto
HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063 Managing Director: Li Peng, Zhong Ronghua
Dne 21. 01. 22 v 17:08 Roberto Sassu via devel napsal(a):
(note for the infrastructure mailing list: please check if the changes I'm proposing could be tested in the Fedora infrastructure, like Copr)
Copr does not use `rpmsign` at all. Copr uses `obs-sign` [1]. The benefits of obs-sign is that you do not need to send whole package to the signing machine and then back. You send just hash there and signature back.
So to test this in Copr the --signverity has to be first implemented in obs-sign.
[1] https://github.com/openSUSE/obs-sign
Miroslav