Hi guys, I would like to use obs-sign for signing packages in Copr. Since this will be important part of Fedora, I would be glad if more people can see it and point out some problem if there are any. I already discussed it with Mitr, but more eyes is always better.
This is little bit premature, since I do not have rpm package ready yet (will be in 2 weeks aprox.). The code is here: https://github.com/openSUSE/obs-sign Docs here: http://en.opensuse.org/openSUSE:Build_Service_Signer
On Fri, May 23, 2014 at 10:17:03AM +0200, Miroslav Suchý wrote:
Hi guys, I would like to use obs-sign for signing packages in Copr.
More generally, it'd be nice to have security review of this plan: https://lists.fedoraproject.org/pipermail/infrastructure/2014-May/014345.htm...
On Fri, 2014-05-23 at 09:18 -0400, Matthew Miller wrote:
On Fri, May 23, 2014 at 10:17:03AM +0200, Miroslav Suchý wrote:
Hi guys, I would like to use obs-sign for signing packages in Copr.
More generally, it'd be nice to have security review of this plan: https://lists.fedoraproject.org/pipermail/infrastructure/2014-May/014345.htm...
As far as I understand the plan in that e-mail is: "What we can have is have signing machine in VM with restrictive SW defined network. If that VM can be only one VM on host, then it would be great." That's really minimal information there. Is there a smart card being used? Where do the keys reside? What is the impact of a key leakage?
Any way let me define some criteria: 1. audit (find out what was signed in time) 2. keys must not be exportable (no-one should get those keys)
I've not mentioned accountability as I understand that only one user signs.
For (2) I'd note that the most embarrassing issue on diginotar failure was the fact that they had no log of what was signed. Depending on the impact of such a leak, this may or may not be important.
For (2) I don't think a VM satisfies the isolation from the internet, and cannot protect the keys. If the VM resides on a developer's machine, one may just copy it over the internet. On similar designs the keys are stored externally (smart cards or HSMs), and locked somewhere when not in use.
regards, Nikos
On Fri, 23 May 2014, Matthew Miller wrote:
More generally, it'd be nice to have security review of this plan: https://lists.fedoraproject.org/pipermail/infrastructure/2014-May/014345.htm...
The point of the plan seems to be to isolate signing keys from the rest of Copr backend and put them in a separate box--let us call it a "signing oracle"--presumably in order to protect those keys if the rest of backend were compromised, but it is not stated explicitly.
It is a good idea in general but I am afraid there is a catch if the keys are used to sign code. The oracle may be able to protect the keys against theft but it is much less likely it would protect them against abuse.
Let us assume an attacker who got to the point where he or she would able to steal the key if it stayed on the build system and were not stashed in the oracle. It seems to me such an attacker would probably be able to coax the oracle to sign an arbitrary package, most likely a package including a trojan horse. Even if the breach is discovered and further abuse of the oracle is prevented, the existing signed "trojanized" package or packages will not disappear.
Is there a way to neutralize such packages that does not involve explicit replacement of signing keys on every system trusting the abused keys? I suspect the answer is negative and the impact of a key having been abused would not be much lower than the impact of a key having been stolen.
Indeed, there are other benefits of stashing keys in a signing oracle.
First, the keys do not leak when the confidentiality (but not integrity) if the main system is compromised, e.g. when a backup copy is made public by accident. (Of course, this assumes such blunders can be avoided when the oracle itself is being handled.)
Second, one can keep a trusted record of all signing operations at the oracle. It would not protect keys against abuse but at least it could make it possible to determine--after the fact--whether a particular key has been abused. (This has been already mentioned by Nikos Mavrogiannopoulos.)
obs-sign appears to record every operation but signed data are represented by their hash. I think it would be better if more detailed information about the data, perhaps a complete copy, were recorded. On the other hand, that would make it necessary to transfer a complete file from the client to the daemon.
On the other hand, there are also risks because the oracle adds more points of failure. Namely, it must make a correct decision whether a certain signing operation is authorized or not.
This seems to be a rather weak point of obs-sign because the daemon relies on identity sent (in plaintext!) by the client and the assumption that unprivileged user are not able to spoof the expected source IP and a low source port--a mechanism that should have gone extinct together with rsh & al. Not to mention that all code seems to be supposed to run as root. Setuid-root at the client to make things more... stimulating for adrenal glands.
I think checks based on source IP & port should be supplemented (if not replaced) with an appropriate cryptographic mechanism, such as digital signatures. Moreover the client should be split into an ordinary non-setuid program doing all complicating things and sending requests via a unix socket to a trusted proxy that would do nothing but add (trusted) information about the client's identity and forward the request to the oracle.
One final comment: The plan suggests to put all eggs into one basket. It goes as far as to call that "ideal (and most paranoid) setup" assuming the basket is sturdy enough. But we all know sturdiness of baskets is often overrated. I suggest to consider an alternative approach where private keys are split using some N out of M scheme among multiple parties with sufficient diversity to reduce the risk of N+ parties being compromised simultaneously.
On 06/04/2014 11:38 PM, Pavel Kankovsky wrote:
Let us assume an attacker who got to the point where he or she would able to steal the key if it stayed on the build system and were not stashed in the oracle. It seems to me such an attacker would probably be able to coax
Key will *not* be stored in build system. Just in Signing machine.
Is there a way to neutralize such packages that does not involve explicit replacement of signing keys on every system trusting the abused keys?
I am not aware of any method.
Second, one can keep a trusted record of all signing operations at the oracle. It would not protect keys against abuse but at least it could make it possible to determine--after the fact--whether a particular key has been abused. (This has been already mentioned by Nikos Mavrogiannopoulos.)
obs-sign appears to record every operation but signed data are represented by their hash. I think it would be better if more detailed information about the data, perhaps a complete copy, were recorded. On the other hand, that would make it necessary to transfer a complete file from the client to the daemon.
I plan to record hash *and* filenames. Which should be enough for forensics analysis. Storing whole data payload is not possible (read it as "we have no money for that"), because we plan to build and sign (docker) images as well.
often overrated. I suggest to consider an alternative approach where private keys are split using some N out of M scheme among multiple parties with sufficient diversity to reduce the risk of N+ parties being compromised simultaneously.
No money for that.
On Thu, Jun 05, 2014 at 10:46:14AM +0200, Miroslav Suchý wrote:
Is there a way to neutralize such packages that does not involve explicit replacement of signing keys on every system trusting the abused keys?
I am not aware of any method.
At one of my previous jobs, we planned but never had to use an approach for this: an update to the '-release' RPM which included a post script to remove the compromised key from systems.
On Thu, 2014-06-05 at 08:13 -0400, Matthew Miller wrote:
On Thu, Jun 05, 2014 at 10:46:14AM +0200, Miroslav Suchý wrote:
Is there a way to neutralize such packages that does not involve explicit replacement of signing keys on every system trusting the abused keys?
I am not aware of any method.
At one of my previous jobs, we planned but never had to use an approach for this: an update to the '-release' RPM which included a post script to remove the compromised key from systems.
The problem is not just the compromised key, but compromised packages, though I guess you could re-sign all packages, but then you also have to ship those signatures out of band (you cannot force people to re-install all packages right ?).
One way to mitigate the impact is also to create subkeys (say one every week) so that you can "repudiate" a window of time by marking only a set of subkeys as compromised. This requires a more complicated signing and verification process though.
Simo.
On Thu, Jun 05, 2014 at 08:19:28 -0400, Simo Sorce simo@redhat.com wrote:
The problem is not just the compromised key, but compromised packages, though I guess you could re-sign all packages, but then you also have to ship those signatures out of band (you cannot force people to re-install all packages right ?).
Didn't we do something like this in response to: https://www.redhat.com/archives/fedora-announce-list/2009-March/msg00010.htm...
On Thu, 5 Jun 2014, Miroslav Suchý wrote:
Let us assume an attacker who got to the point where he or she would able to steal the key if it stayed on the build system [...]
Key will *not* be stored in build system. Just in Signing machine.
Indeed. The conditional in question was counterfactual.
obs-sign appears to record every operation but [...]
I plan to record hash *and* filenames. Which should be enough for forensics analysis.
Filenames will probably make log records more readable but they are not included in signed data and any data can be signed under any name. But the things are even worse: an insidious attacker might inject a Trojan horse into a build initiated by a legitimate user. (*)
(*) It seems to me Copr and Mock (well, Mock seems to be the primary offender here) are quite happy to pull sources & additional repos for the build environment over cleartext HTTP with no integrity checks. And users are encouraged to configure it that way. Therefore one might be able to plan a Trojan horse into a signed package without breaking into the Copr backend. Have I missed anything?
Also: Who will (be able to) review the logs?
Storing whole data payload is not possible (read it as "we have no money for that"), because we plan to build and sign (docker) images as well.
I agree. It would be, ahem, too excessive to store full images.
Would it be possible to extract and store the list of contained files and their properties that would allow one to compare the result with a build created elsewhere? (Of course, this assumes the build process is reproducible--to a sufficient degree.)
No money for that.
Man, you talk like a manager. ;)
On Thu, 5 Jun 2014, Matthew Miller wrote:
At one of my previous jobs, we planned but never had to use an approach for this: an update to the '-release' RPM which included a post script to remove the compromised key from systems.
That would probably work. But it might lead to serious confusion:
The update would need to be signed with the old compromised key, while all other new packages would be signed with the new key. (An RPM file could carry multiple signatures of the same type but I am afraid the program would not be able to handle them.)
Anyone will have to install the update to *-release *after* any older updates (assuming they have not been re-signed with the new key) and *before* any newer updates (including older updates that have been re-signed and any newer updates to *-release itself).
I can imagine how an attempt to replace the signing key that way could result in many systems in an inconsistent state, requiring a manual intervention to recover.
On Thu, 5 Jun 2014, Simo Sorce wrote:
The problem is not just the compromised key, but compromised packages, though I guess you could re-sign all packages, but then you also have to ship those signatures out of band (you cannot force people to re-install all packages right ?).
Do you have to ship the new signatures to anyone? Does anyone re-check signatures of packages that have already been installed?
One way to mitigate the impact is also to create subkeys (say one every week) so that you can "repudiate" a window of time by marking only a set of subkeys as compromised. This requires a more complicated signing and verification process though.
You have a problem. You implement PKI to solve the problem. Now you have two problems... I am sorry, I could not resist. :)
All right, I am going to be (mostly) serious again. Some kind of (nontrivial) PKI is probably desirable.
On Thu, 5 Jun 2014, Bruno Wolff III wrote:
Didn't we do something like this in response to: https://www.redhat.com/archives/fedora-announce-list/2009-March/msg00010.htm...
Hmm... the announcement links to https://fedoraproject.org/wiki/New_signing_key and that page outlines a 13-step plan where step number 12 is as follows:
During development and release of Fedora 10, consider and create better ways to handle automatic key migration. (in progress)
I guess that part of the plan has not made much progress since 2009.
security@lists.fedoraproject.org
-
Bruno Wolff III -
Matthew Miller -
Miroslav Suchý -
Nikos Mavrogiannopoulos -
Pavel Kankovsky -
Simo Sorce