Review of obs-sign

[Miroslav Suchý]

Hi guys,
I would like to use obs-sign for signing packages in Copr.
Since this will be important part of Fedora, I would be glad if more people can see it and
point out some problem if 
there are any.
I already discussed it with Mitr, but more eyes is always better.

This is little bit premature, since I do not have rpm package ready yet (will be in 2
weeks aprox.).
The code is here:
https://github.com/openSUSE/obs-sign
Docs here:
http://en.opensuse.org/openSUSE:Build_Service_Signer

-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Matthew Miller]

On Fri, May 23, 2014 at 10:17:03AM +0200, Miroslav Suchý wrote:
>  Hi guys,
>  I would like to use obs-sign for signing packages in Copr. 
More generally, it'd be nice to have security review of this plan:
https://lists.fedoraproject.org/pipermail/infrastructure/2014-May/014345....

-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Nikos Mavrogiannopoulos]

On Fri, 2014-05-23 at 09:18 -0400, Matthew Miller wrote:
>  On Fri, May 23, 2014 at 10:17:03AM +0200, Miroslav Suchý wrote:
>  > Hi guys,
>  > I would like to use obs-sign for signing packages in Copr.
>  
>  More generally, it'd be nice to have security review of this plan:
>  https://lists.fedoraproject.org/pipermail/infrastructure/2014-May/014345.... 
As far as I understand the plan in that e-mail is:
"What we can have is have signing machine in VM with restrictive SW
defined network. If that VM can be only one VM on 
host, then it would be great."
That's really minimal information there. Is there a smart card being
used? Where do the keys reside? What is the impact of a key leakage? 

Any way let me define some criteria:
1. audit (find out what was signed in time)
2. keys must not be exportable (no-one should get those keys)

I've not mentioned accountability as I understand that only one user
signs.

For (2) I'd note that the most embarrassing issue on diginotar failure
was the fact that they had no log of what was signed. Depending on the
impact of such a leak, this may or may not be important.

For (2) I don't think a VM satisfies the isolation from the internet,
and cannot protect the keys. If the VM resides on a developer's machine,
one may just copy it over the internet. On similar designs the keys are
stored externally (smart cards or HSMs), and locked somewhere when not
in use.

regards,
Nikos

[Pavel Kankovsky]

On Fri, 23 May 2014, Matthew Miller wrote:

>  More generally, it'd be nice to have security review of this
> plan:
>  https://lists.fedoraproject.org/pipermail/infrastructure/2014-May/014345.... 
The point of the plan seems to be to isolate signing keys from the rest of
Copr backend and put them in a separate box--let us call it a "signing
oracle"--presumably in order to protect those keys if the rest of backend
were compromised, but it is not stated explicitly.

It is a good idea in general but I am afraid there is a catch if the keys
are used to sign code. The oracle may be able to protect the keys against
theft but it is much less likely it would protect them against abuse.

Let us assume an attacker who got to the point where he or she would able
to steal the key if it stayed on the build system and were not stashed in
the oracle. It seems to me such an attacker would probably be able to coax
the oracle to sign an arbitrary package, most likely a package including a
trojan horse. Even if the breach is discovered and further abuse of the
oracle is prevented, the existing signed "trojanized" package or packages
will not disappear.

Is there a way to neutralize such packages that does not involve explicit
replacement of signing keys on every system trusting the abused keys?
I suspect the answer is negative and the impact of a key having been 
abused would not be much lower than the impact of a key having been 
stolen.

Indeed, there are other benefits of stashing keys in a signing oracle.

First, the keys do not leak when the confidentiality (but not integrity)
if the main system is compromised, e.g. when a backup copy is made public
by accident. (Of course, this assumes such blunders can be avoided when
the oracle itself is being handled.)

Second, one can keep a trusted record of all signing operations at the
oracle. It would not protect keys against abuse but at least it could make
it possible to determine--after the fact--whether a particular key has
been abused. (This has been already mentioned by Nikos Mavrogiannopoulos.)

obs-sign appears to record every operation but signed data are represented
by their hash. I think it would be better if more detailed information
about the data, perhaps a complete copy, were recorded. On the other hand,
that would make it necessary to transfer a complete file from the client
to the daemon.

On the other hand, there are also risks because the oracle adds more
points of failure. Namely, it must make a correct decision whether a
certain signing operation is authorized or not.

This seems to be a rather weak point of obs-sign because the daemon relies
on identity sent (in plaintext!) by the client and the assumption that
unprivileged user are not able to spoof the expected source IP and a low
source port--a mechanism that should have gone extinct together with rsh
& al. Not to mention that all code seems to be supposed to run as root.
Setuid-root at the client to make things more... stimulating for adrenal
glands.

I think checks based on source IP & port should be supplemented (if not
replaced) with an appropriate cryptographic mechanism, such as digital
signatures. Moreover the client should be split into an ordinary
non-setuid program doing all complicating things and sending requests via
a unix socket to a trusted proxy that would do nothing but add (trusted)
information about the client's identity and forward the request to the 
oracle.

One final comment: The plan suggests to put all eggs into one basket. It
goes as far as to call that "ideal (and most paranoid) setup" assuming
the basket is sturdy enough. But we all know sturdiness of baskets is
often overrated. I suggest to consider an alternative approach where
private keys are split using some N out of M scheme among multiple parties
with sufficient diversity to reduce the risk of N+ parties being 
compromised simultaneously. 

-- 
Pavel Kankovsky aka Peak                      "Que sais-je?"

[Miroslav Suchý]

On 06/04/2014 11:38 PM, Pavel Kankovsky wrote:
>  Let us assume an attacker who got to the point where he or she would
> able
>  to steal the key if it stayed on the build system and were not stashed in
>  the oracle. It seems to me such an attacker would probably be able to coax 
Key will *not* be stored in build system. Just in Signing machine.

>  Is there a way to neutralize such packages that does not involve
> explicit
>  replacement of signing keys on every system trusting the abused keys? 
I am not aware of any method.

>  Second, one can keep a trusted record of all signing operations at
> the
>  oracle. It would not protect keys against abuse but at least it could make
>  it possible to determine--after the fact--whether a particular key has
>  been abused. (This has been already mentioned by Nikos Mavrogiannopoulos.)
>
>  obs-sign appears to record every operation but signed data are represented
>  by their hash. I think it would be better if more detailed information
>  about the data, perhaps a complete copy, were recorded. On the other hand,
>  that would make it necessary to transfer a complete file from the client
>  to the daemon. 
I plan to record hash *and* filenames. Which should be enough for forensics analysis.
Storing whole data payload is not possible (read it as "we have no money for
that"), because we plan to build and sign 
(docker) images as well.

>  often overrated. I suggest to consider an alternative approach where
>  private keys are split using some N out of M scheme among multiple parties
>  with sufficient diversity to reduce the risk of N+ parties being
>  compromised simultaneously. 
No money for that.



-- 
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Matthew Miller]

On Thu, Jun 05, 2014 at 10:46:14AM +0200, Miroslav Suchý wrote:
>  >Is there a way to neutralize such packages that does not involve
> explicit
>  >replacement of signing keys on every system trusting the abused keys?
>  I am not aware of any method. 
At one of my previous jobs, we planned but never had to use an approach for
this: an update to the '-release' RPM which included a post script to remove
the compromised key from systems.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Simo Sorce]

On Thu, 2014-06-05 at 08:13 -0400, Matthew Miller wrote:
>  On Thu, Jun 05, 2014 at 10:46:14AM +0200, Miroslav Suchý wrote:
>  > >Is there a way to neutralize such packages that does not involve explicit
>  > >replacement of signing keys on every system trusting the abused keys?
>  > I am not aware of any method.
>  
>  At one of my previous jobs, we planned but never had to use an approach for
>  this: an update to the '-release' RPM which included a post script to remove
>  the compromised key from systems. 
The problem is not just the compromised key, but compromised packages,
though I guess you could re-sign all packages, but then you also have to
ship those signatures out of band (you cannot force people to re-install
all packages right ?).

One way to mitigate the impact is also to create subkeys (say one every
week) so that you can "repudiate" a window of time by marking only a set
of subkeys as compromised. This requires a more complicated signing and
verification process though.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Bruno Wolff III]

On Thu, Jun 05, 2014 at 08:19:28 -0400,
  Simo Sorce <simo(a)redhat.com&gt; wrote:
> The problem is not just the compromised key, but compromised packages,
> though I guess you could re-sign all packages, but then you also have to
> ship those signatures out of band (you cannot force people to re-install
> all packages right ?). 
Didn't we do something like this in response to:
https://www.redhat.com/archives/fedora-announce-list/2009-March/msg00010....

[Pavel Kankovsky]

On Thu, 5 Jun 2014, Miroslav Suchý wrote:

>  > Let us assume an attacker who got to the point where he or she
> would
>  > able to steal the key if it stayed on the build system [...]
>  Key will *not* be stored in build system. Just in Signing machine. 
Indeed. The conditional in question was counterfactual.

>  > obs-sign appears to record every operation but [...]
>  I plan to record hash *and* filenames. Which should be enough for
>  forensics analysis. 
Filenames will probably make log records more readable but they are not 
included in signed data and any data can be signed under any name.
But the things are even worse: an insidious attacker might inject a 
Trojan horse into a build initiated by a legitimate user. (*)

(*) It seems to me Copr and Mock (well, Mock seems to be the primary
offender here) are quite happy to pull sources & additional repos for the
build environment over cleartext HTTP with no integrity checks. And users
are encouraged to configure it that way. Therefore one might be able to
plan a Trojan horse into a signed package without breaking into the Copr
backend. Have I missed anything?

Also: Who will (be able to) review the logs?

>  Storing whole data payload is not possible (read it as "we have
> no money
>  for that"), because we plan to build and sign (docker) images as well. 
I agree. It would be, ahem, too excessive to store full images.

Would it be possible to extract and store the list of contained files and 
their properties that would allow one to compare the result with a build 
created elsewhere? (Of course, this assumes the build process is 
reproducible--to a sufficient degree.)

>  No money for that. 
Man, you talk like a manager. ;)


On Thu, 5 Jun 2014, Matthew Miller wrote:

>  At one of my previous jobs, we planned but never had to use an
> approach
>  for this: an update to the '-release' RPM which included a post script
>  to remove the compromised key from systems. 
That would probably work. But it might lead to serious confusion:

The update would need to be signed with the old compromised key, while all
other new packages would be signed with the new key. (An RPM file could
carry multiple signatures of the same type but I am afraid the program
would not be able to handle them.)

Anyone will have to install the update to *-release *after* any older
updates (assuming they have not been re-signed with the new key) and
*before* any newer updates (including older updates that have been
re-signed and any newer updates to *-release itself).

I can imagine how an attempt to replace the signing key that way could 
result in many systems in an inconsistent state, requiring a manual 
intervention to recover.


On Thu, 5 Jun 2014, Simo Sorce wrote:

>  The problem is not just the compromised key, but compromised
> packages,
>  though I guess you could re-sign all packages, but then you also have to
>  ship those signatures out of band (you cannot force people to re-install
>  all packages right ?). 
Do you have to ship the new signatures to anyone? Does anyone re-check
signatures of packages that have already been installed?

>  One way to mitigate the impact is also to create subkeys (say one
> every
>  week) so that you can "repudiate" a window of time by marking only a set
>  of subkeys as compromised. This requires a more complicated signing and
>  verification process though. 
You have a problem. You implement PKI to solve the problem. Now you 
have two problems... I am sorry, I could not resist. :)

All right, I am going to be (mostly) serious again. Some kind of 
(nontrivial) PKI is probably desirable.


On Thu, 5 Jun 2014, Bruno Wolff III wrote:

>  Didn't we do something like this in response to:
>  https://www.redhat.com/archives/fedora-announce-list/2009-March/msg00010.... 
Hmm... the announcement links to
<https://fedoraproject.org/wiki/New_signing_key> and that page outlines
a 13-step plan where step number 12 is as follows:

  During development and release of Fedora 10, consider and create better 
  ways to handle automatic key migration. (in progress)

I guess that part of the plan has not made much progress since 2009.


-- 
Pavel Kankovsky aka Peak                      "Que sais-je?"