8:56 p.m.
In the light of recent RPM signing intrusions, maybe we should resurrect
the RPM feature where multiple signatures are allowed (i.e. --addsign is
different to --resign)? With this we could then require N good
signatures (and no bad ones) on each package before yum would trust the
content.
What I'm getting at with this is distributed package signing, which
would make the job of breaking the trust much harder for attackers, as
they would have to crack private keys of many people around the world in
order to subvert Fedora packages.
For instance, an attacker being in the position of injecting a bad
package and signing it with Fedora key would still get nowhere, as he'd
need to convince other signatories to sign those packages before them
being any threat to Fedora users. Before signing, signatories could
require that original contributor that built the package for a
particular tag sends a signed e-mail (containing that tag and package
checksums - valid only once) to the signatories, therefore requiring yet
another compromised private key in order to perform an attack.
Signatories could also use alternative build systems with no public
access (e.g. their own, Matt's at Dell etc.) to verify package checksums
before signing, in order to avoid trusting a compromised Fedora build
system.
This would require more distributed resources and would slow the update
process down somewhat, but may avoid single point of intrusion as being
sufficient to break the distro.
Comments?
--
Bojan
9:22 p.m.
On Tue, 2008-08-26 at 11:56 +1000, Bojan Smojver wrote:
In the light of recent RPM signing intrusions, maybe we should
resurrect
the RPM feature where multiple signatures are allowed (i.e. --addsign is
different to --resign)? With this we could then require N good
signatures (and no bad ones) on each package before yum would trust the
content.
Signatories could also use alternative build systems with no public
access (e.g. their own, Matt's at Dell etc.) to verify package checksums
before signing, in order to avoid trusting a compromised Fedora build
system.
I think the checksums would be the hardest part. Build times, hosts
and other details are very often embedded into a build.
Andrew Bartlett
--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Red Hat Inc.
9:39 p.m.
Andrew Bartlett <abartlet <at> samba.org> writes:
I think the checksums would be the hardest part. Build times,
hosts
and other details are very often embedded into a build.
Yeah, good point. We do have checksums of individual files inside the RPM,
right? Maybe we can leverage that in order to provide a build system neutral
checksum that can be verified independently?
--
Bojan
9:49 p.m.
Bojan Smojver <bojan <at> rexursive.com> writes:
Yeah, good point. We do have checksums of individual files inside the
RPM,
right? Maybe we can leverage that in order to provide a build system neutral
checksum that can be verified independently?
Or maybe we could even rely on checksums of cpio archives produced by rpm2cpio?
--
Bojan
10:09 p.m.
Once upon a time, Bojan Smojver <bojan(a)rexursive.com> said:
Andrew Bartlett <abartlet <at> samba.org> writes:
> I think the checksums would be the hardest part. Build times, hosts
> and other details are very often embedded into a build.
Yeah, good point. We do have checksums of individual files inside the RPM,
right? Maybe we can leverage that in order to provide a build system neutral
checksum that can be verified independently?
That still doesn't help; some things embed the compile time and info in
the files. See for example 'uname -v' (although that one is pretty
easily controlled IIRC) and 'perl -V'.
One possible way to handle builds that do this would be to do something
like use the timestamp of the spec file or last CVS update time for
example and force such builds to use that instead of the current time.
That doesn't help the 'perl -V' example though, since it includes the
'uname -r' and 'uname -v' output in the resulting binary; for example,
you can see that the current perl RPM on F9/x86_64 was built on a RHEL5
(or derivative; somebody could tell from the version string) system and
what kernel it was running at the time.
--
Chris Adams <cmadams(a)hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
10:29 p.m.
Chris Adams <cmadams <at> hiwaay.net> writes:
That still doesn't help; some things embed the compile time and
info in
the files. See for example 'uname -v' (although that one is pretty
easily controlled IIRC) and 'perl -V'.
One possible way to handle builds that do this would be to do something
like use the timestamp of the spec file or last CVS update time for
example and force such builds to use that instead of the current time.
That doesn't help the 'perl -V' example though, since it includes the
'uname -r' and 'uname -v' output in the resulting binary; for example,
you can see that the current perl RPM on F9/x86_64 was built on a RHEL5
(or derivative; somebody could tell from the version string) system and
what kernel it was running at the time.
Right. No very good.
Are these things exceptions to the rule or do majority of package have this kind
of thing built in? If 95% of packages don't have it, the rest can always be
checked by hand by running binary diff or something...
--
Bojan
10:57 p.m.
Bojan Smojver <bojan <at> rexursive.com> writes:
Are these things exceptions to the rule or do majority of package
have this
kind of thing built in?
Actually, it should be quite easy to verify this. If someone from Red Hat could
run 'ls *.rpm | sort | while read pkg; do echo -en "$pkg\t"; rpm2cpio <
$pkg |
sha1sum; done' for all Fedora packages built in koji of a distro/arch (say
F9/i386) and if Matt could do the same on his Dell build farm, we'll clearly see
what gives different checksums of cpio archives.
--
Bojan
11:35 p.m.
On Tue, 2008-08-26 at 03:57 +0000, Bojan Smojver wrote:
Bojan Smojver <bojan <at> rexursive.com> writes:
> Are these things exceptions to the rule or do majority of package have this
> kind of thing built in?
Actually, it should be quite easy to verify this. If someone from Red Hat could
run 'ls *.rpm | sort | while read pkg; do echo -en "$pkg\t"; rpm2cpio <
$pkg |
sha1sum; done' for all Fedora packages built in koji of a distro/arch (say
F9/i386) and if Matt could do the same on his Dell build farm, we'll clearly see
what gives different checksums of cpio archives.
why do you want that?
rpm -qp --dump pkg.rpm
-sv
12:22 a.m.
Seth Vidal <skvidal <at> fedoraproject.org> writes:
why do you want that?
rpm -qp --dump pkg.rpm
Because I didn't read rpm manual page? ;-)
Yeah, that's really useful - thanks for that hint. Makes it really simple for
people to compare content of packages.
You reckon this multi-key signing thing could be done in any practical fashion
in Fedora?
--
Bojan
6:38 a.m.
On Tue, 2008-08-26 at 05:22 +0000, Bojan Smojver wrote:
Seth Vidal <skvidal <at> fedoraproject.org> writes:
> why do you want that?
>
> rpm -qp --dump pkg.rpm
Because I didn't read rpm manual page? ;-)
Yeah, that's really useful - thanks for that hint. Makes it really simple for
people to compare content of packages.
You reckon this multi-key signing thing could be done in any practical fashion
in Fedora?
I think it will complicate things a lot for users to verify and it's not
obvious how much we'll gain in terms of security.
-sv
6:41 p.m.
Seth Vidal <skvidal <at> fedoraproject.org> writes:
I think it will complicate things a lot for users to verify
Users wouldn't actually have to verify anything by hand. The idea was that yum
does that for them. I don't see how that would be any more complicated then now.
Say there are 10 signatories in the pool. Yum would check that:
- the package is signed with the Fedora key
- the package is signed by at least N (say 2) other keys from the pool
- failing the above, it would not accept the package
N could even be configurable in yum for smooth transition from the single key
scenario.
and it's not
obvious how much we'll gain in terms of security.
It is similar to what a reporter does to confirm a story. One source, not so
reliable. Two sources, more reliable. Many sources, most likely reliable.
In terms of attacks, right now if somebody gets a hold of the password of the
Fedora key, it's game over. Ditto if someone compromises the build system to
start producing bad binaries.
With the multi-key, multi-build system, an attacker would need to get his hands
on a lot of private key passwords, break multiple independent build systems etc.
I always think of flight attendants and how they are told by the captain to
secure the doors and cross-check. I'm sure there must be a good reason for that
cross-check :-)
--
Bojan
8:15 p.m.
On Tue, Aug 26, 2008 at 5:41 PM, Bojan Smojver <bojan(a)rexursive.com> wrote:
Seth Vidal <skvidal <at> fedoraproject.org> writes:
> I think it will complicate things a lot for users to verify
Users wouldn't actually have to verify anything by hand. The idea was that yum
does that for them. I don't see how that would be any more complicated then now.
Say there are 10 signatories in the pool. Yum would check that:
- the package is signed with the Fedora key
- the package is signed by at least N (say 2) other keys from the pool
- failing the above, it would not accept the package
N could even be configurable in yum for smooth transition from the single key
scenario.
> and it's not
> obvious how much we'll gain in terms of security.
It is similar to what a reporter does to confirm a story. One source, not so
reliable. Two sources, more reliable. Many sources, most likely reliable.
There is a specific "named" fallacy to that logic. I can't remember
the mathematical name for it, but basically the issue is that having
multiple sources doesn't help if they all get their information from
the same top level source. The big issue with multiple signatures is
that they are going to be automated somehow to deal with the thousands
upon thousands of packages being dealt with... and you are going to
have to come up with an additional income source to pay for the extra
bureaucracy that is being added.
In terms of attacks, right now if somebody gets a hold of the
password of the
Fedora key, it's game over. Ditto if someone compromises the build system to
start producing bad binaries.
With the multi-key, multi-build system, an attacker would need to get his hands
on a lot of private key passwords, break multiple independent build systems etc.
Or just be in the right place somewhere. The build systems will not be
completely independent or they would not be able to produce identical
binaries..
--
Stephen J Smoogen. -- BSD/GNU/Linux
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"
11:08 p.m.
Stephen John Smoogen <smooge <at> gmail.com> writes:
There is a specific "named" fallacy to that logic. I
can't remember
the mathematical name for it, but basically the issue is that having
multiple sources doesn't help if they all get their information from
the same top level source.
Yeah, no kidding.
The point of open source is supposed to be that more eyes see better. That's why
we have package reviews, pre-release checks (alpha, beta, rc) and so on. You can
never achieve 100% independence, of course. That's why we have bugs :-)
The big issue with multiple signatures is
that they are going to be automated somehow to deal with the thousands
upon thousands of packages being dealt with...
If there is any chance of this being automated, it cannot work at all and there
is no point doing it.
and you are going to
have to come up with an additional income source to pay for the extra
bureaucracy that is being added.
True. All security has a price.
Or just be in the right place somewhere.
More than one, actually.
The build systems will not be
completely independent or they would not be able to produce identical
binaries..
Say someone breaks into Fedora build system and subverts the process in such a
way that there is their own gcc inserted just at the right time in order to
produce the binaries they want. A package is built by the packager and a signed
e-mail is sent to the signatories to sign it, because it's an update.
Given this is a new update, another build system, located elsewhere and not
publicly accessible, pulls in the package source and builds it. If that other
system wasn't broken into, it will produce a different binary for sure.
Immediate alarm bells for signatories.
Sure, this is a difficult thing to do right. It doesn't fix all intrusion issues
(nothing can). Takes a lot of effort etc. But it does provide at least some
checks and balances before packages are swallowed by users out there.
No offence, but right now we have a single point of failure that we already know
can be cracked. And that single point of failure is the single point of users'
trust. Not a very safe combo, IMHO.
Never mind, it was just an idea. Probably not even a good one. Back to the
drawing board... ;-)
--
Bojan
7:11 p.m.
Bojan Smojver <bojan <at> rexursive.com> writes:
Say there are 10 signatories in the pool. Yum would check that:
- the package is signed with the Fedora key
- the package is signed by at least N (say 2) other keys from the pool
- failing the above, it would not accept the package
Just for completeness, yum could alternatively accept say 5 keys from the pool
(but no Fedora key), so that any compromise of the central key does not cause
the current "change the Fedora key" hoopla. Simply resign by others and
continue.
--
Bojan
4:40 a.m.
On 30/08/2008, Bojan Smojver <bojan(a)rexursive.com> wrote:
Just for completeness, yum could alternatively accept say 5 keys from
the
pool
(but no Fedora key), so that any compromise of the central key does not
cause
the current "change the Fedora key" hoopla. Simply resign by others and
continue.
What might be good, is only signing packages with one or two keys, but
only allowing those keys' public parts to be updated in rpm database
(or wherever) if signed by a much larger number of keys, which would
be owned by some trusted people from the fedora project. Then
automated rollover could be done by simply providing a new "keyring"
in updates.
4:48 a.m.
Bill Crawford escreveu:
On 30/08/2008, Bojan Smojver <bojan(a)rexursive.com> wrote:
> Just for completeness, yum could alternatively accept say 5 keys from the
> pool
> (but no Fedora key), so that any compromise of the central key does not
> cause
> the current "change the Fedora key" hoopla. Simply resign by others and
> continue.
>
What might be good, is only signing packages with one or two keys, but
only allowing those keys' public parts to be updated in rpm database
(or wherever) if signed by a much larger number of keys, which would
be owned by some trusted people from the fedora project. Then
automated rollover could be done by simply providing a new "keyring"
in updates.
BTW, updates are still frozen. What's the schedule for the normalization
of yum services? Will it be necessary a special procedure to be adopted
by users?
Best regards,
CdAB
11:12 p.m.
Bill Crawford <billcrawford1970 <at> gmail.com> writes:
What might be good, is only signing packages with one or two keys,
but
only allowing those keys' public parts to be updated in rpm database
(or wherever) if signed by a much larger number of keys, which would
be owned by some trusted people from the fedora project. Then
automated rollover could be done by simply providing a new "keyring"
in updates.
Exactly. Imagine if yum trusted fedora-release package signed by 5 keys of
signatories from the pool. This could have been delivered immediately upon
(potential) compromise, which would mean:
- seamless transition to new Fedora key without ever trusting the old key
- automatic delivery of revocation certificate for the old key
- instant suitability of all packages to be resigned by the new key
So, not only would attackers be unable to subvert the packages by stealing
Fedora key (because they'd need other signatories to agree to sign bad
packages), but any (potential) compromise would be quickly dealt with.
--
Bojan
10:11 p.m.
Bojan Smojver <bojan(a)rexursive.com> writes:
For instance, an attacker being in the position of injecting a bad
package and signing it with Fedora key would still get nowhere, as he'd
need to convince other signatories to sign those packages before them
being any threat to Fedora users. Before signing, signatories could
require that original contributor that built the package for a
particular tag sends a signed e-mail (containing that tag and package
checksums - valid only once) to the signatories, therefore requiring yet
another compromised private key in order to perform an attack.
Yup, packagers are going to do that, sure...
Most of us are overworked already. We aren't going to jump through any
hoops for third-party signatories.
regards, tom lane
10:27 p.m.
Tom Lane <tgl <at> redhat.com> writes:
Yup, packagers are going to do that, sure...
That was the intention, yes. Packagers would notify all signatories (with a
signed e-mail) that they've built a new package destined for updates and that
signatories should review and sign it. We're still working out the details of
making sure packages are are genuine in another thread :-)
I guess from Red Hat's point of view, the only difference would be that Fedora
packages would not be valid unless signed and uploaded back to updates by
(required number of) other signatories.
--
Bojan
2:13 a.m.
On Tue, Aug 26, 2008 at 03:27:43 +0000,
Bojan Smojver <bojan(a)rexursive.com> wrote:
I guess from Red Hat's point of view, the only difference would be that Fedora
packages would not be valid unless signed and uploaded back to updates by
(required number of) other signatories.
I don't think you are really going to gain much from doing that. And there
is certainly going to be a lot of pain associated with that. It creates
extra work, adds delays, and adds a dependence on third parties. And it
doesn't completely prevent people from getting bad code signed.
2:38 a.m.
Bruno Wolff III <bruno <at> wolff.to> writes:
I don't think you are really going to gain much from doing that.
This depends on a particular point of view, of course. If it so happened that
Fedora (and/or RHEL) signing key was compromised during the most recent
intrusion, it would have been game over for users. Not so if packages had to be
signed by multiple keys before being accepted by yum.
and adds a dependence on third parties
I see that as a feature, actually. It eliminates single point of failure.
And it doesn't completely prevent people from getting bad code
signed.
I don't think it is possible to design a system that does that completely. But,
at least you have more folks looking over the packages (from multiple sources)
before signing them - more chance of spotting inconsistencies.
--
Bojan
2:55 a.m.
On Tue, Aug 26, 2008 at 07:38:46 +0000,
Bojan Smojver <bojan(a)rexursive.com> wrote:
Bruno Wolff III <bruno <at> wolff.to> writes:
> and adds a dependence on third parties
I see that as a feature, actually. It eliminates single point of failure.
And adds another. If one of those third parties goes belly up, then Fedora
is going to have to take extraordinary measures to get packages signed in
a way that will be axxepted again.
3:51 a.m.
Bruno Wolff III <bruno <at> wolff.to> writes:
And adds another. If one of those third parties goes belly up, then
Fedora
is going to have to take extraordinary measures to get packages signed in
a way that will be axxepted again.
Not true. As I mentioned before, the criteria would be that package is signed
with N good keys. So, resigning with someone else's key would be sufficient to
overcome this.
BTW, third parties do not have to be companies. They can be trusted Fedora
contributors, for instance.
--
Bojan
4:40 a.m.
On Tue, 2008-08-26 at 11:56 +1000, Bojan Smojver wrote:
Comments?
I'm not an expert on these things, but I feel some flaws in your scheme.
First, it relies on that building a specifically tagged package will
always produce the same result. Others have already pointed out that
some software embeds non-deterministic data in files. I'd like to point
out additionally that the build dependencies aren't 100% deterministic
as well: when you build a new version of a build dependency and build a
dependent package soon afterwards, chances are high (enough) that some
builders build against the new and some against the old package, which
likely has an influence on the produced binaries. Should we ever employ
optimization techniques like profile-based compilation, builds between
separate build systems (that have different profile data) will
inevitably differ.
Second, it makes the update system more vulnerable to DOS attacks,
insofar as it only takes an attacker to hack himself into one of the
signatories to produce bad signatures on updates he wants to block (if
they e.g. contain security fixes). The risk of this is at least
proportionally higher with multiple targets to choose from (think RAID-0
and what it does to the probability of errors in such a set of disks).
Third, unless a signatory runs his own build system to verify package
builds (and disregarding my first point), his signature doesn't have
much value as he has to rely on the checksum data provided by the
package maintainer in his signed email -- which relies on the build
system not being compromised to begin with.
Then there's the additional burden on maintainers -- yet another
bureaucratic hurdle.
Note that I'd appreciate being able to have multiple signatures on a
package, but I don't think it helps us with this issue.
Nils
--
Nils Philippsen "Those who would give up Essential Liberty to purchase
Red Hat a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com nor Safety." -- Benjamin Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
6:33 a.m.
Nils Philippsen <nils <at> redhat.com> writes:
builds between
separate build systems (that have different profile data) will
inevitably differ.
That is a problem, I admit.
Second, it makes the update system more vulnerable to DOS attacks,
insofar as it only takes an attacker to hack himself into one of the
signatories to produce bad signatures on updates he wants to block (if
they e.g. contain security fixes). The risk of this is at least
proportionally higher with multiple targets to choose from (think RAID-0
and what it does to the probability of errors in such a set of disks).
If a signatory is producing bad signatures, just use another one. It is trivial
to discard packages signed by bad keys. It is also trivial to ask another
signatory from the pool to check and sign instead.
For instance, if there are 10 signatories in the pool and one of them starts
signing with a bad key etc., this will affect some number of packages (not all)
for a little while, all of which can be resigned by any of the other 9
signatories to make it up to N (which can be say 3 or something like that). That
DOS would not really last very long. In RAID terminology, you have plenty of hot
spares in the RAID-5 array.
Ditto if someone's key get compromised. Just revoke and let others sign. You
still have many eyes going over it.
Compare that to breaking the password on Fedora private key now. Much sweeter
for the attacker, as everything becomes wide open. In RAID terminology, you
single drive just died.
Third, unless a signatory runs his own build system to verify
package
builds (and disregarding my first point), his signature doesn't have
much value as he has to rely on the checksum data provided by the
package maintainer in his signed email -- which relies on the build
system not being compromised to begin with.
That is also not true. If the signatory receives a signed e-mail from the
packager with checksums in it (i.e. rpm -qp --dump, as per sv), the signatory
can verify that against someone else's build system that is not publicly
writeable (e.g. Matt's at Dell). Not everyone would need to run their own.
And remember, the attacker would have to break the build system _and_ fake the
signed e-mail from the packager AT THE SAME TIME and then get signatories to
sign without any checking. Much less likely then just compromising the build
system (which is what they can do now).
It is better to check against something then not to check at all, IMHO.
You know, maybe this would be a good opportunity for some cross-distro
cooperation. They could build our stuff on their build farms, we build their
stuff on ours (not everything - just updates). Then attackers need to compromise
all in order to get just one scalp. Quite an effort.
Then there's the additional burden on maintainers -- yet another
bureaucratic hurdle.
Yeah, security is always like that - pain in the arse. I still remember some
devs in the office chmod-ing everything to 777 because "it's easier" :-)
--
Bojan
5:30 a.m.
On Tue, 2008-08-26 at 11:33 +0000, Bojan Smojver wrote:
Nils Philippsen <nils <at> redhat.com> writes:
> builds between
> separate build systems (that have different profile data) will
> inevitably differ.
That is a problem, I admit.
> Second, it makes the update system more vulnerable to DOS attacks,
> insofar as it only takes an attacker to hack himself into one of the
> signatories to produce bad signatures on updates he wants to block (if
> they e.g. contain security fixes). The risk of this is at least
> proportionally higher with multiple targets to choose from (think RAID-0
> and what it does to the probability of errors in such a set of disks).
If a signatory is producing bad signatures, just use another one. It is trivial
to discard packages signed by bad keys. It is also trivial to ask another
signatory from the pool to check and sign instead.
[snip]
But that's not what you described in the original posting (emphasis on
"no bad ones"):
"""
With this we could then require N good signatures (and no bad ones)
on
each package before yum would trust the content.
"""
> Third, unless a signatory runs his own build system to verify
package
> builds (and disregarding my first point), his signature doesn't have
> much value as he has to rely on the checksum data provided by the
> package maintainer in his signed email -- which relies on the build
> system not being compromised to begin with.
That is also not true. If the signatory receives a signed e-mail from the
packager with checksums in it (i.e. rpm -qp --dump, as per sv), the signatory
can verify that against someone else's build system that is not publicly
writeable (e.g. Matt's at Dell). Not everyone would need to run their own.
The Matt or whoever else has a separate build system are the only
valuable signatories. Anybody else can only say "me too" but that
doesn't add anything to the trust value.
And remember, the attacker would have to break the build system _and_
fake the
signed e-mail from the packager AT THE SAME TIME and then get signatories to
sign without any checking. Much less likely then just compromising the build
system (which is what they can do now).
No, without independent verification builds, an attacker just has to
break the build system because a normal packager has no (sensible) way
to tell if e.g. a compromised compiler inserted trojan code into the
executables. Thus, a normal packager -- trusting what comes out of the
build system -- would just sign off whatever the resulting package is.
As independent verification builds aren't the deterministic thing that
you seem to think, that doesn't sound very helpful to me.
It is better to check against something then not to check at all,
IMHO.
It's like with spam filtering: bad positives are really bad. Do you
think repeating the hoopla of the last days just because of a bad
positive caused by a fluke in an independent build system we have no
control over is a good thing?
You know, maybe this would be a good opportunity for some
cross-distro
cooperation. They could build our stuff on their build farms, we build their
stuff on ours (not everything - just updates). Then attackers need to compromise
all in order to get just one scalp. Quite an effort.
Our packages built in say OpenSUSE's or Ubuntu's build system will
differ from what koji spits out, with a probability bordering on
certainty. I don't know what that gains us.
> Then there's the additional burden on maintainers -- yet
another
> bureaucratic hurdle.
Yeah, security is always like that - pain in the arse. I still remember some
devs in the office chmod-ing everything to 777 because "it's easier" :-)
I can live with PITA as long as there's a tangible benefit from it. I
don't see that here -- just because something is a pain in the arse, it
doesn't necessarily improve security.
Nils
--
Nils Philippsen "Those who would give up Essential Liberty to purchase
Red Hat a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com nor Safety." -- Benjamin Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
8:52 a.m.
Nils Philippsen <nils <at> redhat.com> writes:
But that's not what you described in the original posting
(emphasis on
"no bad ones"):
"""
> With this we could then require N good signatures (and no bad ones) on
> each package before yum would trust the content.
"""
I did not say that yum should use another one. The distribution system in charge
of accepting signatures would discard bad ones. The package would have to be
signed by someone else before distribution and before yum would trust it. Just a
slight delay, that's all.
The Matt or whoever else has a separate build system are the only
valuable signatories. Anybody else can only say "me too" but that
doesn't add anything to the trust value.
I don't think that's right. As long as you have something independent to compare
to, it is not just one person that can run the comparison. If you spread the
effort to more people, you lower the load.
No, without independent verification builds, an attacker just has to
break the build system because a normal packager has no (sensible) way
to tell if e.g. a compromised compiler inserted trojan code into the
executables. Thus, a normal packager -- trusting what comes out of the
build system -- would just sign off whatever the resulting package is.
I did not say that packagers would be signing the packages. I said that
signatories would be doing that. Having independent verification builds is a
must, because signatories would make sure builds match.
As independent verification builds aren't the deterministic thing
that
you seem to think, that doesn't sound very helpful to me.
Well, just because we base our build comparisons right now on something as crude
as raw checksums, doesn't mean this has to be like that forever. We may find
ways of comparing builds differently to determine if they were compromised, by
explicitly excluding well known differences within binaries.
It's like with spam filtering: bad positives are really bad. Do
you
think repeating the hoopla of the last days just because of a bad
positive caused by a fluke in an independent build system we have no
control over is a good thing?
The hoopla of the last few days would not be repeated. There would be no danger
to users, because multiple signatures would be required to inject a bad package
into users' machines. That's kinda the point of double checking. Nothing would
have to be taken down immediately and reinstalled.
Our packages built in say OpenSUSE's or Ubuntu's build system
will
differ from what koji spits out, with a probability bordering on
certainty. I don't know what that gains us.
I didn't say SUSE or Ubuntu folks would have to build Fedora packages on their
build systems, just their build farms (i.e. machines). Isn't koji open source?
If it is, they could run that just fine.
But hey - I get it, people don't like the additional hassle. That's OK. Consider
my proposal withdrawn until the time someone invents a way of reliably comparing
two independent builds.
--
Bojan
10:39 a.m.
On Wed, Aug 27, 2008 at 5:52 AM, Bojan Smojver <bojan(a)rexursive.com> wrote:
Well, just because we base our build comparisons right now on
something as crude
as raw checksums, doesn't mean this has to be like that forever. We may find
ways of comparing builds differently to determine if they were compromised, by
explicitly excluding well known differences within binaries.
How about you back up and just work on this very specific problem of
deterministically doing build comparisons across disparate build
systems ..before we even begin to discuss how a multiple sigantory
process which relies on that.
I didn't say SUSE or Ubuntu folks would have to build Fedora
packages on their
build systems, just their build farms (i.e. machines). Isn't koji open source?
If it is, they could run that just fine.
They would have to choose to run koji, instead of their own setups.
I'm not going to hold my breath on that. If you were going to lobby
them to use koji, you should start with OpenSuse and help them
integrate the features they need into the koji base. Even if the
multiple signatory idea does not pan out, having OpenSuse as koji user
and contributor would be great.
-jef
2:16 p.m.
Jeff Spaleta wrote:
On Wed, Aug 27, 2008 at 5:52 AM, Bojan Smojver
<bojan(a)rexursive.com> wrote:
> Well, just because we base our build comparisons right now on something as crude
> as raw checksums, doesn't mean this has to be like that forever. We may find
> ways of comparing builds differently to determine if they were compromised, by
> explicitly excluding well known differences within binaries.
How about you back up and just work on this very specific problem of
deterministically doing build comparisons across disparate build
systems ..before we even begin to discuss how a multiple sigantory
process which relies on that.
Virtual machines, virtually identical as starting points? But what if
it is the src rpm that is compromised so the builds will be identical
because they both contain the modification? Now you need to have the
packager verify again that the source that produced the 2 builds had
only the changes he intended.
And if you are really paranoid you have to wonder about the compiler and
any existing libraries too: http://cm.bell-labs.com/who/ken/trust.html.
--
Les Mikesell
lesmikesell(a)gmail.com
4:42 p.m.
Les Mikesell <lesmikesell <at> gmail.com> writes:
But what if
it is the src rpm that is compromised so the builds will be identical
because they both contain the modification?
That is not exactly the compromise of the build system and/or Fedora key, now is
it? If your own contributors are subverting the system by uploading borked
source, the mutli-key system isn't going to help (and I never claimed that).
For people that are not convinced in the usefulness of this (in principle), go
the a bank and try to open an account. See if they'll be OK with you producing
just one piece of ID.
--
Bojan
4:52 p.m.
On Wed, 2008-08-27 at 21:42 +0000, Bojan Smojver wrote:
Les Mikesell <lesmikesell <at> gmail.com> writes:
> But what if
> it is the src rpm that is compromised so the builds will be identical
> because they both contain the modification?
That is not exactly the compromise of the build system and/or Fedora key, now is
it? If your own contributors are subverting the system by uploading borked
source, the mutli-key system isn't going to help (and I never claimed that).
For people that are not convinced in the usefulness of this (in principle), go
the a bank and try to open an account. See if they'll be OK with you producing
just one piece of ID.
Not to fan the flames, but last time they did just that :-). And I'm
pretty sure they'll do it again, possibly because the IDs in question
are very hard to fake.
Nils
--
Nils Philippsen "Those who would give up Essential Liberty to purchase
Red Hat a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com nor Safety." -- Benjamin Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
5:12 p.m.
Nils Philippsen <nils <at> redhat.com> writes:
Not to fan the flames, but last time they did just that . And
I'm
pretty sure they'll do it again, possibly because the IDs in question
are very hard to fake.
May we all know the name of this bank? Just so that we can stay away from such
backyard operators.
--
Bojan
3:49 a.m.
We already discussed that in private, but let me rehash this for the
record:
On Wed, 2008-08-27 at 22:12 +0000, Bojan Smojver wrote:
Nils Philippsen <nils <at> redhat.com> writes:
> Not to fan the flames, but last time they did just that . And I'm
> pretty sure they'll do it again, possibly because the IDs in question
> are very hard to fake.
May we all know the name of this bank? Just so that we can stay away from such
backyard operators.
Just about every bank in Germany would operate like this. After all, we
do have a national ID that is hard enough to fake(*) and if you present
it to entities like banks, they usually make a copy and can verify the
data on the ID with the "Einwohnermeldeamt" (possibly "residents
registration office").
(*):
http://www.bundesdruckerei.de/en/products/products_idSystem/idSystem_docu...
Before someone uses terms like "Orwellian", "1984" and so on, there
are
some very tangible benefits from how this system is employed here: I
don't have to jump through hurdles to vote in elections, instead of
having to register for voting I'm invited to do that (by my
"Einwohnermeldeamt") every time there is one, since the day I turned 18.
It's very easy for me to prove (beyond reasonable doubt) that I am who I
am where that is necessary. Note that the police don't have the need to
know who you are without reasons. And it's quite comforting to know that
in situations where they can demand it, the can't make up their own
rules what they accept as ID and what not.
Nils
--
Nils Philippsen "Those who would give up Essential Liberty to purchase
Red Hat a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com nor Safety." -- Benjamin Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
5:07 a.m.
On Thu, 2008-08-28 at 10:49 +0200, Nils Philippsen wrote:
Just about every bank in Germany would operate like this. After all,
we
do have a national ID that is hard enough to fake(*) and if you present
it to entities like banks, they usually make a copy and can verify the
data on the ID with the "Einwohnermeldeamt" (possibly "residents
registration office").
In other words, they only marginally trust that ID card. Therefore, they
cross-check with the office (but most likely NOT the person that issued
the card) that what is on that document is ACTUALLY true.
Think of the card as the RPM coming out of Fedora build system, signed
with Fedora key (i.e. all the hard to get paraphernalia that the card is
made of). Think of the phone call to the office as a comparison to an
independently built RPM. Think of the act of opening the bank account as
an independent signatory signing the RPM.
--
Bojan
5:48 a.m.
On Thu, 2008-08-28 at 20:07 +1000, Bojan Smojver wrote:
On Thu, 2008-08-28 at 10:49 +0200, Nils Philippsen wrote:
> Just about every bank in Germany would operate like this. After all, we
> do have a national ID that is hard enough to fake(*) and if you present
> it to entities like banks, they usually make a copy and can verify the
> data on the ID with the "Einwohnermeldeamt" (possibly "residents
> registration office").
In other words, they only marginally trust that ID card. Therefore, they
cross-check with the office (but most likely NOT the person that issued
the card) that what is on that document is ACTUALLY true.
I don't know if they do that at all or with everybody, in fact I'm
pretty sure they will accept the IDs at face value. Maybe they verify
old documents that don't have all the gizmos that make the ID hard to
fake today. I just wanted to point out that they can do that.
Think of the card as the RPM coming out of Fedora build system,
signed
with Fedora key (i.e. all the hard to get paraphernalia that the card is
made of). Think of the phone call to the office as a comparison to an
independently built RPM. Think of the act of opening the bank account as
an independent signatory signing the RPM.
All this doesn't help much if somebody manages to feed the process with
corrupt data in the beginning -- be that a fake ID when applying for
replacement for an expired ID, or somebody corrupting the repository
that keeps the source that is fed to the builders. Besides the
difficulties in making distributed build verification work solidly, we
have to harden the system/process that feeds the builders in the first
place -- which is already much easier than say hacking the builders.
Otherwise it's just GIGO, all pain and no gain.
Nils
--
Nils Philippsen "Those who would give up Essential Liberty to purchase
Red Hat a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com nor Safety." -- Benjamin Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
6:15 a.m.
On Thu, 2008-08-28 at 12:48 +0200, Nils Philippsen wrote:
I don't know if they do that at all or with everybody, in fact
I'm
pretty sure they will accept the IDs at face value. Maybe they verify
old documents that don't have all the gizmos that make the ID hard to
fake today. I just wanted to point out that they can do that.
Yeah, it's like customs. You can scrape by, but if they do cross-check,
you're history if you have nasties in your bag :-)
All this doesn't help much if somebody manages to feed the
process with
corrupt data in the beginning
I think I mentioned already that such a thing would be (obviously)
beyond the scope of this.
But look, it is obvious that we (and by this I mean myself and the
majority of people in the thread) have fundamentally contradicting views
about this, so let's agree to disagree, OK? I don't want to drag this
forever.
Consider my proposal withdrawn.
--
Bojan
6:27 a.m.
Nils Philippsen wrote:
On Thu, 2008-08-28 at 20:07 +1000, Bojan Smojver wrote:
> On Thu, 2008-08-28 at 10:49 +0200, Nils Philippsen wrote:
>
>> Just about every bank in Germany would operate like this. After all, we
>> do have a national ID that is hard enough to fake(*) and if you present
>> it to entities like banks, they usually make a copy and can verify the
>> data on the ID with the "Einwohnermeldeamt" (possibly "residents
>> registration office").
> In other words, they only marginally trust that ID card. Therefore, they
> cross-check with the office (but most likely NOT the person that issued
> the card) that what is on that document is ACTUALLY true.
I don't know if they do that at all or with everybody, in fact I'm
pretty sure they will accept the IDs at face value. Maybe they verify
old documents that don't have all the gizmos that make the ID hard to
fake today. I just wanted to point out that they can do that.
> Think of the card as the RPM coming out of Fedora build system, signed
> with Fedora key (i.e. all the hard to get paraphernalia that the card is
> made of). Think of the phone call to the office as a comparison to an
> independently built RPM. Think of the act of opening the bank account as
> an independent signatory signing the RPM.
All this doesn't help much if somebody manages to feed the process with
corrupt data in the beginning -- be that a fake ID when applying for
replacement for an expired ID, or somebody corrupting the repository
that keeps the source that is fed to the builders. Besides the
difficulties in making distributed build verification work solidly, we
have to harden the system/process that feeds the builders in the first
place -- which is already much easier than say hacking the builders.
Otherwise it's just GIGO, all pain and no gain.
Nils
It is (almost) all about process. Documented, auditable, verifiable
process that is reviewed periodically by someone other than those using
it to make sure it is in fact being followed, verified, audited, etc.
In a sense, fedora and rht just had a hostile audit. They failed. Now
close the feedback loop and use what you've learned to fix the process.
New keys are nice, but ultimately useless w/o a better process to
safeguard them, to ensure their correct usage, by the correct people on
the correct content, and to enable their revokation if it happens again.
As Schneirer says, good security is hard. This is not a new path,
nothing needs to be invented, just some good solid principles
(separation of duty, verification of trust, independent review and
audit, etc) followed.
Apparently, there was not enough process in place, or the processes that
were in place were not followed, or they are (hopefully were) the wrong
processes.
--
----------------------------------------------------------------------
Jim Wildman, CISSP, RHCE jim(a)rossberry.com http://www.rossberry.com
"Society in every state is a blessing, but Government, even in its best
state, is a necessary evil; in its worst state, an intolerable one."
Thomas Paine
6:41 p.m.
Bojan Smojver wrote:
> But what if
> it is the src rpm that is compromised so the builds will be identical
> because they both contain the modification?
That is not exactly the compromise of the build system and/or Fedora key, now is
it?
Is one significantly harder than the other? If it goes unnoticed the end
result could be the same.
If your own contributors are subverting the system by uploading
borked
source, the mutli-key system isn't going to help (and I never claimed that).
I'm not proposing an intentional trojan source submission, but a
compromise that modifies it in an unexpected way. I'd think if you go to
the trouble to compare builds you'd also want an end-to-end validity
check on the input to be sure it wasn't compromised either at the source
or in transit.
--
Les Mikesell
lesmikesell(a)gmail.com
7:16 p.m.
Les Mikesell <lesmikesell <at> gmail.com> writes:
Is one significantly harder than the other? If it goes unnoticed the
end
result could be the same.
This depends on many factors and it has no simple answer. Packagers usually rely
on checksums and signatures of upstream source before uploading to Fedora CVS
(or so I hope :-). But, there is always a possibility that this can be
compromised, of course.
The result is, however, not the same. Compromise of Fedora build system and/or
key can compromise all packages in Fedora. Compromise of a single package cannot
(generally speaking - there are exceptions).
In any event, just because one security measure doesn't help with every possible
compromise, doesn't mean it doesn't help at all.
I'm not proposing an intentional trojan source submission, but a
compromise that modifies it in an unexpected way. I'd think if you go to
the trouble to compare builds you'd also want an end-to-end validity
check on the input to be sure it wasn't compromised either at the source
or in transit.
See above. But yeah, we may have signatories sign off on source RPMs first,
before they are being built by alternative, independent build systems. It's a
valid point.
The main idea remains the same: "You can fool some of the people all of the
time, and all of the people some of the time, but you can not fool all of the
people all of the time."
--
Bojan
4:37 p.m.
Jeff Spaleta <jspaleta <at> gmail.com> writes:
How about you back up and just work on this very specific problem of
deterministically doing build comparisons across disparate build
systems ..before we even begin to discuss how a multiple sigantory
process which relies on that.
We didn't even find out how many packages differ in significant ways by doing
raw checksums. But look, I already said that my proposal is withdrawn until we
invent a way of doing this correctly.
They would have to choose to run koji, instead of their own setups.
If you re-read the thread, you'll notice that I said that we'd also build stuff
for them (i.e. we'd also run their setups). You know, this thing called
cooperation. It would not be just one way.
--
Bojan
6:10 p.m.
Jeff Spaleta <jspaleta <at> gmail.com> writes:
They would have to choose to run koji, instead of their own setups.
I'm not going to hold my breath on that. If you were going to lobby
them to use koji, you should start with OpenSuse and help them
integrate the features they need into the koji base. Even if the
multiple signatory idea does not pan out, having OpenSuse as koji user
and contributor would be great.
I completely missed here that you seem to be under the impression that I'm
suggesting that everyone should accept Fedora's build system as their own.
Absolutely not (monoculture sucks anyway). This would all be in addition to what
they already have and completely unrelated to how they build their own software.
--
Bojan
6:31 p.m.
This is an interesting idea. Cross checking on multiple build mashines
with private signed email (containg the checksums of source and
binary, stripped away from machine specific data) send to the
signatories - this all makes it alot difficult for someone to crack.
5735
days inactive
5744
days old
41 comments
14 participants
participants (14)
-
Andrew Bartlett -
Bill Crawford -
Bojan Smojver -
Bruno Wolff III -
casimiro barreto -
Chris Adams -
Jef Spaleta -
Jim Wildman -
Joshua C. -
Les Mikesell -
Nils Philippsen -
Seth Vidal -
Stephen John Smoogen -
Tom Lane