Lookaside: Move away from md5

[Mathieu Bridon]

We should move to something more secure than md5 for the uploaded sources.

This patch series implements the client-side part of this change.

We might want to drop the md5 fallback once we have migrated completely, that
is when:

1. all archives on the lookaside have been moved to a stronger hash
2. the "sources" file in all git repos has been updated to the same hash

https://fedorahosted.org/rel-eng/ticket/5846

[PATCH 1/2] Verify source hashes with a fallback on md5

[Mathieu Bridon]

We are about to move away from md5 for hashes.

However, at the time we move, some source tarballs might still exist
only with a md5 hash.

This allows verifying these old archives by falling back on md5.

We might want to just revert this commit eventually, when are confident
that md5 is completely gone (i.e that we have migrated the whole
lookaside cache to the new hash).
---
 src/fedpkg/__init__.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py
index 910e448..55ce0e6 100644
--- a/src/fedpkg/__init__.py
+++ b/src/fedpkg/__init__.py
@@ -249,6 +249,15 @@ class Commands(pyrpkg.Commands):
     def build(self, *args, **kwargs):
         return(super(Commands, self).build(*args, **kwargs))
 
+    def _verify_file(self, file, hash, hashtype):
+        valid = super(Commands, self)._verify_file(file, hash, hashtype)
+
+        if not valid:
+            # We're moving away from md5, so try falling back on md5 for some time
+            valid = super(Commands, self)._verify_file(file, hash, "md5")
+
+        return valid
+
     # New functionality
     def _create_curl(self):
         """Common curl setup options used for all requests to
lookaside."""
-- 
1.8.5.3

[PATCH 1/2] Verify source hashes with a fallback on md5

[Pavol Babincak]

Just for the record I'm not going to merge this one as we will work with 
different approach as outlined in buildsys mailing list 
https://lists.fedoraproject.org/pipermail/buildsys/2015-February/ (I 
haven't found the latest e-mail via web archive yet).


On 03/17/2014 01:05 PM, Mathieu Bridon wrote:
>  We are about to move away from md5 for hashes.
>
>  However, at the time we move, some source tarballs might still exist
>  only with a md5 hash.
>
>  This allows verifying these old archives by falling back on md5.
>
>  We might want to just revert this commit eventually, when are confident
>  that md5 is completely gone (i.e that we have migrated the whole
>  lookaside cache to the new hash).
>  ---
>    src/fedpkg/__init__.py | 9 +++++++++
>    1 file changed, 9 insertions(+)
>
>  diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py
>  index 910e448..55ce0e6 100644
>  --- a/src/fedpkg/__init__.py
>  +++ b/src/fedpkg/__init__.py
>  @@ -249,6 +249,15 @@ class Commands(pyrpkg.Commands):
>        def build(self, *args, **kwargs):
>            return(super(Commands, self).build(*args, **kwargs))
>
>  +    def _verify_file(self, file, hash, hashtype):
>  +        valid = super(Commands, self)._verify_file(file, hash, hashtype)
>  +
>  +        if not valid:
>  +            # We're moving away from md5, so try falling back on md5 for some time
>  +            valid = super(Commands, self)._verify_file(file, hash, "md5")
>  +
>  +        return valid
>  +
>        # New functionality
>        def _create_curl(self):
>            """Common curl setup options used for all requests to
> lookaside."""
>  

-- 
Pavol Babincak
Release Engineering, Red Hat

[PATCH 1/2] Verify source hashes with a fallback on md5

[Pierre-Yves Chibon]

On Tue, Feb 10, 2015 at 02:38:12PM +0100, Pavol Babincak wrote:
>  Just for the record I'm not going to merge this one as we will
> work with
>  different approach as outlined in buildsys mailing list
>  https://lists.fedoraproject.org/pipermail/buildsys/2015-February/ (I haven't
>  found the latest e-mail via web archive yet). 
When you find the email could you send it back this way?
I'm curious about which approach you want to take.


Thanks,
Pierre

>  
>  On 03/17/2014 01:05 PM, Mathieu Bridon wrote:
>  >We are about to move away from md5 for hashes.
>  >
>  >However, at the time we move, some source tarballs might still exist
>  >only with a md5 hash.
>  >
>  >This allows verifying these old archives by falling back on md5.
>  >
>  >We might want to just revert this commit eventually, when are confident
>  >that md5 is completely gone (i.e that we have migrated the whole
>  >lookaside cache to the new hash).
>  >---
>  >  src/fedpkg/__init__.py | 9 +++++++++
>  >  1 file changed, 9 insertions(+)
>  >
>  >diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py
>  >index 910e448..55ce0e6 100644
>  >--- a/src/fedpkg/__init__.py
>  >+++ b/src/fedpkg/__init__.py
>  >@@ -249,6 +249,15 @@ class Commands(pyrpkg.Commands):
>  >      def build(self, *args, **kwargs):
>  >          return(super(Commands, self).build(*args, **kwargs))
>  >
>  >+    def _verify_file(self, file, hash, hashtype):
>  >+        valid = super(Commands, self)._verify_file(file, hash, hashtype)
>  >+
>  >+        if not valid:
>  >+            # We're moving away from md5, so try falling back on md5 for some
> time
>  >+            valid = super(Commands, self)._verify_file(file, hash, "md5")
>  >+
>  >+        return valid
>  >+
>  >      # New functionality
>  >      def _create_curl(self):
>  >          """Common curl setup options used for all requests to
> lookaside."""
>  >
>  
>  
>  -- 
>  Pavol Babincak
>  Release Engineering, Red Hat
>  _______________________________________________
>  rel-eng mailing list
>  rel-eng(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/rel-eng

[PATCH 1/2] Verify source hashes with a fallback on md5

[Pavol Babincak]

On 02/10/2015 02:59 PM, Pierre-Yves Chibon wrote:
>  On Tue, Feb 10, 2015 at 02:38:12PM +0100, Pavol Babincak wrote:
> > Just for the record I'm not going to merge this one as we will work with
> > different approach as outlined in buildsys mailing list
> > https://lists.fedoraproject.org/pipermail/buildsys/2015-February/ (I haven't
> > found the latest e-mail via web archive yet).
>
>  When you find the email could you send it back this way?
>  I'm curious about which approach you want to take.
https://lists.fedoraproject.org/pipermail/buildsys/2015-February/004514.html

And Mathieu already sent patches for pyrpkg (which I'm going to review 
soon):

https://lists.fedoraproject.org/pipermail/buildsys/2015-February/thread.h...
-- 
Pavol Babincak
Release Engineering, Red Hat

[PATCH 2/2] Upload source files with our preferred hash

[Mathieu Bridon]

We are moving away from md5 for hashes.

This gets the upload command to just upload the files with the
configured lookaside hash.

It only requires a corresponding change to the lookaside cache when we
change the actual config to something else than 'md5'.

That will come in a future commit, though.
---
 src/fedpkg/__init__.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py
index 55ce0e6..1a5c385 100644
--- a/src/fedpkg/__init__.py
+++ b/src/fedpkg/__init__.py
@@ -286,7 +286,8 @@ class Commands(pyrpkg.Commands):
         # This is overloaded to add in the fedora user's cert
         cmd = ['curl', '-k', '--cert', self.cert_file,
'--fail', '-o',
                '/dev/null', '--show-error', '--progress-bar',
'-F',
-               'name=%s' % self.module_name, '-F', 'md5sum=%s' %
file_hash,
+               'name=%s' % self.module_name,
+               '-F', '%ssum=%s' % (self.lookasidehash, file_hash),
                '-F', 'file=@%s' % file]
         if self.quiet:
             cmd.append('-s')
-- 
1.8.5.3

[PATCH 2/2] Upload source files with our preferred hash

[Pavol Babincak]

Applied finally. Sorry for the delay.

On 03/17/2014 01:05 PM, Mathieu Bridon wrote:
>  We are moving away from md5 for hashes.
>
>  This gets the upload command to just upload the files with the
>  configured lookaside hash.
>
>  It only requires a corresponding change to the lookaside cache when we
>  change the actual config to something else than 'md5'.
>
>  That will come in a future commit, though.
>  ---
>    src/fedpkg/__init__.py | 3 ++-
>    1 file changed, 2 insertions(+), 1 deletion(-)
>
>  diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py
>  index 55ce0e6..1a5c385 100644
>  --- a/src/fedpkg/__init__.py
>  +++ b/src/fedpkg/__init__.py
>  @@ -286,7 +286,8 @@ class Commands(pyrpkg.Commands):
>            # This is overloaded to add in the fedora user's cert
>            cmd = ['curl', '-k', '--cert', self.cert_file,
> '--fail', '-o',
>                   '/dev/null', '--show-error', '--progress-bar',
> '-F',
>  -               'name=%s' % self.module_name, '-F', 'md5sum=%s' %
> file_hash,
>  +               'name=%s' % self.module_name,
>  +               '-F', '%ssum=%s' % (self.lookasidehash, file_hash),
>                   '-F', 'file=@%s' % file]
>            if self.quiet:
>                cmd.append('-s')
>  

-- 
Pavol Babincak
Release Engineering, Red Hat

[Mathieu Bridon]

So, a couple of things came out of the meeting on Monday concerning
these changes.

----

First, people seemed to be in agreement that having the hash type in the
path on the lookaside cache was a good idea.

That should just be a trivial change to one of the patches:
https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/01418...

I'll submit a new version of that one soon.

----

However, the discussion showed that we didn't have a clear consensus on
whether we would use only one hash or several, and which one(s) we'd
use.

To me, one thing is clear: we should keep the existing md5 hashes around
for some time, at least as long as some package modules still use a md5
hash in their "sources" file.

Dennis wants us to "convert" all of these to stronger hashes, which we
could certainly do. (I'll start looking into writing a script that does
that soon)

But the question is about what to do on new uploads.

I would personally prefer that we stored only one hash, and that we'd
use the strongest currently available. (that would be sha512?)

But Till made the point that storing multiple hashes makes it easier to
compare our sources with other projects or upstreams (as they might have
hashes for different algorithms).

At this point, I don't mind implementing that, but it seems others
preferred having only one hash...

As I said in the meeting, I'm happy to implement whatever ends up being
the consensus. Can we build one? :)


-- 
Mathieu

[Bruno Wolff III]

On Wed, Mar 19, 2014 at 13:23:27 +0800,
   Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
> I would personally prefer that we stored only one hash, and that we'd
> use the strongest currently available. (that would be sha512?) 
The sha-3 competition is done (over a year ago) and Keccak was selected. I 
think it would make more sense to go to that rather than sha512.
https://en.wikipedia.org/wiki/SHA-3

[Kevin Fenzi]

On Wed, 19 Mar 2014 09:58:20 -0500
Bruno Wolff III <bruno(a)wolff.to&gt; wrote:

>  On Wed, Mar 19, 2014 at 13:23:27 +0800,
>     Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
>  >
>  >I would personally prefer that we stored only one hash, and that we'd
>  >use the strongest currently available. (that would be sha512?)
>  
>  The sha-3 competition is done (over a year ago) and Keccak was
>  selected. I think it would make more sense to go to that rather than
>  sha512. https://en.wikipedia.org/wiki/SHA-3 
I did a quick look around, but likely missed it... are there actually
any tools that implement this yet?

kevin

[Bruno Wolff III]

On Wed, Mar 19, 2014 at 09:36:52 -0600,
   Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
> On Wed, 19 Mar 2014 09:58:20 -0500
> Bruno Wolff III <bruno(a)wolff.to&gt; wrote:
>
> > On Wed, Mar 19, 2014 at 13:23:27 +0800,
> >    Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
> > >
> > >I would personally prefer that we stored only one hash, and that we'd
> > >use the strongest currently available. (that would be sha512?)
> >
> > The sha-3 competition is done (over a year ago) and Keccak was
> > selected. I think it would make more sense to go to that rather than
> > sha512. https://en.wikipedia.org/wiki/SHA-3
>
> I did a quick look around, but likely missed it... are there actually
> any tools that implement this yet? 
Looking into this further, even though Keccak was picked over a year ago, 
there is still a standardization process to pick some of the parameters 
that doesn't look like has happened yet. There appears to be a good 
project to watch for this:
https://github.com/maandree/sha3sum

But I was wrong about this being a good time to switch to it. If we use it 
now, we might end up needing to recompute the hashes again later. So for 
now it looks like one of the sha-2 algorthims should be used.