Stronger hashes for the lookaside cache: Migration plan

[Mathieu Bridon]

Hi,

At the April 1st meeting, I promised to send a migration path for ticket
5846:

    https://fedorahosted.org/rel-eng/ticket/5846

I've been a bit busy lately, so I couldn't get to it before today.

So, here's how I saw things, and what my last patchset allows.

----------

A few notes before I start: in the plan below I'm going to assume we
decided to move to using only sha512. This is just for the sake of
keeping the writing simpler, but we could decide on another hash, or to
use several ones.

----------

The first three patches to be merged would be on the infrastructure
side:
https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/014190.html
https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/014188.html
https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/014189.html
https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/014191.html

With that deployed in production, pretty much nothing changes for
clients: they can still upload tarballs using md5, just like right now.

However, the server-side starts accepting uploads with sha512.

At the same time, we can apply the client-side patch to fedpkg:
https://lists.fedoraproject.org/pipermail/rel-eng/2014-March/017458.html

(apologies for the 2/2 there, that's a mistake, there is only one patch)

That simply removes the assumption that we're always uploading with md5,
instead respecting the 'lookaside_hash' parameter
in /etc/rpkg/$site.conf... which is still md5 right now.

So with all the above applied and deployed, nothing changes. :)

The last things to do are:

- set the `lookaside_hash` parameter to `sha512` in fedpkg.conf
(one-line patch not yet submitted)

- copy all the archives currently on the lookaside cache into the proper
path based on their sha512 hash (with hardlinking to save space)

- on a cut-off date, drop the md5 fallback from the server-side cgi
script (4 lines to remove, patch not yet submitted)

The last point above would obviously need to happen **after** the first
one (i.e a new fedpkg landed in stable with that new configuration
value), pretty much as soon as we're ready.

At that point, new archives are uploaded exclusively using sha512, but
old archives are available both as md5 or sha512.

----------

Does the above plan seem agreeable? Anything I missed?

One thing that was requested earlier was to have the hash type as part
of the path on the lookaside cache. I'll need to submit a new patch for
that, but it should be trivial.

Cheers,


-- 
Mathieu