[PATCH] distgit: Upload files to both the new and old path

[Mathieu Bridon]

On Wed, 2015-06-10 at 10:33 -0500, Dennis Gilmore wrote:
> I think we should be linking to the old location and a sha512sum
> location not md5 but the general idea is okay

Doing the new location for md5 as well has a pretty big benefit though:
it means we can update fedpkg to make it download from the new location
independently from the move to sha512.

Decoupling the two makes for an easier migration:

- On the server, new uploads are stored in both locations **right now**
(the patches we were discussing in this thread have all been merged and
deployed to prod)
- I have the fedpkg patches to change the download location, I just
need to submit them. Also, as I have written it, fedpkg will try the
new location first and fallback to the old one if needed.

So as soon as I send those fedpkg patches (still need a bit of time),
we can release a new fedpkg with them, and the migration to the new
path will effectively be done.

At that point, for an source file which had only be uploaded **before**
we changed the upload.cgi script, we'd get this:

  $ fedpkg sources
  Downloading libcangjie-1.3.tar.xz
  ######################################################### 100.0%
  Download failed, falling back to the old URL
  Downloading libcangjie-1.3.tar.xz
  ######################################################### 100.0%

But for a source file uploaded **after** the changes to the upload.cgi
scrip, we'd get this:

  $ fedpkg sources
  Downloading libcangjie-1.3.tar.xz
  ######################################################### 100.0%

Which means at that point, the migration to the new path would be
pretty much done.

<optional step>
I have a script ready, to be run on the server, which will hardlink all
existing uploads from their old to their new path.

We could run it to get all files in their new path with md5, which
would remove the warning from the above "fedpkg sources" output even
for old uploads, but that's a cosmetic issue, so it's not even entirely
required.
</optional step>

And then, I have patches ready for fedpkg that I'll submit at that
point, which just switch to sha512 and the new 'sources' file format.
(the BSD-style one, which contains the hashtype)

We'd push that out, and it would be enough to migrate to sha512:
- fedpkg would upload new source files with sha512, they'd appear only
in their new path
- fedpkg would know what hashtype to use for downloading, based on the
'sources' file, so it would still download just fine old uploads with
md5.

<optional step>
And if we really want to completely get rid of all md5, then we could
run a script on the server-side to get the files in their new path for
sha512 as well, even for old uploads.

And then we'd push an update to fedpkg which would ask package
maintainers to run a "fedpkg new-sources" on their source files if they
are still using md5 hashes, so that all the 'sources' files in distgit
will end up pointing to sha512 hashes.
</optional step>

----

All in all, that makes for a pretty smooth migration, with no flag day
or outage during which we'd need to cut uploads temporarily.

Which is why the upload.cgi script now puts all files in their new
path, even for md5 uploads. :)



-- 
Mathieu