[PATCH] distgit: Upload files to both the new and old path

[Mathieu Bridon]

On Fri, 2015-06-12 at 10:14 -0500, Dennis Gilmore wrote:
> On Friday, June 12, 2015 05:02:00 PM Mathieu Bridon wrote:
> > On Fri, 2015-06-12 at 09:18 -0500, Dennis Gilmore wrote:
> > > On Thursday, June 11, 2015 04:59:22 PM Mathieu Bridon wrote:
> > > > On Wed, 2015-06-10 at 10:33 -0500, Dennis Gilmore wrote:
> > > > > I think we should be linking to the old location and a 
> > > > > sha512sum
> > > > > location not md5 but the general idea is okay
> > > > 
> > > > Doing the new location for md5 as well has a pretty big benefit
> > > > though:
> > > > it means we can update fedpkg to make it download from the new
> > > > location
> > > > independently from the move to sha512.
> > > 
> > > > Decoupling the two makes for an easier migration:
> > > I strongly want a flag day for the migration. because there is 
> > > other
> > > things
> > > that need to be coupled with it. such as the changes to ca setup 
> > > to
> > > use our ca
> > > for logins but a well known ca for koji.fp.o I want to decouple
> > > thekoji hub
> > > and web interface also, which will mean we need a new hostname 
> > > for
> > > the hub.
> > > all of these changes have a hard requirement of a flag day as end
> > > users will
> > > need to make changes on the client side.
> > 
> > Why do you want to couple this change with other, unrelated ones?
> I want to couple the change to sha512 with the others so we have a 
> single path 
> of communication. 

Sure, for the client-side changes it makes sense to bundle them.

Remember that this thread was about the server-side changes of the
path/sha512 migration, which are all done now, and needed no outage.

> > We're almost done on this change (changing the download path), and 
> > very
> > few more is needed to switch to sha512.
> > 
> > I really see no reason to block all the server-side changes on
> > unrelated changes like the koji CA.
> > 
> > Switching to a well-known CA requires very few code change to 
> > fedpkg,
> > so sure, that can be coupled with switching fedpkg to sha512.
> This is fine. it requires all users to get new koji configs as the 
> old ones 
> will no longer work. unless we move the urls for koji entirely.

Yup.

I'm not yet ready to send the changes to fedpkg that actually do the
switch anyway, but when I send them, I'll make it clear that they
should only be pushed as an update along with the other changes you
want.

> > > > - On the server, new uploads are stored in both locations
> > > > **right
> > > > now**
> > > > (the patches we were discussing in this thread have all been 
> > > > merged
> > > > and
> > > > deployed to prod)
> > > > - I have the fedpkg patches to change the download location, I 
> > > > just
> > > > need to submit them. Also, as I have written it, fedpkg will 
> > > > try
> > > > the
> > > > new location first and fallback to the old one if needed.
> > > > 
> > > > So as soon as I send those fedpkg patches (still need a bit of
> > > > time),
> > > > we can release a new fedpkg with them, and the migration to the 
> > > > new
> > > > path will effectively be done.
> > > > 
> > > > At that point, for an source file which had only be uploaded
> > > > **before**
> > > > 
> > > > we changed the upload.cgi script, we'd get this:
> > > >   $ fedpkg sources
> > > >   Downloading libcangjie-1.3.tar.xz
> > > >   ######################################################### 
> > > > 100.0%
> > > >   Download failed, falling back to the old URL
> > > >   Downloading libcangjie-1.3.tar.xz
> > > >   ######################################################### 
> > > > 100.0%
> > > > 
> > > > But for a source file uploaded **after** the changes to the
> > > > upload.cgi
> > > > 
> > > > scrip, we'd get this:
> > > >   $ fedpkg sources
> > > >   Downloading libcangjie-1.3.tar.xz
> > > >   ######################################################### 
> > > > 100.0%
> > > > 
> > > > Which means at that point, the migration to the new path would 
> > > > be
> > > > pretty much done.
> > > > 
> > > > <optional step>
> > > > I have a script ready, to be run on the server, which will 
> > > > hardlink
> > > > all
> > > > existing uploads from their old to their new path.
> > > 
> > > this is not optional. we have to link all the tarballs to the 
> > > sha512
> > > locations.
> > 
> > That's not at all what this paragraph was about. I assume you 
> > instead
> > wanted to reply to the other optional thing I mention a few 
> > paragraphs
> > below.
> > 
> > I'm fine with making it required if you want, even though I don't 
> > see
> > why it would matter.
> The reason I want it mandatory is that it would be less confusion for 
> users and potentially less bandwidth if they go to upload a new 
> tarball thats already there,

Nope.

This is something Toshio requested earlier in this thread, and I
implemented it. It's already deployed in production.

I already explained how it works, but basically, if a packager tries to
upload a file which only exists at the old path, then it will get
hardlinked to the new path.

There will be no actual new upload, no additional bandwidth used.

>  it's just cosmetic.

Indeed it is.

> > It wouldn't matter because fedpkg will decide which hash to use for
> > **downloads** based on the content of the 'sources' file.
> > 
> > And for already uploaded tarballs, the 'sources' file will tell 
> > fedpkg
> > to use md5, not sha512.
> > 
> > As a result, linking all the old tarballs to the sha512 location is 
> > not
> > needed at all for the fedpkg users (packagers, koji builds, ...) to
> > continue working as expected.
> > 
> > The only reason it could be needed is if we decided that after a
> > certain point, fedpkg would only ever download with sha512. But 
> > that
> > would require updating all the 'sources' files for all branches of 
> > all
> > packages, and I remember quite clearly that you told me we wouldn't 
> > do
> > that. (in that tiny cubicle we were sharing at the Red Hat office 
> > in
> > Brno, around Devconf)
> > 
> > In any case, what I'm saying is that it's not **needed**. But I'm 
> > not
> > against doing it anyway if you prefer, as it's somewhat cleaner to 
> > have
> > all paths working with sha512, and it wouldn't be disruptive at all
> > anyway. :)
> I think we are on the same page here

Cool, glad it was just a misunderstanding. :)

> > > > We could run it to get all files in their new path with md5, 
> > > > which
> > > > would remove the warning from the above "fedpkg sources" output
> > > > even
> > > > for old uploads, but that's a cosmetic issue, so it's not even
> > > > entirely
> > > > required.
> > > > </optional step>
> > > > 
> > > > And then, I have patches ready for fedpkg that I'll submit at 
> > > > that
> > > > point, which just switch to sha512 and the new 'sources' file
> > > > format.
> > > > (the BSD-style one, which contains the hashtype)
> > > > 
> > > > We'd push that out, and it would be enough to migrate to 
> > > > sha512:
> > > > - fedpkg would upload new source files with sha512, they'd 
> > > > appear
> > > > only
> > > > in their new path
> > > > - fedpkg would know what hashtype to use for downloading, based 
> > > > on
> > > > the
> > > > 'sources' file, so it would still download just fine old 
> > > > uploads
> > > > with
> > > > md5.
> > > > 
> > > > <optional step>
> > > > And if we really want to completely get rid of all md5, then we
> > > > could
> > > > run a script on the server-side to get the files in their new 
> > > > path
> > > > for
> > > > sha512 as well, even for old uploads.
> > > > 
> > > > And then we'd push an update to fedpkg which would ask package
> > > > maintainers to run a "fedpkg new-sources" on their source files 
> > > > if
> > > > they
> > > > are still using md5 hashes, so that all the 'sources' files in
> > > > distgit
> > > > will end up pointing to sha512 hashes.
> > > > </optional step>
> > > > 
> > > > ----
> > > > 
> > > > All in all, that makes for a pretty smooth migration, with no 
> > > > flag
> > > > day
> > > > or outage during which we'd need to cut uploads temporarily.
> > > 
> > > This is not even close to the only reason for a flag day. we have 
> > > to
> > > have one
> > > and we had a plan to roll it all into one, please do not go 
> > > changing
> > > that.
> > 
> > So let me clarify a bit.
> > 
> > What I meant is that with the migration plan I detailed above, 
> > there
> > wouldn't be a day on which we cut uploads temporarily, do some 
> > things,
> > then reopen uploads.
> > 
> > There would certainly be a flag day when we decide to stop 
> > accepting md5 uploads, but that needs to be after fedpkg is set to 
> > upload with sha512, and that update has reached stable.
> I do not think it has to be stable. I think updates-testing is fine.

Sure, that doesn't change much to my point though. :)


-- 
Mathieu