RFC: Old packages remain on the mirrors for one week

[Dennis Gilmore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 15 Aug 2013 23:48:46 -0500
Dennis Gilmore <dennis at ausil.us> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Mon, 12 Aug 2013 14:47:03 +0100
> Richard Hughes <hughsient at gmail.com> wrote:
> 
> > Hi all,
> > 
> > I'd like to ask for comments on a feature I need for the Fedora
> > Application Installer. The current yum backend in PackageKit does
> > something like this:
> > 
> > * yum install foo
> > * depsolve transaction using cached metadata
> > * download foo-0.1.noarch.rpm
> > * error! foo-0.1.noarch.rpm doesn't exist
> > * download latest repomd, primary
> > * re-depsolve
> > * download latest filelists
> > * continue to re-depsolve
> > * download foo-0.2.noarch.rpm
> > * install foo using librpm
> > 
> > Now, we do this as the metadata is cached on the client side for up
> > to a week as we don't want to unconditionally update the metadata
> > for every transaction, but we don't know if we can download the
> > package without downloading all the metadata beforehand. This is
> > incompatible with the swish UX in the application installer where
> > we can search for things straight away without having
> > "Downloading..." in the UI appearing at odd times. So my proposal
> > is thus:
> > 
> > 1. We retain old packages on the mirrors for a minimum of 7 days.
> 
> without completely rewriting how we compose the trees this is not a
> possibility.
> 
> > 2. We regenerate the metadata on every compose like before
> > 3. We only include the latest package version in the metadata
> 
> this would need the tools to be completely rewritten also.
> 
> > 4. If the user is installing an "old" package we check if the new
> > package is a security or important update and re-download all
> > metadata if so
> this means downloading all the metadata
an option here would be to have a separate security updates repo. we
would need to take steps to make sure it doesn't result in broken deps.
but it would mean you could download security updates metadata
separately which would be much smaller and simpler to consume.

It would require some separate tags in koji and changes to the bodhi
masher processes to make the security repo separately. this would have
the side effect that we could quickly push out important security
updates.

> > Point 3 means that the metadata size does not explode, and CLI tools
> > like yum don't spend minutes depsolving a much larger set of
> > packages. Although this increases the amount of space required on
> > the mirrors (by about 15% for fedora-19 by my approximation), the
> > amount of bandwidth saved is huge. By my calculations, over the
> > last 7 weeks [with ~10 offline updates, and hundreds of 'yum'
> > commands] over 60% of my traffic from the mirrors is metadata!
> > 
> > FWIW; 1,2,3 is what Debian and Ubuntu do. Comments welcome, thanks.
> > 
> > Richard
> 
> with the schedules as they have been I really dont know when Id get
> any time to work on the tooling for composing. certainly not before
> Fedora 21 likely later than that.
> 
> Dennis
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
> 
> iEYEARECAAYFAlINry4ACgkQkSxm47BaWffHQACfQizrrtfRi+qX4+wBK6lQyUQh
> jMoAniirXFSRfkcgc63o0TIzISSBrtQI
> =s6rn
> -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlIPlLAACgkQkSxm47BaWfcL5ACglmuk1eD7/VWD+9k7xqMR1OY2
ZagAoL6Qgp+TO8DkPCUJOoJcEhi6Y0GC
=SUJG
-----END PGP SIGNATURE-----