Keeping old versions of packages

[Richard Hughes]

Using PackageKit and yum on the command line is often painful as we
have to always download metadata unless it's less than a few hours
old. Being able to update the metadata once a week would be awesome
(with the possible exception of security updates) so that we could
schedule the 20Mb+ metadata update when the user is idle rather than
waiting for updates.

I'm wondering what the interest would be in keeping packages
referenced in metadata on the mirrors for say a month? We'd probably
need a separate fedora-security repo too that's designed to be kept
small enough so that metadata checks every day would be not costly in
terms of bandwidth and time.

If anyone is interested in doing this, you'd be awesome. Thanks.

Richard

[Reindl Harald]

Am 09.04.2013 11:10, schrieb Richard Hughes:
>  Using PackageKit and yum on the command line is often painful as we
>  have to always download metadata unless it's less than a few hours
>  old. Being able to update the metadata once a week would be awesome
>  (with the possible exception of security updates) so that we could
>  schedule the 20Mb+ metadata update when the user is idle rather than
>  waiting for updates 
[root@rh:~]$ cat /etc/yum.repos.d/fedora.repo
[fedora]
name=Fedora $releasever - $basearch
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releas...
enabled=1
metadata_expire=7d

>  I'm wondering what the interest would be in keeping packages
>  referenced in metadata on the mirrors for say a month? We'd probably
>  need a separate fedora-security repo too that's designed to be kept
>  small enough so that metadata checks every day would be not costly in
>  terms of bandwidth and time 
this does not work for many reasons and having more repos would
make things worser with low bandwidth because you have to load
more metadata at all and if i want search for updates i use
"yum clean metadata && yum upgrade" which is a real pain on
slow bandwith

the reason is that the metadata getting larger and larger because
all of the shiny ideas what additional ones would be nice while
the developers are on LAN or at least very fast LAN networks

[Richard Hughes]

On 9 April 2013 10:21, Reindl Harald <h.reindl(a)thelounge.net&gt; wrote:
>  metadata_expire=7d 
> From a package manager point of view, this happens: 
1 check expire timeout, all okay
2 depsolve update set
3 download
4 package not found!
5 download needed metadata based on some heuristic
6 goto 2

Richard

[Nico Kadel-Garcia]

7 days is way, way, way too long for an active repository such as any of
the currently supported Fedoras. And yes, it *is* burdensome to download
and keep updating all the time.

If running more than a few servers, I always maintain a local nightly
mirror and point my clients to that by default. The metadata download is
still burdensome, but the traffic is local and much faster. And I
completely exclude "drpms" from the mirror, there's just no point to it if
you've using a local mirror.


On Tue, Apr 9, 2013 at 5:21 AM, Reindl Harald <h.reindl(a)thelounge.net&gt;wrote:

>  Am 09.04.2013 11:10, schrieb Richard Hughes:
>  > Using PackageKit and yum on the command line is often painful as we
>  > have to always download metadata unless it's less than a few hours
>  > old. Being able to update the metadata once a week would be awesome
>  > (with the possible exception of security updates) so that we could
>  > schedule the 20Mb+ metadata update when the user is idle rather than
>  > waiting for updates
>
>  [root@rh:~]$ cat /etc/yum.repos.d/fedora.repo
>  [fedora]
>  name=Fedora $releasever - $basearch
>  failovermethod=priority
>  #baseurl=
>
> http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/E...
>
>
> mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releas...
>  enabled=1
>  metadata_expire=7d
>
>  > I'm wondering what the interest would be in keeping packages
>  > referenced in metadata on the mirrors for say a month? We'd probably
>  > need a separate fedora-security repo too that's designed to be kept
>  > small enough so that metadata checks every day would be not costly in
>  > terms of bandwidth and time
>
>  this does not work for many reasons and having more repos would
>  make things worser with low bandwidth because you have to load
>  more metadata at all and if i want search for updates i use
>  "yum clean metadata && yum upgrade" which is a real pain on
>  slow bandwith
>
>  the reason is that the metadata getting larger and larger because
>  all of the shiny ideas what additional ones would be nice while
>  the developers are on LAN or at least very fast LAN networks
>
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>

[Matthew Miller]

On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote:
>  I'm wondering what the interest would be in keeping packages
>  referenced in metadata on the mirrors for say a month? We'd probably
>  need a separate fedora-security repo too that's designed to be kept
>  small enough so that metadata checks every day would be not costly in
>  terms of bandwidth and time.
>  If anyone is interested in doing this, you'd be awesome. Thanks. 
I've heard of a plan in development about batching non-critical updates into
monthly sets. It seems like these two things could go together.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Reindl Harald]

Am 09.04.2013 14:27, schrieb Matthew Miller:
>  On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote:
> > I'm wondering what the interest would be in keeping packages
> > referenced in metadata on the mirrors for say a month? We'd probably
> > need a separate fedora-security repo too that's designed to be kept
> > small enough so that metadata checks every day would be not costly in
> > terms of bandwidth and time.
> > If anyone is interested in doing this, you'd be awesome. Thanks.
>  
>  I've heard of a plan in development about batching non-critical updates into
>  monthly sets. It seems like these two things could go together 
a terrible idea for a distribution with a new release all 6 months
if i want monthly pacthdays i use Microsoft or Oracle

you can hardly classify which bug is for which user critical!

depends often on the workload and currect jobs and the cirtical
apllication wheer a bug will hurt you much may change from project
to project

there are months where i would not care if LibreOffice does not
start at all and there are weeks where i need it all day long

[Matthew Miller]

On Tue, Apr 09, 2013 at 02:48:40PM +0200, Reindl Harald wrote:
>  depends often on the workload and currect jobs and the cirtical
>  apllication wheer a bug will hurt you much may change from project
>  to project 
As I understand it, you'll be able to opt for an all-updates track. In fact,
that will be encouraged for active developers and testers.


>  there are months where i would not care if LibreOffice does not
>  start at all and there are weeks where i need it all day long 
So, presumably, you don't want it gratuitously updating during those times
you need it all day long, and you don't care either way the rest of the
time. Sounds perfect.



-- 
Matthew Miller           mattdm(a)mattdm.org          <http://mattdm.org/>

[Richard Hughes]

On 9 April 2013 13:48, Reindl Harald <h.reindl(a)thelounge.net&gt; wrote:
>  if i want monthly pacthdays i use Microsoft or Oracle 
Not at all. Patchdays make perfect sense for planning
reboots/downtime/maintenance and that kind of thing.

>  you can hardly classify which bug is for which user critical! 
Sure you can, Red Hat does it for updates in RHEL, and you just have
to define what the terms mean. 90% of the updates I'm looking at right
now for F18 are really not important at all. Spec file fixups, new
versions without bugfixes, updated artwork; that can all wait until a
certain point in the month.

Richard.

[Lars Seipel]

On Tuesday 09 April 2013 16:02:14 Richard Hughes wrote:
>  now for F18 are really not important at all. Spec file fixups, new
>  versions without bugfixes, updated artwork; that can all wait until a
>  certain point in the month. 
Security-critical updates are already tagged as such, aren't they? So users 
are already able to defer installing non-critical updates to a point in time 
where it's convenient for them. There's even a yum plugin for that.

Maybe the existing tools could be enhanced but changing stuff on the repo side? 
I'm not so sure...

Lars

[Reindl Harald]

Am 09.04.2013 17:02, schrieb Richard Hughes:
>  On 9 April 2013 13:48, Reindl Harald <h.reindl(a)thelounge.net&gt;
> wrote:
> > if i want monthly pacthdays i use Microsoft or Oracle
>  
>  Not at all. Patchdays make perfect sense for planning
>  reboots/downtime/maintenance and that kind of thing. 
there where i need this test-machines and internal repos exists
but i do NOT need anybody to hold back updates for me

> > you can hardly classify which bug is for which user critical!
>  
>  Sure you can, Red Hat does it for updates in RHEL, and you just have
>  to define what the terms mean. 90% of the updates I'm looking at right
>  now for F18 are really not important at all. Spec file fixups, new
>  versions without bugfixes, updated artwork; that can all wait until a
>  certain point in the month 
but Fedora IS NOT RHEL
if you want the RHEL way use it

[Kevin Kofler]

Reindl Harald wrote:
>  there where i need this test-machines and internal repos exists
>  but i do NOT need anybody to hold back updates for me 
Agreed 100%, this is really silly. If you don't want updates, just don't 
update!

>  but Fedora IS NOT RHEL
>  if you want the RHEL way use it 
+1

That horrible idea has been floated at DevConf Brno and I already complained 
there about it (sadly not in the session because question time was too 
short, but only in small-group discussions hours later). I really hope 
reason will prevail!

        Kevin Kofler

[Dan Mashal]

On Wed, Apr 10, 2013 at 5:24 PM, Reindl Harald <h.reindl(a)thelounge.net&gt; wrote:
>  but Fedora IS NOT RHEL
>  if you want the RHEL way use it
>  
WHO ARE YOU KIDDING?

[Reindl Harald]

Am 14.04.2013 12:51, schrieb Dan Mashal:
>  On Wed, Apr 10, 2013 at 5:24 PM, Reindl Harald
> <h.reindl(a)thelounge.net&gt; wrote:
> > but Fedora IS NOT RHEL
> > if you want the RHEL way use it
> >
>  
>  WHO ARE YOU KIDDING? 
??

if someone aks for a monthly patchday for Fedora
instead push updates after bugs are fixed who is
kidding there?

who the hell needs this?
if you do not want updates - FINE do not install them
why do someone need handholding by such simple decisions?

[Kevin Fenzi]

On Sun, 14 Apr 2013 12:59:06 +0200
Reindl Harald <h.reindl(a)thelounge.net&gt; wrote:

>  if someone aks for a monthly patchday for Fedora
>  instead push updates after bugs are fixed who is
>  kidding there? 
I don't think anyone actually was asking for that. The proposal around
monthly update packs includes the idea that people who want to use the
current flow of updates process can continue to do so just fine. It's
an additional setup for people who do not want a constant flow of
updates. 

In any case it's not formally been proposed yet that I know of, so it's
kind of silly to speculate about a plan the full details of which
aren't available. 

kevin

[Hans de Goede]

Hi,

Am 09.04.2013 14:27, schrieb Matthew Miller:
>  On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote:
> > I'm wondering what the interest would be in keeping packages
> > referenced in metadata on the mirrors for say a month? We'd probably
> > need a separate fedora-security repo too that's designed to be kept
> > small enough so that metadata checks every day would be not costly in
> > terms of bandwidth and time.
> > If anyone is interested in doing this, you'd be awesome. Thanks.
>
>  I've heard of a plan in development about batching non-critical updates into
>  monthly sets. It seems like these two things could go together 
I'm sorry, but that is a very bad idea. When users report bugs, and I mean
real bugs here, like crashes or non working functionality. I always do
my best to get them a fixed package asap, and AFAIK they really appreciate
this.

Moreover this is just a very non Fedora thing to do, one of the things
Fedora is, is about being First. A lot of out users expect us to quickly give
them new packages after upstream bug-fix releases. Lumping these all together
in a single day in the month just does not feel like a Fedora thing to do.

Also many packages in Fedora are maintained by volunteers lumping all the
updates together will mean a flag day where all of the packages maintained
by someone will get pushed at once, leading to a peak in work load, since
despite testing, etc. There will be regressions as well as new packages
sometimes leading to questions. And there also will be a peak workload
a few days before the flag day to try and get things in now, instead
of needing to wait a month.  Having such peak workloads is not a good
idea in general, and esp. not with volunteers.

Regards,

Hans

[Richard Hughes]

On 9 April 2013 16:16, Hans de Goede <hdegoede(a)redhat.com&gt; wrote:
>  them new packages after upstream bug-fix releases. Lumping these all
>  together in a single day in the month just does not feel like a Fedora thing to do.
You can't QA a trickle. If packages are small and self-contained then
sure, it might work, but applications depending on ~30 other libraries
and ~20 other daemons not so much.

Richard.

[Kevin Kofler]

Richard Hughes wrote:
>  You can't QA a trickle. 
Sure you can! It's a basic rule of QA that small isolated changes can be 
debugged much better than a huge hodgepodge of many totally unrelated 
changes.

>  If packages are small and self-contained then sure, it might work,
> but
>  applications depending on ~30 other libraries and ~20 other daemons not so
>  much. 
The packages are sometimes not (small and self-contained), but the changes 
usually are!

        Kevin Kofler

[Richard Hughes]

On 13 April 2013 23:09, Kevin Kofler <kevin.kofler(a)chello.at&gt; wrote:
>  Sure you can! It's a basic rule of QA that small isolated changes
> can be
>  debugged much better than a huge hodgepodge of many totally unrelated
>  changes. 
Not true for the majority of GNOME and KDE packages. You can't test
gnome-control-center 3.9.1 without installing gnome-settings-daemon
3.9.1 as well. Not because of any library interface issue, but because
g-s-d provides the D-Bus API used by g-c-c. The desktop, like it or
not, is getting more monolithic, not less. FWIW, that's a totally good
thing.

Richard.

[Kevin Kofler]

Richard Hughes wrote:
>  Not true for the majority of GNOME and KDE packages. You can't
> test
>  gnome-control-center 3.9.1 without installing gnome-settings-daemon
>  3.9.1 as well. Not because of any library interface issue, but because
>  g-s-d provides the D-Bus API used by g-c-c. The desktop, like it or
>  not, is getting more monolithic, not less. FWIW, that's a totally good
>  thing. 
Speaking of KDE because that's what I'm familiar with:

* The KDE version upgrades are already being pushed as groups. All the 
4.10.2 packages are in a single update in Bodhi. That makes a lot of sense 
because upstream releases them together at the same time. So doing monthly 
batching in Fedora will only mean we'll be out of sync with upstream, it 
would not provide any grouping that is not already being done.

* In KDE land, you can actually use old kde* on new kdelibs (at least you 
should be able to; it's not really tested or supported for the core KDE 
packages because they are updated at the same time as kdelibs and thus 
expected to be used together), but not new kde* on old kdelibs. So it would 
be possible to test the updated packages separately in reverse dependency 
order. It's just neither practical (time-wise) nor necessary. The monthly 
grouping is already done by upstream there.

* KDE is getting less monolithic, not more. KDE Frameworks 5 will even be 
taking the splitting to extreme levels.


Basically, where upstream releases a large set of version updates 
simultaneously (e.g. KDE coordinated releases), we are already pushing them 
as one group and downstream batching would bring no benefits whatsoever, 
whereas for small leaf packages (e.g. colord-kde), the updates can easily be 
tested in isolation (even much better than when mixed with a huge batch of 
mostly unrelated changes; if, say, a batch of colord + kdelibs + colord-kde 
breaks colord-kde, how do we know which updated package broke it?).

        Kevin Kofler

[Miloslav Trmač]

On Sun, Apr 14, 2013 at 12:31 PM, Richard Hughes <hughsient(a)gmail.com&gt;wrote:

>  On 13 April 2013 23:09, Kevin Kofler <kevin.kofler(a)chello.at&gt;
> wrote:
>  > Sure you can! It's a basic rule of QA that small isolated changes can be
>  > debugged much better than a huge hodgepodge of many totally unrelated
>  > changes.
>
>  Not true for the majority of GNOME and KDE packages. You can't test
>  gnome-control-center 3.9.1 without installing gnome-settings-daemon
>  3.9.1 as well.
>  
A desktop environment should be a comparatively small subset of the overall
universe of packages (in particular, when compared to the applications
written for that desktop environment), so optimizing the distribution
update system for desktop environment updates doesn't make that much
sense.  Desktop environment updates can be published as a group with the
current model already.
    Mirek

[Matthew Miller]

On Tue, Apr 09, 2013 at 05:16:45PM +0200, Hans de Goede wrote:
>  >I've heard of a plan in development about batching
> non-critical updates into
>  >monthly sets. It seems like these two things could go together
>  I'm sorry, but that is a very bad idea. When users report bugs, and I mean
>  real bugs here, like crashes or non working functionality. I always do
>  my best to get them a fixed package asap, and AFAIK they really appreciate
>  this. 
To be clear, the plan I heard (which isn't mine and I don't think is
finished anyway) isn't to *withhold* updates until a certain date; it's to
batch them up and make them available as a collection by default. If want
all *or some* updates as soon as they become available, you could still do
that.


>  Also many packages in Fedora are maintained by volunteers lumping all
> the
>  updates together will mean a flag day where all of the packages maintained
>  by someone will get pushed at once, leading to a peak in work load, since
>  despite testing, etc. There will be regressions as well as new packages
>  sometimes leading to questions. And there also will be a peak workload
>  a few days before the flag day to try and get things in now, instead
>  of needing to wait a month.  Having such peak workloads is not a good
>  idea in general, and esp. not with volunteers. 
Overall, it's a more predictable workload, which *is* a good idea, for both
volunteer and otherwise. It's also more effective to QA packages in sets,
and more effective can mean more efficient.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Mathieu Bridon]

On Wednesday, April 10, 2013 12:09 AM, Matthew Miller wrote:
>  On Tue, Apr 09, 2013 at 05:16:45PM +0200, Hans de Goede wrote:
> >> I've heard of a plan in development about batching non-critical updates into
> >> monthly sets. It seems like these two things could go together
> > I'm sorry, but that is a very bad idea. When users report bugs, and I mean
> > real bugs here, like crashes or non working functionality. I always do
> > my best to get them a fixed package asap, and AFAIK they really appreciate
> > this.
>
>  To be clear, the plan I heard (which isn't mine and I don't think is
>  finished anyway) isn't to *withhold* updates until a certain date; it's to
>  batch them up and make them available as a collection by default. If want
>  all *or some* updates as soon as they become available, you could still do
>  that. 
For what it's worth, this is exactly what we do at $dayjob.

The way I set our repos up is that there are 1 testing repo, and 2 
stable repos: "current" and "released".

We use Bodhi to move things from "testing" to "current", as they get 
QA-ed, just like in Fedora.

But by default, neither the "current" nor "testing" repositories are 
enabled. Only the "released" repository is.

Once a month, we synchronize "current" into "released".

This way, we have the monthly « Patch Tuesday » by default, and it's 
trivial to move to the model of getting updates as they are published if 
you want to.

It also allows users to be selective: they can get an update without 
waiting if it's important to them, but without updating everything else 
right now.


-- 
Mathieu

[Kevin Kofler]

Matthew Miller wrote:
>  Overall, it's a more predictable workload, which *is* a good
> idea, for
>  both volunteer and otherwise. 
No, sorry, but as volunteers, we have other commitments which mean we cannot 
always do our Fedora work when some central Fedora schedule dictates it. The 
mad rushes at release time are already bad enough, adding ones for update 
batches every month is VERY unhelpful. The schedule will inevitably conflict 
with real life. The current flexible approach is much better suited for 
volunteer developers, allowing us to schedule our work on updates when WE 
have time.

And for what is worth, the current approach is also more flexible for our 
users, for similar reasons. Not to mention that there are users like me who 
WANT as frequent updates as possible. (I'd personally like instant or hourly 
pushes, or at least consistent daily pushes (independently of whether it's a 
business day etc.).) But even those users who want to update only once a 
month will want to do it on THEIR schedule, not on the Fedora project's.

>  It's also more effective to QA packages in sets, and more
> effective can
>  mean more efficient. 
How is it more effective to QA a huge batch of totally unrelated changes 
which may or may not have interactions as opposed to QAing small isolated 
changes individually? This just does not make any sense whatsoever.

        Kevin Kofler

[Simo Sorce]

On Tue, 2013-04-09 at 17:16 +0200, Hans de Goede wrote:
>  Hi,
>  
>  Am 09.04.2013 14:27, schrieb Matthew Miller:
>  > On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote:
>  >> I'm wondering what the interest would be in keeping packages
>  >> referenced in metadata on the mirrors for say a month? We'd probably
>  >> need a separate fedora-security repo too that's designed to be kept
>  >> small enough so that metadata checks every day would be not costly in
>  >> terms of bandwidth and time.
>  >> If anyone is interested in doing this, you'd be awesome. Thanks.
>  >
>  > I've heard of a plan in development about batching non-critical updates into
>  > monthly sets. It seems like these two things could go together
>  
>  I'm sorry, but that is a very bad idea. When users report bugs, and I mean
>  real bugs here, like crashes or non working functionality. I always do
>  my best to get them a fixed package asap, and AFAIK they really appreciate
>  this.
>  
>  Moreover this is just a very non Fedora thing to do, one of the things
>  Fedora is, is about being First. A lot of out users expect us to quickly give
>  them new packages after upstream bug-fix releases. Lumping these all together
>  in a single day in the month just does not feel like a Fedora thing to do.
>  
>  Also many packages in Fedora are maintained by volunteers lumping all the
>  updates together will mean a flag day where all of the packages maintained
>  by someone will get pushed at once, leading to a peak in work load, since
>  despite testing, etc. There will be regressions as well as new packages
>  sometimes leading to questions. And there also will be a peak workload
>  a few days before the flag day to try and get things in now, instead
>  of needing to wait a month.  Having such peak workloads is not a good
>  idea in general, and esp. not with volunteers. 
Can't they get them from updates-testing if they need a fix "right
now" ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Vít Ondruch]

Dne 9.4.2013 18:14, Simo Sorce napsal(a):
>  On Tue, 2013-04-09 at 17:16 +0200, Hans de Goede wrote:
> > Hi,
> >
> > Am 09.04.2013 14:27, schrieb Matthew Miller:
> >> On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote:
> >>> I'm wondering what the interest would be in keeping packages
> >>> referenced in metadata on the mirrors for say a month? We'd probably
> >>> need a separate fedora-security repo too that's designed to be kept
> >>> small enough so that metadata checks every day would be not costly in
> >>> terms of bandwidth and time.
> >>> If anyone is interested in doing this, you'd be awesome. Thanks.
> >> I've heard of a plan in development about batching non-critical updates into
> >> monthly sets. It seems like these two things could go together
> > I'm sorry, but that is a very bad idea. When users report bugs, and I mean
> > real bugs here, like crashes or non working functionality. I always do
> > my best to get them a fixed package asap, and AFAIK they really appreciate
> > this.
> >
> > Moreover this is just a very non Fedora thing to do, one of the things
> > Fedora is, is about being First. A lot of out users expect us to quickly give
> > them new packages after upstream bug-fix releases. Lumping these all together
> > in a single day in the month just does not feel like a Fedora thing to do.
> >
> > Also many packages in Fedora are maintained by volunteers lumping all the
> > updates together will mean a flag day where all of the packages maintained
> > by someone will get pushed at once, leading to a peak in work load, since
> > despite testing, etc. There will be regressions as well as new packages
> > sometimes leading to questions. And there also will be a peak workload
> > a few days before the flag day to try and get things in now, instead
> > of needing to wait a month.  Having such peak workloads is not a good
> > idea in general, and esp. not with volunteers.
>  Can't they get them from updates-testing if they need a fix "right
>  now" ?
>
>  Simo.
>  
Actually, if update gets enough karma, it could be released immediately. 
The karma signals that somebody cares to give a karma. If there is no 
karma, it can be release in one month batch, since it is probably just 
minor, non-interesting fix.

Vít

[Bill Nottingham]

Richard Hughes (hughsient(a)gmail.com) said: 
>  Using PackageKit and yum on the command line is often painful as we
>  have to always download metadata unless it's less than a few hours
>  old. Being able to update the metadata once a week would be awesome
>  (with the possible exception of security updates) so that we could
>  schedule the 20Mb+ metadata update when the user is idle rather than
>  waiting for updates.
>  
>  I'm wondering what the interest would be in keeping packages
>  referenced in metadata on the mirrors for say a month? We'd probably
>  need a separate fedora-security repo too that's designed to be kept
>  small enough so that metadata checks every day would be not costly in
>  terms of bandwidth and time.
>  
>  If anyone is interested in doing this, you'd be awesome. Thanks. 
The thing that creates the repos is 'mash', so that's where such a change
needs to be. It currently talks to koji to get the builds for a particular
tag, and has the following config settings:

 latest = True

Grab only the latest package

 latest = False

Grab *all* the versions of the packages.

It only supports these two variations, because that's what koji supports.

If you wanted to keep more versions on the mirrors, you have the following
options:

1) Have mash create everything, and then run a script that prunes versions
older than X, and re-runs createrepo.

We used to do this in the old days. It's an utter hack, and not very
efficient.

2) Have mash try and implement a date-based expiry itself from what it
requests from koji.

The problem here is that mash pulls from koji, and only has two times
that it could use as a key:

- build time (not really what you want, as packages can build and sit
  for a while)

- time it was tagged into the repo (which is affected by rel-eng operations
  if packages need to be retagged or moved)

3) Have mash sort through the packages retrieved by koji, and pull some
subset less than 'all' (the last 2/3/4 versions).

This is probably doable - it requires requesting all tagged packages from
koji and then doing some postprocessing on the list before you download
them. It's merely a matter of code.

So, needless to say, I'd suggest anyone interested in this to look at option
#3. Note that enabling something like that on rawhide would have a large
effect on the repository creation time - there's only so many ways to speed
up scanning across 50000 packages & 100GB on a SAN.

Bill (former mash maintainer)

[Mathieu Bridon]

On Tuesday, April 09, 2013 11:52 PM, Bill Nottingham wrote:
>  If you wanted to keep more versions on the mirrors, you have the
> following
>  options:
>
>  1) Have mash create everything, and then run a script that prunes versions
>  older than X, and re-runs createrepo. 
[... snip ...]
>  2) Have mash try and implement a date-based expiry itself from what
> it
>  requests from koji. 
[... snip ...]
>  3) Have mash sort through the packages retrieved by koji, and pull
> some
>  subset less than 'all' (the last 2/3/4 versions).
>
>  This is probably doable - it requires requesting all tagged packages from
>  koji and then doing some postprocessing on the list before you download
>  them. It's merely a matter of code. 
It seems to me that all of these solutions focus on making mash do more 
stuff, without ever changing Koji.

But couldn't the Koji API grow a new parameter, which would be used for 
specifying how many versions of each package to get at most?

The current behaviour would be obtained by setting it to 1, and setting 
it to 2 would already be a positive change as it would allow downgrading 
a package if the update went wrong.


-- 
Mathieu

[Bruno Wolff III]

On Wed, Apr 10, 2013 at 00:05:45 +0800,
   Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
> The current behaviour would be obtained by setting it to 1, and 
> setting it to 2 would already be a positive change as it would allow 
> downgrading a package if the update went wrong. 
I don't think that is really what you want either. The idea is to keep 
recently obsoleted updates around, not 2 or 3 versions of everything.

The change has some other benefits. Reverting bad updates in rawhide 
would be easier. You can use yum downgrade instead of having to going 
look at koji and download builds. Dealing with packages dropping out 
of repos when moving between test and updates. The latter issue is 
especially bad with branched during freezes.

[Seth Vidal]

On Tue, 9 Apr 2013 11:18:54 -0500
Bruno Wolff III <bruno(a)wolff.to&gt; wrote:

>  On Wed, Apr 10, 2013 at 00:05:45 +0800,
>     Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
>  >The current behaviour would be obtained by setting it to 1, and 
>  >setting it to 2 would already be a positive change as it would allow 
>  >downgrading a package if the update went wrong.
>  
>  I don't think that is really what you want either. The idea is to
>  keep recently obsoleted updates around, not 2 or 3 versions of
>  everything.
>  
>  The change has some other benefits. Reverting bad updates in rawhide 
>  would be easier. You can use yum downgrade instead of having to going 
>  look at koji and download builds. Dealing with packages dropping out 
>  of repos when moving between test and updates. The latter issue is 
>  especially bad with branched during freezes. 

So - this is just an idea - and not necessarily a good one - but what
about moving older pkgs which are not in the initial release repo into
an updates-archive repo.

We could leave the repo disabled by default and only keep 2 copies of
any single pkg name in the repo at a time.

That way in the best of all possible worlds you'd have at most 4 copies
of a pkg in total:
1 - in the base release 'everything' repo
1 - in updates
2 - in updates-archive


-sv

[Jan Zelený]

On 9. 4. 2013 at 12:25:56, seth vidal wrote:
>  On Tue, 9 Apr 2013 11:18:54 -0500
>  
>  Bruno Wolff III <bruno(a)wolff.to&gt; wrote:
>  > On Wed, Apr 10, 2013 at 00:05:45 +0800,
>  > 
>  >    Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
>  > >The current behaviour would be obtained by setting it to 1, and
>  > >setting it to 2 would already be a positive change as it would allow
>  > >downgrading a package if the update went wrong.
>  > 
>  > I don't think that is really what you want either. The idea is to
>  > keep recently obsoleted updates around, not 2 or 3 versions of
>  > everything.
>  > 
>  > The change has some other benefits. Reverting bad updates in rawhide
>  > would be easier. You can use yum downgrade instead of having to going
>  > look at koji and download builds. Dealing with packages dropping out
>  > of repos when moving between test and updates. The latter issue is
>  > especially bad with branched during freezes.
>  
>  So - this is just an idea - and not necessarily a good one - but what
>  about moving older pkgs which are not in the initial release repo into
>  an updates-archive repo.
>  
>  We could leave the repo disabled by default and only keep 2 copies of
>  any single pkg name in the repo at a time.
>  
>  That way in the best of all possible worlds you'd have at most 4 copies
>  of a pkg in total:
>  1 - in the base release 'everything' repo
>  1 - in updates
>  2 - in updates-archive 
I'm not sure this solves the initial problem - downloading new metadata every 
6 hours or so ...

[Bruno Wolff III]

On Wed, Apr 10, 2013 at 09:24:49 +0200,
   Jan Zelený <jzeleny(a)redhat.com&gt; wrote:
> I'm not sure this solves the initial problem - downloading new metadata every
> 6 hours or so ... 
Does the metadata really need to be downloaded or just checked to see if 
it is current?

[Reindl Harald]

Am 10.04.2013 15:47, schrieb Bruno Wolff III:
>  On Wed, Apr 10, 2013 at 09:24:49 +0200,
>    Jan Zelený <jzeleny(a)redhat.com&gt; wrote:
> >
> > I'm not sure this solves the initial problem - downloading new metadata every
> > 6 hours or so ...
>  
>  Does the metadata really need to be downloaded or just checked to see if it is current?
normally not if proper implemented in the client to
check last creation time

FTP:  no problem to check timestamp
HTTP: HEAD-request and E-Tags

[Seth Vidal]

On Wed, 10 Apr 2013 08:47:38 -0500
Bruno Wolff III <bruno(a)wolff.to&gt; wrote:

>  On Wed, Apr 10, 2013 at 09:24:49 +0200,
>     Jan Zelený <jzeleny(a)redhat.com&gt; wrote:
>  >
>  >I'm not sure this solves the initial problem - downloading new
>  >metadata every 6 hours or so ...
>  
>  Does the metadata really need to be downloaded or just checked to see
>  if it is current? 
The metadata doesn't get downloaded if it hasn't changed - the problem
though is that the metadata DOES change often. Normally everyday.

-sv

>  From: seth vidal <skvidal(a)fedoraproject.org&gt;
>  
>  The metadata doesn't get downloaded if it hasn't changed - the problem
>  though is that the metadata DOES change often. Normally everyday. 
Is there anything that could be done to make it unnecessary to pull the 
complete metadata for every update?  For example, IIRC this is all sqlite 
data, but what if this was in a plain-text data dump form where something 
like rsync could be used to efficiently transfer only those bits that have 
changed.  Client CPU time to reconstruct the DB is probably cheaper than 
the bandwidth.  Maybe such a mode would only be used if the DB size 
exceeded some threshold.

--
John Florian

[Seth Vidal]

On Wed, 10 Apr 2013 10:53:25 -0400
John.Florian(a)dart.biz wrote:

>  > From: seth vidal <skvidal(a)fedoraproject.org&gt;
>  > 
>  > The metadata doesn't get downloaded if it hasn't changed - the
>  > problem though is that the metadata DOES change often. Normally
>  > everyday.
>  
>  Is there anything that could be done to make it unnecessary to pull
>  the complete metadata for every update?  For example, IIRC this is
>  all sqlite data, but what if this was in a plain-text data dump form
>  where something like rsync could be used to efficiently transfer only
>  those bits that have changed.  Client CPU time to reconstruct the DB
>  is probably cheaper than the bandwidth.  Maybe such a mode would only
>  be used if the DB size exceeded some threshold.
>   
You'll have to talk to those folks who are trying to work on a delta
metadata format.

And rsync is not efficient from a bunch of clients to a mirror server.
It would cripple a mirror server in a moment.

-sv

[Jan Zelený]

On 10. 4. 2013 at 10:53:25, John.Florian(a)dart.biz wrote:
>  > From: seth vidal <skvidal(a)fedoraproject.org&gt;
>  > 
>  > The metadata doesn't get downloaded if it hasn't changed - the problem
>  > though is that the metadata DOES change often. Normally everyday.
>  
>  Is there anything that could be done to make it unnecessary to pull the
>  complete metadata for every update?  For example, IIRC this is all sqlite
>  data, but what if this was in a plain-text data dump form where something
>  like rsync could be used to efficiently transfer only those bits that have
>  changed.  Client CPU time to reconstruct the DB is probably cheaper than
>  the bandwidth.  Maybe such a mode would only be used if the DB size
>  exceeded some threshold. 
Adding Zdenek to CC, as he is already working on a proposal to reduce metadata 
download size.

Thanks
Jan

[Chris Adams]

Once upon a time, John.Florian(a)dart.biz <John.Florian(a)dart.biz&gt; said:
>  Is there anything that could be done to make it unnecessary to pull
> the 
>  complete metadata for every update?  For example, IIRC this is all sqlite 
>  data, but what if this was in a plain-text data dump form where something 
>  like rsync could be used to efficiently transfer only those bits that have 
>  changed.  Client CPU time to reconstruct the DB is probably cheaper than 
>  the bandwidth.  Maybe such a mode would only be used if the DB size 
>  exceeded some threshold. 
The metadata starts in XML before being loaded into an SQLite DB file,
and the XML is in the repodata directory with the DB.  However, both are
compressed, as they are large.  For example, the current
updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is
41M (9M bzip2 compressed).  I'm guessing there are historical reasons
why different compression is used; both could be made noticeably smaller
with xz (XML to just over 3M, DB to 7M), but that's still a lot of data
to download (and there are also other metadata files that have to be
downloaded sometimes, especially the filelists.xml.gz, which is 10M gzip
compressed).

I'm not sure when the XML is downloaded instead of (or in addition to)
the DB, but it does appear to happen (I see one example in my mirror
server web logs this morning for example).

All the metadata changes with every push (usually once per day for the
updates repo), so it has to be downloaded constantly.  Your only
practical choice for distribution is HTTP; rsync has much higher server
overhead and only available on some mirrors.  If you want anything other
than the full download, it'll have to be in the form of additional
repodata files generated as part of the push.

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

>  From: Chris Adams <cmadams(a)hiwaay.net&gt;
>  The metadata starts in XML before being loaded into an SQLite DB file,
>  and the XML is in the repodata directory with the DB.  However, both are
>  compressed, as they are large.  For example, the current
>  updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is
>  41M (9M bzip2 compressed).  I'm guessing there are historical reasons
>  why different compression is used; both could be made noticeably smaller
>  with xz (XML to just over 3M, DB to 7M), but that's still a lot of data
>  to download (and there are also other metadata files that have to be
>  downloaded sometimes, especially the filelists.xml.gz, which is 10M gzip
>  compressed). 
I was thinking there had been some xz integration recently.  Maybe that 
was with the delta rpm support.  I don't follow that though since we have 
a local mirror, there's not much point in rsycing, storing, etc. the 
deltas.

>  I'm not sure when the XML is downloaded instead of (or in
> addition to)
>  the DB, but it does appear to happen (I see one example in my mirror
>  server web logs this morning for example). 
I've wondered about that too.  I sure hope we didn't add the efficient 
method on top of the inefficient method, rather than replace it.

--
John Florian

[Kevin Fenzi]

On Wed, 10 Apr 2013 12:24:58 -0400
John.Florian(a)dart.biz wrote:

>  I was thinking there had been some xz integration recently.  Maybe
>  that was with the delta rpm support.  I don't follow that though
>  since we have a local mirror, there's not much point in rsycing,
>  storing, etc. the deltas. 
There's a rel-eng ticket to enable xz compressed repodata: 
https://fedorahosted.org/rel-eng/ticket/5362

Basically yum and friends should be able to handle it fine, but
createrepo doesn't default to it, so out repos would be different from
most everyone elses. This didn't happen for f19, but perhaps we could
enable it for rawhide... 

kevin

[Seth Vidal]

On Wed, 10 Apr 2013 12:24:58 -0400
John.Florian(a)dart.biz wrote:

>  I was thinking there had been some xz integration recently.  Maybe
>  that was with the delta rpm support.  I don't follow that though
>  since we have a local mirror, there's not much point in rsycing,
>  storing, etc. the deltas.
>   

yum and createrepo both support xz now - we can really only do it from
f18->f19 iirc - just ot make sure everyone has the required version of
yum and friends.


>  > I'm not sure when the XML is downloaded instead of (or in
> addition
>  > to) the DB, but it does appear to happen (I see one example in my
>  > mirror server web logs this morning for example).
>  
>  I've wondered about that too.  I sure hope we didn't add the
>  efficient method on top of the inefficient method, rather than
>  replace it. 

See my earlier email explaining how that works.

-sv

[Seth Vidal]

On Wed, 10 Apr 2013 10:33:43 -0500
Chris Adams <cmadams(a)hiwaay.net&gt; wrote:

>  The metadata starts in XML before being loaded into an SQLite DB
> file,
>  and the XML is in the repodata directory with the DB.  However, both
>  are compressed, as they are large.  For example, the current
>  updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is
>  41M (9M bzip2 compressed).  I'm guessing there are historical reasons
>  why different compression is used; both could be made noticeably
>  smaller with xz (XML to just over 3M, DB to 7M), but that's still a
>  lot of data to download (and there are also other metadata files that
>  have to be downloaded sometimes, especially the filelists.xml.gz,
>  which is 10M gzip compressed).
>  
>  I'm not sure when the XML is downloaded instead of (or in addition to)
>  the DB, but it does appear to happen (I see one example in my mirror
>  server web logs this morning for example).
>   

Here's how it works.

the xml metadata put together over a decade ago. It is the canonical
representation of the metadata.

The sqlite was added maybe 8ish years ago as a way of more quickly
reading the same data and not eating up so much memory. At the time
bzip2 was the new hotness so we used it instead of gz.

the primary, filelists and other xml should not ever be downloaded at
this point unless you hit a mirror which is out of sync, badly.

the only xml files that should be getting downloaded:
1. repomd.xml - it's fairly small and the index for everything else
2. comps.xml (or groups.xml) - which is where comps is stored per-repo
3. updatemd.xml which is just the security/update info for describing
updates



yum will grab repomd.xml and look to see if it is newer than what it
has already. Then go from there about updating the rest of the metadata.


Hope that helps explain it a bit more.

-sv

[Seth Vidal]

On Wed, 10 Apr 2013 09:24:49 +0200
Jan Zelený <jzeleny(a)redhat.com&gt; wrote:

>  On 9. 4. 2013 at 12:25:56, seth vidal wrote:
>  > On Tue, 9 Apr 2013 11:18:54 -0500
>  > 
>  > Bruno Wolff III <bruno(a)wolff.to&gt; wrote:
>  > > On Wed, Apr 10, 2013 at 00:05:45 +0800,
>  > > 
>  > >    Mathieu Bridon <bochecha(a)fedoraproject.org&gt; wrote:
>  > > >The current behaviour would be obtained by setting it to 1, and
>  > > >setting it to 2 would already be a positive change as it would
>  > > >allow downgrading a package if the update went wrong.
>  > > 
>  > > I don't think that is really what you want either. The idea is to
>  > > keep recently obsoleted updates around, not 2 or 3 versions of
>  > > everything.
>  > > 
>  > > The change has some other benefits. Reverting bad updates in
>  > > rawhide would be easier. You can use yum downgrade instead of
>  > > having to going look at koji and download builds. Dealing with
>  > > packages dropping out of repos when moving between test and
>  > > updates. The latter issue is especially bad with branched during
>  > > freezes.
>  > 
>  > So - this is just an idea - and not necessarily a good one - but
>  > what about moving older pkgs which are not in the initial release
>  > repo into an updates-archive repo.
>  > 
>  > We could leave the repo disabled by default and only keep 2 copies
>  > of any single pkg name in the repo at a time.
>  > 
>  > That way in the best of all possible worlds you'd have at most 4
>  > copies of a pkg in total:
>  > 1 - in the base release 'everything' repo
>  > 1 - in updates
>  > 2 - in updates-archive
>  
>  I'm not sure this solves the initial problem - downloading new
>  metadata every 6 hours or so ... 

I wasn't trying to solve that problem. The problem I did solve was the
updates repodata growing forever if we keep more than one version of
the pkgs in there.

-sv

[Bill Nottingham]

Mathieu Bridon (bochecha(a)fedoraproject.org) said: 
>  On Tuesday, April 09, 2013 11:52 PM, Bill Nottingham wrote:
>  >If you wanted to keep more versions on the mirrors, you have the following
>  >options:
>  >
>  >1) Have mash create everything, and then run a script that prunes versions
>  >older than X, and re-runs createrepo.
>  [... snip ...]
>  >2) Have mash try and implement a date-based expiry itself from what it
>  >requests from koji.
>  [... snip ...]
>  >3) Have mash sort through the packages retrieved by koji, and pull some
>  >subset less than 'all' (the last 2/3/4 versions).
>  >
>  >This is probably doable - it requires requesting all tagged packages from
>  >koji and then doing some postprocessing on the list before you download
>  >them. It's merely a matter of code.
>  
>  It seems to me that all of these solutions focus on making mash do
>  more stuff, without ever changing Koji. 
It's the (somewhat) more agile component in that it can be changed
with less downstream dependencies, fewer worries about deployment on
infrastructure, etc.

>  But couldn't the Koji API grow a new parameter, which would be
> used
>  for specifying how many versions of each package to get at most? 
It certainly could. However, Koji's concept of 'newer' is (IIRC) entirely
tied to build time/tag time in the repo, not NVR. So it may not be what
we want.

Bill

[Chris Adams]

Once upon a time, Bill Nottingham <notting(a)redhat.com&gt; said:
>  So, needless to say, I'd suggest anyone interested in this to
> look at option
>  #3. Note that enabling something like that on rawhide would have a large
>  effect on the repository creation time - there's only so many ways to speed
>  up scanning across 50000 packages & 100GB on a SAN. 
It would also have a significant impact on the mirror servers' disk
space (speaking for a mirror that is running low on disk and no $$ to
upgrade).
-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Kevin Kofler]

Richard Hughes wrote:
>  Using PackageKit and yum on the command line is often painful as we
>  have to always download metadata unless it's less than a few hours
>  old. Being able to update the metadata once a week would be awesome
>  (with the possible exception of security updates) so that we could
>  schedule the 20Mb+ metadata update when the user is idle rather than
>  waiting for updates. 
For me, only once a week is totally unacceptable. I have Apper check for 
updates hourly. If the metadata isn't actually going to be refreshed on 
those hourly runs, that'd really suck!

        Kevin Kofler

[Reindl Harald]

Am 14.04.2013 00:03, schrieb Kevin Kofler:
>  Richard Hughes wrote:
> > Using PackageKit and yum on the command line is often painful as we
> > have to always download metadata unless it's less than a few hours
> > old. Being able to update the metadata once a week would be awesome
> > (with the possible exception of security updates) so that we could
> > schedule the 20Mb+ metadata update when the user is idle rather than
> > waiting for updates.
>  
>  For me, only once a week is totally unacceptable. I have Apper check for 
>  updates hourly. If the metadata isn't actually going to be refreshed on 
>  those hourly runs, that'd really suck! 
OK, hourly is a little bit too much
but daily - YES!
and weekly - NO!