RFE: prevent broken rawhide network install due to repo change

[Andrew Farris]

RFE:  prevent broken rawhide network install due to repo change

Problem: network installs will currently fail if the repository changes during 
the installation, which can take 2-4 hours or more, and changed files are 
removed from the repo, making anaconda fail to download the expected version of 
that changed file

If a user has started a network install then any change in files not yet 
downloaded will completely break that install.  The user will see anaconda fail 
to get the file 10 times, then suggest rebooting and starting the install again. 
  This wastes time and network traffic, since the repository could have only 
changed the last file that installer needed... and they will subsequently 
download them all again (this time hopefully missing the period the repo changes).

To solve the problem all that is necessary is for the repo to hold all changed 
files for a period (probably 3-4 hours for the typical install time) so that any 
installs that have begun before the changes to the repo data can finish.  The 
depsolving in anaconda does not recognize a change in the repodata and attempt 
to resume the install with the new data, so either 1) that needs implemented, or 
2) the mirrors need to hold the files for awhile even though the repodata no 
longer has them included.

Would it be possible for the repository data to be created while excluding any 
files that were updated during that build, leaving the old files out of 
listings, but actually leave the file present and accessible by direct request?

This would improve the network install experience because it is difficult to 
anticipate when the repo changes will break the install process for \insert 
random mirror\.

The caveat to the solution is that the repository needs to be cleaned some time 
later by another process which would know which files are now old (not included 
in the repo data) and delete them.  In the meantime (again suggesting 3-4 hour 
overlap) there will be a slightly larger disk space needed because of the 
duplication.

Any mirrors that sync to rawhide while there are duplicated files should have 
those files already (from the day before) and would only fetch the new files. 
Those mirrors could either 1) keep the duplicate files all day until the next 
sync, or 2) have their own cleaning process such as a cron job.  In either case, 
anyone requesting an installation while the duplicates are there would not be 
effected.

I've run into this problem several times when doing network rawhide installs, 
especially if started late night in California (meaning the repo build and 
mirror syncs are likely to overlap the install).

I am unaware obviously of where the infrastructure would need to be changed to 
handle this, though I assume a combination of koji and elsewhere (both in 
building the repo data and afterward in deleting the temporarily remaining files).

If there is anything already in place to solve/work on this issue point me in 
the right direction, because its not working. ;)

Also I think this is only an issue with rawhide since the release mirrors keep 
the original released files and then updates go elsewhere.  Development on the 
other hand has issues with the changes because files disappear.

I'd be happy to move this suggestion to where it belongs (bugzilla against ?)

-- 
Andrew Farris <lordmorgul(a)gmail.com&gt; <ajfarris(a)gmail.com&gt;
  gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
----                                                                       ----

[Andrew Farris]

Andrew Farris wrote:
>  Any mirrors that sync to rawhide while there are duplicated files
> should 
>  have those files already (from the day before) and would only fetch the 
>  new files. Those mirrors could either 1) keep the duplicate files all 
>  day until the next sync, or 2) have their own cleaning process such as a 
>  cron job.  In either case, anyone requesting an installation while the 
>  duplicates are there would not be effected. 
To clarify 'cleaning process' I envisioned that koji would create a list of the 
files changed during that build, those that are then going to be removed, and 
include that in the repo top level directory.  A cron job on the mirrors could 
just read that 'files removed during build' list and remove them after that 
'overlap period'.

-- 
Andrew Farris <lordmorgul(a)gmail.com&gt; <ajfarris(a)gmail.com&gt;
  gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
----                                                                       ----

[Andrew Farris]

Andrew Farris wrote:
>  RFE:  prevent broken rawhide network install due to repo change
I just love replying to myself so I'll do it again.  I realized the fedora 
infrastructure trac is probably the place for this, so opened ticket:

https://fedorahosted.org/fedora-infrastructure/ticket/354

-- 
Andrew Farris <lordmorgul(a)gmail.com&gt; <ajfarris(a)gmail.com&gt;
  gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
----                                                                       ----

[Till Maas]

On Thu January 24 2008, Andrew Farris wrote:
>  RFE:  prevent broken rawhide network install due to repo change
>
>  Problem: network installs will currently fail if the repository changes
>  during the installation, which can take 2-4 hours or more, and changed
>  files are removed from the repo, making anaconda fail to download the
>  expected version of that changed file 
>  I'd be happy to move this suggestion to where it belongs
> (bugzilla against
>  ?) 
Imho this is something that can be improved in anacanda, e.g. make it use 
another mirror in case an rpm is missinga and in case an rpm is missing, it 
could update its metadata and recalculate which new rpms could be used to 
finish the installation, or it could first download every rpm and then 
install them. When anaconda supports adding repositories with updates, this 
problem could also occur for normal installs.

Regards,
Till

[Andrew Farris]

Till Maas wrote:
>  On Thu January 24 2008, Andrew Farris wrote:
> > RFE:  prevent broken rawhide network install due to repo change
> >
> > Problem: network installs will currently fail if the repository changes
> > during the installation, which can take 2-4 hours or more, and changed
> > files are removed from the repo, making anaconda fail to download the
> > expected version of that changed file
>  
> > I'd be happy to move this suggestion to where it belongs (bugzilla against
> > ?)
>  
>  Imho this is something that can be improved in anacanda, e.g. make it use 
>  another mirror in case an rpm is missinga and in case an rpm is missing, it 
>  could update its metadata and recalculate which new rpms could be used to 
>  finish the installation, or it could first download every rpm and then 
>  install them. When anaconda supports adding repositories with updates, this 
>  problem could also occur for normal installs.
>  
>  Regards,
>  Till
>   
I think the problem really needs to be fixed BOTH in the mirror system and in 
anaconda.  If someone were to attempt to get a single snapshot of a 
self-consistent repo, and happened to rsync during the mirror updating his repo 
would be inconsistent with itself.  There is really no need for this to happen, 
since the only detriment to holding the obsolete files is diskspace (and time 
for infrastructure to develop the changes).  Since every file is requested 
directly by name there should be no conflicts caused by a different version of 
some of the rpms to be there.

The only conflict would occur if a file were replaced with the same name... 
something that should never happen anyway.

The problem for anaconda is, I doubt its a trivial adjustment to make (yes I 
agree it would be a good thing though).  Since anaconda is supposed to be able 
to handle external repos soon, its crucial that it is able to deal with the repo 
changing during install... it seems like it would need to:
- try to fetch the file multiple times, try multiple mirrors if possible
- realize the file is no longer available
- remember what is already been installed, and what was supposed to be installed
- refetch the repodata
- depsolve, noting the files already installed which now need to be updated (any 
that changed since installed)
- build new install list, keep going

I suggested the changes to repository handling because that sounds non-trivial 
at best to get implemented into anaconda soonish.  Essentially the problem is 
circumvented by just not deleting files from the repo so fast, at the cost of 
diskspace.

-- 
Andrew Farris <lordmorgul(a)gmail.com&gt; <ajfarris(a)gmail.com&gt;
  gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
----                                                                       ----