[EPEL-devel] Outage Report: EPEL 7 repository corruption problem
Problem Description:
On 2018-05-11 04:00 UTC reports started coming into centos IRC
channels about EPEL being corrupted and causing breakages. These were
then reported to #fedora-admin and #epel-devel. The problem would show
up as something like:
One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
The problem was examined and turned out to be that an NFS problem on
the backend systems causing the createrepo_c to create the
repositories to create a corrupted SQL file. A program which was to
catch this did not work for some reason still being investigated and
the corrupted sqllite file was mirrored out.
Admins began filling up the #epel and #centos channel asking why their
systems were broken. I would like to thank avij, tmz and others who
worked on answering as many of the people as possible.
Solution:
Because of the way mirroring works, this problem may affect clients
for hours after the fix has been made on the server. There are three
things a client can do:
1. If you have a dedicated mirror, have the mirror update itself with
the upstream mirrors.
2. On client systems you may need to do a 'yum clean all' in order to
remove the bad sql in case yum thinks it is still good to cache from.
3. You can skip yum on updates with:
yum --disablerepo=epel update
Further updates may occur as research is done.
--
Stephen J Smoogen.