5:26 p.m.
As several people have noted, the file directory structure of EPEL has
changed recently. This layout may require changes in both (1) scripts
written with hard-coded locations, and (2) mirrors which were unable
to get daily updates from the main mirrors. While the changes were
communicated in meetings, I did not adequately comprehend their
effects to let mirrors and EPEL users know about it. This meant this
announcement was delayed over two weeks.
===============
What Happened
===============
The updates in the build system were to add new features and make the
release engineering code more manageble. The old release style used by
EPEL in EL-6 and EL-7 was different from how all other releases were
done and caused several problems for the release code and mirrors.
1. Due to all the files of the release being in one directory, any
code which needed to stat (2) the directory caused the server to go
over thousands of files before returning. With EPEL being a large
amount of downloads, this negatively impacted systems. Servers
mirroring the data could find long delays in rsyncing the data
down.
2. The code that generated this was a 'special' case in the Fedora
releng release process which was fragile and tended to cause
problems for updates and releases in both EPEL and the normal
release.
3. The layouts were different from the current Fedora release so that
people grabbing software from multiple places also had to special
case their scripts.
During the updates to the release system with a new version of pungi,
it was decided to remove this special case and have all software
Fedora created laid out in the same structure by the build tools. This
would hopefully make things much more maintainable and improve
performance.
In order to safely transition, there would be a time where the old
files would remain on the server in the old trees and also be
hardlinked to their new location. This was intended to allow for
mirrors to get the files with the minimum amount of bandwidth. However
there were some problems which showed up.
1. As I said before, I didn't grasp that the change was going to
affect EPEL and didn't communicate this to the lists.
2. The transition time for removing the hardlinks was in days versus
weeks. While most mirrors do daily updates, some only do weekly or
monthly rsync's. They missed the hardlinks completely and had to
download data twice.
3. In the usual rule of three, various top level mirrors
(mirrors.kernel.org and some others) had un-related mirroring
problems at the end of October. When these servers caught up with
the new layout, the hardlinked files were gone. This meant that
mirrors taking data from a couple of tier1 sites had large uploads.
=================================
How to deal with current things
=================================
The current layout structure should be 'solid' for the next couple of
years. With the break down of packages into alphabetical subtrees, the
'load' per server should not require a re-ordering in the near
future.
If you have written scripts which downloaded a specific file from the
mirrors, (aka
http://dl.fedoraproject.org/pub/archive/epel/5/i386/epel-release-5-4.noar...
or some similar link), you should instead use a stable linked package
like http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
The epel-release packages get updated regularly to get new macros or
other changes so linking to a specific file is very error prone.
Otherwise one should use yum/dnf related commands to get the files
from the mirrors. This is useful for mirror sites which may alter the
directory structure themselves and thus only the repodata is 'safe' to
figure out what to download.
--
Stephen J Smoogen.