Mirroring errors from kernel.org

[Erinn Looney-Triggs]

Folks we mirror from kernel.org (who I have been trying to get into 
contact with to allow us to mirror without rate limiting, but that's 
another thing) we have in the last week started to get errors like the 
following:

rsync: link_stat 
"/epel/7/aarch64/drpms/devilspie2-0.38-3.el7_0.43-1.el7.aarch64
.drpm" (in fedora-buffet) failed: No such file or directory (2)
rsync: link_stat 
"/epel/7/aarch64/repodata/3559f42dfb922344b198f8e81abc039fb826b
ead571500e3700813f4c2814359-filelists.sqlite.bz2" (in fedora-buffet) 
failed: No
such file or directory (2)
rsync: link_stat 
"/epel/7/aarch64/repodata/5572f062fe81d76019eb328da86d1d3e9dc00
ab0a43c38f04275773a5747fad8-filelists.xml.gz" (in fedora-buffet) failed: 
No such
  file or directory (2)
rsync: link_stat 
"/epel/7/aarch64/repodata/8b0c3c51b10d80a00725aa412f253c36ef5b6
33518845f6abe77ee88b9e8dfd5-updateinfo.xml.bz2" (in fedora-buffet) 
failed: No su
ch file or directory (2)
rsync: link_stat 
"/epel/7/aarch64/repodata/90a97310bf0808f404e5d726f68c083001079
bb5ed29684e26a1da03cef6f8f2-other.xml.gz" (in fedora-buffet) failed: No 
such fil
e or directory (2)
rsync: link_stat 
"/epel/7/aarch64/repodata/9132af8d3084acc749e252f0be14ba021ed0a
d8e7169a207b775ba494b2fb9a4-prestodelta.xml.gz" (in fedora-buffet) 
failed: No su
ch file or directory (2)

Any insight here would be appreciated or I'll just keep digging.

-Erinn

[Adrian Reber]

On Sun, Apr 22, 2018 at 04:23:13PM -0600, Erinn Looney-Triggs wrote:
>  Folks we mirror from kernel.org (who I have been trying to get into
> contact
>  with to allow us to mirror without rate limiting, but that's another thing)
>  we have in the last week started to get errors like the following:
>  
>  rsync: link_stat
>  "/epel/7/aarch64/drpms/devilspie2-0.38-3.el7_0.43-1.el7.aarch64
>  .drpm" (in fedora-buffet) failed: No such file or directory (2)
>  rsync: link_stat
>  "/epel/7/aarch64/repodata/3559f42dfb922344b198f8e81abc039fb826b
>  ead571500e3700813f4c2814359-filelists.sqlite.bz2" (in fedora-buffet) failed:
>  No
>  such file or directory (2)
>  rsync: link_stat
>  "/epel/7/aarch64/repodata/5572f062fe81d76019eb328da86d1d3e9dc00
>  ab0a43c38f04275773a5747fad8-filelists.xml.gz" (in fedora-buffet) failed: No
>  such
>   file or directory (2)
>  rsync: link_stat
>  "/epel/7/aarch64/repodata/8b0c3c51b10d80a00725aa412f253c36ef5b6
>  33518845f6abe77ee88b9e8dfd5-updateinfo.xml.bz2" (in fedora-buffet) failed:
>  No su
>  ch file or directory (2)
>  rsync: link_stat
>  "/epel/7/aarch64/repodata/90a97310bf0808f404e5d726f68c083001079
>  bb5ed29684e26a1da03cef6f8f2-other.xml.gz" (in fedora-buffet) failed: No such
>  fil
>  e or directory (2)
>  rsync: link_stat
>  "/epel/7/aarch64/repodata/9132af8d3084acc749e252f0be14ba021ed0a
>  d8e7169a207b775ba494b2fb9a4-prestodelta.xml.gz" (in fedora-buffet) failed:
>  No su
>  ch file or directory (2)
>  
>  Any insight here would be appreciated or I'll just keep digging. 
Not much we can do from Fedora's side. I am sending this email also to
the kernel.org mirror admin (according to MirrorManager); maybe they
know what is happening.

		Adrian

[Jason L Tibbitts III]

> >>>> "EL" == Erinn Looney-Triggs
> <looneytr(a)Colorado.EDU&gt; writes: 
EL> rsync: link_stat
EL> "/epel/7/aarch64/drpms/devilspie2-0.38-3.el7_0.43-1.el7.aarch64
EL> .drpm" (in fedora-buffet) failed: No such file or directory (2)

So from the error it looks like you're using quick-fedora-mirror.  In
case it's not obvious what has happened, qfm has grabbed the file lists,
decided that you are missing files and specifically requested them from
your upstream mirror.  But that mirror doesn't have the files.

This can happen if it is taking you a long time to sync up and in the
meantime the server has pulled new content which resulted in those files
being deleted.  But that situation should be transient; qfm may retry to
make sure but the next run should get the new file lists and work from
those.

The only other situation I can think of is if your upstream mirror is
using filters so that it doesn't carry that content.  But I'm relatively
certain that kernel.org doesn't do that (and you wouldn't really want to
use qfm against such a host anyway, unless you match their filters).

So the real question is whether this is a persistent problem (with the
same set of missing files).  If not then I would expect it to sort
itself out over time.  But if it is persistent then kernel.org has lost
some content somewhere and they either haven't been able to complete an
rsync run or just aren't mirroring.

I do need to look at the way quick-fedora-mirror handles that precise
rsync error to change the way it retries (or to make sure it doesn't
retry), since subsequent retries without re-fetching the file lists will
just fail in the same manner.

 - J<