q-f-m or master fullfiletimelist problem?

[Chuck Anderson]

I have one system I keep pointed directly at my mirror (not relying on
mirrorlist) so I can find problems with my local mirror.  Today it says:

Cannot download 'http://download.wpi.edu/pub/fedora/linux/updates/25/x86_64/': Yum repo downloading error: Downloading error(s): repodata/0cbc1e966823acb0f4addccb4db219617335c54e85554fed2f3bca01c6383ea4-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/3e93bf5f3f9f9e242a136cd8487f9b917880373791533ec41c429f3942d1b61f-prestodelta.xml.xz - Cannot download, all mirrors were already tried without success; repodata/32d7de1743b6533476c1dfbcc9a7ded8fb71cf389b21e61602ede8313559d5dc-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/fc618e356dcf2d17199eb8118651274f146c4bb4720b1b9ab8a731036daed7da-updateinfo.xml.xz - Cannot download, all mirrors were already tried without success.
Error: Failed to synchronize cache for repo 'updates'

I traced this problem to a stale repomd.xml from Apr 3:

-rw-r--r--.  1 mirror mirror     4848 Apr  3 18:12 repomd.xml

Compare this to the one on the master:

-rw-r--r--        4848 2017/04/05 14:27:41 repomd.xml

Same size.  Uh oh.  q-f-m doesn't think this file needs to be
synced--it isn't showing up in the mastertransferlist:
> grep updates/25/x86_64/repodata/repomd.xml quick-mirror.YksUUJhBIF/*

quick-mirror.YksUUJhBIF/fullfiletimelist-fedora:1491369612      f       4848    linux/updates/25/x86_64/repodata/repomd.xml
quick-mirror.YksUUJhBIF/fullfiletimelist-fedora:e2fcb793861a193d2c3d8ed255723bfa241c2a34        linux/updates/25/x86_64/repodata/repomd.xml
> sha1sum /srv/ftp/pub/fedoraproject.org/fedora/linux/updates/25/x86_64/repodata/repomd.xml ~/repomd.xml

5145e1fcb2207cce33a0d202e6834b5fe792a3e1  /srv/ftp/pub/fedoraproject.org/fedora/linux/updates/25/x86_64/repodata/repomd.xml
d76ad1ef03095a97e2d5e4c9a36cf4598c4b4477  /home/cra/repomd.xml

(the last file being one I directly downloaded from the master just now)

[Jason L Tibbitts III]

> > > > > "CA" == Chuck Anderson  writes:

CA> I traced this problem to a stale repomd.xml from Apr 3:

CA> -rw-r--r--.  1 mirror mirror 4848 Apr 3 18:12 repomd.xml

CA> Compare this to the one on the master:

CA> -rw-r--r-- 4848 2017/04/05 14:27:41 repomd.xml

CA> Same size.  Uh oh.  q-f-m doesn't think this file needs to be
CA> synced--it isn't showing up in the mastertransferlist:

repomd.xml files will always have the same size.  That's why the file
list contains the [Checksums SHA1] section.  When a file appears there, it will
be transferred if its SHA1 checksum fails to match.

I don't really have enough information to tell if that's not happening.
It does look to me as if the file lists have the [Checksums SHA1]
section and that it contains the proper data.  However, you can double check that
yourself trivially.

1) cd to a directory that contains the file list.
2) Copy that file list to /tmp/foo.
3) Edit to remove everything besides the [Checksums SHA1] section.
4) Remove the [End] tag at the end as well.
5)  Run "sha1sum -c /tmp/foo".

That should tell you if the files are current or not according to the
data that's in the file list.  You can also fetch the most current file
list from the master mirrors if you want.  And you can also verify that
every repomd.xml file in that module is represented in the checksums
section if you like.  I just did that and those files are indeed in
there.

- J<

[Chuck Anderson]

On Mon, Apr 10, 2017 at 01:31:48PM -0500, Jason L Tibbitts III wrote:
> cd to a directory that contains the file list.
> Copy that file list to /tmp/foo.
> Edit to remove everything besides the [Checksums SHA1] section.
> Remove the [End] tag at the end as well.
> Run "sha1sum -c /tmp/foo".
>
> That should tell you if the files are current or not according to the
> data that's in the file list.  You can also fetch the most current file
> list from the master mirrors if you want.  And you can also verify that
> every repomd.xml file in that module is represented in the checksums
> section if you like.  I just did that and those files are indeed in
> there.

I re-downloaded fullfiletimelist-fedora and compared against my local
copy--they are identical.

My EL6 system which runs q-f-m reports this error:
> sha1sum -c /tmp/fullfiletimelist-fedora-checksums

sha1sum: /tmp/fullfiletimelist-fedora-checksums: no properly formatted SHA1 checksum lines found

But if I replace the tab characters with two spaces I get this result
showing many failed checksums (showing just FAILED below):

linux/development/26/Everything/armhfp/debug/tree/repodata/repomd.xml: FAILED
linux/development/26/Everything/armhfp/os/repodata/repomd.xml: FAILED
linux/development/26/Everything/source/tree/repodata/repomd.xml: FAILED
linux/development/26/Everything/x86_64/debug/tree/repodata/repomd.xml: FAILED
linux/development/26/Everything/x86_64/os/repodata/repomd.xml: FAILED
linux/development/rawhide/Everything/armhfp/debug/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Everything/armhfp/os/repodata/repomd.xml: FAILED
linux/development/rawhide/Everything/source/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Everything/x86_64/debug/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Everything/x86_64/os/repodata/repomd.xml: FAILED
linux/development/rawhide/Server/armhfp/debug/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Server/armhfp/os/repodata/repomd.xml: FAILED
linux/development/rawhide/Server/source/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Server/x86_64/debug/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Server/x86_64/os/repodata/repomd.xml: FAILED
linux/development/rawhide/Workstation/armhfp/debug/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Workstation/armhfp/os/repodata/repomd.xml: FAILED
linux/development/rawhide/Workstation/source/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Workstation/x86_64/debug/tree/repodata/repomd.xml: FAILED
linux/development/rawhide/Workstation/x86_64/os/repodata/repomd.xml: FAILED
linux/updates/24/SRPMS/repodata/repomd.xml: FAILED
linux/updates/24/armhfp/debug/repodata/repomd.xml: FAILED
linux/updates/24/armhfp/repodata/repomd.xml: FAILED
linux/updates/24/i386/debug/repodata/repomd.xml: FAILED
linux/updates/24/i386/repodata/repomd.xml: FAILED
linux/updates/24/x86_64/debug/repodata/repomd.xml: FAILED
linux/updates/24/x86_64/repodata/repomd.xml: FAILED
linux/updates/testing/24/SRPMS/repodata/repomd.xml: FAILED
linux/updates/testing/24/armhfp/debug/repodata/repomd.xml: FAILED
linux/updates/testing/24/armhfp/repodata/repomd.xml: FAILED
linux/updates/testing/24/i386/debug/repodata/repomd.xml: FAILED
linux/updates/testing/24/i386/repodata/repomd.xml: FAILED
linux/updates/testing/24/x86_64/repodata/repomd.xml: FAILED
linux/updates/testing/25/SRPMS/repodata/repomd.xml: FAILED
linux/updates/testing/25/armhfp/repodata/repomd.xml: FAILED
linux/updates/testing/25/i386/debug/repodata/repomd.xml: FAILED
linux/updates/testing/25/i386/repodata/repomd.xml: FAILED
linux/updates/testing/25/x86_64/debug/repodata/repomd.xml: FAILED
linux/updates/testing/25/x86_64/repodata/repomd.xml: FAILED
linux/updates/testing/26/SRPMS/repodata/repomd.xml: FAILED
linux/updates/testing/26/armhfp/debug/repodata/repomd.xml: FAILED
linux/updates/testing/26/x86_64/debug/repodata/repomd.xml: FAILED
sha1sum: WARNING: 42 of 191 computed checksums did NOT match

[Jason L Tibbitts III]

> > > > > "CA" == Chuck Anderson  writes:

CA> My EL6 system which runs q-f-m reports this error:
> > sha1sum -c /tmp/fullfiletimelist-fedora-checksums

CA> sha1sum: /tmp/fullfiletimelist-fedora-checksums: no properly
CA> formatted SHA1 checksum lines found

Well that's interesting.  EL6 is pretty ancient and I guess it's
possible that the format that the sha1sum command will accept is
different there.  It's just extracted from the file list and piped
straight in, and that doesn't cause any issues where I've tested it.  I
will see if I can get a Centos 6 machine going to see exactly what it
doesn't like.  It wouldn't be hard to massage the output if necessary.

CA> But if I replace the tab characters with two spaces I get this
CA> result showing many failed checksums (showing just FAILED below):

Well, yes, that would indicate that those files are out of date.

- J<

[Chuck Anderson]

On Mon, Apr 17, 2017 at 02:27:01PM -0500, Jason L Tibbitts III wrote:
> > > > > > "CA" == Chuck Anderson  writes:
>
> CA> My EL6 system which runs q-f-m reports this error:
> > > sha1sum -c /tmp/fullfiletimelist-fedora-checksums
>
> CA> sha1sum: /tmp/fullfiletimelist-fedora-checksums: no properly
> CA> formatted SHA1 checksum lines found
>
> Well that's interesting.  EL6 is pretty ancient and I guess it's

If Fedora wants to encourage mirrors to use q-f-m, then at least EL6
needs to be supported.
> possible that the format that the sha1sum command will accept is
> different there.  It's just extracted from the file list and piped
> straight in, and that doesn't cause any issues where I've tested it.  I
> will see if I can get a Centos 6 machine going to see exactly what it
> doesn't like.  It wouldn't be hard to massage the output if necessary.
>
> CA> But if I replace the tab characters with two spaces I get this
> CA> result showing many failed checksums (showing just FAILED below):
>
> Well, yes, that would indicate that those files are out of date.

q-f-m says this on several runs after that:
> > Log: No change in file list for fedora-enchilada

No change in file list checksum. Skipping fedora-enchilada.

I deleted fullfiletimelist-fedora and made a simple change that caused
q-f-m to find the changed repomd.xml files and transfer the new ones.
This should be forward-compatible, because two spaces is the standard
delimiter of a sha1sum checksum file.

--- /usr/local/bin/quick-fedora-mirror~ 2017-04-17 15:57:07.733736546 -0400
+++ /usr/local/bin/quick-fedora-mirror  2017-04-17 15:57:59.964953999 -0400
@@ -610,7 +610,7 @@
         cat updatedfiles-$module >> transferlist-$module

# Extract and verify checksums
-        awk -F '\t' "/^[Checksums/ {s=1; next} /^$/ {s=0; next} {if (s) print $1 "\t$moduledir/" $2}" $fl > checksums-$module
+        awk -F '\t' "/^[Checksums/ {s=1; next} /^$/ {s=0; next} {if (s) print $1 "  $moduledir/" $2}" $fl > checksums-$module
         pushd $DESTD > /dev/null 2>&1
         sha1sum --check --quiet $tempd/checksums-$module 2> /dev/null \
             | grep -i 'failed$' \

[Jason L Tibbitts III]

Please try this patch and let me know if it helps.

diff --git a/quick-fedora-mirror b/quick-fedora-mirror
index 2d35648..89080e9 100755
--- a/quick-fedora-mirror
+++ b/quick-fedora-mirror
@@ -894,7 +894,7 @@ for module in $MODULES; do
         cat updatedfiles-$module >> transferlist-$module

# Extract and verify checksums
-        awk -F '\t' "/^[Checksums/ {s=1; next} /^$/ {s=0; next} {if (s) print $1 "\t$moduledir/" $2}" $fl > checksums-$module
+        awk -F '\t' "/^[Checksums/ {s=1; next} /^$/ {s=0; next} {if (s) print $1 "  $moduledir/" $2}" $fl > checksums-$module
         pushd $DESTD > /dev/null 2>&1
         sha1sum --check --quiet $tempd/checksums-$module 2> /dev/null \
             | grep -i 'failed$' \

[Jason L Tibbitts III]

> > > > > "CA" == Chuck Anderson  writes:

CA> If Fedora wants to encourage mirrors to use q-f-m, then at least EL6
CA> needs to be supported.

I don't recall saying that it wasn't.  I had opened
https://pagure.io/quick-fedora-mirror/issue/49 to track this and I've
already sent you a patch for testing which I guess is exactly the same
as what you sent me below.

- J<