I have a script that uses rsync to pull the Fedora repositories nightly. If I run it manually, it works flawlessly. But if it runs via cron, I get an error on all the Fedora 28 repositories. I do not get the error on Fedora 27 or RPMFusion 27 or 28 repositories. The script sets timeout values to 3600 seconds but fails in only a few seconds. (I haven't started to sync F29 repos yet.)
The error message is,
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(276) [Receiver=3.1.3]
The commands in the script are,
# cat mirror #!/bin/sh /root/bin/mirror-fedora-27 /root/bin/mirror-fedora-28 /root/bin/mirror-rpmfusion-27 /root/bin/mirror-rpmfusion-28
And the F28 subordinate script is,
# cat mirror-fedora-28 #!/bin/sh opt="-aHv --timeout=3600 --contimeout=3600 --exclude=repoview/**" opt+=" --usermap=:apache --groupmap=:apache --exclude=0ad-*" opt+=" --exclude=/*/armhfp/** --exclude=debug/**" opt+=" --exclude=repoview/** --exclude=*.html" opt+=" --exclude=/*/SRPMS/** --delete-excluded" src="rsync://mirrors.kernel.org/fedora" dst="/var/www/html/pub/fedora/linux" r="releases"; u="updates"; s="Server"; e="Everything"; w="Workstation" doit () { mkdir -p $dst/$3 rsync $opt $src/$2 $dst/$3 } doit "$e x86_64" $r/28/$e/x86_64/os/ $r/28/$e/x86_64/os doit "$s x86_64" $r/28/$s/x86_64/os/ $r/28/$s/x86_64/os doit "$w x86_64" $r/28/$w/x86_64/os/ $r/28/$w/x86_64/os doit "$u x86_64" $u/28/$e/x86_64/ $u/28/$e/x86_64/os
Other scripts are similar with the obvious substitutions.
On 12/20/18 3:10 PM, CLOSE Dave wrote:
I have a script that uses rsync to pull the Fedora repositories nightly. If I run it manually, it works flawlessly. But if it runs via cron, I get an error on all the Fedora 28 repositories. I do not get the error on Fedora 27 or RPMFusion 27 or 28 repositories. The script sets timeout values to 3600 seconds but fails in only a few seconds. (I haven't started to sync F29 repos yet.)
The error message is,
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(276) [Receiver=3.1.3]
The commands in the script are,
# cat mirror #!/bin/sh /root/bin/mirror-fedora-27 /root/bin/mirror-fedora-28 /root/bin/mirror-rpmfusion-27 /root/bin/mirror-rpmfusion-28
And the F28 subordinate script is,
# cat mirror-fedora-28 #!/bin/sh opt="-aHv --timeout=3600 --contimeout=3600 --exclude=repoview/**" opt+=" --usermap=:apache --groupmap=:apache --exclude=0ad-*" opt+=" --exclude=/*/armhfp/** --exclude=debug/**" opt+=" --exclude=repoview/** --exclude=*.html" opt+=" --exclude=/*/SRPMS/** --delete-excluded" src="rsync://mirrors.kernel.org/fedora" dst="/var/www/html/pub/fedora/linux" r="releases"; u="updates"; s="Server"; e="Everything"; w="Workstation" doit () { mkdir -p $dst/$3 rsync $opt $src/$2 $dst/$3 } doit "$e x86_64" $r/28/$e/x86_64/os/ $r/28/$e/x86_64/os doit "$s x86_64" $r/28/$s/x86_64/os/ $r/28/$s/x86_64/os doit "$w x86_64" $r/28/$w/x86_64/os/ $r/28/$w/x86_64/os doit "$u x86_64" $u/28/$e/x86_64/ $u/28/$e/x86_64/os
Other scripts are similar with the obvious substitutions.
Running interactively but not under cron usually indicates that you've forgotten that running under cron, you don't have the same environment as you do in an interactive session. So, make sure your environment is set correctly in the script (such as the PATH and such). Just for giggles, try creating a script:
#!/bin/bash env >/tmp/environ.txt
Run it from the command line look at the content of /tmp/environ.txt. Then run it again as a cron job and look at the output. You'll see the differences. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - Huked on foniks reely wurked for me! - ----------------------------------------------------------------------
Rick Stevens wrote:
Running interactively but not under cron usually indicates that you've forgotten that running under cron, you don't have the same environment as you do in an interactive session. So, make sure your environment is set correctly in the script (such as the PATH and such).
Thanks for the reply, Rick. I'm certainly aware of the issue. But note that I have four nearly identical sub-scripts, three of which work flawlessly and one of which reports the error only under cron. The error message seems network related, not environment related, though I don't know for sure what the error means.
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(276) [Receiver=3.1.3]
The only online reference I could find to that message suggested possible missing directories. But the scripts all use absolute pathnames and all of them do exist.
Yesterday, I wrote:
I have a script that uses rsync to pull the Fedora repositories nightly. If I run it manually, it works flawlessly. But if it runs via cron, I get an error on all the Fedora 28 repositories. I do not get the error on Fedora 27 or RPMFusion 27 or 28 repositories. The script sets timeout values to 3600 seconds but fails in only a few seconds. (I haven't started to sync F29 repos yet.)
The error message is,
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(276) [Receiver=3.1.3]
The commands in the script are,
# cat mirror #!/bin/sh /root/bin/mirror-fedora-27 /root/bin/mirror-fedora-28 /root/bin/mirror-rpmfusion-27 /root/bin/mirror-rpmfusion-28
Just for the heck of it, I changed this script last night to reverse the order of the first two subordinate scripts:
# cat mirror #!/bin/sh /root/bin/mirror-fedora-28 /root/bin/mirror-fedora-27 /root/bin/mirror-rpmfusion-27 /root/bin/mirror-rpmfusion-28
And the problem changed. Now it was the F27 updates which failed with the error above. (And, yes, I know F27 is no longer being updated so, even if it worked, there shouldn't have been any updates loaded.)
And the F27 subordinate script is,
# cat mirror-fedora-27 #!/bin/sh opt="-aHv --timeout=3600 --contimeout=3600 --exclude=repoview/**" opt+=" --usermap=:apache --groupmap=:apache --exclude=0ad-*" opt+=" --exclude=/*/armhfp/** --exclude=debug/**" opt+=" --exclude=repoview/** --exclude=*.html" opt+=" --exclude=/*/SRPMS/** --delete-excluded" src="rsync://mirrors.kernel.org/fedora" dst="/var/www/html/pub/fedora/linux" r="releases"; u="updates"; s="Server"; e="Everything"; w="Workstation" doit () { mkdir -p $dst/$3 rsync $opt $src/$2 $dst/$3 } doit "$e x86_64" $r/27/$e/x86_64/os/ $r/27/$e/x86_64/os doit "$s x86_64" $r/27/$s/x86_64/os/ $r/27/$s/x86_64/os doit "$w x86_64" $r/27/$w/x86_64/os/ $r/27/$w/x86_64/os doit "$u x86_64" $u/27/$e/x86_64/ $u/27/$e/x86_64/os
Other scripts are similar with the obvious substitutions.
Note that, even after the failures in the F27 updates last night or the F28 updates previously, the RPMFusion updates did not fail.
On 12/21/18 12:05 PM, CLOSE Dave wrote:
Yesterday, I wrote:
I have a script that uses rsync to pull the Fedora repositories nightly. If I run it manually, it works flawlessly. But if it runs via cron, I get an error on all the Fedora 28 repositories. I do not get the error on Fedora 27 or RPMFusion 27 or 28 repositories. The script sets timeout values to 3600 seconds but fails in only a few seconds. (I haven't started to sync F29 repos yet.)
The error message is,
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(276) [Receiver=3.1.3]
The commands in the script are,
# cat mirror #!/bin/sh /root/bin/mirror-fedora-27 /root/bin/mirror-fedora-28 /root/bin/mirror-rpmfusion-27 /root/bin/mirror-rpmfusion-28
Just for the heck of it, I changed this script last night to reverse the order of the first two subordinate scripts:
# cat mirror #!/bin/sh /root/bin/mirror-fedora-28 /root/bin/mirror-fedora-27 /root/bin/mirror-rpmfusion-27 /root/bin/mirror-rpmfusion-28
And the problem changed. Now it was the F27 updates which failed with the error above. (And, yes, I know F27 is no longer being updated so, even if it worked, there shouldn't have been any updates loaded.)
And the F27 subordinate script is,
# cat mirror-fedora-27 #!/bin/sh opt="-aHv --timeout=3600 --contimeout=3600 --exclude=repoview/**" opt+=" --usermap=:apache --groupmap=:apache --exclude=0ad-*" opt+=" --exclude=/*/armhfp/** --exclude=debug/**" opt+=" --exclude=repoview/** --exclude=*.html" opt+=" --exclude=/*/SRPMS/** --delete-excluded" src="rsync://mirrors.kernel.org/fedora" dst="/var/www/html/pub/fedora/linux" r="releases"; u="updates"; s="Server"; e="Everything"; w="Workstation" doit () { mkdir -p $dst/$3 rsync $opt $src/$2 $dst/$3 } doit "$e x86_64" $r/27/$e/x86_64/os/ $r/27/$e/x86_64/os doit "$s x86_64" $r/27/$s/x86_64/os/ $r/27/$s/x86_64/os doit "$w x86_64" $r/27/$w/x86_64/os/ $r/27/$w/x86_64/os doit "$u x86_64" $u/27/$e/x86_64/ $u/27/$e/x86_64/os
Other scripts are similar with the obvious substitutions.
Note that, even after the failures in the F27 updates last night or the F28 updates previously, the RPMFusion updates did not fail.
Ok, so it looks like one of the scripts is leaving you in a bogus directory so the next script fails. Like I said, environment things can be subtle.
In scripts, I typically have an explicit "cd /some/base/dir" command inside a loop so I know each iteration of the loop is executing from a known base directory and not wherever the last iteration of the loop left me. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - LOOK OUT!!! BEHIND YOU!!! - ----------------------------------------------------------------------
On Thu, Dec 20, 2018 at 11:10:47PM +0000, CLOSE Dave wrote:
I have a script that uses rsync to pull the Fedora repositories nightly. If I run it manually, it works flawlessly. But if it runs via cron, I
I encourage you to use Quick Fedora Mirror instead: https://pagure.io/quick-fedora-mirror.
From the description:
A full rsync of fedora-buffet0 can take hours just to receive the file list, due to the fact that there are over 11 million files. This has to happen for every client, every time they want to update, which is murderous on the download servers and worse on the backend NFS server. It also slows the propagation of important updates because mirrors simply can't poll often.
By generating a simple database of file and directory timestamps and sizes, it becomes easy for the client to determine which files have changed since the last mirror run and only mirror those files. This also provides enough information to handle files which have been deleted on the server and files which are missing on the client. In most situations, it also allows hardlinks to be copied as hardlinks instead of being downloaded multiple times.
-
CLOSE Dave -
Matthew Miller -
Rick Stevens