FBR: add exclude to mirror crawler

[Adrian Reber]

The rsync based crawler has two excludes hardcoded:

cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"

I am just in discussions with a new mirror and the rsync based crawler
fails there with:

rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)

To avoid this in this case and also on other mirrors I would like to
apply following patch:

diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
index 2bdd60273..69c930b23 100644
--- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
+++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
@@ -161,7 +161,7 @@ CHECK_SESSION_IP = True

# Specify additional rsync parameters for the crawler
 # # --timeout 14400: abort rsync crawl after 4 hours
-CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
+CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'

###
 # Configuration options used by the crons

This excludes 'lost+found' from being scanned on all hosts. Can I get two +1
to update the crawler's configuration?

Adrian

[Patrick Uiterwijk]

On Thu, Mar 8, 2018 at 4:37 PM, Adrian Reber  wrote:
> The rsync based crawler has two excludes hardcoded:
>
> cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
>
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
>
> rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
>
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
>
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>
> # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400
> --exclude=lost+found'
>
> ###
>  # Configuration options used by the crons
>
> This excludes 'lost+found' from being scanned on all hosts. Can I get two
> +1
> to update the crawler's configuration?

+1
>             Adrian
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to infrastructure-leave@lists.
> fedoraproject.org

[Pierre-Yves Chibon]

On Thu, Mar 08, 2018 at 04:37:41PM +0100, Adrian Reber wrote:
> The rsync based crawler has two excludes hardcoded:
>
> cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
>
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
>
> rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
>
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
>
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>
> # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'

+1 for me as well

Pierre

[Adrian Reber]

Pushed and live on both MM crawlers.

Adrian

On Thu, Mar 08, 2018 at 04:37:41PM +0100, Adrian Reber wrote:
> The rsync based crawler has two excludes hardcoded:
>
> cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
>
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
>
> rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
>
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
>
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>
> # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'
>
> ###
>  # Configuration options used by the crons
>
> This excludes 'lost+found' from being scanned on all hosts. Can I get two +1
> to update the crawler's configuration?
> Adrian
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

Adrian