FW: [Fedora Infrastructure] #1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date mirrors

[Matt Domsch]

Requesting +1s to apply this hotfix, which only affects the MM crawler.

--
Matt Domsch
Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

-----Original Message-----
From: Fedora Infrastructure [mailto:[e-mail]]
Sent: Mon 10/5/2009 10:38 AM
To: [e-mail]
Cc: [e-mail]; [e-mail]
Subject: [Fedora Infrastructure] #1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date mirrors

#1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date
mirrors
-----------------------------+----------------------------------------------
 Reporter:  mdomsch          |       Owner:  mdomsch   
     Type:  bug              |      Status:  new       
 Priority:  major            |   Milestone:  Fedora 12 
Component:  Web Application  |     Version:  Production
 Severity:  Normal           |    Keywords:  hotfix    
-----------------------------+----------------------------------------------
 = phenomenon =
 nb reported that running report_mirror would report hundreds of
 directories deleted on each run.

= reason =
 crawler is creating HostCategoryDir entries for not up-to-date directories
 (in fact, dirs that nb has excluded on his mirror).  report_mirror then
 deletes these entries.

= recommendation =
 commit 20986e503db481d4760e6e3ea74b07863a8e1cf9
 Author: Matt Domsch 
 Date:   Sun Oct 4 22:25:15 2009 -0500

crawler: don't create HCDs for directories which aren't up2date

diff --git a/server/crawler_perhost b/server/crawler_perhost
 index 9553fb8..8964c7e 100755
 --- a/server/crawler_perhost
 +++ b/server/crawler_perhost
 @@ -303,6 +303,9 @@ def sync_hcds(host, host_category_dirs):
          if hcd.count() > 0:
              hcd = hcd[0]
          else:
 +            # don't create HCDs for directories which aren't up2date on
 the mirror
 +            # chances are the mirror is excluding that directory
 +            if not up2date: continue
              hcd = HostCategoryDir(host_category=hc, path=path,
 directory=d)

if hcd.directory is None:

[Toshio Kuratomi]

On 10/05/2009 08:40 AM, [e-mail] wrote:
> Requesting +1s to apply this hotfix, which only affects the MM crawler.
>
> --
> Matt Domsch
> Technology Strategist, Dell Office of the CTO
> linux.dell.com & www.dell.com/linux
>
> -----Original Message-----
> From: Fedora Infrastructure [mailto:[e-mail]]
> Sent: Mon 10/5/2009 10:38 AM
> To: [e-mail]
> Cc: [e-mail]; [e-mail]
> Subject: [Fedora Infrastructure] #1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date mirrors
>
> #1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date
> mirrors
> -----------------------------+----------------------------------------------
>  Reporter:  mdomsch          |       Owner:  mdomsch   
>      Type:  bug              |      Status:  new       
>  Priority:  major            |   Milestone:  Fedora 12 
> Component:  Web Application  |     Version:  Production
>  Severity:  Normal           |    Keywords:  hotfix    
> -----------------------------+----------------------------------------------
>  = phenomenon =
>  nb reported that running report_mirror would report hundreds of
>  directories deleted on each run.
>
> = reason =
>  crawler is creating HostCategoryDir entries for not up-to-date directories
>  (in fact, dirs that nb has excluded on his mirror).  report_mirror then
>  deletes these entries.
>
> = recommendation =
>  commit 20986e503db481d4760e6e3ea74b07863a8e1cf9
>  Author: Matt Domsch 
>  Date:   Sun Oct 4 22:25:15 2009 -0500
>  crawler: don't create HCDs for directories which aren't up2date
>
> diff --git a/server/crawler_perhost b/server/crawler_perhost
>  index 9553fb8..8964c7e 100755
>  --- a/server/crawler_perhost
>  +++ b/server/crawler_perhost
>  @@ -303,6 +303,9 @@ def sync_hcds(host, host_category_dirs):
>           if hcd.count() > 0:
>               hcd = hcd[0]
>           else:
>        # don't create HCDs for directories which aren't up2date on
>
> the mirror
>        # chances are the mirror is excluding that directory
>        if not up2date: continue
>        hcd = HostCategoryDir(host_category=hc, path=path,
>
> directory=d)
>       if hcd.directory is None:

Looks good to me +1.

-Toshio

[Ricky Zhou]

On 2009-10-05 10:40:31 AM, [e-mail] wrote:
> Requesting +1s to apply this hotfix, which only affects the MM crawler.
>
> --
> Matt Domsch
> Technology Strategist, Dell Office of the CTO
> linux.dell.com & www.dell.com/linux
>
> -----Original Message-----
> From: Fedora Infrastructure [mailto:[e-mail]]
> Sent: Mon 10/5/2009 10:38 AM
> To: [e-mail]
> Cc: [e-mail]; [e-mail]
> Subject: [Fedora Infrastructure] #1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date mirrors
>
> #1719: MM hotfix: crawler incorrectly creating dir entries for not up-to-date
> mirrors
> -----------------------------+----------------------------------------------
>  Reporter:  mdomsch          |       Owner:  mdomsch   
>      Type:  bug              |      Status:  new       
>  Priority:  major            |   Milestone:  Fedora 12 
> Component:  Web Application  |     Version:  Production
>  Severity:  Normal           |    Keywords:  hotfix    
> -----------------------------+----------------------------------------------
>  = phenomenon =
>  nb reported that running report_mirror would report hundreds of
>  directories deleted on each run.
>
> = reason =
>  crawler is creating HostCategoryDir entries for not up-to-date directories
>  (in fact, dirs that nb has excluded on his mirror).  report_mirror then
>  deletes these entries.
>
> = recommendation =
>  commit 20986e503db481d4760e6e3ea74b07863a8e1cf9
>  Author: Matt Domsch 
>  Date:   Sun Oct 4 22:25:15 2009 -0500
>  crawler: don't create HCDs for directories which aren't up2date
>
> diff --git a/server/crawler_perhost b/server/crawler_perhost
>  index 9553fb8..8964c7e 100755
>  --- a/server/crawler_perhost
>  +++ b/server/crawler_perhost
>  @@ -303,6 +303,9 @@ def sync_hcds(host, host_category_dirs):
>           if hcd.count() > 0:
>               hcd = hcd[0]
>           else:
>        # don't create HCDs for directories which aren't up2date on
>
> the mirror
>        # chances are the mirror is excluding that directory
>        if not up2date: continue
>        hcd = HostCategoryDir(host_category=hc, path=path,
>
> directory=d)
>       if hcd.directory is None:
>
> --

+1

Thanks,
Ricky