squid issues

[Jesse Keating]

I noticed something while composing this release.  Packages gotten from
kojipkgs (and thus the proxy) all have a timestamp of when the file was
downloaded, vs the timestamp of the file on the fileserver.  This has
caused me some... frustration in getting my composes synced on the
master mirror using hardlinks to rawhide.

Can somebody help me debug why this is?  I'm not familiar with our squid
setup, and I couldn't find it in a quick wiki search for 'squid'.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Stephen John Smoogen]

2009/2/2 Jesse Keating <jkeating(a)redhat.com&gt;:
>  I noticed something while composing this release.  Packages gotten
> from
>  kojipkgs (and thus the proxy) all have a timestamp of when the file was
>  downloaded, vs the timestamp of the file on the fileserver.  This has
>  caused me some... frustration in getting my composes synced on the
>  master mirror using hardlinks to rawhide.
>
>  Can somebody help me debug why this is?  I'm not familiar with our squid
>  setup, and I couldn't find it in a quick wiki search for 'squid'. 
Hmmm that sounds like something that we ran into a loooong time ago.
Sorry I can't remember the solution off hand. I think Andrew came up
with it but will see if I kept it somewhere in notes.



-- 
Stephen J Smoogen. -- BSD/GNU/Linux
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

Change request: reenable last-modified (was Re: squid issues)

[Ricky Zhou]

On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
>  I noticed something while composing this release.  Packages gotten
> from
>  kojipkgs (and thus the proxy) all have a timestamp of when the file was
>  downloaded, vs the timestamp of the file on the fileserver.  This has
>  caused me some... frustration in getting my composes synced on the
>  master mirror using hardlinks to rawhide.
>  
>  Can somebody help me debug why this is?  I'm not familiar with our squid
>  setup, and I couldn't find it in a quick wiki search for 'squid'. 
Oops,
I cleared the Last-Modified headers in the configs hoping that it
might somehow save some NFS load.  Here's a patch to remove those lines
- can I get two +1s?

diff --git a/configs/system/fedora-updates-push b/configs/system/fedora-updates-push
index 8c09e27..380af87 100755
--- a/configs/system/fedora-updates-push
+++ b/configs/system/fedora-updates-push
@@ -38,7 +38,8 @@ done
 pushd /pub/fedora/
 TMPFILE=$(mktemp -p /tmp/)
 find * -print > $TMPFILE
-diff $TMPFILE fullfilelist >/dev/null
-if [ "$?" = "1" ]; then
+if diff $TMPFILE fullfilelist > /dev/null; then
+    rm -f $TMPFILE
+else
     mv $TMPFILE fullfilelist
 fi
diff --git a/manifests/services/mirrormaster.pp b/manifests/services/mirrormaster.pp
index ca01522..b7c2b47 100644
--- a/manifests/services/mirrormaster.pp
+++ b/manifests/services/mirrormaster.pp
@@ -18,7 +18,7 @@ class mirrormaster {
     }
 
     cron { epel-sync:
-        command => 'rsync -aHz --numeric-ids --delete-after
--exclude=".snapshot" --exclude="*/build-logs"
buildsys.fedoraproject.org::epel/ /pub/epel/ &>/dev/null && TMPFILE=`mktemp
-p /tmp`; pushd /pub/epel; find * -print > $TMPFILE; diff $TMPFILE fullfilelist
>/dev/null; if [ "$?" = "1" ]; then mv $TMPFILE fullfilelist;
fi',
+        command => 'rsync -aHz --numeric-ids --delete-after
--exclude=".snapshot" --exclude="*/build-logs"
buildsys.fedoraproject.org::epel/ /pub/epel/ &>/dev/null &&
TMPFILE=$(mktemp -p /tmp); pushd /pub/epel; find * -print > $TMPFILE; if diff $TMPFILE
fullfilelist > /dev/null; then rm -f $TMPFILE; else mv $TMPFILE fullfilelist; fi',
         user => ftpsync,
         minute => [ 10, 40 ],
         ensure => present,

Thanks,
Ricky

Change request: reenable last-modified (was Re: squid issues)

[Jesse Keating]

On Mon, 2009-02-02 at 21:08 -0500, Ricky Zhou wrote:
>   Oops, I cleared the Last-Modified headers in the configs hoping that
> it
>  might somehow save some NFS load.  Here's a patch to remove those lines
>  - can I get two +1s? 
Wrong diff?

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

Change request: reenable last-modified (was Re: squid issues)

[Ricky Zhou]

On 2009-02-02 09:08:54 PM, Ricky Zhou wrote:
>  On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
>  > I noticed something while composing this release.  Packages gotten from
>  > kojipkgs (and thus the proxy) all have a timestamp of when the file was
>  > downloaded, vs the timestamp of the file on the fileserver.  This has
>  > caused me some... frustration in getting my composes synced on the
>  > master mirror using hardlinks to rawhide.
>  > 
>  > Can somebody help me debug why this is?  I'm not familiar with our squid
>  > setup, and I couldn't find it in a quick wiki search for 'squid'.
>  Oops, I cleared the Last-Modified headers in the configs hoping that it
>  might somehow save some NFS load.  Here's a patch to remove those lines
>  - can I get two +1s? 
I'm sorry, that was the wrong patch.  Here is the right
one:

diff --git a/configs/web/applications/kojipkgs.conf
b/configs/web/applications/kojipkgs.conf
index 7077546..6ad116c 100644
--- a/configs/web/applications/kojipkgs.conf
+++ b/configs/web/applications/kojipkgs.conf
@@ -6,7 +6,6 @@ Alias /packages /mnt/koji/packages
 <Directory /mnt/koji/packages>
     Options Indexes
     FileETag None
-    Header unset Last-Modified
 </Directory>
 
 Alias /repos /mnt/koji/repos
@@ -14,7 +13,6 @@ Alias /repos /mnt/koji/repos
 <Directory /mnt/koji/repos>
     Options Indexes
     FileETag None
-    Header unset Last-Modified
 </Directory>
 
 Alias /scratch /mnt/koji/scratch
@@ -22,7 +20,6 @@ Alias /scratch /mnt/koji/scratch
 <Directory /mnt/koji/scratch>
     Options Indexes
     FileETag None
-    Header unset Last-Modified
 </Directory>
 
 Alias /work /mnt/koji/work
@@ -30,7 +27,6 @@ Alias /work /mnt/koji/work
 <Directory /mnt/koji/work>
     Options Indexes
     FileETag None
-    Header unset Last-Modified
 </Directory>
 
 Alias /static-repos /mnt/koji/static-repos
@@ -38,7 +34,6 @@ Alias /static-repos /mnt/koji/static-repos
 <Directory /mnt/koji/static-repos>
     Options Indexes FollowSymLinks
     FileETag None
-    Header unset Last-Modified
 </Directory>
 
 Alias /mash /mnt/koji/mash
@@ -46,6 +41,5 @@ Alias /mash /mnt/koji/mash
 <Directory /mnt/koji/mash>
     Options Indexes FollowSymLinks
     FileETag None
-    Header unset Last-Modified
 </Directory>
 
Thanks,
Ricky

Change request: reenable last-modified (was Re: squid issues)

[Mike McGrath]

On Mon, 2 Feb 2009, Ricky Zhou wrote:

>  On 2009-02-02 09:08:54 PM, Ricky Zhou wrote:
>  > On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
>  > > I noticed something while composing this release.  Packages gotten from
>  > > kojipkgs (and thus the proxy) all have a timestamp of when the file was
>  > > downloaded, vs the timestamp of the file on the fileserver.  This has
>  > > caused me some... frustration in getting my composes synced on the
>  > > master mirror using hardlinks to rawhide.
>  > >
>  > > Can somebody help me debug why this is?  I'm not familiar with our squid
>  > > setup, and I couldn't find it in a quick wiki search for 'squid'.
>  > Oops, I cleared the Last-Modified headers in the configs hoping that it
>  > might somehow save some NFS load.  Here's a patch to remove those lines
>  > - can I get two +1s?
>  I'm sorry, that was the wrong patch.  Here is the right one:
>
>  diff --git a/configs/web/applications/kojipkgs.conf
> b/configs/web/applications/kojipkgs.conf
>  index 7077546..6ad116c 100644
>  --- a/configs/web/applications/kojipkgs.conf
>  +++ b/configs/web/applications/kojipkgs.conf
>  @@ -6,7 +6,6 @@ Alias /packages /mnt/koji/packages
>   <Directory /mnt/koji/packages>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>
>   Alias /repos /mnt/koji/repos
>  @@ -14,7 +13,6 @@ Alias /repos /mnt/koji/repos
>   <Directory /mnt/koji/repos>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>
>   Alias /scratch /mnt/koji/scratch
>  @@ -22,7 +20,6 @@ Alias /scratch /mnt/koji/scratch
>   <Directory /mnt/koji/scratch>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>
>   Alias /work /mnt/koji/work
>  @@ -30,7 +27,6 @@ Alias /work /mnt/koji/work
>   <Directory /mnt/koji/work>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>
>   Alias /static-repos /mnt/koji/static-repos
>  @@ -38,7 +34,6 @@ Alias /static-repos /mnt/koji/static-repos
>   <Directory /mnt/koji/static-repos>
>       Options Indexes FollowSymLinks
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>
>   Alias /mash /mnt/koji/mash
>  @@ -46,6 +41,5 @@ Alias /mash /mnt/koji/mash
>   <Directory /mnt/koji/mash>
>       Options Indexes FollowSymLinks
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>  
+1 from me.

	-Mike

Change request: reenable last-modified (was Re: squid issues)

[Seth Vidal]

On Mon, 2009-02-02 at 21:15 -0500, Ricky Zhou wrote:
>  On 2009-02-02 09:08:54 PM, Ricky Zhou wrote:
>  > On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
>  > > I noticed something while composing this release.  Packages gotten from
>  > > kojipkgs (and thus the proxy) all have a timestamp of when the file was
>  > > downloaded, vs the timestamp of the file on the fileserver.  This has
>  > > caused me some... frustration in getting my composes synced on the
>  > > master mirror using hardlinks to rawhide.
>  > > 
>  > > Can somebody help me debug why this is?  I'm not familiar with our squid
>  > > setup, and I couldn't find it in a quick wiki search for 'squid'.
>  > Oops, I cleared the Last-Modified headers in the configs hoping that it
>  > might somehow save some NFS load.  Here's a patch to remove those lines
>  > - can I get two +1s?
>  I'm sorry, that was the wrong patch.  Here is the right one:
>  
>  diff --git a/configs/web/applications/kojipkgs.conf
> b/configs/web/applications/kojipkgs.conf
>  index 7077546..6ad116c 100644
>  --- a/configs/web/applications/kojipkgs.conf
>  +++ b/configs/web/applications/kojipkgs.conf
>  @@ -6,7 +6,6 @@ Alias /packages /mnt/koji/packages
>   <Directory /mnt/koji/packages>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>   
>   Alias /repos /mnt/koji/repos
>  @@ -14,7 +13,6 @@ Alias /repos /mnt/koji/repos
>   <Directory /mnt/koji/repos>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>   
>   Alias /scratch /mnt/koji/scratch
>  @@ -22,7 +20,6 @@ Alias /scratch /mnt/koji/scratch
>   <Directory /mnt/koji/scratch>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>   
>   Alias /work /mnt/koji/work
>  @@ -30,7 +27,6 @@ Alias /work /mnt/koji/work
>   <Directory /mnt/koji/work>
>       Options Indexes
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>   
>   Alias /static-repos /mnt/koji/static-repos
>  @@ -38,7 +34,6 @@ Alias /static-repos /mnt/koji/static-repos
>   <Directory /mnt/koji/static-repos>
>       Options Indexes FollowSymLinks
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>   
>   Alias /mash /mnt/koji/mash
>  @@ -46,6 +41,5 @@ Alias /mash /mnt/koji/mash
>   <Directory /mnt/koji/mash>
>       Options Indexes FollowSymLinks
>       FileETag None
>  -    Header unset Last-Modified
>   </Directory>
>    
+1

-sv