5:25 a.m.
Dear Matt,
we are looking into it right now. It seems that, some rsync processes hang.
We will also take into account your recommendations.
Thanks, and sorry for the inconvenience!
Regards,
Dimitris ...
--------------BEGIN PUBLIC INFO BLOCK--------------
Counalakis E. Dimitris-Nektarios [coudim(a)cc.uoc.gr]
Sys & Net Group - Computer Center, http://www.cc.uoc.gr
University of Crete - Greece
-------------- END PUBLIC INFO BLOCK --------------
On 27/07/2013 07:11 πμ, Matt Domsch wrote:
Greetings. I'm the Mirror Wrangler for the Fedora Project, and
author
of the tool we use to know which mirrors are up-to-date:
MirrorManager.
MirrorManager is experiencing several problems with your mirror.
1) the /fedora/fullfilelist file timestamp is July 16, so the mirror
has gotten hung somehow or is not complete updating as you might
expect.
2) by running report_mirror even when the above is failing (as you
are), MirrorManager believes your mirror is actually up-to-date, so
is trying to send users your way.
3) Our crawler is timing out after 2 hours, unable to complete the
HTTP HEAD calls necessary to judge that your mirror is up-to-date.
To try to alleviate #3, because I note you do have an rsync target for
the "Fedora Linux" content (but oddly, not for Fedora EPEL and Fedora
Secondary Arches which you already carry), I added the rsync URL to
your information in the MirrorManager database.
https://admin.fedoraproject.org/mirrormanager/host/488
For most mirrors, rsync directory listings are the fastest way to get
a full list of what content your mirror has. It falls back to doing
individual HTTP HEAD requests on a subset of the files, if rsync isn't
available, and FTP DIR calls if HTTP isn't available.
In your case, the crawler takes a relatively long time to retrieve the
directory listing from your mirror. You seem to not have HTTP
KeepAlives enabled, which, given as much content as you mirror, would
slow down the crawler enough to push you over the 2 hour timeout.
Now, for other mirrors serving rsync such as mirrors.us.kernel.org
shown here, we see times such as:
07/27/2013 12:29:05 AM Starting crawl
07/27/2013 12:29:05 AM scanning Category Fedora Linux
07/27/2013 12:31:26 AM rsync time: 0:01:14.965211
07/27/2013 12:34:21 AM scanning Category Fedora EPEL
07/27/2013 12:34:41 AM rsync time: 0:00:12.548085
07/27/2013 12:34:59 AM scanning Category Fedora Secondary Arches
07/27/2013 12:40:12 AM rsync time: 0:02:49.361520
07/27/2013 12:47:02 AM scanning Category Fedora Other
07/27/2013 12:47:13 AM rsync time: 0:00:06.161032
07/27/2013 12:57:27 AM Total directories: 5805
07/27/2013 12:57:27 AM Changed to up2date: 0
07/27/2013 12:57:27 AM Changed to not up2date: 0
07/27/2013 12:57:27 AM Unchanged: 5805
07/27/2013 12:57:27 AM Unknown disposition: 0
07/27/2013 12:57:27 AM New HostCategoryDirs created: 87
07/27/2013 12:57:27 AM HostCategoryDirs now deleted on the master, marked not up2date: 0
07/27/2013 12:57:27 AM Ending crawl
The whole process takes under 30 minutes.
I raise this because I know you do a good job running your mirror
generally, so this seems anomalous.
In the past, mirror admins have suggested reducing the value of
/proc/sys/vm/vfs_cache_pressure, from the default value 100, to a
lower number, causing the kernel to prefer to keep dentries when under
memory pressure:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
vfs_cache_pressure
------------------
Controls the tendency of the kernel to reclaim the memory which is
used for caching of directory and inode objects.
At the default value of vfs_cache_pressure=100 the kernel will attempt
to reclaim dentries and inodes at a "fair" rate with respect to
pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes
the kernel to prefer to retain dentry and inode caches. When
vfs_cache_pressure=0, the kernel will never reclaim dentries and
inodes due to memory pressure and this can easily lead to
out-of-memory conditions. Increasing vfs_cache_pressure beyond 100
causes the kernel to prefer to reclaim dentries and inodes.
Please take a look and see if a change is warranted on your side to
either or both enabling HTTP KeepAlives or adding rsync targets for
the other Fedora content carried (and then adding those URLs into the
MirrorManager database), or if you see different behaviour than I am.
Thanks,
Matt