On Thu, 26 Feb 2015 08:11:50 -0700
Kevin Fenzi <kevin(a)scrye.com> wrote:
We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users
and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it
starts getting really slow. Downloads go from 25M/s to 200k/sec and
sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days.
There is never any errors on the box, i/o, load and everything is
fine.
I looked this morning a bunch at options and adjusted the memory cache
down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we
can do to fix this once and for all (if these changes don't do so).
So, I finally tracked down the issues with smp mode on squid. ;)
It's two slightly different bugs on Fedora and RHEL.
Fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1176318
RHEL:
https://bugzilla.redhat.com/show_bug.cgi?id=1102842
In Fedora squid is set to use a /var/run/squid dir for pid/ipc state,
but it doesn't have a tmpfiles.d config to make that dir on boot, so it
only works until you reboot.
In RHEL, they compiled it to use /var/run/squid.pid for the pid file,
and the ipc state files can't be written to /var/run/ as squid user, so
they nicely silently fail to talk to each other and no one binds to the
port and nothing works.
We can work around this issue for now on RHEL with:
* Set
pid_filename /var/run/squid/squid.pid
in the squid.conf.
* Add a selinux local policy or set permissive mode:
#============= squid_t ==============
allow squid_t var_run_t:sock_file { read write create };
* Add a /etc/tmpfiles.d/squid.conf with:
D /var/run/squid 0755 squid squid -
So, I guess we can wait and see if the slowdown happens again, and if
so, try the above to enable smp mode and see if it helps any?
kevin