On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote:
> On 07/01/2009 05:39 PM, Mike McGrath wrote:
> > On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
> >
> >> On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrath<mmcgrath(a)redhat.com>
wrote:
> >>> On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
> >>>
> >>>> On Wed, Jul 1, 2009 at 3:33 PM, Mike
McGrath<mmcgrath(a)redhat.com> wrote:
> >>>>> I'm not sure if we have any memcached experience on the
list but I thought
> >>>>> I'd ask. Can anyone explain this:
> >>>>>
> >>>>>
http://pastebin.ca/1481219
> >>>>>
> >>>>> Notice how memcached1 has a much higher hit rate and memcached2
has a much
> >>>>> lower hit rate?
> >>>>>
> >>>>
> >>>> The time for memcached1 is 5x less than memcached2 being up. That
can
> >>>> have an effect on caching as right after a system comes up its
rates
> >>>> are usually much higher and then over time fall off (iirc). I think
it
> >>>> would take bringing both up at the same time to figure out if there
is
> >>>> a true disparity over caching.
> >>>>
> >>>
> >>> I thought that exact same thing :)
> >>>
> >>> memcahed1:
> >>> STAT uptime 9143
> >>> STAT get_hits 311736
> >>> STAT get_misses 11255
> >>>
> >>> memcached2:
> >>> STAT uptime 9144
> >>> STAT get_hits 49679
> >>> STAT get_misses 11116
> >>>
> >>
> >> Now that shows something not kosher. My guess is some app is not
> >> talking to both? What apps use memcached for what?
> >>
> >
> > I was just talking to ricky about this a bit in IRC. So here's the scoop.
> >
> > We've got mediawiki using memcached for a couple of things, including
> > session data (which is weird and wrong but fast).
> >
> > The recent addition to the group is Fedora community, specifically in it's
> > implementation of beaker. I'm going to get ahold of luke tomorrow to
> > verify and test some stuff but I think this line in the config:
> >
> > beaker.cache.url = memcached1;memcached
> >
> > I'm not sure how beaker reads that, but I suspect it might be only sending
> > information to memcached1 and ignoring memcached2 altogether. If this
> > theory holds it'd explain why memcached1 not only has a higher request
> > rate but also a higher hit rate because I suspect fedoracommunity requests
> > some of the same info over and over again compared to the wiki which
> > probably has a broader data pool it pulls from.
> >
> easy test: reverse that:
>
> beaker.cache.url = memcached2;memcached1
>
> and see if the cache hit ratio reverses itself.
>
> -Toshio
So, I've been poking at this a little bit lately, and I'm thinking there are
some problems with the way Fedora Community is utilizing it's Beaker cache &
memcached.
The fact that something is wrong with the caching setup becomes obvious when
playing with the Bugzilla grid *should* cache the first 5 pages, and be very
snappy, as it is in my local instance. However, that is not the case, which
makes me think it's not hitting our memcached servers.
I wrote up a little test script on app1::
from beaker.cache import CacheManager
memcached1 = CacheManager(type='ext:memcached', url='memcached1',
lock_dir='.')
memcached2 = CacheManager(type='ext:memcached', url='memcached2',
lock_dir='.')
def createfunc(*args):
print "createfunc(%s)" % (args,)
def get_value(cache, value):
cache_1 = memcached1.get_cache(cache)
cache_2 = memcached2.get_cache(cache)
result1 = cache_1.get_value(key=value, createfunc=createfunc)
result2 = cache_2.get_value(key=value, createfunc=createfunc)
print "memcached1[%s][%s] = %s" % (cache, value, result1)
print "memcached2[%s][%s] = %s" % (cache, value, result2)
return result1, result2
get_value('fedoracommunity_alerts_global', 'today')
get_value('bodhi', 'dashboard_None')
Which produces::
memcached1[fedoracommunity_alerts_global][today] = None
memcached2[fedoracommunity_alerts_global][today] = None
memcached1[bodhi][dashboard_None] = None
memcached2[bodhi][dashboard_None] = None
I also tried using netcat to query for these by hand, to no avail.
So, it looks like we need to look a bit deeper into what is going on here.
Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere.
I did a little more testing with this today.
On app1 I was unable to get fedoracommunity to put any values in
memcached, and I was not seeing any errors either. I tried this with
both the mod_wsgi deployment, and running paster locally. I then ran
`curl
to trigger the
cache, and then ran the script I mentioned earlier to see if I could
retrieve those values.
I was able to reproduce this failure on a local RHEL5 VM. However,
after upgrading to python-beaker-1.4 *and* restarting memcached, my
script *worked*. (the app just silently returned None for all requests
if I didn't restart memcached after upgrading Beaker)
Looking at the ChangeLog for Beaker 1.4, I'm thinking this could have
done the trick:
* Fixed bug with CacheMiddleware overwriting configuration with default
arguments despite prior setting.
It'll be tough to test this new package in staging, as earlier today I
couldn't get any of the cached apps to load due to koji/fas problems in
staging. When we deploy this production, we'll need to make sure to
restart our memcache daemons.
luke