On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote:
On 07/01/2009 05:39 PM, Mike McGrath wrote:
> On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
>
>> On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrath<mmcgrath(a)redhat.com> wrote:
>>> On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
>>>
>>>> On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrath<mmcgrath(a)redhat.com>
wrote:
>>>>> I'm not sure if we have any memcached experience on the list but
I thought
>>>>> I'd ask. Can anyone explain this:
>>>>>
>>>>>
http://pastebin.ca/1481219
>>>>>
>>>>> Notice how memcached1 has a much higher hit rate and memcached2 has
a much
>>>>> lower hit rate?
>>>>>
>>>>
>>>> The time for memcached1 is 5x less than memcached2 being up. That can
>>>> have an effect on caching as right after a system comes up its rates
>>>> are usually much higher and then over time fall off (iirc). I think it
>>>> would take bringing both up at the same time to figure out if there is
>>>> a true disparity over caching.
>>>>
>>>
>>> I thought that exact same thing :)
>>>
>>> memcahed1:
>>> STAT uptime 9143
>>> STAT get_hits 311736
>>> STAT get_misses 11255
>>>
>>> memcached2:
>>> STAT uptime 9144
>>> STAT get_hits 49679
>>> STAT get_misses 11116
>>>
>>
>> Now that shows something not kosher. My guess is some app is not
>> talking to both? What apps use memcached for what?
>>
>
> I was just talking to ricky about this a bit in IRC. So here's the scoop.
>
> We've got mediawiki using memcached for a couple of things, including
> session data (which is weird and wrong but fast).
>
> The recent addition to the group is Fedora community, specifically in it's
> implementation of beaker. I'm going to get ahold of luke tomorrow to
> verify and test some stuff but I think this line in the config:
>
> beaker.cache.url = memcached1;memcached
>
> I'm not sure how beaker reads that, but I suspect it might be only sending
> information to memcached1 and ignoring memcached2 altogether. If this
> theory holds it'd explain why memcached1 not only has a higher request
> rate but also a higher hit rate because I suspect fedoracommunity requests
> some of the same info over and over again compared to the wiki which
> probably has a broader data pool it pulls from.
>
easy test: reverse that:
beaker.cache.url = memcached2;memcached1
and see if the cache hit ratio reverses itself.
-Toshio
So, I've been poking at this a little bit lately, and I'm thinking there are
some problems with the way Fedora Community is utilizing it's Beaker cache &
memcached.
The fact that something is wrong with the caching setup becomes obvious when
playing with the Bugzilla grid *should* cache the first 5 pages, and be very
snappy, as it is in my local instance. However, that is not the case, which
makes me think it's not hitting our memcached servers.
I wrote up a little test script on app1::
from beaker.cache import CacheManager
memcached1 = CacheManager(type='ext:memcached', url='memcached1',
lock_dir='.')
memcached2 = CacheManager(type='ext:memcached', url='memcached2',
lock_dir='.')
def createfunc(*args):
print "createfunc(%s)" % (args,)
def get_value(cache, value):
cache_1 = memcached1.get_cache(cache)
cache_2 = memcached2.get_cache(cache)
result1 = cache_1.get_value(key=value, createfunc=createfunc)
result2 = cache_2.get_value(key=value, createfunc=createfunc)
print "memcached1[%s][%s] = %s" % (cache, value, result1)
print "memcached2[%s][%s] = %s" % (cache, value, result2)
return result1, result2
get_value('fedoracommunity_alerts_global', 'today')
get_value('bodhi', 'dashboard_None')
Which produces::
memcached1[fedoracommunity_alerts_global][today] = None
memcached2[fedoracommunity_alerts_global][today] = None
memcached1[bodhi][dashboard_None] = None
memcached2[bodhi][dashboard_None] = None
I also tried using netcat to query for these by hand, to no avail.
So, it looks like we need to look a bit deeper into what is going on here.
Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere.
luke