wiki madness

[Matt Domsch]

On Fri, Nov 02, 2007 at 11:06:11AM -0700, Toshio Kuratomi wrote:
> Chuck Anderson wrote:
> 
> >Won't there be performance problems with a TurboGears-based wiki?  I 
> >thought MirrorManager was having issues with TG performance and had to 
> >enable form-data caching to get acceptable performance at the cost of 
> >possibly stale data.  I don't know the details behind it, but that was 
> >the reason I was given for why when you edit forms in MM it sometimes 
> >returns old pre-edit field values.
> >
> We might have performance issues but I'm confident they'll be different 
> performance issues than we're currently experiencing ;-)
> 
> The issues we're running into with moin right now are largely caused by 
> Moin's philosophy of having to run off the filesystem, not a db.  This 
> means 1) we're unable to spread the load among multiple different app 
> servers so we are constrained to a single server's memory and CPU 
> resources, 2) it makes multiple views of data much harder than it needs 
> to (in the subscription list case, Moin has to walk the filesystem, 
> finding each user's prefs file, parsing it for a watchlist, if the 
> watchlist exists, checking if the page and page categories are in that 
> watchlist, and finally being able to send the notification.  With a db, 
> we'd have a separate table for the watchlist and have indexes for the 
> userid and the pagename.  Searching for a page wouldn't have to open a 
> file for every single one of our users, instead it would access a single 
> table and pull out the users which were in the watchlist.)
> 
> With MirrorManager I know we've had memory and db query speed issues 
> trying to serve the mirrorlist directly from the TG app.  I wasn't aware 
> that mirrormanager was having trouble keeping up with it's management 
> functionality, Matt is that still true or is caching a leftover from 
> when the two functions were combined?

I'm sure it's still true, it predated having any mirrorlist
functionality at all.

The short story is, TG (well, SQLObject) either caches data very
aggresively, so you can see stale data on changes, or not at all, so
each field read in each row results in a DB query.  Even with
object.sync() calls scattered through the UI actions like I did,
leaving caching enabled we do still see stale data on occasion.
Disabling caching, generating the UI pages or certainly the publiclist
pages takes _forever_, hundreds of thousands of small DB queries.

Maybe SQLAlchemy has a better caching mechanism, I don't know.

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux