Tagger & Packages interdependencies

[Toshio Kuratomi]

On Thu, Jan 26, 2012 at 02:57:03PM -0500, Luke Macken wrote:
> During the meeting today, Seth asked me to write up details on the
> interdependencies of the services that we're currently running on
> community01.dev.
> 
>     Indexer Cron Job
>         - Runs nightly, hits Koji, downloads RPMs using yum from the master
>           mirror.
>         - Creates a xapian database, and swaps out the old with the new
> 
>     Git repo updater cron job
>         - Queries koji for a list of builds since $LAST_RUN
>         - Does a git pull of all updated package repos
> 
>     Fedora Packages (https://community.dev.fedoraproject.org/packages)
>         - Queries the local xapain db for search queries and package info
>         - Queries bodhi, koji, pkgdb, bugzilla, via python-{fedora,bugzilla}
>         - Uses fedpkg to clones git repos
> 
>     Fedora Tagger (https://community.dev.fedoraproject.org/tagger)
>         - Queries the local xapain db for package info
>         - Uses postgres on db02.stg currently.
> 
> 
> So as far as production deployment goes, we could potentially have a
> separate box just for our indexer & git repos cron jobs, then we could
> rsync them over to our multiple app frontends. What do you guys think?
> 
A bunch of us talked about this on IRC few updates:

* xapian may be usable remotely -- if so we could treat as just a different
  kind of database server.
* The git checkouts (and xapian if it turns out it can't be used remotely)
  would be good candidates to start trying out glusterfs/hekafs/other cloud
  filesystems on.
* Packages is also using yum repodata for provides-requires and filelists.
  That's also a good candidate to store on glusterfs.
* If cloud filesystems are a bust, having a bapp-style box and then rsyncing
  is a possibility due to the relatively small size of the data so far.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20120126/9184a41d/attachment-0001.sig>