fedmsg/datanommer/datagrepper upgrade in production

[Ian Weller]

On Mon, Oct 14, 2013 at 03:12:03PM -0600, Kevin Fenzi wrote:
> On Sat, 12 Oct 2013 16:52:40 -0500
> Ian Weller <ian at ianweller.org> wrote:
> 
> > Here is my plan for upgrading fedmsg/datanommer/datagrepper. I'm not
> > sure it'll be able to get done before freeze on Tuesday at this
> > point... but I'd like to try to. I need somebody else to tell me if
> > this is crazy.
> 
> You're crazy! :) 
> 
> just kidding. 
> 
> > Everything is working as expected in staging. Production has two
> > servers, which adds a little bit more difficulty.
> > 
> > stage 0: preliminary things
> > - need a sysadmin-db to create a datagrepper database
> 
> I think toshio has this set for you now. 
> 
> > - need to know domain/port for a memcached server
> 
> Either memcached03 or memcached04. 
> 
> > - need shared storage on datagrepper{01,02} in
> >   /var/www/datagrepper_output (only datagrepper01 will be writing to
> > it, but both will need to serve it via httpd)
> 
> I've setup a shared gluster volume for this. 
>  
> > stage 1: upgrade fedmsg and datanommer on busgateway01
> > - yum update fedmsg python-datanommer-{models,consumer}
> > - run the two alembic upgrades (which need to be run in reverse order)
> > - restart all the fedmsg services
> > - verify everything is working:
> >   - fedmsg irc is spitting out messages
> >   - datanommer database is receiving new messages (verify with
> >     datanommer-latest)
> 
> Sounds good. 
> 
> > stage 2: upgrade datagrepper on datagrepper{01,02}
> > - add datagrepper::runner to datagrepper01 manifest
> > - merge datagrepper from modules-staging into modules
> > - yum update fedmsg python-datanommer-models datagrepper
> > - create tables in datagrepper database
> > - run-puppet nowait
> 
> ok. 
>  
> > stage 3: upgrade fedmsg on the rest of production
> > - a sysadmin-main (presumably) will need to do this
> 
> I think we are already updated from the mass updates last week, but if
> not we can do this.

Yes, fedmsg is 0.7.0 across the board.

Status at end of day today:
- datanommer 0.6.0 is in production
- datagrepper 0.2.1 is in production

Various bugfixes and enhancements happened. My favorite: each message
now has a permalink:
  https://apps.fedoraproject.org/datagrepper/id?id=2013-a441f82b-9a42-41c7-a05e-e283c6cdf560

Logging in doesn't work since Flask-OpenID's way of handling stateless
auth stores data in /tmp/flask-openid on the system which doesn't work
when you have multiple servers. Planning to move to the FAS-OpenID
plugin for Flask. So the job queue system can't even be used yet -- but
when I get that fixed, it's ready :)

Additionally there are some bugs with overreaching exception handlers on
long-running jobs, so that'll be taken care of as well.

Fortunately all the things that have bugs are new features, and
everything that used to work still does. :)

datagrepper 0.2.2 should be out the door this week, and not long after
we should see a change request on this list (since freeze starts
tomorrow).

-- 
Ian Weller <ian at ianweller.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20131014/c16697ce/attachment-0001.sig>