Moving Taskotron to Production

Tim Flink tflink at redhat.com
Tue Jul 22 19:21:45 UTC 2014


On Tue, 22 Jul 2014 12:12:36 -0600
Kevin Fenzi <kevin at scrye.com> wrote:

> On Tue, 22 Jul 2014 10:56:12 -0600
> Tim Flink <tflink at redhat.com> wrote:
> 
> > Now that Taskotron staging is up and running, I'm not sure what's
> > generally required for services before moving to production.
> > 
> > I'm still working out a few kinks in the initial production systems
> > but I think that the only big thing left is the proxy settings,
> > which I'm leaving alone for the moment.
> > 
> > Is there any review process or checklist for moving things from stg
> > to prod?
> 
> http://infrastructure.fedoraproject.org/infra/docs/requestforresources.txt

Thanks, I don't think that I would have looked in that doc.

> I'd like monitoring added. What should we monitor to make sure it's
> working as expected? 

It depends on how complicated we want to get, I suppose. The most basic
things would be to check to make sure resultsdb, resultsdb_frontend and
buildbot are responding to http but there are json apis for
resultsdb and buildbot which would give more details. I'd like to
monitor free disk space if that's not already done by default.

Are most other services monitored with a GET to a url to see if they're
up?

> Also, do we need any SOP's written up? Anything special in
> stopping/starting things? updates order or anything?

Yeah, the triggered jobs need to be buffered during downtime for
playback after everything comes up. I've always used my local system
for that but I'm not sure what the best choice is for infra.

> Also, I need to make sure we have backups going on the db server at
> least. Anything else that needs backing up?

/home/buildmaster/master on the taskotron server (taskotron01.qa for
production) should be backed up as that's where all the logs are
stored. Everything else that needs to be backed up is in the databases.

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20140722/3068917d/attachment.sig>


More information about the infrastructure mailing list