Planning: Fedmsg Integration

[Ralph Bean]

On Fri, Dec 12, 2014 at 01:01:43PM -0700, Tim Flink wrote:
> On Fri, 12 Dec 2014 11:59:30 -0500
> Ralph Bean <rbean at redhat.com> wrote:
> 
> > On Thu, Dec 11, 2014 at 11:10:55PM -0700, Tim Flink wrote:
> > > Almost forgot about this, even though I was talking about it earlier
> > > today.
> > > 
> > > The current plan for bodhi2, as I understand it, is shooting for
> > > production deployment in January or February. This affects us
> > > because we will no longer be submitting bodhi comments as a
> > > feedback mechanism for completed tasks.
> > > 
> > > Before this happens, we will need to enhance resultsdb-backend to
> > > send out fedmsgs that can be processed by fedmsg-notifier (fmn [1])
> > > 
> > > [1] https://apps.fedoraproject.org/notifications/
> > > 
> > > Outside of the technical requirements here, we need to figure out
> > > whether or not receiving notifications will be optional for
> > > maintainers or not. This will require some conversation with FESCo
> > > as it's more of a policy thing but will need driving from our end.
> > 
> > IMHO, it should be optional, but opt-out.  (So we would need to
> > include it in the new defaults that go out with the next release of
> > fmn).
> 
> Normally, I'd agree with you but if we're going to start gating updates
> or builds based on automated check results, that changes things a bit.
> 
> Then again, it's not like that's going to happen right away so it's
> probably something we can deal with later.

Cool.  It would just easier on the FMN end to make everything opt-out
instead of having some rules be special-cased as mandatory.  Can do,
though.

> > > The more technical bits that I'm aware of at this point are:
> > >  - Decide on a schema for the messages we emit
> > >    * org.fedoraproject.taskotron.result.package.foo-1.2-3.fc99 etc.
> > >      (not even sure that's a valid message)
> > 
> > I'd rather not have a variable in the package topic -- that will give
> > us an enormous number of total unique topic in the datagrepper
> > database.
> > 
> > How about something like:
> > * org.fedoraproject.taskotron.result.new
> > 
> > ..and embed the update, build, and/or package name in the body of the
> > message.
> 
> Yeah, that would work for me. I've not spent much time looking into
> fedmsg structure yet and the one I listed was just the first thing I
> thought of.
> 
> If we did embed the task name and other data in the message itself,
> would fmn be able to differentiate between results for different
> affected packages? I don't think that anyone wants emails for every
> result from taskotron - I don't even have that set up.

Agreed on the undifferentiated emails point.  FMN would need to be able
to distinguish.  The way this is currently done with other services is by
relying on a call to ``fedmsg.meta.msg2packages(msg, **config)`` on
the recipient side.  The implementation for that function is
plugin-based.  We have a plugin for every service that publishes
messages on our bus and we would need to write a new one for Taskotron
messages.  Those are all here
https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure

The implementation varies from service to service, but it would be
easiest if taskotron could just embed a list of the affected packages
in the original message, and then fedmsg.meta.msg2packages could just
extract that list and return it.  If for some reason the list cannot
be embedded in the original message, you could embed the list of
updates and then the fedmsg.meta plugin could query bodhi for the list
of packages to which those updates relate.

> Another thing we're going to have to figure out is how to send
> notifications for depcheck. Update title isn't always consistent, so it
> makes for a poor method to create notification rules by. We could start
> posting results per-package but that is going to require some new code
> to figure out what all would be affected if a build was witheld.

Eh, I see what you mean.  If you just sent a single notification:
'org.fedoraproject.taskotron.result.new' with the body noting that
the check was 'depcheck' and including the list of updates
(and their packages?) that were fed through... then everyone who had
an update in that check would get a notification even if theirs wasn't
the package responsible for the depcheck failure.

Perhaps send one message just indicating that the depcheck run result
was stored, and then send N subsequent failure message for each
package that seems to be at fault in some way (does depcheck know that
kind of causal stuff or does it only know that 'something is wrong
somewhere'?).

> > >  - Figure out what information should be in the emitted messages
> > > 
> > >  - Figure out where the messages should be emitted from and
> > > add/modify code to get that happening (resultsdb-backend makes the
> > > most sense to me right now but am open to other ideas)

resultsdb-backend makes sense to me.

> > >  - Get certs for the emitting machines

There's a script for generating them.  Easy to do -- just ask whenever
you'd like them created and pushed out.

> > >  - Start emitting messages in staging
> > > 
> > > 
> > > As a reward when it's done:
> > > 
> > >  - rip out the bodhi comment code from litaskotron and burn it over
> > > a bonfire while we toast marshmallows for s'mores

I'll bring the cider!

> > > I'd like to have this ready for deployment in staging by early
> > > January, so that it can run for a while before we move it into
> > > production but I'm not sure if that's compatible with vacation
> > > schedules.
> > 
> > For what its worth, bodhi2 (unreleased) currently just queries the
> > resultsdb JSON API to figure out how it should mark or block
> > updates... all without fedmsg.  fedmsg just gives us a nice
> > notification layer for end users.
> 
> Yeah, I'm looking forward to that. The code we're currently using to
> make bodhi comments is ... not well liked or a great piece of code but
> it was the best reasonable solution we had at the time.
> 
> > There is another layer of integration that we've not yet begun to stub
> > out in bodhi2, which is to publish a message when its backend begins a
> > mash.  We have been hoping that taskotron could listen for that new
> > message and begin a depcheck run against all the packages in the koji
> > tag that bodhi is about to mash -- and then later, publish a message
> > when that check is done.  Bodhi would block, waiting for that
> > response, and then only proceed with the mash if that depcheck run
> > passes.
> 
> Yeah, we've talked about doing that in the past. Most of the stuff to
> do that is in place for Taskotron but it would probably require some
> tweaking to return depcheck results in a useful format for mash -
> "which builds in this set are OK to push?". Has there been any farther
> discussion on when we'd want to start doing that and how we'd handle
> the nitty-gritty of actually doing it (overriding bad results, exact
> policy on what gets pushed and what doesn't etc.)?

Yeah, there's some code in there now that let packagers override it
at the update level.  By default, a handful of checks are required to
pass but those can be changed -- even without admin privileges: a
packager can alter the taskotron requirements on her update.
Furthermore, admins can go in and remove certain requirement for an
update if they feel they know better than our systems.

Currently, the masher just removes any update from the mash that
hasn't met all of its stated requirements from the get-go.  Triggering
an extra depcheck run at the beginning of the mash would just ensure
that those results are as up-to-date as possible.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20141215/bfddd35f/attachment.sig>