sorry for the delay.
On Wed, Mar 25, 2015 at 08:51:29AM -0400, Kamil Paral wrote:
> That's exactly why I thought about runtask. To be more
> currently working on rpmgrill. It ships it's own fetch-build script
> which is currently tied to Fedoras Koji.
Just a note, IIRC you don't need just all the rpms, but also the build log for
certain subtests. That's currently not provided by our koji directive, and I'm not
sure if we'll want to add it or not. In such cases, you might need to implement some
bits on your own (e.g. fetching the build log).
Maybe talking about just builds is wrong, but perhaps a better term
would be build artefacts. In order to accommodate scanners which not
only check the build itself but logs producing the build, I'd think that
logs given the scanners would make a good addition.
Perhaps better to be discussed separately tho.
> IMHO rpmgrill shouldn't be concerned about where the builds come from.
> If I can use 'runtask' to fetch the builds, let rpmgrill handle the
> analysis and report back to bodhi or any other similar system that would
> be very beneficial. Furthermore that can run easily in a Jenkins and the
> exit status would provide a way for the CI to figure if the package is
> not shipping with nasty problems.
I see, so you want to use the exit status as a super-simple way to distinguishing
pass/fail in Jenkins, instead of providing some more advanced structure that the CI
understands (junit, etc). I think that's a valid request.
I rather thought to be consistent the way typical test runner behave. I
also know that some of them allow you to configure the behaviour since
there are instances you want to check PASS or FAIL based on the logs.
> > [...] NEEDS_INSPECTION case
I think NEEDS_INSPECTION would also be considered as failed, and the exit code would be a
predefined value (e.g. 100), so that we can still distinguish execution errors from a task
> > 2. Some tasks do not check just a single item. For example,
> > upgradepath check everything that is currently proposed for stable or
> > testing updates. They can return a dozen test results. What should happen
> > when there are mixed results?
> I'd assume to default to the worst result. [...]
We had a long discussion with Josef about this, and our conclusion is that it should be
the script author who is in business of deciding what the overall result is. The easiest
way to implement this seems to be to let the author define one additional result in TAP
(the very last result in TAP), according to any logic he or she needs, and we will
consider this last result to be the decision of what exit code to return.
what if the script author does not provide this additional
step? What would be the fall back?
So, if you have just a single result in TAP, it's the one that is
used for exit code decision. If you have multiple of them, the last one is used.
This means you will *have to* return at least a basic TAP from rpmgrill (there's a
perl library for that), or from the wrapper around rpmgrill (e.g. converting its json to
TAP). But TAP itself is extremely simple, so it shouldn't be a problem doing it even
manually, i.e. in tests written in bash (we don't support this yet, but plan it in the
future). This is the simplest TAP we can consume (I just discovered a bug here, but
I'll fix it):
TAP version 13
And this is the simplest version that makes sense if you want to report to resultsdb:
TAP version 13
> > One further thought, instead of implementing this by
default, we can
> > also create a new directive or variable which will allow task creators
> > to set the exit code, according to TAP or any other logic they want.
> That sounds like a good idea as well. Based on that, what we could also
> do is, add an argument to runtask like:
> which means if the result of the runtask is not PASSED or INFO it'll
> exit with 1. Less flexible than your idea, but perhaps less work
I like it. By using this cmdline option, you would trigger the behavior I described
earlier (using the last TAP result as an exit code for the runner).
So, in summary, for your rpmgrill in Jenkins use case, you would run it with --make-fatal
and you would make sure that the last result in TAP (or the only result in TAP, if you
have just one) reflects the exit code you want runtask to return (FAILED and
NEEDS_INSPECTION means non-zero). Does that sound reasonable (to everyone)?
good to me. I only wonder if we simply fall back to the current
state (exit 0) if the precondition (last tap result outcome) and
--make-fatal is not given.
Software Engineer, PnT DevOps - Developer (Brisbane)
email: rjoost(a)redhat.com | tz: UTC+10 | GPG ID: 0xBE2B559D at pgp.mit.edu
irc: #pnt-devops #rpmdiff