resultsdb roadmap

[Róman Joost]

Dear Tim,

On Thu, Feb 12, 2015 at 10:31:23AM +0100, Tim Flink wrote:
> > [...]
> > I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
> > test result translates to an entry in Resultsdb as an outcome (PASSED,
> > FAILED). Yet the question for the developer as to why it failed is
> > important.
> 
> I don't think that desire is unique to rpmgrill and something that we
> decided to leave out of the initial Taskotron system.
> 
> My initial thought is that that a discrete state (PASS, FAIL,
> NEEDS_INSPECTION etc.) a one-line-or-so summary and link to output
> from rpmgrill would work well. Are there other things that you'd like
> to see in terms of output?
From what Kamil pointed out, there is no state missing.

> > 1. Ed Santiago is currently running RPMGrill here:
> > 
> >     http://rpmgrill-fc20.edsantiago.com:5000/recent
> > 
> >   which in terms of presenting results to the user provides a
> > different experience. The results are grouped by test and provide
> > more insight as to why a test failed. Test results in Resultsdb
> > currently seem to have only one outcome and technical details are
> > left on the build master.
> 
> This was less of a desired final outcome for us than it was the easiest
> way to get started.
> 
> I think that our current method of having gigantic text-only log files
> is far from optimal (and something that we did years ago in AutoQA
> before making the logs more easily accessible) but we haven't improved
> the log accessibility/readability in taskotron yet.

Good to know. I guess this is something to discuss and work on?


> >   How does a representation like RPMGrill translate into Resultsdb?
> > Are there plans/ideas on how to provide a more diversified result
> >   representation in Resultsdb (e.g. error reason, hint on how to solve
> >   an error).
> 
> I think that having rpmgrill output static html would likely be the
> best approach here unless you're planning to continue supporting the
> webapp that you linked to (and from previous conversations, I'm under
> the impression that was never intended to be a long-term thing).

If we'd have a way to turn the TAP output into details retrievable from
resultsdb, it'll be better than rpmgrill printing HTML?


> > 2. Are there plans/ideas about implementing a (fulltext-) search over
> >   results?
> 
> Nothing written down, no but that's been on my wishlist since before
> Taskotron really got started. Unless there's more demand for the
> fulltext indexing than I think there is, we'd probably start with
> something less complicated like indexing the summary statements.
> 
> I'm open to other ideas, though. Just concerned about what fulltext
> indexing of all logs might entail.

True. We've had quite big logs if some big packages contain a lot of
regressions. Good to know that it is on the wish list.

> > 3. Are there plans/ideas on implementing a waiver mechanism?
> 
> Yeah, it'll have to happen before we start gating any
> rawhide/branched/stable builds or updates for Fedora. We don't have a
> design yet but it'll happen at some point.

Ok. This also seems like something we might be able to contribute to.
Perhaps better to discuss this in another thread?

> > [1] - https://git.fedorahosted.org/git/rpmgrill.git
> > 
> > PS: Yes - I'm aware that there is already a bitbucket repository of
> > rpmgrill for Taskotron, but have only had a quick glimpse at it.
> 
> Someone was planning to work on getting rpmgrill working in taskotron
> and that bitbucket repo was meant to hold the task that's executed in
> taskotron.
> 
> From our end, that is being tracked as:
> 
> https://phab.qadevel.cloud.fedoraproject.org/T281
> 
> Let us know if you have any other questions or concerns.

Thanks. I'll have a look at the tasks. I don't want to make any
promises, but I hope we'll find some time to get the rpmgrill tasks in a
'done' state.

Thanks and Regards,
-- 
Róman Joost
Software Engineer, HSS - Software Engineering and Development (Brisbane)
email: rjoost at redhat.com | tz: UTC+10
irc: #hss #rpmdiff
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20150216/301cf5f6/attachment.sig>