On Mon, 2017-03-06 at 12:00 -0500, Kamil Paral wrote:
> > If we implement a 'scenario' key, we can then just teach Bodhi to look
> > for that and combine it with the item, then maybe if it can't find one,
> > treat 'testname-arch' as the scenario (as a fallback for old results,
>
> We don't have that many tests yet, and all of them are basically
> under our control, so if we end up doing this, I think we should
> simply push a patch to all of them (should be fairly easy) and avoid
> any legacy handling.
Well, the case I was thinking of is specifically Bodhi: even if we
adjust things so that all results *going forward* have a scenario key,
*existing* results will not, and often updates sit around in Bodhi for
a long time before going stable. If we make Bodhi only work if the
result has a 'scenario' key, it'll either break or not be able to
display test results for older updates, I think. (And I think Bodhi
shows results even for stable updates, so really, don't we need it to
be able to handle the existing results for all non-EOL releases?)
I was under the impression that Bodhi would consider the scenario key only if present,
otherwise consider all testcase+item-based results unique. So it should work both for
future and past results (with the slight exception of disregarding arch-specific results).
Am I missing something? Or are you hinting at the arch-specific results (depcheck)?
> > results from Taskotron until we teach it to write 'scenario' items
into
> > its results, and a general fallback for cases where 'scenario' is
> > missing for some reason). That could be the general convention, I
> > guess: first look for a 'scenario' key, otherwise try and construct
one
> > from very commonly available keys.
>
> I'd like to avoid the guessing game here. The system will be reliable
> only if it is obvious and simple. If scenario turns out to work well
> for us, let's go all the way. If there's no scenario, results should
> be unique when identified by testcase name + item. If that's not
> enough, scenario needs to be there, period. Of course people can
> submit whatever results they want, but once we want to employ them in
> gating, the results need to adhere to the rules. We're open source, I
> believe we should send patches to fix the task reporting if needed,
> rather than extending the guessing algorithms in our consumers.
I agree we should implement the 'scenario' reporting for all systems
ASAP (should I send a patch for taskotron, if I can figure it out?),
but I don't think that's actually sufficient, for the above reason.
I can definitely take care of adjusting the core taskotron once we decide to go full steam
ahead. From tasks, it seems only depcheck/rpmdeplint will be affected (nothing else uses
any extradata fields). Of course ideally we should also document this in libtaskotron docs
and/or resultsdb_conventions. But I'd leave that until we test it out and realize
whether it's working correctly or not.