Package Update Acceptance Test Plan - final call

[Kamil Paral]

----- "James Laska" <jlaska at redhat.com> wrote:

> > = Test Priority =
> > 
> > 1. Do we want to have the strict requirement that introspection and
> > advisory tests may be started only after all mandatory tests have
> > finished? In other words, do we want to have the testing sequence
> > like this:
> > 
> > -> mandatory tests -> (wait for completion) -> introspection + 
> > advisory tests
> > 
> > or this:
> > 
> > -> mandatory + introspection + advisory tests (no particular order)
> > 
> > Reason for (the first sequence): It prioritizes the most important
> > tests, they will be finished sooner. It can save resources in case
> > some mandatory test fails and we won't run any subsequent tests
> > (see question 2).
> > Reason against: More difficult to implement. Less effective from
> > overall performance view.
> 
> My preference would be the path of least resistance, which seems like
> the second approach.  Should we suffer from slow mandatory test
> results
> due to introspection and advisory tests being run first, we have test
> scheduling options to explore to address the problem.  Whether it's
> prioritized/weighted jobs or grouping the tests into the three
> buckets
> I'm not sure yet ... but I feel like that's something we can address
> in
> the future.

Same opinion here, thanks, document updated.

> 
> > = Test Pass/Fail Criteria =
> > 
> > 2. Do we want to continue executing tests when some mandatory
> > test fails?
> > Reason for: We will have more results, maybe the maintainer will
> > have a look at all the results and may fix more issues at once
> > (not only the failed mandatory test).
> > Reason against: It wastes resources - the update will not be
> > accepted anyway. When a mandatory test fails (like installability
> > or repo sanity), many follow-up tests may fail because of that,
> > so they may not produce interesting output anyway.
> 
> Let's make our lives simpler at first ... schedule all the tests. 
> Even
> if mandatory results fail, having the introspection and advisory
> results
> available for later review or comparison will be helpful.
> 
> That said, as soon as the mandatory tests have failed, we can
> initiate
> whatever process needs to happen to ensure that package update is not
> accepted.  However, the other tests would continue to run and report
> results into the appropriate place.

Again I fully agree. Updated.

> 
> > 3. We should complement requirement "all tests have finished" with
> > a definition what happens if some test crashes. It is obvious that
> > we can't accept package for which some substantial (read mandatory
> > or introspection) test has crashed. So we can add a requirement
> > like this:
> > "all mandatory and introspection tests have finished cleanly (no
> > test crashes)"
> > The question remains - what about advisory tests? Do we require
> that
> > they also don't crash? Or even if some of them crashes it won't be
> > an obstacle for accepting the update?
> > Reason for: Advisory tests are not important for accepting the
> > update, a test crash should not cause rejection.
> > Reason against: Some information won't be available. It could
> happen
> > that those information would cause the maintainer to withdraw/renew
> > the updated package.
> 
> Long-term, all these tests need to run and results presented to the
> maintainer.  Short-term while we are piloting this effort, I don't
> have
> a problem allowing a build to proceed once mandatory tests have
> completed, but advisory/introspection tests failed or aborted.

I would prefer to cover longer-term objectives in the document
rather then pilot stage. I have added a requirement:
"# all mandatory and introspection tests have finished cleanly 
(not crashed or otherwise aborted)"

> 
> > = Introspection tests =
> > 
> > 4. Rpmlint - "no errors present" vs "no new errors present": It
> > is obvious we have to provide an option for package maintainers to
> > whitelist some rpmlint messages. The actual implementation of the
> > whitelist is still to be discovered, but that doesn't matter, it
> > will be there. In this respect is seems to me that "no new errors"
> > requirement has no benefit over "no errors" requirement, because
> > the latter one does everything the prior one and even more. When
> > it is possible to whitelist errors then there's no reason for us
> > to allow any errors in the output.
> > Implementation note: "no new errors" is more an rpmguard's task
> > than rpmlint's. We can implement that as an rpmguard check and put
> > it to introspection tests, that's possible. But it's not needed if
> > we agree on "no errors" requirement.
> 
> "No new errors" seems like an achievable short-term approach to add
> value while not overloading the maintainer with a potentially large
> list
> of rpmlint failures (/me thinking kernel) that they haven't been
> tracking since the package was initially incorporated into Fedora. 
> Down
> the road, I agree a whitelist mechanism would be ideal (as well as a
> shortcut to file a bug against rpmlint).

The whitelist mechanism is surely a must-have, otherwise we would
end up impaled on a stakes, quartered and burnt alive :)

"No new errors" could work for some packages, but it wouldn't work
for many others, especially when a version number is mentioned in the
message - it will change for every release, so we would see it
as a new error anyway. Kernel package is a nice example of this.
A few references:

https://fedorahosted.org/pipermail/autoqa-results/2010-April/013921.html
https://fedorahosted.org/pipermail/autoqa-results/2010-April/016838.html
https://fedorahosted.org/pipermail/autoqa-results/2010-April/016436.html

I believe when AutoQA goes live it will not be enforcing anything
for a long time, just providing additional info. There will be a
lot of time when maintainers may solve these issues. I wouldn't
worry too much, we can also disable some test if it seems to
be causing troubles, until it is resolved.

> 
> > =  Responsibilities and permissions =
> > 
> > 5. "Introspection tests failures may be waived by:"
> > 5a. Should Release Engineering be there?
> 
> No, let's just go with below.  And anyone can demonstrate required
> test
> skills and join the group, including release engineers.
> 
> > 5b. Should we add the new proventesters group?
> 
> Oh good catch, yes!

I have removed RelEng and also QA (on jkeating's suggestion) and
replaced them with proventesters team. Or would it be better
to leave the whole QA team in?

> 
> Hope this helps,
> James
> 
> -- 
> test mailing list
> test at lists.fedoraproject.org
> To unsubscribe: 
> https://admin.fedoraproject.org/mailman/listinfo/test