As I've been working on the email reduction feature, I've
been rather
frustrated with how difficult it is to test AutoQA as a framework.
I've
written scripts to make testing depcheck (minus commenting and build
gathering) easier but I have yet to find a reasonable way to exercise
the whole AutoQA stack other than running jobs on my dev system with
comments disabled and hope that they finish before production does.
Before I go farther, I realize that what I'm discussing would be a
non-trivial amount of work and I'm kind of thinking that it would be a
little too much for 0.5.0. Then again, more testing could help improve
the release. Thoughts on that part would be welcome.
I can think of two methods that we could use to make testing easier
and
start down the road of retiring our mascot:
- Refactor bodhi_utils and koji_utils so that they can be stubbed out
in testing
- Find an alternative to the production instances of koji and bodhi
Either way, setting up the conditions for self-tests (making mock
rpms,
setting up the metadata, creating repos) is going to be a decent
amount
of work and will likely end up with fragile, high maintenance tests if
we go beyond basic smoke tests.
There were concerns over the refactoring I proposed for koji_utils a
while back so I assume that isn't the way we want to proceed for now.
I looked into setting up test instances of bodhi and koji but I'm
rather
intimidated by the amount of maintenance and hacking that would be
required to induce the conditions that we would need for more complete
testing on a regular basis.
With this in mind, I started hacking at a mockup for replacing bodhi
and
koji for testing purposes that does nothing more than implement the
parts of those interfaces that we're using in AutoQA. It took a little
while to reverse engineer the interfaces to bodhi and koji but I do
have
some code that is able to fool the koji client with hard-coded results
(haven't tried it with AutoQA yet nor have I gotten very far on
mocking
up bodhi). I can send out code if anyone wants to see it (kind of ugly
ATM, though).
While finishing the mockup would also be non-trivial, I think that it
will be more manageable and flexible in the long run in addition to
being better suited to our needs since we can add in testability
features as needed without having to bug other teams about features
that
other people may never use.
Anyhow, thoughts on the concepts? On the timing (try to have some for
0.5.0 or not)? Anything that I didn't consider?
Tim
Tim, that sounds great, thank you for this work. I agree that replacing parts of koji and
bodhi library to serve our testing purposes seems a good way to go. I also considered
using custom Koji/Bodhi instances in the past (or using the Bodhi staging instance
instead), but it always looks like a huge maintenance burden.
I expect all this testing/stubbing/mocking to require considerable effort and time. I may
be wrong, but I don't currently see as reasonable to try to put some of it into
upcoming 0.5.0. I would rather make a release targeted specifically at enabling testing.
Or at least half of the release, I can imagine two people working on enabling testing and
two people working on ResultDB, or similarly. Should it be the very next release (0.6.0)?
I don't know, let's discuss and plan that once we release 0.5.0.