AutoQA Self-Test Proof of Concept

[Tim Flink]

This took a little longer than I thought it would but I have finished an 
initial proof of concept for unit testing surrounding a rather trivial 
fix for #265. The code is up in github 
(https://github.com/tflink/autoqa-devel/tree/pytest).

There are certainly some parts of it that are sub-optimal but overall, I 
think that its enough for a proof of concept. Unless people are thrilled 
with the py.test implementation that I currently have, I'm planning to 
do another proof of concept using nose/unittest.

Instructions for running the tests are in README.test.

In the interest of not sending out a huge email, I wrote up some 
documentation on why I structured things the way I did and the goals, 
possible tools etc. that could be used to test AutoQA. That document is 
also in the git repo 
(https://github.com/tflink/autoqa-devel/blob/pytest/doc/design/selfTesting...).

I tend to write in LaTeX and since I didn't see any other conventions I 
just wrote in what I'm most used to. It wouldn't be hard to convert it 
to something else (rst etc.) if that is what people prefer.

Feel free to modify the document or suggest changes. Just because I 
wrote it up doesn't make it final in any sense. Also, I'm sure that I 
forgot to explain something - questions are good :)

Tim

[Kamil Paral]

----- Original Message -----
>  This took a little longer than I thought it would but I have
> finished
>  an
>  initial proof of concept for unit testing surrounding a rather trivial
>  fix for #265. The code is up in github
>  (https://github.com/tflink/autoqa-devel/tree/pytest). 
Yay, we haven't added you to gitautoqa group, have we? You're sponsored now. You
should be able to checkout autoqa over ssh now and have write access. Can you re-push your
branch into our repository? Thanks.

>  
>  There are certainly some parts of it that are sub-optimal but overall,
>  I
>  think that its enough for a proof of concept. Unless people are
>  thrilled
>  with the py.test implementation that I currently have, I'm planning to
>  do another proof of concept using nose/unittest.
>  
>  Instructions for running the tests are in README.test. 
I'll see the contents later today.

>  
>  In the interest of not sending out a huge email, I wrote up some
>  documentation on why I structured things the way I did and the goals,
>  possible tools etc. that could be used to test AutoQA. That document
>  is
>  also in the git repo
>  (https://github.com/tflink/autoqa-devel/blob/pytest/doc/design/selfTesting...).
Ah, documentation! Very appreciated :-)

>  
>  I tend to write in LaTeX and since I didn't see any other conventions
>  I
>  just wrote in what I'm most used to. It wouldn't be hard to convert it
>  to something else (rst etc.) if that is what people prefer. 
Usually people write to mailing list or create a wiki page in Category:AutoQA [1] for more
permanent/larger docs, but do whatever you're used to/consider best, no policies in
there.

[1] https://fedoraproject.org/wiki/Category:AutoQA

>  
>  Feel free to modify the document or suggest changes. Just because I
>  wrote it up doesn't make it final in any sense. Also, I'm sure that I
>  forgot to explain something - questions are good :)
>  
>  Tim
>  _______________________________________________
>  autoqa-devel mailing list
>  autoqa-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/autoqa-devel

[Tim Flink]

On 02/25/2011 02:33 AM, Kamil Paral wrote:
>  Yay, we haven't added you to gitautoqa group, have we? You're
> sponsored now. You should be able to checkout autoqa over ssh now and have write access.
> Can you re-push your branch into our repository? Thanks. 
I just pushed up to
git.fedorahosted.org. Learning more about how to use 
git every day :)

> >
> > I tend to write in LaTeX and since I didn't see any other conventions
> > I
> > just wrote in what I'm most used to. It wouldn't be hard to convert it
> > to something else (rst etc.) if that is what people prefer.
>
>  Usually people write to mailing list or create a wiki page in Category:AutoQA [1] for
> more permanent/larger docs, but do whatever you're used to/consider best, no policies
> in there.
>
>  [1] https://fedoraproject.org/wiki/Category:AutoQA 
I can convert to a wiki page if that works better. That would probably 
reduce the barrier for others to edit the document, anyway. I guess I'm 
just in the habit of keeping documentation with the code instead of in a 
separate location.

Tim

[Kamil Paral]

----- Original Message -----
>  This took a little longer than I thought it would but I have
> finished
>  an
>  initial proof of concept for unit testing surrounding a rather trivial
>  fix for #265. The code is up in github
>  (https://github.com/tflink/autoqa-devel/tree/pytest).
>  
>  There are certainly some parts of it that are sub-optimal but overall,
>  I
>  think that its enough for a proof of concept. Unless people are
>  thrilled
>  with the py.test implementation that I currently have, I'm planning to
>  do another proof of concept using nose/unittest.
>  
>  Instructions for running the tests are in README.test.
>  
>  In the interest of not sending out a huge email, I wrote up some
>  documentation on why I structured things the way I did and the goals,
>  possible tools etc. that could be used to test AutoQA. That document
>  is
>  also in the git repo
>  (https://github.com/tflink/autoqa-devel/blob/pytest/doc/design/selfTesting...).
>   
* The documentation is great, thank you. I find it nicely structured. But I'm missing
comparison with standard python unittest framework (I never used it) [1]. Can you add a
few lines about that?

* README.test is probably missing "python-py" package among dependencies.
But after installing that package, I still receive an error:

  http://fpaste.org/zw5A/

pytest is located at:
/usr/lib/python2.7/site-packages/py/_cmdline/pytest.py

Where's the catch?

* By removing inheritance of SimpleKojiClientSession from koji.ClientSession, you'll
disable calling many methods on that object. Is that a desired behavior? Why's that
necessary?

Thanks,
Kamil


[1] http://docs.python.org/library/unittest.html

[Tim Flink]

On 02/28/2011 09:01 AM, Kamil Paral wrote:
>  * The documentation is great, thank you. I find it nicely structured. But I'm missing
> comparison with standard python unittest framework (I never used it) [1]. Can you add a
> few lines about that? 
I'm planning to do another proof of concept with
unittest/nose this 
week. I'll make sure to go into more depth on the py.test/unitttest 
comparison then.

>  * README.test is probably missing "python-py" package among
> dependencies. 
Those dependencies are handled in the virtualenv. The version of pylib

thats currently in the Fedora repos is too old for the way that those 
tests are written.

>  But after installing that package, I still receive an error:
>
>     http://fpaste.org/zw5A/
>
>  pytest is located at:
>  /usr/lib/python2.7/site-packages/py/_cmdline/pytest.py
>
>  Where's the catch? 
Whoops, my bad. I forgot that I have pip symlinked with a
non-standard 
name. It's pip-python by default. I pushed up a fixed shell script and 
it should run now.

>  * By removing inheritance of SimpleKojiClientSession from
> koji.ClientSession, you'll disable calling many methods on that object. Is that a
> desired behavior? Why's that necessary? 
I don't see any broken functionality
in what we're currently using, was 
there anything in particular you were worried about? Is there something 
that I'm missing?

My rationale for removing the inheritance was twofold:
1) Getting rid of the tightly coupled code
Tightly coupled code is, by nature, difficult to isolate since you are 
less able to mock/stub any of the inherited methods. If all of the code 
was under our control, we could refactor the parent class to help with 
the isolation but I sincerely doubt that a patch of that nature would be 
accepted in koji.

By using dependency injection and writing wrapper methods where needed, 
we can have more easily isolated code which lends to less complication 
and dirty hacks in the unit tests in addition to protecting ourselves 
from potential changes upstream (I'm not predicting changes, just 
speaking generally).

2) Efficiency and Sanity
While it would normally be possible to isolate the child class from the 
parent class through the use of mocks and monkeypatching, 
KojiClientSession is a bit special here. It uses some introspection 
magic to transform methods on the proxy class into results from the 
remote server.

Unfortunately, this is in direct conflict with the way that 
monkeypatching and mocking work in python and greatly increases the 
complexity, fragility and size of the supporting code used for unit tests.


It would be possible to restore the inheritance but that would require a 
decent amount of work and would result in more complex and fragile 
testing code. I'm of the opinion that this is an improvement over what 
is currently in master. It is more isolated from code we have no control 
over and is more easily testable.

Tim