split test into small tests

[Hongqing Yang]

Greetings,

Now I am trying to integrate mediakit_sanity test into AutoQA, mediakit sanity test  includes
tests of media size, checksum, repoclosure, conflicts. I find it is trouble to deal with the
test results, since it is impossible to specify which test is FAILED, we can only give a general
result to all tests self.detail.result=FAILED even some tests passed.

So do you think it it OK that I split the test into small tests? apparently, there will be many
test in the tests directory. Or do you have any suggestion to solve this? Thanks.

Hongqing

[Kamil Paral]

> Greetings,
>
> Now I am trying to integrate mediakit_sanity test into AutoQA,
> mediakit sanity test includes
> tests of media size, checksum, repoclosure, conflicts. I find it is
> trouble to deal with the
> test results, since it is impossible to specify which test is FAILED,
> we can only give a general
> result to all tests self.detail.result=FAILED even some tests passed.
>
> So do you think it it OK that I split the test into small tests?
> apparently, there will be many
> test in the tests directory. Or do you have any suggestion to solve
> this? Thanks.
>
> Hongqing

Hi,

you can report multiple results in a single test run. For example upgradepath test checks all proposed updates and reports PASS/FAIL for every one of them individually. The code is then a little more complicated. Basically you create a separate TestDetail instance for every "item" you want to have reported (in your case one for media size, one for checksum, ...):
http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=tests/upgradepath/upg...
In the end you reports all those TestDetail results manually:
http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=tests/upgradepath/upg...

But I wonder, is it all really necessary? You can use Summary Details section to provide something like this:
Media size: PASS
Checksum: PASS
Repoclosure: PASS
Conflicts: FAIL
and then set the overall result to PASS only if all the four subsections are PASS. This way if we encounter failed mediakit_sanity, it's easy to open the log and immediately see which parts failed.

Kamil