Thoughts about Travis-CI integration

[Alexander Todorov]

На 18.12.2013 20:09, Tim Flink написа:
> On a side note, it might be interesting to find out what percentage of
> packages are running things in %check. I don't know what we would do
> with that metric, but I think it would be interesting :)
>

I have something in the works which can be modified to extract this info. It 
should be fairly easy to make a good guesstimate of the presence/non presence of 
test suites as well.  I think I'll start from here.

>> Does Fedora have its own CI infrastructure coupled with Koji ?
>
> That depends on what you consider CI to be, I guess. I consider both
> autoqa and taskotron to be forms of CI but not the exact same thing as
> travis or jenkins.
>
> Both autoqa and taskotron are designed to run various things based on a
> package's state in either koji or bodhi.
>

Thanks, I have to look into these. I gave Travis as example because it is well 
known and because somebody else decided to make use of it in python-bugzilla.


>
> I'm still unclear on what you're looking to do. Are you talking about
> looking for test suites in package upstreams and running those tests,
> regardless of whether they're run in %check during build? Are you
> talking about creating and maintaining an out-of-band set of unit tests
> for packages without an upstream unit test suite? Are you talking about
> creating a set of package-specific functional/integration test suites
> that are run when packages are built?
>

Pretty much all of these in the long term but I wanted to get a general feeling 
so I can concentrate the effort on something specific.


* For packages which have upstream test suites (e.g. parted) - contribute more 
tests where necessary; Ideally covering bugs reported against Fedora or RHEL for 
which there isn't a test case.

* For packages without upstream test suite my preference is to create one and 
contribute it back upstream. If upstream doesn't accept it and the package is 
critical enough maybe maintain that in the Fedora branch.

* Functional/integration test suites (e.g. using Beaker tasks) are less priority 
goal for the moment but also need to be considered. For some packages these are 
more suitable (e.g. anaconda), for others maybe not (e.g. pykickstart seems to 
have fairly good amount of unit tests).


Whatever the test suite(s) come out to be we need an environment(or 
environments) where to execute them and a method of starting the suite. I can 
see how some test cases are part of %check and can halt the build process and 
others are started async after the build and are left for Fedora QA and devel to 
investigate.

In the case of Travis, test runs are triggered by git commits and Travis is not 
connected to any build infrastructure from what I know. It provides the results 
to whoever is interested in them.