Thoughts about Travis-CI integration

[Tim Flink]

On Wed, 18 Dec 2013 17:52:59 +0200
Alexander Todorov <atodorov at redhat.com> wrote:

> Hi folks,
> this message sparked an interest:
> https://lists.fedorahosted.org/pipermail/python-bugzilla/2013-December/000200.html
> 
> In addition to that I've inspected around 30 packages which seem to
> be missing an upstream test suite (a few have one but it is not
> automatically executed in %check section in the spec file).
> 
> I'm pretty sure many more packages are like this, I just didn't have
> the time to investigate all several thousands of them.  

On a side note, it might be interesting to find out what percentage of
packages are running things in %check. I don't know what we would do
with that metric, but I think it would be interesting :)

> My idea is simple - starting after the holidays to call for help in
> writing test suites (or more test cases) for packages. This can be
> coupled with settings to execute them in Travis CI or another CI
> system of choice.
> 
> 
> 
> My questions are:
> 
> * What is the general feeling of using Travis CI in Fedora? It is
> well established in Ruby and Python circles but I know we like to
> keep dependency on external services to minumum.  

I've not used it personally, so I can't speak much about Travis in
particular. Depending on what you're looking to do, I would hesitate to
add yet another CI system to Fedora - infra already has a jenkins
instance set up and qa-devel is starting to use buildbot for both CI
and as a part of taskotron (the project formerly known as taskbot).

> Does Fedora have its own CI infrastructure coupled with Koji ?  

That depends on what you consider CI to be, I guess. I consider both
autoqa and taskotron to be forms of CI but not the exact same thing as
travis or jenkins.

Both autoqa and taskotron are designed to run various things based on a
package's state in either koji or bodhi.

> Maybe deploy our own instance or contribute to Travis with a pool of
> systems sponsored by Fedora?
> 
> What to do with packages whose test suite is not suitable to be
> executed during build (e.g. due to requirements or limitations on the
> build servers) ?
> 
> What's your take ?
> 
> ( Adding Tim Flink to CC to answer from the infrastructure side. )  

I'm still unclear on what you're looking to do. Are you talking about
looking for test suites in package upstreams and running those tests,
regardless of whether they're run in %check during build? Are you
talking about creating and maintaining an out-of-band set of unit tests
for packages without an upstream unit test suite? Are you talking about
creating a set of package-specific functional/integration test suites
that are run when packages are built?

> * Are there any volunteers to join me in planning and coordinating
> this project? We need to somehow prioritize which packages need
> inspection and working on, count the available test cases, report
> bugs if missing, produce patches, etc. It will be a long run one and
> needs lots of work just because the great number of packages.
> 
> 
> * Who else should I be talking to ?  

Kevin already mentioned this, but definitely infra.

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/test/attachments/20131218/9e48b824/attachment.sig>