On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
On (22/10/14 11:55), Nikolai Kondrashov wrote:
> On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
>> The attached patch removes Clang analyzer run from CI as it takes a long time
>> and its results are ignored.
>>
>> I initially expected it to be taken into use quickly, but as that is not
>> happening, we'd better remove it to save valuable CI server time.
>>
>> We can always add it back when a solution to checking it is agreed on and
>> implemented.
>
> Jakub, can we get this merged? It will saves a good deal of time on the CI
> servers.
BTW: Developers could be still interested in html report from clang.
This is the reason why execution on "git push instance" and execution on local
machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with
--test="a,b,c"
We could have some templates which would describe order of executed test.
(for example in YAML). Developer could easily modify order of test. He/she
could have his/her own templates
a) make discheck, mock
b) just code coverage
c) just static analysis
d) all tests.
It would also allow us to easily disable problematic test on server side.
(temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora)
Example:
template(configuration file)
https://github.com/karelzak/util-linux/blob/master/.travis.yml
bash function used in template
https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
They use configuration file, because the actual CI code is outside the repo.
Travis CI is a CI server, and the configuration file is an interface to it. It
would be nice if we could do the same with Jenkins. In our case
Jenkins-interfacing code lives in Jenkins configuration and what we have in
the repo is only the tests themselves.
Regardless of CI system used, I'd like to keep our CI tests independent of its
architecture so we can easily run them locally and change the CI system if
needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I
would have still kept the .travis.yml as minimal as possible, basically just
invoking contrib/ci/run.
The above is not an argument, but just a clarification and separation of
apples from oranges.
The ability to reorder tests might be useful, but it seems to me the actual
practical benefit would be swamped by the design, coding, maintenance and
performance overhead. Consider a situation where several tests require common
setup, but then are reordered to have another test requiring conflicting setup
in between them. I.e. hard to do in the general case.
The ability to select tests for execution (including more than one) is
essential for anything that lasts longer than, say, 10 seconds and is involved
in code/test cycle. The existing three sets are, perhaps, bearable for the
current number and granularity of tests, but it would be nice to have more
flexible selection and we'll definitely need it as number/length of our tests
grows.
That said, I would say a special file with test lists would be unnecessary as
long as most of us don't need a number of long and highly-tuned, complicated
lists of tests to run. With test selection on the command line we can always
have shell aliases or functions defined by users in their .bashrc, executing
appropriate CI commands as necessary.
Regarding enabling the tests which are excluded from official CI for whatever
reason, but are still useful to have, such as the Clang analyzer at the
moment, we can have a solution for enabling them from the command line in a
similar way.
FWIW, Epoxy allows hierarchical organization of tests and at the moment would
allow selecting tests on the command line as such:
contrib/ci/run /debug/valgrind /coverage/clang
Or like this:
contrib/ci/run /debug
Or this (Bash extglob syntax):
contrib/ci/run '/@(debug|coverage)/make*'
Enabling disabled tests works like this, currently (patterns work as well):
contrib/ci/run --enable=/coverage/clang
The above would run all enabled tests, plus /coverage/clang. If you need to
run just that one disabled test you'll need to specify it twice (as enabling
and selection are separate):
contrib/ci/run --enable=/coverage/clang /coverage/clang
Disabling a test involves editing the source and adding a single-letter option
to the test description command.
The above is just an example. We don't have to use Epoxy, although it might
make these things easier. We don't have to do it the exact same way, but it
might be that just the command line interface will be enough.
Nick