On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
On (22/10/14 11:55), Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
BTW: Developers could be still interested in html report from clang. This is the reason why execution on "git push instance" and execution on local machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with --test="a,b,c" We could have some templates which would describe order of executed test. (for example in YAML). Developer could easily modify order of test. He/she could have his/her own templates a) make discheck, mock b) just code coverage c) just static analysis d) all tests. It would also allow us to easily disable problematic test on server side. (temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora) Example: template(configuration file) https://github.com/karelzak/util-linux/blob/master/.travis.yml bash function used in template https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
They use configuration file, because the actual CI code is outside the repo. Travis CI is a CI server, and the configuration file is an interface to it. It would be nice if we could do the same with Jenkins. In our case Jenkins-interfacing code lives in Jenkins configuration and what we have in the repo is only the tests themselves.
Regardless of CI system used, I'd like to keep our CI tests independent of its architecture so we can easily run them locally and change the CI system if needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I would have still kept the .travis.yml as minimal as possible, basically just invoking contrib/ci/run.
The above is not an argument, but just a clarification and separation of apples from oranges.
The ability to reorder tests might be useful, but it seems to me the actual practical benefit would be swamped by the design, coding, maintenance and performance overhead. Consider a situation where several tests require common setup, but then are reordered to have another test requiring conflicting setup in between them. I.e. hard to do in the general case.
The ability to select tests for execution (including more than one) is essential for anything that lasts longer than, say, 10 seconds and is involved in code/test cycle. The existing three sets are, perhaps, bearable for the current number and granularity of tests, but it would be nice to have more flexible selection and we'll definitely need it as number/length of our tests grows.
That said, I would say a special file with test lists would be unnecessary as long as most of us don't need a number of long and highly-tuned, complicated lists of tests to run. With test selection on the command line we can always have shell aliases or functions defined by users in their .bashrc, executing appropriate CI commands as necessary.
Regarding enabling the tests which are excluded from official CI for whatever reason, but are still useful to have, such as the Clang analyzer at the moment, we can have a solution for enabling them from the command line in a similar way.
FWIW, Epoxy allows hierarchical organization of tests and at the moment would allow selecting tests on the command line as such:
contrib/ci/run /debug/valgrind /coverage/clang
Or like this:
contrib/ci/run /debug
Or this (Bash extglob syntax):
contrib/ci/run '/@(debug|coverage)/make*'
Enabling disabled tests works like this, currently (patterns work as well):
contrib/ci/run --enable=/coverage/clang
The above would run all enabled tests, plus /coverage/clang. If you need to run just that one disabled test you'll need to specify it twice (as enabling and selection are separate):
contrib/ci/run --enable=/coverage/clang /coverage/clang
Disabling a test involves editing the source and adding a single-letter option to the test description command.
The above is just an example. We don't have to use Epoxy, although it might make these things easier. We don't have to do it the exact same way, but it might be that just the command line interface will be enough.
Nick