On (22/10/14 18:21), Nikolai Kondrashov wrote:
On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
>On (22/10/14 11:55), Nikolai Kondrashov wrote:
>>On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
>>>The attached patch removes Clang analyzer run from CI as it takes a long time
>>>and its results are ignored.
>>>
>>>I initially expected it to be taken into use quickly, but as that is not
>>>happening, we'd better remove it to save valuable CI server time.
>>>
>>>We can always add it back when a solution to checking it is agreed on and
>>>implemented.
>>
>>Jakub, can we get this merged? It will saves a good deal of time on the CI
>>servers.
>
>BTW: Developers could be still interested in html report from clang.
>This is the reason why execution on "git push instance" and execution on
local
>machine are not the same use case. Ticket #2435 would be solution.
>
>I can imagine even better solution than Michal's proposal with
--test="a,b,c"
>We could have some templates which would describe order of executed test.
>(for example in YAML). Developer could easily modify order of test. He/she
>could have his/her own templates
> a) make discheck, mock
> b) just code coverage
> c) just static analysis
> d) all tests.
>It would also allow us to easily disable problematic test on server side.
>(temporary use non default template)
>
>
>Travis CI use similar approach. (but it does not work with fedora)
>Example:
> template(configuration file)
>
https://github.com/karelzak/util-linux/blob/master/.travis.yml
> bash function used in template
>
https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
They use configuration file, because the actual CI code is outside the repo.
Travis CI is a CI server, and the configuration file is an interface to it. It
would be nice if we could do the same with Jenkins. In our case
Jenkins-interfacing code lives in Jenkins configuration and what we have in
the repo is only the tests themselves.
Regardless of CI system used, I'd like to keep our CI tests independent of its
architecture so we can easily run them locally and change the CI system if
needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I
would have still kept the .travis.yml as minimal as possible, basically just
invoking contrib/ci/run.
The above is not an argument, but just a clarification and separation of
apples from oranges.
You didn't get my point. I don't want to use Travis CI. I just like how
they
have implemented this. It is not important whether test are executed by Travis
or Jenkins or locally with ci script. The most important part is that
tests (.travis-functions.sh) are separated from the "test plan" (.travis.yml)
The ability to reorder tests might be useful, but it seems to me the
actual
I was more interested in skipping some test rather than reordering test.
practical benefit would be swamped by the design, coding, maintenance
and
performance overhead. Consider a situation where several tests require common
setup, but then are reordered to have another test requiring conflicting setup
in between them. I.e. hard to do in the general case.
Tests should be independent, they can share some setup and teardown function,
but it is totally different. Tests should pass in any order.
The ability to select tests for execution (including more than one)
is
essential for anything that lasts longer than, say, 10 seconds and is involved
in code/test cycle. The existing three sets are, perhaps, bearable for the
current number and granularity of tests, but it would be nice to have more
flexible selection and we'll definitely need it as number/length of our tests
grows.
Totally agree.
That said, I would say a special file with test lists would be
unnecessary as
long as most of us don't need a number of long and highly-tuned, complicated
lists of tests to run. With test selection on the command line we can always
have shell aliases or functions defined by users in their .bashrc, executing
appropriate CI commands as necessary.
Regarding enabling the tests which are excluded from official CI for whatever
reason, but are still useful to have, such as the Clang analyzer at the
moment, we can have a solution for enabling them from the command line in a
similar way.
FWIW, Epoxy allows hierarchical organization of tests and at the moment would
allow selecting tests on the command line as such:
contrib/ci/run /debug/valgrind /coverage/clang
Or like this:
contrib/ci/run /debug
Or this (Bash extglob syntax):
contrib/ci/run '/@(debug|coverage)/make*'
Enabling disabled tests works like this, currently (patterns work as well):
contrib/ci/run --enable=/coverage/clang
The above would run all enabled tests, plus /coverage/clang. If you need to
run just that one disabled test you'll need to specify it twice (as enabling
and selection are separate):
contrib/ci/run --enable=/coverage/clang /coverage/clang
Currently, we have just small set of tests. Imagine near future :-). We will
have lots of integration tests. Developer changes something in IPA provider.
He is sure that test will not break ldap, krb5 or ad provider. For development,
he wants to use just minimal set of integration test from IPA provider and some
specific fest for nss responder and ifp responder.
(Of course, developer will run full test set at the end)
I agree that it is possible to do it with command line arguments and simplify
with aliases. The question is what is more comfortable for developer.
IMHO, modification of file with all tests is simpler that modification of
alias or very long oneliner; adding tests with arguments
--enable=... --enable=... --exclude=... --enable=...
Travis has yaml for this purpose. Beaker uses xml. We can have json or plain
text file in worst case. Structured declaration has some advantages.
LS