Hi everyone,
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Nick
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
And here is the result of a successfull CI run with it: http://sssd-ci.duckdns.org/logs/job/1/91/summary.html
Nick
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
Thank you.
Nick
On Wed, Oct 22, 2014 at 11:55:43AM +0300, Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
Thank you.
Nick
OK, the patch looks fine to me and doesn't break CI.
ACK
On Wed, Oct 22, 2014 at 12:55:10PM +0200, Jakub Hrozek wrote:
On Wed, Oct 22, 2014 at 11:55:43AM +0300, Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
Thank you.
Nick
OK, the patch looks fine to me and doesn't break CI.
ACK
* master: e373fffbb8e06d0d7682d095c734e8df8a499ba0
On 10/22/2014 02:01 PM, Jakub Hrozek wrote:
On Wed, Oct 22, 2014 at 12:55:10PM +0200, Jakub Hrozek wrote:
On Wed, Oct 22, 2014 at 11:55:43AM +0300, Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
Thank you.
Nick
OK, the patch looks fine to me and doesn't break CI.
ACK
- master: e373fffbb8e06d0d7682d095c734e8df8a499ba0
Thanks a lot, Jakub!
Nick
On (22/10/14 11:55), Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
BTW: Developers could be still interested in html report from clang. This is the reason why execution on "git push instance" and execution on local machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with --test="a,b,c" We could have some templates which would describe order of executed test. (for example in YAML). Developer could easily modify order of test. He/she could have his/her own templates a) make discheck, mock b) just code coverage c) just static analysis d) all tests. It would also allow us to easily disable problematic test on server side. (temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora) Example: template(configuration file) https://github.com/karelzak/util-linux/blob/master/.travis.yml bash function used in template https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
LS
On 10/22/2014 02:22 PM, Lukas Slebodnik wrote:
On (22/10/14 11:55), Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
BTW: Developers could be still interested in html report from clang. This is the reason why execution on "git push instance" and execution on local machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with --test="a,b,c" We could have some templates which would describe order of executed test. (for example in YAML). Developer could easily modify order of test. He/she could have his/her own templates a) make discheck, mock b) just code coverage c) just static analysis d) all tests. It would also allow us to easily disable problematic test on server side. (temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora) Example: template(configuration file) https://github.com/karelzak/util-linux/blob/master/.travis.yml bash function used in template https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
LS
+1 for something like template file. That would be more flexible than command line arguments. If we want we can have something like default/fallback template file in git that will be used if user does not specify his own (and could also server as guide to craft custom template files).
Michal
On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
On (22/10/14 11:55), Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
BTW: Developers could be still interested in html report from clang. This is the reason why execution on "git push instance" and execution on local machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with --test="a,b,c" We could have some templates which would describe order of executed test. (for example in YAML). Developer could easily modify order of test. He/she could have his/her own templates a) make discheck, mock b) just code coverage c) just static analysis d) all tests. It would also allow us to easily disable problematic test on server side. (temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora) Example: template(configuration file) https://github.com/karelzak/util-linux/blob/master/.travis.yml bash function used in template https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
They use configuration file, because the actual CI code is outside the repo. Travis CI is a CI server, and the configuration file is an interface to it. It would be nice if we could do the same with Jenkins. In our case Jenkins-interfacing code lives in Jenkins configuration and what we have in the repo is only the tests themselves.
Regardless of CI system used, I'd like to keep our CI tests independent of its architecture so we can easily run them locally and change the CI system if needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I would have still kept the .travis.yml as minimal as possible, basically just invoking contrib/ci/run.
The above is not an argument, but just a clarification and separation of apples from oranges.
The ability to reorder tests might be useful, but it seems to me the actual practical benefit would be swamped by the design, coding, maintenance and performance overhead. Consider a situation where several tests require common setup, but then are reordered to have another test requiring conflicting setup in between them. I.e. hard to do in the general case.
The ability to select tests for execution (including more than one) is essential for anything that lasts longer than, say, 10 seconds and is involved in code/test cycle. The existing three sets are, perhaps, bearable for the current number and granularity of tests, but it would be nice to have more flexible selection and we'll definitely need it as number/length of our tests grows.
That said, I would say a special file with test lists would be unnecessary as long as most of us don't need a number of long and highly-tuned, complicated lists of tests to run. With test selection on the command line we can always have shell aliases or functions defined by users in their .bashrc, executing appropriate CI commands as necessary.
Regarding enabling the tests which are excluded from official CI for whatever reason, but are still useful to have, such as the Clang analyzer at the moment, we can have a solution for enabling them from the command line in a similar way.
FWIW, Epoxy allows hierarchical organization of tests and at the moment would allow selecting tests on the command line as such:
contrib/ci/run /debug/valgrind /coverage/clang
Or like this:
contrib/ci/run /debug
Or this (Bash extglob syntax):
contrib/ci/run '/@(debug|coverage)/make*'
Enabling disabled tests works like this, currently (patterns work as well):
contrib/ci/run --enable=/coverage/clang
The above would run all enabled tests, plus /coverage/clang. If you need to run just that one disabled test you'll need to specify it twice (as enabling and selection are separate):
contrib/ci/run --enable=/coverage/clang /coverage/clang
Disabling a test involves editing the source and adding a single-letter option to the test description command.
The above is just an example. We don't have to use Epoxy, although it might make these things easier. We don't have to do it the exact same way, but it might be that just the command line interface will be enough.
Nick
On (22/10/14 18:21), Nikolai Kondrashov wrote:
On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
On (22/10/14 11:55), Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
BTW: Developers could be still interested in html report from clang. This is the reason why execution on "git push instance" and execution on local machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with --test="a,b,c" We could have some templates which would describe order of executed test. (for example in YAML). Developer could easily modify order of test. He/she could have his/her own templates a) make discheck, mock b) just code coverage c) just static analysis d) all tests. It would also allow us to easily disable problematic test on server side. (temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora) Example: template(configuration file) https://github.com/karelzak/util-linux/blob/master/.travis.yml bash function used in template https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
They use configuration file, because the actual CI code is outside the repo. Travis CI is a CI server, and the configuration file is an interface to it. It would be nice if we could do the same with Jenkins. In our case Jenkins-interfacing code lives in Jenkins configuration and what we have in the repo is only the tests themselves.
Regardless of CI system used, I'd like to keep our CI tests independent of its architecture so we can easily run them locally and change the CI system if needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I would have still kept the .travis.yml as minimal as possible, basically just invoking contrib/ci/run.
The above is not an argument, but just a clarification and separation of apples from oranges.
You didn't get my point. I don't want to use Travis CI. I just like how they have implemented this. It is not important whether test are executed by Travis or Jenkins or locally with ci script. The most important part is that tests (.travis-functions.sh) are separated from the "test plan" (.travis.yml)
The ability to reorder tests might be useful, but it seems to me the actual
I was more interested in skipping some test rather than reordering test.
practical benefit would be swamped by the design, coding, maintenance and performance overhead. Consider a situation where several tests require common setup, but then are reordered to have another test requiring conflicting setup in between them. I.e. hard to do in the general case.
Tests should be independent, they can share some setup and teardown function, but it is totally different. Tests should pass in any order.
The ability to select tests for execution (including more than one) is essential for anything that lasts longer than, say, 10 seconds and is involved in code/test cycle. The existing three sets are, perhaps, bearable for the current number and granularity of tests, but it would be nice to have more flexible selection and we'll definitely need it as number/length of our tests grows.
Totally agree.
That said, I would say a special file with test lists would be unnecessary as long as most of us don't need a number of long and highly-tuned, complicated lists of tests to run. With test selection on the command line we can always have shell aliases or functions defined by users in their .bashrc, executing appropriate CI commands as necessary.
Regarding enabling the tests which are excluded from official CI for whatever reason, but are still useful to have, such as the Clang analyzer at the moment, we can have a solution for enabling them from the command line in a similar way.
FWIW, Epoxy allows hierarchical organization of tests and at the moment would allow selecting tests on the command line as such:
contrib/ci/run /debug/valgrind /coverage/clang
Or like this:
contrib/ci/run /debug
Or this (Bash extglob syntax):
contrib/ci/run '/@(debug|coverage)/make*'
Enabling disabled tests works like this, currently (patterns work as well):
contrib/ci/run --enable=/coverage/clang
The above would run all enabled tests, plus /coverage/clang. If you need to run just that one disabled test you'll need to specify it twice (as enabling and selection are separate):
contrib/ci/run --enable=/coverage/clang /coverage/clang
Currently, we have just small set of tests. Imagine near future :-). We will have lots of integration tests. Developer changes something in IPA provider. He is sure that test will not break ldap, krb5 or ad provider. For development, he wants to use just minimal set of integration test from IPA provider and some specific fest for nss responder and ifp responder. (Of course, developer will run full test set at the end)
I agree that it is possible to do it with command line arguments and simplify with aliases. The question is what is more comfortable for developer. IMHO, modification of file with all tests is simpler that modification of alias or very long oneliner; adding tests with arguments --enable=... --enable=... --exclude=... --enable=...
Travis has yaml for this purpose. Beaker uses xml. We can have json or plain text file in worst case. Structured declaration has some advantages.
LS
On 10/26/2014 12:46 AM, Lukas Slebodnik wrote:
On (22/10/14 18:21), Nikolai Kondrashov wrote:
On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
On (22/10/14 11:55), Nikolai Kondrashov wrote:
On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
The attached patch removes Clang analyzer run from CI as it takes a long time and its results are ignored.
I initially expected it to be taken into use quickly, but as that is not happening, we'd better remove it to save valuable CI server time.
We can always add it back when a solution to checking it is agreed on and implemented.
Jakub, can we get this merged? It will saves a good deal of time on the CI servers.
BTW: Developers could be still interested in html report from clang. This is the reason why execution on "git push instance" and execution on local machine are not the same use case. Ticket #2435 would be solution.
I can imagine even better solution than Michal's proposal with --test="a,b,c" We could have some templates which would describe order of executed test. (for example in YAML). Developer could easily modify order of test. He/she could have his/her own templates a) make discheck, mock b) just code coverage c) just static analysis d) all tests. It would also allow us to easily disable problematic test on server side. (temporary use non default template)
Travis CI use similar approach. (but it does not work with fedora) Example: template(configuration file) https://github.com/karelzak/util-linux/blob/master/.travis.yml bash function used in template https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
They use configuration file, because the actual CI code is outside the repo. Travis CI is a CI server, and the configuration file is an interface to it. It would be nice if we could do the same with Jenkins. In our case Jenkins-interfacing code lives in Jenkins configuration and what we have in the repo is only the tests themselves.
Regardless of CI system used, I'd like to keep our CI tests independent of its architecture so we can easily run them locally and change the CI system if needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I would have still kept the .travis.yml as minimal as possible, basically just invoking contrib/ci/run.
The above is not an argument, but just a clarification and separation of apples from oranges.
You didn't get my point. I don't want to use Travis CI.
Well, at least I got *that* :)
I just like how they have implemented this. It is not important whether test are executed by Travis or Jenkins or locally with ci script. The most important part is that tests (.travis-functions.sh) are separated from the "test plan" (.travis.yml)
Ah, understood now, sorry.
The ability to reorder tests might be useful, but it seems to me the actual
I was more interested in skipping some test rather than reordering test.
Good :)
practical benefit would be swamped by the design, coding, maintenance and performance overhead. Consider a situation where several tests require common setup, but then are reordered to have another test requiring conflicting setup in between them. I.e. hard to do in the general case.
Tests should be independent, they can share some setup and teardown function, but it is totally different. Tests should pass in any order.
Sure.
When costly shared setup/teardown is taken into consideration, performance of arbitrary reordering drops.
Consider this:
setup_A test_A1 test_A2 teardown_A setup_B test_B1 test_B2 teardown_B
To interleave tests you would have to do this:
setup_A test_A1 teardown_A setup_B test_B1 teardown_B setup_A test_A2 teardown_A setup_B test_B2 teardown_B
With arbitrary ordering, increasing number of tests for each fixture and increasing fixture cost, performance drops sharply. But since you agree that reordering is not that important, we're in the clear.
The ability to select tests for execution (including more than one) is essential for anything that lasts longer than, say, 10 seconds and is involved in code/test cycle. The existing three sets are, perhaps, bearable for the current number and granularity of tests, but it would be nice to have more flexible selection and we'll definitely need it as number/length of our tests grows.
Totally agree.
Great :)!
That said, I would say a special file with test lists would be unnecessary as long as most of us don't need a number of long and highly-tuned, complicated lists of tests to run. With test selection on the command line we can always have shell aliases or functions defined by users in their .bashrc, executing appropriate CI commands as necessary.
Regarding enabling the tests which are excluded from official CI for whatever reason, but are still useful to have, such as the Clang analyzer at the moment, we can have a solution for enabling them from the command line in a similar way.
FWIW, Epoxy allows hierarchical organization of tests and at the moment would allow selecting tests on the command line as such:
contrib/ci/run /debug/valgrind /coverage/clangOr like this:
contrib/ci/run /debugOr this (Bash extglob syntax):
contrib/ci/run '/@(debug|coverage)/make*'Enabling disabled tests works like this, currently (patterns work as well):
contrib/ci/run --enable=/coverage/clangThe above would run all enabled tests, plus /coverage/clang. If you need to run just that one disabled test you'll need to specify it twice (as enabling and selection are separate):
contrib/ci/run --enable=/coverage/clang /coverage/clangCurrently, we have just small set of tests. Imagine near future :-). We will have lots of integration tests. Developer changes something in IPA provider. He is sure that test will not break ldap, krb5 or ad provider. For development, he wants to use just minimal set of integration test from IPA provider and some specific fest for nss responder and ifp responder. (Of course, developer will run full test set at the end)
I agree that it is possible to do it with command line arguments and simplify with aliases. The question is what is more comfortable for developer. IMHO, modification of file with all tests is simpler that modification of alias or very long oneliner; adding tests with arguments --enable=... --enable=... --exclude=... --enable=...
Travis has yaml for this purpose. Beaker uses xml. We can have json or plain text file in worst case. Structured declaration has some advantages.
Alright, I think we can implement something like that once we get to a considerable number of tests. It'll be easier to figure out the details when we actually have them.
Nick
sssd-devel@lists.fedorahosted.org
-
Jakub Hrozek -
Lukas Slebodnik -
Michal Židek -
Nikolai Kondrashov