Adjusting the way Fedora runs Python's test suite

[Nick Coghlan]

Hi folks,

While working on bootstrapping a new community sclo-python collection,
I ran into some readability problems with the way the SCL Python spec
file runs the test suite: it uses --verbose mode, which means there's
a lot of noisy output to sort through to find the actual failure when
something goes wrong.

Charalompos pointed out that since the SCL spec file is ultimately
derived from the Fedora one, Fedora would be the best place to file an
issue suggesting some changes, which I've now done:
https://bugzilla.redhat.com/show_bug.cgi?id=1473130

The main suggestion there is one that's a clear win: replace the
current "--verbose" with "-wW" instead, so that the tests are run in
non-verbose mode by default, but if one fails, not only will the test
suite print out any captured output, but it will also immediately
rerun the failed test in verbose mode.

However, I also noted some other possibilities that seem worth discussing:

* checking for refleaks means we can't run parallel tests across
multiple processes. If we were to move refleak testing out to an
integration test in Taskotron, then the tests could be run with the
"-j4" option, significantly reducing the time needed to run the %check
step

* the `--failfast` option causes the tests to be aborted as soon as
one fails. While that doesn't make any difference in the success case,
it will mean faster turnaround cycles in the case where one fails. I'm
less sure of the value of this one, as the downside is that if you're
getting multiple failures, you only find out about them one at a time.

* there's a `--slowest` flag in recent releases that lists the 10
slowest tests in a given run. As far as I know, it doesn't add much
overhead to the test execution process, and provides an indicator of
where to focus efforts if the goal is to make the tests more
efficient.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan(a)gmail.com   |   Brisbane, Australia

[Bohuslav Kabrda]

On Thu, Jul 20, 2017 at 7:51 AM, Nick Coghlan <ncoghlan(a)gmail.com&gt; wrote:

>  Hi folks,
>
>  While working on bootstrapping a new community sclo-python collection,
>  I ran into some readability problems with the way the SCL Python spec
>  file runs the test suite: it uses --verbose mode, which means there's
>  a lot of noisy output to sort through to find the actual failure when
>  something goes wrong.
>
>  Charalompos pointed out that since the SCL spec file is ultimately
>  derived from the Fedora one, Fedora would be the best place to file an
>  issue suggesting some changes, which I've now done:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1473130
>
>  The main suggestion there is one that's a clear win: replace the
>  current "--verbose" with "-wW" instead, so that the tests are run in
>  non-verbose mode by default, but if one fails, not only will the test
>  suite print out any captured output, but it will also immediately
>  rerun the failed test in verbose mode.
>
>  
+1, that sounds great.


>  However, I also noted some other possibilities that seem worth
> discussing:
>
>  * checking for refleaks means we can't run parallel tests across
>  multiple processes. If we were to move refleak testing out to an
>  integration test in Taskotron, then the tests could be run with the
>  "-j4" option, significantly reducing the time needed to run the %check
>  step
>
>  
If it's possible to move this testing to taskotron, then it sounds ok to me.


>  * the `--failfast` option causes the tests to be aborted as soon as
>  one fails. While that doesn't make any difference in the success case,
>  it will mean faster turnaround cycles in the case where one fails. I'm
>  less sure of the value of this one, as the downside is that if you're
>  getting multiple failures, you only find out about them one at a time.
>
>  
I'd suggest to not use this. Back when I was Python maintainer I used this
option sometimes when updating Python and building locally, but for builds
in Koji, you don't want to use it. What you want is to see logs of all the
failed tests in Koji logs and then be able to work on the first, submit a
build when you have a fix and *in the meanwhile* (which in terms of
building Python can be couple dozen minutes) work to fix other bugs... And
repeat that until you fix all. If you did use this flag, you'd have to
build, fix one bug, submit a build, wait a long time, fix another bug,
submit a build, wait a long time, fix another bug, etc.


>  * there's a `--slowest` flag in recent releases that lists the
> 10
>  slowest tests in a given run. As far as I know, it doesn't add much
>  overhead to the test execution process, and provides an indicator of
>  where to focus efforts if the goal is to make the tests more
>  efficient.
>
>  
Yeah, I think improving the performance of the slowest tests could be gain
for everyone, not just for Fedora.


>  Cheers,
>  Nick.
>
>  --
>  Nick Coghlan   |   ncoghlan(a)gmail.com   |   Brisbane, Australia
>  _______________________________________________
>  python-devel mailing list -- python-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
>  


-- 
Regards,
Slavek Kabrda

[Miro Hrončok]

On 20.7.2017 08:12, Bohuslav Kabrda wrote:
>  On Thu, Jul 20, 2017 at 7:51 AM, Nick Coghlan <ncoghlan(a)gmail.com
>
>  <mailto:[e-mail]>> wrote:
>  
>      Hi folks,
>  
>      While working on bootstrapping a new community sclo-python collection,
>      I ran into some readability problems with the way the SCL Python spec
>      file runs the test suite: it uses --verbose mode, which means there's
>      a lot of noisy output to sort through to find the actual failure when
>      something goes wrong.
>  
>      Charalompos pointed out that since the SCL spec file is ultimately
>      derived from the Fedora one, Fedora would be the best place to file an
>      issue suggesting some changes, which I've now done:
>      https://bugzilla.redhat.com/show_bug.cgi?id=1473130
>      <https://bugzilla.redhat.com/show_bug.cgi?id=1473130>
>  
>      The main suggestion there is one that's a clear win: replace the
>      current "--verbose" with "-wW" instead, so that the tests are run
> in
>      non-verbose mode by default, but if one fails, not only will the test
>      suite print out any captured output, but it will also immediately
>      rerun the failed test in verbose mode.
>  
>  
>  +1, that sounds great.
>  
>      However, I also noted some other possibilities that seem worth
>      discussing:
>  
>      * checking for refleaks means we can't run parallel tests across
>      multiple processes. If we were to move refleak testing out to an
>      integration test in Taskotron, then the tests could be run with the
>      "-j4" option, significantly reducing the time needed to run the %check
>      step
>  
>  
>  If it's possible to move this testing to taskotron, then it sounds ok to me.
>  
>      * the `--failfast` option causes the tests to be aborted as soon as
>      one fails. While that doesn't make any difference in the success case,
>      it will mean faster turnaround cycles in the case where one fails. I'm
>      less sure of the value of this one, as the downside is that if you're
>      getting multiple failures, you only find out about them one at a time.
>  
>  
>  I'd suggest to not use this. Back when I was Python maintainer I used 
>  this option sometimes when updating Python and building locally, but for 
>  builds in Koji, you don't want to use it. What you want is to see logs 
>  of all the failed tests in Koji logs and then be able to work on the 
>  first, submit a build when you have a fix and *in the meanwhile* (which 
>  in terms of building Python can be couple dozen minutes) work to fix 
>  other bugs... And repeat that until you fix all. If you did use this 
>  flag, you'd have to build, fix one bug, submit a build, wait a long 
>  time, fix another bug, submit a build, wait a long time, fix another 
>  bug, etc. 
Exactly what I thought.

>  
>      * there's a `--slowest` flag in recent releases that lists the 10
>      slowest tests in a given run. As far as I know, it doesn't add much
>      overhead to the test execution process, and provides an indicator of
>      where to focus efforts if the goal is to make the tests more
>      efficient.
>  
>  
>  Yeah, I think improving the performance of the slowest tests could be 
>  gain for everyone, not just for Fedora.
>  
>      Cheers,
>      Nick.
>  
>      --
>      Nick Coghlan   | ncoghlan(a)gmail.com <mailto:[e-mail]>   | 
>        Brisbane, Australia
>      _______________________________________________
>      python-devel mailing list -- python-devel(a)lists.fedoraproject.org
>      <mailto:[e-mail]>
>      To unsubscribe send an email to
>      python-devel-leave(a)lists.fedoraproject.org
>      <mailto:[e-mail]>
>  
>  
>  
>  
>  -- 
>  Regards,
>  Slavek Kabrda
>  
>  
>  _______________________________________________
>  python-devel mailing list -- python-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
>   
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Nick Coghlan]

On 20 July 2017 at 16:12, Bohuslav Kabrda <bkabrda(a)redhat.com&gt; wrote:
>  On Thu, Jul 20, 2017 at 7:51 AM, Nick Coghlan
> <ncoghlan(a)gmail.com&gt; wrote:
> > * checking for refleaks means we can't run parallel tests across
> > multiple processes. If we were to move refleak testing out to an
> > integration test in Taskotron, then the tests could be run with the
> > "-j4" option, significantly reducing the time needed to run the %check
> > step
> >
>
>  If it's possible to move this testing to taskotron, then it sounds ok to me. 
Based on my changes to the sclo-python spec, I can confirm that
passing "-j0" (which autoscales to the number of CPUs +2 to allow for
blocking tests) cuts my local RPM build times from just under 40
minutes to just over 15.

So I'll use that to speed up the sclo-python bootstrapping builds, and
then we can wait for the dist-git move into Pagure and the
introduction of https://fedoraproject.org/wiki/Changes/InvokingTests
before looking at changing the main build.

> > * the `--failfast` option causes the tests to be aborted as soon
> as
> > one fails. While that doesn't make any difference in the success case,
> > it will mean faster turnaround cycles in the case where one fails. I'm
> > less sure of the value of this one, as the downside is that if you're
> > getting multiple failures, you only find out about them one at a time.
>
>  I'd suggest to not use this. 
Agreed, especially as it's easy enough to add in to the spec file if
you actually want it in a mock build for some reason.

> > * there's a `--slowest` flag in recent releases that lists
> the 10
> > slowest tests in a given run. As far as I know, it doesn't add much
> > overhead to the test execution process, and provides an indicator of
> > where to focus efforts if the goal is to make the tests more
> > efficient.
>
>  Yeah, I think improving the performance of the slowest tests could be gain
>  for everyone, not just for Fedora. 
One caveat on this option is that it's a relatively recent addition
that was never backported to the 3.5 branch, so it's only applicable
to builds of 3.6+.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan(a)gmail.com   |   Brisbane, Australia

[Charalampos Stratakis]

First step is done [0]

[0]
http://pkgs.fedoraproject.org/cgit/rpms/python3.git/commit/?id=5b6a3e0e34...

Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat


----- Original Message -----
From: "Nick Coghlan" <ncoghlan(a)gmail.com&gt;
To: "Fedora Python SIG" <python-devel(a)lists.fedoraproject.org&gt;
Sent: Thursday, July 20, 2017 11:54:33 AM
Subject: Re: Adjusting the way Fedora runs Python's test suite

On 20 July 2017 at 16:12, Bohuslav Kabrda <bkabrda(a)redhat.com&gt; wrote:
>  On Thu, Jul 20, 2017 at 7:51 AM, Nick Coghlan
> <ncoghlan(a)gmail.com&gt; wrote:
> > * checking for refleaks means we can't run parallel tests across
> > multiple processes. If we were to move refleak testing out to an
> > integration test in Taskotron, then the tests could be run with the
> > "-j4" option, significantly reducing the time needed to run the %check
> > step
> >
>
>  If it's possible to move this testing to taskotron, then it sounds ok to me. 
Based on my changes to the sclo-python spec, I can confirm that
passing "-j0" (which autoscales to the number of CPUs +2 to allow for
blocking tests) cuts my local RPM build times from just under 40
minutes to just over 15.

So I'll use that to speed up the sclo-python bootstrapping builds, and
then we can wait for the dist-git move into Pagure and the
introduction of https://fedoraproject.org/wiki/Changes/InvokingTests
before looking at changing the main build.

> > * the `--failfast` option causes the tests to be aborted as soon
> as
> > one fails. While that doesn't make any difference in the success case,
> > it will mean faster turnaround cycles in the case where one fails. I'm
> > less sure of the value of this one, as the downside is that if you're
> > getting multiple failures, you only find out about them one at a time.
>
>  I'd suggest to not use this. 
Agreed, especially as it's easy enough to add in to the spec file if
you actually want it in a mock build for some reason.

> > * there's a `--slowest` flag in recent releases that lists
> the 10
> > slowest tests in a given run. As far as I know, it doesn't add much
> > overhead to the test execution process, and provides an indicator of
> > where to focus efforts if the goal is to make the tests more
> > efficient.
>
>  Yeah, I think improving the performance of the slowest tests could be gain
>  for everyone, not just for Fedora. 
One caveat on this option is that it's a relatively recent addition
that was never backported to the 3.5 branch, so it's only applicable
to builds of 3.6+.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan(a)gmail.com   |   Brisbane, Australia
_______________________________________________
python-devel mailing list -- python-devel(a)lists.fedoraproject.org
To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org

[Nick Coghlan]

On 21 July 2017 at 02:03, Charalampos Stratakis <cstratak(a)redhat.com&gt; wrote:
>  First step is done [0]
>
>  [0]
> http://pkgs.fedoraproject.org/cgit/rpms/python3.git/commit/?id=5b6a3e0e34...
Nice!

So that just leaves the --findleaks -> -j0 change, and that's gated on
both the upcoming pkgdb -> Pagure--in-front-of-dist-git migration and
subsequently adding a new integration test to replace "--findleaks".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan(a)gmail.com   |   Brisbane, Australia