4:10 p.m.
I spent a bit of time fixing minor issues in our test suite and makefiles and would like
to do the following further changes across all our taskotron projects:
1. run the test suite while inside virtualenv with simple `pytest` command
2. run the test suite outside of virtualenv with `make test` or `doit test` and recommend
this approach in readme
3. use a separate virtualenv when running under `make test`, without
--system-site-packages if possible, and ensure up-to-date deps are always installed, to
eliminate any differences that can occur on different setups
4. configure pytest to print out coverage by default, so that it shows up for both
`pytest` and `make test`
Any concerns or different suggestions?
2:01 a.m.
On Fri, Feb 3, 2017 at 11:05 PM, Kamil Paral <kparal(a)redhat.com> wrote:
I spent a bit of time fixing minor issues in our test suite and
makefiles
and would like to do the following further changes across all our taskotron
projects:
1. run the test suite while inside virtualenv with simple `pytest` command
2. run the test suite outside of virtualenv with `make test` or `doit
test` and recommend this approach in readme
3. use a separate virtualenv when running under `make test`, without
--system-site-packages if possible, and ensure up-to-date deps are always
installed, to eliminate any differences that can occur on different setups
The only problem I see here, is that some of the packages that you'd need
to install into the test-virtualenv need some C-compilation when installing
from PyPi, and that (if used without the --system-site-packages) would mean
having to install not only gcc but also plenty of -dev packages.
Not a _huge_ ussue but an issue nevertheless.
4. configure pytest to print out coverage by default, so that it
shows up
for both `pytest` and `make test`
Any concerns or different suggestions?
_______________________________________________
qa-devel mailing list -- qa-devel(a)lists.fedoraproject.org
To unsubscribe send an email to qa-devel-leave(a)lists.fedoraproject.org
6:35 a.m.
On Fri, Feb 3, 2017 at 11:05 PM, Kamil Paral < kparal(a)redhat.com
> wrote:
> I spent a bit of time fixing minor issues in our test suite and
makefiles
> and
> would like to do the following further changes across all our taskotron
> projects:
> 1. run the test suite while inside virtualenv with simple
`pytest` command
> 2. run the test suite outside of virtualenv with `make test` or `doit test`
> and recommend this approach in readme
> 3. use a separate virtualenv when running under `make test`, without
> --system-site-packages if possible, and ensure up-to-date deps are always
> installed, to eliminate any differences that can occur on different setups
The only problem I see here, is that some of the packages that
you'd need to
install into the test-virtualenv need some C-compilation when installing
from PyPi, and that (if used without the --system-site-packages) would mean
having to install not only gcc but also plenty of -dev packages.
Not a _huge_ ussue but an issue nevertheless.
That's a good point. But do we have a good alternative here? If we depend on packages
like that, I see only two options:
a) ask the person to install pyfoo as an RPM (in readme)
b) ask the person to install gcc and libfoo-devel as an RPM (in readme) and pyfoo will be
then compiled and installed from pypi
Approach a) is somewhat easier and does not require compilation stack and devel libraries.
OTOH it requires using virtualenv with --system-site-packages, which means people get
different results on different setups. That's exactly what I'm trying to eliminate
(or at least reduce). E.g. https://phab.qa.fedoraproject.org/D1111 where I can run the
test suite from makefile and you can't, and it's quite difficult to figure out
why.
With b) approach, you need compilation stack on the system. I don't think it's
such a huge problem, because you're a developer after all. The advantage is that
virtualenv can be created without --system-site-packages, which means locally installed
libraries do not affect the execution/test suite results. Also, pyfoo is installed with
exactly the right version, further reducing differences between setups. The only thing
that can differ is the version of libfoo-devel, which can affect the behavior. But the
likeliness of that happening is much smaller than having pyfoo of a different version or
pulling any deps from the system site packages.
The reason why I want to recommend `make test` for running the test suite (at least in
readme), is because in the makefile we can ensure that a clean virtualenv with correct
properties is created, and only and exactly the right versions of deps from
requirements.txt are installed. We can perform further necessary steps, like installing
the project . That further increases reliability. Compare this to manually running
`pytest`- a custom virtualenv must be active; it can be configured differently than
recommended in readme, it can be out of date, or it can have more packages installed than
needed; you might forget some necessary steps.
Of course nothing prevents you from simply running the test suite using `pytest`. It's
the same approach that Phab will do when submitting a patch. However, when some issues
arises, I'd like all parties to be able to run `make test` and it should return the
same result. That should be the most reliable method, and if it doesn't return the
same thing, it means we have an important problem somewhere, and it's not just "a
wrongly configured project on one dev machine".
So, I see these main use cases for `make test` and b) approach:
* good a reliable default for newcomers, an approach that's the least likely to go
wrong
* determining the reason for failures that only one party sees and the other doesn't
* `make test-ci` target, that will hopefully be used one day to perform daily/per-commit
CI testing of our codebases. Again, using the most reliable method available.
For some codebases this is not viable anyway, e.g. libtaskotron, because they depend on
packages not available in pypi (koji) and thus need --system-site-packages. But e.g.
resultsdb projects seem that they could go without it.
7:11 a.m.
> 3. use a separate virtualenv when running under `make test`,
without
> --system-site-packages if possible, and ensure up-to-date deps are always
> installed, to eliminate any differences that can occur on different setups
> The only problem I see here, is that some of the packages that
you'd need
> to
> install into the test-virtualenv need some C-compilation when installing
> from PyPi, and that (if used without the --system-site-packages) would mean
> having to install not only gcc but also plenty of -dev packages.
> Not a _huge_ ussue but an issue nevertheless.
That's a good point. But do we have a good alternative here? If
we depend on
packages like that, I see only two options:
a) ask the person to install pyfoo as an RPM (in readme)
b) ask the person to install gcc and libfoo-devel as an RPM (in readme) and
pyfoo will be then compiled and installed from pypi
Approach a) is somewhat easier and does not require compilation stack
and
devel libraries. OTOH it requires using virtualenv with
--system-site-packages, which means people get different results on
different setups. That's exactly what I'm trying to eliminate (or at least
reduce). E.g. https://phab.qa.fedoraproject.org/D1111 where I can run the
test suite from makefile and you can't, and it's quite difficult to figure
out why.
With b) approach, you need compilation stack on the system. I
don't think
it's such a huge problem, because you're a developer after all. The
advantage is that virtualenv can be created without --system-site-packages,
which means locally installed libraries do not affect the execution/test
suite results. Also, pyfoo is installed with exactly the right version,
further reducing differences between setups. The only thing that can differ
is the version of libfoo-devel, which can affect the behavior. But the
likeliness of that happening is much smaller than having pyfoo of a
different version or pulling any deps from the system site packages.
Sigh, OTOH Josef has a very good point in https://phab.qa.fedoraproject.org/D1112#20744
that figuring out which devel packages are needed on the system for pypi module
compilation is quite a non-trivial task. Seems harder than figuring out which packages you
need to install on the system when using --system-site-packages.
So, I don't really see a simple solution here that would guard us from inconsistent
dev setups (system site packages) or not require us to have a painful deps maintenance (no
system site packages, but devel packages) :/
Even if we decide to keep --system-site-packages for simplicity, I can implement all the
rest of the proposed improvements. Just the reliability will not be as great as I wished.
The main question is now, I believe, whether we want to install compilation-dependent
packages from pypi or rpm.
7:11 a.m.
On Mon, Feb 6, 2017 at 1:35 PM, Kamil Paral <kparal(a)redhat.com> wrote:
That's a good point. But do we have a good alternative here? If we depend
on packages like that, I see only two options:
a) ask the person to install pyfoo as an RPM (in readme)
b) ask the person to install gcc and libfoo-devel as an RPM (in readme)
and pyfoo will be then compiled and installed from pypi
Approach a) is somewhat easier and does not require compilation stack and
devel libraries. OTOH it requires using virtualenv with
--system-site-packages, which means people get different results on
different setups. That's exactly what I'm trying to eliminate (or at least
reduce). E.g. https://phab.qa.fedoraproject.org/D1111 where I can run the
test suite from makefile and you can't, and it's quite difficult to figure
out why.
With b) approach, you need compilation stack on the system. I don't think
it's such a huge problem, because you're a developer after
all. The
advantage is that virtualenv can be created without --system-site-packages,
which means locally installed libraries do not affect the execution/test
suite results. Also, pyfoo is installed with exactly the right version,
further reducing differences between setups. The only thing that can differ
is the version of libfoo-devel, which can affect the behavior. But the
likeliness of that happening is much smaller than having pyfoo of a
different version or pulling any deps from the system site packages.
The reason why I want to recommend `make test` for running the test suite
(at least in readme), is because in the makefile we can ensure that a
clean
virtualenv with correct properties is created, and only and exactly the
right versions of deps from requirements.txt are installed. We can perform
further necessary steps, like installing the project
<https://phab.qa.fedoraproject.org/D1111>;. That further increases
reliability. Compare this to manually running `pytest`- a custom virtualenv
must be active; it can be configured differently than recommended in
readme, it can be out of date, or it can have more packages installed than
needed; you might forget some necessary steps.
Sure, I am a devel, but not a C-devel... As I told you in our other
conversation - I see what you are trying to accomplish, but for me the gain
does not even balance the issues. With variant 'a', all you need to do is
make sure "these python packages are installed" to run the test suite. I'd
rather have something like `requirements_testing.txt` where all the deps
are speeled out in the proper versions, and using that as a base for the
virtualenv population (I guess we could easily make do with the
requirements.py we have now). Either you have the right version in your
system (or in your own development virtualenv from which you are running
the tests), or the right version will be installed for you from pip.
Yes, we might get down to people having to install bunch of header files,
and gcc, if for some reason their system is so different that they can not
obtain the right version in any other way, but it will work most of the
time.
Of course nothing prevents you from simply running the test suite
using
`pytest`. It's the same approach that Phab will do when submitting a patch.
However, when some issues arises, I'd like all parties to be able to run
`make test` and it should return the same result. That should be the most
reliable method, and if it doesn't return the same thing, it means we have
an important problem somewhere, and it's not just "a wrongly configured
project on one dev machine".
So, I see these main use cases for `make test` and b) approach:
* good a reliable default for newcomers, an approach that's the
least
likely to go wrong
* determining the reason for failures that only one party sees and the
other doesn't
* `make test-ci` target, that will hopefully be used one day to perform
daily/per-commit CI testing of our codebases. Again, using the most
reliable method available.
Sure, nobody forces _me_ to do it this way, but I still fail to see the
overall general benefit. If a random _python web app_ project that I wanted
to submit a patch for wanted me to install gcc and tons of -devel libs, I'd
be going to the next door. We were talking "accessibility" a lot with Phab,
and one of the arguments against it (not saying it was you in particular)
was that "it is complicated, and needs additional packages installed". This
is even worse version of the same. At least to me.
On top of that - who is going to be syncing up the versions of said
packages between Fedora (our target) and the requirements.txt? What release
are we going to be using as the target? And is it even the right place and
way to do it?
For some codebases this is not viable anyway, e.g. libtaskotron,
because
they depend on packages not available in pypi (koji) and thus need
--system-site-packages. But e.g. resultsdb projects seem that they could go
without it.
If this is not to be the same on our whole stack, then I can see the point
of doing it even less.
I'm not against doing the rigorous testing somewhere. I just plain don't
believe that the developer's machine is the place for this kind of "making
sure every bit of the whole setup is the same for all the people, as for
our deployment setup".
I'd much rather have some "hardened" acceptance testing done for the diff
requests, in a known env, but out of the devs machine - let's tell the devs
what we do, and let them know how to achieve the same level of proofing,
but leave the default path simple.
IMHO 90% of the time, this will be a non-issue (as if was so far), and in
the 10% that cause the troubles, human interaction will be required anyway.
All in all, I absolutely see the point of what you want to achieve, I just
think that the way to do it is not ideal. From my POW, we should stick to
approach 'A'.
J.
8:21 a.m.
Well, after more discussions with kparal, we are still unsure about the
"right" way to tackle this.
Our current call would be:
1) sync requirements.txt versions with fedora (mostly done)
2) allow --system-site-packages in the test_env
3) do `pip install -r requirements.txt` (with possible flags to enforce
versions) to the makefile virtualenv creation step
4) add info to readme, that testing needs installation of packages from
pypi, and that some of them need compilation
4-1) put together a list of packages that need to be installed (the
python-foobar kind, not -devel + gcc) to the system, in order to "skip" the
stuff that needs to be compiled
Sounds reasonable, Kamil? Others?
Joza
On Mon, Feb 6, 2017 at 2:11 PM, Kamil Paral <kparal(a)redhat.com> wrote:
3. use a separate virtualenv when running under `make test`, without
--system-site-packages if possible, and ensure up-to-date deps are always
installed, to eliminate any differences that can occur on different setups
The only problem I see here, is that some of the packages that you'd need
to install into the test-virtualenv need some C-compilation when installing
from PyPi, and that (if used without the --system-site-packages) would mean
having to install not only gcc but also plenty of -dev packages.
Not a _huge_ ussue but an issue nevertheless.
That's a good point. But do we have a good alternative here? If we depend
on packages like that, I see only two options:
a) ask the person to install pyfoo as an RPM (in readme)
b) ask the person to install gcc and libfoo-devel as an RPM (in readme)
and pyfoo will be then compiled and installed from pypi
Approach a) is somewhat easier and does not require compilation stack and
devel libraries. OTOH it requires using virtualenv with
--system-site-packages, which means people get different results on
different setups. That's exactly what I'm trying to eliminate (or at least
reduce). E.g. https://phab.qa.fedoraproject.org/D1111 where I can run the
test suite from makefile and you can't, and it's quite difficult to figure
out why.
With b) approach, you need compilation stack on the system. I don't think
it's such a huge problem, because you're a developer after all. The
advantage is that virtualenv can be created without --system-site-packages,
which means locally installed libraries do not affect the execution/test
suite results. Also, pyfoo is installed with exactly the right version,
further reducing differences between setups. The only thing that can differ
is the version of libfoo-devel, which can affect the behavior. But the
likeliness of that happening is much smaller than having pyfoo of a
different version or pulling any deps from the system site packages.
Sigh, OTOH Josef has a very good point in https://phab.qa.fedoraproject.
org/D1112#20744 that figuring out which devel packages are needed on the
system for pypi module compilation is quite a non-trivial task. Seems
harder than figuring out which packages you need to install on the system
when using --system-site-packages.
So, I don't really see a simple solution here that would guard us from
inconsistent dev setups (system site packages) or not require us to have a
painful deps maintenance (no system site packages, but devel packages) :/
Even if we decide to keep --system-site-packages for simplicity, I can
implement all the rest of the proposed improvements. Just the reliability
will not be as great as I wished.
The main question is now, I believe, whether we want to install
compilation-dependent packages from pypi or rpm.
_______________________________________________
qa-devel mailing list -- qa-devel(a)lists.fedoraproject.org
To unsubscribe send an email to qa-devel-leave(a)lists.fedoraproject.org
11:04 a.m.
Well, after more discussions with kparal, we are still unsure about
the
"right" way to tackle this.
Our current call would be:
1) sync requirements.txt versions with fedora (mostly done)
2) allow --system-site-packages in the test_env
3) do `pip install -r requirements.txt` (with possible flags to enforce
versions) to the makefile virtualenv creation step
4) add info to readme, that testing needs installation of packages from pypi,
and that some of them need compilation
4-1) put together a list of packages that need to be installed (the
python-foobar kind, not -devel + gcc) to the system, in order to "skip" the
stuff that needs to be compiled
Sounds reasonable, Kamil? Others?
I went back and forth on this. I thought it would be a really simple change, and as usual,
it seems more pain than gain. So, I went forward with this:
1. add tox.ini to projects to allow simple test suite execution with `pytest`
(non-controversial)
2. configure tox.ini to print out test coverage (non-controversial)
3. remove --system-site-packages from all places (readme, makefile) for those projects,
that can be *fully* installed from pypi *without any compilation* (hopefully
non-controversial).
4. keep (or add) --system-site-packages to readme/makefile for the remaining projects, and
add readme info how to deal with pypi compilation or local rpm installation
What Josef mentioned is that he wouldn't try to replicate a perfect environment
directly on dev machine, because that's a lot of work. Instead, use the current
non-perfect environment on dev machines (which should be fine most of the time anyway) and
have a separate CI service (hopefully in the future) with more strict environment
configuration. I guess that's the most practical solution.
We might even want to reopen the question how to version deps in requirements.txt vs spec
file, but I'd keep that for a separate thread, if needed.
My current patches for resultsdb projects are these:
https://phab.qa.fedoraproject.org/D1114
https://phab.qa.fedoraproject.org/D1116
https://phab.qa.fedoraproject.org/D1117
4:02 p.m.
+1 I'm glad that our usual flame-war yielded a common ground that we can
agree upon :) Hope that it was not as painfull as usual.
J.
On Mon, Feb 6, 2017 at 6:04 PM, Kamil Paral <kparal(a)redhat.com> wrote:
Well, after more discussions with kparal, we are still unsure about
the
"right" way to tackle this.
Our current call would be:
1) sync requirements.txt versions with fedora (mostly done)
2) allow --system-site-packages in the test_env
3) do `pip install -r requirements.txt` (with possible flags to enforce
versions) to the makefile virtualenv creation step
4) add info to readme, that testing needs installation of packages from
pypi, and that some of them need compilation
4-1) put together a list of packages that need to be installed (the
python-foobar kind, not -devel + gcc) to the system, in order to "skip" the
stuff that needs to be compiled
Sounds reasonable, Kamil? Others?
I went back and forth on this. I thought it would be a really simple
change, and as usual, it seems more pain than gain. So, I went forward with
this:
1. add tox.ini to projects to allow simple test suite execution with
`pytest` (non-controversial)
2. configure tox.ini to print out test coverage (non-controversial)
3. remove --system-site-packages from all places (readme, makefile) for
those projects, that can be *fully* installed from pypi *without any
compilation* (hopefully non-controversial).
4. keep (or add) --system-site-packages to readme/makefile for the
remaining projects, and add readme info how to deal with pypi compilation
or local rpm installation
What Josef mentioned is that he wouldn't try to replicate a perfect
environment directly on dev machine, because that's a lot of work.
Instead, use the current non-perfect environment on dev machines (which
should be fine most of the time anyway) and have a separate CI service
(hopefully in the future) with more strict environment configuration. I
guess that's the most practical solution.
We might even want to reopen the question how to version deps in
requirements.txt vs spec file, but I'd keep that for a separate thread, if
needed.
My current patches for resultsdb projects are these:
https://phab.qa.fedoraproject.org/D1114
https://phab.qa.fedoraproject.org/D1116
https://phab.qa.fedoraproject.org/D1117
_______________________________________________
qa-devel mailing list -- qa-devel(a)lists.fedoraproject.org
To unsubscribe send an email to qa-devel-leave(a)lists.fedoraproject.org
8:41 a.m.
On Fri, 3 Feb 2017 17:05:04 -0500 (EST)
Kamil Paral <kparal(a)redhat.com> wrote:
I spent a bit of time fixing minor issues in our test suite and
makefiles and would like to do the following further changes across
all our taskotron projects:
1. run the test suite while inside virtualenv with simple `pytest`
command
2. run the test suite outside of virtualenv with `make test` or `doit
test` and recommend this approach in readme
3. use a separate virtualenv when running under `make test`, without
--system-site-packages if possible, and ensure up-to-date deps are
always installed, to eliminate any differences that can occur on
different setups
4. configure pytest to print out coverage by default, so that it
shows up for both `pytest` and `make test`
Any concerns or different suggestions?
Most of this makes sense but I wanted to propose some small tweaks to
what you're talking about here.
tox is designed to deal with virtualenv creation/manipulation, I wonder
if it would be wise to start going in that direction instead of doing
it in a makefile.
I had been hoping to take the dodo.py file from libtaskotron and use
most of it across all of our related projects but doit is a much less
common tool than make is. I do think that many things would be easier
to do with doit than make, though:
- we could check for the presense of required RPMs with doit, though -
something along the lines of 'doit test' coming back with "you need
to install X, Y and Z rpms before running the test suite"
- the docs building that I built into the buildbot on qa.fp.o pretty
much needs doit and some of the generic-ish methods in the
libtaskotron dodo.py file.
I wonder if effort would be better spent getting those dodo.py files
working instead of improving the makefiles.
Either way, it'd be nice to improve things a bit.
Tim
2659
days inactive
2664
days old
qa-devel@lists.fedoraproject.org
8 comments
3 participants
participants (3)
-
Josef Skladanka -
Kamil Paral -
Tim Flink