Macronize %pytest

[Miro Hrončok]

Hello Python packagers,

since there is no upstream supported universal test invocation for Python 
(`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't 
always do what we want in RPM's %check and/or is not always used by upstreams) 
and a lot of upstreams use pytest, I'd like to propose a %pytest macros, for 
standardizing the way pytest is run.

Usage:

   %check
   %pytest

Or with options passed to pyets:

   %check
   %pytest -v -m "not network" tests/


(Here ends the tl;dr version.)


Why not just use `pytest` or `%{python3} -m pytest` instead of `%pytest`?

We want to *test the code we ship*. In the general case this involves:

  1. Prepending sys.path with %{buildroot}%{python3_sitelib} (and/or sitearch)
  2. Prepending $PATH with %{buildroot}%{_bindir}
  3. Ensuring $PWD is not in sys.path


So the idea is (untested):

%pytest_cmd /usr/bin/pytest

%pytest %{expand:
PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"

\
PATH="%{buildroot}%{_bindir}:$PATH" \
%pytest_cmd
}


Where PYTHONPATH solves (1), PATH solves (2) and using `/usr/bin/pytest` over 
`%{python3} -m pytest` kinda solves (3).

I say "kinda" because using `python -m pytest` explicitly adds $PWD to sys.path,

but using `pytest` doesn't always prevent it.

https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-pytho...

I've been trying to completely solve (3) in my head in a general way, but it 
always involves `cd`ing to a temporary directory before running the tests, but 
plenty of test suites cannot handle that gracefully.

Unfortunately Python interpreter doesn't have a flag that says "don't add 
current directory to sys.path, but respect $PYTHONPATH". In the long term, we 
might propose to add it.



Where to have the macro defined? (Skip this part if not interested.)

I was thinking:

  a) pytest package
  b) pytest subpackage (required if rpm-build)
  c) python3-rpm-macros
  d) python-rpm-macros
  e) pyproject-rpm-macros

And here is the pros and cons:

a+) self contained, can change with upstream if needed
a+) always present with pytest
a-) pytest must co-own macro dir or depend on rpm-build


b+) same as a+, but not a-
b-) additional layer of complexity

c+) easy
c+) the macro doesn't need to require pytest, can be used with arbitrary command
c+) can be generalized to non-pytest later if desired
c-) when pytest CLI changes significantly, we need to change it in different 
package (but we are not using pytets CLI now at all, just Python/Shell 
environment variables)
c~) technically not always present with pytest, but always present with 
python3-devel

d+-) same as (c)
d-) the macro uses python3 specific things

e+) the new fancy macros are there
e-) the other macros from there (actually pyproject related) are not required 
for this
e-) not always present with pytest nor python3-devel


Hence, I'd go with (c) python3-rpm-macros.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Scott Talbert]

On Mon, 27 Apr 2020, Miro Hrončok wrote:

>  Hello Python packagers,
>
>  since there is no upstream supported universal test invocation for Python 
>  (`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't 
>  always do what we want in RPM's %check and/or is not always used by 
>  upstreams) and a lot of upstreams use pytest, I'd like to propose a %pytest 
>  macros, for standardizing the way pytest is run.
>
>  Usage:
>
>   %check
>   %pytest
>
>  Or with options passed to pyets:
>
>   %check
>   %pytest -v -m "not network" tests/
>
>
>  (Here ends the tl;dr version.)
>
>
>  Why not just use `pytest` or `%{python3} -m pytest` instead of `%pytest`?
>
>  We want to *test the code we ship*. In the general case this involves:
>
>  1. Prepending sys.path with %{buildroot}%{python3_sitelib} (and/or sitearch)
>  2. Prepending $PATH with %{buildroot}%{_bindir}
>  3. Ensuring $PWD is not in sys.path
>
>
>  So the idea is (untested):
>
>  %pytest_cmd /usr/bin/pytest
>
>  %pytest %{expand:
>
> PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"
>
>  \
>  PATH="%{buildroot}%{_bindir}:$PATH" \
>  %pytest_cmd
>  }
>
>
>  Where PYTHONPATH solves (1), PATH solves (2) and using `/usr/bin/pytest` over 
>  `%{python3} -m pytest` kinda solves (3).
>
>  I say "kinda" because using `python -m pytest` explicitly adds $PWD to 
>  sys.path, but using `pytest` doesn't always prevent it.
>
>  https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-pytho...
>
>  I've been trying to completely solve (3) in my head in a general way, but it 
>  always involves `cd`ing to a temporary directory before running the tests, 
>  but plenty of test suites cannot handle that gracefully.
>
>  Unfortunately Python interpreter doesn't have a flag that says "don't add 
>  current directory to sys.path, but respect $PYTHONPATH". In the long term, we 
>  might propose to add it. 
I like the idea!

I have run into the issue of not wanting $PWD in sys.path before as well, 
so it would be great to solve that.

The only other thing that I might suggest is to 
set PYTHONDONTWRITEBYTECODE=1 so we don't get pytest bytecode written out.

Scott

[Miro Hrončok]

On 27. 04. 20 15:47, Scott Talbert wrote:
>  On Mon, 27 Apr 2020, Miro Hrončok wrote:
>  
> > Hello Python packagers,
> >
> > since there is no upstream supported universal test invocation for Python 
> > (`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't 
> > always do what we want in RPM's %check and/or is not always used by upstreams) 
> > and a lot of upstreams use pytest, I'd like to propose a %pytest macros, for 
> > standardizing the way pytest is run.
> >
> > Usage:
> >
> >  %check
> >  %pytest
> >
> > Or with options passed to pyets:
> >
> >  %check
> >  %pytest -v -m "not network" tests/
> >
>  
>  I like the idea!
>  
>  I have run into the issue of not wanting $PWD in sys.path before as well, so it 
>  would be great to solve that. 
Will pu(n)t it to my far future TODO.

>  The only other thing that I might suggest is to set
> PYTHONDONTWRITEBYTECODE=1 so 
>  we don't get pytest bytecode written out. 
Good point, will do. Thanks.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 27. 04. 20 11:34, Miro Hrončok wrote:
>  Hello Python packagers,
>  
>  since there is no upstream supported universal test invocation for Python 
>  (`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't 
>  always do what we want in RPM's %check and/or is not always used by upstreams) 
>  and a lot of upstreams use pytest, I'd like to propose a %pytest macros, for 
>  standardizing the way pytest is run.
>  
>  Usage:
>  
>     %check
>     %pytest
>  
>  Or with options passed to pyets:
>  
>     %check
>     %pytest -v -m "not network" tests/
>  
>  
>  (Here ends the tl;dr version.)
>  
>  
>  Why not just use `pytest` or `%{python3} -m pytest` instead of `%pytest`?
>  
>  We want to *test the code we ship*. In the general case this involves:
>  
>    1. Prepending sys.path with %{buildroot}%{python3_sitelib} (and/or sitearch)
>    2. Prepending $PATH with %{buildroot}%{_bindir}
>    3. Ensuring $PWD is not in sys.path
>  
>  
>  So the idea is... 
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/55

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok