Cython build failing on longintrepr.h not found

[Sandro]

Hi Pythonians,

I have a favor to ask. I'm trying to package scikit-misc as a dependency 
for plotnine (for neuro-sig).

The build fails with:

fatal error: longintrepr.h: No such file or directory

I'm aware of the fact that the file has been moved in Python 3.11, but 
was under the impression this change was implemented in Cython 0.29.5.

I opened an issue upstream, but I would appreciate a second pair of eyes 
on the build. The file that tries to include the header file is 
generated, so nothing I could work around with using a patch (with my 
limited knowledge that is).

Build: 
https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4869979/
Issue: https://github.com/has2k1/scikit-misc/issues/17

-- Sandro

[Lumír Balhar]

Hi Sandro.

Make sure that the build does not use the pyx file from upstream. It 
seems to me that the file generated by Cython is in the source tarball 
(skmisc/loess/src/_loess.pyx) and I did not find any mention of use of 
Cython in the build log. The file is probably generated by an older 
version of Cython and that is causing you the problem.

Hope this helps.

Have a nice day.

Lumír

On 9/27/22 01:48, Sandro wrote:
> Hi Pythonians,
>
> I have a favor to ask. I'm trying to package scikit-misc as a 
> dependency for plotnine (for neuro-sig).
>
> The build fails with:
>
> fatal error: longintrepr.h: No such file or directory
>
> I'm aware of the fact that the file has been moved in Python 3.11, but 
> was under the impression this change was implemented in Cython 0.29.5.
>
> I opened an issue upstream, but I would appreciate a second pair of 
> eyes on the build. The file that tries to include the header file is 
> generated, so nothing I could work around with using a patch (with my 
> limited knowledge that is).
>
> Build: 
> https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4869979/
> Issue: https://github.com/has2k1/scikit-misc/issues/17
>
> -- Sandro
> _______________________________________________
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue

[Petr Viktorin]

The guidelines have more info: 
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_using_cyt...

On 27. 09. 22 8:17, Lumír Balhar wrote:
> Hi Sandro.
>
> Make sure that the build does not use the pyx file from upstream. It 
> seems to me that the file generated by Cython is in the source tarball 
> (skmisc/loess/src/_loess.pyx) and I did not find any mention of use of 
> Cython in the build log. The file is probably generated by an older 
> version of Cython and that is causing you the problem.
>
> Hope this helps.
>
> Have a nice day.
>
> Lumír
>
> On 9/27/22 01:48, Sandro wrote:
> > Hi Pythonians,
> >
> > I have a favor to ask. I'm trying to package scikit-misc as a 
> > dependency for plotnine (for neuro-sig).
> >
> > The build fails with:
> >
> > fatal error: longintrepr.h: No such file or directory
> >
> > I'm aware of the fact that the file has been moved in Python 3.11, but 
> > was under the impression this change was implemented in Cython 0.29.5.
> >
> > I opened an issue upstream, but I would appreciate a second pair of 
> > eyes on the build. The file that tries to include the header file is 
> > generated, so nothing I could work around with using a patch (with my 
> > limited knowledge that is).
> >
> > Build: 
> > https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4869979/
> > Issue: https://github.com/has2k1/scikit-misc/issues/17
> >
> > -- Sandro
> > _______________________________________________
> > python-devel mailing list -- [e-mail]
> > To unsubscribe send an email to 
> > [e-mail]
> > Fedora Code of Conduct: 
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/[e-mail]...
> > Do not reply to spam, report it: 
> > https://pagure.io/fedora-infrastructure/new_issue
>
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue

[Sandro]

On 27-09-2022 08:17, Lumír Balhar wrote:
> Make sure that the build does not use the pyx file from upstream. It
> seems to me that the file generated by Cython is in the source tarball
> (skmisc/loess/src/_loess.pyx) and I did not find any mention of use of
> Cython in the build log. The file is probably generated by an older
> version of Cython and that is causing you the problem.

Thank you, Lumír, for pointing me in the right direction.

The offending pregenerated C file was indeed in the source tarball along 
with a precompiled library for good measure.

I was looking at the source on GitHub, which does not have 
skmisc/loess/src/_loess.c. I'm sure that's what you meant. It's 
generated from skmisc/loess/src/_loess.pyx

I have switched to the GitHub source tarball, which seems to be aimed at 
developers, who like to build everything from scratch, and comes without 
pregenerated files nor binaries.

However, I haven't been able to build the package, yet. It looks like 
the tools/cythonize.py script, which is called from setup.py, generates 
output, that throws off %pyproject_buildrequires:

No matching package to install: 'Processing'
No matching package to install: 'changed'

Is there a standard way of handling noisy scripts? Or am I just out of 
luck using pyproject macros? Or, my bet, am I missing something?

https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4875750/

-- Sandro

[Miro Hrončok]

On 27. 09. 22 17:55, Sandro wrote:
> On 27-09-2022 08:17, Lumír Balhar wrote:
> > Make sure that the build does not use the pyx file from upstream. It
> > seems to me that the file generated by Cython is in the source tarball
> > (skmisc/loess/src/_loess.pyx) and I did not find any mention of use of
> > Cython in the build log. The file is probably generated by an older
> > version of Cython and that is causing you the problem.
>
> Thank you, Lumír, for pointing me in the right direction.
>
> The offending pregenerated C file was indeed in the source tarball along with a 
> precompiled library for good measure.
>
> I was looking at the source on GitHub, which does not have 
> skmisc/loess/src/_loess.c. I'm sure that's what you meant. It's generated from 
> skmisc/loess/src/_loess.pyx
>
> I have switched to the GitHub source tarball, which seems to be aimed at 
> developers, who like to build everything from scratch, and comes without 
> pregenerated files nor binaries.
>
> However, I haven't been able to build the package, yet. It looks like the 
> tools/cythonize.py script, which is called from setup.py, generates output, 
> that throws off %pyproject_buildrequires:
>
> No matching package to install: 'Processing'
> No matching package to install: 'changed'
>
> Is there a standard way of handling noisy scripts? Or am I just out of luck 
> using pyproject macros? Or, my bet, am I missing something?
>
> https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4875750/

Looking at the code, this happens because:

1) their setup.py uses subprocess

https://github.com/has2k1/scikit-misc/blob/v0.1.4/setup.py#L92

2) the script executed via subprocess prints to stdout:

https://github.com/has2k1/scikit-misc/blob/v0.1.4/tools/cythonize.py#L177

3) %pyproject_buildrequires uses contextlib.redirect_stdout:

https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/a3e31c/f/pyproj...

4) contextlib.redirect_stdout does not support stdout of subprocesses

https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout

We might need a more robust way of redirecting all of stdout:

https://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python...

As a temporary workaround, you might need to sed/patch the prints away or 
convince upstream to print status information to stderr.

[Sandro]

On 27-09-2022 19:41, Miro Hrončok wrote:
> On 27. 09. 22 17:55, Sandro wrote:
> > On 27-09-2022 08:17, Lumír Balhar wrote:
> > > Make sure that the build does not use the pyx file from upstream. It
> > > seems to me that the file generated by Cython is in the source tarball
> > > (skmisc/loess/src/_loess.pyx) and I did not find any mention of use of
> > > Cython in the build log. The file is probably generated by an older
> > > version of Cython and that is causing you the problem.
> >
> > Thank you, Lumír, for pointing me in the right direction.
> >
> > The offending pregenerated C file was indeed in the source tarball along with a
> > precompiled library for good measure.
> >
> > I was looking at the source on GitHub, which does not have
> > skmisc/loess/src/_loess.c. I'm sure that's what you meant. It's generated from
> > skmisc/loess/src/_loess.pyx
> >
> > I have switched to the GitHub source tarball, which seems to be aimed at
> > developers, who like to build everything from scratch, and comes without
> > pregenerated files nor binaries.
> >
> > However, I haven't been able to build the package, yet. It looks like the
> > tools/cythonize.py script, which is called from setup.py, generates output,
> > that throws off %pyproject_buildrequires:
> >
> > No matching package to install: 'Processing'
> > No matching package to install: 'changed'
> >
> > Is there a standard way of handling noisy scripts? Or am I just out of luck
> > using pyproject macros? Or, my bet, am I missing something?
> >
> > https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4875750/
>
> Looking at the code, this happens because:
> their setup.py uses subprocess

<snip>
> We might need a more robust way of redirecting all of stdout:
>
> https://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python...
>
> As a temporary workaround, you might need to sed/patch the prints away or
> convince upstream to print status information to stderr.

Thanks, Miro, I managed to get past the build stage and opened a PR 
upstream.

-- Sandro

[Miro Hrončok]

On 27. 09. 22 19:41, Miro Hrončok wrote:
> On 27. 09. 22 17:55, Sandro wrote:
> > On 27-09-2022 08:17, Lumír Balhar wrote:
> > > Make sure that the build does not use the pyx file from upstream. It
> > > seems to me that the file generated by Cython is in the source tarball
> > > (skmisc/loess/src/_loess.pyx) and I did not find any mention of use of
> > > Cython in the build log. The file is probably generated by an older
> > > version of Cython and that is causing you the problem.
> >
> > Thank you, Lumír, for pointing me in the right direction.
> >
> > The offending pregenerated C file was indeed in the source tarball along with 
> > a precompiled library for good measure.
> >
> > I was looking at the source on GitHub, which does not have 
> > skmisc/loess/src/_loess.c. I'm sure that's what you meant. It's generated 
> > from skmisc/loess/src/_loess.pyx
> >
> > I have switched to the GitHub source tarball, which seems to be aimed at 
> > developers, who like to build everything from scratch, and comes without 
> > pregenerated files nor binaries.
> >
> > However, I haven't been able to build the package, yet. It looks like the 
> > tools/cythonize.py script, which is called from setup.py, generates output, 
> > that throws off %pyproject_buildrequires:
> >
> > No matching package to install: 'Processing'
> > No matching package to install: 'changed'
> >
> > Is there a standard way of handling noisy scripts? Or am I just out of luck 
> > using pyproject macros? Or, my bet, am I missing something?
> >
> > https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4875750/
>
> Looking at the code, this happens because:
> their setup.py uses subprocess
>
> https://github.com/has2k1/scikit-misc/blob/v0.1.4/setup.py#L92
> the script executed via subprocess prints to stdout:
>
> https://github.com/has2k1/scikit-misc/blob/v0.1.4/tools/cythonize.py#L177
> %pyproject_buildrequires uses contextlib.redirect_stdout:
>
> https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/a3e31c/f/pyproj...
> contextlib.redirect_stdout does not support stdout of subprocesses
>
> https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout
>
> We might need a more robust way of redirecting all of stdout...

And this now finally has a fix ready:

https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/353

[Sandro]

On 07-02-2023 01:10, Miro Hrončok wrote:
> > contextlib.redirect_stdout does not support stdout of subprocesses
> >
> > https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout
> >
> > We might need a more robust way of redirecting all of stdout...
>
> And this now finally has a fix ready:
>
> https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/353

Thanks Miro. Upstream actually fixed that and you contributed to it. [1] 
Unfortunately, there has not been a release since. That's why the patch 
was still needed.

[1] https://github.com/has2k1/scikit-misc/pull/18

Cheers,

Sandro