Question regarding setuptools automatic discovery (pyproject.toml)

[Sandro]

Hi,

I ran into a build failure for a Python package that dropped setup.py in 
the latest update and uses pyproject.toml for metadata and setuptools.

The build failed due to 'error: Installed (but unpackaged) file(s) 
found' [1]. However, all the erroneously installed modules should be 
excluded by default, if I understand the setuptools automatic discovery 
documentation [2] correctly. They are listed in 
'FlatLayoutPackageFinder.DEFAULT_EXCLUDE'. In this case 'docs', 
'scripts' and 'test'.

Since this is still beta when using pyproject.toml, I was wondering if 
someone else has come across this misbehavior. Or, maybe, it's something 
I failed to spot in pyproject.toml, that's causing it.

I can make it work by adding a 'include = ["palettable"]' in the 
tool.setuptools.packages.find table [3]. But I'd like a second opinion 
before submitting a PR upstream.

[1] https://kojipkgs.fedoraproject.org//work/tasks/5770/101775770/build.log
[2] 
https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#flat-l...
[3] https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/6000976/

Cheers,

[Miro Hrončok]

On 04. 06. 23 1:54, Sandro wrote:
> Hi,

Hey.
> I ran into a build failure for a Python package that dropped setup.py in the 
> latest update and uses pyproject.toml for metadata and setuptools.
>
> The build failed due to 'error: Installed (but unpackaged) file(s) found' [1]. 
> However, all the erroneously installed modules should be excluded by default, 
> if I understand the setuptools automatic discovery documentation [2] correctly. 
> They are listed in 'FlatLayoutPackageFinder.DEFAULT_EXCLUDE'. In this case 
> 'docs', 'scripts' and 'test'.
>
> Since this is still beta when using pyproject.toml, I was wondering if someone 
> else has come across this misbehavior. Or, maybe, it's something I failed to 
> spot in pyproject.toml, that's causing it.
>
> I can make it work by adding a 'include = ["palettable"]' in the 
> tool.setuptools.packages.find table [3]. But I'd like a second opinion before 
> submitting a PR upstream.

First of all, upstream wheel downloaded from 
https://pypi.org/project/palettable/3.3.3/#files includes build, docs and test 
as well.

I think the elicit exclude configured at 
https://github.com/jiffyclub/palettable/blob/v3.3.3/pyproject.toml#L35 migth 
override the DEFAULT_EXCLUDE value.

I don't know how tool.setuptools.packages.find.exclude behaves, but I have seen 
this behavior with pytest, where the default was to exclude a bunch of 
directories including ".*", but projects that set their own excludes (without 
".*")  suddenly started to collect tests from our .pyproject-* folders.

[Sandro]

On 04-06-2023 21:27, Miro Hrončok wrote:
> > I ran into a build failure for a Python package that dropped setup.py 
> > in the latest update and uses pyproject.toml for metadata and setuptools.
> >
> > The build failed due to 'error: Installed (but unpackaged) file(s) 
> > found' [1]. However, all the erroneously installed modules should be 
> > excluded by default, if I understand the setuptools automatic 
> > discovery documentation [2] correctly. They are listed in 
> > 'FlatLayoutPackageFinder.DEFAULT_EXCLUDE'. In this case 'docs', 
> > 'scripts' and 'test'.
> >
> > Since this is still beta when using pyproject.toml, I was wondering if 
> > someone else has come across this misbehavior. Or, maybe, it's 
> > something I failed to spot in pyproject.toml, that's causing it.
> >
> > I can make it work by adding a 'include = ["palettable"]' in the 
> > tool.setuptools.packages.find table [3]. But I'd like a second opinion 
> > before submitting a PR upstream.
>
> First of all, upstream wheel downloaded from 
> https://pypi.org/project/palettable/3.3.3/#files includes build, docs 
> and test as well.

I know. I tried a local 'pthon3 -m build' and the wheel was stuffed with 
everything. Terrible oversight by upstream, it seems.
> I think the elicit exclude configured at 
> https://github.com/jiffyclub/palettable/blob/v3.3.3/pyproject.toml#L35 
> migth override the DEFAULT_EXCLUDE value.

That sounds plausible. I also tried modifying pyproject.toml not using 
include or exclude. Thus relying on the automatic find mechanism, but I 
couldn't get it to work.

The exclude is kinda a no-op, since there are no files named *.test 
anywhere.

Adding an explicit include is probably the easiest way out. I'll suggest 
that upstream.
> I don't know how tool.setuptools.packages.find.exclude behaves, but I 
> have seen this behavior with pytest, where the default was to exclude a 
> bunch of directories including ".*", but projects that set their own 
> excludes (without ".*")  suddenly started to collect tests from our 
> .pyproject-* folders.

Thanks for the feedback. I'll keep that in mind.

-- Sandro

[Mikel Olasagasti]

Hi Sandro,

Hau idatzi du Sandro ([e-mail]) erabiltzaileak (2023 eka.
4(a), ig. (01:55)):
> Hi,
>
> I ran into a build failure for a Python package that dropped setup.py in
> the latest update and uses pyproject.toml for metadata and setuptools.
>
> The build failed due to 'error: Installed (but unpackaged) file(s)
> found' [1]. However, all the erroneously installed modules should be
> excluded by default, if I understand the setuptools automatic discovery
> documentation [2] correctly. They are listed in
> 'FlatLayoutPackageFinder.DEFAULT_EXCLUDE'. In this case 'docs',
> 'scripts' and 'test'.
>
> Since this is still beta when using pyproject.toml, I was wondering if
> someone else has come across this misbehavior. Or, maybe, it's something
> I failed to spot in pyproject.toml, that's causing it.
>
> I can make it work by adding a 'include = ["palettable"]' in the
> tool.setuptools.packages.find table [3]. But I'd like a second opinion
> before submitting a PR upstream.

I ran to the same problem updating pg_activity last week. After a
conversation with upstream[1] and exploring different options they
preferred to remove packages.find.

Regards,
Mikel

[1] https://github.com/dalibo/pg_activity/pull/378

[Sandro]

On 05-06-2023 11:19, Mikel Olasagasti wrote:
> > I can make it work by adding a 'include = ["palettable"]' in the
> > tool.setuptools.packages.find table [3]. But I'd like a second opinion
> > before submitting a PR upstream.
>
> I ran to the same problem updating pg_activity last week. After a
> conversation with upstream[1] and exploring different options they
> preferred to remove packages.find.

That's another option. It seems the automatic find using pyproject.toml 
is not working as expected. But again, it's still BETA.

I have it working in one of my own packages [1]. But I'm using setup.cfg 
for backwards compatibility with older setuptools. Otherwise, the 
package cannot be build for EL.

[1] https://github.com/munin-monitoring/PyMunin3

-- Sandro