flit

[Orion Poplawski]

So - is it possible to build rpms with flit?

[Nick Coghlan]

On 16 November 2017 at 12:30, Orion Poplawski  wrote:
> So - is it possible to build rpms with flit?

I haven't checked recently, but I believe flit still omits `setup.py` from
its sdists by default, and the Python RPM macros are still expecting to be
able to call that directly.

If that's still the case, then the answer would be "Yes, but only if you
make sure the sdist still contains a suitable setup.py shim".

However, if flit is now adding its own shim implcitly, then the answer
would just be "Yes".

Cheers,
Nick.

[Elliott Sales de Andrade]

On 16 November 2017 at 01:31, Nick Coghlan  wrote:
> On 16 November 2017 at 12:30, Orion Poplawski  wrote:
> > So - is it possible to build rpms with flit?

There are already some packages that are built using flit:

$ dnf repoquery --disablerepo=* --enablerepo=fedora-source --arch=src
--whatrequires python3-flit
python-entrypoints-0:0.2.2-7.fc26.src
python-testpath-0:0.3.1-1.fc26.src

The latter is a slightly better template as it doesn't contain any
workarounds for old versions of flit.
Another sample is python-jupyter-kernel-test (not sure why it didn't show;
it should be in F26, too).
> I haven't checked recently, but I believe flit still omits `setup.py` from
> its sdists by default, and the Python RPM macros are still expecting to be
> able to call that directly.
>
> If that's still the case, then the answer would be "Yes, but only if you
> make sure the sdist still contains a suitable setup.py shim".
>
> However, if flit is now adding its own shim implcitly, then the answer
> would just be "Yes".

There's no need for a shim; basically use flit to build a wheel and then
use the wheel installation macros instead.
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   [e-mail]   |   Brisbane, Australia

[Nick Coghlan]

On 16 November 2017 at 16:51, Elliott Sales de Andrade <
[e-mail]> wrote:
> On 16 November 2017 at 01:31, Nick Coghlan  wrote:
> > However, if flit is now adding its own shim implcitly, then the answer
> > would just be "Yes".
> There's no need for a shim; basically use flit to build a wheel and then
> use the wheel installation macros instead.

Wait, we have wheel installation macros?

/me goes and checks
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...

Huh, so we do. Somehow I managed to miss that :)

How would folks feel about adding a link to the dist-git macro file to the
macros section in https://fedoraproject.org/wiki/Packaging:Python#Macros ?

I'm not currently inclined to add the information directly to the wiki page
as:

- I don't like wikis in general as a docs management tool
- the wiki only shows one version, but the RPM with the default macros is
branched along with the rest of Fedora

We could potentially add something to https://fedora-python.readthedocs.io/
about Fedora's RPM packaging tools for Python projects, but that would be
significantly more work than just linking to dist-git.

Cheers,
Nick.

[Miro Hrončok]

On 16.11.2017 08:17, Nick Coghlan wrote:
> On 16 November 2017 at 16:51, Elliott Sales de Andrade 
> <[e-mail] > wrote:
> On 16 November 2017 at 01:31, Nick Coghlan <[e-mail]
> <mailto:[e-mail]>> wrote:
>
>     However, if flit is now adding its own shim implcitly, then the
>     answer would just be "Yes".
>
> There's no need for a shim; basically use flit to build a wheel and
> then use the wheel installation macros instead.
>
>
> Wait, we have wheel installation macros?
>
> /me goes and checks 
> https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
>
> Huh, so we do. Somehow I managed to miss that :)
>
> How would folks feel about adding a link to the dist-git macro file to 
> the macros section in 
> https://fedoraproject.org/wiki/Packaging:Python#Macros ?
>
> I'm not currently inclined to add the information directly to the wiki 
> page as:
> I don't like wikis in general as a docs management tool
> the wiki only shows one version, but the RPM with the default macros
>
> is branched along with the rest of Fedora

Adding the link makes sense to me. Adding all the macros definition to 
the wiki does not make sense to me, but form different reasons. I think 
that having %py3_build_egg and %py3_install_egg there is just not 
necessary. Since there are more files at [0] I'd just add that link.

[0] https://src.fedoraproject.org/rpms/python-rpm-macros/tree/master

About flit:

* is this way of creating Python distributions getting more and
    more popular?
  * if so, should we add a new section of the guidelines? something like
    "Packaging setup.py-less projects"?
> We could potentially add something to 
> https://fedora-python.readthedocs.io/ about Fedora's RPM packaging tools 
> for Python projects, but that would be significantly more work than just 
> linking to dist-git.
>
> Cheers,
> Nick.
>
> -- 
> Nick Coghlan   | [e-mail]    |   
> Brisbane, Australia
>
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Nick Coghlan]

On 16 November 2017 at 22:33, Miro Hrončok  wrote:
> Adding the link makes sense to me. Adding all the macros definition to the
> wiki does not make sense to me, but form different reasons. I think that
> having %py3_build_egg and %py3_install_egg there is just not necessary.
> Since there are more files at [0] I'd just add that link.
>
> [0] https://src.fedoraproject.org/rpms/python-rpm-macros/tree/master

Even though it's just a new informational link, I'm guessing we still need
to file an FPC ticket for that?
> About flit:
> is this way of creating Python distributions getting more and
> more popular?

It is - Thomas Kluyver (flit's creator) is actually driving a lot of work
to get pip's pluggable build system support working properly.

The key differences between it and setuptools are:

1. It's designed specifically for pure Python projects
2. It assumes it can get the list of files to package from git

Those two assumptions sweep away a *lot* of the complexity in setuptools,
and hence make flit far more beginner friendly.
> if so, should we add a new section of the guidelines? something like
> "Packaging setup.py-less projects"?

Rather than emphasising the absence of setup.py, I'd emphasise the use of
wheel files:

* "Defining an RPM based on a wheel build process"
* "Defining an RPM based on a setup.py file"

The latter would just be a renamed
https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file -
that is the setup.py based build process, but it isn't currently obvious
that `pyX_build` and `pyX_install` assume the use of a setup.py file.

Cheers,
Nick.

[Charalampos Stratakis]

----- Original Message -----
> From: "Nick Coghlan" 
> To: "Fedora Python SIG" 
> Sent: Thursday, November 16, 2017 1:59:24 PM
> Subject: Re: flit
> Even though it's just a new informational link, I'm guessing we still need to
> file an FPC ticket for that?

That is correct, ideally with a draft page of the proposed changes.

Regards,

Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat

[Toshio Kuratomi]

On Nov 16, 2017 4:59 AM, "Nick Coghlan"  wrote:

On 16 November 2017 at 22:33, Miro Hrončok  wrote:
> Adding the link makes sense to me. Adding all the macros definition to the
> wiki does not make sense to me, but form different reasons. I think that
> having %py3_build_egg and %py3_install_egg there is just not necessary.
> Since there are more files at [0] I'd just add that link.
>
> [0] https://src.fedoraproject.org/rpms/python-rpm-macros/tree/master

Even though it's just a new informational link, I'm guessing we still need
to file an FPC ticket for that?

I think that macros we care about should be on that page, not just at that
link.  However, you should check with tibbs/FPC about whether the
definitions/list of macros are an altogether dated concept.   When I was
there we used those because spec files are just shell scripts.  For a
sysadmin to become a packager, they just needed to understand a few
concepts about the structure of the spec file but otherwise they just put
their steps from the command line into the spec.

Defining the macros on the page allowed those users to see how the macro
replaced some steps of their manual procedure.

However, that may no longer be the audience of the guidelines.  It may be
that they're now targeting programmers or packagers who don't have the same
intimate relationship to the command line as sysadmin and have a greater
comfort learning a domain specific language to do a job.

In that case, perhaps the entire concept of enumerating the macros is
unneeded for that target audience.  Instead, simply introducing the macros
when they're used in the guidelines is enough.  Talk to tibbs/FPC about
what their thoughts are.
> if so, should we add a new section of the guidelines? something like
> "Packaging setup.py-less projects"?

Rather than emphasising the absence of setup.py, I'd emphasise the use of
wheel files:

* "Defining an RPM based on a wheel build process"
* "Defining an RPM based on a setup.py file"

I would not emphasize the use of wheel files as they are not source and
from flit's documentation, it doesn't appear that wheels are even central
to it (contrast how much wheel is mentioned in  its documentation compared
to pyproject.tom).  Instead I would emphasize flit itself as the build tool
which we're using to transform the source into the files on our systems.
If there's ever an alternative to flit which builds with wheels as part of
that process we'll need new guidelines based on that so using wheel as the
prime keyword that people associate with this build process instead of flit
is not future proof either.

-Toshio

[Jason L Tibbitts III]

> > > > > "TK" == Toshio Kuratomi  writes:

TK> However, you should check with tibbs/FPC about
TK> whether the definitions/list of macros are an altogether dated
TK> concept.

I think it's reasonable to document macros which are going to need to
use.  Python packaging just isn't sufficiently regularized (or simple)
that people can avoid having to come into more "intimate" contact with
the macros.

That documentation doesn't really have to be in the guidelines page, of
course, and we could certainly move them out to a separate document.
(We recommend but don't document %autosetup, for example.  There's
plenty of stuff you have to search for, which is unfortunate.)
Currently we have them in a list that's collapsed by default so at least
they don't take up too much space on the page.

Note also that the guidelines are in the wiki for convenience, but
there have been at least two efforts to move them to a "better" format.
And I would truly love to have a companion set of documents which aren't
guidelines but which expand on them and document things like this.
Whether that ends up being maintained as part of the guidelines or
separately hasn't even been discussed yet, but at least you'll be able
to send a PR in pagure to get something updated.

- J<

[Nick Coghlan]

On 17 November 2017 at 04:50, Jason L Tibbitts III 
wrote:
> Note also that the guidelines are in the wiki for convenience, but
> there have been at least two efforts to move them to a "better" format.
> And I would truly love to have a companion set of documents which aren't
> guidelines but which expand on them and document things like this.
> Whether that ends up being maintained as part of the guidelines or
> separately hasn't even been discussed yet, but at least you'll be able
> to send a PR in pagure to get something updated.

OK, it sounds like putting something into
https://fedora-python.readthedocs.io/en/latest/ will actually be the best
way to go, then.

That way, the update to the guidelines page can just be a link to a page
maintained in https://github.com/fedora-python/fedora-python

Cheers,
Nick.

P.S. The latter page is mainly in GitHub because I couldn't figure out how
to get Pagure's RTD integration to work, but I also see some benefits in
lowering barriers to contribution for Python folks that already have GitHub
accounts, but don't have Fedora accounts yet.

[Nick Coghlan]

On 17 November 2017 at 01:51, Toshio Kuratomi  wrote:
> On Nov 16, 2017 4:59 AM, "Nick Coghlan"  wrote:
> Rather than emphasising the absence of setup.py, I'd emphasise the use of
> wheel files:
> "Defining an RPM based on a wheel build process"
> "Defining an RPM based on a setup.py file"
>
> I would not emphasize the use of wheel files as they are not source and
> from flit's documentation, it doesn't appear that wheels are even central
> to it (contrast how much wheel is mentioned in  its documentation compared
> to pyproject.tom).  Instead I would emphasize flit itself as the build tool
> which we're using to transform the source into the files on our systems.
> If there's ever an alternative to flit which builds with wheels as part of
> that process we'll need new guidelines based on that so using wheel as the
> prime keyword that people associate with this build process instead of flit
> is not future proof either.

It's not just flit - it's enscons and other PEP 517/518 backends, whereby
the only thing we know for sure about the sdist in the long term is that
"pip wheel" (and other PEP 517 frontends) will be able to convert it into a
Python wheel in the %build phase, which can then be unpacked by the wheel
installation macro in the %install phase.

So the two possible approaches are:

* traditional sdist: "setup.py build", "setup.py install"
* Current wheel macros: "setup.py bdist_wheel", "pip install <wheel file>"

If we tweak the %py_build_wheel macro to use `pip wheel` [1] rather than
calling `setup.py bdist_wheel` directly, then even the wheel build macro
would be usable without a setup.py shim (once pip itself fully supports PEP
517/518)

Cheers,
Nick.

[1] https://pip.pypa.io/en/stable/reference/pip_wheel/

[Toshio Kuratomi]

On Thu, Nov 16, 2017 at 5:37 PM, Nick Coghlan  wrote:
> On 17 November 2017 at 01:51, Toshio Kuratomi  wrote:
> > On Nov 16, 2017 4:59 AM, "Nick Coghlan"  wrote:
> > Rather than emphasising the absence of setup.py, I'd emphasise the use of
> > wheel files:
> > "Defining an RPM based on a wheel build process"
> > "Defining an RPM based on a setup.py file"
> >
> > I would not emphasize the use of wheel files as they are not source and
> > from flit's documentation, it doesn't appear that wheels are even central to
> > it (contrast how much wheel is mentioned in  its documentation compared to
> > pyproject.tom).  Instead I would emphasize flit itself as the build tool
> > which we're using to transform the source into the files on our systems.  If
> > there's ever an alternative to flit which builds with wheels as part of that
> > process we'll need new guidelines based on that so using wheel as the prime
> > keyword that people associate with this build process instead of flit is not
> > future proof either.
>
> It's not just flit - it's enscons and other PEP 517/518 backends, whereby
> the only thing we know for sure about the sdist in the long term is that
> "pip wheel" (and other PEP 517 frontends) will be able to convert it into a
> Python wheel in the %build phase, which can then be unpacked by the wheel
> installation macro in the %install phase.
>
> So the two possible approaches are:
> traditional sdist: "setup.py build", "setup.py install"
> Current wheel macros: "setup.py bdist_wheel", "pip install <wheel file>"
>
> If we tweak the %py_build_wheel macro to use `pip wheel` [1] rather than
> calling `setup.py bdist_wheel` directly, then even the wheel build macro
> would be usable without a setup.py shim (once pip itself fully supports PEP
> 517/518)

I'm not sure what you're saying.  I must be further outside of the
packaging loop than I thought.

The two questions that I need to know the answer to to make sure we're
even vaguley on the same page are:

* Is the rpm Source: line still going to be an sdist?
* Are the files that are in the built rpm going to be the same as now?

-Toshio

[Nick Coghlan]

On 17 November 2017 at 11:55, Toshio Kuratomi  wrote:
> On Thu, Nov 16, 2017 at 5:37 PM, Nick Coghlan  wrote:
> > So the two possible approaches are:
> > traditional sdist: "setup.py build", "setup.py install"
> > Current wheel macros: "setup.py bdist_wheel", "pip install <wheel file>"
> >
> > If we tweak the %py_build_wheel macro to use `pip wheel` [1] rather than
> > calling `setup.py bdist_wheel` directly, then even the wheel build macro
> > would be usable without a setup.py shim (once pip itself fully supports PEP
> > 517/518)
>
> I'm not sure what you're saying.  I must be further outside of the
> packaging loop than I thought.
>
> The two questions that I need to know the answer to to make sure we're
> even vaguley on the same page are:
> Is the rpm Source: line still going to be an sdist?
> Are the files that are in the built rpm going to be the same as now?

Yes, those are both unchanged.

The issue is specifically with the example in
https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file,
which only shows the %pyX_build and %pyX_install macros, which assume
that the right way to install a Python project inside the buildroot is
to run:

"setup.py build" in the %build step
    "setup.py install" in the %install step

That works as long as the project either uses setup.py natively (i.e.
distutils/setuptools projects), or provides a backwards compatibility
shim, but won't work consistently for newer projects that rely on
pyproject.toml and the static build dependency declarations from PEPs
517 & 518.

Switching to a wheel based build doesn't change either the starting
point (which is still an sdist) or the end point (which is still a
policy compliant RPM), it changes the internal interface between the
build step and the install step from being a distutils build directory
to a wheel archive:

"pip wheel" (or "setup.py bdist_wheel") in the %build step
    "pip install" in the %install step

At a policy management level, I think it makes sense to separate the
"these are the policy decisions you *must* abide by" guidelines (which
are the domain of FPC) from the "Here are the helper macros that we
provide to make it easier to abide by those guidelines". The
distinction is that you can still pass a package review without using
the helper macros, but you'll still want to use them in most cases
simply because they make ongoing package maintenance easier (since the
helper macros will adjust automatically to rebases and policy changes,
while handcrafted spec files might not).

Cheers,
Nick.

[Toshio Kuratomi]

On Thu, Nov 16, 2017 at 8:43 PM, Nick Coghlan  wrote:
> On 17 November 2017 at 11:55, Toshio Kuratomi  wrote:
> > On Thu, Nov 16, 2017 at 5:37 PM, Nick Coghlan  wrote:
> > > So the two possible approaches are:
> > > traditional sdist: "setup.py build", "setup.py install"
> > > Current wheel macros: "setup.py bdist_wheel", "pip install <wheel file>"
> > >
> > > If we tweak the %py_build_wheel macro to use `pip wheel` [1] rather than
> > > calling `setup.py bdist_wheel` directly, then even the wheel build macro
> > > would be usable without a setup.py shim (once pip itself fully supports PEP
> > > 517/518)
> >
> > I'm not sure what you're saying.  I must be further outside of the
> > packaging loop than I thought.
> >
> > The two questions that I need to know the answer to to make sure we're
> > even vaguley on the same page are:
> > Is the rpm Source: line still going to be an sdist?
> > Are the files that are in the built rpm going to be the same as now?
>
> Yes, those are both unchanged.
>
> The issue is specifically with the example in
> https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file,
> which only shows the %pyX_build and %pyX_install macros, which assume
> that the right way to install a Python project inside the buildroot is
> to run:
> "setup.py build" in the %build step
> "setup.py install" in the %install step
>
> That works as long as the project either uses setup.py natively (i.e.
> distutils/setuptools projects), or provides a backwards compatibility
> shim, but won't work consistently for newer projects that rely on
> pyproject.toml and the static build dependency declarations from PEPs
> 517 & 518.
>
> Switching to a wheel based build doesn't change either the starting
> point (which is still an sdist) or the end point (which is still a
> policy compliant RPM), it changes the internal interface between the
> build step and the install step from being a distutils build directory
> to a wheel archive:
> "pip wheel" (or "setup.py bdist_wheel") in the %build step
> "pip install" in the %install step

So This seems more like "Building packages via pip"  or "Building
packages that use pyproject.toml".  wheels are actually an
implementation detail here that doesn't really surface to the either
the input side or the output side.  we don't care if the build tool
(pip in this example) takes the sdist with a pyproject.toml file and
builds a wheel, a _build/ directory, or some other intermediate
format.  We just care that it builds the appropriate files in this
step and then installs from that built resource in the second step.
This is why the name and focus seems wrong to me.  the importance to
us when building from source is the metadata format and the tools that
transform the source code with that metadata into a built resource on
our filesystem.
> At a policy management level, I think it makes sense to separate the
> "these are the policy decisions you *must* abide by" guidelines (which
> are the domain of FPC) from the "Here are the helper macros that we
> provide to make it easier to abide by those guidelines". The
> distinction is that you can still pass a package review without using
> the helper macros, but you'll still want to use them in most cases
> simply because they make ongoing package maintenance easier (since the
> helper macros will adjust automatically to rebases and policy changes,
> while handcrafted spec files might not).

So, How to write a package that complies with the guidelines is also
the domain of the FPC (at least, right now).  So you need to have
something inside of the guidelines that shows how to use these things.
I leaned towards spelling out the manual steps and then showing that
there were macros that encapsulated some of those steps but that may
no longer be the preferred direction.  The guidelines must have at
least one or the other, though.  This doesn't conflict with your last
statement (whether the package does or does not have to use the
macros) but it does conflict with the implicit idea that the FPC is
limited to the policy decisions that you must abide by and also is a
problem when you talk about not having any in-guidelines information
on how to build the packages that have pyproject.toml.

-Toshio

[Nick Coghlan]

On 18 November 2017 at 06:54, Toshio Kuratomi  wrote:
> On Thu, Nov 16, 2017 at 8:43 PM, Nick Coghlan  wrote:
> > Switching to a wheel based build doesn't change either the starting
> > point (which is still an sdist) or the end point (which is still a
> > policy compliant RPM), it changes the internal interface between the
> > build step and the install step from being a distutils build directory
> > to a wheel archive:
> > "pip wheel" (or "setup.py bdist_wheel") in the %build step
> > "pip install" in the %install step
>
> So This seems more like "Building packages via pip"  or "Building
> packages that use pyproject.toml".  wheels are actually an
> implementation detail here that doesn't really surface to the either
> the input side or the output side.  we don't care if the build tool
> (pip in this example) takes the sdist with a pyproject.toml file and
> builds a wheel, a _build/ directory, or some other intermediate
> format.  We just care that it builds the appropriate files in this
> step and then installs from that built resource in the second step.
> This is why the name and focus seems wrong to me.  the importance to
> us when building from source is the metadata format and the tools that
> transform the source code with that metadata into a built resource on
> our filesystem.

Ah, OK - this makes sense.

Framing it in terms of the required input file, the two most suitable
names would be:

* "pyproject.toml compatible spec files"
* "setup.py only spec files"

I'd suggest keeping the current wheel-centric macro names, though, and
just adjust the wheel building macro to accept pyproject.toml files in
addition to setup.py files.
> > At a policy management level, I think it makes sense to separate the
> > "these are the policy decisions you *must* abide by" guidelines (which
> > are the domain of FPC) from the "Here are the helper macros that we
> > provide to make it easier to abide by those guidelines". The
> > distinction is that you can still pass a package review without using
> > the helper macros, but you'll still want to use them in most cases
> > simply because they make ongoing package maintenance easier (since the
> > helper macros will adjust automatically to rebases and policy changes,
> > while handcrafted spec files might not).
>
> So, How to write a package that complies with the guidelines is also
> the domain of the FPC (at least, right now).  So you need to have
> something inside of the guidelines that shows how to use these things.
> I leaned towards spelling out the manual steps and then showing that
> there were macros that encapsulated some of those steps but that may
> no longer be the preferred direction.  The guidelines must have at
> least one or the other, though.

In that case, I'd lean towards emphasising the macros in the
guidelines, and then pointing towards the macro definitions in
dist-git if folks really want to dig into the details of what the
macros actually do. However, we'd also keep the various sections that
explain how to do various lower level things correctly (like
precompiling Python files), since those providing the "Why?" of
various decisions, which simply reading the macro source files won't
give you.

Cheers,
NIck.

[Toshio Kuratomi]

On Sat, Nov 18, 2017 at 5:28 AM, Nick Coghlan  wrote:
> Ah, OK - this makes sense.
>
> Framing it in terms of the required input file, the two most suitable
> names would be:
> "pyproject.toml compatible spec files"
> "setup.py only spec files"
>
> I'd suggest keeping the current wheel-centric macro names, though, and
> just adjust the wheel building macro to accept pyproject.toml files in
> addition to setup.py files.

+1, Sound good
> In that case, I'd lean towards emphasising the macros in the
> guidelines, and then pointing towards the macro definitions in
> dist-git if folks really want to dig into the details of what the
> macros actually do. However, we'd also keep the various sections that
> explain how to do various lower level things correctly (like
> precompiling Python files), since those providing the "Why?" of
> various decisions, which simply reading the macro source files won't
> give you.

<nod>m his makes sense to me too.

Thanks Nick!
-Toshio

[Jason L Tibbitts III]

> > > > > "NC" == Nick Coghlan  writes:

NC> Even though it's just a new informational link, I'm guessing we
NC> still need to file an FPC ticket for that?

You can just ask me for super minor things, but it does sound like this
could use a bit of exposition in the guidelines, and that would at least
need a ticket.

- J<