Template for modules that use pbr?

[Tim Landscheidt]

Hi,

I'm trying to update git-review to 1.24.  With that version,
git-review uses pbr.  I have no deeper knowledge of Python
packaging in Fedora and only tweaked the spec file so that
it works for me
(cf. https://github.com/scfc/fedora-git-review/commit/b35c26010d67affd5da0305af2c...),
so I wanted to solicit some advice on how to do it
right(TM).

1. Previously, Fedora used GitHub "releases" that appear to
   be verbatim snapshots of the source repository.  But with
   https://github.com/openstack-infra/git-review/archive/1.24.tar.gz,
   this no longer works as "python setup.py build" gives:
   "error in setup command: Error parsing
   /tmp/tmp.ITVvwuYttj/git-review-1.24/setup.cfg: Exception:
   Versioning for this project requires either an sdist tar-
   ball, or access to an upstream git repository."  However,
   using
   http://pypi.python.org/packages/source/g/git-review/git-review-1.24.tar.gz
   works fine.

Comparing the two shows a number of additional files in
   the latter (AUTHORS, ChangeLog, git_review.egg-info,
   PKG-INFO).  Am I correct to assume that this is a problem
   upstream, i. e. that the tarballs should not have been
   published in this state, and/or is there a way to work
   around that in the spec file?

2. By default, pbr seems to pull any dependencies from the
   network.
   http://docs.openstack.org/developer/pbr/packagers.html
   suggests that this can and should be avoided by setting
   SKIP_PIP_INSTALL=1.  Is there a template for how to do
   that properly, i. e. in a macro or just prepending to
   %{__python}, etc.?

3. Testing with Koji
   (cf. http://koji.fedoraproject.org/koji/taskinfo?taskID=7977881),
   the build log
   (cf. https://kojipkgs.fedoraproject.org//work/tasks/7881/7977881/build.log)
   starts with:

| Mock Version: 1.1.41
   | Mock Version: 1.1.41
   | ENTER do(['bash', '--login', '-c', 'rpmbuild -bs --target noarch --nodeps builddir/build/SPECS/git-review.spec'], False, '/var/lib/mock/f19-build-2538894-429281/root/', None, 86400, True, False, 1000, 425, None, False, {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'PROMPT_COMMAND': 'echo -n "<mock-chroot>"', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}, logger=<mockbuild.trace_decorator.getLog object at 0x1725dd0>)
   | Executing command: ['bash', '--login', '-c', 'rpmbuild -bs --target noarch --nodeps builddir/build/SPECS/git-review.spec'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'PROMPT_COMMAND': 'echo -n "<mock-chroot>"', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
   | sh: /usr/bin/python: No such file or directory
   | sh: /usr/bin/python: No such file or directory
   | [...]

Is that to be expected?

Of course, any other advice is also much appreciated.

TIA,
Tim

[Bohuslav Kabrda]

----- Original Message -----
> Hi,
>
> I'm trying to update git-review to 1.24.  With that version,
> git-review uses pbr.  I have no deeper knowledge of Python
> packaging in Fedora and only tweaked the spec file so that
> it works for me
> (cf.
> https://github.com/scfc/fedora-git-review/commit/b35c26010d67affd5da0305af2c...),
> so I wanted to solicit some advice on how to do it
> right(TM).

Hi Tim,
not sure if my answers will be right(TM), but I hope that they'll be useful, at least :)
> Previously, Fedora used GitHub "releases" that appear to
> be verbatim snapshots of the source repository.  But with
> https://github.com/openstack-infra/git-review/archive/1.24.tar.gz,
> this no longer works as "python setup.py build" gives:
> "error in setup command: Error parsing
> /tmp/tmp.ITVvwuYttj/git-review-1.24/setup.cfg: Exception:
> Versioning for this project requires either an sdist tar-
> ball, or access to an upstream git repository."  However,
> using
> http://pypi.python.org/packages/source/g/git-review/git-review-1.24.tar.gz
> works fine.
>
> Comparing the two shows a number of additional files in
> the latter (AUTHORS, ChangeLog, git_review.egg-info,
> PKG-INFO).  Am I correct to assume that this is a problem
> upstream, i. e. that the tarballs should not have been
> published in this state, and/or is there a way to work
> around that in the spec file?

The important thing here is that upstreams do not "publish" tarballs on Github. Github dynamically generates tarballs of repos for given tag/commit hash when you request it. This generally means that the tarball downloaded from Github is not what upstream meant to release (unlike pypi, which *is* what upstream released).
The interesting thing about the tarball from pypi is, that all files in the archive are owned by "jenkins" user. I didn't check, but my guess is, that when upstream wants to do a release, they let Jenkins (with access to upstream git repository, which is what pbr asks you to have) generate the final tarball - and Jenkins generates all the extra files before packing the tarball. If you look at the files missing from the first archive, they do look autogenerated, which seems to confirm my thoughts, but I'm not 100 % sure.
> By default, pbr seems to pull any dependencies from the
> network.
> http://docs.openstack.org/developer/pbr/packagers.html
> suggests that this can and should be avoided by setting
> SKIP_PIP_INSTALL=1.  Is there a template for how to do
> that properly, i. e. in a macro or just prepending to
> %{__python}, etc.?

I'd prepend that to macro, e.g.
SKIP_PIP_INSTALL=1 %{__python2} <something>
(Note, that it is recommended to use %{__python2} or %{__python3} explicitly, rather than using %{__python}.)
> Testing with Koji
> (cf. http://koji.fedoraproject.org/koji/taskinfo?taskID=7977881),
> the build log
> (cf.
> https://kojipkgs.fedoraproject.org//work/tasks/7881/7977881/build.log)
> starts with:
>
> | Mock Version: 1.1.41
> | Mock Version: 1.1.41
> | ENTER do(['bash', '--login', '-c', 'rpmbuild -bs --target noarch
> | --nodeps builddir/build/SPECS/git-review.spec'], False,
> | '/var/lib/mock/f19-build-2538894-429281/root/', None, 86400, True,
> | False, 1000, 425, None, False, {'LANG': 'en_US.UTF-8', 'TERM': 'vt100',
> | 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'PROMPT_COMMAND': 'echo -n
> | "<mock-chroot>"', 'HOME': '/builddir', 'PATH':
> | '/usr/bin:/bin:/usr/sbin:/sbin'},
> | logger=<mockbuild.trace_decorator.getLog object at 0x1725dd0>)
> | Executing command: ['bash', '--login', '-c', 'rpmbuild -bs --target
> | noarch --nodeps builddir/build/SPECS/git-review.spec'] with env
> | {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash',
> | 'HOSTNAME': 'mock', 'PROMPT_COMMAND': 'echo -n "<mock-chroot>"',
> | 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
> | sh: /usr/bin/python: No such file or directory
> | sh: /usr/bin/python: No such file or directory
> | [...]
>
> Is that to be expected?

These lines are usually caused by macro definitions in specfile containing calls to not yet installed binaries. When Koji creates the minimal mock buildroot, it tries to expand the specfile to get dependencies (BuildRequires) to install. Since Python is not in the minimal buildroot when the specfile is expanded, mock will produce this warning. I'm not sure where exactly this comes from the git-review specfile, but I'm quite sure that it's not a problem.

Slavek
> Of course, any other advice is also much appreciated.
>
> TIA,
> Tim

[Nick Coghlan]

On 10/30/2014 06:57 PM, Bohuslav Kabrda wrote:
> ----- Original Message -----
> > Comparing the two shows a number of additional files in
> >    the latter (AUTHORS, ChangeLog, git_review.egg-info,
> >    PKG-INFO).  Am I correct to assume that this is a problem
> >    upstream, i. e. that the tarballs should not have been
> >    published in this state, and/or is there a way to work
> >    around that in the spec file?
>
> The important thing here is that upstreams do not "publish" tarballs on Github. Github dynamically generates tarballs of repos for given tag/commit hash when you request it. This generally means that the tarball downloaded from Github is not what upstream meant to release (unlike pypi, which *is* what upstream released).
> The interesting thing about the tarball from pypi is, that all files in the archive are owned by "jenkins" user. I didn't check, but my guess is, that when upstream wants to do a release, they let Jenkins (with access to upstream git repository, which is what pbr asks you to have) generate the final tarball - and Jenkins generates all the extra files before packing the tarball. If you look at the files missing from the first archive, they do look autogenerated, which seems to confirm my thoughts, but I'm not 100 % sure.

Right, Python sdists include some metadata files that can either be
maintained manually (which is error prone) or automatically generated
from source control (which introduces a build step into the release
archive generation, so the tarballs automatically generated by GitHub
are incomplete).

In this case, if you don't have the pregenerated metadata files from the
sdist, then pbr is going to need access to the original git repo in
order to generate them (see
http://docs.openstack.org/developer/pbr/#what-it-does for the things it
can automatically derive).

Regards,
Nick.

[Haïkel]

I suggest you take a look at openstack packages spec and see how we
solved the network issue with PBR.
http://pkgs.fedoraproject.org/cgit/openstack-glance.git/tree/openstack-glanc...

Regards,
H.