mock build results for same .spec build different for local & online/COPR builds -- local OK, @copr FAILS ?

[PGNet Dev]

(i'd been discussing this issue with praiskup @ copr-devel/buildsys; he suggested that
I bring it here ...)

This spec

	https://download.copr.fedorainfracloud.org/results/pgfed/nginx-mainline/f...

which uses forgemeta to pull multiple SCM sources, and uses some git/bash scripting in
%defines,

builds locally, on F32, via rpmbuild or mock build, from spec or srpm, with NO error. 
resulting rpms are installable, usable & pass testing.

submitting the _same_ spec to COPR for online build FAILS @, supposedly, similar Mock
build

Here's a diff

	https://www.diffchecker.com/izjQYkUF

comparing the log output of

	SUCCESSFUL LOCAL BUILD	
	mock --buildsrpm --spec=~/rpmbuild/SPECS/nginx.spec --sources=~/rpmbuild/SOURCES	
	cat build.log	

and
	
	FAILED COPR BUILD
	copr-cli build nginx-mainline ~/rpmbuild/SPECS/nginx.spec
	https://download.copr.fedorainfracloud.org/results/pgfed/nginx-mainline/f...
	
The goal is to have the same spec generate the same Mock build, regardless of
environment.

Why does the COPR build's Mock build stage fail?

IIUC, mock builds _should_ be portable between mock envs, at least for the same chroot.

Do I need additional prep of the spec prior to submit?  Something in my foregemeta/scm
usage that's env-dependent? Something else I've missed?

Or is there an issue with the Mock build env @ COPR?

[Igor Raits]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, 2020-07-02 at 18:00 -0700, PGNet Dev wrote:
>  (i'd been discussing this issue with praiskup @
> copr-devel/buildsys;
>  he suggested that I bring it here ...)
>  
>  This spec
>  
>  	
>
> https://download.copr.fedorainfracloud.org/results/pgfed/nginx-mainline/f...
>  
>  which uses forgemeta to pull multiple SCM sources, and uses some
>  git/bash scripting in %defines,
>  
>  builds locally, on F32, via rpmbuild or mock build, from spec or
>  srpm, with NO error.  resulting rpms are installable, usable & pass
>  testing.
>  
>  submitting the _same_ spec to COPR for online build FAILS @,
>  supposedly, similar Mock build 
By default COPR does not allow you to use internet though it can be
enabled in the project settings.

>  Here's a diff
>  
>  	https://www.diffchecker.com/izjQYkUF
>  
>  comparing the log output of
>  
>  	SUCCESSFUL LOCAL BUILD	
>  	mock --buildsrpm --spec=~/rpmbuild/SPECS/nginx.spec --
>  sources=~/rpmbuild/SOURCES	
>  	cat build.log	
>  
>  and
>  	
>  	FAILED COPR BUILD
>  	copr-cli build nginx-mainline ~/rpmbuild/SPECS/nginx.spec
>  	
>
> https://download.copr.fedorainfracloud.org/results/pgfed/nginx-mainline/f...
>  	
>  The goal is to have the same spec generate the same Mock build,
>  regardless of environment.
>  
>  Why does the COPR build's Mock build stage fail?
>  
>  IIUC, mock builds _should_ be portable between mock envs, at least
>  for the same chroot.
>  
>  Do I need additional prep of the spec prior to submit?  Something in
>  my foregemeta/scm usage that's env-dependent? Something else I've
>  missed?
>  
>  Or is there an issue with the Mock build env @ COPR?
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: 
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail] 
- -- 
Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7+vesACgkQEV1auJxc
> Hh6few//fpCK/6V7F+7FWmo5l6e+kXJgtl3KZZfsusAo07ETto5meeOjZibjM13N
> 4adwQMY83uUJkD13jQwfWBzrcuK8ER6rhAGZ8SnTLKo7wrCWuvO2b2OBZxyXgaVW
> 1TcJzZsLLxEvBX1OPzYYlymyZSFfvePul6K1ByFLkYVpAn1xtYOCiB1QUT0BmX4m
> AQihgGJ7w0P8BY8f4crERM4U/pmxzBH6TeSMiTyq4JQSqfeloVvdFjzAiZ6MBp1Z
> rb+XtMhu6vu6ZYYYFf4bqPzYann5Bu8KGylhanuS+Tvvo9TmfKEQYz1Ec5XFkWwL
> wyw3QrlqcfllcX3zd3bgQYpPLuJST1PJho6Pv2lYyFVjhj8GC9kjXskm42CXeauz
> oF4D3m/Iz6kN+YQw/w+tg2BAnD+BOJVZaaI/T3dQIo2WE7fZVQ7fPCCtOkwiZYbc
> +eoBybumEFspShLKykoi5ENWbi3qA95qE69h+286btdgakKX+zv4RfMa2/k4jmMj
> r/Qohlph20mHJYJXf3VBO3Sng+pB2+fp2nhf46JE7OjcXefMMBuyja7ko7wBAMC1
> vdfQc1ppCAQMvxxc+O17TNdP3f5sitzlf7J1XeiaPWHsPVv1XbZjnQTER7lYMYTy
> PqvDnoIEYfY2pVXHmvczYxeCGtbjA04bRPpmx1PLC1wZl2jcL/s=
> =KWRR
> -----END PGP SIGNATURE-----

[Pavel Raiskup]

On Friday, July 3, 2020 3:00:48 AM CEST PGNet Dev wrote:
> https://download.copr.fedorainfracloud.org/results/pgfed/nginx-mainline/f...
There are things like:

    %global forgeurl1 https://github.com/openresty/headers-more-nginx-module
    %define _ctag1    HEAD
    %global commit1   %( git ls-remote %{forgeurl1} | grep %{_ctag1} | cut -f1 )

So basically it needs to have the /bin/git installed in the chroot (it is
not by default), and it needs to have the internet enabled at that time to
get the contents of %commitX.

So the alternative local mock command would be:

    mock --buildsrpm --spec nginx.spec --enable-network

I'm not familiar with the %forge* macros, but I don't think it is expected
that you will add commands that need the Internet into the macro
definition.  Simply do:

    - %global commit1   %( git ls-remote %{forgeurl1} | grep %{_ctag1} | cut -f1 )
    + %global commit1 <commit|tag|branch>

Alternatively, if you really need the dynamic %commitX generator -- you
can use one of the Copr SCM methods to "generate" the spec file and bake
the _static_ %commitX one step before (before the package is imported into
the Copr DistGit).

>  builds locally, on F32, via rpmbuild or mock build, from spec or
> srpm,
>  with NO error.  resulting rpms are installable, usable & pass testing. 
I'm not sure how `/bin/git` got installed into your mock chroot, and how
it comes you have the Internet there.

>  	FAILED COPR BUILD
>  	copr-cli build nginx-mainline ~/rpmbuild/SPECS/nginx.spec
>
> 	https://download.copr.fedorainfracloud.org/results/pgfed/nginx-mainline/f...
>  	
>  The goal is to have the same spec generate the same Mock build,
>  regardless of environment. 
Already deleted.

>  Do I need additional prep of the spec prior to submit?  Something in
> my
>  foregemeta/scm usage that's env-dependent? Something else I've missed? 
Probably, see above.

Pavel

[Nicolas Mailhot]

Le vendredi 03 juillet 2020 à 07:26 +0200, Pavel Raiskup a écrit :

Hi,

>  I'm not familiar with the %forge* macros, but I don't think
> it is
>  expected that you will add commands that need the Internet into the
>  macro definition.   
It’s neither expected nor unexpected. For a Fedora-oriented spec, of
course you should never depend on Internet ability. For you own needs,
both mock and copr allow enabling the internet. forge macros do not
enforce a specific policy.

The main showstopper will be that if the additional commands need to
process files in sources, you need to run %prep first, and %prep
depends on sources being set, so you’ll be in a chicken and egg
situation.

Now, we’ve been moving to more dynamic spec contruction in the last
years, so I can see it being possible in the near feature if everyone
makes a goodwill effort, but right now that is not possible.

With the dynamic buildrequires part rpm upstream added, and with
https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_mac...

it will certainly be possible to compute a second level of sources
during the dynamic buildrequires first pass over prep, and the change
makes the forge macro code modular enough the second level will be
auto-registered in sourcelist, but how is the buildsystem supposed to
provide sources that did not exist during its first pass over the spec
file?

That probably involves running spectool by default over %sourcelist at
the start of %prep, which, again, should be doable easily with 
https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_mac...

and Fabio’s most excellent rewrite of the spectool code (if you have
not used it yet, try it, it’s good). However someone needs to add
lookaside or buildsys integration to spectool to spectool so spectool
has something to source new sources from in a restricted build context.

If anyone is interested in doing the spectool/buildsys integration
part, we can fill a change for F34. It’s another example, like
autobumping, of high level build features that were devilishly hard to
implement before something like %auto_call was available, and is pretty
simple to do with %auto_call implemented

Regards,

-- 
Nicolas Mailhot

[Pavel Raiskup]

On Friday, July 3, 2020 9:51:20 AM CEST Nicolas Mailhot via devel wrote:
>  it will certainly be possible to compute a second level of sources
>  during the dynamic buildrequires first pass over prep, and the change
>  makes the forge macro code modular enough the second level will be
>  auto-registered in sourcelist, but how is the buildsystem supposed to
>  provide sources that did not exist during its first pass over the spec
>  file? 
Possible.  But I personally think that "dynamic buidlrequires" should stay
working with BuildRequires, not Sources.

>  and Fabio’s most excellent rewrite of the spectool code (if you have
>  not used it yet, try it, it’s good). However someone needs to add
>  lookaside or buildsys integration to spectool to spectool so spectool
>  has something to source new sources from in a restricted build context. 
Let me just state my opinion that I don't like this situation coming.
Packaging should be simple task, and should be made simpler, obvious.

That said, I like that package build is just:
SPEC + sources + patches + build dependencies

I'd appreciate the link to spectool rewrite, though.
Pavel

[Nicolas Mailhot]

Le vendredi 03 juillet 2020 à 11:03 +0200, Pavel Raiskup a écrit :
>  On Friday, July 3, 2020 9:51:20 AM CEST Nicolas Mailhot via devel
>  wrote:
>  > it will certainly be possible to compute a second level of sources
>  > during the dynamic buildrequires first pass over prep, and the
>  > change
>  > makes the forge macro code modular enough the second level will be
>  > auto-registered in sourcelist, but how is the buildsystem supposed
>  > to
>  > provide sources that did not exist during its first pass over the
>  > spec
>  > file?
>  
>  Possible.  But I personally think that "dynamic buidlrequires" should
>  stay
>  working with BuildRequires, not Sources.
>  
>  > and Fabio’s most excellent rewrite of the spectool code (if you
>  > have
>  > not used it yet, try it, it’s good). However someone needs to add
>  > lookaside or buildsys integration to spectool to spectool so
>  > spectool
>  > has something to source new sources from in a restricted build
>  > context.
>  
>  Let me just state my opinion that I don't like this situation coming.
>  Packaging should be simple task, and should be made simpler, obvious. 
I understand. There is just this huge pressure in npm, and go and rust,
and all other modern languages, and even in git itself via submodules,
to create huge dependency graphs of interlinked components.

You can attempt, like we (me at least) to make individual component
packaging as easy and cheap as possible. That still costs you one spec
file per git repo, and forces you to treat git submodules as separate
projects.

It is possible the pressure will continue to grow to a point that a
spec needs to automate not just the top-level component but all the
other bits upstreams hide bellow this component. If that happens we
will need dynamic sourcing in spec files. There is huge pressure EL
side at least to get this second scenario to work (in part, due to EL
management putting hard constrains on introduction of new srpms,
forcing EL packagers to stuff as many things as possible into existing
spec files).

The requester is clearly attempting the second approach.

Regards,


-- 
Nicolas Mailhot

[Nicolas Mailhot]

Le vendredi 03 juillet 2020 à 11:03 +0200, Pavel Raiskup a écrit :
>  
>  I'd appreciate the link to spectool rewrite, though. 
Here it is:
https://pagure.io/rpmdevtools/blob/master/f/rpmdev-spectool

Regards,

-- 
Nicolas Mailhot

[PGNet Dev]

hi,

>  ... 
All the above^ is an interesting/informative read.

On 7/3/20 2:31 AM, Nicolas Mailhot via devel wrote:
>  The requester is clearly attempting the second approach. 
Well, not explicitly.  I'm not requesting any _specific_ approach.

The goal is simply to 'build it here (locally, via mock), then build it there
(remotely, on COPR)'. And have the results be the same.

My general question is why what works in _mock_ here, locally, fails to work similarly on
COPR.  Not just rpmbuild, which I understand _is_ well integrated into my local env, but
the supposedly _isolated_, basic mock env.

And if it inherently does/can work similarly, what specifically I need to do/configure to
get there.

On 7/2/20 10:26 PM, Pavel Raiskup wrote:
>  and it needs to have the internet enabled at that time to
>  get the contents of %commitX.
>  
>  So the alternative local mock command would be:
>  
>       mock --buildsrpm --spec nginx.spec --enable-network 
in both cases, networking _is_ 'on'

Here, the mock env is setup as

	cat /etc/mock/site-defaults.cfg
		...
		config_opts['rpmbuild_networking'] = True
		config_opts['chroot_additional_packages'] = 'git'
		...
	mock --init

and the copr build as,

	copr-cli create \
	 --chroot fedora-32-x86_64 \
	 --multilib off \
	 --description "x" \
	 --instructions "x" \
	 --use-bootstrap on \
	 --enable-net on \
	 --unlisted-on-hp on \
	<project>


On 7/3/20 12:01 AM, Nicolas Mailhot wrote:
>  You added some processing that depends on the git command (that
>  forgemeta does not use) but forgot to BuildRequire the package
>  providing that command. 
It's _clearly_ already added in my referenced spec:

	BuildRequires:     git

On 7/3/20 1:23 AM, Pavel Raiskup wrote:
>  I'm afraid that for building the src.rpm simply adding git to
> BuildRequires will not help. 
Agreed; it does not.


On 7/3/20 12:01 AM, Nicolas Mailhot wrote:
>  dead stupid (so stupid 
Let's chalk that up this time to 'lost in translation', shall we?

>  As stated before forgemeta does not depend on the git (or hg, or
> svn,
>  or…) command, it’s a pure URL munger, so it won’t pull in your scm of
>  choice in the buildroot.
>
>  Presumably your workflow is so git oriented your local setup always has
>  git installed. 
I've been working with Fedora for < 2 weeks now; i.e., not much 'baggage'
at all.

There _is_ no "so oriented" or "always" workflow.  ALL of what I'm
attempting to do is simply based on new/fresh eyes reading the docs provided, and giving
it a whirl.

I'll certainly acknowledge the _hope_ that there's something similar to my prior
build env's 'Source Services' capability,

	https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.source_s...

doable here.

On 7/2/20 10:26 PM, Pavel Raiskup wrote:

>  So basically it needs to have the /bin/git installed in the chroot
> (it is 
>  not by default),  


git _was_ trivially added to the local mock chroot, for its use, with obvious success, in
the local mock build of the spec.

COPR uses mock.

So far, I have not seen how that's to be similarly done for the COPR env.

Is is possible to, similarly, add git to the COPR mock env?

[PGNet Dev]

On 7/3/20 8:24 AM, PGNet Dev wrote:
>  git _was_ trivially added to the local mock chroot, for its use, with
> obvious success, in the local mock build of the spec.
>  
>  COPR uses mock.
>  
>  So far, I have not seen how that's to be similarly done for the COPR env.
>  
>  Is is possible to, similarly, add git to the COPR mock env? 
FrostyX @ irc was kind enuf to give this a try -- & found the solution.

@ nav to

	https://copr.fedorainfracloud.org/coprs/pgfed/nginx-mainline/edit_chroot/...

adding 'git' to

	Packages: git
		You can add additional packages to the minimal buildroot of this chroot. These packages
will be always present before the build starts.

( from UI, that's 'hidden' behind the "EDIT" button for the select
chroot(s). obvious if you know it ... )

seems to solve the problem.  he was able to successfully bld my spec ...


from cmd line,

	copr-cli edit-chroot <coprchroot> --packages git

looks like it should work as well


ta, FrostyX!

[PGNet Dev]

On 7/3/20 2:07 PM, PGNet Dev wrote:
>  from cmd line,
>  
>  	copr-cli edit-chroot <coprchroot> --packages git
>  
>  looks like it should work as well 
and it does, nicely:

==> 14:26:43 Build 1517366: succeeded

[Nicolas Mailhot]

Le vendredi 03 juillet 2020 à 08:24 -0700, PGNet Dev a écrit :
>  
>  
>  On 7/3/20 12:01 AM, Nicolas Mailhot wrote:
>  > You added some processing that depends on the git command (that
>  > forgemeta does not use) but forgot to BuildRequire the package
>  > providing that command.
>  
>  It's _clearly_ already added in my referenced spec:
>  
>  	BuildRequires:     git 
Then you probably hit one of the cases where the SRPM is prepared
before the full buildroot is assembled.

That means you may need to make sure to enable the container option in
copr (it’s default in mock now but IIRC copr lags a bit) and probably
add some conditionals to your code so it does not error out if executed
before the full buildroot is assembled.

Generally speaking, there’s no reason a spec that buildrequires
everything it needs should not execute in mock or copr, but you have to
be extra careful with preamble macros, because preamble macros are not
executed once but multiple times during a build.

Because BuildRequires can be generated rpmbuild needs to execute the
preamble to check what BuildRequires exist, therefore a preamble macro
should fail gracefully when one of its BuildRequires is not available
yet.

(A good way to discover how convoluted this part is is to add some
%{echo: markers in a working spec and look when the echoes trigger in a
full mock/copr build).

If you don’t care you can force a specific buildroot in mock/corp
configuration, of course.

>  > dead stupid (so stupid
>  
>  Let's chalk that up this time to 'lost in translation', shall we? 
That was a compliment, actually, the only reason the humain brain does
not see stupid mistakes is that you’re asking it to operate at another
level. Computers are dumb as a brick and have no problem seeing all the
dumb little things the humain brain optimises away.

>  There _is_ no "so oriented" or "always" workflow.
>   
You’re formatted like we all are by your past experiences. Just because
you did not read about a pattern in a book, does not mean you are not
following a pattern, because you’ve been exposed to the work of others
that followed the same pattern (knowingly or not).

Regar,ds


-- 
Nicolas Mailhot

[Nicolas Mailhot]

Le jeudi 02 juillet 2020 à 18:00 -0700, PGNet Dev a écrit :

Hi,

As usual for those things the reason is dead stupid (so stupid the
human brain refuses to see it)

>  submitting the _same_ spec to COPR for online build FAILS @,
>  supposedly, similar Mock build
>  
>  Here's a diff
>  
>  	https://www.diffchecker.com/izjQYkUF
>   
The fail log is full of

        sh: git: command not found
	sh: git: command not found
	sh: git: command not found
	sh: git: command not found
	sh: git: command not found
	sh: git: command not found
	sh: git: command not found
	sh: git: command not found
	sh: git: command not found

You added some processing that depends on the git command (that
forgemeta does not use) but forgot to BuildRequire the package
providing that command.

As stated before forgemeta does not depend on the git (or hg, or svn,
or…) command, it’s a pure URL munger, so it won’t pull in your scm of
choice in the buildroot.

Presumably your workflow is so git oriented your local setup always has
git installed.

Regards,

-- 
Nicolas Mailhot

[Pavel Raiskup]

On Friday, July 3, 2020 9:01:16 AM CEST Nicolas Mailhot wrote:
>  Le jeudi 02 juillet 2020 à 18:00 -0700, PGNet Dev a écrit :
>  
>  Hi,
>  
>  As usual for those things the reason is dead stupid (so stupid the
>  human brain refuses to see it)
>  
>  > submitting the _same_ spec to COPR for online build FAILS @,
>  > supposedly, similar Mock build
>  > 
>  > Here's a diff
>  > 
>  > 	https://www.diffchecker.com/izjQYkUF
>  > 
>  
>  The fail log is full of
>  
>          sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  	sh: git: command not found
>  
>  You added some processing that depends on the git command (that
>  forgemeta does not use) but forgot to BuildRequire the package
>  providing that command. 
I'm afraid that for building the src.rpm simply adding git to BuildRequires
will not help.

Pavel

>  As stated before forgemeta does not depend on the git (or hg, or
> svn,
>  or…) command, it’s a pure URL munger, so it won’t pull in your scm of
>  choice in the buildroot.
>  
>  Presumably your workflow is so git oriented your local setup always has
>  git installed.
>  
>  Regards,
>  
>