Fwd: %forgemeta support for `git` tasks in checked-out code?

[Pavel Raiskup]

Just forwarding this question to more appropriate list.

----------  Forwarded Message  ----------

Subject: %forgemeta support for `git` tasks in checked-out code?
Date: Thursday, June 25, 2020, 6:42:42 PM CEST
From: PGNet Dev 
To: [e-mail]

I'm building a package using multiple sources.

I'm using $forgemeta to get/manage sources, as per example here:

https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/#_multip...

One of my sources pulls from upstream scm (github), and needs a subsequent 'git submodule--init'.

Is seems that what %forgemeta pulls/tars/expands is not initilialize git repo; rather, just the relevant target's files.

E.g., for a given scm source (here, ngx_brotli), the %forgemeta pull unpacks

tree -L 1 -a ngx_brotli
    	ngx_brotli
    	├── config
    	├── CONTRIBUTING.md
    	├── deps
    	├── filter
    	├── .gitmodules
    	├── LICENSE
    	├── README.md
    	├── script
    	├── static
    	└── .travis.yml

note, no ".git" folder, and no ability as is to `git submodule --init` in the tree.

otoh, a `git clone` of the same source results, as expected, in

tree -L 1 -a ngx_brotli  
    	ngx_brotli
    	├── config
    	├── CONTRIBUTING.md
    	├── deps
    	├── filter
!!		├── .git
    	├── .gitmodules
    	├── LICENSE
    	├── README.md
    	├── script
    	├── static
    	└── .travis.yml

currently, to workaround this, I DISABLE the forge for the source in question, and do the pull & submodule init in %prep,

e.g., this spec

%global forgeurl0 https://github.com/nginx/nginx
    Version:          1.19.0
    %global tag0      release-%{version}

%global forgeurl1 https://github.com/openresty/headers-more-nginx-module
    %global tag1      master

# DISABLE global forgeurl1 https://github.com/google/ngx_brotli
    %global tag2      master

%forgemeta -i -z 0
    %forgemeta -i -z 1
    # DISABLE forgemeta -i -z 2

Name:           %{name}
    %define dist    %{scm0}
    Release:        %{dist}
    Summary:        %{name}
    License:        BSD
    URL:            %{forgeurl0}

Source0:        %{forgesource0}
    Source1:        %{forgesource1}
    Source2:        https://github.com/google/ngx_brotli/archive/master/ngx_brotli-master.tar.gz
    ...

%prep
    pushd .
    cd %{_sourcedir}
    rm -rf ngx_brotli
    git clone https://github.com/google/ngx_brotli
    cd ngx_brotli
    git submodule update --init
    cd ..
    tar --remove-files -czf ngx_brotli.tar.gz ngx_brotli
    popd

%setup -T -b 0 -n  nginx-%{tag0}
    %setup -T -b 1 -n  headers-more-nginx-module-%{tag1}
    ...

%build
    export DESTDIR=%{buildroot}
    cd %{_builddir}/%{name}-%{tag0}
    ...

%install
    cd %{_builddir}/%{name}-%{tag0}
    make install DESTDIR=%{buildroot} INSTALLDIRS=vendor
    ...

appears to work, and the `git submodule --init` is taken care of correctly.

but, it's messy.

_is_ there a %forgemeta alternative to the manual git clone that will init/unpack the source _as_ a repo, allowing the submodule task to be exec'd?

this isn't, of course, ONLY a COPR question, but one of rpmbuild more generally.

_my_ goal _is_ to get it working on COPR, eventually.

that said, if this^ belongs on some other list, let me know.
_______________________________________________
copr-devel mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

-----------------------------------------

[Nicolas Mailhot]

Hi,

forgemeta works in release mode, with release archives published over
http(s). It does not talk at all to source projects using the git
protocol (and that is intentional, since a lot ot things tooling-side
do not talk the git protocol and will never talk the git protocol,
starting with rpm itself, spectool, etc).

git is not the only scm system out there. Nor its protocol is set in
stone. It would be very tricky to push it down to layers like spec
files that operate on timescales measured in years/decades and are used
by a huge amount of third-party tools that expect http(s) source URLs.

From a pure auditing side, it is also correct to have all the sources
of things you package listed explicitly, and not have recursive sources
that install themselves other sources behind auditor’s backs.

Note that your spec as it stands is inherently unsafe since you are
pulling sources from a a moving reference (master) instead of commiting
to a specific commit or tag state¹ (another proof git is not safe to
use as direct packaging layer, since it pretends to give you a fixed
state while submoduling things that are not fixed).

The correct auditable way to use git in rpm is to declare all the git
sources you actually use, with a specific tag or commit hash for each
of them.

So you should have a long list of

%forgeurlX
%tagX or %commitX

and a single %forgemeta -a at the end

That will change soonish BTW, I’m currently doing final testing on code
that will use a slightly different syntax:

%forge_urlX
%forge_tagX or %forge_commitX
%forge_patchlist %{expand:
foo1.patch
foo2.patch
}

…
%auto_init

…
%sourcelist
%auto_sources

%patchlist
%auto_patches

%prep
%auto_prep

Because not making -a the default and emphasising -z in the
documentation was comfusing users. -z should be a last resort debuging
help, not the first thing you look at when packaging multiple sources.

That will leave you with each individual source unpacked and patched in
%{_builddir}, and needing a some symlinks between %{_builddir}
subdirectories to construct a unified source trees, but that’s how
multisource works deep down in rpm, nothing I invented myself.

Regards,

¹ I made it work in the macros because I knew people would attempt to
do it, and I’d rather have it work in my code than have people butcher
it to achiever their aims, but from a Fedora packaging guidelines POW
it is a mistake).

[PGNet Dev]

hi,

On 6/25/20 11:58 PM, Nicolas Mailhot wrote:
> forgemeta works in release mode, with release archives published over
> http(s). It does not talk at all to source projects using the git
> protocol (and that is intentional, since a lot ot things tooling-side
> do not talk the git protocol and will never talk the git protocol,
> starting with rpm itself, spectool, etc).

noted.  not clear initially from reading the docs; this helps. thx!
> git is not the only scm system out there.

(snip)

sure.  i'm trying to get forge playing nice with not-included-yet hg sources atm :-)
> From a pure auditing side

...
> Note that your spec as it stands is inherently unsafe since

...
> The correct auditable way

...

(snip)

yup.  and for the moment -- while I'm getting my 'end product' sorted out -- that's intentional, and I'm not concerned with audit trail.

yet.

point taken otherwise.
> So you should have a long list of
> %forgeurlX

that's already changed in my latest ...
> %tagX or %commitX

fair enuf, now that the above is nice-n-clear ...
> and a single %forgemeta -a at the end

was having weird artifacts doing that earlier, so split into one-per-source-target.

once a forge bug (other thread) gets ironed out, that'll get revisited, too.
> That will change soonish BTW, I’m currently doing final testing on code
> that will use a slightly different syntax:
>
> %forge_urlX

(snip)

that'll get announced ... here?

do you have a _rough_ timeframe for when that'll be consistently supported/usable in rpmbuild, mock & COPR?
bunch of moving parts to get in sync ...
> Because not making -a the default and emphasising -z in the
> documentation was comfusing users. -z should be a last resort debuging
> help

it was helpful for debug, here.  and at my early stage, necessary ...
> not the first thing you look at when packaging multiple sources.

yup
> That will leave you with each individual source unpacked and patched in
> %{_builddir}, and needing a some symlinks between %{_builddir}
> subdirectories to construct a unified source trees, but that’s how
> multisource works deep down in rpm, nothing I invented myself.

that's what i'd naively assumed was to cleanly happen when i'd started with this multisource spec.
if this cleans that up, then +1 !
> ... have people butcher it to achiever their aims ...

but, that's _our_ job! ;-)

cheers.

[Nicolas Mailhot]

Le vendredi 26 juin 2020 à 07:41 -0700, PGNet Dev a écrit :
> hi,
>
> On 6/25/20 11:58 PM, Nicolas Mailhot wrote:
> > forgemeta works in release mode, with release archives published
> > over
> > http(s). It does not talk at all to source projects using the git
> > protocol (and that is intentional, since a lot ot things tooling-
> > side
> > do not talk the git protocol and will never talk the git protocol,
> > starting with rpm itself, spectool, etc).
>
> noted.  not clear initially from reading the docs; this helps. thx!
> > git is not the only scm system out there.
>
> (snip)
>
> sure.  i'm trying to get forge playing nice with not-included-yet hg
> sources atm :-)

I know some people have used it successfully with birbucket hg
projects, for example. The code is pretty SCM agnostic, it "only"
constructs API URLs for the major forge services out there.
> yup.  and for the moment -- while I'm getting my 'end product' sorted
> out -- that's intentional, and I'm not concerned with audit trail.

That’s fine as long as you know the limitations, and that is why I made
the technical side of the macros support this workflow too.
> > So you should have a long list of
> > %forgeurlX
>
> that's already changed in my latest ...
> > %tagX or %commitX
>
> fair enuf, now that the above is nice-n-clear ...
>
> that'll get announced ... here?

That will be announced on devel and packaging once there is a redhat-
rpm-config PR and a set of example copr builds to look at. The aim is
to get it done soonish, so i18n can decide if they want to push a
change that depends on it before the F33 change deadline (30th june).

First set of unit tests and test builds is green, so it should not take
long now.
> do you have a _rough_ timeframe for when that'll be consistently
> supported/usable in rpmbuild, mock & COPR?

This is very low-level work that only depends on rpm (and often not a
particular rpm version). So typically when it is done it just works in
all upper level tools (spectool, rpmlint, mock, copr, koji, etc).

Now get it done includes a redhat-rpm-config PR merge, and I do not
control how long those take (likewise, it will imply an update of the
forge and fonts packaging guidelines since they both use the same macro
engine, and FPC can ponder things ponderously sometimes).

Regards,

[Nicolas Mailhot]

Le vendredi 26 juin 2020 à 07:41 -0700, PGNet Dev a écrit :
> hi,
>
> On 6/25/20 11:58 PM, Nicolas Mailhot wrote:
> > forgemeta works in release mode, with release archives published
> > over
> > http(s). It does not talk at all to source projects using the git
> > protocol (and that is intentional, since a lot ot things tooling-
> > side
> > do not talk the git protocol and will never talk the git protocol,
> > starting with rpm itself, spectool, etc).
>
> noted.  not clear initially from reading the docs; this helps. thx!

The documentation part of the initial redhat-rpm-config forge
implementation was never merged, and I gave up on refreshing it, so
you’re in traditional unwritten rpm lore land.

Regards,