6:05 a.m.
Hi, all,
I'd like to discuss how we can add Build tag in the RPM.
As one of the key points is to turn it into a common standard for rpm
packages across the ecosystem, the conversation is currently opened
upstream [1] and in RHEL Engineering. And I'd like to get Fedora
community on board.
This is not a finalized proposal and I think it is not ready yet to be
a Fedora Change.
But I want to start a conversation and ask for opinions. There are
also some open questions which need help, especially the requirements
around build reason. And alternative suggestions are welcome as well.
I've posted long version at
https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954
You can comment there (simply login with your FAS id) or here on the
mailing list.
And I am going to update that post with the new feedback.
Shorter version:
================================================
We'd like to introduce Build Number/Tag/Id in the rpm metadata.
By this change we would like to:
* Provide a possibility to change build environment and rebuild rpm
packages without changing their content: neither sources nor spec
files.
* Set a common standard for the RPM-based ecosystem, which can be used
not just within Fedora, but also by Remixes, downstreams, SIGs and
other distributions.
The most visible impact of the proposal would be the filename change:
Current: dnf-4.9.0-12.fc36.noarch.rpm
Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
It can be implemented in two steps:
1) Now → Compatibility mode
* Introduce Build tag in the rpm metadata
* Introduce “build reason” to be added to rpm changelog as a top entry
* Enable passing Build tag value to the build in Koji. The value of
the Build tag will be set from Koji build id.
* Introduce macro in Release field of the rpm spec files, which adds
Build tag after the usual disttag
Release: 12.%{?dist}%{?build:.%{build}}
* Introduce option to pass “build reason” to a Koji build via koji cli
and fedpkg/centpkg tooling.
2) Compatibility mode → Final
* Implement support for the upgrade path on the rpm side in a
compatible way. So that NV(R’=R+B) and NVRB are treated the same.
* Remove %{build} part from Release tag and use independent Build tag
for versioning.
================================================
[1] https://github.com/rpm-software-management/rpm/discussions/2031
--
Aleksandra Fedorova
CI for Fedora/CentOS Stream/RHEL
bookwar on Matrix/IRC
6:24 a.m.
On 18/06/2022 13:05, Aleksandra Fedorova wrote:
* Provide a possibility to change build environment and rebuild rpm
packages without changing their content: neither sources nor spec
files.
I have a better solution - let's move all packages to %autorelease +
%autochangelog.
Release: 12.%{?dist}%{?build:.%{build}}
Looks ugly.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
6:43 a.m.
Hi,
On Sat, Jun 18, 2022 at 1:24 PM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 18/06/2022 13:05, Aleksandra Fedorova wrote:
> * Provide a possibility to change build environment and rebuild rpm
> packages without changing their content: neither sources nor spec
> files.
I have a better solution - let's move all packages to %autorelease +
%autochangelog.
rpmautospec is a shortcut which allows you to not fill Release tag in
the dist-git manually. But to rebuild a package you still need at
least one additional commit to dist-git, which will change the value
that macro calculates. In the end the number is tied to dist-git
history.
rpmautospec automates the Release tag itself (as a version for spec
files), it doesn’t have a Build component in it. Thus, while
rpmautospec is a valuable tool for working on dist-git sources, its
development doesn’t overlap with the Build tag topic.
And, for example, rpmautospec will not help in the case we need to
update a build on pull request update: When you work with
pull-requests you don’t necessarily add commits, you rework the
history of a branch from which you run a PR. Sometimes even reducing
the number of commits in it.
(I added this answer as a section to the main post.)
> Release: 12.%{?dist}%{?build:.%{build}}
Looks ugly.
1) This is a temporary solution until we get full support of the Build
tag upstream
2) It probably can be simplified.
I am no expert in RPM macro language, but, for example, if we agree to
set default Build value to 0, it can be just Release:
12.%{?dist}.%{build}
But this is indeed one of the open questions, where we are looking for
more options.
--
Aleksandra Fedorova
bookwar
6:57 a.m.
On 18/06/2022 13:43, Aleksandra Fedorova wrote:
And, for example, rpmautospec will not help in the case we need to
update a build on pull request update: When you work with
pull-requests you don’t necessarily add commits, you rework the
history of a branch from which you run a PR. Sometimes even reducing
the number of commits in it.
When a PR is merged, the Release will be incremented and a new changelog
entry will be added based on the merged commits.
I am no expert in RPM macro language, but, for example, if we agree
to
set default Build value to 0, it can be just Release:
12.%{?dist}.%{build}
It will produce $name-$version-12.%{?dist}.0. Instead, you should use
%{nil} to omit this field.
1) This is a temporary solution until we get full support of the
Build
tag upstream
I think a new RPM field BuildTag needs to be introduced. It it is set,
it will be added to the package name (just like the optional Epoch).
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
3:47 a.m.
V Sat, Jun 18, 2022 at 01:57:30PM +0200, Vitaly Zaitsev via devel napsal(a):
On 18/06/2022 13:43, Aleksandra Fedorova wrote:
> And, for example, rpmautospec will not help in the case we need to
> update a build on pull request update: When you work with
> pull-requests you don’t necessarily add commits, you rework the
> history of a branch from which you run a PR. Sometimes even reducing
> the number of commits in it.
When a PR is merged, the Release will be incremented and a new changelog
entry will be added based on the merged commits.
I think Aleksandra wants to (non-scratch) build all pull requestes before
merging. Those would have identical NVR without buildid. Imagine you have
A commit in dist-git and two competing pull requests B and C:
dist-git HEAD → A
|\
B C
With the build id, you you can build both B and C. Then you can test them.
When you are fine with one of them, let's say B, you merge it with
fast-forward:
A
|\
dist-git HEAD → B C
And you don't need to rebuild and test B again because it has already been
performed.
-- Petr
2:47 p.m.
On Sat, Jun 18, 2022 at 01:05:31PM +0200, Aleksandra Fedorova wrote:
We'd like to introduce Build Number/Tag/Id in the rpm metadata.
I agree with Vitaly: %rpmautospec+%rpmautorelease seem to cover
most of this functionality in a reasonable way. Maybe there are other
motivations / use-cases that weren't discussed that couldn't be solved
by our existing tools, but so far I don't see that.
There were two/three reasons listed:
By this change we would like to:
* Provide a possibility to change build environment and rebuild rpm
packages without changing their content: neither sources nor spec
files.
Strictly speaking, rpmautospec covers this already.
On Mon, Jun 20, 2022 at 01:45:39PM +0200, Vitaly Zaitsev via devel wrote:
On 20/06/2022 10:47, Petr Pisar wrote:
> I think Aleksandra wants to (non-scratch) build all pull requestes before
> merging.
Official non-scratch builds for non-merged pull requests? Looks very
dangerous.
Yeah. I think we should think about scratch and non-scratch builds
separately. For scratch builds, e.g. in a pull request, it's just fine
to do repeated builds with the same NEVR. For non-scratch builds, the
fact that you cannot do that is a feature.
* Set a common standard for the RPM-based ecosystem, which can be
used
not just within Fedora, but also by Remixes, downstreams, SIGs and
other distributions.
So far downstreams and other distros used a different dist tag. I think
this is reasonable. And if they're doing other modifications apart
from a rebuild, an %autochangelog entry generated from a dist-git commit
seems better than a yet-another mechanism to provide information about
the rebuild. In particular, a build-counter doesn't see to be good fit
for this, since we'd want to distinguish different vendors that do
the build, and with a counter we can't do that and in fact risk confusion
if multiple downstreams do builds with the same number.
The most visible impact of the proposal would be the filename
change:
Current: dnf-4.9.0-12.fc36.noarch.rpm
Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
The history of development of rpm and the ecosystem has shown that
modifications that are visibile at the level of the output binary rpm
have a long implementation tail for the ecosystem. In particular, if
we allow add the build-number information, many many consumers will
need to adjust for this, from trivial things like regexps to match
'%dist.rpm' in the filename to complicated things that extract and
make use of the version field. So if we want to add a new feature
here, a much strong justification why what we have already is not
enough would need to be provided.
Zbyszek
5:13 p.m.
On Mon, Jun 20, 2022 at 3:48 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Sat, Jun 18, 2022 at 01:05:31PM +0200, Aleksandra Fedorova wrote:
> We'd like to introduce Build Number/Tag/Id in the rpm metadata.
I agree with Vitaly: %rpmautospec+%rpmautorelease seem to cover
most of this functionality in a reasonable way. Maybe there are other
motivations / use-cases that weren't discussed that couldn't be solved
by our existing tools, but so far I don't see that.
There were two/three reasons listed:
> By this change we would like to:
> * Provide a possibility to change build environment and rebuild rpm
> packages without changing their content: neither sources nor spec
> files.
Strictly speaking, rpmautospec covers this already.
It does not, because the necessary features to support build counters
were deleted from rpmautospec. That said, this feature should not
require adoption of rpmautospec, and there's no technical reason for
that. Coupling this to rpmautospec is unnecessary and IMO unwanted.
On Mon, Jun 20, 2022 at 01:45:39PM +0200, Vitaly Zaitsev via devel
wrote:
> On 20/06/2022 10:47, Petr Pisar wrote:
> > I think Aleksandra wants to (non-scratch) build all pull requestes before
> > merging.
>
> Official non-scratch builds for non-merged pull requests? Looks very
> dangerous.
Yeah. I think we should think about scratch and non-scratch builds
separately. For scratch builds, e.g. in a pull request, it's just fine
to do repeated builds with the same NEVR. For non-scratch builds, the
fact that you cannot do that is a feature.
A merge should trigger a non-scratch build in a side-tag that can
automerge once Koschei rebuilds are completed in the side tag.
> * Set a common standard for the RPM-based ecosystem, which can
be used
> not just within Fedora, but also by Remixes, downstreams, SIGs and
> other distributions.
So far downstreams and other distros used a different dist tag. I think
this is reasonable. And if they're doing other modifications apart
from a rebuild, an %autochangelog entry generated from a dist-git commit
seems better than a yet-another mechanism to provide information about
the rebuild. In particular, a build-counter doesn't see to be good fit
for this, since we'd want to distinguish different vendors that do
the build, and with a counter we can't do that and in fact risk confusion
if multiple downstreams do builds with the same number.
We have a better way to do it at the DNF level: using sticky vendors.
It's not turned on yet, but it is something I'd like to do eventually.
It uses the Vendor tag to differentiate sources. Fedora, COPR, OBS,
RPM Fusion, CentOS, CentOS SIGs, and Red Hat all use different vendors
now, so this will be possible. I'm already trying this feature out in
CentOS Hyperscale with decent success.
> The most visible impact of the proposal would be the filename
change:
>
> Current: dnf-4.9.0-12.fc36.noarch.rpm
> Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
The history of development of rpm and the ecosystem has shown that
modifications that are visibile at the level of the output binary rpm
have a long implementation tail for the ecosystem. In particular, if
we allow add the build-number information, many many consumers will
need to adjust for this, from trivial things like regexps to match
'%dist.rpm' in the filename to complicated things that extract and
make use of the version field. So if we want to add a new feature
here, a much strong justification why what we have already is not
enough would need to be provided.
This is already something people have to expect, since our existing
policy permits a tailing number and it is used for various purposes.
--
真実はいつも一つ！/ Always, there's only one truth!
5:27 p.m.
On Tue, Jun 21, 2022 at 12:15 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Mon, Jun 20, 2022 at 3:48 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Sat, Jun 18, 2022 at 01:05:31PM +0200, Aleksandra Fedorova wrote:
> > The most visible impact of the proposal would be the filename change:
> >
> > Current: dnf-4.9.0-12.fc36.noarch.rpm
> > Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
>
> The history of development of rpm and the ecosystem has shown that
> modifications that are visibile at the level of the output binary rpm
> have a long implementation tail for the ecosystem. In particular, if
> we allow add the build-number information, many many consumers will
> need to adjust for this, from trivial things like regexps to match
> '%dist.rpm' in the filename to complicated things that extract and
> make use of the version field. So if we want to add a new feature
> here, a much strong justification why what we have already is not
> enough would need to be provided.
>
This is already something people have to expect, since our existing
policy permits a tailing number and it is used for various purposes.
Well, it's complicated.
There's currently a 100% foolproof way to parse NEVRs into their
components, because there are *always* two "-" separators (when
counting from the end) that separate "name", "epoch:version", and
"release" (i.e. you can do nevr.rsplit("-", 2) in Python, and you get
100% correct results for 100% of packages in Fedora repositories).
If anything, the "Build" tag should be separated by a hyphen, not by a
dot, because that makes it look like it's part of the Release tag or
the %dist macro (i.e. it should be N-E:V-R-B, not N-E:V-R.B).
However, by adding another component to the end that's also separated
by a hyphen the fool-proof parsing breaks catastrophically, and you'd
have to fall back to (possibly incorrect) heuristics for checking
whether the NEVR has three components or is in the new NEVRB format.
But separating the Build tag from the Release tag by a dot as
suggested in this proposal would similarly fail to parse unambiguously
- because you can't know whether it's a weirdly high post-%dist
Release increment, or an actual Build tag, as they would both be
valid, convey different semantics, but result in a string with the
same syntax.
Fabio
6:09 p.m.
On Mon, Jun 20, 2022 at 6:28 PM Fabio Valentini <decathorpe(a)gmail.com> wrote:
On Tue, Jun 21, 2022 at 12:15 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
>
> On Mon, Jun 20, 2022 at 3:48 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
> >
> > On Sat, Jun 18, 2022 at 01:05:31PM +0200, Aleksandra Fedorova wrote:
> > > The most visible impact of the proposal would be the filename change:
> > >
> > > Current: dnf-4.9.0-12.fc36.noarch.rpm
> > > Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
> >
> > The history of development of rpm and the ecosystem has shown that
> > modifications that are visibile at the level of the output binary rpm
> > have a long implementation tail for the ecosystem. In particular, if
> > we allow add the build-number information, many many consumers will
> > need to adjust for this, from trivial things like regexps to match
> > '%dist.rpm' in the filename to complicated things that extract and
> > make use of the version field. So if we want to add a new feature
> > here, a much strong justification why what we have already is not
> > enough would need to be provided.
> >
>
> This is already something people have to expect, since our existing
> policy permits a tailing number and it is used for various purposes.
Well, it's complicated.
There's currently a 100% foolproof way to parse NEVRs into their
components, because there are *always* two "-" separators (when
counting from the end) that separate "name", "epoch:version", and
"release" (i.e. you can do nevr.rsplit("-", 2) in Python, and you
get
100% correct results for 100% of packages in Fedora repositories).
If anything, the "Build" tag should be separated by a hyphen, not by a
dot, because that makes it look like it's part of the Release tag or
the %dist macro (i.e. it should be N-E:V-R-B, not N-E:V-R.B).
However, by adding another component to the end that's also separated
by a hyphen the fool-proof parsing breaks catastrophically, and you'd
have to fall back to (possibly incorrect) heuristics for checking
whether the NEVR has three components or is in the new NEVRB format.
But separating the Build tag from the Release tag by a dot as
suggested in this proposal would similarly fail to parse unambiguously
- because you can't know whether it's a weirdly high post-%dist
Release increment, or an actual Build tag, as they would both be
valid, convey different semantics, but result in a string with the
same syntax.
The rule is already violated when you evaluate the full NEVRA.
Changing it to NEVRABA doesn't make it significantly worse.
If it were up to me, I'd rejigger things to have NEVRBDA instead:
name-epoch:version-release.build.disttag.arch. However, doing such a
thing has more consequences for the semantics of version comparison.
I've seen and worked with EVRD before, and it's quite handy, but
that's not an upstream thing.
EVRB would be more useful since it solves concrete issues we have
right now for rebuild management (note not *tracking* which is
something I explicitly don't care about). We can choose to use a dash
to separate release and build, or keep using a period. It doesn't
matter to me one way or another.
--
真実はいつも一つ！/ Always, there's only one truth!
4:58 a.m.
On Tue, Jun 21, 2022 at 12:27:57AM +0200, Fabio Valentini wrote:
On Tue, Jun 21, 2022 at 12:15 AM Neal Gompa
<ngompa13(a)gmail.com> wrote:
>
> On Mon, Jun 20, 2022 at 3:48 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
> >
> > On Sat, Jun 18, 2022 at 01:05:31PM +0200, Aleksandra Fedorova wrote:
> > > The most visible impact of the proposal would be the filename change:
> > >
> > > Current: dnf-4.9.0-12.fc36.noarch.rpm
> > > Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
> >
> > The history of development of rpm and the ecosystem has shown that
> > modifications that are visibile at the level of the output binary rpm
> > have a long implementation tail for the ecosystem. In particular, if
> > we allow add the build-number information, many many consumers will
> > need to adjust for this, from trivial things like regexps to match
> > '%dist.rpm' in the filename to complicated things that extract and
> > make use of the version field. So if we want to add a new feature
> > here, a much strong justification why what we have already is not
> > enough would need to be provided.
> >
>
> This is already something people have to expect, since our existing
> policy permits a tailing number and it is used for various purposes.
Well, it's complicated.
There's currently a 100% foolproof way to parse NEVRs into their
components, because there are *always* two "-" separators (when
counting from the end) that separate "name", "epoch:version", and
"release" (i.e. you can do nevr.rsplit("-", 2) in Python, and you
get
100% correct results for 100% of packages in Fedora repositories).
That is already broken today with modules where the stream which is included in
the name can have '-' in them.
Pierre
6:45 a.m.
V Tue, Jun 21, 2022 at 11:58:02AM +0200, Pierre-Yves Chibon napsal(a):
On Tue, Jun 21, 2022 at 12:27:57AM +0200, Fabio Valentini wrote:
> There's currently a 100% foolproof way to parse NEVRs into their
> components, because there are *always* two "-" separators (when
> counting from the end) that separate "name", "epoch:version",
and
> "release" (i.e. you can do nevr.rsplit("-", 2) in Python, and
you get
> 100% correct results for 100% of packages in Fedora repositories).
That is already broken today with modules where the stream which is included in
the name can have '-' in them.
I believe that the hyphen is replaced with an underscore when building
a modular package.
-- Petr
2:50 p.m.
On Mon, Jun 20, 2022 at 09:47:27PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
> Official non-scratch builds for non-merged pull requests? Looks
very
> dangerous.
Yeah. I think we should think about scratch and non-scratch builds
separately. For scratch builds, e.g. in a pull request, it's just fine
to do repeated builds with the same NEVR. For non-scratch builds, the
fact that you cannot do that is a feature.
The advantage I see in the proposal is that the final build gets promoted,
so you have an artifact which has, theoretically at least, been under
scrutiny, rather than a new one which we _hope_ is the same.
But I'm not sure that really buys us a lot in quality, since the build
environment updates. If promoting the scratch build avoids a problem that
would have happened if it were rebuilt... well, that problem is going to
come up later during a mass or automatic rebuild later anyway.
It does, of course, reduce extraneous throw-away computing.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3:28 a.m.
V Tue, Jun 21, 2022 at 03:50:31PM -0400, Matthew Miller napsal(a):
On Mon, Jun 20, 2022 at 09:47:27PM +0200, Zbigniew Jędrzejewski-Szmek
wrote:
> > Official non-scratch builds for non-merged pull requests? Looks very
> > dangerous.
>
> Yeah. I think we should think about scratch and non-scratch builds
> separately. For scratch builds, e.g. in a pull request, it's just fine
> to do repeated builds with the same NEVR. For non-scratch builds, the
> fact that you cannot do that is a feature.
The advantage I see in the proposal is that the final build gets promoted,
so you have an artifact which has, theoretically at least, been under
scrutiny, rather than a new one which we _hope_ is the same.
But I'm not sure that really buys us a lot in quality, since the build
environment updates. If promoting the scratch build avoids a problem that
would have happened if it were rebuilt... well, that problem is going to
come up later during a mass or automatic rebuild later anyway.
The same applies to nonscratch builds. We do not rebuild packages when moving
them from updates-testing to updates repository only to make sure they still
build or fit to the environment.
-- Petr
4:07 a.m.
On Mon, Jun 20, 2022 at 1:46 PM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 20/06/2022 10:47, Petr Pisar wrote:
> I think Aleksandra wants to (non-scratch) build all pull requestes before
> merging.
Official non-scratch builds for non-merged pull requests? Looks very
dangerous.
I actually think it's a good idea.
Scratch builds always sounded like a strange idea, mainly b/c it's
forcing us to decide if the output of a build is going to be released
*before* the build is triggered, when it's impossible to tell whether
that output is going to be release-worthy at all.
I can imagine a setup where the build system doesn't differentiate
between builds. There are just builds. And the update system (or
something else) is where the decision is taken to promote a build as
an update or not, *after* the output of the build is known. When a
build is released, it's marked in the build system as "official".
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
5:23 a.m.
On Wed, Jun 22, 2022 at 11:08 AM Hunor Csomortáni <csomh(a)redhat.com> wrote:
On Mon, Jun 20, 2022 at 1:46 PM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> On 20/06/2022 10:47, Petr Pisar wrote:
> > I think Aleksandra wants to (non-scratch) build all pull requestes before
> > merging.
>
> Official non-scratch builds for non-merged pull requests? Looks very
> dangerous.
I actually think it's a good idea.
Actually, it's not. There are multiple good technical and legal
reasons to not do that.
Scratch builds always sounded like a strange idea, mainly b/c
it's
forcing us to decide if the output of a build is going to be released
*before* the build is triggered, when it's impossible to tell whether
that output is going to be release-worthy at all.
I can imagine a setup where the build system doesn't differentiate
between builds. There are just builds. And the update system (or
something else) is where the decision is taken to promote a build as
an update or not, *after* the output of the build is known. When a
build is released, it's marked in the build system as "official".
Strange - what you're describing here is what we already have:
Gating (for Rawhide) and the updates-testing repo (for stable releases).
Those two mechanisms provide exactly the "bless this as good"
mechanism for "official" builds that you want.
If a build doesn't pass gating tests, or is voted down in bodhi, it
won't be pushed to buildroots or to users.
No scratch builds involved.
In my understanding (and this is also how I use them), scratch builds
are just that - test builds that can be thrown away.
Most of the time, I just need to test whether a build succeeds in
koji, most often because I need to test an architecture that is very
slow to emulate locally.
It's "nice" that PRs on src.fp.o trigger test scratch builds, just
because it tells me whether the change actually builds or not.
But the idea to reuse such test builds as "official builds" has never
even crossed my mind.
Additionally, *IF* we'd want to do something like that, we'd need a
*very strict* policy that determines exactly *which* scratch builds
would actually eligible for being "promoted" to an "official build":
- was it built against the correct buildroot (i.e. fNN-build or a
valid side tag)?
- was it built recently enough (i.e. there haven't been any SONAME or
Python or Perl or X version bumps that would affect the build in the
meantime)?
- was it built from the correct dist-git repo (i.e. not a fork)?
- or if it *was* built from a fork, has the commit also been merged
into the correct dist-git branch?
- is the commit it was built from on the correct dist-git branch (i.e.
the branch matches the target release)?
- was it *DEFINITELY 100% SURE* not built from a manually uploaded SRPM file?
- etc.
At this point, I'd wager most scratch builds wouldn't even qualify. :)
Fabio
6:10 a.m.
V Wed, Jun 22, 2022 at 12:23:21PM +0200, Fabio Valentini napsal(a):
On Wed, Jun 22, 2022 at 11:08 AM Hunor Csomortáni
<csomh(a)redhat.com> wrote:
>
> On Mon, Jun 20, 2022 at 1:46 PM Vitaly Zaitsev via devel
> <devel(a)lists.fedoraproject.org> wrote:
> >
> > On 20/06/2022 10:47, Petr Pisar wrote:
> > > I think Aleksandra wants to (non-scratch) build all pull requestes before
> > > merging.
> >
> > Official non-scratch builds for non-merged pull requests? Looks very
> > dangerous.
>
> I actually think it's a good idea.
Actually, it's not. There are multiple good technical and legal
reasons to not do that.
I have probably shocking news for you: Pull requests do not live in a parallel
space. When a pull request is opened, a system copies the commits into the
target repository. So the repository is already tainted whether you do or
don't do the build. The same feature allows you to retain the sources even
though the forked repository is deleted.
Although the pull-requesting systems usually make the pull request commits
available under handy ref names (e.g. Pagure offers refs/pull/<PR#>/head), the
commits are always reachable with their commit ID. See this GitHub afair when
"Linus deleted a Linux"
<https://github.com/torvalds/linux/tree/8bcab0346d4fcf21b97046eb44db8cf37d...;.
You can try to clone rpms/cvs repository and then do "git fetch origin
b3a9a1f11152fef8dc52080ee85040b9a5cb8630". You will get a commit which has
never been merged into Fedora's dist-git.
I know, dist-git is not only a git. It's also a lookaside cache for sources.
And I can calm the audience: Koji won't built from that commit because it's
not in the list of refs which are fetched by git by default
<https://koji.fedoraproject.org/koji/taskinfo?taskID=88588941>;.
-- Petr
9 a.m.
On Wed, Jun 22, 2022 at 01:10:57PM +0200, Petr Pisar wrote:
I have probably shocking news for you: Pull requests do not live in a
parallel
This seems like it's getting a little heated and going towards
personally-directed rhetoric. Can we please not do that? Thank you.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
5:50 a.m.
On 22/06/2022 11:07, Hunor Csomortáni wrote:
I actually think it's a good idea.
Scratch builds always sounded like a strange idea, mainly b/c it's
forcing us to decide if the output of a build is going to be released
*before* the build is triggered, when it's impossible to tell whether
that output is going to be release-worthy at all.
Everyone can open a pull request. Thus, they can easily include code
with malware or backdoors inside, or a copyrighted content.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
7:05 a.m.
Am 18.06.22 um 13:05 schrieb Aleksandra Fedorova:
Hi, all,
I'd like to discuss how we can add Build tag in the RPM.
As one of the key points is to turn it into a common standard for rpm
packages across the ecosystem, the conversation is currently opened
upstream [1] and in RHEL Engineering. And I'd like to get Fedora
community on board.
This is not a finalized proposal and I think it is not ready yet to be
a Fedora Change.
But I want to start a conversation and ask for opinions. There are
also some open questions which need help, especially the requirements
around build reason. And alternative suggestions are welcome as well.
I've posted long version at
https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954
You can comment there (simply login with your FAS id) or here on the
mailing list.
And I am going to update that post with the new feedback.
Shorter version:
================================================
We'd like to introduce Build Number/Tag/Id in the rpm metadata.
I am strongly opposed to this proposal.
It is just useless featuritis and doesn't make any sense at all, breaks
lots of tools/scripts, breaks 3rd party packaging conventions, and many
more ...
Ralf
7:09 a.m.
On 18/06/2022 14:05, Ralf Corsépius wrote:
It is just useless featuritis and doesn't make any sense at all,
breaks
lots of tools/scripts, breaks 3rd party packaging conventions, and many
more ...
If an optional RPM BuildTag is introduced, it won't break anything. If
you don't need it, you can simply omit it from SPEC, just like Epoch.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
3:05 p.m.
On Sat, Jun 18, 2022 at 01:05:31PM +0200, Aleksandra Fedorova wrote:
Hi, all,
I'd like to discuss how we can add Build tag in the RPM.
As one of the key points is to turn it into a common standard for rpm
packages across the ecosystem, the conversation is currently opened
upstream [1] and in RHEL Engineering. And I'd like to get Fedora
community on board.
This is not a finalized proposal and I think it is not ready yet to be
a Fedora Change.
But I want to start a conversation and ask for opinions. There are
also some open questions which need help, especially the requirements
around build reason. And alternative suggestions are welcome as well.
I've posted long version at
https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954
You can comment there (simply login with your FAS id) or here on the
mailing list.
And I am going to update that post with the new feedback.
Shorter version:
================================================
We'd like to introduce Build Number/Tag/Id in the rpm metadata.
By this change we would like to:
* Provide a possibility to change build environment and rebuild rpm
packages without changing their content: neither sources nor spec
files.
* Set a common standard for the RPM-based ecosystem, which can be used
not just within Fedora, but also by Remixes, downstreams, SIGs and
other distributions.
The most visible impact of the proposal would be the filename change:
Current: dnf-4.9.0-12.fc36.noarch.rpm
Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
It can be implemented in two steps:
1) Now → Compatibility mode
* Introduce Build tag in the rpm metadata
* Introduce “build reason” to be added to rpm changelog as a top entry
* Enable passing Build tag value to the build in Koji. The value of
the Build tag will be set from Koji build id.
* Introduce macro in Release field of the rpm spec files, which adds
Build tag after the usual disttag
Release: 12.%{?dist}%{?build:.%{build}}
* Introduce option to pass “build reason” to a Koji build via koji cli
and fedpkg/centpkg tooling.
2) Compatibility mode → Final
* Implement support for the upgrade path on the rpm side in a
compatible way. So that NV(R’=R+B) and NVRB are treated the same.
* Remove %{build} part from Release tag and use independent Build tag
for versioning.
================================================
[1] https://github.com/rpm-software-management/rpm/discussions/2031
I think I understand the problem this is trying to solve, but I have issues
with the proposed implementation. Of course, this assumes I understand the
problem to solve. My understanding is that we can't look at a built RPM and
know where it was built and in what environment it was built. If I have
missed something, please let me know.
Issues/concerns:
1) Overloading Release again. We did it first with dist tags and now this
proposal wants to put build-id in there too. Does the build-id really need to
be exposed to users? We have long established the NVR, or NEVRA, naming
mechanism and even to this day people are confused by the dist tag since they
see that in the built package but not in the spec file. Adding build-id will
lead to more confusion and break existing workflows. I am not convinced the
build-id needs to be user exposed.
2) Similar to modularity, build-id should have its own tag in the RPM header.
If this is going to be done, we should do it right from the beginning. I am
aware that modularity dumped a bunch of stuff in Release as well, but I think
that was a mistake too.
3) From reading the thread and proposal, a lot of the build-id proposal feels
like it's trying to make the depsorting work so you always get the latest
version. OK, but that means build-id is being used as another Release value,
which I don't think is a good approach. Release is a string, but in practice
we've always more or less used it as an int if you strip the dist tag part.
Each rebuild of a package that should go to a user should increment this
field. Updating the Version should reset Release to 1. Epoch allows for
resetting Release as well. We've got a lot of knobs here for depsorting and
version sorting to ensure users get the latest build. Let's not add more. If
build-id changing or incrementing does mean a user should get a new build,
then the Release should increment, which leads me to...
4) As others have stated or suggested, we should go all in on rpmautospec. If
we get build-id tied in correctly, we should fully implement rpmautospec so
Release is also constructed by the build system entirely at rpmbuild time. We
can keep spec files clean and maintain that pull request workflow and
cherry-picking across branches without the ever-present collision on Release
and changelog.
5) Also go all in on automating (or removing) the spec file changelog. I
don't understand what the "build reason" is for above, but isn't every
changelog entry in a spec file technically the build reason?
I would rather skip the compatibility mode in the proposal and just have it
focus directly on the final mode.
Thanks,
--
David Cantrell <dcantrell(a)redhat.com>
Red Hat, Inc. | Boston, MA | EST5EDT
2:45 a.m.
On 21/06/2022 22:05, David Cantrell wrote:
5) Also go all in on automating (or removing) the spec file
changelog. I
don't understand what the "build reason" is for above, but isn't every
changelog entry in a spec file technically the build reason?
I also think that every package change (including rebuild) must be
tracked in changelog.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
6:09 a.m.
I also think that every package change (including rebuild) must be
tracked in changelog.
I think that convolution is at the very heart of the problem:
As it is, dist-git tracks "packaging sources", i.e. spec and source hashes or
files, and this determines the content of the src.rpm and its version.
What you get when you build a binary rpm from that src.spm depends very much on the
environment. And that environment is not reflected in the version nor in the built rpm
(besides Build Host and Date).
We sometimes still are in the old "dist-cvs mindset" where cvs really was not
used as a vcs at all, but as as a way to drive the build infrastructure. But that mindset
will not serve as any longer. We see that problem with every mass rebuild, such as now
with pyth 3.11: Basically, one has to grep the changelog, i.e. unstructured textual
information, to find out which pakcage has been rebuilt. If you built your package in
rawhide after py3.11 was merged but didn't have that changelog entry it got rebuilt
again, with an extra changelog entry. (If you pushed a fake changelog entry before the
merge then...). Merge rawhide into f36 to get clean dist-git branches and you have a
wrong/misleading changelog entry. (No, there is no py3.11 nor rebuild on f36.)
In addition, you don't know which defines were in effect during a build. (rpkg kind of
solves that by having unparsed and parsed spec, at least for the rpkg macros.)
I really think we need to stop abusing spec/changelog for things which are purely
buildroot related. Let dist-git track whatever is needed to adjust the *package(ing)* to
newer buildroots. Let the binary rpm track what buildroot it has been built against (be it
a builtroot identifier/hash, macro settings used etc), or track it in koji/bodhi where the
binary rpm has been built resp. the update has been pushed.
12:45 p.m.
On Wed, Jun 22, 2022 at 11:09:13AM -0000, Michael J Gruber wrote:
>
> I also think that every package change (including rebuild) must be
> tracked in changelog.
I think that convolution is at the very heart of the problem:
As it is, dist-git tracks "packaging sources", i.e. spec and source hashes or
files, and this determines the content of the src.rpm and its version.
What you get when you build a binary rpm from that src.spm depends very much on the
environment. And that environment is not reflected in the version nor in the built rpm
(besides Build Host and Date).
Well, it's reflected in the requires/provides/contents of the rpm too
though right?
If I build foo-1.0-1 against libbar-2.0-1 and then again against
libbar-4.0-1 its going to require different libs at install/run time.
We sometimes still are in the old "dist-cvs mindset" where
cvs really was not used as a vcs at all, but as as a way to drive the build
infrastructure. But that mindset will not serve as any longer. We see that problem with
every mass rebuild, such as now with pyth 3.11: Basically, one has to grep the changelog,
i.e. unstructured textual information, to find out which pakcage has been rebuilt. If you
built your package in rawhide after py3.11 was merged but didn't have that changelog
entry it got rebuilt again, with an extra changelog entry. (If you pushed a fake changelog
entry before the merge then...). Merge rawhide into f36 to get clean dist-git branches and
you have a wrong/misleading changelog entry. (No, there is no py3.11 nor rebuild on f36.)
Well, this proposal would try and inject some 'build changelog', but I'm
not sure that really tells the entire story either. ;(
In addition, you don't know which defines were in effect during a
build. (rpkg kind of solves that by having unparsed and parsed spec, at least for the rpkg
macros.)
I really think we need to stop abusing spec/changelog for things which are purely
buildroot related. Let dist-git track whatever is needed to adjust the *package(ing)* to
newer buildroots. Let the binary rpm track what buildroot it has been built against (be it
a builtroot identifier/hash, macro settings used etc), or track it in koji/bodhi where the
binary rpm has been built resp. the update has been pushed.
I'm not convinced.
This adds a lot to complexity... there's yet another place to look for
changes, a number to look at to see if your version is the latest one.
kevin
2:28 a.m.
On Wednesday, June 22, 2022 7:45:09 PM CEST Kevin Fenzi wrote:
On Wed, Jun 22, 2022 at 11:09:13AM -0000, Michael J Gruber wrote:
> > I also think that every package change (including rebuild) must be
> > tracked in changelog.
>
> I think that convolution is at the very heart of the problem:
>
> As it is, dist-git tracks "packaging sources", i.e. spec and source
hashes
> or files, and this determines the content of the src.rpm and its version.
>
> What you get when you build a binary rpm from that src.spm depends very
> much on the environment. And that environment is not reflected in the
> version nor in the built rpm (besides Build Host and Date).
Well, it's reflected in the requires/provides/contents of the rpm too
though right?
If I build foo-1.0-1 against libbar-2.0-1 and then again against
libbar-4.0-1 its going to require different libs at install/run time.
The problem is that sometimes you build against libbar-4.0-1 vs. libbar-4.0-2
and you get qualitatively different binary RPMs as the result in each case,
despite you triggered both the builds from the same dist-git commit. This
difference is not always reflected in metadata of the resulting RPMs.
Kamil
4:52 a.m.
On Thu, Jun 23, 2022 at 09:28:47AM +0200, Kamil Dudka wrote:
On Wednesday, June 22, 2022 7:45:09 PM CEST Kevin Fenzi wrote:
> On Wed, Jun 22, 2022 at 11:09:13AM -0000, Michael J Gruber wrote:
> > > I also think that every package change (including rebuild) must be
> > > tracked in changelog.
> >
> > I think that convolution is at the very heart of the problem:
> >
> > As it is, dist-git tracks "packaging sources", i.e. spec and source
hashes
> > or files, and this determines the content of the src.rpm and its version.
> >
> > What you get when you build a binary rpm from that src.spm depends very
> > much on the environment. And that environment is not reflected in the
> > version nor in the built rpm (besides Build Host and Date).
> Well, it's reflected in the requires/provides/contents of the rpm too
> though right?
>
> If I build foo-1.0-1 against libbar-2.0-1 and then again against
> libbar-4.0-1 its going to require different libs at install/run time.
The problem is that sometimes you build against libbar-4.0-1 vs. libbar-4.0-2
and you get qualitatively different binary RPMs as the result in each case,
despite you triggered both the builds from the same dist-git commit. This
difference is not always reflected in metadata of the resulting RPMs.
That is true. But this is a hard problem to solve: some information about
the build environment ends up in the Provides/Requires metadata. We could
include more, that'd take up space in each binary rpm. The proposal that
started this thread adds a free-form field which obviously is not enough
to capture this.
Maybe we should generate a JSON file that describes the build root and
include it in the -debuginfo or -debugsource package? fedpkg or dnf could
then learn how to fetch it nicely.
Zbyszek
5:15 a.m.
On 18. 06. 22 13:05, Aleksandra Fedorova wrote:
Hi, all,
I'd like to discuss how we can add Build tag in the RPM.
As one of the key points is to turn it into a common standard for rpm
packages across the ecosystem, the conversation is currently opened
upstream [1] and in RHEL Engineering. And I'd like to get Fedora
community on board.
This is not a finalized proposal and I think it is not ready yet to be
a Fedora Change.
But I want to start a conversation and ask for opinions. There are
also some open questions which need help, especially the requirements
around build reason. And alternative suggestions are welcome as well.
I've posted long version at
https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954
You can comment there (simply login with your FAS id) or here on the
mailing list.
And I am going to update that post with the new feedback.
Shorter version:
================================================
We'd like to introduce Build Number/Tag/Id in the rpm metadata.
By this change we would like to:
* Provide a possibility to change build environment and rebuild rpm
packages without changing their content: neither sources nor spec
files.
* Set a common standard for the RPM-based ecosystem, which can be used
not just within Fedora, but also by Remixes, downstreams, SIGs and
other distributions.
The most visible impact of the proposal would be the filename change:
Current: dnf-4.9.0-12.fc36.noarch.rpm
Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
It can be implemented in two steps:
1) Now → Compatibility mode
* Introduce Build tag in the rpm metadata
* Introduce “build reason” to be added to rpm changelog as a top entry
* Enable passing Build tag value to the build in Koji. The value of
the Build tag will be set from Koji build id.
* Introduce macro in Release field of the rpm spec files, which adds
Build tag after the usual disttag
Release: 12.%{?dist}%{?build:.%{build}}
* Introduce option to pass “build reason” to a Koji build via koji cli
and fedpkg/centpkg tooling.
2) Compatibility mode → Final
* Implement support for the upgrade path on the rpm side in a
compatible way. So that NV(R’=R+B) and NVRB are treated the same.
* Remove %{build} part from Release tag and use independent Build tag
for versioning.
================================================
[1] https://github.com/rpm-software-management/rpm/discussions/2031
I have couple concerns/questions.
When modularity was introduced, local builds (outside of MSB) were side tracked
not to be part of the MVP. Implementing this later proved out to be rather
complicated. So let's focus on the following gotchas from the start here
instead please:
1) An user rebuilds a package from Fedora dist-git in local mock, what will be
the value of the build tag? How will the local build sort over the official
Fedora builds?
2) A packager rebuilds packages from Fedora dist-git in a Copr repo, what will
be the value of the build tag? How will the Copr build sort over the official
Fedora builds? (This is essentially the same question but the answer might differ.)
3) If every Fedora packager can rebuild anything without a commit, what do we
do prevent accidental builds?
As for your open questions, I think that build reason is useful as a changelog
message (and should be exposed to users).
I also agree with others in this thread that exposing the build ID so
user-visibly isn't great.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
7:24 a.m.
Hi, Miro,
On Thu, Jun 23, 2022 at 12:15 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 18. 06. 22 13:05, Aleksandra Fedorova wrote:
> Hi, all,
>
> I'd like to discuss how we can add Build tag in the RPM.
>
> As one of the key points is to turn it into a common standard for rpm
> packages across the ecosystem, the conversation is currently opened
> upstream [1] and in RHEL Engineering. And I'd like to get Fedora
> community on board.
>
> This is not a finalized proposal and I think it is not ready yet to be
> a Fedora Change.
> But I want to start a conversation and ask for opinions. There are
> also some open questions which need help, especially the requirements
> around build reason. And alternative suggestions are welcome as well.
>
> I've posted long version at
> https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954
>
> You can comment there (simply login with your FAS id) or here on the
> mailing list.
> And I am going to update that post with the new feedback.
>
> Shorter version:
> ================================================
> We'd like to introduce Build Number/Tag/Id in the rpm metadata.
>
> By this change we would like to:
> * Provide a possibility to change build environment and rebuild rpm
> packages without changing their content: neither sources nor spec
> files.
> * Set a common standard for the RPM-based ecosystem, which can be used
> not just within Fedora, but also by Remixes, downstreams, SIGs and
> other distributions.
>
> The most visible impact of the proposal would be the filename change:
>
> Current: dnf-4.9.0-12.fc36.noarch.rpm
> Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
>
> It can be implemented in two steps:
>
> 1) Now → Compatibility mode
>
> * Introduce Build tag in the rpm metadata
> * Introduce “build reason” to be added to rpm changelog as a top entry
> * Enable passing Build tag value to the build in Koji. The value of
> the Build tag will be set from Koji build id.
> * Introduce macro in Release field of the rpm spec files, which adds
> Build tag after the usual disttag
> Release: 12.%{?dist}%{?build:.%{build}}
> * Introduce option to pass “build reason” to a Koji build via koji cli
> and fedpkg/centpkg tooling.
>
> 2) Compatibility mode → Final
>
> * Implement support for the upgrade path on the rpm side in a
> compatible way. So that NV(R’=R+B) and NVRB are treated the same.
> * Remove %{build} part from Release tag and use independent Build tag
> for versioning.
> ================================================
>
> [1] https://github.com/rpm-software-management/rpm/discussions/2031
I have couple concerns/questions.
When modularity was introduced, local builds (outside of MSB) were side tracked
not to be part of the MVP. Implementing this later proved out to be rather
complicated. So let's focus on the following gotchas from the start here
instead please:
I think these are very valid questions. I didn't think about them at
first, but then added at the last moment under the name:
"Cross-distribution upgrade paths" in the doc. But I like how you
mention Copr and local builds as especially important cases of this,
so I will adjust the wording.
I don't have a ready to go answer, because I am not sure about the
goal. How do we want this to work?
Here are just some considerations.
1) An user rebuilds a package from Fedora dist-git in local mock, what will be
the value of the build tag? How will the local build sort over the official
Fedora builds?
Afaik currently, if you do a local build, you need to bump an NVR to
ensure the upgrade path is set. Then if the base distribution upgrades
or rebuilds the package, you would need to bump your version again if
you like to preserve the upgrade path.
This approach would work exactly the same way after the change, as
bumping the Release tag in the sources makes build-related versioning
irrelevant.
We can agree that by default Build tag in mock environment is unset.
Local builds have no build id, and to make clean upgrades bump of
Release tag is required.
But there are some new possibilities:
If you want to create a local rebuild without Release bump, you would
be able to pass the build tag value manually and set to whatever you
want. For example you can add 1 to existing build tag. Or you could
set it to 0. Or to 99999999.
The latter is less likely though.
If I were to create local builds which are always higher or always
lower than builds from the base system, I would probably play with the
dist-tag, not build id:
$ rpmdev-vercmp 0:1.2.3-12.fc37.15 0:1.2.3-12.localfc37.1
0:1.2.3-12.fc37.15 < 0:1.2.3-12.localfc37.1
There is also a possibility to set default local build tag to be the
local timestamp. But I am not sure if it is useful for a local
development.
2) A packager rebuilds packages from Fedora dist-git in a Copr repo,
what will
be the value of the build tag? How will the Copr build sort over the official
Fedora builds? (This is essentially the same question but the answer might differ.)
Similar as above, the questions would be how does it work now, do we
want a change or do we want to keep the current setup?
CC @Miroslav Suchý
3) If every Fedora packager can rebuild anything without a commit,
what do we
do prevent accidental builds?
I think each rebuild should be treated as a new package, thus it would
require a new bodhi update, testing, and signing. Which means it will
be less likely to accidentally ship it.
But as I mentioned in the post, right now we'd like to propose the
refactoring, and not a change of the development workflow. Thus I
would initially restrict the rebuild possibility to the admin group
which handles mass-rebuilds and other admin tasks. Then I would
gradually open it up case by case, each case through a separate
conversation.
Saying that, probably the first case, which I would consider is: is
there a problem of an accidental rebuild of a merged code for Fedora
Rawhide? What would be the reasons for us to _not_ allow it?
As for your open questions, I think that build reason is useful as a changelog
message (and should be exposed to users).
I also agree with others in this thread that exposing the build ID so
user-visibly isn't great.
I am not convinced about it. Technically we can configure filename to
not show build-id at all.
But then we will have a confusion, that upgrade path is not visible
from a filename. And as it was mentioned elsewhere [1] we already have
Epoch tag which works this way, and users hate it for this exact
reason.
I think that cleaner tags and filenames is a good goal.
But I think the requirement "filename should contain all information
which is required to calculate the upgrade path between two builds" is
valid and probably should be included in the definition of that
"cleanliness".
[1]
https://github.com/rpm-software-management/rpm/discussions/2031#discussio...
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
--
Aleksandra Fedorova
bookwar
7:32 a.m.
On Thu, Jun 23, 2022 at 8:25 AM Aleksandra Fedorova <alpha(a)bookwar.info> wrote:
Hi, Miro,
On Thu, Jun 23, 2022 at 12:15 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>
> On 18. 06. 22 13:05, Aleksandra Fedorova wrote:
> > Hi, all,
> >
> > I'd like to discuss how we can add Build tag in the RPM.
> >
> > As one of the key points is to turn it into a common standard for rpm
> > packages across the ecosystem, the conversation is currently opened
> > upstream [1] and in RHEL Engineering. And I'd like to get Fedora
> > community on board.
> >
> > This is not a finalized proposal and I think it is not ready yet to be
> > a Fedora Change.
> > But I want to start a conversation and ask for opinions. There are
> > also some open questions which need help, especially the requirements
> > around build reason. And alternative suggestions are welcome as well.
> >
> > I've posted long version at
> > https://discussion.fedoraproject.org/t/rfc-build-tag-in-rpms/39954
> >
> > You can comment there (simply login with your FAS id) or here on the
> > mailing list.
> > And I am going to update that post with the new feedback.
> >
> > Shorter version:
> > ================================================
> > We'd like to introduce Build Number/Tag/Id in the rpm metadata.
> >
> > By this change we would like to:
> > * Provide a possibility to change build environment and rebuild rpm
> > packages without changing their content: neither sources nor spec
> > files.
> > * Set a common standard for the RPM-based ecosystem, which can be used
> > not just within Fedora, but also by Remixes, downstreams, SIGs and
> > other distributions.
> >
> > The most visible impact of the proposal would be the filename change:
> >
> > Current: dnf-4.9.0-12.fc36.noarch.rpm
> > Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
> >
> > It can be implemented in two steps:
> >
> > 1) Now → Compatibility mode
> >
> > * Introduce Build tag in the rpm metadata
> > * Introduce “build reason” to be added to rpm changelog as a top entry
> > * Enable passing Build tag value to the build in Koji. The value of
> > the Build tag will be set from Koji build id.
> > * Introduce macro in Release field of the rpm spec files, which adds
> > Build tag after the usual disttag
> > Release: 12.%{?dist}%{?build:.%{build}}
> > * Introduce option to pass “build reason” to a Koji build via koji cli
> > and fedpkg/centpkg tooling.
> >
> > 2) Compatibility mode → Final
> >
> > * Implement support for the upgrade path on the rpm side in a
> > compatible way. So that NV(R’=R+B) and NVRB are treated the same.
> > * Remove %{build} part from Release tag and use independent Build tag
> > for versioning.
> > ================================================
> >
> > [1] https://github.com/rpm-software-management/rpm/discussions/2031
>
> I have couple concerns/questions.
>
> When modularity was introduced, local builds (outside of MSB) were side tracked
> not to be part of the MVP. Implementing this later proved out to be rather
> complicated. So let's focus on the following gotchas from the start here
> instead please:
I think these are very valid questions. I didn't think about them at
first, but then added at the last moment under the name:
"Cross-distribution upgrade paths" in the doc. But I like how you
mention Copr and local builds as especially important cases of this,
so I will adjust the wording.
I don't have a ready to go answer, because I am not sure about the
goal. How do we want this to work?
Here are just some considerations.
>
> 1) An user rebuilds a package from Fedora dist-git in local mock, what will be
> the value of the build tag? How will the local build sort over the official
> Fedora builds?
Afaik currently, if you do a local build, you need to bump an NVR to
ensure the upgrade path is set. Then if the base distribution upgrades
or rebuilds the package, you would need to bump your version again if
you like to preserve the upgrade path.
This approach would work exactly the same way after the change, as
bumping the Release tag in the sources makes build-related versioning
irrelevant.
We can agree that by default Build tag in mock environment is unset.
Local builds have no build id, and to make clean upgrades bump of
Release tag is required.
But there are some new possibilities:
If you want to create a local rebuild without Release bump, you would
be able to pass the build tag value manually and set to whatever you
want. For example you can add 1 to existing build tag. Or you could
set it to 0. Or to 99999999.
The latter is less likely though.
If I were to create local builds which are always higher or always
lower than builds from the base system, I would probably play with the
dist-tag, not build id:
$ rpmdev-vercmp 0:1.2.3-12.fc37.15 0:1.2.3-12.localfc37.1
0:1.2.3-12.fc37.15 < 0:1.2.3-12.localfc37.1
There is also a possibility to set default local build tag to be the
local timestamp. But I am not sure if it is useful for a local
development.
I think it makes sense to just set it to zero by default.
> 2) A packager rebuilds packages from Fedora dist-git in a Copr
repo, what will
> be the value of the build tag? How will the Copr build sort over the official
> Fedora builds? (This is essentially the same question but the answer might differ.)
Similar as above, the questions would be how does it work now, do we
want a change or do we want to keep the current setup?
CC @Miroslav Suchý
I think we'll want to introduce sticky vendors to the Fedora ecosystem
to deal with this. Otherwise we'll have EVRB games between sources.
Users generally want packages coming from a particular source and
following that track makes sense.
DNF has the capability to do this if you set
"allow_vendor_change=False" in dnf.conf.
See: https://dnf.readthedocs.io/en/latest/conf_ref.html#allow-vendor-change-label
> 3) If every Fedora packager can rebuild anything without a
commit, what do we
> do prevent accidental builds?
I think each rebuild should be treated as a new package, thus it would
require a new bodhi update, testing, and signing. Which means it will
be less likely to accidentally ship it.
But as I mentioned in the post, right now we'd like to propose the
refactoring, and not a change of the development workflow. Thus I
would initially restrict the rebuild possibility to the admin group
which handles mass-rebuilds and other admin tasks. Then I would
gradually open it up case by case, each case through a separate
conversation.
Saying that, probably the first case, which I would consider is: is
there a problem of an accidental rebuild of a merged code for Fedora
Rawhide? What would be the reasons for us to _not_ allow it?
The most compelling reason would be storage. Koji keeps the binaries
of every successful build. We need some policy changes to keep this
from turning into even more of a problem than it is now.
>
> As for your open questions, I think that build reason is useful as a changelog
> message (and should be exposed to users).
>
> I also agree with others in this thread that exposing the build ID so
> user-visibly isn't great.
>
I am not convinced about it. Technically we can configure filename to
not show build-id at all.
But then we will have a confusion, that upgrade path is not visible
from a filename. And as it was mentioned elsewhere [1] we already have
Epoch tag which works this way, and users hate it for this exact
reason.
I think that cleaner tags and filenames is a good goal.
But I think the requirement "filename should contain all information
which is required to calculate the upgrade path between two builds" is
valid and probably should be included in the definition of that
"cleanliness".
[1]
https://github.com/rpm-software-management/rpm/discussions/2031#discussio...
It's important for a non-obvious reason: it's pretty much the only way
to ensure when repodata is made, that multiple builds can live safely
in the same repository.
--
真実はいつも一つ！/ Always, there's only one truth!
8:01 a.m.
On 23. 06. 22 14:24, Aleksandra Fedorova wrote:
> 2) A packager rebuilds packages from Fedora dist-git in a Copr
repo, what will
> be the value of the build tag? How will the Copr build sort over the official
> Fedora builds? (This is essentially the same question but the answer might differ.)
Similar as above, the questions would be how does it work now, do we
want a change or do we want to keep the current setup?
Examples from before the Python 3.11 side tag was merged.
Fedora rawhide: python3-tomli-2.0.1-2.fc37 (built with Python 3.10)
Python 3.11 copr: python3-tomli-2.0.1-2.fc37 (built with Python 3.11)
The NEVRs are identical. Other builds in the Python 3.11 copr install the 3.11
build.
Now when we managed to not pick up a certain bump:
Fedora rawhide: python3-tomli-2.0.1-2.fc37 (built with Python 3.10)
Python 3.11 copr: python3-tomli-2.0.1-1.fc37 (built with Python 3.11)
The rawhide's NEVR is higher. Other builds in the Python 3.11 copr install try
to install the 3.10 build and that conflicts with other stuff in the copr and
the dependency resolution fails.
We certainly don't want to regress in that regard.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
2:30 a.m.
New subject: [Fedora-packaging] Re: [RFC] Build tag in RPM: from NVR to NVRB
On Thursday, 23 June 2022 at 15:01, Miro Hrončok wrote:
On 23. 06. 22 14:24, Aleksandra Fedorova wrote:
> > 2) A packager rebuilds packages from Fedora dist-git in a Copr repo, what will
> > be the value of the build tag? How will the Copr build sort over the official
> > Fedora builds? (This is essentially the same question but the answer might
differ.)
> Similar as above, the questions would be how does it work now, do we
> want a change or do we want to keep the current setup?
Examples from before the Python 3.11 side tag was merged.
Fedora rawhide: python3-tomli-2.0.1-2.fc37 (built with Python 3.10)
Python 3.11 copr: python3-tomli-2.0.1-2.fc37 (built with Python 3.11)
The NEVRs are identical. Other builds in the Python 3.11 copr install the
3.11 build.
Now when we managed to not pick up a certain bump:
Fedora rawhide: python3-tomli-2.0.1-2.fc37 (built with Python 3.10)
Python 3.11 copr: python3-tomli-2.0.1-1.fc37 (built with Python 3.11)
The rawhide's NEVR is higher. Other builds in the Python 3.11 copr install
try to install the 3.10 build and that conflicts with other stuff in the
copr and the dependency resolution fails.
We certainly don't want to regress in that regard.
You can build almost anything in COPR and with random NEVRA.
Accommodating what you described above would be nice to have, but should
definitely not be a requirement. Adding build ID would not help here,
either, as they will be different between different build systems.
COPR builds have different buildhost and are signed with a different
signature. You cannot expect two packages with identical NEVRA but
different buildhost and signature to have identical dependencies and
contents.
Regards,
Dominik
--
Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
-- from "Collected Sayings of Muad'Dib" by the Princess Irulan
3:03 a.m.
New subject: [Fedora-packaging] Re: [RFC] Build tag in RPM: from NVR to NVRB
On 24. 06. 22 9:30, Dominik 'Rathann' Mierzejewski wrote:
On Thursday, 23 June 2022 at 15:01, Miro Hrončok wrote:
> On 23. 06. 22 14:24, Aleksandra Fedorova wrote:
>>> 2) A packager rebuilds packages from Fedora dist-git in a Copr repo, what
will
>>> be the value of the build tag? How will the Copr build sort over the
official
>>> Fedora builds? (This is essentially the same question but the answer might
differ.)
>> Similar as above, the questions would be how does it work now, do we
>> want a change or do we want to keep the current setup?
>
> Examples from before the Python 3.11 side tag was merged.
>
>
> Fedora rawhide: python3-tomli-2.0.1-2.fc37 (built with Python 3.10)
> Python 3.11 copr: python3-tomli-2.0.1-2.fc37 (built with Python 3.11)
>
> The NEVRs are identical. Other builds in the Python 3.11 copr install the
> 3.11 build.
>
>
> Now when we managed to not pick up a certain bump:
>
> Fedora rawhide: python3-tomli-2.0.1-2.fc37 (built with Python 3.10)
> Python 3.11 copr: python3-tomli-2.0.1-1.fc37 (built with Python 3.11)
>
> The rawhide's NEVR is higher. Other builds in the Python 3.11 copr install
> try to install the 3.10 build and that conflicts with other stuff in the
> copr and the dependency resolution fails.
>
> We certainly don't want to regress in that regard.
You can build almost anything in COPR and with random NEVRA.
Accommodating what you described above would be nice to have, but should
definitely not be a requirement. Adding build ID would not help here,
either, as they will be different between different build systems.
No, adding build ID would break this.
COPR builds have different buildhost and are signed with a different
signature. You cannot expect two packages with identical NEVRA but
different buildhost and signature to have identical dependencies and
contents.
I don't.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
8:03 a.m.
Dne 23. 06. 22 v 14:24 Aleksandra Fedorova napsal(a):
> 1) An user rebuilds a package from Fedora dist-git in local mock,
what will be
> the value of the build tag? How will the local build sort over the official
> Fedora builds?
Afaik currently, if you do a local build, you need to bump an NVR to
ensure the upgrade path is set. Then if the base distribution upgrades
or rebuilds the package, you would need to bump your version again if
you like to preserve the upgrade path.
This approach would work exactly the same way after the change, as
bumping the Release tag in the sources makes build-related versioning
irrelevant.
We can agree that by default Build tag in mock environment is unset.
Local builds have no build id, and to make clean upgrades bump of
Release tag is required.
But there are some new possibilities:
If you want to create a local rebuild without Release bump, you would
be able to pass the build tag value manually and set to whatever you
want. For example you can add 1 to existing build tag. Or you could
would be able => most people will not do that.
set it to 0. Or to 99999999.
Any final guidelines should recommend the semantics of the number. At least.
It can be monotonous sequence (1, 2, 3...) or a date used in SOA in DNS (2022062301). Or
anything else. Just document it.
The latter is less likely though.
If I were to create local builds which are always higher or always
lower than builds from the base system, I would probably play with the
dist-tag, not build id:
$ rpmdev-vercmp 0:1.2.3-12.fc37.15 0:1.2.3-12.localfc37.1
0:1.2.3-12.fc37.15 < 0:1.2.3-12.localfc37.1
There is also a possibility to set default local build tag to be the
local timestamp. But I am not sure if it is useful for a local
development
> 2) A packager rebuilds packages from Fedora dist-git in a Copr repo, what will
> be the value of the build tag? How will the Copr build sort over the official
> Fedora builds? (This is essentially the same question but the answer might differ.)
Similar as above, the questions would be how does it work now, do we
want a change or do we want to keep the current setup?
While for local build users may set the tag to some value, Copr should set it. And Copr
developers needs to know to what
value it will be.
I, personally, do not participate in this thread as I am in neutral position. At the end,
I would appreciate issue
created against Mock and Copr where will be clear guidance what and how it should be set.
Miroslav
8:10 a.m.
On 23. 06. 22 14:24, Aleksandra Fedorova wrote:
> 3) If every Fedora packager can rebuild anything without a
commit, what do we
> do prevent accidental builds?
I think each rebuild should be treated as a new package, thus it would
require a new bodhi update, testing, and signing. Which means it will
be less likely to accidentally ship it.
But as I mentioned in the post, right now we'd like to propose the
refactoring, and not a change of the development workflow. Thus I
would initially restrict the rebuild possibility to the admin group
which handles mass-rebuilds and other admin tasks. Then I would
gradually open it up case by case, each case through a separate
conversation.
Saying that, probably the first case, which I would consider is: is
there a problem of an accidental rebuild of a merged code for Fedora
Rawhide? What would be the reasons for us to_not_ allow it?
A specific example is this workflow:
Imagine 200 packages need to be rebuilt with boost 1.99.
1) I build boost 1.99 in a side tag
2) I commit a bump to 200 packages
3) I submit a side tag build for 200 packages
4) I repeat (3) until it seems futile
This solves the dependency order issues quite well. I also don't need to think
about that much and scripting it is trivial. If the build previously succeeded,
it won't do anything. If it failed, it will try again.
I realize this isn't a very clever workflow. In fact, that isn't my exact
workflow. Really, in (4) I exclude the packages that already rebuilt
successfully. But I don't need to be that careful when checking. Is my list of
finished builds slightly out of date? No problem. Are some builds still running
and I try to submit them again? No problem. Nothing happens.
With the proposed build ID think, every workflow like this would either need to
be extremely more robust, or we would end up with 20 useless concurrent
libreoffice builds very soon.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
670
days inactive
676
days old
34 comments
17 participants
participants (17)
-
Aleksandra Fedorova -
David Cantrell -
Dominik 'Rathann' Mierzejewski -
Fabio Valentini -
Hunor Csomortáni -
Kamil Dudka -
Kevin Fenzi -
Matthew Miller -
Michael J Gruber -
Miro Hrončok -
Miroslav Suchý -
Neal Gompa -
Petr Pisar -
Pierre-Yves Chibon -
Ralf Corsépius -
Vitaly Zaitsev -
Zbigniew Jędrzejewski-Szmek