Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons can be found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be happy to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out: - You can easily opt-in by using the macros - You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution: - The changelog will be automatically generated using an external `changelog` file as well as the commit history - When you opt-in, you will simply move the existing changelog to an external file `changelog`, which is placed in the dist-git repo and add the appropriate macro. - Upon building, the macro will generate the changelog using all the commits of the repo up to the last commit touching the `changelog` file. Of all these commits it will only consider the commits following these rules: - There are generally two approaches regarding what to include by default: 1. commits touching only the `sources`, `.gitignore`, `dead.package` files, `tests` folder will be ignored by default, i.e. a blacklist 2. only commits touching the spec file or patches will be included by default, i.e. a whitelist ==> Which approach do you think is better/will work in most cases? - empty commits (not touching any files) will be included on the assumption that this is their purpose - commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be - Finally the content of the changelog file specified will be appended to the changelog generated from the git history
If you wanted to edit the changelog, you would: - Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog` - Edit `changelog` as desired - Commit and push the changes
Since the macro will only consider the commits up to the first commit touching `changelog` (in other words, the macro will only consider the commits after this one) and include `changelog` file as is, your edits will appear in the RPM changelog as you want.
One thing to note is that rebuilds from the same commit will have the same %changelog, they will not get a new entry. If you want a new changelog entry, you have to create a new commit with the desired changelog entry as commit message (the commit itself can be empty).
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements: - the number of commits at this version - the number of builds at this commit - the dist-tag being added after them The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ? | B -- version: 1.0 -- release: 1.0 | C -- version: 1.0 -- release: 2.0 | D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros: - Easy to replicate locally - Every changelog entry can be mapped to a `version-release` (No changes to the packaging guidelines) - Allows triggering two builds from the same commit without any manual change (simplifies mass-rebuilds) - Easy to link a certain build with a specific commit - Easy to “guess” the next release value before triggering the build
Cons: - Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had 45 commits leading to this) - Relies on information from the build system for the build number (but can be closely simulated locally since the <n_build> info is only used for rebuilds) - Upgrade path may be problematic if Fn is upgraded to version X with one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros: - Allows triggering two builds from the same commit without manual intervention - Emulates what many maintainers do manually today for most use cases - Can cater for pre-releases (e.g.: by offering different macros or macro options for the different use cases)
Cons: - Needs the build system for information about builds in this and other Fedora versions - Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo - Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
[1]: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/ [2]: https://hackmd.io/kuXOPe74RfepuztBz7pBsg
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
If you need to rebuild for a libpingouisawesome soname bump, you just do an empty commit with the explanation.
If you merge that empty commit to a branch that did not need it, it would have a bogus changelog entry (status quo). If you care, you would not merge but cherry-pick anything thta comes next (which is now much easier given the benefit of not having the %changelog and release).
With the proposed solution that includes "successful build count" you always bump and build even if it is not needed and also you make the release number depend on a specific build system, which I think is kinda wrong.
i.e. if you do two "fedpkg build" in a row without a commit, I think that the second one should still fail with "already been build" kind of message.
On Mon, Feb 24, 2020, 18:38 Miro Hrončok mhroncok@redhat.com wrote:
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two
options
are more appealing to us:
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
If you need to rebuild for a libpingouisawesome soname bump, you just do an empty commit with the explanation.
If you merge that empty commit to a branch that did not need it, it would have a bogus changelog entry (status quo). If you care, you would not merge but cherry-pick anything thta comes next (which is now much easier given the benefit of not having the %changelog and release).
With the proposed solution that includes "successful build count" you always bump and build even if it is not needed and also you make the release number depend on a specific build system, which I think is kinda wrong.
i.e. if you do two "fedpkg build" in a row without a commit, I think that the second one should still fail with "already been build" kind of message.
What if build environment has changed?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le lundi 24 février 2020 à 18:13 +0100, Miro Hrončok a écrit :
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this?
Packages have a life in review before they get a Fedora git (sometimes the life can be years in copr, OBS or elsewhere as our review process is none too fast)
Regards,
On 24. 02. 20 19:30, Nicolas Mailhot via devel wrote:
Le lundi 24 février 2020 à 18:13 +0100, Miro Hrončok a écrit :
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this?
Packages have a life in review before they get a Fedora git (sometimes the life can be years in copr, OBS or elsewhere as our review process is none too fast)
Yet even if we count builds, the package needs to be in git first.
On Mon, Feb 24, 2020 at 07:30:15PM +0100, Nicolas Mailhot via devel wrote:
Le lundi 24 février 2020 à 18:13 +0100, Miro Hrončok a écrit :
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this?
Packages have a life in review before they get a Fedora git (sometimes the life can be years in copr, OBS or elsewhere as our review process is none too fast)
I guess this means the first version that lands in Fedora will not have a -1 release, is that a real problem? Don't we already have this anyway since often we bump the release and changelog during the review process to address the reviewer's comments?
Pierre
Dne 24. 02. 20 v 18:13 Miro Hrončok napsal(a):
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
+1
Or the combination of: Get the release field from the annotation of the git tag and Get the release field using the number of tags since the last version change If the regexp does not match, just count the tags (or the commits).
And BTW I like the changelog proposal.
On Mon, Feb 24, 2020 at 10:05:31PM +0100, Miroslav Suchý wrote:
Dne 24. 02. 20 v 18:13 Miro Hrončok napsal(a):
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
+1
Or the combination of: Get the release field from the annotation of the git tag and Get the release field using the number of tags since the last version change If the regexp does not match, just count the tags (or the commits).
We addressed git tags in the hackmd doc and we were not very thrilled with the idea. It seems fragile (parsing/regex) and if the changelog is included in the git tag, packagers still have to maintain 3 different changelogs.
And BTW I like the changelog proposal.
Thanks :) It's a bit of a tricky situation since we tend to consider both release and changelog separately but they are also very much related since the release appears in the changelog. So if we want to keep the current changelog format, we need to consider them together (of we accept to change the structure of the changelog)
Pierre
Dne 25. 02. 20 v 9:55 Pierre-Yves Chibon napsal(a):
On Mon, Feb 24, 2020 at 10:05:31PM +0100, Miroslav Suchý wrote:
Dne 24. 02. 20 v 18:13 Miro Hrončok napsal(a):
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
+1
Or the combination of: Get the release field from the annotation of the git tag and Get the release field using the number of tags since the last version change If the regexp does not match, just count the tags (or the commits).
We addressed git tags in the hackmd doc and we were not very thrilled with the idea. It seems fragile (parsing/regex) and if the changelog is included in the git tag, packagers still have to maintain 3 different changelogs.
We should keep git tags for Bodhi updates O:-)
Vít
And BTW I like the changelog proposal.
Thanks :) It's a bit of a tricky situation since we tend to consider both release and changelog separately but they are also very much related since the release appears in the changelog. So if we want to keep the current changelog format, we need to consider them together (of we accept to change the structure of the changelog)
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Feb 24, 2020 at 06:13:21PM +0100, Miro Hrončok wrote:
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
It has another con which is indeed not mentioned in this section but in the next one (n_commits+n_build) (I've fixed that). Upgrade path may be problematic if you update Fn to a version in less commit than the update for Fn-1 (ie: you update F32 to 1.0 in 1 commit and update F31 to 1.0 in 2 commits, suddenly you have F32 with 1.0-1 and F31 with 1.0-2).
If you need to rebuild for a libpingouisawesome soname bump, you just do an empty commit with the explanation.
If you merge that empty commit to a branch that did not need it, it would have a bogus changelog entry (status quo). If you care, you would not merge but cherry-pick anything thta comes next (which is now much easier given the benefit of not having the %changelog and release).
With the proposed solution that includes "successful build count" you always bump and build even if it is not needed and also you make the release number depend on a specific build system, which I think is kinda wrong.
One of the benefit of the approach using number of commits + number of builds (when it's > 1) is that mass-rebuild could be made even simpler and faster. Instead of doing git commit + koji build, you would simply do koji build.
Pierre
On 25. 02. 20 9:50, Pierre-Yves Chibon wrote:
Upgrade path may be problematic if you update Fn to a version in less commit than the update for Fn-1 (ie: you update F32 to 1.0 in 1 commit and update F31 to 1.0 in 2 commits, suddenly you have F32 with 1.0-1 and F31 with 1.0-2).
I don't consider that an issue. It's not super elegant, but since we do distro-sync on upgrades, it shuld be fine.
On Tue, Feb 25, 2020 at 05:42:11PM +0100, Miro Hrončok wrote:
On 25. 02. 20 9:50, Pierre-Yves Chibon wrote:
Upgrade path may be problematic if you update Fn to a version in less commit than the update for Fn-1 (ie: you update F32 to 1.0 in 1 commit and update F31 to 1.0 in 2 commits, suddenly you have F32 with 1.0-1 and F31 with 1.0-2).
I don't consider that an issue. It's not super elegant, but since we do distro-sync on upgrades, it shuld be fine.
Hmm, I don't do distro-sync and in general I think upgrade path is something that should be preserved.
What about doing <name>-<version>-<dist>.<commits-since-version-bump>? This means that upgrade path not affected by the number of commits or builds in the older release.
The numbers <commits-since-version-bump> in different branches cannot be meaningfully compared. Those numbers only make sense in the context of a specific branch, so they should be ordered after <dist>.
Zbyszek
On Thu, 27 Feb 2020 at 18:07, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Tue, Feb 25, 2020 at 05:42:11PM +0100, Miro Hrončok wrote:
On 25. 02. 20 9:50, Pierre-Yves Chibon wrote:
Upgrade path may be problematic if you update Fn to a version in less commit than the update for Fn-1 (ie: you update F32 to 1.0 in 1 commit and update F31 to 1.0 in 2 commits, suddenly you have F32 with 1.0-1 and F31 with 1.0-2).
I don't consider that an issue. It's not super elegant, but since we do distro-sync on upgrades, it shuld be fine.
Hmm, I don't do distro-sync and in general I think upgrade path is something that should be preserved.
What about doing <name>-<version>-<dist>.<commits-since-version-bump>?
This is a very good point really. Either it should have been always like that or we will lose something by removing that number just after the last dash.
I.e. what's the definition of Release number (the number just after the last dash). Does it have a definition? I was looking for it in packaging guidelines but couldn't find it.
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
Thank you clime
This means that upgrade path not affected by the number of commits or builds in the older release.
The numbers <commits-since-version-bump> in different branches cannot be meaningfully compared. Those numbers only make sense in the context of a specific branch, so they should be ordered after <dist>.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le samedi 29 février 2020 à 14:28 +0100, clime a écrit :
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
You need to evaluate evr with a neutral dist value to decide to bump or not the auto-dynamic part of release or not. Because the whole point of the auto-dynamic part of release would be to track rebuilds from the same spec, all othert parts of EVR being equal
Changelog-side and package build side you need the full EVR without neutralization
On Sat, 29 Feb 2020 at 14:47, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 14:28 +0100, clime a écrit :
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
You need to evaluate evr with a neutral dist value to decide to bump or not the auto-dynamic part of release or not. Because the whole point of the auto-dynamic part of release would be to track rebuilds from the same spec, all othert parts of EVR being equal
Changelog-side and package build side you need the full EVR without neutralization
Thank you very much Nicolas but you reacted to a question which was actually unrelated to your proposal. That particular question about the meaning of ENVR when you strip the distribution tag (i.e. .fc32 or .el7) was intended to be generic, i.e. i want to know how if e.g. python3-alembic-1.1.0-1 has any meaning alone without, for example, .fc31 appended to it (or if it should have any meaning which is e.g. not respected these days).
I posted a separate set of questions to understand your proposal which is very interesting by the way but the description was a bit fuzzy at some points so I needed more explanation. If you could rather react there, it would be awesome.
Thank you again clime
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le samedi 29 février 2020 à 15:12 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 14:47, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 14:28 +0100, clime a écrit :
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
You need to evaluate evr with a neutral dist value to decide to bump or not the auto-dynamic part of release or not. Because the whole point of the auto-dynamic part of release would be to track rebuilds from the same spec, all othert parts of EVR being equal
Changelog-side and package build side you need the full EVR without neutralization
Thank you very much Nicolas but you reacted to a question which was actually unrelated to your proposal. That particular question about the meaning of ENVR when you strip the distribution tag (i.e. .fc32 or .el7) was intended to be generic, i.e. i want to know how if e.g. python3-alembic-1.1.0-1 has any meaning alone without, for example, .fc31 appended to it (or if it should have any meaning which is e.g. not respected these days).
And I answered you. From a changelog POW you need the full dist because the exact same stripped/neutralized Release may (and does) exist today in different branches, pointing to different spec states.
From a dynamic release POW dist is irrelevant to compute the next bump point and needs stripping. All that matters is that the result evrs differently, ignoring dist. Keeping branches synchronized or not is a packager decision (choosing to desync requires specific upgrade path care, but it is a valid use case today).
And yes that info oriented my proposal. Any other proposal will have to take it into account too.
Regards,
On Sat, 29 Feb 2020 at 16:24, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 15:12 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 14:47, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 14:28 +0100, clime a écrit :
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
You need to evaluate evr with a neutral dist value to decide to bump or not the auto-dynamic part of release or not. Because the whole point of the auto-dynamic part of release would be to track rebuilds from the same spec, all othert parts of EVR being equal
Changelog-side and package build side you need the full EVR without neutralization
Thank you very much Nicolas but you reacted to a question which was actually unrelated to your proposal. That particular question about the meaning of ENVR when you strip the distribution tag (i.e. .fc32 or .el7) was intended to be generic, i.e. i want to know how if e.g. python3-alembic-1.1.0-1 has any meaning alone without, for example, .fc31 appended to it (or if it should have any meaning which is e.g. not respected these days).
And I answered you. From a changelog POW you need the full dist because the exact same stripped/neutralized Release may (and does) exist today in different branches, pointing to different spec states.
Can you give an example of such package?
I mean, of course, technically it is possible and not forbidden anywhere in the guidelines as far as I know. But...
If I, as a user, see e.g. python3-alembic-1.1.0-13 in f31 and then I will see python3-alembic-1.1.0-13 in f32 (this time with .fc32 disttag), I am going to assume nothing has changed for that package. I think that is the intuitive user's expectation here. By providing the same NVR (except for disttag) into the new Fedora release, as user had installed before, I can tell to the user: "Nothing has changed for you here". I mean, that's how I would interpret it.
If that stripped NVR suddenly starts depending on build count for the given branch, I will start getting quite random numbers where some package in the new Fedora release looks like "nothing has changed" (according to its stripped NVR) but in fact a lot have changed. The similar problem is with bumping by commit count too.
If we do these build-counter, commit-counter (without the leading "pivot" number) approaches, then it would imho really be better to have python3-alembic-1.1.0-fc31.13 and python3-alembic-1.1.0-fc32.13 which is quite a huge change.
From a dynamic release POW dist is irrelevant to compute the next bump point and needs stripping. All that matters is that the result evrs differently, ignoring dist. Keeping branches synchronized or not is a packager decision (choosing to desync requires specific upgrade path care, but it is a valid use case today).
And yes that info oriented my proposal. Any other proposal will have to take it into account too.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le samedi 29 février 2020 à 20:57 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 16:24, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 15:12 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 14:47, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 14:28 +0100, clime a écrit :
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
You need to evaluate evr with a neutral dist value to decide to bump or not the auto-dynamic part of release or not. Because the whole point of the auto-dynamic part of release would be to track rebuilds from the same spec, all othert parts of EVR being equal
Changelog-side and package build side you need the full EVR without neutralization
Thank you very much Nicolas but you reacted to a question which was actually unrelated to your proposal. That particular question about the meaning of ENVR when you strip the distribution tag (i.e. .fc32 or .el7) was intended to be generic, i.e. i want to know how if e.g. python3-alembic-1.1.0-1 has any meaning alone without, for example, .fc31 appended to it (or if it should have any meaning which is e.g. not respected these days).
And I answered you. From a changelog POW you need the full dist because the exact same stripped/neutralized Release may (and does) exist today in different branches, pointing to different spec states.
Can you give an example of such package?
I mean, of course, technically it is possible and not forbidden anywhere in the guidelines as far as I know. But...
That’s pretty much inevitable when you hit release-specific problems that require pushing release-specific changes or patches in one or several branches. No matter what clever numbering conventions you try to invent, things are eventually going to collide.
%{dist} is not here just to prettify things. Branching is real branching. syncing branches is a packager optimization. It’s not possible in all cases. The only hard requirement is to keep evr lower in older branches, syncing is optional and not done when it’s just a lot of pain for little win.
You can say “start from the -devel evr, add .number when adding a patch”. That only gets you as far as the first patch. What if f33 state needs patching one way in f32 and another in f31 (ignoring el7 and el8 for now)? Your convention already hit its limits. And it’s just the first patch step.
The usual packager reaction in that case is either add spaguetti dist ifdefs in their spec, or just give up on syncing (I definitely prefer the second strategy). After all, no harm done if the branches de-sync, as long as cross-release upgrade pathes work.
If I, as a user, see e.g. python3-alembic-1.1.0-13 in f31 and then I will see python3-alembic-1.1.0-13 in f32 (this time with .fc32 disttag), I am going to assume nothing has changed for that package.
And you’ll be wrong, because we do branch. I we could have avoided branching, we’d have avoided it. Branching brings quite a lot of complexity. However we do branch because releases overlap but are not technically equal.
What we *can* do is make fedpkg merge master work to avoid packager work when the branches can be synced (or re-synced). And fedpkg merge master should work with my proposal (as long as the older branch has not already autobumped further than master, but a packager that allows that already broke the upgrade path, and earned manual clean-up work).
If we do these build-counter, commit-counter (without the leading "pivot" number) approaches, then it would imho really be better to have python3-alembic-1.1.0-fc31.13 and python3-alembic-1.1.0-fc32.13 which is quite a huge change.
The system provides a %{dynrel} counter. The packager can stick it before or after %{dist} (or not use it at all), the mecanism will work the same (obviously, without autobumping if the packager does not use %{dynrel} in his release string).
%{dynrel} is sufficient to handle autobumping. If you try to own the whole release string, you’ll hit all kinds of complexity (starting with pre/post release, then the ruby thing, etc)
%{dist} (especially taking %{distprefix} into account, but even without it) does not sort well. elx/fcx does not sort. Third part packages may use a sort-able dist or not. if you want evr to work, you put the relevant part of r before dist, and keep it only as last resort to distinguish between synced fedora branches.
Regards,
On Sat, 29 Feb 2020 at 22:42, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:57 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 16:24, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 15:12 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 14:47, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 14:28 +0100, clime a écrit :
Does ENVR without %{dist} means something with respect to the content from which the package was built or with respect to features that it offers for the given distribution version?
You need to evaluate evr with a neutral dist value to decide to bump or not the auto-dynamic part of release or not. Because the whole point of the auto-dynamic part of release would be to track rebuilds from the same spec, all othert parts of EVR being equal
Changelog-side and package build side you need the full EVR without neutralization
Thank you very much Nicolas but you reacted to a question which was actually unrelated to your proposal. That particular question about the meaning of ENVR when you strip the distribution tag (i.e. .fc32 or .el7) was intended to be generic, i.e. i want to know how if e.g. python3-alembic-1.1.0-1 has any meaning alone without, for example, .fc31 appended to it (or if it should have any meaning which is e.g. not respected these days).
And I answered you. From a changelog POW you need the full dist because the exact same stripped/neutralized Release may (and does) exist today in different branches, pointing to different spec states.
Can you give an example of such package?
I mean, of course, technically it is possible and not forbidden anywhere in the guidelines as far as I know. But...
That’s pretty much inevitable when you hit release-specific problems that require pushing release-specific changes or patches in one or several branches. No matter what clever numbering conventions you try to invent, things are eventually going to collide.
%{dist} is not here just to prettify things. Branching is real branching. syncing branches is a packager optimization. It’s not possible in all cases. The only hard requirement is to keep evr lower in older branches, syncing is optional and not done when it’s just a lot of pain for little win.
You can say “start from the -devel evr, add .number when adding a patch”. That only gets you as far as the first patch. What if f33 state needs patching one way in f32 and another in f31 (ignoring el7 and el8 for now)? Your convention already hit its limits. And it’s just the first patch step.
If I wanted to be super-clean, I would simply do a minor bump for f32 and f31 on top of the f33 patch. Minor bumps do not give users any false impressions because they are placed after the dist tag.
The usual packager reaction in that case is either add spaguetti dist ifdefs in their spec, or just give up on syncing (I definitely prefer the second strategy). After all, no harm done if the branches de-sync, as long as cross-release upgrade pathes work.
If I, as a user, see e.g. python3-alembic-1.1.0-13 in f31 and then I will see python3-alembic-1.1.0-13 in f32 (this time with .fc32 disttag), I am going to assume nothing has changed for that package.
And you’ll be wrong, because we do branch. I we could have avoided branching, we’d have avoided it. Branching brings quite a lot of complexity. However we do branch because releases overlap but are not technically equal.
We still should take into account user's intuitive expectations and the way numbers are layed out in the Release string that raises those expectations.
Yes, we always work in a context of a certain distribution branch so it's very inconvenient to assign to a certain number meaning that should somehow cross boundaries of that particular branch. Maybe the way dist-git branches are layed out (per distribution version) is not totally perfect for everything though and we should be aware of that.
What we *can* do is make fedpkg merge master work to avoid packager work when the branches can be synced (or re-synced). And fedpkg merge master should work with my proposal (as long as the older branch has not already autobumped further than master, but a packager that allows that already broke the upgrade path, and earned manual clean-up work).
If we do these build-counter, commit-counter (without the leading "pivot" number) approaches, then it would imho really be better to have python3-alembic-1.1.0-fc31.13 and python3-alembic-1.1.0-fc32.13 which is quite a huge change.
The system provides a %{dynrel} counter. The packager can stick it before or after %{dist} (or not use it at all), the mecanism will work the same (obviously, without autobumping if the packager does not use %{dynrel} in his release string).
Yes, that's a cool thing.
%{dynrel} is sufficient to handle autobumping. If you try to own the whole release string, you’ll hit all kinds of complexity (starting with pre/post release, then the ruby thing, etc)
%{dist} (especially taking %{distprefix} into account, but even without it) does not sort well. elx/fcx does not sort. Third part packages may use a sort-able dist or not. if you want evr to work, you put the relevant part of r before dist, and keep it only as last resort to distinguish between synced fedora branches.
Regards,
Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Sat, Feb 29, 2020 at 10:40:51PM +0100, Nicolas Mailhot via devel wrote:
Le samedi 29 février 2020 à 20:57 +0100, clime a écrit :
If I, as a user, see e.g. python3-alembic-1.1.0-13 in f31 and then I will see python3-alembic-1.1.0-13 in f32 (this time with .fc32 disttag), I am going to assume nothing has changed for that package.
And you’ll be wrong, because we do branch.
Succinct, but true.
To give a concrete example: foo-99-1 is in current rawhide and F32 branches. In rawhide, the mass rebuild happens, but so we get foo-99-2. But this is a noarch package and foo-99-1.fc32 and foo-99-2.fc33 are effectively identical, despite having different NVr (where "r" is just the number part without the dist tag). Now a bug reported that only affects F32 or the maintainer rebuilds in F32 only. A rebuild with the fix is labelled foo-99-2.fc32. Now the packages are functionally different, despite having same NVr.
(And this is all correct packaging, according to update guidelines. The upgrade path is preserved at all times, and that is the only thing that is required.)
So yeah, I stand by my original statement that
The numbers <commits-since-version-bump> in different branches cannot be meaningfully compared. Those numbers only make sense in the context of a specific branch, so they should be ordered after <dist>.
Zbyszek
On Mon, Feb 24, 2020 at 06:13:21PM +0100, Miro Hrončok wrote:
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
This.
And a "just use the git changelog" feature as well.
In fact both should be the default.
Rich.
On Mon, 2020-02-24 at 18:13 +0100, Miro Hrončok wrote:
On 24. 02. 20 17:48, Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
Can we please have a "git is the only source of truth" version of this? I.e. "Compute the release field from the number of commits since the last version change" in the document. It seem to only have one con (breaks if two builds are triggered from the same commit) which is the status quo.
The <#commits>.<#builds> approach also doesn't address "snapshot" prereleases, i.e. which can't be dealt with having appending "~prereleasename" to the version, which is a second con (or a "pro" of the "try to emulate traditional manual release numbers" approach, if you will).
Nils
Hello!
On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons can be found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be happy to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
- You can easily opt-in by using the macros
- You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
- When you opt-in, you will simply move the existing changelog to an external file `changelog`, which is placed in the dist-git repo and add the appropriate macro.
- Upon building, the macro will generate the changelog using all the commits of the repo up to the last commit touching the `changelog` file. Of all these commits it will only consider the commits following these rules:
- There are generally two approaches regarding what to include by default:
==> Which approach do you think is better/will work in most cases?
- commits touching only the `sources`, `.gitignore`, `dead.package` files, `tests` folder will be ignored by default, i.e. a blacklist
- only commits touching the spec file or patches will be included by default, i.e. a whitelist
- empty commits (not touching any files) will be included on the assumption that this is their purpose
- commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be
- Finally the content of the changelog file specified will be appended to the changelog generated from the git history
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
Since the macro will only consider the commits up to the first commit touching `changelog` (in other words, the macro will only consider the commits after this one) and include `changelog` file as is, your edits will appear in the RPM changelog as you want.
One thing to note is that rebuilds from the same commit will have the same %changelog, they will not get a new entry. If you want a new changelog entry, you have to create a new commit with the desired changelog entry as commit message (the commit itself can be empty).
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ? | B -- version: 1.0 -- release: 1.0 | C -- version: 1.0 -- release: 2.0 | D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to the packaging guidelines)
- Allows triggering two builds from the same commit without any manual change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had 45 commits leading to this)
- Relies on information from the build system for the build number (but can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
* Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1 - manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
Ad. the following approach for calculating release: - Compute the release field from the number of commits since the last version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
To go back to your original post:
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
This approach is taken in the rpkg-util project (https://pagure.io/rpkg-util). I really recommend looking at it as I spent more than a year solving this particular problem with changelog and release (but actually not only that problem) and I also optimized the macros there as much as I possibly could.
If you want to play around with it, you can download the latest version from here:
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
and try it on this:
clime
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Nice, I have been trying to fight through the 'git context already missing' with pure lua rpm macros, and so far was hitting walls left and right :-)
Will look at https://pagure.io/rpkg-util, might have more questions :-)
Adam
On Tue, Feb 25, 2020 at 12:20 AM clime clime@fedoraproject.org wrote:
Hello!
On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This topic has already been discussed a few times over the past month,
but Adam
Saleh, Nils Philippsen and myself have had the opportunity to invest
some time
on it with the hope of making the packager's life simpler as well as
making it
easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons
can be
found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be
happy
to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM
macros would
satisfy the requirements we have for opting-in/out:
- You can easily opt-in by using the macros
- You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external
`changelog`
file as well as the commit history
- When you opt-in, you will simply move the existing changelog to an
external
file `changelog`, which is placed in the dist-git repo and add the appropriate macro.
- Upon building, the macro will generate the changelog using all the
commits
of the repo up to the last commit touching the `changelog` file. Ofall
these commits it will only consider the commits following theserules:
- There are generally two approaches regarding what to include bydefault:
1. commits touching only the `sources`, `.gitignore`,`dead.package`
files, `tests` folder will be ignored by default, i.e. ablacklist
2. only commits touching the spec file or patches will be includedby
default, i.e. a whitelist ==> Which approach do you think is better/will work in most cases? - empty commits (not touching any files) will be included on theassumption
that this is their purpose - commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be
- Finally the content of the changelog file specified will be appended
to the
changelog generated from the git historyIf you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
Since the macro will only consider the commits up to the first commit
touching
`changelog` (in other words, the macro will only consider the commits
after this
one) and include `changelog` file as is, your edits will appear in the
RPM
changelog as you want.
One thing to note is that rebuilds from the same commit will have the
same
%changelog, they will not get a new entry. If you want a new changelog
entry,
you have to create a new commit with the desired changelog entry as
commit
message (the commit itself can be empty).
However, for the release field, we are struggling a little bit more, two
options
are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ? | B -- version: 1.0 -- release: 1.0 | C -- version: 1.0 -- release: 2.0 | D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the
first one
succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No
changes to the
packaging guidelines)
- Allows triggering two builds from the same commit without any manual
change
(simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only
had 45
commits leading to this)
- Relies on information from the build system for the build number
(but can
be closely simulated locally since the <n_build> info is only usedfor
rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with
one
commit while Fn-1 is upgrade to version X with 2 commits (or more)B) The release field is automatically generated taking existing builds
in all
current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This
means for
builds of the same epoch:version, find a new release that (if at all
possible)
ensures upgradability from older Fedora versions to newer ones, adhering
to the
structure of a release tag documented in the Versioning Guidelines[1].
Going
from the (RPM-wise) "latest build" that the new one should surpass, this
can
mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released
in years
to still benefit from automatically generated releases.
The following examples would use a macro for the release field as
outlined in a
separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative
macro (or
option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in
the
spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or
macro
options for the different use cases)Cons:
- Needs the build system for information about builds in this and
other Fedora
versions
- Not easy to reproduce locally because we don’t have
machine-consumable
knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our
packaging
guidelines)So these are the results of our current investigations, we are very much
eager
to get your feedback on them and even more eager if you have ideas on
how to
approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
- Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
To go back to your original post:
For both the release and the changelog fields we believe using RPM
macros would
satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
This approach is taken in the rpkg-util project (https://pagure.io/rpkg-util). I really recommend looking at it as I spent more than a year solving this particular problem with changelog and release (but actually not only that problem) and I also optimized the macros there as much as I possibly could.
If you want to play around with it, you can download the latest version from here:
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
and try it on this:
clime
[1]:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Hello Adam!
On Tue, 25 Feb 2020 at 08:58, Adam Saleh asaleh@redhat.com wrote:
Nice, I have been trying to fight through the 'git context already missing' with pure lua rpm macros, and so far was hitting walls left and right :-)
Will look at https://pagure.io/rpkg-util, might have more questions :-)
You have probably already noticed...the docs at https://docs.pagure.org/rpkg-util/index.html are for rpkg-util version that is currently in Fedora but it contains some nice introduction, nevertheless.
The (newer) git version from the pagure repository has some features (e.g. git_release macro), which are not yet documented there.
man page of the git version ($ man rpkg) is up to date, however.
Thank you for looking at it. clime
Adam
On Tue, Feb 25, 2020 at 12:20 AM clime clime@fedoraproject.org wrote:
Hello!
On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons can be found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be happy to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
- You can easily opt-in by using the macros
- You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
- When you opt-in, you will simply move the existing changelog to an external file `changelog`, which is placed in the dist-git repo and add the appropriate macro.
- Upon building, the macro will generate the changelog using all the commits of the repo up to the last commit touching the `changelog` file. Of all these commits it will only consider the commits following these rules:
- There are generally two approaches regarding what to include by default:
==> Which approach do you think is better/will work in most cases?
- commits touching only the `sources`, `.gitignore`, `dead.package` files, `tests` folder will be ignored by default, i.e. a blacklist
- only commits touching the spec file or patches will be included by default, i.e. a whitelist
- empty commits (not touching any files) will be included on the assumption that this is their purpose
- commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be
- Finally the content of the changelog file specified will be appended to the changelog generated from the git history
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
Since the macro will only consider the commits up to the first commit touching `changelog` (in other words, the macro will only consider the commits after this one) and include `changelog` file as is, your edits will appear in the RPM changelog as you want.
One thing to note is that rebuilds from the same commit will have the same %changelog, they will not get a new entry. If you want a new changelog entry, you have to create a new commit with the desired changelog entry as commit message (the commit itself can be empty).
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ? | B -- version: 1.0 -- release: 1.0 | C -- version: 1.0 -- release: 2.0 | D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to the packaging guidelines)
- Allows triggering two builds from the same commit without any manual change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had 45 commits leading to this)
- Relies on information from the build system for the build number (but can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
- Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
To go back to your original post:
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
This approach is taken in the rpkg-util project (https://pagure.io/rpkg-util). I really recommend looking at it as I spent more than a year solving this particular problem with changelog and release (but actually not only that problem) and I also optimized the macros there as much as I possibly could.
If you want to play around with it, you can download the latest version from here:
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
and try it on this:
clime
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, 25 Feb 2020 at 09:31, clime clime@fedoraproject.org wrote:
Hello Adam!
On Tue, 25 Feb 2020 at 08:58, Adam Saleh asaleh@redhat.com wrote:
Nice, I have been trying to fight through the 'git context already missing' with pure lua rpm macros, and so far was hitting walls left and right :-)
Will look at https://pagure.io/rpkg-util, might have more questions :-)
You have probably already noticed...the docs at https://docs.pagure.org/rpkg-util/index.html are for rpkg-util version that is currently in Fedora but it contains some nice introduction, nevertheless.
The (newer) git version from the pagure repository has some features (e.g. git_release macro), which are not yet documented there.
man page of the git version ($ man rpkg) is up to date, however.
Thank you for looking at it. clime
...in case of any questions i am on this email or on #fedora-apps. Again, thanks for looking at it. Cheers.
Adam
On Tue, Feb 25, 2020 at 12:20 AM clime clime@fedoraproject.org wrote:
Hello!
On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons can be found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be happy to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
- You can easily opt-in by using the macros
- You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
- When you opt-in, you will simply move the existing changelog to an external file `changelog`, which is placed in the dist-git repo and add the appropriate macro.
- Upon building, the macro will generate the changelog using all the commits of the repo up to the last commit touching the `changelog` file. Of all these commits it will only consider the commits following these rules:
- There are generally two approaches regarding what to include by default:
==> Which approach do you think is better/will work in most cases?
- commits touching only the `sources`, `.gitignore`, `dead.package` files, `tests` folder will be ignored by default, i.e. a blacklist
- only commits touching the spec file or patches will be included by default, i.e. a whitelist
- empty commits (not touching any files) will be included on the assumption that this is their purpose
- commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be
- Finally the content of the changelog file specified will be appended to the changelog generated from the git history
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
Since the macro will only consider the commits up to the first commit touching `changelog` (in other words, the macro will only consider the commits after this one) and include `changelog` file as is, your edits will appear in the RPM changelog as you want.
One thing to note is that rebuilds from the same commit will have the same %changelog, they will not get a new entry. If you want a new changelog entry, you have to create a new commit with the desired changelog entry as commit message (the commit itself can be empty).
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ? | B -- version: 1.0 -- release: 1.0 | C -- version: 1.0 -- release: 2.0 | D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to the packaging guidelines)
- Allows triggering two builds from the same commit without any manual change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had 45 commits leading to this)
- Relies on information from the build system for the build number (but can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
- Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
To go back to your original post:
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
This approach is taken in the rpkg-util project (https://pagure.io/rpkg-util). I really recommend looking at it as I spent more than a year solving this particular problem with changelog and release (but actually not only that problem) and I also optimized the macros there as much as I possibly could.
If you want to play around with it, you can download the latest version from here:
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
and try it on this:
clime
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Feb 25, 2020 at 12:18:24AM +0100, clime wrote:
Hello!
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
It makes it easier to do rebuilds which means it makes it easier and simpler to do mass-rebuilds.
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
- Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Interesting idea.
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
We very much agree that removing git tags is not a good idea, that's why we listed it as a con :)
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
If you make the build system provide the ${dirty_appendix} and drop the ${pivot} (because we want to generate the release, so there is no input specified), you get very close to what we described.
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
This is basically what I had in mind.
To go back to your original post:
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
We had a discussion with Neal about this topic on #fedora-apps yesterday and I believe one of the outcome of this discussion was that we should indeed use this approach, for the reasons you're mentioning.
Thanks for your feedbacks and thoughts (and links!)
Pierre
Le 2020-02-25 10:24, Pierre-Yves Chibon a écrit :
If you make the build system provide the ${dirty_appendix} and drop the ${pivot} (because we want to generate the release, so there is no input specified), you get very close to what we described.
BTW, regardless of how things up, we have existing logic that computes dist prefixes (unpstream git commit numbers…) and appendixes (bootstrap…).
The easy way to do things is to let things as they stand today and limit the automation to the main release number, but that still requires adding those things to changelog entries (if we want to trace them in changelogs; we probably want because a project that does not releases will be stuck at version 0 forever, only the distprefix disambiguates what version 0 means)
OTOH, if people feel ambitious, and want to reexamine all the dist prefix/appendix logic, there is quite a lot of work to audit the hundreds of packages that use them today and define a migration plan and logic. Because this amount of spec logic is not going to rewrite itself, and expecting someone else (packagers) to do it will throw a non trivial part of the distro under the bus.
(The correct way to do things would be to evolve rpm to have dedicated fields for all the things we crammed into Release in the past. That’s a non trivial endeavour.)
Regards,
Hey pingou!
On Tue, 25 Feb 2020 at 10:26, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Tue, Feb 25, 2020 at 12:18:24AM +0100, clime wrote:
Hello!
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
It makes it easier to do rebuilds which means it makes it easier and simpler to do mass-rebuilds.
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
- Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Interesting idea.
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
We very much agree that removing git tags is not a good idea, that's why we listed it as a con :)
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
If you make the build system provide the ${dirty_appendix} and drop the ${pivot} (because we want to generate the release, so there is no input specified), you get very close to what we described.
${dirty_appendix} is a part intended for development purposes, i.e. to be able to have versioned working tree when working locally without committing. It will be empty if working tree is clean, which should always be the case in a build system (there is more info here https://pagure.io/rpkg-util/blob/master/f/macros/macros.d/git.bash#_9 on this).
I am guessing you think about number-of-builds as the ${dirty_appendix} ? That would need to be a normal rpm macro provided by a build system (similarly to %dist). Anyway, if the number-of-builds part is just to optimize mass rebuild process, I would try to look if we can tweak the build system to be able to do the mass rebuilds without making new commits back to dist-git and even without that %number-of-builds macro. In my opinion, it should be possible. There is already BUILDTIME in rpm headers that can be used to differentiate between newer and older rpm with otherwise the same nevr.
In rpkg-util, ${pivot} (the main release number) is generated automatically (when the git_release macro is employed) but packager can also hard-set it at any time to any value. I think with the pivot number, the upgrade-path problem would be solved cause the commit-count appendix is only used as delta since the last pivot change, which is marked by an annotated tag in the git tree. So the commit (having an annotated tag on it) in older and newer Fedora release (branch) would generate an rpm with the same NVR and just the %dist decides then (there can also be two separate tags if %dist is used in the Release string and if it is configured not to be empty as i do it now rpkg-util :) but this is another discussion).
The pivot is also useful for post/pre-releases and other cases described here: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_more_...
It is then useful to have something that can be configured by hand.
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
This is basically what I had in mind.
Yes, it takes slightly more computational time than being navigated by tags themselves that can be used for version bumps as well as release bumps.
To go back to your original post:
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
We had a discussion with Neal about this topic on #fedora-apps yesterday and I believe one of the outcome of this discussion was that we should indeed use this approach, for the reasons you're mentioning.
Thanks for your feedbacks and thoughts (and links!)
Right, I will be very happy if I can collaborate on this. clime
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, 25 Feb 2020 at 12:47, clime clime@fedoraproject.org wrote:
Hey pingou!
On Tue, 25 Feb 2020 at 10:26, Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Tue, Feb 25, 2020 at 12:18:24AM +0100, clime wrote:
Hello!
What is the point of including number of builds into release? I think the Miro's approach solves it. Or is there any other problem except soname bumps?
It makes it easier to do rebuilds which means it makes it easier and simpler to do mass-rebuilds.
Ad. document - annotated git tags: (-) Editing the changelog would mean allowing to remove the git tags, thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by introducing arguments since_tag and until_tag for git_changelog macro (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). That's how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
- Mon Feb 24 2020 clime clime@fedoraproject.org 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Interesting idea.
Removing already pushed git tags is probably not a good idea anyway: https://git-scm.com/docs/git-tag#_on_re_tagging
We very much agree that removing git tags is not a good idea, that's why we listed it as a con :)
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except the release part has more subparts than just two (including %{dist}).
If you make the build system provide the ${dirty_appendix} and drop the ${pivot} (because we want to generate the release, so there is no input specified), you get very close to what we described.
${dirty_appendix} is a part intended for development purposes, i.e. to be able to have versioned working tree when working locally without committing. It will be empty if working tree is clean, which should always be the case in a build system (there is more info here https://pagure.io/rpkg-util/blob/master/f/macros/macros.d/git.bash#_9 on this).
I am guessing you think about number-of-builds as the ${dirty_appendix} ? That would need to be a normal rpm macro provided by a build system (similarly to %dist). Anyway, if the number-of-builds part is just to optimize mass rebuild process, I would try to look if we can tweak the build system to be able to do the mass rebuilds without making new commits back to dist-git and even without that %number-of-builds macro. In my opinion, it should be possible. There is already BUILDTIME in rpm headers that can be used to differentiate between newer and older rpm with otherwise the same nevr.
In rpkg-util, ${pivot} (the main release number) is generated automatically (when the git_release macro is employed) but packager can also hard-set it at any time to any value. I think with the pivot number, the upgrade-path problem would be solved cause the commit-count appendix is only used as delta since the last pivot change, which is marked by an annotated tag in the git tree. So the commit (having an annotated tag on it) in older and newer Fedora release (branch) would generate an rpm with the same NVR and just the %dist decides then (there can also be two separate tags if %dist is used in the Release string and if it is configured not to be empty as i do it now rpkg-util :) but this is another discussion).
The pivot is also useful for post/pre-releases and other cases described here: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_more_...
It is then useful to have something that can be configured by hand.
The full description is here: https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 but the main difference is that it counts commits from the latest annotated tag which contains (in its name) the current version and the current release number which are extracted from the spec file when creating the tag unless they are specified manually on command line. Commit count is only appended to it if we build from commit which is on top of some annotated tag (i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky. You would need to go through all the past commits that touched that spec file, keep checking these out and look if the version is changed when compared to the one parsed from the head commit. Possible though.
This is basically what I had in mind.
Yes, it takes slightly more computational time than being navigated by tags themselves that can be used for version bumps as well as release bumps.
To go back to your original post:
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms because it will be only possible to build them when the git context is present but not when they become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mock...
That's why I think that such macros should be of a different kind: macros that are computed once when srpm is created and result of which is put _verbatim_ into the spec file so that there is no (re)computation later when srpm is being built and when the git context is already missing.
We had a discussion with Neal about this topic on #fedora-apps yesterday and I believe one of the outcome of this discussion was that we should indeed use this approach, for the reasons you're mentioning.
Thanks for your feedbacks and thoughts (and links!)
Right, I will be very happy if I can collaborate on this. clime
Sry for the weird line breaks, i need to tweak my mail "client" not to do that.
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
- commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be
Could we please use the usual git commit keyword syntax? I.e. the e-mail header format (Signed-off-by:). E.g.:
RPM-Changelog: include RPM-Changelog: exclude
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
If the changelog file name is hard-coded, I recommend "fedpkg generate-changelog" command to write directly into the "changelog" file.
-- Petr
On Tue, Feb 25, 2020 at 09:06:35AM +0100, Petr Pisar wrote:
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
- commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may beCould we please use the usual git commit keyword syntax? I.e. the e-mail header format (Signed-off-by:). E.g.:
RPM-Changelog: include RPM-Changelog: exclude
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
If the changelog file name is hard-coded, I recommend "fedpkg generate-changelog" command to write directly into the "changelog" file.
Two good ideas, thanks :)
Pierre
Dne 24. 02. 20 v 17:48 Pierre-Yves Chibon napsal(a):
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons can be found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be happy to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
- You can easily opt-in by using the macros
- You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
- When you opt-in, you will simply move the existing changelog to an external file `changelog`, which is placed in the dist-git repo and add the appropriate macro.
I assume you are aware about my PR:
https://pagure.io/packaging-committee/pull-request/942
- Upon building, the macro will generate the changelog using all the commits of the repo up to the last commit touching the `changelog` file.
I proposed, that the changelog file is either included in the repository or it is not included (probably listed in .gitignore). So it is either used as it is or if later, then it is generated.
You propose mixture if I understand it correctly. I.e. the changelog file is always present in the dist-git and it is always is used. But if there are more recent commits without changelog modifications, they are prepended to the changelog, but the changelog file itself stays untouched. E.g. if my latest commit modifies the changelog, the changelog as it is present in the repo will be used without any modifications.
I like it. It is opt-in, because I have to place some macro into .spec file and it is even more or less bacward compatible, because the automation kicks in only if I don't maintain the changelog manually.
Of all these commits it will only consider the commits following these rules: - There are generally two approaches regarding what to include by default: 1. commits touching only the `sources`, `.gitignore`, `dead.package` files, `tests` folder will be ignored by default, i.e. a blacklist 2. only commits touching the spec file or patches will be included by default, i.e. a whitelist ==> Which approach do you think is better/will work in most cases? - empty commits (not touching any files) will be included on the assumption that this is their purpose - commits containing "magic keyword" (#changelog_exclude, #changelog_include?) will be ignored or included as the case may be
- Finally the content of the changelog file specified will be appended to the changelog generated from the git history
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
This also helps to completely modify the changelog and possible fix some past errors in git changelog or what not.
- Edit `changelog` as desired
- Commit and push the changes
Since the macro will only consider the commits up to the first commit touching `changelog` (in other words, the macro will only consider the commits after this one) and include `changelog` file as is, your edits will appear in the RPM changelog as you want.
One thing to note is that rebuilds from the same commit will have the same %changelog, they will not get a new entry. If you want a new changelog entry, you have to create a new commit with the desired changelog entry as commit message (the commit itself can be empty).
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ? | B -- version: 1.0 -- release: 1.0 | C -- version: 1.0 -- release: 2.0 | D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to the packaging guidelines)
- Allows triggering two builds from the same commit without any manual change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had 45 commits leading to this)
- Relies on information from the build system for the build number (but can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
I am not really sure about this. How do you envision this is going to be implemented? Is there going to be "release" file, similarly to changelog, which would help me to override this? Because, currently, it seems that both methods are going to need a lot of information from the build system. They will need to parse .spec file etc. I don't see this to be able to handle even a bit more complex stuff like e.g. https://src.fedoraproject.org/rpms/ruby/blob/master/f/ruby.spec#_26
Vít
Looking forward for the discussion,
Pierre For Adam, Nils and myself
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Feb 25, 2020 at 12:59:39PM +0100, Vít Ondruch wrote:
Dne 24. 02. 20 v 17:48 Pierre-Yves Chibon napsal(a):
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
We have investigated a few ideas, the full list with their pros and cons can be found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev If you have any questions about some of these, please ask them, we'll be happy to answer them and potentially complete this document.
For both the release and the changelog fields we believe using RPM macros would satisfy the requirements we have for opting-in/out:
- You can easily opt-in by using the macros
- You can easily opt-out by removing the macros from your spec file
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
- When you opt-in, you will simply move the existing changelog to an external file `changelog`, which is placed in the dist-git repo and add the appropriate macro.
I assume you are aware about my PR:
https://pagure.io/packaging-committee/pull-request/942
- Upon building, the macro will generate the changelog using all the commits of the repo up to the last commit touching the `changelog` file.
I proposed, that the changelog file is either included in the repository or it is not included (probably listed in .gitignore). So it is either used as it is or if later, then it is generated.
You propose mixture if I understand it correctly. I.e. the changelog file is always present in the dist-git and it is always is used. But if there are more recent commits without changelog modifications, they are prepended to the changelog, but the changelog file itself stays untouched. E.g. if my latest commit modifies the changelog, the changelog as it is present in the repo will be used without any modifications.
I like it. It is opt-in, because I have to place some macro into .spec file and it is even more or less bacward compatible, because the automation kicks in only if I don't maintain the changelog manually.
That is correct. I was aware of the PR to the FPC which actually contributed to this idea (figuring out the last commit that changed the `changelog` file is way easier than the last commit that changed %changelog in the spec file). So thanks for this!
[About the release field]
I am not really sure about this. How do you envision this is going to be implemented? Is there going to be "release" file, similarly to changelog, which would help me to override this? Because, currently, it seems that both methods are going to need a lot of information from the build system.
The first method using number of commits and number of builds, pull limited information from the build system (just: how many builds succeed before this one with this version and release?). The second method does rely heavily on the build system.
They will need to parse .spec file etc. I don't see this to be able to handle even a bit more complex stuff like e.g. https://src.fedoraproject.org/rpms/ruby/blob/master/f/ruby.spec#_26
We've been trying to find something that fit the majority of the packages but we are well aware that there are and will always be some special snowflakes that will not be able to adhere to this. I'd be happy if this works makes life/packaging easier for 50% of our packages (I'd even probably be happy if it's 20%). Once we have the simple case out and we can test it, see how it behaves, what its limits are, we can start building on more complex cases, but I don't really believe on building the perfect solution in one go.
Pierre
Dne 25. 02. 20 v 15:03 Pierre-Yves Chibon napsal(a):
On Tue, Feb 25, 2020 at 12:59:39PM +0100, Vít Ondruch wrote:
[About the release field]
I am not really sure about this. How do you envision this is going to be implemented? Is there going to be "release" file, similarly to changelog, which would help me to override this?
I just wanted to stress out that I believe it would be good idea to use similar approach for changelog as well as for release. I.e. both should be opt-in, stored externally, overridable, and the automation should kick in only if they were not touched by the last commit.
Vít
Because, currently, it seems that both methods are going to need a lot of information from the build system.
The first method using number of commits and number of builds, pull limited information from the build system (just: how many builds succeed before this one with this version and release?). The second method does rely heavily on the build system.
They will need to parse .spec file etc. I don't see this to be able to handle even a bit more complex stuff like e.g. https://src.fedoraproject.org/rpms/ruby/blob/master/f/ruby.spec#_26
We've been trying to find something that fit the majority of the packages but we are well aware that there are and will always be some special snowflakes that will not be able to adhere to this. I'd be happy if this works makes life/packaging easier for 50% of our packages (I'd even probably be happy if it's 20%). Once we have the simple case out and we can test it, see how it behaves, what its limits are, we can start building on more complex cases, but I don't really believe on building the perfect solution in one go.
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 24/02/2020 à 17:48, Pierre-Yves Chibon a écrit :
- You can easily opt-in by using the macros
Please keep opt-in as a mandatory need for such a change.
To be clear, I will be (perhaps the only) one to not use it.
For now spec file are self-contained, which is nice.
I don't like the idea of generated / external stuff related to "storage" or "build system"
Sorry, to be again the old bad guy which don't like changes.
Remi
On Tue, Feb 25, 2020 at 2:47 PM Remi Collet Fedora@famillecollet.com wrote:
Le 24/02/2020 à 17:48, Pierre-Yves Chibon a écrit :
- You can easily opt-in by using the macros
Please keep opt-in as a mandatory need for such a change.
To be clear, I will be (perhaps the only) one to not use it.
For now spec file are self-contained, which is nice.
I don't like the idea of generated / external stuff related to "storage" or "build system"
Sorry, to be again the old bad guy which don't like changes.
Remi
FWIW, I agree. Maybe I'm getting old as well >:-D
I don't think it's a good idea to use any information from outside the dist-git repository as a source of truth for anything. The big benefit of only using the git repository as source of information is that it is immutable, reproducible, and cannot be changed after commits have been pushed. The git repository data is also available for working on packages *offline*, in contrast to having to ask koji for the number of builds since X ...
Fabio
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Feb 25, 2020 at 02:55:37PM +0100, Fabio Valentini wrote:
On Tue, Feb 25, 2020 at 2:47 PM Remi Collet Fedora@famillecollet.com wrote:
Le 24/02/2020 à 17:48, Pierre-Yves Chibon a écrit :
- You can easily opt-in by using the macros
Please keep opt-in as a mandatory need for such a change.
To be clear, I will be (perhaps the only) one to not use it.
For now spec file are self-contained, which is nice.
I don't like the idea of generated / external stuff related to "storage" or "build system"
Sorry, to be again the old bad guy which don't like changes.
Remi
FWIW, I agree. Maybe I'm getting old as well >:-D
I don't think it's a good idea to use any information from outside the dist-git repository as a source of truth for anything. The big benefit of only using the git repository as source of information is that it is immutable, reproducible, and cannot be changed after commits have been pushed. The git repository data is also available for working on packages *offline*, in contrast to having to ask koji for the number of builds since X ...
The way I see it is this: With the number of commits+number of build idea, you get the same results locally and in bodhi. Locally fedpkg build or rpmbuild -ba will override the existing RPM In koji, it will simply append a .1 to the release to avoid overriding the existing RPM. But the content and release, except for two characters, will be the same.
That being said, there seems to be a consensus forming about wanting to rely only on number of commits (though, we still have the upgrade path issue to sort out).
Pierre
On Tue, Feb 25, 2020 at 3:12 PM Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Tue, Feb 25, 2020 at 02:55:37PM +0100, Fabio Valentini wrote:
On Tue, Feb 25, 2020 at 2:47 PM Remi Collet Fedora@famillecollet.com wrote:
Le 24/02/2020 à 17:48, Pierre-Yves Chibon a écrit :
- You can easily opt-in by using the macros
Please keep opt-in as a mandatory need for such a change.
To be clear, I will be (perhaps the only) one to not use it.
For now spec file are self-contained, which is nice.
I don't like the idea of generated / external stuff related to "storage" or "build system"
Sorry, to be again the old bad guy which don't like changes.
Remi
FWIW, I agree. Maybe I'm getting old as well >:-D
I don't think it's a good idea to use any information from outside the dist-git repository as a source of truth for anything. The big benefit of only using the git repository as source of information is that it is immutable, reproducible, and cannot be changed after commits have been pushed. The git repository data is also available for working on packages *offline*, in contrast to having to ask koji for the number of builds since X ...
The way I see it is this: With the number of commits+number of build idea, you get the same results locally and in bodhi. Locally fedpkg build or rpmbuild -ba will override the existing RPM In koji, it will simply append a .1 to the release to avoid overriding the existing RPM. But the content and release, except for two characters, will be the same.
(snip)
That being said, there seems to be a consensus forming about wanting to rely only on number of commits (though, we still have the upgrade path issue to sort out).
Hi Pierre,
After reporting a few upgrade path bugs for (I think) fedora 28 and 29, I was told that "we don't care about upgrade path anymore", since "dnf system-upgrade" operates in "distro-sync" mode by default, since a few releases ago.
So I don't see upgrade path as a (big) concern here. There may be package downgrades at system-upgrade time, but that's already the case today - most of the time because either people forget to build for fedora-branched after the branch point, or because they forget to submit bodhi updates after update-testing activation point. Whereas those two are "real" downgrades, any downgrades caused by the new commit counting would only be "downgrade by number but upgrade in content".
Fabio
Side note: I've been meaning to propose dropping Epoch because of this "we don't care about upgrade path anymore", but I've not gotten around to do that yet 😈️
Pierre _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 25. 02. 20 15:27, Fabio Valentini wrote:
Side note: I've been meaning to propose dropping Epoch because of this "we don't care about upgrade path anymore", but I've not gotten around to do that yet
Unfortunately, that breaks rawhide users.
On Tue, 2020-02-25 at 17:45 +0100, Miro Hrončok wrote:
On 25. 02. 20 15:27, Fabio Valentini wrote:
Side note: I've been meaning to propose dropping Epoch because of this "we don't care about upgrade path anymore", but I've not gotten around to do that yet
We still need Epoch to have the ability to back-pedal on a (hypothetical) horribly broken version update in a stable release which wasn't caught during testing.
Unfortunately, that breaks rawhide users.
Well, if we officially were to break with the upgrade path constraints, we'd have to document this. While we're at it, we should then document that Rawhide users should use "dnf distro-sync".
Nils
Le 2020-02-26 11:14, Nils Philippsen a écrit :
Well, if we officially were to break with the upgrade path constraints, we'd have to document this. While we're at it, we should then document that Rawhide users should use "dnf distro-sync".
That won’t work because (for example) rawhide is pulling forward *right* *now*, while F32 is not finished, so you have a *huge* time windows during which there is nothing to distro-sync with, and you still need to keep compat between rawhide and future-stable numbering.
If you don’t keep compat you can not solve post-beta-freeze issues by pulling in future-stable the rawhide packages that solve those issues. And there will be some of those. Otherwise beta would be the same thing as definite release.
Reagrds,
On Wed, 2020-02-26 at 11:29 +0100, Nicolas Mailhot via devel wrote:
Le 2020-02-26 11:14, Nils Philippsen a écrit :
Well, if we officially were to break with the upgrade path constraints, we'd have to document this. While we're at it, we should then document that Rawhide users should use "dnf distro-sync".
That won’t work because (for example) rawhide is pulling forward *right* *now*, while F32 is not finished, so you have a *huge* time windows during which there is nothing to distro-sync with, and you still need to keep compat between rawhide and future-stable numbering.
Heh, I wasn't volunteering to push this. If you ask me, we should stick with having a clean upgrade path.
Nils
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
This all sounds great. I'd also love for there to be a standard way of tagging specific git commit log messages as meant for user consumption, and use that to prepopulate the bodhi release notes field (with an eventual eye towards making this the single source of Fedora package change information).
On Tue, Feb 25, 2020, at 2:53 PM, Matthew Miller wrote:
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
This all sounds great. I'd also love for there to be a standard way of tagging specific git commit log messages as meant for user consumption, and use that to prepopulate the bodhi release notes field (with an eventual eye towards making this the single source of Fedora package change information).
Indeed, it is unfortunate that the Bodhi info for EOL releases is currently lost.
V/r, James Cassell
Few more notes: - having just: Release: <commit-count> means every commit bumps release and hence every commit should likely generate a new record into changelog => changelog basically becomes dumped `git log` (in rpm-compatible format) - i.e. there is no capability to group multiple changes into one changelog record (grouping them implicitly by version changes will be tricky because changelogs will become mutable) - it doesn't help in jumping from specific package name to a specific commit in dist-git unless the macro also generates short git hash into the release but it also doesn't make things worse compared to now - for the solution with external changelog file with the commit fallback - it might be slightly confusing for someone who wants to look through the changes for a package in dist-git - at first, one should look at commits but from a certain commit switch to the file => probably everyone will just stick to reading the commits
I very much like simplicity of the approach but i think it is good to realize that the proposed solution is a direct translation of: "RPM Changelog and Release, you are not needed. Stand aside and don't make any problems!" If this is what packagers usually want, then it is a good solution. I think it also depends on what rpm consumers generally want.
Then from the document: Get the release field from the annotation of the git tag (-) requires manual work on behalf of the maintainer
That doesn't need to be the case with a client-side tooling that generates release automatically into a tag name.
(-) Fragile: this means parsing using a regex the git annotation to extract the release information
I would suggest putting release directly into a tag name, not into its annotation (i.e. subject or body). Basically tag names = NVRs.
There is nothing fragile on parsing release from such name. In python, it is just standard rsplit('-', 1) and taking the last component.
There are two issues here: 1) epochs: they will need to be put into the tag-name as <epoch>/NVR because tag names cannot contain colons 2) tilde for prereleases: git tags cannot contain tilde character ('~'). But i personally believe we could find better ways to mark a prerelease. It used to be forbidden in Fedora...
clime
On Tue, 25 Feb 2020 at 22:20, James Cassell fedoraproject@cyberpear.com wrote:
On Tue, Feb 25, 2020, at 2:53 PM, Matthew Miller wrote:
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
This all sounds great. I'd also love for there to be a standard way of tagging specific git commit log messages as meant for user consumption, and use that to prepopulate the bodhi release notes field (with an eventual eye towards making this the single source of Fedora package change information).
Indeed, it is unfortunate that the Bodhi info for EOL releases is currently lost.
V/r, James Cassell _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Dne 26. 02. 20 v 12:16 clime napsal(a):
Few more notes:
- having just: Release: <commit-count> means every commit bumps
release and hence every commit should likely generate a new record into changelog => changelog basically becomes dumped `git log` (in rpm-compatible format) - i.e. there is no capability to group multiple changes into one changelog record (grouping them implicitly by version changes will be tricky because changelogs will become mutable)
If you care, you'll be able to manually modify the changelog. But I agree that some automatic grouping would be nice. May be there could also be grouping hint, similarly to `RPM-Changelog: exclude`.
Vít
- it doesn't help in jumping from specific package name to a specific
commit in dist-git unless the macro also generates short git hash into the release but it also doesn't make things worse compared to now
- for the solution with external changelog file with the commit
fallback - it might be slightly confusing for someone who wants to look through the changes for a package in dist-git - at first, one should look at commits but from a certain commit switch to the file => probably everyone will just stick to reading the commits
I very much like simplicity of the approach but i think it is good to realize that the proposed solution is a direct translation of: "RPM Changelog and Release, you are not needed. Stand aside and don't make any problems!" If this is what packagers usually want, then it is a good solution. I think it also depends on what rpm consumers generally want.
Then from the document: Get the release field from the annotation of the git tag (-) requires manual work on behalf of the maintainer
That doesn't need to be the case with a client-side tooling that generates release automatically into a tag name.
(-) Fragile: this means parsing using a regex the git annotation to extract the release information
I would suggest putting release directly into a tag name, not into its annotation (i.e. subject or body). Basically tag names = NVRs.
There is nothing fragile on parsing release from such name. In python, it is just standard rsplit('-', 1) and taking the last component.
There are two issues here:
- epochs: they will need to be put into the tag-name as <epoch>/NVR
because tag names cannot contain colons 2) tilde for prereleases: git tags cannot contain tilde character ('~'). But i personally believe we could find better ways to mark a prerelease. It used to be forbidden in Fedora...
clime
On Tue, 25 Feb 2020 at 22:20, James Cassell fedoraproject@cyberpear.com wrote:
On Tue, Feb 25, 2020, at 2:53 PM, Matthew Miller wrote:
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
This all sounds great. I'd also love for there to be a standard way of tagging specific git commit log messages as meant for user consumption, and use that to prepopulate the bodhi release notes field (with an eventual eye towards making this the single source of Fedora package change information).
Indeed, it is unfortunate that the Bodhi info for EOL releases is currently lost.
V/r, James Cassell _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, 24 Feb 2020 at 11:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
Going through the various discussions, I think we are running into a classic sausage factory problem.. everyone wants to have their favourite meat, but the only way to make it work is mix them all together, grind it up and come up with something else.
Would it make sense to classify packages into 3 different groupings (like cheese)?
Processed cheese: packages which are automatically built with a processed tool that updates most of the items. Just gets a bit of work every now and then from a packager afterwords for major changes or side-cases found. Everything in the package is fully automated with build tools filling in all the needed data for it.
Cheddar cheese: packages which need more work and care, but are still pretty 'standard'... some may be sharper and need some aging but might be ok with some automated parts (%changelog and %release and a couple of things).
Beaufort D'Ete (etc etc): packages which need a lot of work on the side of the maintainer to make it work for multiple reasons. Each packager has their own way of dealing with these packages which makes each a regional cheese.
Currently a lot of packagers have to spend their time making cheeze-wiz packages so that they can even get to working on their specialist cheese. But because they are focused on wanting to get that specialist package out they are worried that any change will make all packages cheeze-wiz.
I am wondering if we just make that part of the system separate with people knowing that they can label a package into a specific mental bucket, they can focus their energies on their Comté vieux.
Hi list,
Stephen John Smoogen smooge@gmail.com writes:
On Mon, 24 Feb 2020 at 11:50, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This topic has already been discussed a few times over the past month, but Adam Saleh, Nils Philippsen and myself have had the opportunity to invest some time on it with the hope of making the packager's life simpler as well as making it easier to build automation around our package maintenance.
Going through the various discussions, I think we are running into a classic sausage factory problem.. everyone wants to have their favourite meat, but the only way to make it work is mix them all together, grind it up and come up with something else.
I have to agree with Stephen here. As someone who has been heavily involved in building packages on OBS, where the Release field is generated automatically by the build server: I don't see a problem in automatically generating the release field, I would actually very much welcome it. But then none of my packages fall into the "Beaufort D'Ete" category, so I might be missing something here.
For the changelog: yes please, generate it from the commit log! They are more or less the same for all my packages and I'm getting tired of copy pasting the same text into %changelog and git commit.
Cheers,
Dan
On Thu, Feb 27, 2020 at 08:08:49AM +0100, Dan Čermák wrote:
For the changelog: yes please, generate it from the commit log! They are more or less the same for all my packages and I'm getting tired of copy pasting the same text into %changelog and git commit.
Another idea would be generating a changelog-entry from git history when creating an update in bodhi, and there is no pre-existing changelog-entry for the current version.
This would not break the option to build a package from just one file. Having it all in one file is a big bonus to fedora, you can just download that file and build your package and not worry about the whole git-workflow, or having to check if you downloaded all files (not completely true in case of patch files).
It also would remove the need to copy messages from git log to the changelog. (some people complained about that - not only this message)
I'm also not really a fan of "git as single source of truth" (has been mentioned a few times in this thread) - for me git is just a tool ensuring that git history was not modified.
The *actual* source of truth is still the .spec file in the commit that is used to build the package - nobody is ever looking at old commits except for checking for malicious changes. (at least for spec files, with code it is useful for bisecting bugs).
For end-users it might be useful to get the changelog alone (for that it does not matter if it is generated or copied from the .spec), but I never had any use for the changelog without the .spec file, as this gives me the context to the changes in the .spec file.
But after all I do not care too much about how changelog is created, as long as the previous functionality is still preserved - my git log messages contain information about the .spec file changes while the changelog contains changes about the functionality of the package. ("what have I changed" (git) vs. "what has changed for you" (cl))
All the best, Astra
On Thu, 27 Feb 2020 at 10:00, David Kaufmann astra@ionic.at wrote:
On Thu, Feb 27, 2020 at 08:08:49AM +0100, Dan Čermák wrote:
For the changelog: yes please, generate it from the commit log! They are more or less the same for all my packages and I'm getting tired of copy pasting the same text into %changelog and git commit.
Another idea would be generating a changelog-entry from git history when creating an update in bodhi, and there is no pre-existing changelog-entry for the current version.
This would not break the option to build a package from just one file. Having it all in one file is a big bonus to fedora, you can just download that file and build your package and not worry about the whole git-workflow, or having to check if you downloaded all files (not completely true in case of patch files).
Yes, I also thing having all relevant information in a single file is a nice feature. At the same time, usually you need to download/clone the whole repository to build the package.
It also would remove the need to copy messages from git log to the changelog. (some people complained about that - not only this message)
But Bodhi changelogs is not what user can read on his/her machine when examining e.g. dnf check-update --changelogs. These are imho rpm changelogs. So the rpm spec changelogs are the most important.
I'm also not really a fan of "git as single source of truth" (has been mentioned a few times in this thread) - for me git is just a tool ensuring that git history was not modified.
The *actual* source of truth is still the .spec file in the commit that is used to build the package - nobody is ever looking at old commits except for checking for malicious changes. (at least for spec files, with code it is useful for bisecting bugs).
For end-users it might be useful to get the changelog alone (for that it does not matter if it is generated or copied from the .spec), but I never had any use for the changelog without the .spec file, as this gives me the context to the changes in the .spec file.
But after all I do not care too much about how changelog is created, as long as the previous functionality is still preserved - my git log messages contain information about the .spec file changes while the changelog contains changes about the functionality of the package. ("what have I changed" (git) vs. "what has changed for you" (cl))
I think it is a great point. Commit message very often just say something like new upstream release. In the final changelog it would be nice to additionally e.g. provide a link to the upstream release summary - but it's also more work.
Anyway, the messages will change if you really try to explain to user what has changed. That's true.
All the best, Astra _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Thu, Feb 27, 2020 at 12:21:48PM +0100, clime wrote:
On Thu, 27 Feb 2020 at 10:00, David Kaufmann astra@ionic.at wrote:
Another idea would be generating a changelog-entry from git history when creating an update in bodhi, and there is no pre-existing changelog-entry for the current version.
But Bodhi changelogs is not what user can read on his/her machine when examining e.g. dnf check-update --changelogs. These are imho rpm changelogs. So the rpm spec changelogs are the most important.
Ah, sorry, what I meant to say is bodhi modifying the .spec file, adding the changelog line, and committing it, before building the package. (similarly like the releng commits when rebuilding stuff for new releases)
But when I am thinking more about it I think it won't work, as bodhi would need to rebuild the package then. (afaik it does not, as koji built it already)
An alternative might be something like modifying "fedpkg build" to check for missing changelogs and ask something like:
"your package does not contain a changelog entry for <version>. should we add the following entryies to the changelog for <version>?: <list of commit-messages>"
maybe even with the option of modifying the lines?
All the best, Astra
On Thu, 27 Feb 2020 at 12:42, David Kaufmann astra@ionic.at wrote:
On Thu, Feb 27, 2020 at 12:21:48PM +0100, clime wrote:
On Thu, 27 Feb 2020 at 10:00, David Kaufmann astra@ionic.at wrote:
Another idea would be generating a changelog-entry from git history when creating an update in bodhi, and there is no pre-existing changelog-entry for the current version.
But Bodhi changelogs is not what user can read on his/her machine when examining e.g. dnf check-update --changelogs. These are imho rpm changelogs. So the rpm spec changelogs are the most important.
Ah, sorry, what I meant to say is bodhi modifying the .spec file, adding the changelog line, and committing it, before building the package. (similarly like the releng commits when rebuilding stuff for new releases)
But when I am thinking more about it I think it won't work, as bodhi would need to rebuild the package then. (afaik it does not, as koji built it already)
Ah, right, I understand now. Thank you for the explanation!
An alternative might be something like modifying "fedpkg build" to check for missing changelogs and ask something like:
"your package does not contain a changelog entry for <version>. should we add the following entryies to the changelog for <version>?: <list of commit-messages>"
maybe even with the option of modifying the lines?
That would be possible, yes, but then fedpkg build would need to create additional commit to maintain what user have inputed so that it is remembered somewhere and present in the resulting rpm, i.e. it would probably need to change repository and do a push during the build launching.
I had a similar idea but executed earlier. When` fedpkg tag` is invoked (packagers usually don't invoke it today), a test-editor window is opened that can be prepopulated from commit message but it could also be prepopulated from an external file depending on configuration. Here user can compose the final changelog record for the given release and this record is then stored in the newly created tag (in its annotation) and from there it is automatically (through a dedicated macro if used) rendered into spec file when needed (e.g. when srpm is built).
It requires one more step (i.e. fepkg commit -> fedpkg tag -> fedpkg push -> fedpkg build) but the build can be also launched automatically if a new tag is pushed so the number of steps would be the same.
A disadvantage is that the spec file is not standalone with respect to important information. It can be, however, made standalone at any time by "rendering" it, i.e. process when the macros are replaced by the actual information pulled from git (the command to do it could be e.g. fedpkg spec).
All the best, Astra _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Thu, 27 Feb 2020 at 13:23, clime clime@fedoraproject.org wrote:
On Thu, 27 Feb 2020 at 12:42, David Kaufmann astra@ionic.at wrote:
On Thu, Feb 27, 2020 at 12:21:48PM +0100, clime wrote:
On Thu, 27 Feb 2020 at 10:00, David Kaufmann astra@ionic.at wrote:
Another idea would be generating a changelog-entry from git history when creating an update in bodhi, and there is no pre-existing changelog-entry for the current version.
But Bodhi changelogs is not what user can read on his/her machine when examining e.g. dnf check-update --changelogs. These are imho rpm changelogs. So the rpm spec changelogs are the most important.
Ah, sorry, what I meant to say is bodhi modifying the .spec file, adding the changelog line, and committing it, before building the package. (similarly like the releng commits when rebuilding stuff for new releases)
But when I am thinking more about it I think it won't work, as bodhi would need to rebuild the package then. (afaik it does not, as koji built it already)
Ah, right, I understand now. Thank you for the explanation!
An alternative might be something like modifying "fedpkg build" to check for missing changelogs and ask something like:
"your package does not contain a changelog entry for <version>. should we add the following entryies to the changelog for <version>?: <list of commit-messages>"
maybe even with the option of modifying the lines?
That would be possible, yes, but then fedpkg build would need to create additional commit to maintain what user have inputed so that it is remembered somewhere and present in the resulting rpm, i.e. it would probably need to change repository and do a push during the build launching.
I had a similar idea but executed earlier. When` fedpkg tag` is invoked (packagers usually don't invoke it today), a test-editor window is opened that can be prepopulated from commit message but it could also be prepopulated from an external file depending on configuration. Here user can compose the final changelog record for the given release and this record is then stored in the newly created tag (in its annotation) and from there it is automatically (through a dedicated macro if used) rendered into spec file when needed (e.g. when srpm is built).
It requires one more step (i.e. fepkg commit -> fedpkg tag -> fedpkg push -> fedpkg build) but the build can be also launched automatically if a new tag is pushed so the number of steps would be the same.
Actually, I thought about your idea doing it during `fedpkg build` (create the changelog record and push) and I started to like it more and more because people wouldn't need to run any extra commands compared to today when they would like to use the new macros.
This could be only an opt-in feature (create_changelog_and_push_before_build or similar) cause some people will not want this.
There are two ways how to create such record and push it without actually changing HEAD commit (which would be good cause otherwise we would be building different commit then we launched the command on, which would lead to confusion)
1) git notes: mutable text records that can be attached to any commit probably or tag in a git repo (see $ man git notes) and they have a separate history 2) git's annoted tags: tags with immutable text records and timestamps that can be signed and are checksummed.
This is a direction of thought where the changelog is stored in git metadata. Proposal with the external changelog file in the repository is, instead, a direction where the changelog is stored in git data (well partially data and partially metadata because of the commit message fallback) - it's a bit different.
Another thing to consider is whether we want a transparent build system where all the description of what will happen when a spec file is sent to it is included in the specfile itself or whether we want some auto-magic where e.g. 'Release' field or %changelog are automatically inserted when missing (something like that is impossible right now due to rpm constraints but just to make a point).
A disadvantage is that the spec file is not standalone with respect to important information. It can be, however, made standalone at any time by "rendering" it, i.e. process when the macros are replaced by the actual information pulled from git (the command to do it could be e.g. fedpkg spec).
All the best, Astra _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Thu, 2020-02-27 at 15:10 +0100, clime wrote:
Another thing to consider is whether we want a transparent build system where all the description of what will happen when a spec file is sent to it is included in the specfile itself or whether we want
But we don't have that today: think of macros defined externally to the SPEC file and RPM which we use, think of "underspecified" build dependencies where what happens to be available and fulfil the BRs will be used and ultimately influence how the package is built. What happens when you build a package is already dependent on outside sources.
some auto-magic where e.g. 'Release' field or %changelog are automatically inserted when missing (something like that is impossible right now due to rpm constraints but just to make a point).
Again, this is opt-in. And the way this will be made opt-in is that people who want to use the feature use a macro, e.g. "%automatic_release", so that it is clear that some value will be filled in there. The idea isn't to have gutted spec files that don't build anywhere else than in our build system, one fixed requirement is that local builds using fedpkg or rpmbuild must continue to work.
Nils
On Thu, 27 Feb 2020 at 17:29, Nils Philippsen nils@redhat.com wrote:
On Thu, 2020-02-27 at 15:10 +0100, clime wrote:
Another thing to consider is whether we want a transparent build system where all the description of what will happen when a spec file is sent to it is included in the specfile itself or whether we want
But we don't have that today: think of macros defined externally to the SPEC file and RPM which we use, think of "underspecified" build dependencies where what happens to be available and fulfil the BRs will be used and ultimately influence how the package is built. What happens when you build a package is already dependent on outside sources.
some auto-magic where e.g. 'Release' field or %changelog are automatically inserted when missing (something like that is impossible right now due to rpm constraints but just to make a point).
Again, this is opt-in. And the way this will be made opt-in is that people who want to use the feature use a macro, e.g. "%automatic_release", so that it is clear that some value will be filled in there. The idea isn't to have gutted spec files that don't build anywhere else than in our build system, one fixed requirement is that local builds using fedpkg or rpmbuild must continue to work.
Yes, that sounds good! I just wouldn't like too much when a build system would be fiddling with spec files without some explicit macro that everyone can see _in_ the spec file telling it explicitly to do it - like e.g. changing values of some fields that packager manually specified for some reason, e.g. auto correction.
Nils
Nils Philippsen "Those who would give up Essential Liberty to Software Engineer purchase a little Temporary Safety, deserve neither Red Hat Liberty nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: D0C1 1576 CDA6 5B6E BBAE 95B2 7D53 7FCA E9F6 395D old: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 2/27/20 12:08 AM, Dan Čermák wrote:
For the changelog: yes please, generate it from the commit log! They are more or less the same for all my packages and I'm getting tired of copy pasting the same text into %changelog and git commit.
Do you know about fedpkg commit --clog ?
On Monday, 24 February 2020 17:48:36 CET Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ?
|
B -- version: 1.0 -- release: 1.0
|
C -- version: 1.0 -- release: 2.0
|
D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to
the packaging guidelines)
- Allows triggering two builds from the same commit without any manual
change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had
45 commits leading to this)
- Relies on information from the build system for the build number (but
can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with
one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro
options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other
Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our
packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
Best regards,
Robert-André
On Wed, Feb 26, 2020 at 8:05 PM Robert-André Mauchin zebob.m@gmail.com wrote:
On Monday, 24 February 2020 17:48:36 CET Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ?
|
B -- version: 1.0 -- release: 1.0
|
C -- version: 1.0 -- release: 2.0
|
D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to
the packaging guidelines)
- Allows triggering two builds from the same commit without any manual
change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had
45 commits leading to this)
- Relies on information from the build system for the build number (but
can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with
one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro
options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other
Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our
packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
You don't use Release for upstream versioning, even for snapshots. For your examples:
* 0-0.1.beta.2 -> 0~beta.2-1 * 0-0.1.20120225gitd6c789a -> 0~git20120225.d6c789a-1
Le mercredi 26 février 2020 à 23:07 -0500, Neal Gompa a écrit :
You don't use Release for upstream versioning, even for snapshots. For your examples:
- 0-0.1.beta.2 -> 0~beta.2-1
- 0-0.1.20120225gitd6c789a -> 0~git20120225.d6c789a-
Sorry but no
You are attempting to redefine the meaning of Version (*upstream* version) to accomodate your release simplification plans
As I wrote the list months ago the upstream Version (Version, %{version}) is zero 0 nil not 0~git20120225.d6c789a-1
The git20120225.d6c789a is a Fedora downstream construct
You can not break the data model with automation the way you break it for humans. Automation does not care about your feelings. Automation input is O as upstream Version. It can add downstream constructs to Release, it can not rewrite Version (bad bad idea to attempt rewriting a core rpm Tag in macros anyway)
Le 2020-02-27 08:35, Nicolas Mailhot via devel a écrit :
Le mercredi 26 février 2020 à 23:07 -0500, Neal Gompa a écrit :
You don't use Release for upstream versioning, even for snapshots. For your examples:
- 0-0.1.beta.2 -> 0~beta.2-1
- 0-0.1.20120225gitd6c789a -> 0~git20120225.d6c789a-
Sorry but no
You are attempting to redefine the meaning of Version (*upstream* version) to accommodate your release simplification plans
To expand a little: as soon as you are in commit land, upstream ordering is effectively undefined.
It is a packager (hopefully educated) judgment call to impose a Fedora packaging order over an upstream commit sequence, to decide if a commit looks like more a pre or post release, do decide to take the commit in a specific upstream branch or not, etc.
Upstream commit ordering is undefined. Upstream did not pledge to evolve commits in a specific non-API breaking linear direction. They can make API breaks, revert them, reorder, fix Fedora problems in a topic branch, merge topic branches in no particular order, give up on topic branches and re-do them some other way, rebase, fork, etc. The Fedora packager may make a mistake in its educated guess and target a different branch or commit point in its next package release.
Even if we had deep integration with upstream’s code repositories (which we have not today, and which would cause deep problems every time upstream shuffles its commits or re-hosts or forks, losing some history, or when an upstream repo URL goes away for another reason) we could not reliably automate an evr ordering from upstream’s commit history (we’d have to effectively epoch every single commit change, and that is moving the problem from release to version to epoch instead of merely moving it from releasez to version).
Release to upstream commit mapping is a downstream Fedora invention. It has no place in Version. Please stop pretending upstream made the commit choice for us. If it was ready to make this choice, it would have tagged a specific release point.
Regards,
On Wed, 2020-02-26 at 23:07 -0500, Neal Gompa wrote:
On Wed, Feb 26, 2020 at 8:05 PM Robert-André Mauchin < zebob.m@gmail.com> wrote:
On Monday, 24 February 2020 17:48:36 CET Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ?
|
B -- version: 1.0 -- release: 1.0
|
C -- version: 1.0 -- release: 2.0
|
D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No
changes to the packaging guidelines)
- Allows triggering two builds from the same commit without any
manual change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the
build
Cons:
- Old packages which are no longer receiving upstream releases
may need special care (for example if they are at the release -48 but only had 45 commits leading to this)
- Relies on information from the build system for the build
number (but can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version
X with one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without
manual intervention
- Emulates what many maintainers do manually today for most use
cases
- Can cater for pre-releases (e.g.: by offering different
macros or macro options for the different use cases)
Cons:
- Needs the build system for information about builds in this
and other Fedora versions
- Not easy to reproduce locally because we don’t have machine-
consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-
release` information for all commits (and this will require a change in our packaging guidelines)
So these are the results of our current investigations, we are very much eager to get your feedback on them and even more eager if you have ideas on how to approach/solve some of the challenges mentioned here.
Looking forward for the discussion,
Pierre For Adam, Nils and myself
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
You don't use Release for upstream versioning, even for snapshots. For your examples:
- 0-0.1.beta.2 -> 0~beta.2-1
- 0-0.1.20120225gitd6c789a -> 0~git20120225.d6c789a-1
This doesn't work reliably as the string "git" can't be sorted sensibly between e.g. "alpha", "beta", "rc" tagged upstream pre-releases, because a git commit can come from anywhere on the timeline. Here's what the Versioning Guidelines[1] start off with: "If you wish to package a prerelease version and are confident that you will need to package only tagged releases and not any snapshots until the next release, you MAY make use of RPM’s tilde (‘~’) notation."
Nils
[1]: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_versi...
On Thu, Feb 27, 2020 at 02:04:41AM +0100, Robert-André Mauchin wrote:
On Monday, 24 February 2020 17:48:36 CET Pierre-Yves Chibon wrote:
However, for the release field, we are struggling a little bit more, two options are more appealing to us:
A) The release field is automatically generated using two elements:
- the number of commits at this version
- the number of builds at this commit
- the dist-tag being added after them
The release field would thus look like: ``<number of commit at version>.<number of build at commit>%{dist}``
So we could have: (A, B, C and D being different commits) A -- version: 0.9 -- release: ?
|
B -- version: 1.0 -- release: 1.0
|
C -- version: 1.0 -- release: 2.0
|
D -- version: 1.1 -- release: 1.0
A rebuild of the commit D, would lead to a release 1.1 (assuming the first one succeeded)/
Pros:
- Easy to replicate locally
- Every changelog entry can be mapped to a `version-release` (No changes to
the packaging guidelines)
- Allows triggering two builds from the same commit without any manual
change (simplifies mass-rebuilds)
- Easy to link a certain build with a specific commit
- Easy to “guess” the next release value before triggering the build
Cons:
- Old packages which are no longer receiving upstream releases may need special care (for example if they are at the release -48 but only had
45 commits leading to this)
- Relies on information from the build system for the build number (but
can be closely simulated locally since the <n_build> info is only used for rebuilds)
- Upgrade path may be problematic if Fn is upgraded to version X with
one commit while Fn-1 is upgrade to version X with 2 commits (or more)
B) The release field is automatically generated taking existing builds in all current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for builds of the same epoch:version, find a new release that (if at all possible) ensures upgradability from older Fedora versions to newer ones, adhering to the structure of a release tag documented in the Versioning Guidelines[1]. Going from the (RPM-wise) "latest build" that the new one should surpass, this can mean bumping in the front (`pkgrel`) or the back (`minorbump`).
This allows packages from "very stable" upstreams who haven't released in years to still benefit from automatically generated releases.
The following examples would use a macro for the release field as outlined in a separate document[2].
Example #1 ("normal" release progression): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F31: 1.0-2.fc32
Example #2 ("hotfix" in an older release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in F30: 1.0-1.fc30.1
Example #3 (pre-release, selected by an alternative macro (or option) in the spec file): Rawhide has: 2.0-1.fc32 F31 has: 1.0-1.fc31 F30 has: 1.0-1.fc30
Next release in Rawhide: 3.0-0.1.20200224git1234abcd
Pros:
- Allows triggering two builds from the same commit without manual intervention
- Emulates what many maintainers do manually today for most use cases
- Can cater for pre-releases (e.g.: by offering different macros or macro
options for the different use cases)
Cons:
- Needs the build system for information about builds in this and other
Fedora versions
- Not easy to reproduce locally because we don’t have machine-consumable knowledge about other builds in e.g. the dist-git repo
- Does not allow to generate changelog entries with `version-release` information for all commits (and this will require a change in our
packaging guidelines)
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
This is something that we will need to investigate and clarify a little more, the answer may very well be: it won't, but let's investigate this first.
We are very well aware that the proposal in its current form will not address all packages. The idea being to improve the situation for some packages, see how this works and see how we can grow this number of packages. Reading this thread you can already see that some packagers are not interested in such a system regardless of how it is, which is fine, we want it to be opt-in.
Pierre
On 27. 02. 20 9:20, Pierre-Yves Chibon wrote:
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
This is something that we will need to investigate and clarify a little more, the answer may very well be: it won't, but let's investigate this first.
There are three ways of of there I can think of ATM:
1. (as said by Pierre) make it opt-in only and don't handle this 2. (as said by Neal) don't do this, use 0.1~beta.2-<release> 3. allow to keep the Release filed if it uses %{baserelease}
%{baserelease} is already respected by rpmdev-bumpspec (and hence mass rebuilds)
%global baserelease 8 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
bumpspec does:
%global baserelease 9 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
So we can reuse that and say: If the Release is manually defined in spec and uses %{baserelease} and %{baserelease} is not defined in spec, the automation sets the %{baserelease} value instead of setting the release directly.
Le 2020-02-27 09:52, Miro Hrončok a écrit :
On 27. 02. 20 9:20, Pierre-Yves Chibon wrote:
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
This is something that we will need to investigate and clarify a little more, the answer may very well be: it won't, but let's investigate this first.
There are three ways of of there I can think of ATM:
- (as said by Pierre) make it opt-in only and don't handle this
- (as said by Neal) don't do this, use 0.1~beta.2-<release>
- allow to keep the Release filed if it uses %{baserelease}
%{baserelease} is already respected by rpmdev-bumpspec (and hence mass rebuilds)
%global baserelease 8 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
bumpspec does:
%global baserelease 9 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
Go (and other) packages do not set Version nor Release. They set distprefixNUMBER, which is defined in https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea...
This automation must be kept or migrated to something else if people do not want to keep it ("just write bogus upstream versions manually is not a forward path)
If changelog is automated, distprefixNUMBER should end up somewhere in it, because that’s important information not present in Version
https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea... also defines some things equivalent to a distpostfix. They probably belong in changelog too.
Regards,
On Thu, 27 Feb 2020 at 10:47, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-02-27 09:52, Miro Hrončok a écrit :
On 27. 02. 20 9:20, Pierre-Yves Chibon wrote:
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
This is something that we will need to investigate and clarify a little more, the answer may very well be: it won't, but let's investigate this first.
There are three ways of of there I can think of ATM:
- (as said by Pierre) make it opt-in only and don't handle this
- (as said by Neal) don't do this, use 0.1~beta.2-<release>
- allow to keep the Release filed if it uses %{baserelease}
%{baserelease} is already respected by rpmdev-bumpspec (and hence mass rebuilds)
%global baserelease 8 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
bumpspec does:
%global baserelease 9 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
Go (and other) packages do not set Version nor Release. They set distprefixNUMBER, which is defined in https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea...
Hello Nicolas,
can you, please, show an example of such package? I was searching through some golang packages because I was curious how it works but couldn't find an example (i.e. those packages had Version and Release set, e.g. https://src.fedoraproject.org/rpms/golang-github-anacrolix-missinggo/blob/ma...).
This automation must be kept or migrated to something else if people do not want to keep it ("just write bogus upstream versions manually is not a forward path)
If changelog is automated, distprefixNUMBER should end up somewhere in it, because that’s important information not present in Version
https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea... also defines some things equivalent to a distpostfix. They probably belong in changelog too.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2020-02-27 12:59, clime a écrit : Hi,
can you, please, show an example of such package? I was searching through some golang packages because I was curious how it works but couldn't find an example
A Go example: https://src.fedoraproject.org/rpms/golang-x-build
A non-Go example: https://src.fedoraproject.org/rpms/impallari-dancing-script-fonts
In both cases the spec calls a macro (not the same one in both specs) that causes one or several disprefixNUMBER variables to be computed
The result then ends in Release via https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea...
The font spec example calls the macro in redhat-rpm-config directly, the go example calls it through a Go-specific macro in fonts-rpm-macros
Regards,
On Thu, 27 Feb 2020 at 16:26, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-02-27 12:59, clime a écrit : Hi,
can you, please, show an example of such package? I was searching through some golang packages because I was curious how it works but couldn't find an example
A Go example: https://src.fedoraproject.org/rpms/golang-x-build
A non-Go example: https://src.fedoraproject.org/rpms/impallari-dancing-script-fonts
In both cases the spec calls a macro (not the same one in both specs) that causes one or several disprefixNUMBER variables to be computed
The result then ends in Release via https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea...
The font spec example calls the macro in redhat-rpm-config directly, the go example calls it through a Go-specific macro in fonts-rpm-macros
Okay, thanks :)!
At least my idea was that there should be a spec file preprocessor, evaluating the macros that need git context, that would produce valid rpm spec that will be evaluated by the rpm engine in the end so everything related to %{dist} generation that is used in those packages will work unchanged.
A small tweak in those spec files to allow dynamic release bumping could be:
Release: 0.<<dynamic_release>>%{?dist}
or alternatively:
Release: <<dynamic_release prefix=0>>%{?dist} depending on the implementation of the dynamic_release macro.
clime
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le jeudi 27 février 2020 à 17:38 +0100, clime a écrit :
On Thu, 27 Feb 2020 at 16:26, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-02-27 12:59, clime a écrit : Hi,
can you, please, show an example of such package? I was searching through some golang packages because I was curious how it works but couldn't find an example
A Go example: https://src.fedoraproject.org/rpms/golang-x-build
A non-Go example: https://src.fedoraproject.org/rpms/impallari-dancing-script-fonts
In both cases the spec calls a macro (not the same one in both specs) that causes one or several distprefixNUMBER variables to be computed
The result then ends in Release via https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea...
The font spec example calls the macro in redhat-rpm-config directly (%forgemeta), the Go example calls it through a Go-specific macro in go-rpm-macros (%gometa)
Okay, thanks :)!
At least my idea was that there should be a spec file preprocessor, evaluating the macros that need git context, that would produce valid rpm spec that will be evaluated by the rpm engine in the end so everything related to %{dist} generation that is used in those packages will work unchanged.
A small tweak in those spec files to allow dynamic release bumping could be:
Release: 0.<<dynamic_release>>%{?dist}
or alternatively:
Release: <<dynamic_release prefix=0>>%{?dist} depending on the implementation of the dynamic_release macro.
Yes, something like that would be possible, my worry was essentially about not snipping commit info from automated changelogs.
%{dynrel} should default to 1 if buildsys info is not available
Regards,
On Thu, 27 Feb 2020 at 09:53, Miro Hrončok mhroncok@redhat.com wrote:
On 27. 02. 20 9:20, Pierre-Yves Chibon wrote:
How would that work with "complex" releases? For example release containing prerelease info like 0.1.beta.2 or 0.1.20120225gitd6c789a? Many Go package have no version, so depend heavily on the Release tag to signal what is the snapshot date and git commit packaged.
This is something that we will need to investigate and clarify a little more, the answer may very well be: it won't, but let's investigate this first.
There are three ways of of there I can think of ATM:
- (as said by Pierre) make it opt-in only and don't handle this
- (as said by Neal) don't do this, use 0.1~beta.2-<release>
- allow to keep the Release filed if it uses %{baserelease}
%{baserelease} is already respected by rpmdev-bumpspec (and hence mass rebuilds)
%global baserelease 8 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
bumpspec does:
%global baserelease 9 Release: iamcrazy1234568andIknowit.%{baserelease}.whatnot~foo666%{?dist}
So we can reuse that and say: If the Release is manually defined in spec and uses %{baserelease} and %{baserelease} is not defined in spec, the automation sets the %{baserelease} value instead of setting the release directly.
Hello Miro,
interesting idea. I have some notes:
If i understand correctly, this would rely on the locally undefined %{baserelease} macro, which is later provided auto-magically by the build system. Should this macro be populated also locally if not defined?
I already mentioned it earlier in this thread but I think the use of bare rpm macro is problematic because they either need to have git context (metadata) around to evaluate or they would even need to do remote network requests. Do you disagree?
But anyway, what packager provides statically in the Release filed without using a dedicated macros should imho stay there and not be somehow magically deleted and overridden by build system. So the solution might be simply:
Release: beta.2.<<dynamic_release_part>>
or it might be:
Release: <<dynamic_release prefix=beta.2>>
if the macro for the release computation needs to know the value of the static prefix (that would be the case for git_release macro defined in rpkg-util).
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 27. 02. 20 11:57, clime wrote:
If i understand correctly, this would rely on the locally undefined %{baserelease} macro, which is later provided auto-magically by the build system. Should this macro be populated also locally if not defined?
Yes, by fedpkg. Now should the specs be buildable standalone? Possibly by defining baserelease_default value in some SRPM macro package.
So the specs would have:
Release: beta.2.%{dynamic_release}
And %{dynamic_release} would be defined as:
%{?_buildsystem_dynamic_release}%{?!_buildsystem_dynamic_release:1}
And fedpkg would populate %_buildsystem_dynamic_release as it does with %fedora etc.
We would just need to teach bumpspec (or more generally releng rebuild scripts) to not try to manually bump releases with %{dynamic_release}.
I already mentioned it earlier in this thread but I think the use of bare rpm macro is problematic because they either need to have git context (metadata) around to evaluate or they would even need to do remote network requests. Do you disagree?
No. I agree.
Hi,
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
...
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
This means that a separate commit needs to be done *after* on top of the commits doing the actual changes. It's a bit disappointing, but on its own would not be too bad, since we can have fedpkg provide a higher level command that combines generate-changelog and build...
One important feature will work: being able to cherry-pick commit between branches w/o spurious conflicts. As long as the "real" change to the spec file are done in separate commits, and the changelog commit is another commit on top, then when cherry-picking to another branch, the "real" commits would be cherry-pickable, and then the changelog commit would be recreated using the automation, OK.
But it doesn't work quite as nicely with something similar: merging branches. A simple 'git merge' would result in conflicts.
Also, if an amendment to the changelog is done, and the same change needs to be done in the changelog in a different branch, trying to cherry-pick the fix commit would most likely result in a merge conflict.
Considering these three drawbacks (separate commit step and resulting log noise, merge conflicts), I'd very much hope for a solution where the changelog is never stored in the version control, and is always autogenerated at srpm creation time. We should never try to chery-pick commits that have changelog entries with actual date or e-v-r texts, since this will inevitably lead to merge conflicts.
A different approach: - we have 'fedpkg generate-changelog' (which does all the git log extraction that was described, I think that part is OK), - the output from this command included in the srpm at srpm generation time and never committed to version control, - the output is annotated with the source commits hashes, so we can see where each line came from.
At any time, the packager can run 'fedpkg generate-changelog' to see how the output looks like. If they see something they don't like, if the commits haven't been pushed out yet, they can immediately run 'git commit --amend' and recheck. If they have been pushed out, an override to the changelog could be committed as part of a commit message text.
Git-changelog-suppress: ad5555b42e or Git-changelog-suppress: ad5555b42e..efefedeadb or Git-changelog-replace: ad5555b42e Some other text with typos fixed that completely overrides whatever was generated from ad5555b42e. or Git-changelog-append: ad5555b42e (additional clarification for commit ad5555b42e, e.g. bug #12345)
This would have the following advantages: - git log is the sole source of truth - there are no cherry-pick or merge conflicts - there is no separate "now I'm done and I need to do another commit before building" thing.
The assumption is that those Git-changelog-* macros would only be used occasionally, if the bad changelog entry was not noticed before pushing the changes out.
(One nitpick: when cherry-picking between branches, hashes of the cherry-picked commits change, so "ad5555b42e" in the example above would stop working. This is handled by using 'git cherry-pick -x'. 'fedpkg generate-changelog' would look at any hash in a "(cherry picked from commit ...)" line.)
As how to hook this up with rpm, looking at https://pagure.io/packaging-committee/pull-request/942#comment-108542, a macro like %generate_changelog could be provided that would simply shell out to 'fedpkg generate-changelog'.
I'll comment separate on the -release part.
Zbyszek
On Thu, 27 Feb 2020 at 16:13, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Hi,
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
...
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
This means that a separate commit needs to be done *after* on top of the commits doing the actual changes. It's a bit disappointing, but on its own would not be too bad, since we can have fedpkg provide a higher level command that combines generate-changelog and build...
One important feature will work: being able to cherry-pick commit between branches w/o spurious conflicts. As long as the "real" change to the spec file are done in separate commits, and the changelog commit is another commit on top, then when cherry-picking to another branch, the "real" commits would be cherry-pickable, and then the changelog commit would be recreated using the automation, OK.
But it doesn't work quite as nicely with something similar: merging branches. A simple 'git merge' would result in conflicts.
Also, if an amendment to the changelog is done, and the same change needs to be done in the changelog in a different branch, trying to cherry-pick the fix commit would most likely result in a merge conflict.
Considering these three drawbacks (separate commit step and resulting log noise, merge conflicts), I'd very much hope for a solution where the changelog is never stored in the version control, and is always autogenerated at srpm creation time. We should never try to chery-pick commits that have changelog entries with actual date or e-v-r texts, since this will inevitably lead to merge conflicts.
A different approach:
- we have 'fedpkg generate-changelog' (which does all the git log extraction that was described, I think that part is OK),
- the output from this command included in the srpm at srpm generation time and never committed to version control,
- the output is annotated with the source commits hashes, so we can see where each line came from.
At any time, the packager can run 'fedpkg generate-changelog' to see how the output looks like. If they see something they don't like, if the commits haven't been pushed out yet, they can immediately run 'git commit --amend' and recheck. If they have been pushed out, an override to the changelog could be committed as part of a commit message text.
Git-changelog-suppress: ad5555b42e or Git-changelog-suppress: ad5555b42e..efefedeadb or Git-changelog-replace: ad5555b42e Some other text with typos fixed that completely overrides whatever was generated from ad5555b42e. or Git-changelog-append: ad5555b42e (additional clarification for commit ad5555b42e, e.g. bug #12345)
This would have the following advantages:
- git log is the sole source of truth
- there are no cherry-pick or merge conflicts
- there is no separate "now I'm done and I need to do another commit before building" thing.
The assumption is that those Git-changelog-* macros would only be used occasionally, if the bad changelog entry was not noticed before pushing the changes out.
(One nitpick: when cherry-picking between branches, hashes of the cherry-picked commits change, so "ad5555b42e" in the example above would stop working. This is handled by using 'git cherry-pick -x'. 'fedpkg generate-changelog' would look at any hash in a "(cherry picked from commit ...)" line.)
Hello Zbyszek,
it's good that it solves conflicts when cherry-picking and merging and git log is the source of truth but simply dropping a line that i don't want in the changelog or appending something to it seems to be quite painful to do.
One advantage you mention is:
- there is no separate "now I'm done and I need to do another commit before building" thing.
But if you want to modify the automatic product, you would need one or more additional commits to do that.
Imho, one would prefer to be able to do that simply by editing a text file (even if temporary). Annotated tags give you that ability by design - the tag annotation can be prepopulated and then freely edited, which is much more pleasant in my opinion. It doesn't clutter commit space and you can point other branches to that particular annotation by merging if desired. But the cherry-picking workflow, on the other hand, would require creating a new tag specific for that branch (that would suggest the tags should contain branchname/%dist at the end as fedpkg does it now). That's were things start getting clumsy unless there is something like fedpkg build --annotation-from f31 to make a new tag with the same content as is the one in f31 branch, push it and build.
Similar workflow with git notes should be imho also considered.
I also wonder whether: "Changelog generated must be editable" from the document means after pushing or before pushing and whether it is such a hard requirement. If I make some "critical" mistake like referencing wrong CVE, it might be better to create a new changelog entry and new release, that will otherwise contain no changes and will simply provide a message of type:
- Changelog correction for release 1.5.1: - 1.5.1 does _not_ fix CVE-2020-3123 - 1.5.1 _fixes_ CVE-2020-3124
I think cases when a maintainer wants to do something like this are rare and if they happen it might be better to model that case like an "email communication", rather than "into the past mutable channel". In case of annotated tags, that would mean some commits would have two attached (rare situation), the second one with the correcting message. With respect to automatically generated release numbers, the machinery in rpkg-util would handle it correctly, idk about other approaches.
One disadvantage of annotated tags might be that you cannot merge them from pull requests but on the other hand, one can at least provide "the prepopulated content" (e.g. simply by messages in the commits being merged) that maintainer can then use when the tag is being created.
I am talking about annotated tags because i am familiar with that approach and i don't have that much to say about other approaches at the moment.
As how to hook this up with rpm, looking at https://pagure.io/packaging-committee/pull-request/942#comment-108542, a macro like %generate_changelog could be provided that would simply shell out to 'fedpkg generate-changelog'.
I'll comment separate on the -release part.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Hi,
Anyway, here is what I would personnaly consider a robust, inclusive, and future-proof design:
— a %{use_dynstate} rpm variable enables dynamic changelog/release behaviour — probably initialy set to false distro-wide, to let volunteers test the thing by setting it to true iin their specs, — then to true (opt-out), when kinks have been fixed, and FPC/FESCO greenlights general availability
– when activated, last changelog, evr and %{dynrel} state are saved in one or several detached files — evr computed using a fixed neutral %{dist} value (for example “-” since it’s forbidden in %{dist})
– those files are given standard rpm variable names and added to Sources: – manual packager Source900: %{dynstate} or whatever – rpm later updated to allow source file declarations via macros at to eliminate boilerplate (I need this in forge and go packages) – or the detached files are set in stone as separate Tags with a default value overridable via the %{dynstate} variable in a new rpm version
– the packager adds %{dynrel} wherever he sees fit in his Release string
– at fedpkg commit time changelog state is updated with info taken from fedora git state, *without* evr and build date – that’s Fedora-specific integration, exact commit/tag syntax to mark things to inject or ignore TBD Fedora-side – fedpkg commit sets a tag in git to mark anything older can now be ignored for changelog generation purposes – detached changelog state remains packager-editable, like the in- spec changelog today. That allows pruning useless no longer relevant stuff and fixing grammar errors
— at rpmbuild -bs stage – evr is computed using the same neutral %{dist} value as before, and the saved %{dynrel} value – if it is equal to the previously saved evr %{dynrel} is bumped and saved in the detached file – a build line is added to the detached changelog, using the current date and full evr (without %{dist} neutralization) — that probably requires defining a rpm variable to track whom the build is attributed to – it can default to "Anonymous coward" or whatever if not explicitely set – Koji, OBS and Copr can set it to whoever asked them to build the package – the result constitutes the new srpm (either first or second level srpm as upstream rpm sees fit) – that’s generic non Fedora-specific behaviour that work as well in rpmbuild -bs, mock, copr, koji, OBS, whatever, with or without Fedora git presence – Koji/copr need to commit the new dynamic dynrel/changelog state to git (a build-striggered state change is commited by the build system)
And yes that requires some work rpm-side. That is necessary to maintain the rpm decentralized design and keep srpms independent from anyone’s git state. Personnaly, I don’t see the point of pretending we can move our infra forward without ever touching rpm. The cardinal sin of Modularity was attempting to create an overlay over existing rpm/dnf behaviour, without changing this behaviour when it needed improving.
Contrat that with dynamic buildrequires or weak deps that slotted into our workflows with hardly any ripple. Though they were major feature changes. But, they were done with rpm upstream, instead of trying to bypass it.
Regards,
On Sat, 29 Feb 2020 at 10:15, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Hi,
Anyway, here is what I would personnaly consider a robust, inclusive, and future-proof design:
I will need to ask some questions to really understand it.
— a %{use_dynstate} rpm variable enables dynamic changelog/release behaviour — probably initialy set to false distro-wide, to let volunteers test the thing by setting it to true iin their specs, — then to true (opt-out), when kinks have been fixed, and FPC/FESCO greenlights general availability
– when activated, last changelog, evr and %{dynrel} state are saved in one or several detached files
You mean the last changelog, evr, and %{dynrel} are stored once %{use_dynstate} is set and and after one invokes fedpkg commit? I think there should be some explicit action (e.g. the commit) to generate the files after I set the %{use_dynstate} value to true in the spec file.
How is %{dynrel} computed here at this stage - does it have something to do with the number of commits from the latest version change or similar?
— evr computed using a fixed neutral %{dist} value (for example “-” since it’s forbidden in %{dist})
– those files are given standard rpm variable names and added to Sources: – manual packager Source900: %{dynstate} or whatever – rpm later updated to allow source file declarations via macros at to eliminate boilerplate (I need this in forge and go packages) – or the detached files are set in stone as separate Tags with a default value overridable via the %{dynstate} variable in a new rpm version
– the packager adds %{dynrel} wherever he sees fit in his Release string
– at fedpkg commit time changelog state is updated with info taken from fedora git state, *without* evr and build date – that’s Fedora-specific integration, exact commit/tag syntax to mark things to inject or ignore TBD Fedora-side – fedpkg commit sets a tag in git to mark anything older can now be ignored for changelog generation purposes – detached changelog state remains packager-editable, like the in- spec changelog today. That allows pruning useless no longer relevant stuff and fixing grammar errors
— at rpmbuild -bs stage – evr is computed using the same neutral %{dist} value as before, and the saved %{dynrel} value – if it is equal to the previously saved evr %{dynrel} is bumped and saved in the detached file
Is the intention here that with each new build of the same package, the value of %{dynrel} is bumped and committed back?
Only if other parts of release (other from %{dist} and %{dynrel}) or version or release have changed, nothing is done? That means the evr file read from the repository needs to contain somehow outdated values at this point (when srpm is being built in build system), otherwise %{dynrel} would be always bumped?
– a build line is added to the detached changelog, using the current date and full evr (without %{dist} neutralization) — that probably requires defining a rpm variable to track whom the build is attributed to – it can default to "Anonymous coward" or whatever if not explicitely set
I think today's changelogs do not contain name of the person who built the pacakge (unless it is a mass rebuild), do you think something like this is needed? Usually a person responsible for release of the package (for the given "evr") is mentioned in the changelog record.
– Koji, OBS and Copr can set it to whoever asked them to build the package – the result constitutes the new srpm (either first or second level srpm as upstream rpm sees fit)
You mean there would be different kinds of built srpms? Or otherwise i don't under why upstream rpm (i understand it as https://github.com/rpm-software-management/rpm) should be involved here. What is meant by the "levels"?
– that’s generic non Fedora-specific behaviour that work as well in rpmbuild -bs, mock, copr, koji, OBS, whatever, with or without Fedora git presence – Koji/copr need to commit the new dynamic dynrel/changelog state to git (a build-striggered state change is commited by the build system)
For copr, it is not possible, because it has read-only access to git repositories being built.
And yes that requires some work rpm-side. That is necessary to maintain the rpm decentralized design and keep srpms independent from anyone’s git state. Personnaly, I don’t see the point of pretending we can move our infra forward without ever touching rpm.
I think there are good examples where some things can be done without rpm changes - e.g. the simple-koji-ci to do automatic scratch builds on a new commit.
The cardinal sin of Modularity was attempting to create an overlay over existing rpm/dnf behaviour, without changing this behaviour when it needed improving.
Contrat that with dynamic buildrequires or weak deps that slotted into our workflows with hardly any ripple. Though they were major feature changes. But, they were done with rpm upstream, instead of trying to bypass it.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le samedi 29 février 2020 à 13:49 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 10:15, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Hi,
Anyway, here is what I would personnaly consider a robust, inclusive, and future-proof design:
I will need to ask some questions to really understand it.
— a %{use_dynstate} rpm variable enables dynamic changelog/release behaviour — probably initialy set to false distro-wide, to let volunteers test the thing by setting it to true iin their specs, — then to true (opt-out), when kinks have been fixed, and FPC/FESCO greenlights general availability
– when activated, last changelog, evr and %{dynrel} state are saved in one or several detached files
You mean the last changelog, evr, and %{dynrel} are stored once %{use_dynstate} is set and and after one invokes fedpkg commit? I think there should be some explicit action (e.g. the commit) to generate the files after I set the %{use_dynstate} value to true in the spec file.
Once a spec sets %{use_dynstate} to true changelog, last computed changelog, ev, neutralized-r, and %{dynrel} are taken from detached files. neutralized-r is the evaluation of Release with %{dist} set to a neutral value (for example “-”).
State is stored in files included in the srpm to be able to import to and from Fedora git (pretty much a hard requirement if tooling is to be kept Fedora-git neutral, which is itself a hard requirement to be able to interact with packaging work existing outside Fedora git).
You need the last computed ev and neutralized-r to decide whether %{dynrel} can be kept, needs bumping, or should be reset to 1.
I don’t trust mixed mode, KISS, changelog is detached or not, don’t try to have it both ways, here lies madness and confusion.
How is %{dynrel} computed here at this stage
Nothing is computed at this stage.
Detached changelog state, is a commit property (except for build date). It is computed at fedpkg commit time: – if detached changelog data is absent → move in-spec changelog data to the detached file if data exists, → otherwise init detached data to nothing – after the first fedpkg commit with %{use_dynstate} set to true, the detached changelog file exists and the in-spec changelog has been removed. – others have detailed possible fedpkg commit strategies to stuff changelog with new info
%{dynrel} and build changelog info are a build property. They are computed at build time: – if computed ev ≠ last saved ev or last %{dynrel} is undefined → reset %{dynrel} to 1 – otherwise if computed neutralized-r = last neutralized-r → bump %{dynrel} – otherwise, do nothing, something already took care of things – save the new computed ev, computed neutralized-r and dynrel to the detached files – add a build line with the full evr (and full dist) to detached changelog – at build time, there is no relationship with magic git state or magic buildsys info, the srpms are 100% autonomous as before. The detached changelog has already been populated manually or by distro automation or manual processes when it reaches build stage and a new build is recorded (with a new dynrel value, if necessary).
Is the intention here that with each new build of the same package, the value of %{dynrel} is bumped and committed back?
It is bumped (or reseted to 1) whenever comparison with last saved state shows a bump or reset is needed.
Build-time state changes need to be commited back, of course (otherwise the produced srpm needs to be re-imported manually)
That means the evr file read from the repository needs to contain somehow outdated values at this point (when srpm is being built in build system), otherwise %{dynrel} would be always bumped?
Final dynrel state is not computed before a build, yes. The build will decide if it needs to bump dynrel, or can reset it to 1. I had forgotten to document the reset, sorry.
There is no need to compute a new dynrel before actual build, the packager may yet change ev or r.
– a build line is added to the detached changelog, using the current date and full evr (without %{dist} neutralization) — that probably requires defining a rpm variable to track whom the build is attributed to – it can default to "Anonymous coward" or whatever if not explicitely set
I think today's changelogs do not contain name of the person who built the pacakge
Today’s changelog mixes who made spec changes and who built the result. That what confuses you. Confusion is anthetical with automation. This proposal clearly documents that changelog can change at build time (fedpkg build) and between builds (fedpkg commit).
From a package consumer POW, the only attribution and timestamp that matters is who build the final package and when, because the builder is taking the responsability of pushing a result to users.
A clean changelog is:
%changelog * timestamp builder-id - built-EVR ← set at build time - change ← set at commit time - change ← set at commit time - change ← set at commit time
Not sure at all the change lines need attribution or timestamping. From a package user POW they are virtual potential changes that won’t materialize before a build.
So yes before someone presses the build button the detached changelog will look like this
%changelog
- staged change - staged change - staged change
* timestamp builder-id - built-EVR ← set at build time - past change - past change - past change
The act of building adds the final %changelog line: * timestamp builder-id - built-EVR
You mean there would be different kinds of built srpms?
We already have two level of srpms, one before dynamic BuildRequires processing and one after. It’s up to rpm upstream to decide at which srpm level they would prefer preforming dynrel and dynchangelog munging.
I want upstream rpm involved for dynrel and dynchangelog to be independent from current Fedora git infra, and interoperable.
For copr, it is not possible, because it has read-only access to git repositories being built.
It has some access to private git repos, you see copr initialising them when giving it a new srpm to build.
I think there are good examples where some things can be done without rpm changes - e.g. the simple-koji-ci to do automatic scratch builds on a new commit.
For something that is effectively a format change (moving changelog outside the spec itself) it’s a bad bad idea to do it without rpm upstream vetting and involvment. Sure you can kludge things without them. Operational word: kludge. Will forever be a pain point as long as it is not integrated upstream. Magic git-only or bbuildsys-only state is a misfeature, (s)rpms should cut the git/buildsys cordon as soon as they are born.
scratch build are something else entirely. They are transient non- shared things that do not need to cleanly export or import in other build systems
Regards,
Le samedi 29 février 2020 à 16:11 +0100, Nicolas Mailhot a écrit :
Build-time state changes need to be commited back, of course (otherwise the produced srpm needs to be re-imported manually)
Probably, only on *successful* production build however. We don’t need to record failed or scratch builds in our package changelogs
Regards,
On Sat, 29 Feb 2020 at 16:12, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 13:49 +0100, clime a écrit :
On Sat, 29 Feb 2020 at 10:15, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Hi,
Anyway, here is what I would personnaly consider a robust, inclusive, and future-proof design:
I will need to ask some questions to really understand it.
— a %{use_dynstate} rpm variable enables dynamic changelog/release behaviour — probably initialy set to false distro-wide, to let volunteers test the thing by setting it to true iin their specs, — then to true (opt-out), when kinks have been fixed, and FPC/FESCO greenlights general availability
– when activated, last changelog, evr and %{dynrel} state are saved in one or several detached files
You mean the last changelog, evr, and %{dynrel} are stored once %{use_dynstate} is set and and after one invokes fedpkg commit? I think there should be some explicit action (e.g. the commit) to generate the files after I set the %{use_dynstate} value to true in the spec file.
Once a spec sets %{use_dynstate} to true changelog, last computed changelog, ev, neutralized-r, and %{dynrel} are taken from detached files. neutralized-r is the evaluation of Release with %{dist} set to a neutral value (for example “-”).
State is stored in files included in the srpm to be able to import to and from Fedora git (pretty much a hard requirement if tooling is to be kept Fedora-git neutral, which is itself a hard requirement to be able to interact with packaging work existing outside Fedora git).
You need the last computed ev and neutralized-r to decide whether %{dynrel} can be kept, needs bumping, or should be reset to 1.
I don’t trust mixed mode, KISS, changelog is detached or not, don’t try to have it both ways, here lies madness and confusion.
How is %{dynrel} computed here at this stage
Nothing is computed at this stage.
Detached changelog state, is a commit property (except for build date). It is computed at fedpkg commit time: – if detached changelog data is absent → move in-spec changelog data to the detached file if data exists, → otherwise init detached data to nothing – after the first fedpkg commit with %{use_dynstate} set to true, the detached changelog file exists and the in-spec changelog has been removed. – others have detailed possible fedpkg commit strategies to stuff changelog with new info
%{dynrel} and build changelog info are a build property. They are computed at build time: – if computed ev ≠ last saved ev or last %{dynrel} is undefined → reset %{dynrel} to 1 – otherwise if computed neutralized-r = last neutralized-r → bump %{dynrel} – otherwise, do nothing, something already took care of things – save the new computed ev, computed neutralized-r and dynrel to the detached files – add a build line with the full evr (and full dist) to detached changelog – at build time, there is no relationship with magic git state or magic buildsys info, the srpms are 100% autonomous as before. The detached changelog has already been populated manually or by distro automation or manual processes when it reaches build stage and a new build is recorded (with a new dynrel value, if necessary).
Is the intention here that with each new build of the same package, the value of %{dynrel} is bumped and committed back?
It is bumped (or reseted to 1) whenever comparison with last saved state shows a bump or reset is needed.
Build-time state changes need to be commited back, of course (otherwise the produced srpm needs to be re-imported manually)
That means the evr file read from the repository needs to contain somehow outdated values at this point (when srpm is being built in build system), otherwise %{dynrel} would be always bumped?
Final dynrel state is not computed before a build, yes. The build will decide if it needs to bump dynrel, or can reset it to 1. I had forgotten to document the reset, sorry.
There is no need to compute a new dynrel before actual build, the packager may yet change ev or r.
– a build line is added to the detached changelog, using the current date and full evr (without %{dist} neutralization) — that probably requires defining a rpm variable to track whom the build is attributed to – it can default to "Anonymous coward" or whatever if not explicitely set
I think today's changelogs do not contain name of the person who built the pacakge
Today’s changelog mixes who made spec changes and who built the result. That what confuses you. Confusion is anthetical with automation. This proposal clearly documents that changelog can change at build time (fedpkg build) and between builds (fedpkg commit).
From a package consumer POW, the only attribution and timestamp that matters is who build the final package and when, because the builder is taking the responsability of pushing a result to users.
The person that was responsible for compiling some set of changes into the release and pushed those changes into src.fp.o carries responsibility too and I would say it is the person one should contact if something doesn't work because this person knows the most about the package. (I am assuming this person would be outsourcing the job of actually launching the build to someone else - that someone else doesn't need to know anything except of the commit id to build).
But okay, basically you are saying that for dynamic release, we don't need any git context and we just need a build-counter, which is being maintained by build system and committed back when incremented.
What about fedpkg local and fedpkg verrel then? Either, they will be giving you always "old" %{dynrel} because when you actually invoke fedpkg build you will get a bump on %{dynrel} or they will be giving you always the "future" value - i.e. they will always assume you are only about to call `fedpkg build`. That would work until you invoke fedpkg build. If you add new commit after wards and call `fedpkg local` or `fedpkg nvr`, you will be suddenly getting "old" %{dynrel} value in Release. I think it would be better from user standpoint of view if fedpkg verrel and fedpkg local give you always the correct NVR value - i.e. the exactly the value that you will get when you send the current commit into build system.
As for changelog, generally with build system commiting back to dist-git, there is a problem that I can be concurrently pushing changes while the build system tries to push its changes and build system is not human so it will not know how to resolve such situation. Do you have a solution for situation when I launch a build and then I do another change, commit, and push (the changes to changelog can be quite arbitrary here)? I guess it would require commit ids at places where the build info line should be inserted?
I think for simplicity, there should be only one owner of the changelog file, otherwise you need to solve synchronization issues but maybe you have some nice simple solution for this so that build system doesn't need to solve merge conflicts on its side...
A clean changelog is:
%changelog
- timestamp builder-id - built-EVR ← set at build time
- change ← set at commit time
- change ← set at commit time
- change ← set at commit time
Not sure at all the change lines need attribution or timestamping. From a package user POW they are virtual potential changes that won’t materialize before a build.
So yes before someone presses the build button the detached changelog will look like this
%changelog
- staged change
- staged change
- staged change
- timestamp builder-id - built-EVR ← set at build time
- past change
- past change
- past change
The act of building adds the final %changelog line:
- timestamp builder-id - built-EVR
You mean there would be different kinds of built srpms?
We already have two level of srpms, one before dynamic BuildRequires processing and one after. It’s up to rpm upstream to decide at which srpm level they would prefer preforming dynrel and dynchangelog munging.
I think, this is not a decision of rpm upstream but an infrastructure thing or a mock thing. It's true that mock is under in rpm-software-management group but under "rpm upstream" I really imagined this particular repository: https://github.com/rpm-software-management/rpm but you probably meant the whole rpm-software-management group.
I want upstream rpm involved for dynrel and dynchangelog to be independent from current Fedora git infra, and interoperable.
For copr, it is not possible, because it has read-only access to git repositories being built.
It has some access to private git repos, you see copr initialising them when giving it a new srpm to build.
True, but these repositories are read-only for outside world and it's a copr's implementation detail that they exist at this point. The source repositories are e.g. src.fp.o, pagure.io, github.com, bitbucket.org, etc.
I think your proposal wouldn't quite work for copr because it has no access to those repositories (which especially for src.fp.o is good in your proposal, otherwise copr would be modifying src.fp.o repos).
So If someone sets %{use_dynstate} in those external repos and uses %{dynrel} in Release, copr _can_ bump it in the way you described because the evr file and the file storing %{dynrel} (effectively build counter) can be put into those private repos.
So if someone wanted to have a build counter in the release, yes, this could be an implementation although it would be much easier to simply catch the information from copr's database where this info is stored instead of manipulating files in copr-dist-git to provide the next value of the build conter.
But with changelog, the situation is different because copr couldn't push the updated changelog with build info back to the source repo so in the source repo, there would be always some kind of incomplete changelog file ... in fact there would never be a changelog file at all because nobody uses fedpkg commit in git repos that are not from src.fp.o so this dynamic changelog thing wouldn't be usable for copr at all.
I think there are good examples where some things can be done without rpm changes - e.g. the simple-koji-ci to do automatic scratch builds on a new commit.
For something that is effectively a format change (moving changelog outside the spec itself) it’s a bad bad idea to do it without rpm upstream vetting and involvment. Sure you can kludge things without them. Operational word: kludge. Will forever be a pain point as long as it is not integrated upstream. Magic git-only or bbuildsys-only state is a misfeature, (s)rpms should cut the git/buildsys cordon as soon as they are born.
scratch build are something else entirely. They are transient non- shared things that do not need to cleanly export or import in other build systems
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
What about fedpkg local and fedpkg verrel then?
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build. Which is exactly what you want to make QA and testing workflows work. Those don’t live exclusively in koji.
And because only production builds get committed back the packager can change his mind and stage a few more changes before doing the production build, without polluting the changelog builds that were never pushed to users.
For fedpkg verrel you'll probably want to output a last (saved in detached file) and next line (probably factorizable by externalizing the dynrel bump logic in a separate command). That’s more honest anyway, next may happen or not, when you launch fedpkg verrel, it’s mere potential (the next commit may bump version and invalidate your future build).
As for changelog, generally with build system commiting back to dist-git, there is a problem that I can be concurrently pushing changes while the build system tries to push its changes and build system is not human so it will not know how to resolve such situation.
I understand, but that’s the consequence of automating changelog. Right now the reconciliation is done by the human packager (you can get in a similar situation by working on a change at the time a mass rebuild runs).
Do you have a solution for situation when I launch a build and then I do another change, commit, and push (the changes to changelog can be quite arbitrary here)?
You can set a lock at fedpkg build time, and forbid fedpkg commit in that branch till the lock is released (fedpkg build in the branch succeeded or not). The packager can still git commit things, as long as he does not touch the detached changelog file. Only fedpkg commit syncs git state with detached changelog state.
An alternate simpler strategy would be to mark the fedpkg build as failure in koji if sync-back failed. That would work too. The build system need not be ultra smart, just robust WRT human behaviour.
(If the packager deliberately messes with the detached changelog while a production build for the same branch is ongoing he deserves a build failure – the changelog state is undefined till the build ends, so he’s doing changes on quicksand. If he tried that today he’d probably have to rewrite his changelog if the build failed)
I think, this is not a decision of rpm upstream but an infrastructure thing or a mock thing.
mock and rpm upstreams excel when they work together. My premise is that they are better qualified than us to do rpm/buildsys boundary fine tuning.
Populating changelog from git and syncing back to fedora git are fedpkg/koji responsability, because Fedora git is Fedora specific infra. Handling release autobumping and build recording belongs to the lower layers, however. They’re the same with or without Fedora git, they belong to lower levels.
I think your proposal wouldn't quite work for copr because it has no access to those repositories (which especially for src.fp.o is good in your proposal, otherwise copr would be modifying src.fp.o repos).
copr does not need write access to Fedora git, its builds do not participate to the production build lineage as long as no one re- imports them in koji (which should be a deliberate fedpkg command, not something driven by copr itself).
copr only needs access to its own private git to remember its own last successful build, if people want it to autobump from last successful state. (maybe they don’t). srpm import in copr will overwrite copr state with the state the new srpm contains, which is fine too.
Putting state in detached srpm source files has the following super- duper property. You can import the srpm or scm checkout between systems, and they’ll just pick up from the point the previous system left things, without needing a full scm import. So you do not need to deal with incompatible scms (or lack of scm).
So if someone wanted to have a build counter in the release, yes, this could be an implementation although it would be much easier to simply catch the information from copr's database where this info is stored
It is simpler from the buildsys POW, but it ties state in a specific git and buildsystem. So it will break cross-buildsys workflows.
Cross-buildsys workflows are critical for the project success because they enable sharing work with other distros, and allow packagers to make the best of a palette of build systems (each with its own constrains and limitiations). Fun fact: I noticed today than one of my old Fedora packages was rebuilt by others for AIX. This kind of cross- pollination is one of the strengths of the rpm ecosystem. Don’t break it by making out packages depend on Fedora git or buildsys state.
in fact there would never be a changelog file at all because nobody uses fedpkg commit in git repos that are not from src.fp.o so this dynamic changelog thing wouldn't be usable for copr at all.
When using a buildsys that didn’t automate changelog filling, you can just write in the detached changelog file directly. The build part won’t care how you filled this file with changes, it just needs it to record a build point.
scm to changelog plumbing logic is going to change over time (for example, if we drop pagure for gitlab, or git for the next best thing). It is dangerous to set it in stone.
dynrel autobumping and recording builds in the changelog is more of an rpm format change. It can (and should) be done in an independant way.
Regards,
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
What about fedpkg local and fedpkg verrel then?
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
1) fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because you haven't built that package before) 2) fedpkg build 3) vim <somepkg>.spec and do some change in %description 4) fedpkg commit -m "description improvement" 5) fedpkg local -> produces <somepkg>-1.0-1.fc32 6) fedpkg push -> error because build system pushed meanwhile 7) fedpkg pull --rebase (to quickly fix the error) 8) fedpkg build -> builds <somepkg>-1.0-2.fc32
To get the correct NVR for the package being built by `fedpkg local` you need to wait for some external process to finish first so that you can pull the new state. Only then you get correct value again.
Which is exactly what you want to make QA and testing workflows work. Those don’t live exclusively in koji.
And because only production builds get committed back the packager can change his mind and stage a few more changes before doing the production build, without polluting the changelog builds that were never pushed to users.
That's a good thing, definitely.
For fedpkg verrel you'll probably want to output a last (saved in detached file) and next line (probably factorizable by externalizing the dynrel bump logic in a separate command). That’s more honest anyway, next may happen or not, when you launch fedpkg verrel, it’s mere potential (the next commit may bump version and invalidate your future build).
There is the same problem as above with `fedpkg local`, the command won't be giving you correct values at all times.
As for changelog, generally with build system commiting back to dist-git, there is a problem that I can be concurrently pushing changes while the build system tries to push its changes and build system is not human so it will not know how to resolve such situation.
I understand, but that’s the consequence of automating changelog. Right now the reconciliation is done by the human packager (you can get in a similar situation by working on a change at the time a mass rebuild runs).
It doesn't need to be the consequence of "automating changelog". It's only a consequence in certain implementations. If the changelog is derived purely from the git state, there aren't these problems anymore. The mass rebuilds are easier to handle than your case because with mass rebuilds, you always put the the new changelog record "on top" of other changelogs records. I.e. when you cannot push from build system (while peforming a mass rebuild), you can simply pull and try to do the same thing as before. But in your proposal it's not that easy (unless you do the locking or failing in that case as you described below) because you would need modify/insert into "middle" of the changelog at the right place.
Do you have a solution for situation when I launch a build and then I do another change, commit, and push (the changes to changelog can be quite arbitrary here)?
You can set a lock at fedpkg build time, and forbid fedpkg commit in that branch till the lock is released (fedpkg build in the branch succeeded or not). The packager can still git commit things, as long as he does not touch the detached changelog file. Only fedpkg commit syncs git state with detached changelog state.
The fact that "git commit" is different from "fedpkg commit" is not very convenient imho but could be probably get used to. But you generally don't except commit actions to change the repository content under your hands before actually committing. The locking mechanism you describe can be easily work-arounded by cloning the package repo again and doing your stuff. The only way to avoid it would be to implement locking server side. Either way it's not the best user experience.
An alternate simpler strategy would be to mark the fedpkg build as failure in koji if sync-back failed. That would work too. The build system need not be ultra smart, just robust WRT human behaviour.
I think not being able to work with dist-git while something is building its content is a throughput limitation and doesn't bring the best UX.
(If the packager deliberately messes with the detached changelog while a production build for the same branch is ongoing he deserves a build failure – the changelog state is undefined till the build ends, so he’s doing changes on quicksand. If he tried that today he’d probably have to rewrite his changelog if the build failed)
I think, this is not a decision of rpm upstream but an infrastructure thing or a mock thing.
mock and rpm upstreams excel when they work together. My premise is that they are better qualified than us to do rpm/buildsys boundary fine tuning.
Populating changelog from git and syncing back to fedora git are fedpkg/koji responsability, because Fedora git is Fedora specific infra. Handling release autobumping and build recording belongs to the lower layers, however. They’re the same with or without Fedora git, they belong to lower levels.
Yes, the updated %{dynrel} file and the updated changelog file can be build products either of mock or rpmbuild.
I think your proposal wouldn't quite work for copr because it has no access to those repositories (which especially for src.fp.o is good in your proposal, otherwise copr would be modifying src.fp.o repos).
copr does not need write access to Fedora git, its builds do not participate to the production build lineage as long as no one re- imports them in koji (which should be a deliberate fedpkg command, not something driven by copr itself).
copr only needs access to its own private git to remember its own last successful build, if people want it to autobump from last successful state. (maybe they don’t). srpm import in copr will overwrite copr state with the state the new srpm contains, which is fine too.
Putting state in detached srpm source files has the following super- duper property. You can import the srpm or scm checkout between systems, and they’ll just pick up from the point the previous system left things, without needing a full scm import.
That's nice if there is always just a single provider of the given package in our ecosystem. In case of copr, you can fork a certain repo and continue building there so you will be getting bumped numbers on top of the work done in the original repo. But once the original repo starts producing some builds as well you will be getting that package in two versions in two different coprs and while the original repo might have done less builds than the forked one since the fork happened, it can contain the package with more features because the forked repo was simply playing around.
I.e. as a user you might want to jump to the forked repo at some point because there is some activity and you can update to the package from that repo without any problems (because it builds upon the original package) but once the original repo wakes up, you might want to jump back and at that point, you need to downgrade. But this is a problem that will be always there, just mentioning that situation like this might happen. Not only NEVRAs play the role but their origin too.
I can see the advantage of your proposal that the auto-bumping theoretically doesn't need git to work. But it still needs git because you need push the updated %{dynstate} files somewhere. The only way to avoid it would be to build a new srpm together with the rpm where the srpm would be the same as the input srpm except for the updated %{dynstate} files. So in case we wouldn't have src.fp.o and the development would be fully decentralized, I could see your proposal to be actually the only way to implement auto-bumping and it would be a nice way. The thing is, the way we are developing today is different from that and hence different solutions might possibly be more suitable.
So you do not need to deal with incompatible scms (or lack of scm).
Even if you create certain dedicated macros relying on git metadata and put them into spec file, there should and will be a way to simply export srpm from the repo which no longer has any git dependencies. And then you can import that raw srpm anywhere else and continue bumping manually or start using macros that work with the new SCM or you can start using macros that don't rely on any SCM metadata but rely on manually crafted external changelog (specified without the headers but with new lines between separate changelog record groups), mtime of spec file (you couldn't manually edit patches without changing their names too in that case...but in the end, you could count with mtimes of patch files too), and name-email file stored alongside other files.
So if someone wanted to have a build counter in the release, yes, this could be an implementation although it would be much easier to simply catch the information from copr's database where this info is stored
It is simpler from the buildsys POW, but it ties state in a specific git and buildsystem. So it will break cross-buildsys workflows.
Cross-buildsys workflows are critical for the project success because they enable sharing work with other distros, and allow packagers to make the best of a palette of build systems (each with its own constrains and limitiations). Fun fact: I noticed today than one of my old Fedora packages was rebuilt by others for AIX. This kind of cross- pollination is one of the strengths of the rpm ecosystem. Don’t break it by making out packages depend on Fedora git or buildsys state.
Well, only automatic versioning would be dependent on Fedora git or buildsys state and as I tried to say earlier (downstream) versioning is not universal across the whole rpm universe. But even versioning wouldn't be actually dependent on anything because you always have a way to "export" things as they currently are and import them elsewhere (true, in case the release would be derived from some value stored in copr's db, you would need to manually set the latest dynamic release copr has produced into the spec file you are exporting or manually create the %{dynrel} file before importing it somewhere else).
in fact there would never be a changelog file at all because nobody uses fedpkg commit in git repos that are not from src.fp.o so this dynamic changelog thing wouldn't be usable for copr at all.
When using a buildsys that didn’t automate changelog filling, you can just write in the detached changelog file directly. The build part won’t care how you filled this file with changes, it just needs it to record a build point.
The point is that COPR could and potentially would like to implement the changelog filling but even if it does implement it, there is no way to push the resulting changelog back to the source repo, it could only push it back into its "private" repo but then user would need to manually take that file and put it into his/her source repo manually which is very inconvenient and it's not an automatic changelog generation one would wish for. So basically you are saying, such external repository can have automated changelog but it would need to be generated by some custom tooling and this changelog file would need to be complete with all the timestamps, emails, and verrels in headers. So In other words, for copr+external repo we would have something else than for koji+fedora dist-git.
scm to changelog plumbing logic is going to change over time (for example, if we drop pagure for gitlab, or git for the next best thing). It is dangerous to set it in stone.
It is, however, good to have it implemented only on one place in a way that it is easily reusable.
---
It's great that somebody can come up with such a different approach than what was originally suggested.
Best regards and thanks! clime
dynrel autobumping and recording builds in the changelog is more of an rpm format change. It can (and should) be done in an independant way.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
- fedpkg pull --rebase (to quickly fix the error)
- fedpkg build -> builds <somepkg>-1.0-2.fc32
To get the correct NVR for the package being built by `fedpkg local` you need to wait for some external process to finish first so that you can pull the new state. Only then you get correct value again.
Sure, that‘s the inconvenient of state: it needs syncing. And the whole auto-bump thing intrinsically depends on state not provided manually by human beings.
However putting state in git would be no better, because you’d end up assuming a build exists before it does, in fact, build (and builds do fail because packagers are imperfect humans).
And putting state in buildsys would also not be any better because successful production builds end up in changelogs and you want changelogs to be tracked.
So, no matter how you re-arange things, you’ll always have a buildsys sync-back somewhere (you have it today when humans change their changelogs after a failed build; automating that makes the sync-back explicit).
For fedpkg verrel you'll probably want to output a last (saved in detached file) and next line (probably factorizable by externalizing the dynrel bump logic in a separate command). That’s more honest anyway, next may happen or not, when you launch fedpkg verrel, it’s mere potential (the next commit may bump version and invalidate your future build).
There is the same problem as above with `fedpkg local`, the command won't be giving you correct values at all times.
There is no correct value to give before production build succeeded or not. When fedpkg local gives you something today it is lying (just try it just before someone mass rebuilds, and then try to commit after this mass rebuild: that will fail and you’ll have to merge/rebase).
You’re assuming a simple universe where you’re the only one doing things with a package. That’s not a true assumption, many packages have shared ownership and maintenance (to say nothing of proven-packagers and releng doing things on top of them).
It doesn't need to be the consequence of "automating changelog". It's only a consequence in certain implementations. If the changelog is derived purely from the git state, there aren't these problems anymore.
Sure there are. You're assuming anything committed to git builds before actually trying to build it. And expecting manual packager git fixup when that’s not the case. Just like in the sync-back state except with no tooling enforcement and help.
Putting state in detached srpm source files has the following super- duper property. You can import the srpm or scm checkout between systems, and they’ll just pick up from the point the previous system left things, without needing a full scm import.
That's nice if there is always just a single provider of the given package in our ecosystem. In case of copr, you can fork a certain repo and continue building there so you will be getting bumped numbers on top of the work done in the original repo. But once the original repo starts producing some builds as well you will be getting that package in two versions in two different coprs and while the original repo might have done less builds than the forked one since the fork happened, it can contain the package with more features because the forked repo was simply playing around.
You will never get official Fedora git releases changed based on the status of some copr builds. That takes a deliberate package admin action (and that works in the proposal, just fedpkg import the last srpm built in copr, that syncs repos).
If you want things to work on the other direction, copr will need to read state from Fedora infra one way or another, and it can perfectly read state from detached files committed back to Fedora git by koji. And then you need a second-level counter for copr, and configure copr for using the the second-level counter only. And then if you want OBS to build from copr that builds from koji…
I’m not sure multi-level bumping is in scope for the proposal. If it’s in scope that makes things more complex (but not a lot more complex). Multi-level definitely means the n-th level builder must be able to redefine Release structure, and will want the (n-1)th Release exposed as variable.
I can see the advantage of your proposal that the auto-bumping theoretically doesn't need git to work. But it still needs git because you need push the updated %{dynstate} files somewhere. The only way to avoid it would be to build a new srpm together with the rpm where the srpm would be the same as the input srpm except for the updated %{dynstate} files.
That’s why I wrote that the last-step of release bumping belongs in rpm proper, at srpm creation stage. Only way I see to get in independent from a specific project scm setup. The last build state must be provided in srpm data as detached file, and the bumping logic make decisions based on this data (we already put state in srpm for reproducible builds, IIRC).
So in case we wouldn't have src.fp.o and the development would be fully decentralized, I could see your proposal to be actually the only way to implement auto-bumping and it would be a nice way. The thing is, the way we are developing today is different from that
If you don’t keep things decentralized you’ll be in a word of pain when the scm or buildsys needs to be changed for another implementation (not to mention, that’s not a good way to collaborate with other distros). That will happen eventually. Web apps are not eternal.
Regards,
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git. The src.fp.o is a synchronization point of all developers. Now you are adding there also a build process with which all the developers need to synchronize. I think it is inconvenient.
- fedpkg pull --rebase (to quickly fix the error)
- fedpkg build -> builds <somepkg>-1.0-2.fc32
To get the correct NVR for the package being built by `fedpkg local` you need to wait for some external process to finish first so that you can pull the new state. Only then you get correct value again.
Sure, that‘s the inconvenient of state: it needs syncing. And the whole auto-bump thing intrinsically depends on state not provided manually by human beings.
However putting state in git would be no better, because you’d end up assuming a build exists before it does, in fact, build (and builds do fail because packagers are imperfect humans).
The only assumption is that a certain build will produce certain NEVRA if it succeeds.
And putting state in buildsys would also not be any better because successful production builds end up in changelogs and you want changelogs to be tracked.
So, no matter how you re-arange things, you’ll always have a buildsys sync-back somewhere (you have it today when humans change their changelogs after a failed build; automating that makes the sync-back explicit).
Scratch builds can help here a lot.
For fedpkg verrel you'll probably want to output a last (saved in detached file) and next line (probably factorizable by externalizing the dynrel bump logic in a separate command). That’s more honest anyway, next may happen or not, when you launch fedpkg verrel, it’s mere potential (the next commit may bump version and invalidate your future build).
There is the same problem as above with `fedpkg local`, the command won't be giving you correct values at all times.
There is no correct value to give before production build succeeded or not. When fedpkg local gives you something today it is lying (just try it just before someone mass rebuilds, and then try to commit after this mass rebuild: that will fail and you’ll have to merge/rebase).
It gives you something you can rely on if push succeeds.
You’re assuming a simple universe where you’re the only one doing things with a package. That’s not a true assumption, many packages have shared ownership and maintenance (to say nothing of proven-packagers and releng doing things on top of them).
There is only one person doing the actual build for a given release while multiple persons can participate in development.
It doesn't need to be the consequence of "automating changelog". It's only a consequence in certain implementations. If the changelog is derived purely from the git state, there aren't these problems anymore.
Sure there are. You're assuming anything committed to git builds before actually trying to build it. And expecting manual packager git fixup when that’s not the case. Just like in the sync-back state except with no tooling enforcement and help.
Who will be fixing the bugs it if not human?
Putting state in detached srpm source files has the following super- duper property. You can import the srpm or scm checkout between systems, and they’ll just pick up from the point the previous system left things, without needing a full scm import.
That's nice if there is always just a single provider of the given package in our ecosystem. In case of copr, you can fork a certain repo and continue building there so you will be getting bumped numbers on top of the work done in the original repo. But once the original repo starts producing some builds as well you will be getting that package in two versions in two different coprs and while the original repo might have done less builds than the forked one since the fork happened, it can contain the package with more features because the forked repo was simply playing around.
You will never get official Fedora git releases changed based on the status of some copr builds. That takes a deliberate package admin action (and that works in the proposal, just fedpkg import the last srpm built in copr, that syncs repos).
If you want things to work on the other direction, copr will need to read state from Fedora infra one way or another, and it can perfectly read state from detached files committed back to Fedora git by koji. And then you need a second-level counter for copr, and configure copr for using the the second-level counter only. And then if you want OBS to build from copr that builds from koji…
I’m not sure multi-level bumping is in scope for the proposal. If it’s in scope that makes things more complex (but not a lot more complex). Multi-level definitely means the n-th level builder must be able to redefine Release structure, and will want the (n-1)th Release exposed as variable.
I can see the advantage of your proposal that the auto-bumping theoretically doesn't need git to work. But it still needs git because you need push the updated %{dynstate} files somewhere. The only way to avoid it would be to build a new srpm together with the rpm where the srpm would be the same as the input srpm except for the updated %{dynstate} files.
That’s why I wrote that the last-step of release bumping belongs in rpm proper, at srpm creation stage. Only way I see to get in independent from a specific project scm setup. The last build state must be provided in srpm data as detached file, and the bumping logic make decisions based on this data (we already put state in srpm for reproducible builds, IIRC).
So in case we wouldn't have src.fp.o and the development would be fully decentralized, I could see your proposal to be actually the only way to implement auto-bumping and it would be a nice way. The thing is, the way we are developing today is different from that
If you don’t keep things decentralized you’ll be in a word of pain when the scm or buildsys needs to be changed for another implementation (not to mention, that’s not a good way to collaborate with other distros). That will happen eventually. Web apps are not eternal.
Full decentralization likely means that everyone has its own git repo and we can only sync by mailing list. I think src.fp.o is a good point where things can be done and where we can agree on what the packages that Fedora produce are (meaning we need a canonical source of package sources, otherwise it would be more complex to put a distribution together).
Best regards! clime
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Mar 2, 2020 at 8:46 AM clime clime@fedoraproject.org wrote:
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
If you don’t keep things decentralized you’ll be in a word of pain when the scm or buildsys needs to be changed for another implementation (not to mention, that’s not a good way to collaborate with other distros). That will happen eventually. Web apps are not eternal.
Full decentralization likely means that everyone has its own git repo and we can only sync by mailing list. I think src.fp.o is a good point where things can be done and where we can agree on what the packages that Fedora produce are (meaning we need a canonical source of package sources, otherwise it would be more complex to put a distribution together).
This is not true. Pagure accepts PRs from arbitrary Git servers just fine via the remote PR feature, so we do support decentralized workflows without resorting to sending patches via email or Bugzilla.
Ad. Zbyszek:
What about doing <name>-<version>-<dist>.<commits-since-version-bump>? This means that upgrade path not affected by the number of commits or builds in the older release.
I was thinking how to properly implement this into rpkg-util and then finally, it clicked for me.
First, I added the prefix parameter for git_release macro (below git_dir_release is used instead, which is the recommended form). Hence now, one can specify:
Release: {{{ git_dir_release prefix="0%{?dist}" }}}
which would produce release strings like:
0.fc32.1 0.fc32.2 0.fc32.3
for each tag in f32 branch. The leading zero is not used here for anything but without it, we would get NVRs like somepkg-1.0-.fc32.1 - i.e. dash would be followed by immediate dot, which is not actually invalid but it is strange.
Then it came to me that instead of %{dist} we can simply use branch name and hence (finally) drop "c" from .fcXY:
Release: {{{ git_dir_release prefix="$GIT_BRANCH" }}}
("$GIT_BRANCH" is a macro variable that gives you name of the currently checked out branch)
This will not work for cases when people put some special stuff into %{dist} like Nicolas showed: https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea... but it would work for simple cases and fallback would be possible.
Finally, one can just alias `git_dir_release prefix="$GIT_BRANCH"` with `git_dir_release_branched` (https://pagure.io/rpkg-util/blob/master/f/macros/macros.d/git.bash#_464) and hence get the following:
Release: {{{ git_dir_release_branched }}}
which will be bumping release with respect to the latest tag on the current branch as well as the commits stacked on top of that tag (it will be also bumping release for uncommitted work if your working tree is dirty but i don't want to show it here now because NVRs are then quite long).
I've prepared a test project for this new macro: https://pagure.io/hello_rpkg_release. You need the latest "rpkg" and "rpkg-macros" from https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ for it to work. But I also just dumped the results here: https://pagure.io/hello_rpkg_release/tree/result . You can see what NVRs are generated (by `rpkg nvr` command) at each particular point for the given branch (i.e. hello_rpkg_release-1.14.0-f32.2 for the second tagged commit in f32 branch) -- they are written down after the "-->" arrow. I mention there three NVRs in total:
hello_rpkg_release-1.14.0-master.1 (the first tagged commit master branch hello_rpkg_release-1.14.0-f32.2 (the second tagged commit in the f32 branch) hello_rpkg_release-1.14.0-f31.1.git.1.34684da9 (untagged commit on top of the first tagged commit in the f31 branch)
Any feedback welcome.
On Mon, 2 Mar 2020 at 14:51, Neal Gompa ngompa13@gmail.com wrote:
On Mon, Mar 2, 2020 at 8:46 AM clime clime@fedoraproject.org wrote:
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
If you don’t keep things decentralized you’ll be in a word of pain when the scm or buildsys needs to be changed for another implementation (not to mention, that’s not a good way to collaborate with other distros). That will happen eventually. Web apps are not eternal.
Full decentralization likely means that everyone has its own git repo and we can only sync by mailing list. I think src.fp.o is a good point where things can be done and where we can agree on what the packages that Fedora produce are (meaning we need a canonical source of package sources, otherwise it would be more complex to put a distribution together).
This is not true. Pagure accepts PRs from arbitrary Git servers just fine via the remote PR feature, so we do support decentralized workflows without resorting to sending patches via email or Bugzilla.
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, 3 Mar 2020 at 00:28, clime clime@fedoraproject.org wrote:
Ad. Zbyszek:
What about doing <name>-<version>-<dist>.<commits-since-version-bump>? This means that upgrade path not affected by the number of commits or builds in the older release.
I was thinking how to properly implement this into rpkg-util and then finally, it clicked for me.
First, I added the prefix parameter for git_release macro (below git_dir_release is used instead, which is the recommended form). Hence now, one can specify:
Release: {{{ git_dir_release prefix="0%{?dist}" }}}
which would produce release strings like:
0.fc32.1 0.fc32.2 0.fc32.3
Actually, that wouldn't work because prefix needs to be static, not dependent on rpm macros (we would be searching for tags that contain literally '0%{?dist}' in the last release part after the dash when generating the current release based on past tag names). Only the below versions that depend just on Git would work. I was hoping I can get a fallback for cases like Nicolas or the ruby packages have but that doesn't seem to be possible (at least to me at the moment).
for each tag in f32 branch. The leading zero is not used here for anything but without it, we would get NVRs like somepkg-1.0-.fc32.1 - i.e. dash would be followed by immediate dot, which is not actually invalid but it is strange.
Then it came to me that instead of %{dist} we can simply use branch name and hence (finally) drop "c" from .fcXY:
Release: {{{ git_dir_release prefix="$GIT_BRANCH" }}}
("$GIT_BRANCH" is a macro variable that gives you name of the currently checked out branch)
This will not work for cases when people put some special stuff into %{dist} like Nicolas showed: https://src.fedoraproject.org/rpms/fedora-release/blob/master/f/fedora-relea... but it would work for simple cases and fallback would be possible.
See above, regrettably.
Finally, one can just alias `git_dir_release prefix="$GIT_BRANCH"` with `git_dir_release_branched` (https://pagure.io/rpkg-util/blob/master/f/macros/macros.d/git.bash#_464) and hence get the following:
Release: {{{ git_dir_release_branched }}}
which will be bumping release with respect to the latest tag on the current branch as well as the commits stacked on top of that tag (it will be also bumping release for uncommitted work if your working tree is dirty but i don't want to show it here now because NVRs are then quite long).
I've prepared a test project for this new macro: https://pagure.io/hello_rpkg_release. You need the latest "rpkg" and "rpkg-macros" from https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ for it to work. But I also just dumped the results here: https://pagure.io/hello_rpkg_release/tree/result . You can see what NVRs are generated (by `rpkg nvr` command) at each particular point for the given branch (i.e. hello_rpkg_release-1.14.0-f32.2 for the second tagged commit in f32 branch) -- they are written down after the "-->" arrow. I mention there three NVRs in total:
hello_rpkg_release-1.14.0-master.1 (the first tagged commit master branch hello_rpkg_release-1.14.0-f32.2 (the second tagged commit in the f32 branch) hello_rpkg_release-1.14.0-f31.1.git.1.34684da9 (untagged commit on top of the first tagged commit in the f31 branch)
Any feedback welcome.
On Mon, 2 Mar 2020 at 14:51, Neal Gompa ngompa13@gmail.com wrote:
On Mon, Mar 2, 2020 at 8:46 AM clime clime@fedoraproject.org wrote:
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
If you don’t keep things decentralized you’ll be in a word of pain when the scm or buildsys needs to be changed for another implementation (not to mention, that’s not a good way to collaborate with other distros). That will happen eventually. Web apps are not eternal.
Full decentralization likely means that everyone has its own git repo and we can only sync by mailing list. I think src.fp.o is a good point where things can be done and where we can agree on what the packages that Fedora produce are (meaning we need a canonical source of package sources, otherwise it would be more complex to put a distribution together).
This is not true. Pagure accepts PRs from arbitrary Git servers just fine via the remote PR feature, so we do support decentralized workflows without resorting to sending patches via email or Bugzilla.
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2020-03-03 07:03, clime a écrit :
Actually, that wouldn't work because prefix needs to be static, not dependent on rpm macros
For myself, I would oppose any rpm release process that would not take core rpm mecanisms like macros into account.
Recording builds in changelogs without checking they actually happened is bad engineering.
Simulating rpm behaviour without performing actual spec evaluation in rpm, is also bad engineering. Especially when you know your simulation is horribly simplistic and approximative.
“Who cares if results match most of the time” is terrible workload optimization. You’ll make packagers waste far more time fixing the cases where automation guessed wrong, than you will win when it guesses right. Basic 80/20 rule, the 20% hard cases cost more than the 80% easy cases. Taking care of the 80% easy cases while making the 20% hard cases harder (due to automation mistakes) is not a good deal at all.
Please work on approaches which are reliable by default. Reliability is hard even when you aim for it. When you don’t, it’s not attainable at all.
Reagrds,
On Tue, 3 Mar 2020 at 09:38, Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le 2020-03-03 07:03, clime a écrit :
Actually, that wouldn't work because prefix needs to be static, not dependent on rpm macros
For myself, I would oppose any rpm release process that would not take core rpm mecanisms like macros into account.
Being independent of rpm has one huge potential advantage. You can apply the same approach that you apply for rpm spec files to other kinds of spec files. For example the fedora & containers mailing list contains a thread, when person needs to parametrize FROM clause by branch name to correctly reference base image: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
The approach I am suggesting would help there and it could help on other places too.
Recording builds in changelogs without checking they actually happened is bad engineering.
It's not about recording builds but instead about recording releases which I see as something that takes place already in Git.
Simulating rpm behaviour without performing actual spec evaluation in rpm, is also bad engineering. Especially when you know your simulation is horribly simplistic and approximative.
I am not trying to simulate rpm behavior. I am just trying to introduce macros that are rendered verbatim into the spec file inside srpm so those srpms can be rebuilt at any time when needed. rpm does not have this feature atm. If we should put it into rpm, it would be already third kind of rpm macro but even if it should go into rpm at some point, it is better to have the approach heavily tested by production use first because here it doesn't really add much complexity. And as I mentioned, independence of rpm means you can use it for non-rpm projects too.
It's also not simplistic: what you see in {{{ }}} tags is interpreted by bash, it can be multi-line and you can hence do anything there you can do with rpm macros. The only problem is, people already have some automation in rpm macros that doesn't require the git context, instead just e.g. the tarball content and this functionality would need to be rewritten by using {{{ }}} if it should play together with the git-based release generation. This cannot be avoided as much as I would like to. If we try to avoid it, we will get a weird hybrid that doesn't work.
It's also not approximative. You can define your own logic for generating release. There should be some pre-made recommended macros but in the end you are not limited by them.
“Who cares if results match most of the time” is terrible workload optimization. You’ll make packagers waste far more time fixing the cases where automation guessed wrong, than you will win when it guesses right. Basic 80/20 rule, the 20% hard cases cost more than the 80% easy cases. Taking care of the 80% easy cases while making the 20% hard cases harder (due to automation mistakes) is not a good deal at all.
Please work on approaches which are reliable by default. Reliability is hard even when you aim for it. When you don’t, it’s not attainable at all.
I think the approach I am suggesting is very reliable.
As much as I enjoy this conversation, I will need to actually try to make something happen. Forgive me if I don't go into long disputes hereafter. If you, however, make a detailed document summarizing the infra changes needed for your approach I will try to comment on the individual aspects of it and also respond with the document with the changes needed for what I am suggesting.
Best regards! clime
Reagrds,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le 2020-03-02 14:45, clime a écrit :
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git.
But fedpkg build is a release to production process. It’s not a dev staging process. In release processes, actually doing the release is not an inconvenient optional check.
You're asked to autobump a “Release” field. You’re asked to automate a changelog which is effectively a release changelog, not dev changelog (the dev changelog lives in git, people want a good changelog in rpm proper to track release not git history).
The *single* *only* reason good release numbers and good changelog matter is because the result will be released to third parties. koshei, or scratch builds, or whatever, do not care about those.
A normal git flow would lock the branch at release time to let the release manager do his releasing (merge window closed in Linus’s terms). That’s option 1 I gave you.
Alternatively, you can use option 2, the “I feel lucky” approach, let everything open, and abort the release if merge-back failed. A packager that changes his package before the production build is finished will probably want it canceled anyway (what is the point of risking breaking builds of other packages by pushing an unfinished package to the shared koji buildroot)?
Either option does not stop the packager to do anything he wants in his local branches. They just require him to merge/rebase when syncing with the centralized master Fedora state. That’s a core element of git architecture, be it in Fedora or elsewhere.
You can not avoid a buildsys merge-back when doing production builds. The merge-back can be explicit and automatic (as I proposed), or you can rely on informal meatware for it, but it will exist. Since the whole point of the request is to lower dependence on meatware, why on hell would you want to keep meatware as part of the whole process?
The only “cost” for the packager is to wait for his build to finish before continuing to change the package (or merge/rebase). That’s not an horrific cost. No one forced the packager to start a build before he was ready. No one forced the packager to perform a production build instead of a scratch build if he just wanted to check partial changes. If shit happens, and he realizes before the build end something is not ok, aborting the build on failed merge back is helping the packager.
Or am I missing something? “normally you don't need to anything like that when working with git” is not a clear technical point.
Regards,
On Tue, 3 Mar 2020 at 09:22, Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le 2020-03-02 14:45, clime a écrit :
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git.
But fedpkg build is a release to production process. It’s not a dev staging process. In release processes, actually doing the release is not an inconvenient optional check.
Well, you could have a situation when somebody wants to immediately continue working after making a release. Probably rare but could happen and if build time of a package is long (libreOffice/firefox), this could hit you as an inconvenience. Or do you want to push-back immediately after srpm build when you don't yet know whether the build will succeed? That wasn't clear to me before.
You're asked to autobump a “Release” field. You’re asked to automate a changelog which is effectively a release changelog, not dev changelog (the dev changelog lives in git, people want a good changelog in rpm proper to track release not git history).
I am not suggesting to use raw git commit messages for the changelog but instead content of annotated tags which can be initially prepopulated by commit messages but can be then edited to any extent and even the way the edit window for annotated tag is prepopulated might be configurable (to .e.g. load content from a file in which case you would probably skip the editing part). And I see the annotated tags as the actual releases.
From https://git-scm.com/docs/git-tag: Annotated tags are meant for release while lightweight tags are meant for private or temporary object labels.
They were designed to represent releases and because we are using git I don't see why we should avoid using annotated tags for releases.
On src.fp.o, there is also already support for this: e.g. https://src.fedoraproject.org/rpms/2ping/releases
The release needs building (koji), and distribution (bodhi + mirrors) but that's stuff that needs to be done with the release. Scratch builds can be used to make sure that the package builds and in the worst case, a new release needs to be done that accounts for changed build environment. But I don't think we need to throw the whole concept of annotated tags = releases because of it.
Of course, everything here can be made opt-in. As an approach somebody likes and somebody doesn't.
I think this is where our approaches really differ, you see release as something that happens somewhere else than in git, but I see the release as something that happens in git. And afterwards we need to get that released content to users.
I think we will never be able to reconciliate our views because they are different from the very base. In the end, what matters is implementation complexity and ease of use. I can present to you the things that need to be done for this to work in a detailed way (although it would take a bit of time to put into a document) but I also know it is a doable and it is actually fairly simple when not accounting for time spending on pull requests and getting them accepted. Your approach seems to be intuitively difficult to implement, either locking implementation (i am sure there would be quite unpleasant cases when the repo just stays locked even if it should be already unlocked, at least initially) or the fail after repo change - koji would need to listen for changes in dist-git and then abort builds based on that, which is doable but then you need authentication between the two systems (i.e koji needs to have write access to dist-git which is probably not the case today because the releng script to do mass rebuilds works separately and only invokes builds in koji) and you need some job that gets parsed changelog and NVR and does the pushing back to dist-git, which needs to be strictly separated from srpm/rpm builds due to security. Again, quite fairly complex to introduce something like that into a system like koji. Maybe you would like to bypass koji and create some external service that listens for dist-git and koji events and it is doing what you describe but introduction of this service is again quite a big task especially if one needs access to write to dist-git. And i think I would be able to find out more smaller or bigger obstacles when actually really trying to evaluate how difficult this would be.
The thing is that the I am not that much of a fan of "push into dist-git can make build fail" - it seems to me like something nobody expects so I don't want to be spending time figuring this out so that I can give you arguments that would be acceptable for you. If, however, you will spend time on "what piece of code need to be inserted where" in the infrastructure, I'll spend time on it as well for what I am proposing and we can evaluate what would be easier.
I am basically suggesting a small preprocessing step before sprm build. Cloning dist-git repo is already somewhere there in Koji, there just needs to be one more command invocation before everything continues the same way as before. No write access do dist-git needed which is already a big relief.
The *single* *only* reason good release numbers and good changelog matter is because the result will be released to third parties. koshei, or scratch builds, or whatever, do not care about those.
I agree they matter.
A normal git flow would lock the branch at release time to let the release manager do his releasing (merge window closed in Linus’s terms). That’s option 1 I gave you.
Alternatively, you can use option 2, the “I feel lucky” approach, let everything open, and abort the release if merge-back failed. A packager that changes his package before the production build is finished will probably want it canceled anyway (what is the point of risking breaking builds of other packages by pushing an unfinished package to the shared koji buildroot)?
Either option does not stop the packager to do anything he wants in his local branches. They just require him to merge/rebase when syncing with the centralized master Fedora state. That’s a core element of git architecture, be it in Fedora or elsewhere.
You can not avoid a buildsys merge-back when doing production builds.
Merge backs by build system can be avoided, however. Why do you think they can't be avoided?
There is the case of mass rebuilds but this is pretty much a one-shot event for each branch and it's not a merge-back we are talking about here....I would need to see the implementation today to be sure but in F12, it used be a separate script doing this: https://fedoraproject.org/wiki/Fedora_12_Mass_Rebuild
The merge-back can be explicit and automatic (as I proposed), or you can rely on informal meatware for it, but it will exist. Since the whole point of the request is to lower dependence on meatware, why on hell would you want to keep meatware as part of the whole process?
To me the whole point of the request is "make packager's life easier". If it goes hand-to-hand with automation (which it does by definition), then even better.
The only “cost” for the packager is to wait for his build to finish before continuing to change the package (or merge/rebase). That’s not an horrific cost. No one forced the packager to start a build before he was ready. No one forced the packager to perform a production build instead of a scratch build if he just wanted to check partial changes. If shit happens, and he realizes before the build end something is not ok, aborting the build on failed merge back is helping the packager.
Or am I missing something? “normally you don't need to anything like that when working with git” is not a clear technical point.
It's rather a point about what people are used to and what should be taken into account as well. If we want more projects on src.fp.o (wouldn't be bad, right?), then we shouldn't introduce an unexpected complexity that people would need to learn about in probably not that pleasant manner (unless they read dist-git documentation or there is some banner somewhere saying "don't push while building").
Best regards! clime
Regards,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le 2020-03-03 15:14, clime a écrit :
On Tue, 3 Mar 2020 at 09:22, Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le 2020-03-02 14:45, clime a écrit :
On Mon, 2 Mar 2020 at 12:05, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2020-03-01 02:31, clime a écrit :
On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
Putting %{dynrel} reconciliation in the rpmbuild -bs stage using detached file state means that fedpkg local (or checking out git state and building in mock or copr or OBS or via plain rpmbuild -bs) will give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
- fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before) 2) fedpkg build
At that point state is undefined till the build succeeds or not. If the build succeeds, the buildsystem will write back a new state. Let’s assume it succeeds.
It's undefined if you add more elements to the equation than necessary (i.e. build system), otherwise it would be well-defined in this case.
- vim <somepkg>.spec and do some change in %description
- fedpkg commit -m "description improvement"
- fedpkg local -> produces <somepkg>-1.0-1.fc32
- fedpkg push -> error because build system pushed meanwhile
Yes, here the packager notices something else has been going on, and he needs to merge or rebase. He’d have the same effect if another packager had been doing stuff on the package, or a mass rebuild had been going on. That’s the distributed decentralized aspect of git, except here the packager collided with himself by starting two work threads in parallel (one buildsys-side, one local).
The problem is that you launched some process and you need to wait until it finishes, normally you don't need to anything like that when working with git.
But fedpkg build is a release to production process. It’s not a dev staging process. In release processes, actually doing the release is not an inconvenient optional check.
Well, you could have a situation when somebody wants to immediately continue working after making a release. Probably rare but could happen and if build time of a package is long (libreOffice/firefox), this could hit you as an inconvenience. Or do you want to push-back immediately after srpm build when you don't yet know whether the build will succeed? That wasn't clear to me before.
Either you lock the centralized branch while the build is ongoing or you fail the build if someone pushes to the centralized branch before the end of the build.
That does not stop packagers from preparing the next stage in a local branch, only changing the state of the centralized branch at the same time the build process is changing it (two conflicting release state changes, option 1: buildsystem wins, option 2, packager wins, can’t have an heisenstate where things are both released and not released).
I am not suggesting to use raw git commit messages for the changelog but instead content of annotated tags which can be initially prepopulated by commit messages but can be then edited to any extent and even the way the edit window for annotated tag is prepopulated might be configurable (to .e.g. load content from a file in which case you would probably skip the editing part). And I see the annotated tags as the actual releases.
The only actual release is the package built in koji. So, no matter what mechanism you use git side: – you must represent the real state of package builds and not a git-only fiction → that means a form of buildsys write-back because builds can and will fail – you must produce something packagers can also build in mock or copr or OBS or whatever → that means checkout-ing changelog data in a form that can be read by rpmbuild -bs → ie a file not git metadata → with the associated conflict risk if multiple changes occur between checkouts
You can not avoid a buildsys merge-back when doing production builds.
Merge backs by build system can be avoided, however. Why do you think they can't be avoided?
Because they’re the real state rpm changelogs records. Because that’s what people use rpm changelogs for. Something broke, what is the affected packaged, what where the released package states since last time it worked.
There is the case of mass rebuilds but this is pretty much a one-shot event
Mass rebuilds are not an exception, they’re becoming the norm. Every SIG that deals with modern software released as a huge number of interlinked components needs to perform SIG-level mass rebuilds all the time (directly in rawhide, in a side tag, whatever). koshei can now autobuild all dependent packages when one of their requirements changed. That’s where the project is going and has been going for several years.
All the modularity efforts in the past years have been justified by the need to find a way to mass build things.
Regards,
Dne 27. 02. 20 v 16:11 Zbigniew Jędrzejewski-Szmek napsal(a):
Hi,
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
...
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
This means that a separate commit needs to be done *after* on top of the commits doing the actual changes.
It would not need any extra commit, if you modified manually the changelog with the last entry. Which should be not problem, considering that `fedpkg generate-changelog > changelog` would be executed locally and you can amend the last commit with the changelog.
Vít
It's a bit disappointing, but on its own would not be too bad, since we can have fedpkg provide a higher level command that combines generate-changelog and build...
One important feature will work: being able to cherry-pick commit between branches w/o spurious conflicts. As long as the "real" change to the spec file are done in separate commits, and the changelog commit is another commit on top, then when cherry-picking to another branch, the "real" commits would be cherry-pickable, and then the changelog commit would be recreated using the automation, OK.
But it doesn't work quite as nicely with something similar: merging branches. A simple 'git merge' would result in conflicts.
Also, if an amendment to the changelog is done, and the same change needs to be done in the changelog in a different branch, trying to cherry-pick the fix commit would most likely result in a merge conflict.
Considering these three drawbacks (separate commit step and resulting log noise, merge conflicts), I'd very much hope for a solution where the changelog is never stored in the version control, and is always autogenerated at srpm creation time. We should never try to chery-pick commits that have changelog entries with actual date or e-v-r texts, since this will inevitably lead to merge conflicts.
A different approach:
- we have 'fedpkg generate-changelog' (which does all the git log extraction that was described, I think that part is OK),
- the output from this command included in the srpm at srpm generation time and never committed to version control,
- the output is annotated with the source commits hashes, so we can see where each line came from.
At any time, the packager can run 'fedpkg generate-changelog' to see how the output looks like. If they see something they don't like, if the commits haven't been pushed out yet, they can immediately run 'git commit --amend' and recheck. If they have been pushed out, an override to the changelog could be committed as part of a commit message text.
Git-changelog-suppress: ad5555b42e or Git-changelog-suppress: ad5555b42e..efefedeadb or Git-changelog-replace: ad5555b42e Some other text with typos fixed that completely overrides whatever was generated from ad5555b42e. or Git-changelog-append: ad5555b42e (additional clarification for commit ad5555b42e, e.g. bug #12345)
This would have the following advantages:
- git log is the sole source of truth
- there are no cherry-pick or merge conflicts
- there is no separate "now I'm done and I need to do another commit before building" thing.
The assumption is that those Git-changelog-* macros would only be used occasionally, if the bad changelog entry was not noticed before pushing the changes out.
(One nitpick: when cherry-picking between branches, hashes of the cherry-picked commits change, so "ad5555b42e" in the example above would stop working. This is handled by using 'git cherry-pick -x'. 'fedpkg generate-changelog' would look at any hash in a "(cherry picked from commit ...)" line.)
As how to hook this up with rpm, looking at https://pagure.io/packaging-committee/pull-request/942#comment-108542, a macro like %generate_changelog could be provided that would simply shell out to 'fedpkg generate-changelog'.
I'll comment separate on the -release part.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Fri, Feb 28, 2020 at 12:31:23PM +0100, Vít Ondruch wrote:
Dne 27. 02. 20 v 16:11 Zbigniew Jędrzejewski-Szmek napsal(a):
Hi,
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
...
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
This means that a separate commit needs to be done *after* on top of the commits doing the actual changes.
It would not need any extra commit, if you modified manually the changelog with the last entry. Which should be not problem, considering that `fedpkg generate-changelog > changelog` would be executed locally and you can amend the last commit with the changelog.
Yes, but... Normally I'd do a cycle of "hack + build&test + commit". Now if I do changelog generation, I'd either have to repeat it after every step, or do it once at the end and then the last commit would suddenly contain a changelog update for multiple previous commits. Either way, this is not elegant and rather brittle.
Zbyszek
It's a bit disappointing, but on its own would not be too bad, since we can have fedpkg provide a higher level command that combines generate-changelog and build...
One important feature will work: being able to cherry-pick commit between branches w/o spurious conflicts. As long as the "real" change to the spec file are done in separate commits, and the changelog commit is another commit on top, then when cherry-picking to another branch, the "real" commits would be cherry-pickable, and then the changelog commit would be recreated using the automation, OK.
But it doesn't work quite as nicely with something similar: merging branches. A simple 'git merge' would result in conflicts.
Also, if an amendment to the changelog is done, and the same change needs to be done in the changelog in a different branch, trying to cherry-pick the fix commit would most likely result in a merge conflict.
Considering these three drawbacks (separate commit step and resulting log noise, merge conflicts), I'd very much hope for a solution where the changelog is never stored in the version control, and is always autogenerated at srpm creation time. We should never try to chery-pick commits that have changelog entries with actual date or e-v-r texts, since this will inevitably lead to merge conflicts.
A different approach:
- we have 'fedpkg generate-changelog' (which does all the git log extraction that was described, I think that part is OK),
- the output from this command included in the srpm at srpm generation time and never committed to version control,
- the output is annotated with the source commits hashes, so we can see where each line came from.
At any time, the packager can run 'fedpkg generate-changelog' to see how the output looks like. If they see something they don't like, if the commits haven't been pushed out yet, they can immediately run 'git commit --amend' and recheck. If they have been pushed out, an override to the changelog could be committed as part of a commit message text.
Git-changelog-suppress: ad5555b42e or Git-changelog-suppress: ad5555b42e..efefedeadb or Git-changelog-replace: ad5555b42e Some other text with typos fixed that completely overrides whatever was generated from ad5555b42e. or Git-changelog-append: ad5555b42e (additional clarification for commit ad5555b42e, e.g. bug #12345)
This would have the following advantages:
- git log is the sole source of truth
- there are no cherry-pick or merge conflicts
- there is no separate "now I'm done and I need to do another commit before building" thing.
The assumption is that those Git-changelog-* macros would only be used occasionally, if the bad changelog entry was not noticed before pushing the changes out.
(One nitpick: when cherry-picking between branches, hashes of the cherry-picked commits change, so "ad5555b42e" in the example above would stop working. This is handled by using 'git cherry-pick -x'. 'fedpkg generate-changelog' would look at any hash in a "(cherry picked from commit ...)" line.)
As how to hook this up with rpm, looking at https://pagure.io/packaging-committee/pull-request/942#comment-108542, a macro like %generate_changelog could be provided that would simply shell out to 'fedpkg generate-changelog'.
I'll comment separate on the -release part.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Dne 28. 02. 20 v 15:06 Zbigniew Jędrzejewski-Szmek napsal(a):
On Fri, Feb 28, 2020 at 12:31:23PM +0100, Vít Ondruch wrote:
Dne 27. 02. 20 v 16:11 Zbigniew Jędrzejewski-Szmek napsal(a):
Hi,
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
...
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
This means that a separate commit needs to be done *after* on top of the commits doing the actual changes.
It would not need any extra commit, if you modified manually the changelog with the last entry. Which should be not problem, considering that `fedpkg generate-changelog > changelog` would be executed locally and you can amend the last commit with the changelog.
Yes, but... Normally I'd do a cycle of "hack + build&test + commit". Now if I do changelog generation, I'd either have to repeat it after every step, or do it once at the end and then the last commit would suddenly contain a changelog update for multiple previous commits. Either way, this is not elegant and rather brittle.
But the point of automation is to not edit changelog at all, isn't it? If you are going to edit it, then the automation probably did not work good enough or you have pushed something you should not have.
Changelog is brittle even now. Everybody is free to edit changelog from release to release and there are no restrictions what can be done there. The situation with automation won't be worse then it is now.
Vít
Zbyszek
It's a bit disappointing, but on its own would not be too bad, since we can have fedpkg provide a higher level command that combines generate-changelog and build...
One important feature will work: being able to cherry-pick commit between branches w/o spurious conflicts. As long as the "real" change to the spec file are done in separate commits, and the changelog commit is another commit on top, then when cherry-picking to another branch, the "real" commits would be cherry-pickable, and then the changelog commit would be recreated using the automation, OK.
But it doesn't work quite as nicely with something similar: merging branches. A simple 'git merge' would result in conflicts.
Also, if an amendment to the changelog is done, and the same change needs to be done in the changelog in a different branch, trying to cherry-pick the fix commit would most likely result in a merge conflict.
Considering these three drawbacks (separate commit step and resulting log noise, merge conflicts), I'd very much hope for a solution where the changelog is never stored in the version control, and is always autogenerated at srpm creation time. We should never try to chery-pick commits that have changelog entries with actual date or e-v-r texts, since this will inevitably lead to merge conflicts.
A different approach:
- we have 'fedpkg generate-changelog' (which does all the git log extraction that was described, I think that part is OK),
- the output from this command included in the srpm at srpm generation time and never committed to version control,
- the output is annotated with the source commits hashes, so we can see where each line came from.
At any time, the packager can run 'fedpkg generate-changelog' to see how the output looks like. If they see something they don't like, if the commits haven't been pushed out yet, they can immediately run 'git commit --amend' and recheck. If they have been pushed out, an override to the changelog could be committed as part of a commit message text.
Git-changelog-suppress: ad5555b42e or Git-changelog-suppress: ad5555b42e..efefedeadb or Git-changelog-replace: ad5555b42e Some other text with typos fixed that completely overrides whatever was generated from ad5555b42e. or Git-changelog-append: ad5555b42e (additional clarification for commit ad5555b42e, e.g. bug #12345)
This would have the following advantages:
- git log is the sole source of truth
- there are no cherry-pick or merge conflicts
- there is no separate "now I'm done and I need to do another commit before building" thing.
The assumption is that those Git-changelog-* macros would only be used occasionally, if the bad changelog entry was not noticed before pushing the changes out.
(One nitpick: when cherry-picking between branches, hashes of the cherry-picked commits change, so "ad5555b42e" in the example above would stop working. This is handled by using 'git cherry-pick -x'. 'fedpkg generate-changelog' would look at any hash in a "(cherry picked from commit ...)" line.)
As how to hook this up with rpm, looking at https://pagure.io/packaging-committee/pull-request/942#comment-108542, a macro like %generate_changelog could be provided that would simply shell out to 'fedpkg generate-changelog'.
I'll comment separate on the -release part.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Fri, 28 Feb 2020 at 15:07, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Fri, Feb 28, 2020 at 12:31:23PM +0100, Vít Ondruch wrote:
Dne 27. 02. 20 v 16:11 Zbigniew Jędrzejewski-Szmek napsal(a):
Hi,
On Mon, Feb 24, 2020 at 05:48:36PM +0100, Pierre-Yves Chibon wrote:
For the changelog, we believe we have a potential good solution:
- The changelog will be automatically generated using an external `changelog` file as well as the commit history
...
If you wanted to edit the changelog, you would:
- Generate the changelog locally via a command like: `fedpkg generate-changelog > changelog`
- Edit `changelog` as desired
- Commit and push the changes
This means that a separate commit needs to be done *after* on top of the commits doing the actual changes.
It would not need any extra commit, if you modified manually the changelog with the last entry. Which should be not problem, considering that `fedpkg generate-changelog > changelog` would be executed locally and you can amend the last commit with the changelog.
Yes, but... Normally I'd do a cycle of "hack + build&test + commit". Now if I do changelog generation, I'd either have to repeat it after every step, or do it once at the end and then the last commit would suddenly contain a changelog update for multiple previous commits. Either way, this is not elegant and rather brittle.
I thought the main reason not to combine update in the changelog file with code commits is to avoid conflicts when cherry picking as you described.
I.e. i do minor update specifically in f32 and generate changelog file in the same commit. Then I want to do normal release update for all fedora branches. I start with master (as usual) and add e.g. add a new patch and generate the changelog file because i would like to add more info, then commit. Now I cannot cherry-pick that commit into f32 without conflict.
In this case we wouldn't achieve one the targets of this proposal (afaik) => getting rid of merge conflicts in changelog and release - this is cherry-picking but still it would be nice not to have conflicts there. This target isn't in the document i think but i thought this is one of the goals.
clime
Zbyszek
It's a bit disappointing, but on its own would not be too bad, since we can have fedpkg provide a higher level command that combines generate-changelog and build...
One important feature will work: being able to cherry-pick commit between branches w/o spurious conflicts. As long as the "real" change to the spec file are done in separate commits, and the changelog commit is another commit on top, then when cherry-picking to another branch, the "real" commits would be cherry-pickable, and then the changelog commit would be recreated using the automation, OK.
But it doesn't work quite as nicely with something similar: merging branches. A simple 'git merge' would result in conflicts.
Also, if an amendment to the changelog is done, and the same change needs to be done in the changelog in a different branch, trying to cherry-pick the fix commit would most likely result in a merge conflict.
Considering these three drawbacks (separate commit step and resulting log noise, merge conflicts), I'd very much hope for a solution where the changelog is never stored in the version control, and is always autogenerated at srpm creation time. We should never try to chery-pick commits that have changelog entries with actual date or e-v-r texts, since this will inevitably lead to merge conflicts.
A different approach:
- we have 'fedpkg generate-changelog' (which does all the git log extraction that was described, I think that part is OK),
- the output from this command included in the srpm at srpm generation time and never committed to version control,
- the output is annotated with the source commits hashes, so we can see where each line came from.
At any time, the packager can run 'fedpkg generate-changelog' to see how the output looks like. If they see something they don't like, if the commits haven't been pushed out yet, they can immediately run 'git commit --amend' and recheck. If they have been pushed out, an override to the changelog could be committed as part of a commit message text.
Git-changelog-suppress: ad5555b42e or Git-changelog-suppress: ad5555b42e..efefedeadb or Git-changelog-replace: ad5555b42e Some other text with typos fixed that completely overrides whatever was generated from ad5555b42e. or Git-changelog-append: ad5555b42e (additional clarification for commit ad5555b42e, e.g. bug #12345)
This would have the following advantages:
- git log is the sole source of truth
- there are no cherry-pick or merge conflicts
- there is no separate "now I'm done and I need to do another commit before building" thing.
The assumption is that those Git-changelog-* macros would only be used occasionally, if the bad changelog entry was not noticed before pushing the changes out.
(One nitpick: when cherry-picking between branches, hashes of the cherry-picked commits change, so "ad5555b42e" in the example above would stop working. This is handled by using 'git cherry-pick -x'. 'fedpkg generate-changelog' would look at any hash in a "(cherry picked from commit ...)" line.)
As how to hook this up with rpm, looking at https://pagure.io/packaging-committee/pull-request/942#comment-108542, a macro like %generate_changelog could be provided that would simply shell out to 'fedpkg generate-changelog'.
I'll comment separate on the -release part.
Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Fri, 2020-02-28 at 15:45 +0100, clime wrote:
I thought the main reason not to combine update in the changelog file with code commits is to avoid conflicts when cherry picking as you described.
I.e. i do minor update specifically in f32 and generate changelog file in the same commit. Then I want to do normal release update for all fedora branches. I start with master (as usual) and add e.g. add a new patch and generate the changelog file because i would like to add more info, then commit. Now I cannot cherry-pick that commit into f32 without conflict.
In this case we wouldn't achieve one the targets of this proposal (afaik) => getting rid of merge conflicts in changelog and release - this is cherry-picking but still it would be nice not to have conflicts there. This target isn't in the document i think but i thought this is one of the goals.
Leaving aside the advantages or disadvantages of a detached changelog (vs. in-spec vs. "everything from the commits"), our aim is to reduce the potential for conflicts between concurrent PRs or when cherry- picking between branches. If what we end up with can't avoid them completely for (what should be) corner cases, like when you have to manually fuzz with the generated changelog, that may not be ideal, but it's still a considerable improvement over the present situation.
Nils
On Mon, 2 Mar 2020 at 11:44, Nils Philippsen nils@redhat.com wrote:
On Fri, 2020-02-28 at 15:45 +0100, clime wrote:
I thought the main reason not to combine update in the changelog file with code commits is to avoid conflicts when cherry picking as you described.
I.e. i do minor update specifically in f32 and generate changelog file in the same commit. Then I want to do normal release update for all fedora branches. I start with master (as usual) and add e.g. add a new patch and generate the changelog file because i would like to add more info, then commit. Now I cannot cherry-pick that commit into f32 without conflict.
In this case we wouldn't achieve one the targets of this proposal (afaik) => getting rid of merge conflicts in changelog and release - this is cherry-picking but still it would be nice not to have conflicts there. This target isn't in the document i think but i thought this is one of the goals.
Leaving aside the advantages or disadvantages of a detached changelog (vs. in-spec vs. "everything from the commits"), our aim is to reduce the potential for conflicts between concurrent PRs or when cherry- picking between branches. If what we end up with can't avoid them completely for (what should be) corner cases, like when you have to manually fuzz with the generated changelog, that may not be ideal, but it's still a considerable improvement over the present situation.
Right, but if we can eliminate the problem completely why not do it?
clime
Nils
Nils Philippsen "Those who would give up Essential Liberty to Software Engineer purchase a little Temporary Safety, deserve neither Red Hat Liberty nor Safety." -- Benjamin Franklin, 1759 PGP fingerprint: D0C1 1576 CDA6 5B6E BBAE 95B2 7D53 7FCA E9F6 395D old: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-
Adam Saleh -
clime -
Dan Čermák -
David Kaufmann -
Fabio Valentini -
Igor Gnatenko -
James Cassell -
Matthew Miller -
Miro Hrončok -
Miroslav Suchý -
Neal Gompa -
Nicolas Mailhot -
Nils Philippsen -
Orion Poplawski -
Petr Pisar -
Pierre-Yves Chibon -
Remi Collet -
Richard W.M. Jones -
Robert-André Mauchin -
Stephen John Smoogen -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek