Hello Adam!
On Tue, 25 Feb 2020 at 08:58, Adam Saleh <asaleh(a)redhat.com> wrote:
Nice, I have been trying to fight through the 'git context already missing' with
pure lua rpm macros,
and so far was hitting walls left and right :-)
Will look at
https://pagure.io/rpkg-util, might have more questions :-)
You have probably already noticed...the docs at
https://docs.pagure.org/rpkg-util/index.html
are for rpkg-util version that is currently in Fedora but it contains
some nice introduction, nevertheless.
The (newer) git version from the pagure repository has some features
(e.g. git_release macro),
which are not yet documented there.
man page of the git version ($ man rpkg) is up to date, however.
Thank you for looking at it.
clime
>
> Adam
>
> On Tue, Feb 25, 2020 at 12:20 AM clime <clime(a)fedoraproject.org> wrote:
>>
>> Hello!
>>
>> On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon <pingou(a)pingoured.fr>
wrote:
>> >
>> > Good Morning Everyone,
>> >
>> > This topic has already been discussed a few times over the past month, but
Adam
>> > Saleh, Nils Philippsen and myself have had the opportunity to invest some
time
>> > on it with the hope of making the packager's life simpler as well as
making it
>> > easier to build automation around our package maintenance.
>> >
>> > We have investigated a few ideas, the full list with their pros and cons can
be
>> > found in this document:
https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev
>> > If you have any questions about some of these, please ask them, we'll be
happy
>> > to answer them and potentially complete this document.
>> >
>> >
>> > For both the release and the changelog fields we believe using RPM macros
would
>> > satisfy the requirements we have for opting-in/out:
>> > - You can easily opt-in by using the macros
>> > - You can easily opt-out by removing the macros from your spec file
>> >
>> >
>> > For the changelog, we believe we have a potential good solution:
>> > - The changelog will be automatically generated using an external
`changelog`
>> > file as well as the commit history
>> > - When you opt-in, you will simply move the existing changelog to an
external
>> > file `changelog`, which is placed in the dist-git repo and add the
>> > appropriate macro.
>> > - Upon building, the macro will generate the changelog using all the
commits
>> > of the repo up to the last commit touching the `changelog` file. Of all
>> > these commits it will only consider the commits following these rules:
>> > - There are generally two approaches regarding what to include by
default:
>> > 1. commits touching only the `sources`, `.gitignore`, `dead.package`
>> > files, `tests` folder will be ignored by default, i.e. a blacklist
>> > 2. only commits touching the spec file or patches will be included by
>> > default, i.e. a whitelist
>> > ==> Which approach do you think is better/will work in most cases?
>> > - empty commits (not touching any files) will be included on the
assumption
>> > that this is their purpose
>> > - commits containing "magic keyword" (#changelog_exclude,
>> > #changelog_include?) will be ignored or included as the case may be
>> > - Finally the content of the changelog file specified will be appended to
the
>> > changelog generated from the git history
>> >
>> >
>> > If you wanted to edit the changelog, you would:
>> > - Generate the changelog locally via a command like:
>> > `fedpkg generate-changelog > changelog`
>> > - Edit `changelog` as desired
>> > - Commit and push the changes
>> >
>> > Since the macro will only consider the commits up to the first commit
touching
>> > `changelog` (in other words, the macro will only consider the commits after
this
>> > one) and include `changelog` file as is, your edits will appear in the RPM
>> > changelog as you want.
>> >
>> > One thing to note is that rebuilds from the same commit will have the same
>> > %changelog, they will not get a new entry. If you want a new changelog
entry,
>> > you have to create a new commit with the desired changelog entry as commit
>> > message (the commit itself can be empty).
>> >
>> >
>> >
>> > However, for the release field, we are struggling a little bit more, two
options
>> > are more appealing to us:
>> >
>> > A) The release field is automatically generated using two elements:
>> > - the number of commits at this version
>> > - the number of builds at this commit
>> > - the dist-tag being added after them
>> > The release field would thus look like:
>> > ``<number of commit at version>.<number of build at
commit>%{dist}``
>> >
>> > So we could have: (A, B, C and D being different commits)
>> > A -- version: 0.9 -- release: ?
>> > |
>> > B -- version: 1.0 -- release: 1.0
>> > |
>> > C -- version: 1.0 -- release: 2.0
>> > |
>> > D -- version: 1.1 -- release: 1.0
>> >
>> > A rebuild of the commit D, would lead to a release 1.1 (assuming the first
one
>> > succeeded)/
>> >
>> > Pros:
>> > - Easy to replicate locally
>> > - Every changelog entry can be mapped to a `version-release` (No changes to
the
>> > packaging guidelines)
>> > - Allows triggering two builds from the same commit without any manual
change
>> > (simplifies mass-rebuilds)
>> > - Easy to link a certain build with a specific commit
>> > - Easy to “guess” the next release value before triggering the build
>> >
>> > Cons:
>> > - Old packages which are no longer receiving upstream releases may need
>> > special care (for example if they are at the release -48 but only had
45
>> > commits leading to this)
>> > - Relies on information from the build system for the build number (but
can
>> > be closely simulated locally since the <n_build> info is only
used for
>> > rebuilds)
>> > - Upgrade path may be problematic if Fn is upgraded to version X with
one
>> > commit while Fn-1 is upgrade to version X with 2 commits (or more)
>> >
>> >
>> > B) The release field is automatically generated taking existing builds in
all
>> > current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means
for
>> > builds of the same epoch:version, find a new release that (if at all
possible)
>> > ensures upgradability from older Fedora versions to newer ones, adhering to
the
>> > structure of a release tag documented in the Versioning Guidelines[1].
Going
>> > from the (RPM-wise) "latest build" that the new one should
surpass, this can
>> > mean bumping in the front (`pkgrel`) or the back (`minorbump`).
>> >
>> > This allows packages from "very stable" upstreams who haven't
released in years
>> > to still benefit from automatically generated releases.
>> >
>> > The following examples would use a macro for the release field as outlined
in a
>> > separate document[2].
>> >
>> > Example #1 ("normal" release progression):
>> > Rawhide has: 2.0-1.fc32
>> > F31 has: 1.0-1.fc31
>> > F30 has: 1.0-1.fc30
>> >
>> > Next release in F31: 1.0-2.fc32
>> >
>> >
>> > Example #2 ("hotfix" in an older release, selected by an
alternative macro (or
>> > option) in the spec file):
>> > Rawhide has: 2.0-1.fc32
>> > F31 has: 1.0-1.fc31
>> > F30 has: 1.0-1.fc30
>> >
>> > Next release in F30: 1.0-1.fc30.1
>> >
>> > Example #3 (pre-release, selected by an alternative macro (or option) in
the
>> > spec file):
>> > Rawhide has: 2.0-1.fc32
>> > F31 has: 1.0-1.fc31
>> > F30 has: 1.0-1.fc30
>> >
>> > Next release in Rawhide: 3.0-0.1.20200224git1234abcd
>> >
>> >
>> > Pros:
>> > - Allows triggering two builds from the same commit without manual
>> > intervention
>> > - Emulates what many maintainers do manually today for most use cases
>> > - Can cater for pre-releases (e.g.: by offering different macros or macro
>> > options for the different use cases)
>> >
>> > Cons:
>> > - Needs the build system for information about builds in this and other
Fedora
>> > versions
>> > - Not easy to reproduce locally because we don’t have machine-consumable
>> > knowledge about other builds in e.g. the dist-git repo
>> > - Does not allow to generate changelog entries with `version-release`
>> > information for all commits (and this will require a change in our
packaging
>> > guidelines)
>> >
>> >
>> > So these are the results of our current investigations, we are very much
eager
>> > to get your feedback on them and even more eager if you have ideas on how
to
>> > approach/solve some of the challenges mentioned here.
>> >
>> >
>> > Looking forward for the discussion,
>> >
>> > Pierre
>> > For Adam, Nils and myself
>>
>> What is the point of including number of builds into release? I think
>> the Miro's approach solves it.
>> Or is there any other problem except soname bumps?
>>
>> Ad. document - annotated git tags:
>> (-) Editing the changelog would mean allowing to remove the git tags,
>> thus leading to potential issue in build reproducibility
>>
>> That doesn't need to be the case. In rpkg-util, this was resolved by
>> introducing arguments since_tag and until_tag
>> for git_changelog macro
>> (
https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros).
>> That's
>> how it can be prevented for some annotated tag to contribute to changelog.
>>
>> Example:
>>
>> {{{ git_changelog since_tag=name-1.3-1 }}}
>>
>> * Mon Feb 24 2020 clime <clime(a)fedoraproject.org> 1.2-1
>> - manual changelog entry that is used instead of a tag annotation
>>
>> {{{ git_changelog until_tag=name-1.1-1 }}}
>>
>> Removing already pushed git tags is probably not a good idea anyway:
>>
https://git-scm.com/docs/git-tag#_on_re_tagging
>>
>> Ad. the following approach for calculating release:
>> - Compute the release field from the number of commits since the last
>> version change: <version>-<commits_number>%{dist}
>>
>> I think it is a good idea. In rpkg-util, it is done similarly, except
>> the release part has more subparts than just
>> two (including %{dist}).
>>
>> The full description is here:
>>
https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262
>> but the main difference is that it counts commits from the latest
>> annotated tag which contains (in its name)
>> the current version and the current release number which are extracted
>> from the spec file when
>> creating the tag unless they are specified manually on command line.
>> Commit count is only appended
>> to it if we build from commit which is on top of some annotated tag
>> (i.e. it is itself untagged).
>>
>> Going by just a textual version change in a spec file might be slightly tricky.
>> You would need to go through all the past commits that touched that spec file,
>> keep checking these out and look if the version is changed when compared to the
>> one parsed from the head commit. Possible though.
>>
>> To go back to your original post:
>> > For both the release and the changelog fields we believe using RPM macros
would
>> > satisfy the requirements we have for opting-in/out:
>>
>> By using such RPM macros, you would lose ability to rebuild srpms
>> because it will
>> be only possible to build them when the git context is present but not when they
>> become a standalone thing. I.e. things like this will not work:
>>
>>
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/m...
>>
>> That's why I think that such macros should be of a different kind:
>> macros that are computed
>> once when srpm is created and result of which is put _verbatim_ into
>> the spec file so that
>> there is no (re)computation later when srpm is being built and when
>> the git context is already
>> missing.
>>
>> This approach is taken in the rpkg-util project
>> (
https://pagure.io/rpkg-util). I really
>> recommend looking at it as I spent more than a year solving this
>> particular problem with
>> changelog and release (but actually not only that problem) and I also
>> optimized the macros there
>> as much as I possibly could.
>>
>> If you want to play around with it, you can download the latest
>> version from here:
>>
>>
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
>>
>> and try it on this:
>>
>>
https://pagure.io/hello_rpkg
>>
>> clime
>>
>> >
>> >
>> > [1]:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/
>> > [2]:
https://hackmd.io/kuXOPe74RfepuztBz7pBsg
>> > _______________________________________________
>> > devel mailing list -- devel(a)lists.fedoraproject.org
>> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org