On Tue, 25 Feb 2020 at 09:31, clime <clime(a)fedoraproject.org> wrote:
Hello Adam!
On Tue, 25 Feb 2020 at 08:58, Adam Saleh <asaleh(a)redhat.com> wrote:
>
> Nice, I have been trying to fight through the 'git context already missing'
with pure lua rpm macros,
> and so far was hitting walls left and right :-)
>
> Will look at
https://pagure.io/rpkg-util, might have more questions :-)
You have probably already noticed...the docs at
https://docs.pagure.org/rpkg-util/index.html
are for rpkg-util version that is currently in Fedora but it contains
some nice introduction, nevertheless.
The (newer) git version from the pagure repository has some features
(e.g. git_release macro),
which are not yet documented there.
man page of the git version ($ man rpkg) is up to date, however.
Thank you for looking at it.
clime
...in case of any questions i am on this email or on #fedora-apps. Again,
thanks for looking at it. Cheers.
>
> >
> > Adam
> >
> > On Tue, Feb 25, 2020 at 12:20 AM clime <clime(a)fedoraproject.org> wrote:
> >>
> >> Hello!
> >>
> >> On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon <pingou(a)pingoured.fr>
wrote:
> >> >
> >> > Good Morning Everyone,
> >> >
> >> > This topic has already been discussed a few times over the past month,
but Adam
> >> > Saleh, Nils Philippsen and myself have had the opportunity to invest
some time
> >> > on it with the hope of making the packager's life simpler as well
as making it
> >> > easier to build automation around our package maintenance.
> >> >
> >> > We have investigated a few ideas, the full list with their pros and
cons can be
> >> > found in this document:
https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev
> >> > If you have any questions about some of these, please ask them,
we'll be happy
> >> > to answer them and potentially complete this document.
> >> >
> >> >
> >> > For both the release and the changelog fields we believe using RPM
macros would
> >> > satisfy the requirements we have for opting-in/out:
> >> > - You can easily opt-in by using the macros
> >> > - You can easily opt-out by removing the macros from your spec file
> >> >
> >> >
> >> > For the changelog, we believe we have a potential good solution:
> >> > - The changelog will be automatically generated using an external
`changelog`
> >> > file as well as the commit history
> >> > - When you opt-in, you will simply move the existing changelog to an
external
> >> > file `changelog`, which is placed in the dist-git repo and add the
> >> > appropriate macro.
> >> > - Upon building, the macro will generate the changelog using all the
commits
> >> > of the repo up to the last commit touching the `changelog` file. Of
all
> >> > these commits it will only consider the commits following these
rules:
> >> > - There are generally two approaches regarding what to include by
default:
> >> > 1. commits touching only the `sources`, `.gitignore`,
`dead.package`
> >> > files, `tests` folder will be ignored by default, i.e. a
blacklist
> >> > 2. only commits touching the spec file or patches will be
included by
> >> > default, i.e. a whitelist
> >> > ==> Which approach do you think is better/will work in most
cases?
> >> > - empty commits (not touching any files) will be included on the
assumption
> >> > that this is their purpose
> >> > - commits containing "magic keyword"
(#changelog_exclude,
> >> > #changelog_include?) will be ignored or included as the case may
be
> >> > - Finally the content of the changelog file specified will be
appended to the
> >> > changelog generated from the git history
> >> >
> >> >
> >> > If you wanted to edit the changelog, you would:
> >> > - Generate the changelog locally via a command like:
> >> > `fedpkg generate-changelog > changelog`
> >> > - Edit `changelog` as desired
> >> > - Commit and push the changes
> >> >
> >> > Since the macro will only consider the commits up to the first commit
touching
> >> > `changelog` (in other words, the macro will only consider the commits
after this
> >> > one) and include `changelog` file as is, your edits will appear in the
RPM
> >> > changelog as you want.
> >> >
> >> > One thing to note is that rebuilds from the same commit will have the
same
> >> > %changelog, they will not get a new entry. If you want a new changelog
entry,
> >> > you have to create a new commit with the desired changelog entry as
commit
> >> > message (the commit itself can be empty).
> >> >
> >> >
> >> >
> >> > However, for the release field, we are struggling a little bit more,
two options
> >> > are more appealing to us:
> >> >
> >> > A) The release field is automatically generated using two elements:
> >> > - the number of commits at this version
> >> > - the number of builds at this commit
> >> > - the dist-tag being added after them
> >> > The release field would thus look like:
> >> > ``<number of commit at version>.<number of build at
commit>%{dist}``
> >> >
> >> > So we could have: (A, B, C and D being different commits)
> >> > A -- version: 0.9 -- release: ?
> >> > |
> >> > B -- version: 1.0 -- release: 1.0
> >> > |
> >> > C -- version: 1.0 -- release: 2.0
> >> > |
> >> > D -- version: 1.1 -- release: 1.0
> >> >
> >> > A rebuild of the commit D, would lead to a release 1.1 (assuming the
first one
> >> > succeeded)/
> >> >
> >> > Pros:
> >> > - Easy to replicate locally
> >> > - Every changelog entry can be mapped to a `version-release` (No
changes to the
> >> > packaging guidelines)
> >> > - Allows triggering two builds from the same commit without any manual
change
> >> > (simplifies mass-rebuilds)
> >> > - Easy to link a certain build with a specific commit
> >> > - Easy to “guess” the next release value before triggering the build
> >> >
> >> > Cons:
> >> > - Old packages which are no longer receiving upstream releases may
need
> >> > special care (for example if they are at the release -48 but only
had 45
> >> > commits leading to this)
> >> > - Relies on information from the build system for the build number
(but can
> >> > be closely simulated locally since the <n_build> info is
only used for
> >> > rebuilds)
> >> > - Upgrade path may be problematic if Fn is upgraded to version X
with one
> >> > commit while Fn-1 is upgrade to version X with 2 commits (or
more)
> >> >
> >> >
> >> > B) The release field is automatically generated taking existing builds
in all
> >> > current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This
means for
> >> > builds of the same epoch:version, find a new release that (if at all
possible)
> >> > ensures upgradability from older Fedora versions to newer ones,
adhering to the
> >> > structure of a release tag documented in the Versioning Guidelines[1].
Going
> >> > from the (RPM-wise) "latest build" that the new one should
surpass, this can
> >> > mean bumping in the front (`pkgrel`) or the back (`minorbump`).
> >> >
> >> > This allows packages from "very stable" upstreams who
haven't released in years
> >> > to still benefit from automatically generated releases.
> >> >
> >> > The following examples would use a macro for the release field as
outlined in a
> >> > separate document[2].
> >> >
> >> > Example #1 ("normal" release progression):
> >> > Rawhide has: 2.0-1.fc32
> >> > F31 has: 1.0-1.fc31
> >> > F30 has: 1.0-1.fc30
> >> >
> >> > Next release in F31: 1.0-2.fc32
> >> >
> >> >
> >> > Example #2 ("hotfix" in an older release, selected by an
alternative macro (or
> >> > option) in the spec file):
> >> > Rawhide has: 2.0-1.fc32
> >> > F31 has: 1.0-1.fc31
> >> > F30 has: 1.0-1.fc30
> >> >
> >> > Next release in F30: 1.0-1.fc30.1
> >> >
> >> > Example #3 (pre-release, selected by an alternative macro (or option)
in the
> >> > spec file):
> >> > Rawhide has: 2.0-1.fc32
> >> > F31 has: 1.0-1.fc31
> >> > F30 has: 1.0-1.fc30
> >> >
> >> > Next release in Rawhide: 3.0-0.1.20200224git1234abcd
> >> >
> >> >
> >> > Pros:
> >> > - Allows triggering two builds from the same commit without manual
> >> > intervention
> >> > - Emulates what many maintainers do manually today for most use
cases
> >> > - Can cater for pre-releases (e.g.: by offering different macros or
macro
> >> > options for the different use cases)
> >> >
> >> > Cons:
> >> > - Needs the build system for information about builds in this and
other Fedora
> >> > versions
> >> > - Not easy to reproduce locally because we don’t have
machine-consumable
> >> > knowledge about other builds in e.g. the dist-git repo
> >> > - Does not allow to generate changelog entries with
`version-release`
> >> > information for all commits (and this will require a change in our
packaging
> >> > guidelines)
> >> >
> >> >
> >> > So these are the results of our current investigations, we are very
much eager
> >> > to get your feedback on them and even more eager if you have ideas on
how to
> >> > approach/solve some of the challenges mentioned here.
> >> >
> >> >
> >> > Looking forward for the discussion,
> >> >
> >> > Pierre
> >> > For Adam, Nils and myself
> >>
> >> What is the point of including number of builds into release? I think
> >> the Miro's approach solves it.
> >> Or is there any other problem except soname bumps?
> >>
> >> Ad. document - annotated git tags:
> >> (-) Editing the changelog would mean allowing to remove the git tags,
> >> thus leading to potential issue in build reproducibility
> >>
> >> That doesn't need to be the case. In rpkg-util, this was resolved by
> >> introducing arguments since_tag and until_tag
> >> for git_changelog macro
> >> (
https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros).
> >> That's
> >> how it can be prevented for some annotated tag to contribute to changelog.
> >>
> >> Example:
> >>
> >> {{{ git_changelog since_tag=name-1.3-1 }}}
> >>
> >> * Mon Feb 24 2020 clime <clime(a)fedoraproject.org> 1.2-1
> >> - manual changelog entry that is used instead of a tag annotation
> >>
> >> {{{ git_changelog until_tag=name-1.1-1 }}}
> >>
> >> Removing already pushed git tags is probably not a good idea anyway:
> >>
https://git-scm.com/docs/git-tag#_on_re_tagging
> >>
> >> Ad. the following approach for calculating release:
> >> - Compute the release field from the number of commits since the last
> >> version change: <version>-<commits_number>%{dist}
> >>
> >> I think it is a good idea. In rpkg-util, it is done similarly, except
> >> the release part has more subparts than just
> >> two (including %{dist}).
> >>
> >> The full description is here:
> >>
https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262
> >> but the main difference is that it counts commits from the latest
> >> annotated tag which contains (in its name)
> >> the current version and the current release number which are extracted
> >> from the spec file when
> >> creating the tag unless they are specified manually on command line.
> >> Commit count is only appended
> >> to it if we build from commit which is on top of some annotated tag
> >> (i.e. it is itself untagged).
> >>
> >> Going by just a textual version change in a spec file might be slightly
tricky.
> >> You would need to go through all the past commits that touched that spec
file,
> >> keep checking these out and look if the version is changed when compared to
the
> >> one parsed from the head commit. Possible though.
> >>
> >> To go back to your original post:
> >> > For both the release and the changelog fields we believe using RPM
macros would
> >> > satisfy the requirements we have for opting-in/out:
> >>
> >> By using such RPM macros, you would lose ability to rebuild srpms
> >> because it will
> >> be only possible to build them when the git context is present but not when
they
> >> become a standalone thing. I.e. things like this will not work:
> >>
> >>
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/m...
> >>
> >> That's why I think that such macros should be of a different kind:
> >> macros that are computed
> >> once when srpm is created and result of which is put _verbatim_ into
> >> the spec file so that
> >> there is no (re)computation later when srpm is being built and when
> >> the git context is already
> >> missing.
> >>
> >> This approach is taken in the rpkg-util project
> >> (
https://pagure.io/rpkg-util). I really
> >> recommend looking at it as I spent more than a year solving this
> >> particular problem with
> >> changelog and release (but actually not only that problem) and I also
> >> optimized the macros there
> >> as much as I possibly could.
> >>
> >> If you want to play around with it, you can download the latest
> >> version from here:
> >>
> >>
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
> >>
> >> and try it on this:
> >>
> >>
https://pagure.io/hello_rpkg
> >>
> >> clime
> >>
> >> >
> >> >
> >> > [1]:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/
> >> > [2]:
https://hackmd.io/kuXOPe74RfepuztBz7pBsg
> >> > _______________________________________________
> >> > devel mailing list -- devel(a)lists.fedoraproject.org
> >> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> > List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> >> _______________________________________________
> >> devel mailing list -- devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> >
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org