What would it take to drop release and changelog from our spec files? (and do we want to?)
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this? - Based on dates? - Using an always increasing integer? - Using the number of successful builds since the last time the version field changed? - Another idea?
The third option looks like to be the one closest to our current behavior.
Another question regarding the release field is: how would we deal with pre-releases and other unsortable versions? The current doc relies on <extraver> etc. in the release field as per: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_unsor... Would using the tilde to specify pre-releases in the version field be sufficient (as described in https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_versi... I.e., how can we cater for snapshot releases (e.g. based off a specific git commit)?
With the changelog it becomes a little bit more tricky. We currently have 3 changelogs in Fedora with 3 different target audience (this is how I understand them): - One for the files in the git repository, meant to be "consumed" by our fellow packagers, not meant to leave the Fedora infrastructure - One in the spec file describing the changes applied to it. This one is meant to be accessible to sysadmins who want to know/check what changed in a package - One in bodhi, meant for end-user consumption and which should give some explanation as to why the package was updated or where information about the update can be found
So we need to, somehow, merge two changelogs into one while realizing that some information in one may not be desirable in the other (for example the world famous commit message: "oops I've forgot to upload the sources" does not need to appear in the RPM's changelog). Would it be easier to merge the git changelog with the spec changelog or the spec changelog with the bodhi notes?
For the former one easy way to achieve this is to consider all the commits since the last successful build and have a magic keyword to either include or exclude a commit message in the changelog. For the latter, we discussed the idea of using annotated git tags this fall.
Both have their pros and cons, do people have some experience with either and could share their feedback? Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
To give some context to this discussion, the CPE team is moving toward quarterly planning and for the first months of 2020 a small team of people in the CPE team is willing to do the work to make this happen, but there are two requirements: - The work needs to be scoped and defined by January 20th 2020 (so we can evaluate whether or not we have the knowledge, resources and time to do it) - The work needs to be achievable before March 31st 2020 (cf the quarterly objectives we're working towards)
Thus our call for input to accept or reject the idea and if the former scope/define the system.
Thanks in advance for your help,
Pierre
On Fri, 10 Jan 2020 at 17:45, Pierre-Yves Chibon pingou@pingoured.fr wrote:
With the changelog it becomes a little bit more tricky. We currently have 3 changelogs in Fedora with 3 different target audience (this is how I understand them):
- One for the files in the git repository, meant to be "consumed" by our fellow packagers, not meant to leave the Fedora infrastructure
- One in the spec file describing the changes applied to it. This one is meant to be accessible to sysadmins who want to know/check what changed in a package
- One in bodhi, meant for end-user consumption and which should give some explanation as to why the package was updated or where information about the update can be found
So we need to, somehow, merge two changelogs into one while realizing that some information in one may not be desirable in the other (for example the world famous commit message: "oops I've forgot to upload the sources" does not need to appear in the RPM's changelog). Would it be easier to merge the git changelog with the spec changelog or the spec changelog with the bodhi notes?
For the former one easy way to achieve this is to consider all the commits since the last successful build and have a magic keyword to either include or exclude a commit message in the changelog. For the latter, we discussed the idea of using annotated git tags this fall.
Most of the time, I end up copying the spec changelog in the commit message and I don't change the update template, so the bodhi changelog is also the same. The spec changelog is a pain for me, so I'd vote for git commit messages + tags (unnanotated; otherwise, I don't see much benefit).
Dne 10. 01. 20 v 18:21 Iñaki Ucar napsal(a):
Most of the time, I end up copying the spec changelog in the commit message and I don't change the update template,
+1 Thou, occasionaly I *delete* some of those commits as they are unnessary in changelog. E.g.:
* typo fix * revert of .... * previous fix was not complete...
On Mon, Jan 13, 2020 at 12:20:15PM +0100, Miroslav Suchý wrote:
Dne 10. 01. 20 v 18:21 Iñaki Ucar napsal(a):
Most of the time, I end up copying the spec changelog in the commit message and I don't change the update template,
+1 Thou, occasionaly I *delete* some of those commits as they are unnessary in changelog. E.g.:
- typo fix
- revert of ....
- previous fix was not complete...
It seems I'm using git quite differently.
My spec file changelog describes what the spec changes did in regard to functionality or version changes of the tool.
My git commit messages usually try to summarize, what happened to the (dist-)git-repo - mostly stuff that can be seen in `git show --stat`, although sometimes I describe what I changed inside a file.
The field for bodhi I usually copy from the changelog - but to be honest I only fill it, because it's there - I don't even really know what it is used for, except being shown on the update page.
On the other side when I'm looking for something, because something broke or whatever, usually I go to the spec-file on src.fedoraproject.org[1] first, and if I can't find the reason I'm looking for in the first few seconds I take a look at the changelog, which for convenience is right there.
Didn't even know about `rpm -q xx --changelog` - quite nice, but as I prefer to look at the spec file instructions first anyway I'll probably still check there. (Usually I look for a specific Patch file or configure-flag, so the directly checking the spec is faster)
For me this is a plus and one of the reasons I do enjoy packaging for Fedora more than I enjoy packaging for Debian - one single spec file vs. many files, and each single one of those has a different format.
But I also do understand that separating the changelog from the spec file might have its benefits and that duplication of the messages for people using the git log message as changelog messages is annoying.
All the best, Astra
On Tue, 2020-01-14 at 00:30 +0100, David Kaufmann wrote:
The field for bodhi I usually copy from the changelog - but to be honest I only fill it, because it's there - I don't even really know what it is used for, except being shown on the update page.
Users can read the update text with the updateinfo dnf subcommand. For example:
$ dnf updateinfo --info FEDORA-2019-ed226e6112 Last metadata expiration check: 0:02:22 ago on Wed 15 Jan 2020 01:45:20 PM EST. =============================================================================== xorg-x11-server-1.20.6-1.fc30 =============================================================================== Update ID: FEDORA-2019-ed226e6112 Type: bugfix Updated: 2020-01-14 19:37:24 Bugs: 1752211 - crash of /usr/libexec/Xorg : 1761241 - [abrt] xorg-x11-server-Xwayland: present_wnmd_abort_vblank(): Xwayland killed by SIGABRT Description: xserver 1.20.6 Severity: None
On Fri, Jan 10, 2020, 17:37 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version
field changed?
- Another idea?
What about "number of commits since last version update" (possibly tagged in git)? That should encompass the possibilities you listed above, is well-defined, and should be most like the current behavior.
The number of successful builds doesn't sound like a well-defined / stable number to me, since it relies on information outside of git.
The third option looks like to be the one closest to our current behavior.
Another question regarding the release field is: how would we deal with pre-releases and other unsortable versions? The current doc relies on <extraver> etc. in the release field as per:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_unsor... Would using the tilde to specify pre-releases in the version field be sufficient (as described in
https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_versi... )? I.e., how can we cater for snapshot releases (e.g. based off a specific git commit)?
The tilde notation is not enough for general snapshots, though it works well for tagged alpha / beta / rc releases (because they are real releases, not snapshots).
For snapshot builds, I think something like %releaseprefix of 0." (cf. %distprefix) could be used to indicate a prerelease in the .spec file, and incorporated in the automatically generated Release tag.
With the changelog it becomes a little bit more tricky. We currently have 3 changelogs in Fedora with 3 different target audience (this is how I understand them):
- One for the files in the git repository, meant to be "consumed" by our fellow packagers, not meant to leave the Fedora infrastructure
- One in the spec file describing the changes applied to it. This one is
meant to be accessible to sysadmins who want to know/check what changed in a package
- One in bodhi, meant for end-user consumption and which should give some explanation as to why the package was updated or where information
about the update can be found
I think it would be good to merge git commit messages and the .spec changelog. This is also how many projects handle this on GitHub. Using [skip-changelog] in commit messages or something or something similar to indicate that it should not be mentioned in generated release notes is also common practice (just like [skip-ci]).
So I think this is very much doable, since there's actually a lot of precedent for doing this. Also, the two audiences for git commit messages and RPM changelogs are probably more similar than the audience for bodhi updates (where I usually put more user-facing information).
Fabio
So we need to, somehow, merge two changelogs into one while realizing that some information in one may not be desirable in the other (for example the world famous commit message: "oops I've forgot to upload the sources" does not need to appear in the RPM's changelog). Would it be easier to merge the git changelog with the spec changelog or the spec changelog with the bodhi notes?
For the former one easy way to achieve this is to consider all the commits since the last successful build and have a magic keyword to either include or exclude a commit message in the changelog. For the latter, we discussed the idea of using annotated git tags this fall.
Both have their pros and cons, do people have some experience with either and could share their feedback? Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
To give some context to this discussion, the CPE team is moving toward quarterly planning and for the first months of 2020 a small team of people in the CPE team is willing to do the work to make this happen, but there are two requirements:
- The work needs to be scoped and defined by January 20th 2020 (so we can evaluate whether or not we have the knowledge, resources and time to
do it)
- The work needs to be achievable before March 31st 2020 (cf the
quarterly objectives we're working towards)
Thus our call for input to accept or reject the idea and if the former scope/define the system.
Thanks in advance for your help,
Pierre
[1] https://github.com/hawkowl/towncrier _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Dne 10. 01. 20 v 18:33 Fabio Valentini napsal(a):
On Fri, Jan 10, 2020, 17:37 Pierre-Yves Chibon <pingou@pingoured.fr mailto:pingou@pingoured.fr> wrote:
Good Morning Everyone, This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further. Do we want to drop release and changelog from our spec file? If we do, how would this work? The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this? - Based on dates? - Using an always increasing integer? - Using the number of successful builds since the last time the version field changed? - Another idea?What about "number of commits since last version update" (possibly tagged in git)? That should encompass the possibilities you listed above, is well-defined, and should be most like the current behavior.
That won't work. This assumes that all subpackages have the same version as the main package, but that might not be true (it is definitely not true for Ruby neither for Perl AFAIK). If nothing else, there must be way to override/hint the automation (unless the automation is smart enough to detect such scenarios, which would be cool).
Vít
On Fri, Jan 10, 2020 at 6:52 PM Vít Ondruch vondruch@redhat.com wrote:
Dne 10. 01. 20 v 18:33 Fabio Valentini napsal(a):
On Fri, Jan 10, 2020, 17:37 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
What about "number of commits since last version update" (possibly tagged in git)? That should encompass the possibilities you listed above, is well-defined, and should be most like the current behavior.
(snip)
That won't work. This assumes that all subpackages have the same version as the main package, but that might not be true (it is definitely not true for Ruby neither for Perl AFAIK).
It won't work in all cases, true. But in the predominant case where all produced packages have the same version, it works fine.
If nothing else, there must be way to override/hint the automation (unless the automation is smart enough to detect such scenarios, which would be cool).
For example: For the handful of special cases like ruby it should be easy to define a macro to tell the Release tag generator "hey, count up releases from point X instead" (which could just be a manually specified git ref).
Fabio
Vít
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Fri, Jan 10, 2020 at 12:52 PM Vít Ondruch vondruch@redhat.com wrote:
Dne 10. 01. 20 v 18:33 Fabio Valentini napsal(a):
On Fri, Jan 10, 2020, 17:37 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
What about "number of commits since last version update" (possibly tagged in git)? That should encompass the possibilities you listed above, is well-defined, and should be most like the current behavior.
That won't work. This assumes that all subpackages have the same version as the main package, but that might not be true (it is definitely not true for Ruby neither for Perl AFAIK). If nothing else, there must be way to override/hint the automation (unless the automation is smart enough to detect such scenarios, which would be cool).
Yes it will, because the source version is predictable. Version updates would work off the source RPM version, and that isn't affected by games like that. The query would be something like the following:
$ rpmspec -q --srpm --qf "%{VERSION}-%{RELEASE}\n" foo.spec
That will *always* do the right thing.
On Fri, Jan 10, 2020 at 10:14:39PM -0500, Neal Gompa wrote:
On Fri, Jan 10, 2020 at 12:52 PM Vít Ondruch vondruch@redhat.com wrote:
Dne 10. 01. 20 v 18:33 Fabio Valentini napsal(a):
What about "number of commits since last version update" (possibly tagged in git)? That should encompass the possibilities you listed above, is well-defined, and should be most like the current behavior.
That won't work. This assumes that all subpackages have the same version as the main package, but that might not be true (it is definitely not true for Ruby neither for Perl AFAIK). If nothing else, there must be way to override/hint the automation (unless the automation is smart enough to detect such scenarios, which would be cool).
Yes it will, because the source version is predictable. Version updates would work off the source RPM version, and that isn't affected by games like that.
No, it won't as we have competing %{version}-%{release} strings among multiple packages. E.g. perl source bundles an Encode module. And we know the module is updated frequenly on CPAN. Thus we build perl-Encode-V1-R1 from perl.spec, then we build perl-Encode-V1-R2 from CPAN perl-Encode.spec so that R2 >= R1, then we rebuild perl.spec again with disabled perl-Encode subpackage.
With unpredictable releases we would have to fake version of the bundled module so that it is always lower than a version of the standalone package. Not impossible but, a nuissance.
-- Petr
Dne 13. 01. 20 v 10:46 Petr Pisar napsal(a):
No, it won't as we have competing %{version}-%{release} strings among multiple packages. E.g. perl source bundles an Encode module. And we know the module is updated frequenly on CPAN. Thus we build perl-Encode-V1-R1 from perl.spec, then we build perl-Encode-V1-R2 from CPAN perl-Encode.spec so that R2 >= R1, then we rebuild perl.spec again with disabled perl-Encode subpackage.
Wow! Why it is not in separate package then? It can have the same Source0, but if subpackage have different release cadence, then I see no reason why it is a subpackage.
Le 2020-01-13 11:28, Miroslav Suchý a écrit :
Dne 13. 01. 20 v 10:46 Petr Pisar napsal(a):
No, it won't as we have competing %{version}-%{release} strings among multiple packages. E.g. perl source bundles an Encode module. And we know the module is updated frequenly on CPAN. Thus we build perl-Encode-V1-R1 from perl.spec, then we build perl-Encode-V1-R2 from CPAN perl-Encode.spec so that R2
= R1, then we
rebuild perl.spec again with disabled perl-Encode subpackage.
Wow! Why it is not in separate package then? It can have the same Source0, but if subpackage have different release cadence, then I see no reason why it is a subpackage.
Personaly, I tend to consider that something that gets definitely bundled inside another project, and is no longer released separately, surrenders its right for separate versioning, and inherits this versioning from the bundler, whatever upstream feels about it (if they want to keep versioning separate they can provide a separate release).
Of course if it *is* still released separately packaging the bundled version instead of the real upstream is bad practice.
Regards,
On Mon, Jan 13, 2020 at 11:28:41AM +0100, Miroslav Suchý wrote:
Dne 13. 01. 20 v 10:46 Petr Pisar napsal(a):
No, it won't as we have competing %{version}-%{release} strings among multiple packages. E.g. perl source bundles an Encode module. And we know the module is updated frequenly on CPAN. Thus we build perl-Encode-V1-R1 from perl.spec, then we build perl-Encode-V1-R2 from CPAN perl-Encode.spec so that R2 >= R1, then we rebuild perl.spec again with disabled perl-Encode subpackage.
Wow! Why it is not in separate package then? It can have the same Source0, but if subpackage have different release cadence, then I see no reason why it is a subpackage.
The subpackage is part of a standard library (because it is bundled) and other packages including the standalone one need it for building or for running. Therefore we keep the subpackage available for the short time when bootstrapping a new perl.
(Theoretically if we disabled tests and reimplemented build scripts of the standalone packages, wou could remove the subpackages completely. But we consider that more labour intensive, fragile and risky than the currenct approach.)
-- Petr
Just catching up on this thread...
How about an incremental step? (I don't know how difficult it would be to implement however)...
What about separating the change log to a separate file in dist-git? Something like the traditional "ChangeLog"?
I like to keep my spec files in sync between releases unless there's a good reason for them to diverge but sometimes there are changes to a specific release that aren't really a part of the others.
That way I can keep the spec files in sync (git merge master, et. all) but without merging changelogs that don't actually apply to that release (and dealing with merge conflicts)?
Thanks, Richard
On Mon, Jan 13, 2020 at 08:19:36AM -0600, Richard Shaw wrote:
Just catching up on this thread... How about an incremental step? (I don't know how difficult it would be to implement however)...
This is what Vit was suggesting which seems like a very nice idea :)
What about separating the change log to a separate file in dist-git? Something like the traditional "ChangeLog"? I like to keep my spec files in sync between releases unless there's a good reason for them to diverge but sometimes there are changes to a specific release that aren't really a part of the others. That way I can keep the spec files in sync (git merge master, et. all) but without merging changelogs that don't actually apply to that release (and dealing with merge conflicts)?
The conflicts won't go away, it'll be moved from the .spec file to the ChangeLog file.
Pierre
On Mon, Jan 13, 2020 at 08:19:36AM -0600, Richard Shaw wrote:
Just catching up on this thread...
How about an incremental step? (I don't know how difficult it would be to implement however)...
What about separating the change log to a separate file in dist-git? Something like the traditional "ChangeLog"?
I like to keep my spec files in sync between releases unless there's a good reason for them to diverge but sometimes there are changes to a specific release that aren't really a part of the others.
That way I can keep the spec files in sync (git merge master, et. all) but without merging changelogs that don't actually apply to that release (and dealing with merge conflicts)?
I do a variant of this with rpminspect and some other packages. I automatically generate 'changelog' when I make a new upstream release. This and the tarball then Source0 and Source1. The spec file has:
%changelog %include %{SOURCE1}
I don't upload the changelog to the lookaside cache, but just commit it in dist-git. I'm trying to strictly release from upstream and push builds in to dist-git, but I'm doing this changelog structure so that if there is downstream maintenance needed them commits to the changelog can be separate from spec and patch changes which makes cherry-picking across branches theoretically easier.
There is the problem of Release, which I would like to have in a separate file to that I can also %include. Haven't tried that yet mostly because I haven't had an excuse to.
Thanks,
Dne 11. 01. 20 v 4:14 Neal Gompa napsal(a):
On Fri, Jan 10, 2020 at 12:52 PM Vít Ondruch vondruch@redhat.com wrote:
Dne 10. 01. 20 v 18:33 Fabio Valentini napsal(a):
On Fri, Jan 10, 2020, 17:37 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
What about "number of commits since last version update" (possibly tagged in git)? That should encompass the possibilities you listed above, is well-defined, and should be most like the current behavior.
That won't work. This assumes that all subpackages have the same version as the main package, but that might not be true (it is definitely not true for Ruby neither for Perl AFAIK). If nothing else, there must be way to override/hint the automation (unless the automation is smart enough to detect such scenarios, which would be cool).
Yes it will, because the source version is predictable. Version updates would work off the source RPM version, and that isn't affected by games like that. The query would be something like the following:
$ rpmspec -q --srpm --qf "%{VERSION}-%{RELEASE}\n" foo.spec
That will *always* do the right thing.
I might not get your point, but working on Ruby 2.7, I cannot reset its release to -1, because Ruby 2.6 ships rubygem-net-telnet-0.2.0-124.fc32.noarch.rpm [1], while the version of this subpackage has not changed for Ruby 2.7. If I did the reset, I would end up with rubygem-net-telnet-0.2.0-1.fc32.noarch.rpm which would never override the package coming from Ruby 2.6.
I don't even want to discuss the case Petr described in the follow up, because that is even more complex situation ...
Vít
[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=1397861
Le vendredi 10 janvier 2020 à 17:36 +0100, Pierre-Yves Chibon a écrit :
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
Dropping changelog is easy. Since we have a clean separation of spec repo (src.fedoraproject.org) and project repo (pagure, gitlab or elsewhere) the spec should just be assembled from all the src.fedoraproject.org commit messages not present in the previous generated changelog
(that won't work for thinks ike rehat-rpm-config because it does not separate the project files in a separate repository but it’s high time it behaved likea normal project, the non separation is a major PITA to deal with)
Droping releases is much harder to design for because we don’t have a linear build history, there are branches that split and then re-merge at system release time (sometimes, with excursions in copr or another repo), none of the proposed solutions would accomodate those workflows.
Regards,
On Fri, Jan 10, 2020 at 6:36 PM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le vendredi 10 janvier 2020 à 17:36 +0100, Pierre-Yves Chibon a écrit :
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
Dropping changelog is easy. Since we have a clean separation of spec repo (src.fedoraproject.org) and project repo (pagure, gitlab or elsewhere) the spec should just be assembled from all the src.fedoraproject.org commit messages not present in the previous generated changelog
(that won't work for thinks ike rehat-rpm-config because it does not separate the project files in a separate repository but it’s high time it behaved likea normal project, the non separation is a major PITA to deal with)
(snip)
Droping releases is much harder to design for because we don’t have a linear build history, there are branches that split and then re-merge at system release time (sometimes, with excursions in copr or another repo), none of the proposed solutions would accomodate those workflows.
You can never expect our tooling to do "magic" (TM) and work "just right", no matter which Versions and Releases and Epochs of packages are available from third-party repos and coprs. This has nothing to do with proposed auto-generated Release tag, and it's definitely not a new problem. We've basically ignored consistency with third-party repos until now - and rightly so, IMO - because that's what "dnf distro-sync" is for. (Even upgrade path from fedora N to N+1 doesn't have to be "clean" anymore, because system-upgrade operates in "distro-sync" mode by default now ...)
Fabio
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le vendredi 10 janvier 2020 à 20:53 +0100, Fabio Valentini a écrit :
You can never expect our tooling to do "magic" (TM) and work "just right", no matter which Versions and Releases and Epochs of packages are available from third-party repos and coprs.
Yes, sure, but the current way we manage releases accomodate those worklows.
For example: 1. I hit a fontconfig bug while preparing the new font packaging guidelines, 2. Akira kindly fixed the problem upstream, https://gitlab.freedesktop.org/fontconfig/fontconfig/issues/185 3. it was trivial to build a package matching the Fedora fontconfig with just the fix added in copr, without breaking the release thread https://copr.fedorainfracloud.org/coprs/nim/fonts-rpm-macros/build/1126851/
(it exists just for me in my copre because general availability was not required to advance on the guidelines proposal; general availability would have required a push to rawhide and a support commitment by Akira)
And, it will all converge once FPC finishes its review, Akira releases fontconfig upstream, and the result is rebuilt for Fedora. Had I waited for Akira to wrap up an upstream release and build the result rawhide- side the FPC submission would have been pushed back for months (and then it would have delayed other Fedora changes, it's a cascading effect).
The non-linear progression permitted by current manual release setting allows parallelizing work and getting things done a lot faster within the project. I don’t see how to manage this with the autogeneration proposal
Regards,
On Fri, Jan 10, 2020 at 09:18:35PM +0100, Nicolas Mailhot via devel wrote:
Le vendredi 10 janvier 2020 à 20:53 +0100, Fabio Valentini a écrit :
You can never expect our tooling to do "magic" (TM) and work "just right", no matter which Versions and Releases and Epochs of packages are available from third-party repos and coprs.
Yes, sure, but the current way we manage releases accomodate those worklows.
For example:
- I hit a fontconfig bug while preparing the new font packaging
guidelines, 2. Akira kindly fixed the problem upstream, https://gitlab.freedesktop.org/fontconfig/fontconfig/issues/185 3. it was trivial to build a package matching the Fedora fontconfig with just the fix added in copr, without breaking the release thread https://copr.fedorainfracloud.org/coprs/nim/fonts-rpm-macros/build/1126851/
(it exists just for me in my copre because general availability was not required to advance on the guidelines proposal; general availability would have required a push to rawhide and a support commitment by Akira)
And, it will all converge once FPC finishes its review, Akira releases fontconfig upstream, and the result is rebuilt for Fedora. Had I waited for Akira to wrap up an upstream release and build the result rawhide- side the FPC submission would have been pushed back for months (and then it would have delayed other Fedora changes, it's a cascading effect).
The non-linear progression permitted by current manual release setting allows parallelizing work and getting things done a lot faster within the project. I don’t see how to manage this with the autogeneration proposal
I don't quite see how it conflicts with it either. You may end up having foo-1.0-42 in copr and foo-1.0-32 in koji which would lead to a foo-1.0-32 (or -39) at the next build, but I'm not seeing how it's a problem per say. Could you give a more concrete example?
Thanks, Pierre
Le 2020-01-13 12:07, Pierre-Yves Chibon a écrit :
Hi,
I don't quite see how it conflicts with it either. You may end up having foo-1.0-42 in copr and foo-1.0-32 in koji which would lead to a foo-1.0-32 (or -39) at the next build, but I'm not seeing how it's a problem per say.
Correct handling of release streams means my private copr build is higher than the current koji build, but lower than the next koji build. Therefore whenever Akira decided to do a new official build it will replace my own (in dnf and mock), and I know I need to check if my private patch is still relevant, or if problem that justified the patch is solved in the koji build
Such a packaging workflow (builds that diverge in separate repos and then re-converge in koji) is very common.
Regards,
When removing the changelog in a spec file, users (not package maintainers) using this command are disappointed?
``` $ rpm -q --changelog python3 * Tue Oct 15 2019 Miro Hrončok mhroncok@redhat.com - 3.7.5-1 - Update to 3.7.5 ... ```
On 13. 01. 20 12:40, Jun Aruga wrote:
When removing the changelog in a spec file, users (not package maintainers) using this command are disappointed?
$ rpm -q --changelog python3 * Tue Oct 15 2019 Miro Hrončok <mhroncok@redhat.com> - 3.7.5-1 - Update to 3.7.5 ...
No, this would still work. It would be injected into the rpm during buildtime, from a more sophisticated source than the spec file.
On 1/10/20 7:35 PM, Nicolas Mailhot via devel wrote:
Dropping changelog is easy. Since we have a clean separation of spec repo (src.fedoraproject.org) and project repo (pagure, gitlab or elsewhere) the spec should just be assembled from all the src.fedoraproject.org commit messages not present in the previous generated changelog
(that won't work for thinks ike rehat-rpm-config because it does not separate the project files in a separate repository but it’s high time it behaved likea normal project, the non separation is a major PITA to deal with)
You keep saying that, but maybe you were not involved with redhat-rpm-config back when it was that way. It was the most hideous piece of package I had ever worked with, because the model of external tarballs is just absurd and does not work at all with what it is and does. We've been there, done that and the current model is the first time that redhat-rpm-config is actually nice to maintain.
- Panu -
Le 2020-01-13 11:01, Panu Matilainen a écrit :
You keep saying that, but maybe you were not involved with redhat-rpm-config back when it was that way. It was the most hideous piece of package I had ever worked with, because the model of external tarballs is just absurd and does not work at all with what it is and does. We've been there, done that and the current model is the first time that redhat-rpm-config is actually nice to maintain.
I’ve also been there and done that and contributed to redhat-rpm-config and other macro packages that do not use the redhat-rpm-config model, and redhat-rpm-config is the most annoying to work with, with no directory structure to speak of, file-by-file special casing in the spec file, need to mangle file names because the way redhat-rpm-config is set up breaks pagure assumptions, and probably others I forget. This one-of-a-kind-ism is horrible except for people that only contribute to redhat-rpm-config and are so used to its quirks they don’t feel them anymore.
It is trivial nowadays to point a spec to pagure (or gitlab, or github) with the forge macros, and get an archive spectool understands that matches the version declared in the src.fedora.org spec, with a correct directory structure that can be used to simplify the spec logic
Regards,
* Nicolas Mailhot via devel:
Le vendredi 10 janvier 2020 à 17:36 +0100, Pierre-Yves Chibon a écrit :
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
Dropping changelog is easy. Since we have a clean separation of spec repo (src.fedoraproject.org) and project repo (pagure, gitlab or elsewhere) the spec should just be assembled from all the src.fedoraproject.org commit messages not present in the previous generated changelog
That way, you cannot fix typos, add missing CVE IDs, and the like. It's a significant functionality change.
It may be possible to auto-generate a %changelog section listing new commits since the last %changelog update. Combined with a tool that people can run before editing the %changelog section manually, to make its contents explicit, I expect that to work.
(that won't work for thinks ike rehat-rpm-config because it does not separate the project files in a separate repository but it’s high time it behaved likea normal project, the non separation is a major PITA to deal with)
I have trouble matching this claim to my experience working on redhat-rpm-config. Why is it painful to use Git as it was designed?
I don't think it's an improvement if contributors have to figure out how to generate a new upstream tarball for each change.
Thanks, Florian
Le 2020-01-13 12:52, Florian Weimer a écrit :
I have trouble matching this claim to my experience working on redhat-rpm-config. Why is it painful to use Git as it was designed?
Because redhat-rpm-config is not "using Git as it was designed". It’s using git as a centralized flat and linear SCM (à la SVN/CVS) without putting the project info in the repository but in a static spec mapping.
In sane macro project that use separate sources you can do things like "install rpm/macros.d/* to %{_rpmconfigdir}/macros.d/" (because if you accepted a PR that puts things in rpm/macros.d and tagged the result you *want* this result deployed in %{_rpmconfigdir}/macros.d/, right?). In redhat-rpm-config that does not work. Every single file is a different source with custom spec handling.
Apart from being inefficient this custom handling badly collides whenever two PRs try to add new files. With a real detached source project, with a real tree managed by git, all the eventual tree collisions would be managed by git. With everything-is-a-separate-source files collide (at the source number level) even when they have different filenames, and will be deployed in different places.
Regards,
On 1/13/20 2:44 PM, Nicolas Mailhot via devel wrote:
Le 2020-01-13 12:52, Florian Weimer a écrit :
I have trouble matching this claim to my experience working on redhat-rpm-config. Why is it painful to use Git as it was designed?
Because redhat-rpm-config is not "using Git as it was designed". It’s using git as a centralized flat and linear SCM (à la SVN/CVS) without putting the project info in the repository but in a static spec mapping.
In sane macro project that use separate sources you can do things like "install rpm/macros.d/* to %{_rpmconfigdir}/macros.d/" (because if you accepted a PR that puts things in rpm/macros.d and tagged the result you *want* this result deployed in %{_rpmconfigdir}/macros.d/, right?). In redhat-rpm-config that does not work. Every single file is a different source with custom spec handling.
Apart from being inefficient this custom handling badly collides whenever two PRs try to add new files. With a real detached source project, with a real tree managed by git, all the eventual tree collisions would be managed by git. With everything-is-a-separate-source files collide (at the source number level) even when they have different filenames, and will be deployed in different places.
Well yes, nothing is perfect. These are such minor issues compared to the patches-on-patches-of-patches horror that was before that it fails to register on my annoyance scale at all, it's nothing but a nice tradeoff for being able to use git as deity meant it for this content.
- Panu -
On Mon, Jan 13, 2020 at 8:07 AM Panu Matilainen pmatilai@redhat.com wrote:
On 1/13/20 2:44 PM, Nicolas Mailhot via devel wrote:
Le 2020-01-13 12:52, Florian Weimer a écrit :
I have trouble matching this claim to my experience working on redhat-rpm-config. Why is it painful to use Git as it was designed?
Because redhat-rpm-config is not "using Git as it was designed". It’s using git as a centralized flat and linear SCM (à la SVN/CVS) without putting the project info in the repository but in a static spec mapping.
In sane macro project that use separate sources you can do things like "install rpm/macros.d/* to %{_rpmconfigdir}/macros.d/" (because if you accepted a PR that puts things in rpm/macros.d and tagged the result you *want* this result deployed in %{_rpmconfigdir}/macros.d/, right?). In redhat-rpm-config that does not work. Every single file is a different source with custom spec handling.
Apart from being inefficient this custom handling badly collides whenever two PRs try to add new files. With a real detached source project, with a real tree managed by git, all the eventual tree collisions would be managed by git. With everything-is-a-separate-source files collide (at the source number level) even when they have different filenames, and will be deployed in different places.
Well yes, nothing is perfect. These are such minor issues compared to the patches-on-patches-of-patches horror that was before that it fails to register on my annoyance scale at all, it's nothing but a nice tradeoff for being able to use git as deity meant it for this content.
It's also important to note that splitting is only useful if there are multiple disparate downstreams. In this case, there isn't. Only Fedora (and its downstream RHEL) use redhat-rpm-config. Contrast that to Mageia's rpm-setup and OpenMandriva's rpm-openmandriva-setup, which are managed this way *because* it is the upstream for multiple disparate distributions (rpm-setup is the upstream for Mageia and PCLinuxOS, while rpm-omv-setup is the upstream for OpenMandriva and ROSA).
On Fri, Jan 10, 2020 at 05:36:46PM +0100, Pierre-Yves Chibon wrote:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file? If we do, how would this work?
OpenSUSE proved years and years ago that dropping %changelog is possible, easy and desirable. We should do that IMHO.
I wouldn't personally bother touching the Release header.
Rich.
On Fri, Jan 10, 2020 at 9:46 pm, Richard W.M. Jones rjones@redhat.com wrote:
OpenSUSE proved years and years ago that dropping %changelog is possible, easy and desirable. We should do that IMHO.
They still have %changelog at the bottom of each spec file, but as the last line of the file. The actual changelog is stored as a separate .changes file. That's a *lot* better than what Fedora does now, because it makes it way easier to scroll through the spec file. But getting rid of the changelog entirely would be even nicer. :)
Michael
On Fri, Jan 10, 2020 at 8:20 PM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Fri, Jan 10, 2020 at 9:46 pm, Richard W.M. Jones rjones@redhat.com wrote:
OpenSUSE proved years and years ago that dropping %changelog is possible, easy and desirable. We should do that IMHO.
They still have %changelog at the bottom of each spec file, but as the last line of the file. The actual changelog is stored as a separate .changes file. That's a *lot* better than what Fedora does now, because it makes it way easier to scroll through the spec file. But getting rid of the changelog entirely would be even nicer. :)
openSUSE needs the changes file because the integrated VCS in their build system is awful. History is not able to be preserved across submissions to projects, so the only complete artifact of history is that file. If you want an example of a distribution that completely eliminated the management of an RPM changelog, look at Mageia (and its ancestor, Mandriva). Their Dist-SVN system dynamically generated changelogs from the SVN log.
However, they had two properties that we don't have with Git, and we need some analogues for that:
* Dist-SVN is still SVN, and that means the log data can be edited without editing the revision itself. This means the message could be changed after the fact and the next build would incorporate it. * Dist-SVN relied on tags of successful builds to track the checkpoints for changelog generation and did not use branches in any meaningful capacity, so the history was always fully linear.
I think any successful equivalent of that for Dist-Git will require us to emulate these two properties somehow. My thinking is that annotated tags would be the right way to allow for changelogs to be set without the complexity of figuring out commit filtering. The annotation must be editable. If not, this won't work. We have to have a way to let people clean up changelogs. I think we can enforce linear tracking within a branch for Koji to generate Release values, but it's going to be tricky, given the contortions to the history that Git allows.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 1/10/20 8:14 PM, Michael Catanzaro wrote:
On Fri, Jan 10, 2020 at 9:46 pm, Richard W.M. Jones rjones@redhat.com wrote:
OpenSUSE proved years and years ago that dropping %changelog is possible, easy and desirable. We should do that IMHO.
They still have %changelog at the bottom of each spec file, but as the last line of the file. The actual changelog is stored as a separate .changes file. That's a *lot* better than what Fedora does now, because it makes it way easier to scroll through the spec file. But getting rid of the changelog entirely would be even nicer. :)
changelogs often include CVE information, especially useful when the fixes are backported rather than included as part of the regular update/release process.
How could the CVE info be available in the absence of changelogs?
On Mon, Jan 13, 2020 at 2:02 PM Przemek Klosowski via devel devel@lists.fedoraproject.org wrote:
On 1/10/20 8:14 PM, Michael Catanzaro wrote:
On Fri, Jan 10, 2020 at 9:46 pm, Richard W.M. Jones rjones@redhat.com wrote:
OpenSUSE proved years and years ago that dropping %changelog is possible, easy and desirable. We should do that IMHO.
They still have %changelog at the bottom of each spec file, but as the last line of the file. The actual changelog is stored as a separate .changes file. That's a *lot* better than what Fedora does now, because it makes it way easier to scroll through the spec file. But getting rid of the changelog entirely would be even nicer. :)
changelogs often include CVE information, especially useful when the fixes are backported rather than included as part of the regular update/release process.
How could the CVE info be available in the absence of changelogs?
In Fedora, this information has always been available as part of updateinfo, just like with RHEL. Only CentOS seems to still not have updateinfo published for advisories and including security information.
That said, the information could *still* be in changelogs if the packager deems so.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 1/13/20 2:47 PM, Neal Gompa wrote:
changelogs often include CVE information, especially useful when the fixes are backported rather than included as part of the regular update/release process.
How could the CVE info be available in the absence of changelogs?
In Fedora, this information has always been available as part of updateinfo, just like with RHEL. Only CentOS seems to still not have updateinfo published for advisories and including security information.
You're right, the updateinfo capability in dnf is awesome!! Thanks for bringing it up here, I missed the --cve option. I think specifically
dnf updateinfo list --cve=CVE-2015-2080
should list the packages that address this particular CVE, which would be better than grepping changelog for CVEs, except that it didn't work for me right now somehow. I found very little info about it, e.g. on Oracle pages:
https://docs.oracle.com/en/operating-systems/oracle-linux/8/software-managem...
Is there a better description somewhere, maybe with some examples?
That said, the information could *still* be in changelogs if the packager deems so.
I am all for automating all this---the CVE-in-changelog looked like a manual effort on the part of some packagers, so if there's an automatic workflow that takes care of it in the updateinfo records, I am all for it and won't miss the changelogs.
On Fri, Jan 10, 2020 at 11:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
The third option looks like to be the one closest to our current behavior.
I always envisioned that we'd use a variant of the third option.
The options I've thought of:
* <commit-at-version>%{dist}.<build-at-version> * <commit-at-version>.<build-at-version>%{?dist} * %{dist}.<commit-at-version>.<build-at-version>
The first option aligns with our current guidelines suggesting that such bumps go at the end after the DistTag. I personally have felt those were ugly, but there you go. The second option is actually what I use at $DAYJOB for our environment, and I've been pretty pleased with how that works in practice (our buildsystem builds for all target distros from the same commit at once, so the EVRs are rather strictly matched). However, it might not be workable with the whole "Koji and manual build per target" thing. The third option is how openSUSE Leap is currently done, primarily to make Leap packages lower than SUSE Linux Enterprise packages (openSUSE Leap can "upgrade" to SLE). I'm not a fan of this scheme as it feels rather "weird" to me.
Naturally, there is also an option that would involve enhancing RPM: extending EVR comparisons to factor in the DistTag as a formal property (e.g. EVRD). ALT Linux recently started doing this. I will admit there is some appeal to this, as it makes the distro versioning aspect a property unto itself. This would require making Koji move from NVR to NVRD for its uniqueness constraint, which would be an "interesting" change to implement. This would eliminate the %dist macro entirely, and remove boilerplate entirely from the Release field. This would make all the questions and games about sequencing the values in the Release field to be completely about the unique build versioning of the package. In practice, I think this is just a fancier variant of option two above.
Another question regarding the release field is: how would we deal with pre-releases and other unsortable versions? The current doc relies on <extraver> etc. in the release field as per: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_unsor... Would using the tilde to specify pre-releases in the version field be sufficient (as described in https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_versi... I.e., how can we cater for snapshot releases (e.g. based off a specific git commit)?
Pre-releases can use tilde versioning, and starting with Fedora 31, we can also use carat versions for post-releases.
Carat versioning draft from tibbs from long ago: https://fedoraproject.org/wiki/User:Tibbs/TildeCaretVersioning
On Fri, Jan 10, 2020 at 09:54:59PM -0500, Neal Gompa wrote:
On Fri, Jan 10, 2020 at 11:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
The release field would need to be set by koji ignoring whatever is in the spec file.
Yes, please!!! This is a relatively small step that will make so many other things much easier. This is long overdue.
I think this should be done with two principles:
- that this automatization will coexist for a while with manual release tags and changelogs that we have now, so they need to be compatible. Initially this could be opt-in, and once we get a feeling that things works nicely, this could be made opt-out or even mandatory.
- only support simple versioning (i.e. don't bother supporting pre-release or post-release versions in the dist tag. Packages that need that would either need to use tile/caret versioning or not be supported for automatic tagging.)
How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
The third option looks like to be the one closest to our current behavior.
I always envisioned that we'd use a variant of the third option.
The options I've thought of:
- <commit-at-version>%{dist}.<build-at-version>
- <commit-at-version>.<build-at-version>%{?dist}
- %{dist}.<commit-at-version>.<build-at-version>
(I understand that <commit-at-version> is basically %version.)
In the first variant, builds from the same %version with different dist tags would always compare different, so a build from a later Fedora release would compare higher.
If builds are tagged automatically, the automatic build-number tag cannot be meanigfully compared between different Fedora releases. In the second variant, builds from the same %version would compare by the build number, which seems useless.
The first option aligns with our current guidelines suggesting that such bumps go at the end after the DistTag. I personally have felt those were ugly, but there you go. The second option is actually what I use at $DAYJOB for our environment, and I've been pretty pleased with how that works in practice (our buildsystem builds for all target distros from the same commit at once, so the EVRs are rather strictly matched). However, it might not be workable with the whole "Koji and manual build per target" thing.
Exactly. I don't think this makes sense in Fedora.
The third option is how openSUSE Leap is currently done, primarily to make Leap packages lower than SUSE Linux Enterprise packages (openSUSE Leap can "upgrade" to SLE). I'm not a fan of this scheme as it feels rather "weird" to me.
Agreed.
Naturally, there is also an option that would involve enhancing RPM: extending EVR comparisons to factor in the DistTag as a formal property (e.g. EVRD). ALT Linux recently started doing this. I will admit there is some appeal to this, as it makes the distro versioning aspect a property unto itself. This would require making Koji move from NVR to NVRD for its uniqueness constraint, which would be an "interesting" change to implement. This would eliminate the %dist macro entirely, and remove boilerplate entirely from the Release field. This would make all the questions and games about sequencing the values in the Release field to be completely about the unique build versioning of the package. In practice, I think this is just a fancier variant of option two above.
I don't think we should go down this route, unless we know for sure that we can't make things work otherwise. I hope we could phase-in this automatic tagging in koji, and this will be much easier if we are able to do this without changes to rpm and the version comparison logic (which is also embedded in countless other places).
Another question regarding the release field is: how would we deal with pre-releases and other unsortable versions? The current doc relies on <extraver> etc. in the release field as per: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_unsor... Would using the tilde to specify pre-releases in the version field be sufficient (as described in https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_versi... I.e., how can we cater for snapshot releases (e.g. based off a specific git commit)?
Pre-releases can use tilde versioning, and starting with Fedora 31, we can also use carat versions for post-releases.
Carat versioning draft from tibbs from long ago: https://fedoraproject.org/wiki/User:Tibbs/TildeCaretVersioning
I would limit automatic tagging for packages which do "sane" versioning, i.e. don't embed any pre-release or post-release data in disttag. FPC should finally approve tile+caret versioning guidelines [1,2]. Right now F30 doesn't support carets, but we could proceed anyway. By the time this is ready, F30 should have support and F29 will be EOL.
[1] https://pagure.io/packaging-committee/issue/904 [2] https://pagure.io/packaging-committee/pull-request/908
Zbyszek
On Sat, Jan 11, 2020 at 12:38 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Fri, Jan 10, 2020 at 09:54:59PM -0500, Neal Gompa wrote:
On Fri, Jan 10, 2020 at 11:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
The release field would need to be set by koji ignoring whatever is in the spec file.
Yes, please!!! This is a relatively small step that will make so many other things much easier. This is long overdue.
I think this should be done with two principles:
that this automatization will coexist for a while with manual release tags and changelogs that we have now, so they need to be compatible. Initially this could be opt-in, and once we get a feeling that things works nicely, this could be made opt-out or even mandatory.
only support simple versioning (i.e. don't bother supporting pre-release or post-release versions in the dist tag. Packages that need that would either need to use tile/caret versioning or not be supported for automatic tagging.)
I think we have to make sure there's no option in which automatic tagging is not supportable. Otherwise we can basically never transition fully over.
How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
The third option looks like to be the one closest to our current behavior.
I always envisioned that we'd use a variant of the third option.
The options I've thought of:
- <commit-at-version>%{dist}.<build-at-version>
- <commit-at-version>.<build-at-version>%{?dist}
- %{dist}.<commit-at-version>.<build-at-version>
(I understand that <commit-at-version> is basically %version.)
To clarify, <commit-at-version> is a counter of the number of commits since %version was bump, starting at 1. <build-at-version> is a counter for the number of builds with that %version-<commit-at-version>, starting at 1.
In the first variant, builds from the same %version with different dist tags would always compare different, so a build from a later Fedora release would compare higher.
If builds are tagged automatically, the automatic build-number tag cannot be meanigfully compared between different Fedora releases. In the second variant, builds from the same %version would compare by the build number, which seems useless.
The first option aligns with our current guidelines suggesting that such bumps go at the end after the DistTag. I personally have felt those were ugly, but there you go. The second option is actually what I use at $DAYJOB for our environment, and I've been pretty pleased with how that works in practice (our buildsystem builds for all target distros from the same commit at once, so the EVRs are rather strictly matched). However, it might not be workable with the whole "Koji and manual build per target" thing.
Exactly. I don't think this makes sense in Fedora.
The only reason I mentioned it is because since we distro-sync between releases, it doesn't actually matter as much as it used to.
The third option is how openSUSE Leap is currently done, primarily to make Leap packages lower than SUSE Linux Enterprise packages (openSUSE Leap can "upgrade" to SLE). I'm not a fan of this scheme as it feels rather "weird" to me.
Agreed.
Naturally, there is also an option that would involve enhancing RPM: extending EVR comparisons to factor in the DistTag as a formal property (e.g. EVRD). ALT Linux recently started doing this. I will admit there is some appeal to this, as it makes the distro versioning aspect a property unto itself. This would require making Koji move from NVR to NVRD for its uniqueness constraint, which would be an "interesting" change to implement. This would eliminate the %dist macro entirely, and remove boilerplate entirely from the Release field. This would make all the questions and games about sequencing the values in the Release field to be completely about the unique build versioning of the package. In practice, I think this is just a fancier variant of option two above.
I don't think we should go down this route, unless we know for sure that we can't make things work otherwise. I hope we could phase-in this automatic tagging in koji, and this will be much easier if we are able to do this without changes to rpm and the version comparison logic (which is also embedded in countless other places).
Another way to go here would be to do EVDR instead of EVRD. This more closely mirrors the intent that we have in Fedora for newer distribution releases to always be considered "upgrades" from older ones.
In this case, it would make sense to enhance RPM and leverage DistTag for version comparison.
Unlike the equivalent variant with the Release field (like how openSUSE does), we wouldn't have to take a hit with lots of things looking like it is "downgrading" for a distro upgrade.
Another question regarding the release field is: how would we deal with pre-releases and other unsortable versions? The current doc relies on <extraver> etc. in the release field as per: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_unsor... Would using the tilde to specify pre-releases in the version field be sufficient (as described in https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_versi... I.e., how can we cater for snapshot releases (e.g. based off a specific git commit)?
Pre-releases can use tilde versioning, and starting with Fedora 31, we can also use carat versions for post-releases.
Carat versioning draft from tibbs from long ago: https://fedoraproject.org/wiki/User:Tibbs/TildeCaretVersioning
I would limit automatic tagging for packages which do "sane" versioning, i.e. don't embed any pre-release or post-release data in disttag. FPC should finally approve tile+caret versioning guidelines [1,2]. Right now F30 doesn't support carets, but we could proceed anyway. By the time this is ready, F30 should have support and F29 will be EOL.
[1] https://pagure.io/packaging-committee/issue/904 [2] https://pagure.io/packaging-committee/pull-request/908
And carat versions are being backported to EL8: https://bugzilla.redhat.com/show_bug.cgi?id=1654901
-- 真実はいつも一つ！/ Always, there's only one truth!
Le samedi 11 janvier 2020 à 13:09 -0500, Neal Gompa a écrit :
The only reason I mentioned it is because since we distro-sync between releases, it doesn't actually matter as much as it used to.
rawhide does not distro-sync (and some may say that rawhide does not matter, but early problem detection by rawhide users is one big reason distro sync works so well for other releases)
On Sun, Jan 12, 2020 at 4:03 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 11 janvier 2020 à 13:09 -0500, Neal Gompa a écrit :
The only reason I mentioned it is because since we distro-sync between releases, it doesn't actually matter as much as it used to.
rawhide does not distro-sync (and some may say that rawhide does not matter, but early problem detection by rawhide users is one big reason distro sync works so well for other releases)
It is strongly recommended that you should be using 'dnf distro-sync' for Rawhide if you want to use it as a daily driver. That handles all the cases that occur in a rolling branch effectively.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Sun, Jan 12, 2020 at 04:38:55AM -0500, Neal Gompa wrote:
On Sun, Jan 12, 2020 at 4:03 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le samedi 11 janvier 2020 à 13:09 -0500, Neal Gompa a écrit :
The only reason I mentioned it is because since we distro-sync between releases, it doesn't actually matter as much as it used to.
rawhide does not distro-sync (and some may say that rawhide does not matter, but early problem detection by rawhide users is one big reason distro sync works so well for other releases)
It sometimes does. Sometimes it does not.
It is strongly recommended that you should be using 'dnf distro-sync' for Rawhide if you want to use it as a daily driver. That handles all the cases that occur in a rolling branch effectively.
It does not.
For example, right now on my laptop:
Error: Problem: problem with installed package awscli-1.16.309-1.fc32.noarch - package awscli-1.16.309-1.fc32.noarch requires python3.8dist(botocore) = 1.13.45, but none of the providers can be installed - python3-botocore-1.13.45-2.fc32.noarch does not belong to a distupgrade repository - awscli-1.16.309-1.fc32.noarch does not belong to a distupgrade repository (try to add '--skip-broken' to skip uninstallable packages)
(--skip-broken doesn't do anything here, same exact output)
distro-sync is only going to work if the repo(s) are all consistent for you to sync to. :( Which I think we should get to, but it's not the case yet.
kevin
Dne 11. 01. 20 v 3:54 Neal Gompa napsal(a):
- %{dist}.<commit-at-version>.<build-at-version>
-1 Packages with commitish in release version are usually developers snapshot. We already have few packages with such release in Fedora, but I would dislike to make this standard.
On Wed, Jan 15, 2020 at 7:10 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 11. 01. 20 v 3:54 Neal Gompa napsal(a):
- %{dist}.<commit-at-version>.<build-at-version>
-1 Packages with commitish in release version are usually developers snapshot. We already have few packages with such release in Fedora, but I would dislike to make this standard.
That is not putting the hash in there. That is a counter. It's poorly labeled.
Basically, an NEVRA would look like this:
foo-0:1.0.0-fc32.1.1.noarch
"commit-at-version" is a counter, starting from 1 that indicates the number of commits in the build system of the version of a package. So version 1.1 would start at 1 initially, and every subsequent commit where you _don't_ change the version, this number goes up.
"build-at-version" is a counter, starting from 1 that indicates the number of builds in the build system of the commit at that version of a package. So at version 1.1 with commit-at-version 1 would initially have build-at-version 1 for the first build, but on the fourth rebuild, it would be 5.
Given that, here's what the options I presented would look like the following Release fields:
* <commit-at-version>%{dist}.<build-at-version>: 1.fc32.5 * <commit-at-version>.<build-at-version>%{?dist}: 1.5.fc32 * %{dist}.<commit-at-version>.<build-at-version>: fc32.1.5
-- 真実はいつも一つ！/ Always, there's only one truth!
On Fri, Jan 10, 2020 at 09:54:59PM -0500, Neal Gompa wrote:
On Fri, Jan 10, 2020 at 11:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
The third option looks like to be the one closest to our current behavior.
I always envisioned that we'd use a variant of the third option.
The options I've thought of:
- <commit-at-version>%{dist}.<build-at-version>
- <commit-at-version>.<build-at-version>%{?dist}
- %{dist}.<commit-at-version>.<build-at-version>
I've been thinking a bit about this and been wondering any reason why not to do simply? <build-at-version>%{dist}
This would basically mimic what we are currently doing by hand, it would be the less changes to our current way of working (making opting-in smoother).
Thanks, Pierre
On Tue, Jan 28, 2020 at 7:27 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Fri, Jan 10, 2020 at 09:54:59PM -0500, Neal Gompa wrote:
On Fri, Jan 10, 2020 at 11:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
The third option looks like to be the one closest to our current behavior.
I always envisioned that we'd use a variant of the third option.
The options I've thought of:
- <commit-at-version>%{dist}.<build-at-version>
- <commit-at-version>.<build-at-version>%{?dist}
- %{dist}.<commit-at-version>.<build-at-version>
I've been thinking a bit about this and been wondering any reason why not to do simply? <build-at-version>%{dist}
This would basically mimic what we are currently doing by hand, it would be the less changes to our current way of working (making opting-in smoother).
If we're not doing automatic builds, sure. I've been going on two big assumptions:
1. We're going to do automatic building 2. We need *some* kind of stable leading portion of release for packagers to use for specified dependencies, especially Obsoletes+Provides combos.
Neal Gompa ngompa13@gmail.com writes:
On Tue, Jan 28, 2020 at 7:27 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Fri, Jan 10, 2020 at 09:54:59PM -0500, Neal Gompa wrote:
On Fri, Jan 10, 2020 at 11:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
The third option looks like to be the one closest to our current behavior.
I always envisioned that we'd use a variant of the third option.
The options I've thought of:
- <commit-at-version>%{dist}.<build-at-version>
- <commit-at-version>.<build-at-version>%{?dist}
- %{dist}.<commit-at-version>.<build-at-version>
I've been thinking a bit about this and been wondering any reason why not to do simply? <build-at-version>%{dist}
This would basically mimic what we are currently doing by hand, it would be the less changes to our current way of working (making opting-in smoother).
If we're not doing automatic builds, sure. I've been going on two big assumptions:
- We're going to do automatic building
- We need *some* kind of stable leading portion of release for
packagers to use for specified dependencies, especially Obsoletes+Provides combos.
How is openSUSE taking care of 2 then? OBS bumps the release on each rebuild and that can result in crazily high release numbers. I assume they got a mechanism for Obsoletes+Provides and we could use that too?
Am 10.01.20 um 17:36 schrieb Pierre-Yves Chibon:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file?
Vote: no.
The correct releases and changelogs in the rpms are important to check for security patches made. This need of any admin will override any argument for a removal, as it's the most important source on a working system to gather it's security state.
Best regards, Marius
On Sun, Jan 12, 2020 at 4:20 PM Marius Schwarz fedoradev@cloud-foo.de wrote:
Am 10.01.20 um 17:36 schrieb Pierre-Yves Chibon:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file?
Vote: no.
The correct releases and changelogs in the rpms are important to check for security patches made. This need of any admin will override any argument for a removal, as it's the most important source on a working system to gather it's security state.
The idea is to remove the manual management of them, not the existence of them altogether. I agree with you in that we should not remove them entirely for many of the same reasons.
On 12. 01. 20 22:19, Marius Schwarz wrote:
Am 10.01.20 um 17:36 schrieb Pierre-Yves Chibon:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and spoken about on this very list this fall, so I'd like to push it a little further.
Do we want to drop release and changelog from our spec file?
Vote: no.
The correct releases and changelogs in the rpms are important to check for security patches made. This need of any admin will override any argument for a removal, as it's the most important source on a working system to gather it's security state.
It would stay in the RPM, we would just populate it differently and it would no longer be hardcoded in the spec file in our infrastructure.
On 1/12/20 3:19 PM, Marius Schwarz wrote:
Am 10.01.20 um 17:36 schrieb Pierre-Yves Chibon:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and
spoken
about on this very list this fall, so I'd like to push it a little
further.
Do we want to drop release and changelog from our spec file?
Vote: no.
The correct releases and changelogs in the rpms are important to check for security patches made. This need of any admin will override any argument for a removal, as it's the most important source on a working system to gather it's security state.
Finally the reply I was looking for! As someone who relies the changelog of the RPM for security reasons this whole thread has me worried.
On 1/12/20 3:38 PM, Miro Hrončok wrote:
It would stay in the RPM, we would just populate it differently and it would no longer be hardcoded in the spec file in our infrastructure.
How will it be populated? Will it ensure that the information that is important for security minding end users is still available? Sorry in advance if I missed the details of how it would still be managed and included for end users to consume?
Thanks! Joe
On Mon, 13 Jan 2020 at 15:23, Joe Doss joe@solidadmin.com wrote:
On 1/12/20 3:19 PM, Marius Schwarz wrote:
Am 10.01.20 um 17:36 schrieb Pierre-Yves Chibon:
Good Morning Everyone,
This is not a new idea, it has been presented at flock last year and
spoken
about on this very list this fall, so I'd like to push it a little
further.
Do we want to drop release and changelog from our spec file?
Vote: no.
The correct releases and changelogs in the rpms are important to check for security patches made. This need of any admin will override any argument for a removal, as it's the most important source on a working system to gather it's security state.
Finally the reply I was looking for! As someone who relies the changelog of the RPM for security reasons this whole thread has me worried.
On 1/12/20 3:38 PM, Miro Hrončok wrote:
It would stay in the RPM, we would just populate it differently and it would no longer be hardcoded in the spec file in our infrastructure.
How will it be populated? Will it ensure that the information that is important for security minding end users is still available? Sorry in advance if I missed the details of how it would still be managed and included for end users to consume?
The CVE information there is mostly on the whim of the packager. Some packagers do put items in there and others forget (aka I have forgotten to do so a couple of times).
How will the new way be populated? That is what part of this thread is trying to get to. It has not been decided or implemented but would hopefully be part of getting the changelog out
On 10. 01. 20 17:36, Pierre-Yves Chibon wrote:
Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
Is the idea of using annotated git tags abandoned altogether?
We could even create a tool that would "prefill" a template with all commit messages since the latest annotated tag.
So you would do:
- commit, commit, commit (optional pushes) - fedpkg release - edit the message, inspect the e:v-r, be able to abort if I don't like it - fedpkg push - pushes the tags and branch - fedpkg build
And when somebody attempts to do a untagged build, it would fail, similarly to what it does now when you try to build unpushed changes.
Example template:
# You are about to tag foobar 1.1-1 # Here are the commit messages since 1.0-6 # # This is the 1st commit message:
Update to 1.1
# This is the commit message #2:
Damn sources
# This is the commit message #3:
Fix the tests
# This is the commit message #4:
Typo
# Please amend the commit messages to a %changelog entry. Lines starting # with '#' will be ignored, and an empty message aborts the release. # If you like to change the release number, abort the process and follow # <link to docs> # # Author: Miro Hrončok mhroncok@redhat.com # Date: Sun Jan 12 22:54:32 2020 +0100 # # Use --author and/or --date to change the above.
On Sun, Jan 12, 2020 at 10:56:00PM +0100, Miro Hrončok wrote:
On 10. 01. 20 17:36, Pierre-Yves Chibon wrote:
Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
Is the idea of using annotated git tags abandoned altogether?
It was the sentence just above the section you quoted here ;-)
For memory: -- So we need to, somehow, merge two changelogs into one while realizing that some information in one may not be desirable in the other (for example the world famous commit message: "oops I've forgot to upload the sources" does not need to appear in the RPM's changelog). Would it be easier to merge the git changelog with the spec changelog or the spec changelog with the bodhi notes?
For the former one easy way to achieve this is to consider all the commits since the last successful build and have a magic keyword to either include or exclude a commit message in the changelog. For the latter, we discussed the idea of using annotated git tags this fall. --
We could even create a tool that would "prefill" a template with all commit messages since the latest annotated tag.
So you would do:
- commit, commit, commit (optional pushes)
- fedpkg release
- edit the message, inspect the e:v-r, be able to abort if I don't like it
- fedpkg push - pushes the tags and branch
- fedpkg build
And when somebody attempts to do a untagged build, it would fail, similarly to what it does now when you try to build unpushed changes.
This is definitely an option, it might even be the simplest to implement. One observation about it though is that we would still have three changelogs then, git commit, git tags and bodhi update. So it is easier to implement as we rely on the packager to do the review of what goes into the RPM changelog. We still have Neal's question: do we let packager edit these tags? (Considering we allow editing old changelog entries, I guess we could).
One question, if the release is set in koji and the changelog in git tag. If I tag foo 1.0 with a first changelog entry, then koji builds 1.0-1 with that changelog. I find out that there is a mistake in the changelog, so I edit the tag, won't koji build 1.0-2? Which would then have two repeating changelog entries one for -1 with the error and one for -2 with the fix. Does it matter?
Note that this question may be applicable regardless of the solution we adopt (git tag, git commits or something else?)
Pierre
On 13. 01. 20 12:28, Pierre-Yves Chibon wrote:
On Sun, Jan 12, 2020 at 10:56:00PM +0100, Miro Hrončok wrote:
On 10. 01. 20 17:36, Pierre-Yves Chibon wrote:
Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
Is the idea of using annotated git tags abandoned altogether?
It was the sentence just above the section you quoted here ;-)
"For the latter, we discussed the idea of using annotated git tags this fall."
Sorry about that, I must have missed it.
We could even create a tool that would "prefill" a template with all commit messages since the latest annotated tag.
So you would do:
- commit, commit, commit (optional pushes)
- fedpkg release
- edit the message, inspect the e:v-r, be able to abort if I don't like it
- fedpkg push - pushes the tags and branch
- fedpkg build
And when somebody attempts to do a untagged build, it would fail, similarly to what it does now when you try to build unpushed changes.
This is definitely an option, it might even be the simplest to implement. One observation about it though is that we would still have three changelogs then, git commit, git tags and bodhi update. So it is easier to implement as we rely on the packager to do the review of what goes into the RPM changelog. We still have Neal's question: do we let packager edit these tags? (Considering we allow editing old changelog entries, I guess we could).
If we build from the tag hash in Koji, we cannot. If we build from the commit hash and let Koji "fetch" the changelog from existing tags, we can.
We still need to construct the changelog entries from old tags, to the second option makes sense.
One question, if the release is set in koji and the changelog in git tag. If I tag foo 1.0 with a first changelog entry, then koji builds 1.0-1 with that changelog. I find out that there is a mistake in the changelog, so I edit the tag, won't koji build 1.0-2? Which would then have two repeating changelog entries one for -1 with the error and one for -2 with the fix. Does it matter?
Currently, you cannot change already built existing changelog entry without a bump and rebuild. This would be the same:
1. You change the changelog entry of 1.0-1. 2. You attempt build. 3. Koji tells you 1.0-1 was already build. 4. You add another entry.
It doesn't matter that much. Obviously, the ability to edit tags puts some constrains on how the release is counted:
- if it is counted simply from number of tags, you cannot ever remove them, just edit them to "empty changelog" - locally, it counts all tags, and in Koji, only pushed tags, this may produce confusion in corner cases - even if we only count pushed tags locally, somebody else might have pushed another one in the meantime - Hence, we might count the release number from commits, always
On Mon, 2020-01-13 at 12:28 +0100, Pierre-Yves Chibon wrote:
If I tag foo 1.0 with a first changelog entry, then koji builds 1.0-1 with that changelog.
I think it would be better if the release were part of the git tag, instead of automatically generating it. Not all packages use an integer for the release, and the release also encodes the distro version (though there is a proposal in the thread to change that too) currently. Some packages do this due to a policy for packages that don't have releases from upstream. For example:
$ rpm -q dino dino-0.0-0.15.20191216.git.11c18cd.fc30.x86_64
Dino doesn't make releases upstream, so the release field encodes the commit and commit date, along with an incremental "integer", which is really a float because we want it to sort less than 1, in case Dino ever makes a 0.0 release.
Now, we could change the policy too to get around this, but I think "git tag" is a natural way for me to indicate "I want to build and release this commit to this Fedora release". I can encode the full NEVR in the git tag, and I can use the tag body to encode the end user release notes for Bodhi and for the changelog. This also allows the git commit messages to be logs for packagers to read (I do like the proposal of a fedpkg feature to prefill the git tag message body with the git commits since last release as a starting point though!)
Le 2020-01-13 16:20, Randy Barlow a écrit :
Now, we could change the policy too to get around this, but I think "git tag" is a natural way for me to indicate "I want to build and release this commit to this Fedora release".
Please do not try to infer packaged commit from src.fedoraproject.org commit tags.
As a general principle, inventing complex string structures that require complex one-of-a-king parsers (with heuristics that always fail one way or another) does not work. rpm is not perl (for a long time). Keep a separate variable/field/tag for separate info, do not multiplex them.
Practically, we already have a standard way to set target commit in the distribution, %{commit} and several hundred distro packages depend on this mecanism.
Auto-release, if it exists, should correspond to spec release info and nothing more, with all the packaged project info read from the spec. Or else we’ll get madness like a tag that says something is at version/tag/commit X and the tagged spec that declares it is Y.
Regards,
Dne 12. 01. 20 v 22:56 Miro Hrončok napsal(a):
On 10. 01. 20 17:36, Pierre-Yves Chibon wrote:
Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
Is the idea of using annotated git tags abandoned altogether?
We could even create a tool that would "prefill" a template with all commit messages since the latest annotated tag.
So you would do:
- commit, commit, commit (optional pushes)
- fedpkg release
- edit the message, inspect the e:v-r, be able to abort if I don't
like it
- fedpkg push - pushes the tags and branch
- fedpkg build
And when somebody attempts to do a untagged build, it would fail, similarly to what it does now when you try to build unpushed changes.
Example template:
# You are about to tag foobar 1.1-1 # Here are the commit messages since 1.0-6 # # This is the 1st commit message:
Update to 1.1
# This is the commit message #2:
Damn sources
# This is the commit message #3:
Fix the tests
# This is the commit message #4:
Typo
# Please amend the commit messages to a %changelog entry. Lines starting # with '#' will be ignored, and an empty message aborts the release. # If you like to change the release number, abort the process and follow # <link to docs> # # Author: Miro Hrončok mhroncok@redhat.com # Date: Sun Jan 12 22:54:32 2020 +0100 # # Use --author and/or --date to change the above.
While I like the annotated tag proposal, I would really appreciate if the first step was replacing the:
~~~
%changelog
* Tue Jan 07 2020 Vít Ondruch vondruch@redhat.com - 2.7.0-1 - Upgrade to Ruby 2.7.0. - Drop useless %%{rubygems_default_filter}. - Fix checksec 2.0+ compatibility.
* Tue Jun 25 2019 Vít Ondruch vondruch@redhat.com - 2.6.3-121 - Properly support %%prerelease in %%gemspec_ macros.
... snip ...
~~~
in .spec file by
~~~
%changelog
%include changelog
~~~
where the `changelog` file would be either available with the original changelog content in repository or if it was not available, it would be provided by build infra. The *provided by infra* is the most important thing here. This IMO would allow us incrementally improve the situation.
Then we could discuss how to provide the content of the `changelog` file, while maintainers could still maintain old changelogs, or they could split the changelog into separate `changelog` file and maintain it there, or leave it for infrastructure (simple collecting git commits) or using annotated tags.
Vít
Dne 13. 01. 20 v 14:05 Vít Ondruch napsal(a):
Dne 12. 01. 20 v 22:56 Miro Hrončok napsal(a):
On 10. 01. 20 17:36, Pierre-Yves Chibon wrote:
Is there a different approach, e.g. by using towncrier[1] or something comparable, to track changes outside the spec file?
Is the idea of using annotated git tags abandoned altogether?
We could even create a tool that would "prefill" a template with all commit messages since the latest annotated tag.
So you would do:
- commit, commit, commit (optional pushes)
- fedpkg release
- edit the message, inspect the e:v-r, be able to abort if I don't
like it
- fedpkg push - pushes the tags and branch
- fedpkg build
And when somebody attempts to do a untagged build, it would fail, similarly to what it does now when you try to build unpushed changes.
Example template:
# You are about to tag foobar 1.1-1 # Here are the commit messages since 1.0-6 # # This is the 1st commit message:
Update to 1.1
# This is the commit message #2:
Damn sources
# This is the commit message #3:
Fix the tests
# This is the commit message #4:
Typo
# Please amend the commit messages to a %changelog entry. Lines starting # with '#' will be ignored, and an empty message aborts the release. # If you like to change the release number, abort the process and follow # <link to docs> # # Author: Miro Hrončok mhroncok@redhat.com # Date: Sun Jan 12 22:54:32 2020 +0100 # # Use --author and/or --date to change the above.
While I like the annotated tag proposal, I would really appreciate if the first step was replacing the:
%changelog * Tue Jan 07 2020 Vít Ondruch <vondruch@redhat.com> - 2.7.0-1 - Upgrade to Ruby 2.7.0. - Drop useless %%{rubygems_default_filter}. - Fix checksec 2.0+ compatibility. * Tue Jun 25 2019 Vít Ondruch <vondruch@redhat.com> - 2.6.3-121 - Properly support %%prerelease in %%gemspec_ macros. ... snip ...in .spec file by
%changelog %include changelogwhere the `changelog` file would be either available with the original changelog content in repository or if it was not available, it would be provided by build infra. The *provided by infra* is the most important thing here. This IMO would allow us incrementally improve the situation.
Then we could discuss how to provide the content of the `changelog` file, while maintainers could still maintain old changelogs, or they could split the changelog into separate `changelog` file and maintain it there, or leave it for infrastructure (simple collecting git commits) or using annotated tags.
Actually, I took the liberty and filled this guideline proposal change [1], which just suggests keeping changelogs in `changelog` file, outside of the .spec file.
Vít
Am 13.01.20 um 14:05 schrieb Vít Ondruch:
While I like the annotated tag proposal, I would really appreciate if the first step was replacing the:
(..:)
%changelog
%include changelog
where the `changelog` file would be either available with the original changelog content in repository or if it was not available, it would be provided by build infra. The *provided by infra* is the most important thing here. This IMO would allow us incrementally improve the situation. Then we could discuss how to provide the content of the `changelog` file, while maintainers could still maintain old changelogs, or they could split the changelog into separate `changelog` file and maintain it there, or leave it for infrastructure (simple collecting git commits) or using annotated tags.
I like that proposal. As a volunteer packager with only a few packages this appproach is very easy to understand and I can opt out easily if I dislike the automation (I'd like not to maintain a changelog manually but I also don't want to have it auto-generated based on all commit messages).
If we look at the modularity disaster I think we should first start with a very simple approach which is easy to understand but also provides an opt-out.
Felix
On 1/15/20 2:13 PM, Miroslav Suchý wrote:
Dne 13. 01. 20 v 14:05 Vít Ondruch napsal(a):
%changelog
%include changelog
+1
As I pointed out in https://pagure.io/packaging-committee/pull-request/942, %include is nasty because it breaks the stand-alone attribute of specs. There are umphteen dupes and variants of bugs about systemd.spec's use of %include breaking this and that use pattern that aren't systemd's fault, it's just that %include isn't as useful as it initially seems.
- Panu -
Dne 15. 01. 20 v 13:33 Panu Matilainen napsal(a):
On 1/15/20 2:13 PM, Miroslav Suchý wrote:
Dne 13. 01. 20 v 14:05 Vít Ondruch napsal(a):
%changelog
%include changelog
+1
As I pointed out in https://pagure.io/packaging-committee/pull-request/942, %include is nasty because it breaks the stand-alone attribute of specs. There are umphteen dupes and variants of bugs about systemd.spec's use of %include breaking this and that use pattern that aren't systemd's fault, it's just that %include isn't as useful as it initially seems.
Would it be helpful to open RPM upstream ticket to discuss when the missing included file is problematic and whether there should be other graceful variant of include? May be the %include should always behave gracefully, because (S)RPM build is going to always fail due to missing files specified by Source directive.
Vít
- Panu - _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 1/15/20 3:33 PM, Vít Ondruch wrote:
Dne 15. 01. 20 v 13:33 Panu Matilainen napsal(a):
On 1/15/20 2:13 PM, Miroslav Suchý wrote:
Dne 13. 01. 20 v 14:05 Vít Ondruch napsal(a):
%changelog
%include changelog
+1
As I pointed out in https://pagure.io/packaging-committee/pull-request/942, %include is nasty because it breaks the stand-alone attribute of specs. There are umphteen dupes and variants of bugs about systemd.spec's use of %include breaking this and that use pattern that aren't systemd's fault, it's just that %include isn't as useful as it initially seems.
Would it be helpful to open RPM upstream ticket to discuss when the missing included file is problematic and whether there should be other graceful variant of include? May be the %include should always behave gracefully, because (S)RPM build is going to always fail due to missing files specified by Source directive.
This was actually discussed upstream not too long ago, just can't find the reference offhand.
Basically an %include can never be allowed to fail as it can contain anything at all, including mandatory parts of the spec. In theory you can have a spec consisting of nothing but one or more %include directives.
(S)RPM builds are not an issue, it's spec queries, in particular for build-dependencies but also for other data. Rpm has special logic to allow missing sources and patches for query purposes, because they're only needed for building. Something changelog-specific could also safely ignore the missing file as %changelog is always an optional part of the spec.
- Panu -
On Wed, Jan 15, 2020 at 04:28:56PM +0200, Panu Matilainen wrote:
On 1/15/20 3:33 PM, Vít Ondruch wrote:
Dne 15. 01. 20 v 13:33 Panu Matilainen napsal(a):
On 1/15/20 2:13 PM, Miroslav Suchý wrote:
Dne 13. 01. 20 v 14:05 Vít Ondruch napsal(a):
%changelog
%include changelog
+1
As I pointed out in https://pagure.io/packaging-committee/pull-request/942, %include is nasty because it breaks the stand-alone attribute of specs. There are umphteen dupes and variants of bugs about systemd.spec's use of %include breaking this and that use pattern that aren't systemd's fault, it's just that %include isn't as useful as it initially seems.
Would it be helpful to open RPM upstream ticket to discuss when the missing included file is problematic and whether there should be other graceful variant of include? May be the %include should always behave gracefully, because (S)RPM build is going to always fail due to missing files specified by Source directive.
This was actually discussed upstream not too long ago, just can't find the reference offhand.
Basically an %include can never be allowed to fail as it can contain anything at all, including mandatory parts of the spec. In theory you can have a spec consisting of nothing but one or more %include directives.
(S)RPM builds are not an issue, it's spec queries, in particular for build-dependencies but also for other data. Rpm has special logic to allow missing sources and patches for query purposes, because they're only needed for building. Something changelog-specific could also safely ignore the missing file as %changelog is always an optional part of the spec.
I ran in to some issues when using %include for the changelog and it may be related to this. I made my changelog the Source1 file for the package and did:
%changelog %include %{SOURCE1}
It works for the packages I'm using it in.
Thanks,
On 1/20/20 7:03 PM, David Cantrell wrote:
On Wed, Jan 15, 2020 at 04:28:56PM +0200, Panu Matilainen wrote:
On 1/15/20 3:33 PM, Vít Ondruch wrote:
Dne 15. 01. 20 v 13:33 Panu Matilainen napsal(a):
On 1/15/20 2:13 PM, Miroslav Suchý wrote:
Dne 13. 01. 20 v 14:05 Vít Ondruch napsal(a):
%changelog
%include changelog
+1
As I pointed out in https://pagure.io/packaging-committee/pull-request/942, %include is nasty because it breaks the stand-alone attribute of specs. There are umphteen dupes and variants of bugs about systemd.spec's use of %include breaking this and that use pattern that aren't systemd's fault, it's just that %include isn't as useful as it initially seems.
Would it be helpful to open RPM upstream ticket to discuss when the missing included file is problematic and whether there should be other graceful variant of include? May be the %include should always behave gracefully, because (S)RPM build is going to always fail due to missing files specified by Source directive.
This was actually discussed upstream not too long ago, just can't find the reference offhand.
Basically an %include can never be allowed to fail as it can contain anything at all, including mandatory parts of the spec. In theory you can have a spec consisting of nothing but one or more %include directives.
(S)RPM builds are not an issue, it's spec queries, in particular for build-dependencies but also for other data. Rpm has special logic to allow missing sources and patches for query purposes, because they're only needed for building. Something changelog-specific could also safely ignore the missing file as %changelog is always an optional part of the spec.
I ran in to some issues when using %include for the changelog and it may be related to this. I made my changelog the Source1 file for the package and did:
%changelog %include %{SOURCE1}
It works for the packages I'm using it in.
This will essentially break "dnf builddep" for that spec, because the %{_sourcedir} for root is different from the user. It'd be less of an issue if it always defaulted to current directory, but that's a can of worms of its own.
It'll also cause general breakage and pain for those who expect the spec to be standalone parseable. For example the "all of rawhide" spec snapshot tarball becomes rather useless if specs use %include.
- Panu -
On Fri, Jan 10, 2020 at 05:36:46PM +0100, Pierre-Yves Chibon wrote:
The release field would need to be set by koji ignoring whatever is in the spec file. How do we want to do this?
- Based on dates?
- Using an always increasing integer?
- Using the number of successful builds since the last time the version field changed?
- Another idea?
Obsoletes RPM tag is usually specific up to release value. Therefore the new release scheme must hold two properties:
(1) The value must be predictable. I.e. a packager must know what release the pacakge will be assigned before commiting a change with the Obsoletes tag.
(2) The new values must be larger than all historical values (across all historical Fedora releases). That assures than a new build won't become obsoleted because of a decreased release.
-- Petr
Hey Petr!
On Mon, 2020-01-13 at 10:34 +0100, Petr Pisar wrote:
(2) The new values must be larger than all historical values (across all historical Fedora releases). That assures than a new build won't become obsoleted because of a decreased release.
can you clarify what you mean with "historical values/releases" here? Would it include all currently active releases at some point in time (i.e. everything up to F32/rawhide ATM)?
The way I see it, we should have a differently phrased requirement, ensuring that within an upstream version of a package, the release of a build in a higher Fedora release should be "newer" than of any previous build (for the same version) in an older release.
I mean that we e.g. can't cater for the case where a package is built in an older Fedora release, but no correspondent build is done in a newer one, just as without automation: if a package has say releases -1.fc30, -1.fc31 and -1.fc32 and then the packager only bumps to -2.fc30 and builds in F-30, there's not much we can do about it (other than chiding them for it ;)).
Nils
On Mon, Jan 20, 2020 at 05:20:08PM +0100, Nils Philippsen wrote:
On Mon, 2020-01-13 at 10:34 +0100, Petr Pisar wrote:
(2) The new values must be larger than all historical values (across all historical Fedora releases). That assures than a new build won't become obsoleted because of a decreased release.
can you clarify what you mean with "historical values/releases" here? Would it include all currently active releases at some point in time (i.e. everything up to F32/rawhide ATM)?
Not ontly currently active. Users also upgrade from just end-of-lifed distribution to a next one. It must also cover at least one distribution before the latest active one.
The way I see it, we should have a differently phrased requirement, ensuring that within an upstream version of a package, the release of a build in a higher Fedora release should be "newer" than of any previous build (for the same version) in an older release.
Yes. You are right. My requirement was too strong. Probably stemming from the fact that many packages do not recieve any version bump and the only changing part is the Release string.
-- Petr
On Fri, Jan 10, 2020 at 9:37 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
Thus our call for input to accept or reject the idea and if the former scope/define the system.
Whatever you decide, please try it out on a small set of packages that you personally maintain for a long time. This "field testing" will ensure that you're covering all the corner cases that you know about (and many of the ones that you do not).
With the packages I maintain with rdopkg, we generally do it the other way around: We write a human-readable %changelog entry, and then "rdopkg amend" will amend my dist-git commit log to match the text that I wrote in the .spec file.
- Ken
On Friday, January 10, 2020 5:36:46 PM CET Pierre-Yves Chibon wrote:
Do we want to drop release and changelog from our spec file?
No. People continuously tend to forget that '%changelog' is for end-users. Especially if some distributions already claim they can live fine without %changelog...
Unless product managers say that 'rpm -q --changelog' is not a thing nowadays, we should at least _allow_ being "nice" to end users. So whatever approach we use by default -- the maintainers still have to have a chance to maintain %changelog manually.
That said, to not loose the freedom, yes - we can implement some automation - but only if that can be turned off. By those who care about %changelog.
Regarding automation, I'm sceptic. See how GNU maintains ChangeLog files ... and how difficult is to edit the ChangeLog entries retrospectively when some automation breaks it. If people claim maintaining %changelog is too expensive so they want it generated -- having it generated is even more expensive. I mean if maintainer cares to have '-q --changelog' nice.
With the changelog it becomes a little bit more tricky. We currently have 3 changelogs in Fedora with 3 different target audience (this is how I understand them):
- One for the files in the git repository, meant to be "consumed" by our fellow packagers, not meant to leave the Fedora infrastructure
- One in the spec file describing the changes applied to it. This one is meant to be accessible to sysadmins who want to know/check what changed in a package
- One in bodhi, meant for end-user consumption and which should give some explanation as to why the package was updated or where information about the update can be found
In ideal world, shouldn't the bodhi change description be equivalent to %changelog, or at least a super-set of %changelog? If these were equivalent, maintainers woudl have to think more about %changelog.
So we need to, somehow, merge two changelogs into one while realizing that some information in one may not be desirable in the other (for example the world famous commit message: "oops I've forgot to upload the sources" does not need to appear in the RPM's changelog). Would it be easier to merge the git changelog with the spec changelog or the spec changelog with the bodhi notes?
spec changelog with bodhi notes
For the former one easy way to achieve this is to consider all the commits since the last successful build and have a magic keyword to either include or exclude a commit message in the changelog. For the latter, we discussed the idea of using annotated git tags this fall.
Both have their pros and cons, do people have some experience with either and could share their feedback?
See the GNU (e.g. gnulib) `make ChangeLog`. The annotated tags are IMO used in rpkg-util, and for regular git user they are "magic". People will start asking where the changelog is defined, how to change it, etc.
Pavel
Dne 21. 01. 20 v 9:36 Pavel Raiskup napsal(a):
On Friday, January 10, 2020 5:36:46 PM CET Pierre-Yves Chibon wrote:
Do we want to drop release and changelog from our spec file?
No. People continuously tend to forget that '%changelog' is for end-users. Especially if some distributions already claim they can live fine without %changelog...
I assume that there will be some tags which allow to exclude some specific messages from changelog. E.g. yesterday, I did two rebuilds of libguestfs:
https://src.fedoraproject.org/rpms/libguestfs/c/d33d482cf7b0cf4e1fa48d229be9...
https://koji.fedoraproject.org/koji/taskinfo?taskID=40786204
and
https://src.fedoraproject.org/rpms/libguestfs/c/cbdfa80fe9eb4a26a0807584114a...
https://koji.fedoraproject.org/koji/taskinfo?taskID=40786518
The first failed due to some Koji issues. Therefore I had to bump the release and do another one. If this situation happened when the changelog is generated from git log, I would expect that adding some keyword such as "[skip changelog]" (there are quite commonly used similar hints for CI nowadays [1]) would instruct the generator to leave the second commit out of the changelog, because it does not provide any additional value to user.
Unless product managers say that 'rpm -q --changelog' is not a thing nowadays, we should at least _allow_ being "nice" to end users. So whatever approach we use by default -- the maintainers still have to have a chance to maintain %changelog manually.
That said, to not loose the freedom, yes - we can implement some automation - but only if that can be turned off. By those who care about %changelog.
Regarding automation, I'm sceptic. See how GNU maintains ChangeLog files ... and how difficult is to edit the ChangeLog entries retrospectively when some automation breaks it. If people claim maintaining %changelog is too expensive so they want it generated -- having it generated is even more expensive. I mean if maintainer cares to have '-q --changelog' nice.
With the changelog it becomes a little bit more tricky. We currently have 3 changelogs in Fedora with 3 different target audience (this is how I understand them):
- One for the files in the git repository, meant to be "consumed" by our fellow packagers, not meant to leave the Fedora infrastructure
- One in the spec file describing the changes applied to it. This one is meant to be accessible to sysadmins who want to know/check what changed in a package
- One in bodhi, meant for end-user consumption and which should give some explanation as to why the package was updated or where information about the update can be found
In ideal world, shouldn't the bodhi change description be equivalent to %changelog, or at least a super-set of %changelog? If these were equivalent, maintainers woudl have to think more about %changelog.
Don't forget that single Bodhi update might ship several builds.
Vít
[1] https://docs.travis-ci.com/user/customizing-the-build/#skipping-a-build
So we need to, somehow, merge two changelogs into one while realizing that some information in one may not be desirable in the other (for example the world famous commit message: "oops I've forgot to upload the sources" does not need to appear in the RPM's changelog). Would it be easier to merge the git changelog with the spec changelog or the spec changelog with the bodhi notes?
spec changelog with bodhi notes
For the former one easy way to achieve this is to consider all the commits since the last successful build and have a magic keyword to either include or exclude a commit message in the changelog. For the latter, we discussed the idea of using annotated git tags this fall.
Both have their pros and cons, do people have some experience with either and could share their feedback?
See the GNU (e.g. gnulib) `make ChangeLog`. The annotated tags are IMO used in rpkg-util, and for regular git user they are "magic". People will start asking where the changelog is defined, how to change it, etc.
Pavel
On Tuesday, January 21, 2020 10:20:10 AM CET Vít Ondruch wrote:
I would expect that adding some keyword such as "[skip changelog]" (there are quite commonly used similar hints for CI nowadays [1]) would instruct the generator to leave the second commit out of the changelog, because it does not provide any additional value to user.
Yes, but people can forget to put there the keyword, and push - and later still affect the %changelog - and so we would have to have a way to adjust changelog by "patch" or something.
Again, as example, take a look at gnulib, to build-aux/gitlog-to-changelog and it treats *.amend files, --ignore-matching, --ignore-line, etc.
In ideal world, shouldn't the bodhi change description be equivalent to %changelog, or at least a super-set of %changelog? If these were equivalent, maintainers woudl have to think more about %changelog.
Don't forget that single Bodhi update might ship several builds.
Correct, no need to not provide all the %changelogs.
Pavel
On Tue, Jan 21, 2020 at 4:39 AM Pavel Raiskup praiskup@redhat.com wrote:
On Tuesday, January 21, 2020 10:20:10 AM CET Vít Ondruch wrote:
I would expect that adding some keyword such as "[skip changelog]" (there are quite commonly used similar hints for CI nowadays [1]) would instruct the generator to leave the second commit out of the changelog, because it does not provide any additional value to user.
Yes, but people can forget to put there the keyword, and push - and later still affect the %changelog - and so we would have to have a way to adjust changelog by "patch" or something.
Again, as example, take a look at gnulib, to build-aux/gitlog-to-changelog and it treats *.amend files, --ignore-matching, --ignore-line, etc.
In ideal world, shouldn't the bodhi change description be equivalent to %changelog, or at least a super-set of %changelog? If these were equivalent, maintainers woudl have to think more about %changelog.
Don't forget that single Bodhi update might ship several builds.
Correct, no need to not provide all the %changelogs.
I don't think people realize this, but while from DNF the updateinfo and changelog data are separate, when Bodhi generates update notices that are sent as emails and used in feeds, both pieces of information are presented together.
So ideally, there should be different information in updateinfo and changelog. The former is user-centric and the latter is packager centric.
On Tue, Jan 21, 2020 at 09:36:27AM +0100, Pavel Raiskup wrote:
On Friday, January 10, 2020 5:36:46 PM CET Pierre-Yves Chibon wrote:
Do we want to drop release and changelog from our spec file?
No. People continuously tend to forget that '%changelog' is for end-users. Especially if some distributions already claim they can live fine without %changelog...
Unless product managers say that 'rpm -q --changelog' is not a thing nowadays, we should at least _allow_ being "nice" to end users. So whatever approach we use by default -- the maintainers still have to have a chance to maintain %changelog manually.
rpm -q --changelog is a thing, which is why we're discussing the idea to remove the changelog from the *spec file* not from the (s)RPM.
And yes I agree that whichever approach is designed, it should be made opt-in, if only to allow to gather feedback and improve the process.
Pierre
On Tuesday, January 21, 2020 10:24:53 AM CET Pierre-Yves Chibon wrote:
On Tue, Jan 21, 2020 at 09:36:27AM +0100, Pavel Raiskup wrote:
On Friday, January 10, 2020 5:36:46 PM CET Pierre-Yves Chibon wrote:
Do we want to drop release and changelog from our spec file?
No. People continuously tend to forget that '%changelog' is for end-users. Especially if some distributions already claim they can live fine without %changelog...
Unless product managers say that 'rpm -q --changelog' is not a thing nowadays, we should at least _allow_ being "nice" to end users. So whatever approach we use by default -- the maintainers still have to have a chance to maintain %changelog manually.
rpm -q --changelog is a thing, which is why we're discussing the idea to remove the changelog from the *spec file* not from the (s)RPM.
And yes I agree that whichever approach is designed, it should be made opt-in, if only to allow to gather feedback and improve the process.
Thanks.
While we are on it.. did you consider to implement some nice Release/%changelog workflow in src.fedoraproject.org? I filled now https://pagure.io/pagure/issue/4714
Pavel
-
Dan Čermák -
David Cantrell -
David Kaufmann -
Fabio Valentini -
Felix Schwarz -
Florian Weimer -
Iñaki Ucar -
Joe Doss -
Jun Aruga -
Ken Dreyer -
Kevin Fenzi -
Marius Schwarz -
Michael Catanzaro -
Miro Hrončok -
Miroslav Suchý -
Neal Gompa -
Nicolas Mailhot -
Nils Philippsen -
Panu Matilainen -
Pavel Raiskup -
Petr Pisar -
Pierre-Yves Chibon -
Przemek Klosowski -
Randy Barlow -
Richard Shaw -
Richard W.M. Jones -
Stephen John Smoogen -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek