spec file changes: removing Release: and %changelog

[Colin Walters]

Hi,

I've spent a little bit of time on the shuttle to/from work hacking on
Makefile.common.  The general idea is to remove the Release: and %
changelog fields from spec files, and have them generated automatically.
The goals are:

* Allow for automated (re-)builds without hackish scripts
* Avoid developer time being spent incrementing integers in spec file
* Obviate the need to keep two changelogs in sync
* Resolve clashes about what disttags should be by having them
  determined by the build system 
* Longer term, make it easier to transition to a better source format

For example, with this change we can go from the workflow of:

edit foo.spec
make clog
cvs commit -F clog
make tag build
# discover error
edit foo.spec
make force-tag build

Now:

edit foo.spec2
make build
# discover error
edit foo.spec2
make build

Much nicer!  And we're being generous with the first system - if you
want to do it without the unsafe force-tag it's even longer.  In the new
system, you only need to know two commands: "make local" to do a local
build, and "make build" for Koji.

The attached patch is a work in progress; you can patch your
Makefile.common, delete the Release: line from your spec file, and
rename your spec file to .spec2 (suggestions for a better name are
appreciated).

It works for "make local" which gives you a feel for how the system can
operate.  You can verify that after every 'make local' you get a new RPM
release, and thus you can install them sanely on your system without
using --force.  One tweak I'd like to implement for 'make local' is have
it use '.local' or something for a disttag.

What remains to be implemented is the Koji side of things.  The idea is
that "make build" will run 'cvs commit; make tag koji', and that Koji
will have a database of package Release tags, and increment them itself
for each new build.  It will generate the %changelog section from the
CVS changelog between the last tag and the new one.

[Tom Callaway]

On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  Hi,
>  
>  I've spent a little bit of time on the shuttle to/from work hacking on
>  Makefile.common.  The general idea is to remove the Release: and %
>  changelog fields from spec files, and have them generated automatically. 
Changelog I might buy into, but Release? There are too many complicated
cases (Pre/Post immediately spring to mind) to automatically do this.
Also, we'd be forcing a hardcoded dist tag here. 

-1.

~spot

[Colin Walters]

On Wed, 2008-03-05 at 13:11 -0500, Tom "spot" Callaway wrote:
>  On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > Hi,
>  > 
>  > I've spent a little bit of time on the shuttle to/from work hacking on
>  > Makefile.common.  The general idea is to remove the Release: and %
>  > changelog fields from spec files, and have them generated automatically.
>  
>  Changelog I might buy into, but Release? There are too many complicated
>  cases (Pre/Post immediately spring to mind) to automatically do this. 
Can you explain what the problem is with Pre/Post?

>  Also, we'd be forcing a hardcoded dist tag here.  
There's nothing hardcoded; you should be able to override it.

[Jesse Keating]

On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  What remains to be implemented is the Koji side of things.  The idea
> is
>  that "make build" will run 'cvs commit; make tag koji', and that Koji
>  will have a database of package Release tags, and increment them itself
>  for each new build.  It will generate the %changelog section from the
>  CVS changelog between the last tag and the new one. 
We really don't want koji writing back to the CVS server.  There is a
lot of benefit to koji having only readonly access to the source
control.

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?

[Daniel P. Berrange]

On Wed, Mar 05, 2008 at 01:11:32PM -0500, Tom spot Callaway wrote:
>  On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > Hi,
>  > 
>  > I've spent a little bit of time on the shuttle to/from work hacking on
>  > Makefile.common.  The general idea is to remove the Release: and %
>  > changelog fields from spec files, and have them generated automatically.
>  
>  Changelog I might buy into, but Release? There are too many complicated
>  cases (Pre/Post immediately spring to mind) to automatically do this.
>  Also, we'd be forcing a hardcoded dist tag here.  
We should optimize for the common case - not the edge-case. I'd welcome
this automated release / changelog management. If a handful of packages
need to opt-out that's still a huge win for the common case.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Tom Callaway]

On Wed, 2008-03-05 at 13:15 -0500, Colin Walters wrote:
>  On Wed, 2008-03-05 at 13:11 -0500, Tom "spot" Callaway
> wrote:
>  > On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > > Hi,
>  > > 
>  > > I've spent a little bit of time on the shuttle to/from work hacking on
>  > > Makefile.common.  The general idea is to remove the Release: and %
>  > > changelog fields from spec files, and have them generated automatically.
>  > 
>  > Changelog I might buy into, but Release? There are too many complicated
>  > cases (Pre/Post immediately spring to mind) to automatically do this.
>  
>  Can you explain what the problem is with Pre/Post? 
How will the Makefile know whether the package is a Pre or Post release
package? See:

http://fedoraproject.org/wiki/Packaging/NamingGuidelines#head-5ea39bbc33c...

>  > Also, we'd be forcing a hardcoded dist tag here. 
>  
>  There's nothing hardcoded; you should be able to override it. 
+       (echo -n "Release: " && echo $${release}$(DIST) ...

You're hardcoding the value of the dist tag by default into the spec,
then generating the SRPM. That means that the SRPM will always have that
dist value, even if it is rebuilt.

Also, since this seems to happen on build operations, it means that
packages get a dist tag, even if the packager doesn't want one.

:(

~spot

[Till Maas]

On Wed March 5 2008, Colin Walters wrote:

>  The attached patch is a work in progress; you can patch your
>  Makefile.common, delete the Release: line from your spec file, and
>  rename your spec file to .spec2 (suggestions for a better name are
>  appreciated). 
Imho .spec.in would be a better name. Btw. it seems that you spec would not 
work with releases like:

0.1.pre2%{?dist}

1%{?dist}.1
to update a package in FN without it being newer that the package in FN+1 with 
release 1%{?dist}.

And also not scm snapshots.

Regards,
Till

[Colin Walters]

On Wed, 2008-03-05 at 13:17 -0500, Jesse Keating wrote:
>  On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > What remains to be implemented is the Koji side of things.  The idea is
>  > that "make build" will run 'cvs commit; make tag koji', and that
> Koji
>  > will have a database of package Release tags, and increment them itself
>  > for each new build.  It will generate the %changelog section from the
>  > CVS changelog between the last tag and the new one.
>  
>  We really don't want koji writing back to the CVS server.  There is a
>  lot of benefit to koji having only readonly access to the source
>  control. 
This system wouldn't require koji writing to the CVS server.

[Tom Callaway]

On Wed, 2008-03-05 at 18:18 +0000, Daniel P. Berrange wrote:
>  On Wed, Mar 05, 2008 at 01:11:32PM -0500, Tom spot Callaway wrote:
>  > On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > > Hi,
>  > > 
>  > > I've spent a little bit of time on the shuttle to/from work hacking on
>  > > Makefile.common.  The general idea is to remove the Release: and %
>  > > changelog fields from spec files, and have them generated automatically.
>  > 
>  > Changelog I might buy into, but Release? There are too many complicated
>  > cases (Pre/Post immediately spring to mind) to automatically do this.
>  > Also, we'd be forcing a hardcoded dist tag here. 
>  
>  We should optimize for the common case - not the edge-case. I'd welcome
>  this automated release / changelog management. If a handful of packages
>  need to opt-out that's still a huge win for the common case. 
Its not an optimization, its excluding all possible edge cases. The
kernel is a notable "edge case" here.

~spot

[John Dennis]

Thanks for the work Colin, spec file maintenance has always been 
needlessly difficult and error prone, this should go a long way to helping.

Daniel P. Berrange wrote:
>  We should optimize for the common case - not the edge-case. I'd
> welcome
>  this automated release / changelog management. If a handful of packages
>  need to opt-out that's still a huge win for the common case. 
+1
-- 
John Dennis <jdennis(a)redhat.com&gt;

[Dmitry Butskoy]

Oh, no...
>  I've spent a little bit of time on the shuttle to/from work hacking on
>  Makefile.common.  The general idea is to remove the Release: and %
>  changelog fields from spec files, 
And then I have to maintain two versions of the spec file -- one for 
Fedora build system, other for: another distros, another Fedora-related 
distros, local builds etc...
>   
>  * Allow for automated (re-)builds without hackish scripts
>     
Is that "hackish" my problem? ;)

>  * Avoid developer time being spent incrementing integers in spec
> file
>     
Interesting... Perhaps you will propose the automated "Version:" 
determined by the tarball? Or even tarball's URL?

>  * Obviate the need to keep two changelogs in sync
>     
Actually, I never fill the cvs changelog. Only the spec's changelog. 
Probably I'm wrong, but nobody claim me for it for a while.
>  edit foo.spec
>  make clog
>  cvs commit -F clog
>  make tag build
>  # discover error
>  edit foo.spec
>  make force-tag build
>     
BTW, maybe the better way is to determine the changes in the %changelog 
section (cvs diff etc.), and then create a particular cvs log by that 
chagnes?
>   In the new
>  system, you only need to know two commands: "make local" to do a local
>  build, 
For this, I will prefer to hack with "rpmbuild" at a separate place and 
to control what is happening immediately.


~buc
http://www.fedoraproject.org/wiki/DmitryButskoy

[Jesse Keating]

On Wed, 2008-03-05 at 13:20 -0500, Colin Walters wrote:
>  On Wed, 2008-03-05 at 13:17 -0500, Jesse Keating wrote:
>  > On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > > What remains to be implemented is the Koji side of things.  The idea is
>  > > that "make build" will run 'cvs commit; make tag koji', and
> that Koji
>  > > will have a database of package Release tags, and increment them itself
>  > > for each new build.  It will generate the %changelog section from the
>  > > CVS changelog between the last tag and the new one.
>  > 
>  > We really don't want koji writing back to the CVS server.  There is a
>  > lot of benefit to koji having only readonly access to the source
>  > control.
>  
>  This system wouldn't require koji writing to the CVS server. 
Sorry, I misread that.  Still, why would you pull from koji instead of
the cvs server itself?

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?

[Colin Walters]

On Wed, 2008-03-05 at 13:23 -0500, Jesse Keating wrote:
>  On Wed, 2008-03-05 at 13:20 -0500, Colin Walters wrote:
>  > On Wed, 2008-03-05 at 13:17 -0500, Jesse Keating wrote:
>  > > On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > > > What remains to be implemented is the Koji side of things.  The idea is
>  > > > that "make build" will run 'cvs commit; make tag koji',
> and that Koji
>  > > > will have a database of package Release tags, and increment them itself
>  > > > for each new build.  It will generate the %changelog section from the
>  > > > CVS changelog between the last tag and the new one.
>  > > 
>  > > We really don't want koji writing back to the CVS server.  There is a
>  > > lot of benefit to koji having only readonly access to the source
>  > > control.
>  > 
>  > This system wouldn't require koji writing to the CVS server.
>  
>  Sorry, I misread that.  Still, why would you pull from koji instead of
>  the cvs server itself? 
The overall idea is that the release number is determined by the "build
server", where "build server" can be either localhost (for make local),
or Koji (for Fedora).  So for local builds, you don't hit Koji to
determine a release - we just create one, and then increment it from
there.

We don't want to distribute locally-built RPMs as if they came from
Fedora, so giving them a .local disttag with a locally-determined
release is a good thing, I think.

[Jef Spaleta]

On Wed, Mar 5, 2008 at 9:28 AM, Colin Walters <walters(a)redhat.com&gt; wrote:
>   The overall idea is that the release number is determined by the
> "build
>   server", where "build server" can be either localhost (for make local),
>   or Koji (for Fedora).  So for local builds, you don't hit Koji to
>   determine a release - we just create one, and then increment it from
>   there.
>
>   We don't want to distribute locally-built RPMs as if they came from
>   Fedora, so giving them a .local disttag with a locally-determined
>   release is a good thing, I think. 
I think you have to abstract this a little and rely on a increment
counter macro in the release field to give maintainers the option of
using(or not) and potentially overriding.  That way maintainers can
encode the integer counter into the Release field in a way that makes
sense to keep the update path working...even for wacky 'edge' cases
like the kernel.

-jef

[Jesse Keating]

On Wed, 2008-03-05 at 13:28 -0500, Colin Walters wrote:
>  The overall idea is that the release number is determined by the
> "build
>  server", where "build server" can be either localhost (for make local),
>  or Koji (for Fedora).  So for local builds, you don't hit Koji to
>  determine a release - we just create one, and then increment it from
>  there.
>  
>  We don't want to distribute locally-built RPMs as if they came from
>  Fedora, so giving them a .local disttag with a locally-determined
>  release is a good thing, I think. 
But again, why is it by the build server?  You want to match the changes
to the spec with a number usually, even if that number doesn't get
built, or doesn't get successfully built.  This is why the kernel used
to key the release from the cvs check in number.  Of course that ran
afoul of branch collisions.

-- 
Jesse Keating RHCE      (jkeating.livejournal.com)
Fedora Project          (fedoraproject.org/wiki/JesseKeating)
GPG Public Key          (geek.j2solutions.net/jkeating.j2solutions.pub)

[Colin Walters]

On Wed, 2008-03-05 at 21:23 +0300, Dmitry Butskoy wrote:
>  Oh, no...
>  >
>  > I've spent a little bit of time on the shuttle to/from work hacking on
>  > Makefile.common.  The general idea is to remove the Release: and %
>  > changelog fields from spec files,
>  
>  And then I have to maintain two versions of the spec file -- one for 
>  Fedora build system, other for: another distros, another Fedora-related 
>  distros, local builds etc... 
In my experience, everyone has slightly forked versions of spec files
because they mandate different macros, etc.

I do in fact want to enable cross-distribution sharing of build scripts,
but I see that as a separate, longer term project from improving
Fedora's build process.

I actually started translating some spec files into a new build system
to solve other problems that Fedora has like rampant copy+paste of
scriptlets, etc.  You can see some examples of my thoughts here:

A simple one:
http://cdn.verbum.org/hotwire.py
A lot more complex:
http://cdn.verbum.org/hal.py

The idea is that you'd compile that into a spec file - we're not talking
about changing the binary formats in any way.  But, that's for the
future.  While starting on that I realized that it made more sense to
take an incremental approach of solving some of the more problematic
spec issues.

>  Interesting... Perhaps you will propose the automated
> "Version:" 
>  determined by the tarball? Or even tarball's URL? 
Actually, yes - that was one of the next steps =)

>  Actually, I never fill the cvs changelog. Only the spec's
> changelog.  
I don't blame you, because having two is very confusing.

>  For this, I will prefer to hack with "rpmbuild" at a
> separate place and 
>  to control what is happening immediately. 
Well, we can make 'make local' do whatever it is you want - what is
different?

[Tom Callaway]

On Wed, 2008-03-05 at 09:34 -0900, Jeff Spaleta wrote:
>  On Wed, Mar 5, 2008 at 9:28 AM, Colin Walters
> <walters(a)redhat.com&gt; wrote:
>  >  The overall idea is that the release number is determined by the "build
>  >  server", where "build server" can be either localhost (for make
> local),
>  >  or Koji (for Fedora).  So for local builds, you don't hit Koji to
>  >  determine a release - we just create one, and then increment it from
>  >  there.
>  >
>  >  We don't want to distribute locally-built RPMs as if they came from
>  >  Fedora, so giving them a .local disttag with a locally-determined
>  >  release is a good thing, I think.
>  
>  I think you have to abstract this a little and rely on a increment
>  counter macro in the release field to give maintainers the option of
>  using(or not) and potentially overriding.  That way maintainers can
>  encode the integer counter into the Release field in a way that makes
>  sense to keep the update path working...even for wacky 'edge' cases
>  like the kernel. 
There is some merit in this, but its not as easy as it seems.

Say that we have a %{counter} variable that gets magically incremented
based on the build, this better handles the edge cases, but what about
when we update to a newer version? 

Foo goes from 1.0.1 to 1.0.2. The Release should reset to 1. The
%{counter} needs to know this (while incrementing it forever is
possible, its ugly) and do the right thing.

~spot

[Colin Walters]

On Wed, 2008-03-05 at 13:34 -0500, Jesse Keating wrote:
>  On Wed, 2008-03-05 at 13:28 -0500, Colin Walters wrote:
>  > The overall idea is that the release number is determined by the "build
>  > server", where "build server" can be either localhost (for make
> local),
>  > or Koji (for Fedora).  So for local builds, you don't hit Koji to
>  > determine a release - we just create one, and then increment it from
>  > there.
>  > 
>  > We don't want to distribute locally-built RPMs as if they came from
>  > Fedora, so giving them a .local disttag with a locally-determined
>  > release is a good thing, I think.
>  
>  But again, why is it by the build server?  You want to match the changes
>  to the spec with a number usually, even if that number doesn't get
>  built, or doesn't get successfully built.  This is why the kernel used
>  to key the release from the cvs check in number.  Of course that ran
>  afoul of branch collisions. 
I see three options, in increasing complexity:

First, Koji could keep a mapping from its made-up Release integer to the
CVS commit tag.

Second, we could use some transformation of the CVS commit tag as the
Release.  

Third, we could switch to a revision control system with 

* Atomic commits
* A nice integer sequence mapping for commits on a branch

Subversion and Mercurial both have these properties; I'm not sure if git
has the second.

[Colin Walters]

On Wed, 2008-03-05 at 09:34 -0900, Jeff Spaleta wrote:
>  On Wed, Mar 5, 2008 at 9:28 AM, Colin Walters
> <walters(a)redhat.com&gt; wrote:
>  >  The overall idea is that the release number is determined by the "build
>  >  server", where "build server" can be either localhost (for make
> local),
>  >  or Koji (for Fedora).  So for local builds, you don't hit Koji to
>  >  determine a release - we just create one, and then increment it from
>  >  there.
>  >
>  >  We don't want to distribute locally-built RPMs as if they came from
>  >  Fedora, so giving them a .local disttag with a locally-determined
>  >  release is a good thing, I think.
>  
>  I think you have to abstract this a little and rely on a increment
>  counter macro in the release field to give maintainers the option of
>  using(or not) and potentially overriding.  That way maintainers can
>  encode the integer counter into the Release field in a way that makes
>  sense to keep the update path working...even for wacky 'edge' cases
>  like the kernel. 
My thought on this is that you can only switch to the new system after a
new upstream release.  That avoids the issue of backwards compatibility
mappings, because the new upstream version will override any previous
Release tags.

I don't know what's special about the kernel - can someone needs to
explain to me what need it has that wouldn't be met by this system?

[Colin Walters]

On Wed, 2008-03-05 at 13:19 -0500, Tom "spot" Callaway wrote:

>  How will the Makefile know whether the package is a Pre or Post
> release
>  package? See:
>  
>
> http://fedoraproject.org/wiki/Packaging/NamingGuidelines#head-5ea39bbc33c...
Ah, I thought you were talking about %pre/%post scripts =)

Ok.  So the root of the problem is here:

"When upstream uses versions that attempt to have meaning to humans
instead of being easy for a computer to order."

I think the first thing to at least try is to convince upstream to
switch to a compatible release sequence.

Failing that, the right thing in the new system would be for the
packaging process to include a script to transform the upstream version
sequence into one compatible with RPM.  Say versionscript.py.  That will
be run at build time on the Version: field to determine an
RPM-compatible version which will get included into the spec.

>  > > Also, we'd be forcing a hardcoded dist tag here. 
>  > 
>  > There's nothing hardcoded; you should be able to override it.
>  
>  +       (echo -n "Release: " && echo $${release}$(DIST) ...
>  
>  You're hardcoding the value of the dist tag by default into the spec,
>  then generating the SRPM. That means that the SRPM will always have that
>  dist value, even if it is rebuilt. 
Oh.  Doesn't it work to change $(DIST) there to be %{_dist}?

>  Also, since this seems to happen on build operations, it means that
>  packages get a dist tag, even if the packager doesn't want one. 
Do we want packagers to have a choice?

[Tom Callaway]

On Wed, 2008-03-05 at 13:55 -0500, Colin Walters wrote:
>  Failing that, the right thing in the new system would be for the
>  packaging process to include a script to transform the upstream
>  version
>  sequence into one compatible with RPM.  Say versionscript.py.  That
>  will
>  be run at build time on the Version: field to determine an
>  RPM-compatible version which will get included into the spec. 
Yes, but that script isn't going to be pretty. We're talking about some
rather hackish stuff that a human packager can pretty quickly
accomplish.

>  > > > Also, we'd be forcing a hardcoded dist tag here. 
>  > > 
>  > > There's nothing hardcoded; you should be able to override it.
>  > 
>  > +       (echo -n "Release: " && echo $${release}$(DIST) ...
>  > 
>  > You're hardcoding the value of the dist tag by default into the
>  spec,
>  > then generating the SRPM. That means that the SRPM will always have
>  that
>  > dist value, even if it is rebuilt.
>  
>  Oh.  Doesn't it work to change $(DIST) there to be %{_dist}? 
That would be much better...

>  > Also, since this seems to happen on build operations, it means
> that
>  > packages get a dist tag, even if the packager doesn't want one.
>  
>  Do we want packagers to have a choice? 
Absolutely.

~spot

[Colin Walters]

On Wed, 2008-03-05 at 13:20 -0500, Tom "spot" Callaway wrote:
>  On Wed, 2008-03-05 at 18:18 +0000, Daniel P. Berrange wrote:
>  > On Wed, Mar 05, 2008 at 01:11:32PM -0500, Tom spot Callaway wrote:
>  > > On Wed, 2008-03-05 at 13:03 -0500, Colin Walters wrote:
>  > > > Hi,
>  > > > 
>  > > > I've spent a little bit of time on the shuttle to/from work hacking
> on
>  > > > Makefile.common.  The general idea is to remove the Release: and %
>  > > > changelog fields from spec files, and have them generated automatically.
>  > > 
>  > > Changelog I might buy into, but Release? There are too many complicated
>  > > cases (Pre/Post immediately spring to mind) to automatically do this.
>  > > Also, we'd be forcing a hardcoded dist tag here. 
>  > 
>  > We should optimize for the common case - not the edge-case. I'd welcome
>  > this automated release / changelog management. If a handful of packages
>  > need to opt-out that's still a huge win for the common case.
>  
>  Its not an optimization, its excluding all possible edge cases. The
>  kernel is a notable "edge case" here. 
I'm not suggesting we immediately deploy the patch and force everyone to
use it - the goal was to prototype and get feedback, and this is exactly
the kind of issue that I wanted to know about.

The versionscript.py I mentioned in another response is one possible
solution.  Another would be to add a new field to the spec file like:

VersionTransform: pre
(or post).

That would tell the build system to tweak its Release: generating
accordingly.

[Till Maas]

On Wed March 5 2008, Colin Walters wrote:
>  On Wed, 2008-03-05 at 13:19 -0500, Tom "spot" Callaway
> wrote: 
>  > Also, since this seems to happen on build operations, it means
> that
>  > packages get a dist tag, even if the packager doesn't want one.
>
>  Do we want packagers to have a choice? 
Afaik, the dist tag is not used on packages that are very large and contain 
primary data, e.g. game data, that works in several releases and should not 
be updated when the release is updated.

Regards,
Till

[Colin Walters]

On Wed, 2008-03-05 at 21:33 +0100, Till Maas wrote:
>  On Wed March 5 2008, Colin Walters wrote:
>  > On Wed, 2008-03-05 at 13:19 -0500, Tom "spot" Callaway wrote:
>  
>  > > Also, since this seems to happen on build operations, it means that
>  > > packages get a dist tag, even if the packager doesn't want one.
>  >
>  > Do we want packagers to have a choice?
>  
>  Afaik, the dist tag is not used on packages that are very large and contain 
>  primary data, e.g. game data, that works in several releases and should not 
>  be updated when the release is updated. 
Ah, I see.  Ok, I'll add an option to override in the next patch.

[David Malcolm]

On Wed, 2008-03-05 at 13:37 -0500, Colin Walters wrote:
>  On Wed, 2008-03-05 at 21:23 +0300, Dmitry Butskoy wrote:
>  > Oh, no...
>  > >
>  > > I've spent a little bit of time on the shuttle to/from work hacking on
>  > > Makefile.common.  The general idea is to remove the Release: and %
>  > > changelog fields from spec files,
>  > 
>  > And then I have to maintain two versions of the spec file -- one for 
>  > Fedora build system, other for: another distros, another Fedora-related 
>  > distros, local builds etc...
>  
>  In my experience, everyone has slightly forked versions of spec files
>  because they mandate different macros, etc.
>  
>  I do in fact want to enable cross-distribution sharing of build scripts,
>  but I see that as a separate, longer term project from improving
>  Fedora's build process.
>  
>  I actually started translating some spec files into a new build system
>  to solve other problems that Fedora has like rampant copy+paste of
>  scriptlets, etc.  You can see some examples of my thoughts here:
>  
>  A simple one:
>  http://cdn.verbum.org/hotwire.py
>  A lot more complex:
>  http://cdn.verbum.org/hal.py
>  
>  The idea is that you'd compile that into a spec file - we're not talking
>  about changing the binary formats in any way.  But, that's for the
>  future.  While starting on that I realized that it made more sense to
>  take an incremental approach of solving some of the more problematic
>  spec issues. 
Very cool - the DRY principle in action.  Seems somewhat reminiscent of
conary recipes.

Do you having working code yet to generate the specfile, or is this just
an idea of syntax?

[snip]

[Jim Meyering]

Colin Walters <walters(a)redhat.com&gt; wrote:
...
>  Third, we could switch to a revision control system with
>
>  * Atomic commits
>  * A nice integer sequence mapping for commits on a branch
>
>  Subversion and Mercurial both have these properties; I'm not sure if git
>  has the second. 
It does.  For example, in my coreutils.git tree, running "git describe HEAD"
prints "v6.10-108-gd44893c", which indicates that is the 108th change
set since the v6.10 tag, and its SHA1 starts with "d44893c".

Doing the same for the previous SHA1:

    $ git describe b7a836c0a3524cda8ef79c30c3fe7ea759ae4656
    v6.10-107-gb7a836c

[Colin Walters]

On Wed, 2008-03-05 at 16:22 -0500, David Malcolm wrote:

>  Very cool - the DRY principle in action.  Seems somewhat reminiscent
> of
>  conary recipes. 
Yeah, it's inspired by the conary recipies.

>  Do you having working code yet to generate the specfile, or is this
> just
>  an idea of syntax? 
Those files were another shuttle trip =)  No compiler yet.  I realized
after doing them that it'd be a lot more bang-for-the-buck to do some
incremental changes to the existing spec format.

[Kevin Kofler]

Colin Walters <walters <at> redhat.com> writes:
>  I actually started translating some spec files into a new build
> system
>  to solve other problems that Fedora has like rampant copy+paste of
>  scriptlets, etc.  You can see some examples of my thoughts here:
>  
>  A simple one:
>  http://cdn.verbum.org/hotwire.py
>  A lot more complex:
>  http://cdn.verbum.org/hal.py 
So you're suggesting that in addition to knowing RPM's specfile syntax, 
packagers now have to know Python too? That's going to eliminate all the 
non-programmer packagers and also be a PITA for those of us who don't primarily 
program in Python.

        Kevin Kofler

[Kevin Kofler]

Dmitry Butskoy <buc <at> odusz.so-cdu.ru> writes:
>  > * Obviate the need to keep two changelogs in sync
>  >   
>  
>  Actually, I never fill the cvs changelog. Only the spec's changelog. 
>  Probably I'm wrong, but nobody claim me for it for a while. 
My solution: select the RPM changelog entry I just made, click Commit in 
Cervisia, middle-click-paste the selected changelog entry into the commit 
message, click OK. Where's the problem?

        Kevin Kofler

[Kevin Kofler]

Colin Walters <walters <at> redhat.com> writes:
>  I've spent a little bit of time on the shuttle to/from work
> hacking on
>  Makefile.common.  The general idea is to remove the Release: and %
>  changelog fields from spec files, and have them generated automatically. 
Sorry, but IMHO, for reasons already explained by others in this thread, this 
is a solution looking for a problem.

        Kevin Kofler

[Daniel P. Berrange]

On Wed, Mar 05, 2008 at 01:37:47PM -0500, Colin Walters wrote:
>  On Wed, 2008-03-05 at 21:23 +0300, Dmitry Butskoy wrote:
>  > Oh, no...
>  > >
>  > > I've spent a little bit of time on the shuttle to/from work hacking on
>  > > Makefile.common.  The general idea is to remove the Release: and %
>  > > changelog fields from spec files,
>  > 
>  > And then I have to maintain two versions of the spec file -- one for 
>  > Fedora build system, other for: another distros, another Fedora-related 
>  > distros, local builds etc...
>  
>  In my experience, everyone has slightly forked versions of spec files
>  because they mandate different macros, etc.
>  
>  I do in fact want to enable cross-distribution sharing of build scripts,
>  but I see that as a separate, longer term project from improving
>  Fedora's build process.
>  
>  I actually started translating some spec files into a new build system
>  to solve other problems that Fedora has like rampant copy+paste of
>  scriptlets, etc.  You can see some examples of my thoughts here:
>  
>  A simple one:
>  http://cdn.verbum.org/hotwire.py
>  A lot more complex:
>  http://cdn.verbum.org/hal.py 
The hal example is utterly disgusting

>  
>  The idea is that you'd compile that into a spec file - we're not talking
>  about changing the binary formats in any way.  
This looks like completely the wrong way to solve the copy+paste scriptlets
problem. Turning a declarative configuration format into a programming
language simply to eliminate common config is just nuts. 

If we want to eliminate the common scriptlets, then RPM should get richer 
metadata. By auto-generating the spec files from python you're just pushing
the problem elsewhere & whenever the python wrapper needs to change the
scriptlets you need to re-generate all the distro's spec files / RPMS.

The vast majority of scriptlets can be eliminated if RPM knew what the
file type is - eg, if it sees an ELF library, it should automatically
run ldconfig. If it see a desktop file it should automatically run
the desktop file install script, etc, etc. If done right you can add
more scriptlets / change existing scriptlets for any existing RPM
without needing to re-generate the RPM / spec.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Till Maas]

On Wed March 5 2008, Dmitry Butskoy wrote:

>  Actually, I never fill the cvs changelog. Only the spec's
> changelog.
>  Probably I'm wrong, but nobody claim me for it for a while. 
With the Makefile.common Addons[1] you can use "make commit" to fill the 
commit message with the spec changelog for the current version. It uses the 
clog file from "make clog", which you could use with cvs commit -F clog iirc.

Regards,
Till

[1] 
http://fedoraproject.org/wiki/PackageMaintainers/UsefulScripts#head-74286...

[Matej Cepl]

On 2008-03-05, 18:03 GMT, Colin Walters wrote:
>  The general idea is to remove the Release: and %
>  changelog fields from spec files, and have them generated automatically. 
Aside from negative comments other people have in this thread 
I would that we have bigger problems than bumping up Release: 
manually -- why should everybody fill BuildRoot manually with 
predefined value-de-jour which he copies paste from wiki? This 
whole variable should go in the ideal world, IMHO.

Matěj

[Debarshi Ray]

>   I've spent a little bit of time on the shuttle to/from work
> hacking on
>   Makefile.common.  The general idea is to remove the Release: and %
>   changelog fields from spec files, and have them generated automatically.
>   The goals are:
>
>  [...]
>
>   * Obviate the need to keep two changelogs in sync 
As Dmitry said, I would rather have the cvs log generated from the
%changelog because:

(1) I don't want my Spec files to be "controlled" or "written to"
by
an automated Fedora-specific thing.

(2) I use 'cd /devel/redhat/SPECS && rpmbuild -ba foo.spec' to build
my packages far too many times than the other alternatives. When I
encounter any issues or do some modification, it is far too easy for
me to locally update the %changelog than remember what I did and then
provide all of them during 'cvs commit'. I really do not want to make
a 'cvs commit' every time I make any change to the Spec because for
all I know I might just have to revert the change 30 seconds later.

On the other hand, I would like a way to generate the cvs commit
message from the Spec's %changelog. Something like 'make commit'?

Cheers,
Debarshi
-- 
"From what we get, we can make a living; what we give, however, makes a life."
    -- Arthur Ashe

[Till Maas]

On Thu March 6 2008, Debarshi Ray wrote:

>  On the other hand, I would like a way to generate the cvs commit
>  message from the Spec's %changelog. Something like 'make commit'? 
You can use either

a) Makefile.common Addons to get "make commit":

http://fedoraproject.org/wiki/PackageMaintainers/UsefulScripts#head-74286...

b) make clog; cvs commit -F clog

Regards,
Till

[Nicolas Mailhot]

>  Colin Walters <walters <at> redhat.com> writes:
>  > I actually started translating some spec files into a new build system
>  > to solve other problems that Fedora has like rampant copy+paste of
>  > scriptlets, etc.  You can see some examples of my thoughts here:
>  > 
>  > A simple one:
>  > http://cdn.verbum.org/hotwire.py
>  > A lot more complex:
>  > http://cdn.verbum.org/hal.py 
This is atrocious and quite worse than the XML spec variant that was
kicked out of rpm recently.

If you're going to rewrite the spec syntax I'll ask that:

1. this new syntax is discussed and supported at rpm.org level not at
the Fedora rpm build tool level (ie not a template used to generate
something else à la sendmail, scrollkeeper, etc but the real thing)

2. It's not so obviously welded to a particular computing language, but
is a real descriptive syntax (this bit the xml spec guys had at least
gotten right)

3. this rewrite is the occasion of fixing long-standing problems in spec
syntax such as grouping declaration, disttags/alphatag support, version
range support,clear encoding rules [insert your pet spec peeve there].
And not just an xml-ization, python-ification or whatever of the
existing pile of cruft (haven't you learned nothing from the OOXML
debacle?)

In short this looks awfully like a short-term win for the build tools
and a long-term PITA for everyone else.

-- 
Nicolas Mailhot

[Lubomir Kundrak]

On Wed, 2008-03-05 at 13:42 -0500, Colin Walters wrote:

>  Third, we could switch to a revision control system with 
>  
>  * Atomic commits 
Hm, and your idea is to bump the release on each commit? What if I want
to tag and increment the revision only when building. My workflow is:

Bump revision
Do a change, add it to changlog, commit.
...
Do a change, add it to changlog, commit.
Tag, Build.

How would atomic commits be possibly helpful to this?

-- 
Lubomir Kundrak (Red Hat Security Response Team)

[Rahul Sundaram]

Till Maas wrote:
>  On Thu March 6 2008, Debarshi Ray wrote:
>  
> > On the other hand, I would like a way to generate the cvs commit
> > message from the Spec's %changelog. Something like 'make commit'?
>  
>  You can use either
>  
>  a) Makefile.common Addons to get "make commit":
>  
>
> http://fedoraproject.org/wiki/PackageMaintainers/UsefulScripts#head-74286...
>  
>  b) make clog; cvs commit -F clog 
Can you make an effort to get this all merged into the makefile in 
Fedora CVS? I suspect that a lot of package maintainers are not aware of 
this wiki page/specific enhancement and are losing out the benefits.

Rahul

[Michael Schwendt]

On Thu, 06 Mar 2008 16:37:48 +0530, Rahul Sundaram wrote:

>  Till Maas wrote:
>  > On Thu March 6 2008, Debarshi Ray wrote:
>  > 
>  >> On the other hand, I would like a way to generate the cvs commit
>  >> message from the Spec's %changelog. Something like 'make commit'?
>  > 
>  > You can use either
>  > 
>  > a) Makefile.common Addons to get "make commit":
>  > 
>  >
> http://fedoraproject.org/wiki/PackageMaintainers/UsefulScripts#head-74286...
>  > 
>  > b) make clog; cvs commit -F clog
>  
>  Can you make an effort to get this all merged into the makefile in 
>  Fedora CVS? 
"make clog" is included for a very long time already (~several years).

[Rahul Sundaram]

Michael Schwendt wrote:
>  On Thu, 06 Mar 2008 16:37:48 +0530, Rahul Sundaram wrote:
>  
> > Till Maas wrote:
> >> On Thu March 6 2008, Debarshi Ray wrote:
> >>
> >>> On the other hand, I would like a way to generate the cvs commit
> >>> message from the Spec's %changelog. Something like 'make
> commit'?
> >> You can use either
> >>
> >> a) Makefile.common Addons to get "make commit":
> >>
> >>
> http://fedoraproject.org/wiki/PackageMaintainers/UsefulScripts#head-74286...
> >>
> >> b) make clog; cvs commit -F clog
> > Can you make an effort to get this all merged into the makefile in 
> > Fedora CVS?
>  
>  "make clog" is included for a very long time already (~several years). 
I quoted slightly incorrectly. I was talking about make commit.

Rahul

[Debarshi Ray]

Shame on me that I never cared to look at common/Makefile.common.
There are a lot of cool things there. :-)

However, since we already have 'make clog', what is stopping us from
having a 'make commit'? In my limited view, if one is creating a clog
then he is almost certain to commit to the VCS. It is almost as if
clog is redundant. Why would I want to have the topmost entry of the
changelog in a separate file, even when I have the entire changelog in
the Spec, and not make a commit?

So modifying Colin's original use case scenarios:

>  edit foo.spec
>  make clog
>  cvs commit -F clog
>  make tag build
>  # discover error
>  edit foo.spec
>  make force-tag build
>
>  Now:
>
>  edit foo.spec2
>  make build
>  # discover error
>  edit foo.spec2
>  make build 
...we can have:

edit foo.spec
make commit tag build
# discover error
edit foo.spec
make commit force-tag build

Till now we have assumed that every commit will have a new changelog
entry which can be associated with that commit. Have we considered the
odd commit that a packager might make to fix some trivial issue (eg.,
tabs/spaces) that does not warrant a tag, build, update or a separate
entry in the %changelog?

Maybe we can have a 'make fix' for that? I don't know.

Happy hacking,
Debarshi
-- 
"From what we get, we can make a living; what we give, however, makes a life."
    -- Arthur Ashe

[Till Maas]

On Thu March 6 2008, Debarshi Ray wrote:
>  Shame on me that I never cared to look at common/Makefile.common.
>  There are a lot of cool things there. :-)
>
>  However, since we already have 'make clog', what is stopping us from
>  having a 'make commit'? In my limited view, if one is creating a clog 
Someone needs to write a patch, submit it to  
fedora-infrastructure-list and maybe send a reminder in case the patch was not 
commited to Makefile.common.

>  then he is almost certain to commit to the VCS. It is almost as if
>  clog is redundant. Why would I want to have the topmost entry of the
>  changelog in a separate file, even when I have the entire changelog in
>  the Spec, and not make a commit? 
Maybe someone wants to edit the clog file before commit, e.g. because not 
every change in the specs changelog was done in the current commit, but maybe 
something earlier. Or for some reason something else should be added.

>  edit foo.spec
>  make commit tag build
>  # discover error
>  edit foo.spec
>  make commit force-tag build
>
>  Till now we have assumed that every commit will have a new changelog
>  entry which can be associated with that commit. Have we considered the
>  odd commit that a packager might make to fix some trivial issue (eg.,
>  tabs/spaces) that does not warrant a tag, build, update or a separate
>  entry in the %changelog? 
I do not see what the problem here is. It is possible to run "make commit" 
without doing a tag and build.

Regards,
Till

[Colin Walters]

On Thu, 2008-03-06 at 00:23 +0000, Kevin Kofler wrote:
>  Colin Walters <walters <at> redhat.com> writes:
>  > I've spent a little bit of time on the shuttle to/from work hacking on
>  > Makefile.common.  The general idea is to remove the Release: and %
>  > changelog fields from spec files, and have them generated automatically.
>  
>  Sorry, but IMHO, for reasons already explained by others in this thread, this 
>  is a solution looking for a problem. 
I pretty clearly listed the problems - do you disagree that they are
problems, or that this approach will solve them?

[Colin Walters]

On Thu, 2008-03-06 at 00:32 +0000, Daniel P. Berrange wrote:

>  The hal example is utterly disgusting 
You have looked at the existing hal.spec, right?  I don't think any
version of hal packaging is going to be pretty, at least if you're
operating under the constraint of generating compatible RPMs.  The
hal-libs split is weird, and the architecture deps are a mess.  Though
the latter could be cleaned up with a dpkg-like mini-language for them
perhaps instead of arbitrary code.

>  This looks like completely the wrong way to solve the copy+paste
> scriptlets
>  problem. Turning a declarative configuration format into a programming
>  language simply to eliminate common config is just nuts.  
It's just as declarative as spec files are.  Which is to say, it's
mixed.

>  If we want to eliminate the common scriptlets, then RPM should get
> richer 
>  metadata. By auto-generating the spec files from python you're just pushing
>  the problem elsewhere  
Right - onto people who care about them and know the details, not the
responsibility of someone who wants to add a new package that happens to
use a desktop file.

>  & whenever the python wrapper needs to change the
>  scriptlets you need to re-generate all the distro's spec files / RPMS. 
Nothing about this approach would preclude installing the scripts system
wide and generating code to call them.  I think that's actually a good
idea - we tried to do that with desktop-file-install, for example.

>  The vast majority of scriptlets can be eliminated if RPM knew what
> the
>  file type is - eg, if it sees an ELF library, it should automatically
>  run ldconfig. If it see a desktop file it should automatically run
>  the desktop file install script, etc, etc. If done right you can add
>  more scriptlets / change existing scriptlets for any existing RPM
>  without needing to re-generate the RPM / spec. 
Honestly, I don't have a strong opinion on which layer in the system
this intelligence should live.  If the RPM maintainers think that it
belongs in RPM, that's fine by me.  I think we all do agree that we want
a more automatic system with possible overrides.

[Richard W.M. Jones]

On Thu, Mar 06, 2008 at 12:32:22AM +0000, Daniel P. Berrange wrote:
>  The vast majority of scriptlets can be eliminated if RPM knew what
> the
>  file type is - eg, if it sees an ELF library, it should automatically 
... an ELF library installed in a system directory.  No point in
running ldconfig for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so.

Probably the best thing is to have a

  %ldconfig

directive.

>  run ldconfig. If it see a desktop file it should automatically run
>  the desktop file install script, etc, etc. If done right you can add
>  more scriptlets / change existing scriptlets for any existing RPM
>  without needing to re-generate the RPM / spec. 
But yes, in general many scriptlets can be replaced with automatic
detection or some sort of directive.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top

[Nicolas Mailhot]

Le jeudi 06 mars 2008 à 22:04 -0500, Colin Walters a écrit :
>  On Thu, 2008-03-06 at 00:23 +0000, Kevin Kofler wrote:
>  > Colin Walters <walters <at> redhat.com> writes:
>  > > I've spent a little bit of time on the shuttle to/from work hacking on
>  > > Makefile.common.  The general idea is to remove the Release: and %
>  > > changelog fields from spec files, and have them generated automatically.
>  > 
>  > Sorry, but IMHO, for reasons already explained by others in this thread, this 
>  > is a solution looking for a problem.
>  
>  I pretty clearly listed the problems - do you disagree that they are
>  problems, or that this approach will solve them? 

The problem with your proposal is not that is will solve your problems,
the problem is it will create others you may not care about but others
do.

-- 
Nicolas Mailhot

[Kevin Kofler]

Colin Walters <walters <at> redhat.com> writes:
>  I pretty clearly listed the problems - do you disagree that they are
>  problems, or that this approach will solve them? 
I disagree that they are problems:

>  * Allow for automated (re-)builds without hackish scripts 
The scripts we have work fine (there was only one issue discovered during the 
GCC 4.3 mass rebuilds, which was with the *jpp.*.fc* scheme, and that has 
already been fixed), and if you want maintainers to uniformly accept your 
automated Release tag generation, you will have to implement the exact same 
release bumping schemes those scripts implement.

>  * Avoid developer time being spent incrementing integers in spec file
That's usually a 1-character change, sometimes a 2-character change, it takes 
10 seconds at worst, I fail to see the problem there.

>  * Obviate the need to keep two changelogs in sync 
That's solved by a one (middle) click paste or by solutions like "make clog"
or 
the proposed "make commit". I don't see the real problem there either.

>  * Resolve clashes about what disttags should be by having them
>   determined by the build system 
%{?dist} is universally agreed as the standard these days (except for the cases 
which don't need a disttag at all, like huge noarch binaries of data which are 
just untarred and need no compilation at all, so they can just be carried over 
from release to release) and it's clear what it expands to.

>  * Longer term, make it easier to transition to a better source format
Removing only Release and %changelog from the specfile won't change much there.


In addition, I think your approach creates new unsolved problems which have 
already been mentioned:
* how to pick the versioning scheme to use: at least prerelease packages need a 
special scheme, then there's the branch-only fixes (*.fc*.*) and the 
JPackage-compatible versioning (*jpp.*.fc*) which are also in common use.
* in schemes like prereleases, how to define the prerelease tag ("alphatag") to

use.
* Release number inflation for no good reason (if we fix something in a build 
which failed, it doesn't need a new Release number).
* specfiles needing modification when built outside of the context of the build 
system.

        Kevin Kofler

[Colin Walters]

On Fri, 2008-03-07 at 10:03 +0000, Kevin Kofler wrote:

>  > * Allow for automated (re-)builds without hackish scripts
>  
>  The scripts we have work fine  
I've been at Red Hat quite a long time now, and automated rebuilds have
been a problem that entire time.  This problem is created by the spec
file trying to be a database of build IDs (Revision); there's no innate
reason rebuilding software has to be so painful.

There were still some problems even with the latest rebuilding script,
not because the script was wrong (AFAIK) but because following those
guidelines is very complicated in edge cases, and people get it wrong.

Another way to look at this is - one should never have files modified
both by humans and automated processes stored in revision control.  If
you do, it's a recipe for pointless conflicts and pain.

>  (there was only one issue discovered during the 
>  GCC 4.3 mass rebuilds, which was with the *jpp.*.fc* scheme, and that has 
>  already been fixed), and if you want maintainers to uniformly accept your 
>  automated Release tag generation, you will have to implement the exact same 
>  release bumping schemes those scripts implement. 
Yes, exactly!  We implement it in one place, and don't give a chance for
humans to mess it up.

>  > * Avoid developer time being spent incrementing integers in spec
> file
>  
>  That's usually a 1-character change, sometimes a 2-character change, it takes 
>  10 seconds at worst, I fail to see the problem there. 
The real thing this blocks is more automation in our processes.  I don't
want to have to edit a spec file at all when a new release comes out; we
should have a system where I can look on a web page and see "New
upstream releases of foo, bar available", and click one link to pull
them in for tentative builds and tests, and then if that works tag them.

>  > * Obviate the need to keep two changelogs in sync
>  
>  That's solved by a one (middle) click paste or by solutions like "make
> clog" or 
>  the proposed "make commit". I don't see the real problem there either.
Even with "make commit", there are still two separate changelogs, and
they will be out of sync.  We really don't want more than one way to do
things.

>  In addition, I think your approach creates new unsolved problems
> which have 
>  already been mentioned:
>  * how to pick the versioning scheme to use: at least prerelease packages need a 
>  special scheme,  
As I replied to Tom, these kinds of edge cases can be encoded in a
separate header in the spec, or probably even better in an improved
version of the "sources" file; see my replies to him and also below.

>  then there's the branch-only fixes (*.fc*.*)  
The build system can determine this automatically - remember, we have a
database of all builds.  Again, one less thing for a human to mess up.

>  and the 
>  JPackage-compatible versioning (*jpp.*.fc*) which are also in common use. 
If we want that, it can be handled by a new header such as
"JpackageCompatibleRelease: yes".

>  * in schemes like prereleases, how to define the prerelease tag
> ("alphatag") to 
>  use. 
I like the idea for this one of extending the "sources" file - say a
third field which denotes whether a file is a pre or post release.

>  * Release number inflation for no good reason (if we fix something in
> a build 
>  which failed, it doesn't need a new Release number). 
But Release numbers right now aren't guaranteed to correspond to
anything at all.  They often approximate the number of successful
builds, but...note that with the proposal to get rid of force-tag, you
will no longer be able to make Release correspond one to one with
successful builds even in our current system.

>  * specfiles needing modification when built outside of the context of
> the build 
>  system. 
No, "make srpm" (and by extension "make local") continues to work.

[Debarshi Ray]

> >> * Obviate the need to keep two changelogs in sync 
> > That's solved by a one (middle) click paste or by solutions
> like 
"make clog" or
> > the proposed "make commit". I don't see the real
> problem there either. 
>   Even with "make commit", there are still two separate
> changelogs, and
>   they will be out of sync.  We really don't want more than one way to do
>   things. 
Although quite a few projects use things like 'git log' to generate
the ChangeLog, many still keep them in sync manually. In our case, I
prefer to write the Spec manually and let the rest sort themselves out
either manually or in some other automated way. Apart from the other
reasons that I had mentioned, here is another one.

Suppose Foo Bar submits a review request. The package is not yet in
the CVS so she does not have ready access to the proposed targets
which will generate a %changelog for her. So he does it manually
althrough the review process, but when he imports it into the CVS he
is recommended to remove all the handwritten stuff, because when he
commits, the scripts are going to overwrite all the manually written
stuff. Is it not messy?

Cheerio,
Debarshi
-- 
"From what we get, we can make a living; what we give, however, makes a life."
    -- Arthur Ashe

[Toshio Kuratomi]

Colin Walters wrote:
>  On Fri, 2008-03-07 at 10:03 +0000, Kevin Kofler wrote:
>  There were still some problems even with the latest rebuilding script,
>  not because the script was wrong (AFAIK) but because following those
>  guidelines is very complicated in edge cases, and people get it wrong. 
[snip]

> > (there was only one issue discovered during the 
> > GCC 4.3 mass rebuilds, which was with the *jpp.*.fc* scheme, and that has 
> > already been fixed), and if you want maintainers to uniformly accept your 
> > automated Release tag generation, you will have to implement the exact same 
> > release bumping schemes those scripts implement.
>  
>  Yes, exactly!  We implement it in one place, and don't give a chance for
>  humans to mess it up.
>   
Errr... Read the guidelines again.  That is a valid tag due to java 
packagers having made the case that there is a need to interleave with 
jpackage.  If the buildsystem were to make the release tag up it would 
have to decide whether the packager wanted to stay interleavable with 
jpackage and if so find out what the present jpackage version-release is 
and build our release tag from that.

Or you can try to convince the Java Packagers that interleaving with 
jpackage is not a desirable thing... but we've already been through that 
once so you better have some good arguments :-)

> >> * Avoid developer time being spent incrementing integers in
> spec file
> > That's usually a 1-character change, sometimes a 2-character change, it takes 
> > 10 seconds at worst, I fail to see the problem there.
>  
>  The real thing this blocks is more automation in our processes.  I don't
>  want to have to edit a spec file at all when a new release comes out; we
>  should have a system where I can look on a web page and see "New
>  upstream releases of foo, bar available", and click one link to pull
>  them in for tentative builds and tests, and then if that works tag them.
>   
That would be nice.  It would also have to check that the configure 
script hasn't changed beyond the package's version update (for autoconf 
using build systems... not sure that other build systems will accurately 
capture build requirements in one spot :-( ).  Otherwise, you silently 
go without a new optional feature because configure didn't find it at 
build time.

>  
> > In addition, I think your approach creates new unsolved problems which have 
> > already been mentioned:
> > * how to pick the versioning scheme to use: at least prerelease packages need a 
> > special scheme, 
>  
>  As I replied to Tom, these kinds of edge cases can be encoded in a
>  separate header in the spec, or probably even better in an improved
>  version of the "sources" file; see my replies to him and also below.
>   
But then you're right back to specifying release information in the spec 
file :-(.  (It's more elegant in that it's not encoding the build order 
in the spec but it's still annoying to the packager as they have to add 
it to the file when things change.)  Implemented wrong, it also could be 
a special case to be used and then forgotten... Having the header always 
present would be good:
ReleaseStyle: (standard|manual|jpackage|prerelease|postrelease)[ ,]* \
     (standard|manual|jpackage|prerelease|postrelease)

> > then there's the branch-only fixes (*.fc*.*) 
>  
>  The build system can determine this automatically - remember, we have a
>  database of all builds.  Again, one less thing for a human to mess up.
>   
Err... it can determine that the last build was a branch-only build but 
how can it determine that the current build is a branch only build?

> > * in schemes like prereleases, how to define the prerelease tag
> ("alphatag") to 
> > use.
>  
>  I like the idea for this one of extending the "sources" file - say a
>  third field which denotes whether a file is a pre or post release.
>   
-1

This is one of the differences between Debian dpkg and rpm that I like 
about rpm.  All the build information is in one file.  Our current 
sources file is generated automatically and never needs to be edited by 
the package builder.  It is only there so the buildsystem can 
differentiate between two tarballs that have the same name.

> > * specfiles needing modification when built outside of the
> context of the build 
> > system.
>  
>  No, "make srpm" (and by extension "make local") continues to work.
>   
You're both right.  make srpm could be enhanced to do something 
approaching correct.  But just taking the spec file out of Fedora's cvs 
and attempting to build it won't work any longer.  You no longer can 
build with just rpmbuild, a spec, and a tarball.  You need the Fedora 
Makefiles as well.

Also, btw, your make local doesn't do the right thing WRT disttag. 
disttag is supposed to tell which distribution the package is built for, 
not which repo it is built by.  So disttag == .fc8 means the package was 
built for and intended to run on Fedora 8.  .local is a repotag rather 
than a disttag.  Additionally, whatever it expands to should sort lower 
than the equivalent Fedora package built in the buildsystem.  Otherwise 
you, the package maintainer, have extra steps to perform to go from 
testing your local built package to testing the package built on the 
buildsystem.  (Whereas if it sorts lower, the buildsystem package is 
picked up by the normal yum update).

I like the ideas for change that you're putting out here but not 
necessarily the implementations :-).  Things that can be done from 
within the spec file to affect the buildsystem are vastly preferable to 
me as our current model is that the srpm is the basic, indivisible unit 
that we want to work with.

If you don't see a way to work within that constraint then we as a 
project need to evaluate whether we want to continue to base ourselves 
on the srpm or if we're going to move to a new format as our base (with 
the possibility of generating srpms as an intermediate format).  We 
might take half-steps like you propose to get there but we need to 
decide that that's the direction we want to head.

-Toshio

[Colin Walters]

On Fri, 2008-03-07 at 08:51 -0800, Toshio Kuratomi wrote:

(reordering your reply here)

>  I like the ideas for change that you're putting out here but not
>
>  necessarily the implementations :-).  Things that can be done from 
>  within the spec file to affect the buildsystem are vastly preferable to 
>  me as our current model is that the srpm is the basic, indivisible unit 
>  that we want to work with.
>  
>  If you don't see a way to work within that constraint then we as a 
>  project need to evaluate whether we want to continue to base ourselves 
>  on the srpm or if we're going to move to a new format as our base (with 
>  the possibility of generating srpms as an intermediate format).  We 
>  might take half-steps like you propose to get there but we need to 
>  decide that that's the direction we want to head. 
Thanks for the detailed constructive response!  Yes, I agree this is a
crucial point.  The srpm would no longer be the "preferred form of
modification" to use GPL terminology.  

But looking at this realistically, the only point in our current process
(that I know of) where we use srpms internally now is for review of new
contributions.  But we've been talking about a process where we give new
packages tentative revision control space, so they can be imported
without losing history.  So I think the project is already heading in
the direction of using the pkgcvs as the primary/preferred source
format.

So as you say, a key question is - can the goals be achieved without
taking the approach of generating spec files?  Tom suggested a
%{autorelease} macro or the like instead of using spec.in->spec.  I
think that would work in the sense that you could avoid having the spec
file needing to modified by automated processes, but that still means
you need to take the approach of generating a srpm, because vanilla
rpmbuild won't know how to handle %{autorelease} or the like.

I'm a bit torn on this - it makes things slightly easier for just these
modifications, but for more complex changes like importing the package
description from a wiki, or generating the base %files checklist from a
global set based on package tags, we need to take the approach of
generating a spec.

I'll consider this as I continue to prototype out the changes.  The
differences between the macro and spec.in approaches are not large, so
we can switch if necessary.

>  Errr... Read the guidelines again.  That is a valid tag due to java 
>  packagers having made the case that there is a need to interleave with 
>  jpackage.  If the buildsystem were to make the release tag up it would 
>  have to decide whether the packager wanted to stay interleavable with 
>  jpackage and if so find out what the present jpackage version-release is 
>  and build our release tag from that.
>  
>  Or you can try to convince the Java Packagers that interleaving with 
>  jpackage is not a desirable thing... but we've already been through that 
>  once so you better have some good arguments :-) 
The only part of the Release tag that I care about the build system
creating is incrementing the number, so that I don't have to spend my
time doing that, and we can automate things more.  

I'm totally fine with allowing the spec file to specify what other bits
go into there.

>  >>> * Avoid developer time being spent incrementing integers
> in spec file
>  >> That's usually a 1-character change, sometimes a 2-character change, it
> takes 
>  >> 10 seconds at worst, I fail to see the problem there.
>  > 
>  > The real thing this blocks is more automation in our processes.  I don't
>  > want to have to edit a spec file at all when a new release comes out; we
>  > should have a system where I can look on a web page and see "New
>  > upstream releases of foo, bar available", and click one link to pull
>  > them in for tentative builds and tests, and then if that works tag them.
>  > 
>  That would be nice.  It would also have to check that the configure 
>  script hasn't changed beyond the package's version update (for autoconf 
>  using build systems... not sure that other build systems will accurately 
>  capture build requirements in one spot :-( ).  Otherwise, you silently 
>  go without a new optional feature because configure didn't find it at 
>  build time. 
Absolutely, it is critical in more automated system that it's easy to
pull down the proposed changes and test things locally too.

You could imagine that when you click on that update link, you get to a
"proposed changes" page where you can see things like the diffs of
configure, number of added/removed files, whether there are new .desktop
files, any new setuid files, etc etc.

>  But then you're right back to specifying release information in
> the spec 
>  file :-(.  (It's more elegant in that it's not encoding the build order 
>  in the spec but it's still annoying to the packager as they have to add 
>  it to the file when things change.)  
Essentially, nothing that we want changed by external processes can be
in the spec file.  If as a first pass we want to allow for automating of
rebuilds, then having the pre/post release information in the spec is
fine because a rebuild won't change that.

However, if we also want to automate pulling in new upstream versions or
SVN/hg/git snapshots, then we need to have the pre/post information in a
separate place.

>  Err... it can determine that the last build was a branch-only build
> but 
>  how can it determine that the current build is a branch only build? 
I think the simple solution here is to always automatically use the
branch versioning format for non-rawhide builds.

>  >> * in schemes like prereleases, how to define the prerelease
> tag ("alphatag") to 
>  >> use.
>  > 
>  > I like the idea for this one of extending the "sources" file - say a
>  > third field which denotes whether a file is a pre or post release.
>  > 
>  -1
>  
>  This is one of the differences between Debian dpkg and rpm that I like 
>  about rpm.  All the build information is in one file.  Our current 
>  sources file is generated automatically and never needs to be edited by 
>  the package builder.  It is only there so the buildsystem can 
>  differentiate between two tarballs that have the same name. 
What I'm thinking for this is that the sources file would continue to be
generated automatically; instead of doing:

make new-sources FILES=blah-svn20080307.tar.gz
you'd do
make new-sources FILES=blah-svn20080307.tar.gz PRERELEASE=1

(or possibly this is just autodetected by the system; if you add a new
source file whose version would be lower than the existing one, it marks
it as a prerelease)

But really the payoff is longer term, when we can have a package status
web page which can say:

Package foo:
SVN: 47 new changesets [pull]
Releases: 1 new release [pull]

Or even larger scope:

Packages tagged gnome-desktop:

New upstream releases: [pull all]
 gtk+ 2.14.8 [pull]
   - Currently applied patches:
     * http://bugzilla.gnome.org/438221 (fix frobnicator)   [unapply]
     * http://bugzilla.gnome.org/437891 (fix foo)           [unapply]
 gnome-vfs2 2.22.3 [pull]
 ...

>  Also, btw, your make local doesn't do the right thing WRT
> disttag. 
>  disttag is supposed to tell which distribution the package is built for, 
>  not which repo it is built by.  So disttag == .fc8 means the package was 
>  built for and intended to run on Fedora 8.  .local is a repotag rather 
>  than a disttag.  Additionally, whatever it expands to should sort lower 
>  than the equivalent Fedora package built in the buildsystem.  Otherwise 
>  you, the package maintainer, have extra steps to perform to go from 
>  testing your local built package to testing the package built on the 
>  buildsystem.  (Whereas if it sorts lower, the buildsystem package is 
>  picked up by the normal yum update). 
Makes sense, yep.  The make local changes are a side thing; I'm mostly
interested in the pkgcvs/koji-related changes.

[Jef Spaleta]

On Fri, Mar 7, 2008 at 10:00 AM, Colin Walters <walters(a)redhat.com&gt; wrote:
>   Thanks for the detailed constructive response!  Yes, I agree this is
> a
>   crucial point.  The srpm would no longer be the "preferred form of
>   modification" to use GPL terminology. 
<snip>
>  So I think the project is already heading in
>   the direction of using the pkgcvs as the primary/preferred source
>   format. 
As a Board member, this is indeed something I like to see us get to.
Being able to get to a point where everyone feels comfortable with
working within cvs (or whatever we use for code control) to meet our
(as a project) source distribution requirements for any binary we make
available through our build system.

-jef

[James Antill]

On Thu, 2008-03-06 at 13:26 +0530, Debarshi Ray wrote:

>  On the other hand, I would like a way to generate the cvs commit
>  message from the Spec's %changelog. Something like 'make commit'? 
 "make clog" has been there a long time, having a make target that does
"cvs commit -F clog" would probably be worthwhile, and means I could get
rid of my script to do that :).

-- 
James Antill <james(a)fedoraproject.com&gt;
Fedora

[Debarshi Ray]

> > However, since we already have 'make clog', what is
> stopping us from
> > having a 'make commit'? In my limited view, if one is creating a clog 
>  Someone needs to write a patch, submit it to
>   fedora-infrastructure-list and maybe send a reminder in case the patch was not
>   commited to Makefile.common. 
Here is a way to implement the 'commit' target:
http://rishi.fedorapeople.org/make-commit.patch

Comments?

Happy hacking,
Debarshi
-- 
"From what we get, we can make a living; what we give, however, makes a life."
    -- Arthur Ashe

[Till Maas]

On Sun March 9 2008, Debarshi Ray wrote:
>  >> However, since we already have 'make clog', what is
> stopping us from
>  >> having a 'make commit'? In my limited view, if one is creating a clog
>  >
>  > Someone needs to write a patch, submit it to
>  >  fedora-infrastructure-list and maybe send a reminder in case the patch
>  > was not commited to Makefile.common.
>
>  Here is a way to implement the 'commit' target:
>  http://rishi.fedorapeople.org/make-commit.patch
>
>  Comments? 
The patch looks good. You need to submit it to the fedora buildsys 
mailinglist[1] to get it included into Makefile.common, unless someone with 
commit access notices it here and commits it. :-)

Regards,
Till

[1] https://www.redhat.com/mailman/listinfo/fedora-buildsys-list