Packaging Guidelines - creating tarball from VCS with script

[Tomas Radej]

Hi,

I was wondering if Packaging Guidelines could be amended so that even when creating
tarball from VCS, using a standalone shell script would be mandatory (see
http://fedoraproject.org/wiki/Packaging:SourceURL#Using_Revision_Control ). I believe this
could allow easier reviews and package updates as there would be no need to copy&paste
code from comments, and checking for package's checksum could be (at least partially)
automated for the fedora-review tool.

What do you think?

Regards,
-- 
Tomas Radej <tradej(a)redhat.com&gt;

[Emanuel Rietveld]

On 05/14/2012 03:02 PM, Tomas Radej wrote:
>  Hi,
>
>  I was wondering if Packaging Guidelines could be amended so that even when creating
> tarball from VCS, using a standalone shell script would be mandatory (see
> http://fedoraproject.org/wiki/Packaging:SourceURL#Using_Revision_Control ). I believe this
> could allow easier reviews and package updates as there would be no need to copy&paste
> code from comments, and checking for package's checksum could be (at least partially)
> automated for the fedora-review tool.
>
>  What do you think?
>
>  Regards, 
I think this is a very good idea, and I'm going to do this for my next 
package. Thank you for suggesting it.

Emanuel

[Toshio Kuratomi]

On Mon, May 14, 2012 at 03:02:23PM +0200, Tomas Radej wrote:
>  Hi,
>  
>  I was wondering if Packaging Guidelines could be amended so that even when
>  creating tarball from VCS, using a standalone shell script would be
>  mandatory (see
>  http://fedoraproject.org/wiki/Packaging:SourceURL#Using_Revision_Control
>  ). I believe this could allow easier reviews and package updates as there
>  would be no need to copy&paste code from comments, and checking for
>  package's checksum could be (at least partially) automated for the
>  fedora-review tool.
>  
>  What do you think?
>   
Automating of the package's checksum won't work for many VCS's .  git,
for
instance, does not preserve timestamps.  So the tarball created from a git
snapshot will have a different checksum for each checkout.

I personally prefer to have the checkout instructions in comments.  It makes
it easier to review what the person did and interrupts my thoughts less when
I can see what the person did to produce the tarball in the same window as
I'm looking at the spec file.  Having to open up a second file to see if the
checkout commands are hitting the canonical source repository and that they
contain enough information to checkout only a single version is
a distraction.

-Toshio

[Neal Gompa]

I agree with Toshio on this. Depending on how the VCS behaves with
checkout/cloning, it will be difficult to get predictable results in a
usable way through a script. Commenting in the spec file is the best way to
go in my opinion.
On May 14, 2012 9:22 AM, "Toshio Kuratomi" <a.badger(a)gmail.com&gt; wrote:

>  On Mon, May 14, 2012 at 03:02:23PM +0200, Tomas Radej wrote:
>  > Hi,
>  >
>  > I was wondering if Packaging Guidelines could be amended so that even
>  when
>  > creating tarball from VCS, using a standalone shell script would be
>  > mandatory (see
>  > http://fedoraproject.org/wiki/Packaging:SourceURL#Using_Revision_Control
>  > ). I believe this could allow easier reviews and package updates as there
>  > would be no need to copy&paste code from comments, and checking for
>  > package's checksum could be (at least partially) automated for the
>  > fedora-review tool.
>  >
>  > What do you think?
>  >
>  Automating of the package's checksum won't work for many VCS's .  git, for
>  instance, does not preserve timestamps.  So the tarball created from a git
>  snapshot will have a different checksum for each checkout.
>
>  I personally prefer to have the checkout instructions in comments.  It
>  makes
>  it easier to review what the person did and interrupts my thoughts less
>  when
>  I can see what the person did to produce the tarball in the same window as
>  I'm looking at the spec file.  Having to open up a second file to see if
>  the
>  checkout commands are hitting the canonical source repository and that they
>  contain enough information to checkout only a single version is
>  a distraction.
>
>  -Toshio
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>

[Remi Collet]

Le 14/05/2012 16:22, Toshio Kuratomi a écrit :

> > What do you think?
>
>  I personally prefer to have the checkout instructions in comments. 
+1

Except for some very complex scripts for which it make sense to have a 
shell script.



Remi.

[Tomas Radej]

On Mon, 14 May 2012 16:31:08 +0200
Remi Collet <Fedora(a)FamilleCollet.com&gt; wrote:

>  Le 14/05/2012 16:22, Toshio Kuratomi a écrit :
>  
>  >> What do you think?
>  >
>  > I personally prefer to have the checkout instructions in comments.
>  
>  +1
>  
>  Except for some very complex scripts for which it make sense to have a 
>  shell script.
>   
Discussion with pingou and sochotni on #fedora-java brought us this: What about using an
RPM macro with this grammar:

	%create_tarball git|svn|cvs URL revision [additional commands]

This covers vast majority of the cases, and by using e. g. git archive, we can ensure the
timestamps will be the same. In case you need to make some non-standard changes, like
removing a file, you can still use a stand-alone script in SOURCE1.

What do you think about this?

-- 
Tomas Radej <tradej(a)redhat.com&gt;

[Alec Leamas]

On 05/15/2012 10:19 AM, Tomas Radej wrote:
>  On Mon, 14 May 2012 16:31:08 +0200
>  Remi Collet<Fedora(a)FamilleCollet.com>  wrote:
>
> > Le 14/05/2012 16:22, Toshio Kuratomi a écrit :
> >
> >>> What do you think?
> >> I personally prefer to have the checkout instructions in comments.
> > +1
> >
> > Except for some very complex scripts for which it make sense to have a
> > shell script.
> >
>  Discussion with pingou and sochotni on #fedora-java brought us this: What about using an
> RPM macro with this grammar:
>
>  	%create_tarball git|svn|cvs URL revision [additional commands]
>
>  This covers vast majority of the cases, and by using e. g. git archive, we can ensure the
> timestamps will be the same. In case you need to make some non-standard changes, like
> removing a file, you can still use a stand-alone script in SOURCE1.
>
>  What do you think about this?
>  
Even though the timestamp problem remains unsolved e. g., for svn this 
is a great idea. Maybe expanding syntax to

   %create_tarball0, %create_tarball1...

which  also defines %SOURCE0,%SOURCE1... which makes %prep happy?

[Colin Walters]

On Tue, 2012-05-15 at 10:19 +0200, Tomas Radej wrote:

>  Discussion with pingou and sochotni on #fedora-java brought us this:
> What about using an RPM macro with this grammar:
>  
>  	%create_tarball git|svn|cvs URL revision [additional commands] 
The build system has no network access.

For the present Fedora/Fedpkg/RPM architecture, you really can't get out
of mashing git repositories into tarballs without a serious
architectural rework of everything.

[Oron Peled]

On Tuesday, 15 בMay 2012 17:40:27 Colin Walters wrote:
>  On Tue, 2012-05-15 at 10:19 +0200, Tomas Radej wrote:
>  
>  > Discussion with pingou and sochotni on #fedora-java brought us this: What 
about using an RPM macro with this grammar:
>  > 
>  > 	%create_tarball git|svn|cvs URL revision [additional commands]
>  
>  The build system has no network access. 
Yes, that's an important feature (so our sources are always rebuildable
regardless of extenal changes).

>  For the present Fedora/Fedpkg/RPM architecture, you really can't
> get out
>  of mashing git repositories into tarballs without a serious
>  architectural rework of everything. 
I think the real needed rework is not in the infrastructure, but
adding a new build mode to rpmbuild(1):

 * We currently have:
   1. The '-b[bsa]' build from .spec + sources
   2. The '-t[bsa]' build from tarball with embedded .spec
   3. The '--rebuild' build from SRPM

   Notes:
      - Modes 2, 3 actually extract the parts and continue as 1.

      - Both mode 1,2 can be used to generate SRPM which enable us
        to preserve all components in encapsulated form (no fuzzy
        external dependencies).

 * Which calls for two new modes:
   4. rpmbuild -b[bsa] --from-vcs-repo <url> spec_file
   5. rpmbuild -b[bsa] --from-vcs-wc <directory> spec_file

 * Notes:
   - Obviously mode 4 would be implemented in terms of mode 5
     (clone the URL to temporary wc and use it)

   - Just like mode 1 and 2 the '-bs' would enable us to encapsulate
     the result as SRPM that would not depend anymore in the vcs repo
     or wc -- this SRPM could be uploaded to koji, etc.

   - I think trying to abuse existing %prep section and %setup semantics
     for vcs use, will create more problems than it solves.

     My suggestion is that these command line flags could be used only
     if the .spec file also contains *alternative* %vcs_prep

 * Example:

    Name: acme
    ...
    Source0: %name-%version.tar.gz
    Patch0: foobar.patch
    ...
    Vcs-URL: git://something_we_can_clone_from

    # Would be used when no "--from-vcs-*" flags were given
    # e.g: when we --rebuild from SRPM
    %prep
    %setup
    %patch0 -p1

    # Would be only used for --from-vcs-repo
    %vcs_prep
    # No %setup (no tarballs here, move along)
    # No %patch (the vcs should be used for this)
    # Maybe we can have a nice macro for common cases
    %vcs_setup
    # Or alternatively, can be done manually
    git clone [some-special-flags-I-want] %{VCS_URL}

    # Both %prep and %vcs_prep would be skipped for --from-vcs-wc
    # Only the %_builddir would be set to the given directory

    # The rest is normal, since all execution paths now converge
    # into a "prepared" build directory + .spec file
    %build
    ...

Advantages:
 * All existing packages are unaffected (but --from-vcs-repo cannot
   be used with them yet).

 * A developer that work on a vcs-wc can create a package for testing
   imediately from her expanded wc.

 * A .spec file with the extra %vcs_prep and Vcs-URL can create SRPM
   directly from the vcs-repo. This SRPM can be uploaded to our
   build system and be used for building *without* any interaction
   with the vcs.

Problems:
 * The --from-vcs-wc is very handy and can be easily abused to generate
   RPM's from non-clean working trees:
   - Maybe this mode can mark some "dirty" flag in the RPM header.
     (RPM_WIP_UNRELEASED).

 * With --from-vcs-repo there is no clear model to separate "upstream"
   code from packager changes:
   - Maybe the solution is to map this information from the vcs.
     E.g:
       Vcs-URL-upstream: git://server/path master
       Vcs-URL-build:    git://server/path fc17

Now, is this a science fiction?

-- 
Oron Peled                                 Voice: [phone]
oron(a)actcom.co.il                  http://users.actcom.co.il/~oron
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Kevin Fenzi]

On Tue, 15 May 2012 23:30:27 +0300
Oron Peled <oron(a)actcom.co.il&gt; wrote:

...snip...

>   * A .spec file with the extra %vcs_prep and Vcs-URL can create SRPM
>     directly from the vcs-repo. This SRPM can be uploaded to our
>     build system and be used for building *without* any interaction
>     with the vcs. 
I very much dislike this. Currently offical builds cannot use src.rpms,
they must use VCS. This allows us to be able to easily see what was in
a build. If we allow arbitray src.rpms it opens up a big can of worms: 

- We have to store those src.rpms forever (or many years). 
- Looking at what was used requires you to download a big src.rpm and
  unpack it instead of looking at a git hash in a repo. 
- There's much less auditing. Someone could upload a src.rpm with
  horrible junk in it and the only way to tell would be to download and
  inspect it. 

So, I think thats a big no go. ;) 

Otherwise this sounds like great stuff to talk to rpm upstream
about. ;) 

kevin

[Tomas Radej]

On Wed, 16 May 2012 10:10:17 -0600
Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Tue, 15 May 2012 23:30:27 +0300
>
>  Otherwise this sounds like great stuff to talk to rpm upstream
>  about. ;) 
>   
Since the discussion is pretty much done here, I'll talk to RPM guys and see if they
find this to be a good idea. Thanks for the input, guys.

-- 
Tomas Radej <tradej(a)redhat.com&gt;

[Ville Skyttä]

On 2012-05-14 17:22, Toshio Kuratomi wrote:

>  I personally prefer to have the checkout instructions in comments.
The big drawback of that is that if they're just in comments, they're
not a prerequisite for creating the shipped tarball, and they will
bitrot sooner or later.  Using macros in them is one way to avoid some
cases of bitrot, but it's plain annoying because they need to be
expanded by hand before copy-pasting the commands.

Always creating and using an external script to create the tarball
doesn't suffer from this problem.

[Thomas Moschny]

2012/5/14 Toshio Kuratomi <a.badger(a)gmail.com&gt;:
>  Automating of the package's checksum won't work for many
> VCS's .  git, for
>  instance, does not preserve timestamps.  So the tarball created from a git
>  snapshot will have a different checksum for each checkout. 
While files' modification times in a checkout may be different,
archives created with 'git archive' are stable, because git uses the
datetime of the commit for each file in the archive.

So instead of cloning the repository and creating the archive locally,
the preferred method would imho be to use a download link (if present)
of the used git hosting service for directly preparing an archive, and
to file bugs for the respective hosting service if they do not
properly use this git functionality.

- Thomas

[Alec Leamas]

On 05/14/2012 10:46 PM, Thomas Moschny wrote:
>  2012/5/14 Toshio Kuratomi<a.badger(a)gmail.com>:
> > Automating of the package's checksum won't work for many VCS's .  git,
> for
> > instance, does not preserve timestamps.  So the tarball created from a git
> > snapshot will have a different checksum for each checkout.
>  While files' modification times in a checkout may be different,
>  archives created with 'git archive' are stable, because git uses the
>  datetime of the commit for each file in the archive.
>
>  So instead of cloning the repository and creating the archive locally,
>  the preferred method would imho be to use a download link (if present)
>  of the used git hosting service for directly preparing an archive, and
>  to file bugs for the respective hosting service if they do not
>  properly use this git functionality.
>
>  - Thomas 
I don't find the reference right here, but of the top of my head git 
changes modification times when checking out a branch, but keeps it when 
checking out a commit. But the real problem is the hosting services, and 
while we certainly can file bugs on github or gitorious, we also have to 
live with them.

We just had a thread on the github tarballs with the unreasonable names 
like 1234fcd; no extension... modification time is not the only problem