5:22 p.m.
Hello all,
I hope I am not the first to come across packaging issues for projects that use GitHub as
their upstream source download. For those not familiar, GitHub dynamically generates
downloads for all git 'tags'. So for example:
$ git tag -a -m 'Tagging 1.0.0' 1.0.0
This would create a tag of '1.0.0' in git. On GitHub, this tag would be
'downloadable' via a dynamic app such as:
https://github.com/derks/myapp/zipball/1.0.0
This is the upstream URL that a lot of GitHub users are providing rather than a direct
download link. Unfortunately the above is not usable by RPM because rpmbuild expects the
URL to end with the file. If I were to use the above in my spec, I would get an error
saying something to the affect of 'unknown source file 1.0.0'. Additionally, the
tarbal that is created looks like:
myapp-1.0.0-XXXXXXX.tar.gz
Where XXXXXXX is the last bit of the git commit hash id... which cause yet another pain in
that... you need to update this commit revision every time you update the spec... and it
just makes things a bit less fluid. I'm wondering how other people have resolved
these issues for projects using GitHub as the upstream Source0 download provider.
Finally, debian has a web app to resolve these issues at:
http://githubredir.debian.net/
What would be the thoughts of using that to produce more sane/traditional tarbals of
upstream GitHub source?
---
derks
7:11 p.m.
On Tue, Feb 15, 2011 at 4:22 PM, BJ Dierkes
<wdierkes(a)5dollarwhitebox.org> wrote:
I'm wondering how other people have resolved these issues for
projects using
GitHub as the upstream Source0 download provider.
I package a github-hosted package, and I ran across this when I was
testing with rpmlint. It is too bad that GitHub tries to be so
"clever". After reading
http://fedoraproject.org/wiki/Packaging:SourceURL#Troublesome_URLs,
here is what I have in my .spec:
# Similar to MySQL, github has a mirror redirector for its downloads.
# You can find this tarball at the following canonical URL:
# http://github.com/masterzen/mysql-snmp/tarball/v1.0rc2
Source0: masterzen-mysql-snmp-v1.0rc2-0-gdec3ef8.tar.gz
Finally, debian has a web app to resolve these issues at:
http://githubredir.debian.net/
Cool idea, although it feels a bit like using some sort of URL
shortener. I wish GitHub would just fix their stuff.
7:24 p.m.
Ken Dreyer <ktdreyer(a)ktdreyer.com> writes:
On Tue, Feb 15, 2011 at 4:22 PM, BJ Dierkes
<wdierkes(a)5dollarwhitebox.org> wrote:
> I'm wondering how other people have resolved these issues for projects using
> GitHub as the upstream Source0 download provider.
> Finally, debian has a web app to resolve these issues at:
> http://githubredir.debian.net/
Cool idea, although it feels a bit like using some sort of URL
shortener. I wish GitHub would just fix their stuff.
Seems like really we should lobby GitHub to provide download URLs that
specify the full file name. There's no advantage to the way they are
doing it, AFAICS, and it's not obvious what file format you will get
from their URLs. ("zipball" seems a particularly poor choice here
--- personally I'd have thought it meant a .zip archive ...)
regards, tom lane
5:24 a.m.
Dne 16.2.2011 02:24, Tom Lane napsal(a):
Ken Dreyer<ktdreyer(a)ktdreyer.com> writes:
> On Tue, Feb 15, 2011 at 4:22 PM, BJ Dierkes
> <wdierkes(a)5dollarwhitebox.org> wrote:
>> I'm wondering how other people have resolved these issues for projects using
>> GitHub as the upstream Source0 download provider.
>> Finally, debian has a web app to resolve these issues at:
>> http://githubredir.debian.net/
> Cool idea, although it feels a bit like using some sort of URL
> shortener. I wish GitHub would just fix their stuff.
Seems like really we should lobby GitHub to provide download URLs that
specify the full file name. There's no advantage to the way they are
doing it, AFAICS, and it's not obvious what file format you will get
from their URLs. ("zipball" seems a particularly poor choice here
--- personally I'd have thought it meant a .zip archive ...)
regards, tom lane
Have you already asked GitHub or is it just what we should? Also Kevin
notes should be mentioned. I believe they can do something about it.
Vit
3:49 a.m.
BJ Dierkes wrote:
What would be the thoughts of using that to produce more
sane/traditional
tarbals of upstream GitHub source?
It doesn't fix the third, and main, issue: GitHub-generated tarballs get a
new checksum each time they're regenerated (because some file dates change).
This makes them completely unverifiable.
(This also goes for other "generate tarball from tag" services, e.g.
BitBucket.)
The only reasonable thing to do there is really to have upstream upload a
real, verifiable tarball (explaining the above issue to them).
Kevin Kofler
3:55 a.m.
On 02/16/2011 12:22 AM, BJ Dierkes wrote:
Hello all,
I hope I am not the first to come across packaging issues for projects that use GitHub as
their upstream source download. For those not familiar, GitHub dynamically generates
downloads for all git 'tags'. So for example:
$ git tag -a -m 'Tagging 1.0.0' 1.0.0
This would create a tag of '1.0.0' in git. On GitHub, this tag would be
'downloadable' via a dynamic app such as:
https://github.com/derks/myapp/zipball/1.0.0
This is the upstream URL that a lot of GitHub users are providing rather than a direct
download link. Unfortunately the above is not usable by RPM because rpmbuild expects the
URL to end with the file. If I were to use the above in my spec, I would get an error
saying something to the affect of 'unknown source file 1.0.0'. Additionally, the
tarbal that is created looks like:
myapp-1.0.0-XXXXXXX.tar.gz
Where XXXXXXX is the last bit of the git commit hash id... which cause yet another pain
in that... you need to update this commit revision every time you update the spec... and
it just makes things a bit less fluid. I'm wondering how other people have resolved
these issues for projects using GitHub as the upstream Source0 download provider.
Finally, debian has a web app to resolve these issues at:
http://githubredir.debian.net/
What would be the thoughts of using that to produce more sane/traditional tarbals of
upstream GitHub source?
What I am using in my spec files is something like this:
# git clone git://github.com/sonatype/sisu
# git archive --prefix="sonatype-sisu-1.4.3.2/" --format=tar \
# sisu-1.4.3.2 > sisu-1.4.3.2.tar.gz
Source0: %{name}-%{version}.tar.gz
Which basically takes the tag name and creates a nice tarball from it.
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Associate Software Engineer - Base Operating Systems Brno
PGP: 7B087241
Red Hat Inc. http://cz.redhat.com
4:06 a.m.
On 02/16/2011 10:55 AM, Stanislav Ochotnicky wrote:
On 02/16/2011 12:22 AM, BJ Dierkes wrote:
> Hello all,
>
> I hope I am not the first to come across packaging issues for projects that use
GitHub as their upstream source download. For those not familiar, GitHub dynamically
generates downloads for all git 'tags'. So for example:
>
> $ git tag -a -m 'Tagging 1.0.0' 1.0.0
>
>
> This would create a tag of '1.0.0' in git. On GitHub, this tag would be
'downloadable' via a dynamic app such as:
>
> https://github.com/derks/myapp/zipball/1.0.0
>
>
> This is the upstream URL that a lot of GitHub users are providing rather than a
direct download link. Unfortunately the above is not usable by RPM because rpmbuild
expects the URL to end with the file. If I were to use the above in my spec, I would get
an error saying something to the affect of 'unknown source file 1.0.0'.
Additionally, the tarbal that is created looks like:
>
> myapp-1.0.0-XXXXXXX.tar.gz
>
>
> Where XXXXXXX is the last bit of the git commit hash id... which cause yet another
pain in that... you need to update this commit revision every time you update the spec...
and it just makes things a bit less fluid. I'm wondering how other people have
resolved these issues for projects using GitHub as the upstream Source0 download
provider.
>
> Finally, debian has a web app to resolve these issues at:
>
> http://githubredir.debian.net/
>
>
> What would be the thoughts of using that to produce more sane/traditional tarbals of
upstream GitHub source?
>
What I am using in my spec files is something like this:
# git clone git://github.com/sonatype/sisu
# git archive --prefix="sonatype-sisu-1.4.3.2/" --format=tar \
# sisu-1.4.3.2 > sisu-1.4.3.2.tar.gz
Source0: %{name}-%{version}.tar.gz
Which basically takes the tag name and creates a nice tarball from it.
Although I should pipe it through gzip/bzip2 :-/
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Associate Software Engineer - Base Operating Systems Brno
PGP: 7B087241
Red Hat Inc. http://cz.redhat.com
8:47 a.m.
On 02/16/2011 03:36 PM, Matej Cepl wrote:
Dne 16.2.2011 11:06, Stanislav Ochotnicky napsal(a):
>> Which basically takes the tag name and creates a nice tarball from it.
>
> Although I should pipe it through gzip/bzip2 :-/
And you really don't resolve the issue with unstable MD5 checksum.
Why would that be? I am creating the tarball from the same git tag. Yes
I am using tag name, so theoretically if upstream is ugly enough to
overwrite their tags with different contents, this will be a problem.
But normally I can re-create the tarball using "git archive" and the md5
checksum will stay the same. Can you tell me how exactly this won't work?
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Associate Software Engineer - Base Operating Systems Brno
PGP: 7B087241
Red Hat Inc. http://cz.redhat.com
9:58 a.m.
Stanislav Ochotnicky <sochotnicky(a)redhat.com> writes:
On 02/16/2011 03:36 PM, Matej Cepl wrote:
> Dne 16.2.2011 11:06, Stanislav Ochotnicky napsal(a):
>>> Which basically takes the tag name and creates a nice tarball from it.
>>
>> Although I should pipe it through gzip/bzip2 :-/
>
> And you really don't resolve the issue with unstable MD5 checksum.
Why would that be? I am creating the tarball from the same git tag. Yes
I am using tag name, so theoretically if upstream is ugly enough to
overwrite their tags with different contents, this will be a problem.
But normally I can re-create the tarball using "git archive" and the md5
checksum will stay the same. Can you tell me how exactly this won't work?
That does not mean that the compressed contents will always be the same.
At least the gzip compressed tarballs from github contain a time stamp.
Andreas.
--
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84 5EC7 45C6 250E 6F00 984E
"And now for something completely different."
10:15 a.m.
On 02/16/2011 04:58 PM, Andreas Schwab wrote:
Stanislav Ochotnicky <sochotnicky(a)redhat.com> writes:
> On 02/16/2011 03:36 PM, Matej Cepl wrote:
>> Dne 16.2.2011 11:06, Stanislav Ochotnicky napsal(a):
>>>> Which basically takes the tag name and creates a nice tarball from it.
>>>
>>> Although I should pipe it through gzip/bzip2 :-/
>>
>> And you really don't resolve the issue with unstable MD5 checksum.
>
> Why would that be? I am creating the tarball from the same git tag. Yes
> I am using tag name, so theoretically if upstream is ugly enough to
> overwrite their tags with different contents, this will be a problem.
> But normally I can re-create the tarball using "git archive" and the md5
> checksum will stay the same. Can you tell me how exactly this won't work?
That does not mean that the compressed contents will always be the same.
$ git clone -q git://github.com/sonatype/sisu
$ GIT_DIR=sisu/.git git archive --prefix="sonatype-sisu-1.4.3.2/" \
--format=tar sisu-1.4.3.2 | bzip2 > tarball1.tar.bz2
$ rm -rf sisu/
$ git clone -q git://github.com/sonatype/sisu
$ GIT_DIR=sisu/.git git archive --prefix="sonatype-sisu-1.4.3.2/" \
--format=tar sisu-1.4.3.2 | bzip2 > tarball2.tar.bz2
$ md5sum tarball*
085fc5da0a7b504482e5e6449349316d tarball1.tar.bz2
085fc5da0a7b504482e5e6449349316d tarball2.tar.bz2
True, this is just one test run and yes...it's enough for the git or
bzip2 defaults to change and this will stop working. But as long as they
stay the same the MD5 is stable. Bonus points for using git hash instead
of tag name I guess...
At least the gzip compressed tarballs from github contain a time
stamp.
True, but git-archive doesn't do that by default. That's just their
"improvement" AFAIK
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Associate Software Engineer - Base Operating Systems Brno
PGP: 7B087241
Red Hat Inc. http://cz.redhat.com
3:13 a.m.
Stanislav Ochotnicky <sochotnicky(a)redhat.com> writes:
On 02/16/2011 04:58 PM, Andreas Schwab wrote:
> That does not mean that the compressed contents will always be the same.
$ git clone -q git://github.com/sonatype/sisu
$ GIT_DIR=sisu/.git git archive --prefix="sonatype-sisu-1.4.3.2/" --format=tar
sisu-1.4.3.2 | gzip > tarball1.tar.gz
$ GIT_DIR=sisu/.git git archive --prefix="sonatype-sisu-1.4.3.2/" --format=tar
sisu-1.4.3.2 | gzip > tarball2.tar.gz
$ md5sum tarball*
94101d5c44539875f09db8487b900abc tarball1.tar.gz
e87037cc65b3ff602675b84a23548299 tarball2.tar.gz
q.e.d.
> At least the gzip compressed tarballs from github contain a time
stamp.
^^^^
True, but git-archive doesn't do that by default.
I didn't say anything about git-archive.
Andreas.
--
Andreas Schwab, schwab(a)redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84 5EC7 45C6 250E 6F00 984E
"And now for something completely different."
2:29 p.m.
On 02/21/2011 11:13 AM, Andreas Schwab wrote:
Stanislav Ochotnicky <sochotnicky(a)redhat.com> writes:
> On 02/16/2011 04:58 PM, Andreas Schwab wrote:
>> That does not mean that the compressed contents will always be the same.
>
> $ git clone -q git://github.com/sonatype/sisu
$ GIT_DIR=sisu/.git git archive --prefix="sonatype-sisu-1.4.3.2/" --format=tar
sisu-1.4.3.2 | gzip > tarball1.tar.gz
$ GIT_DIR=sisu/.git git archive --prefix="sonatype-sisu-1.4.3.2/" --format=tar
sisu-1.4.3.2 | gzip > tarball2.tar.gz
$ md5sum tarball*
94101d5c44539875f09db8487b900abc tarball1.tar.gz
e87037cc65b3ff602675b84a23548299 tarball2.tar.gz
q.e.d.
FWIW, s/gzip/gzip -n/ and they'll match.
12:21 p.m.
That does not mean that the compressed contents will always be the
same.
At least the gzip compressed tarballs from github contain a time stamp.
That is true of the tarball-generation feature of gitweb or whatever it's
called last I looked too. It's easily fixed by having it pass -n to gzip
(which I would have thought they could have fixed in gitweb by now), and
already does not apply to bzip2 format.
Thanks,
Roland
5:45 a.m.
I ran into Github-related issues as well and filed a bug report:
http://support.github.com/discussions/repos/4565-sha-in-download-filename...
You can simply append the filename to the URL and it'll work, I'm using
this for example in
http://pkgs.fedoraproject.org/gitweb/?p=lua-xmlrpc.git;a=blob;f=lua-xmlrp...
The remaining problem really is the one filed above and some more people
watching and commenting the issue could bring it higher up on the
priority list.
Tim
--
Tim Niemueller <tim(a)niemueller.de> www.niemueller.de
=================================================================
Imagination is more important than knowledge. (Albert Einstein)
4843
days inactive
4849
days old
14 comments
11 participants
participants (11)
-
Andreas Schwab -
BJ Dierkes -
Ken Dreyer -
Kevin Kofler -
Matej Cepl -
Roland McGrath -
Stanislav Ochotnicky -
Tim Niemueller -
Tom Lane -
Ville Skyttä -
Vít Ondruch