11:54 a.m.
#5362: Use XZ for repository metadata compression
-------------------------+------------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Keywords: | Blocked By:
Blocking: |
-------------------------+------------------------
See [https://bugzilla.redhat.com/show_bug.cgi?id=700020 bug #700020] for
details.
I was informed that yum and createrepo already support xz compression for
metadata, yet in Fedora still use bz2. We should probably switch
repositories to use xz as well.
I was told to talk with rel-eng, I hope this is the right place to do so.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
4:53 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
I talked to dgilmore in person and he agrees this can be changed. In
Rawhide for the time being, just to test it, and once we branch F19, it
will be in F19.
Dennis, could you please have a look at it and change the default for
Rawhide, if possible? Thanks.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:1>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
6:14 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
Dennis, Kevin, I'd really like to have this implemented before F19
branches, because then it's too late and we will have to wait for F20.
It cuts the metadata size by 30%, that's huge! It saves 1/3 of your
waiting time for yum refresh!
Can one of you please have a look at necessary changes? It probably
requires just a minor tweaks in rel-eng scripts. Thanks!
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:2>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
6:01 p.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Changes (by kevin):
* cc: notting (added)
Comment:
I agree this would be nice to do and now would be a good time. ;)
I guess we need:
- change mash to use xz for it's createrepo
- Add pyliblzma to base? Or ask yum folks to add it as a requires?
Adding notting here for comment.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:3>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
8:28 p.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by mattdm):
Replying to [comment:3 kevin]:
- Add pyliblzma to base? Or ask yum folks to add it as a requires?
It's already pulled in as a requirement of yum in F18. So check that off
the list. :)
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:4>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
10:11 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by notting):
mash just uses the createrepo default - is there a reason that couldn't
change?
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:5>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
10:33 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
Replying to [comment:5 notting]:
mash just uses the createrepo default - is there a reason that
couldn't
change?
I'll gladly create an upstream ticket for it if desired. But please note:
1. it most likely won't be done before F19 branches
1. I assume the --xz option was added for the very reason that
distributions might easily change the behavior to suit their needs. It
also hints that there was a reason to stay with gz by default.
1. I think the default compression method won't be changed in createrepo
unless all important distributions that use it have their package managers
adjusted to work with xz. Remember, this is not just about Fedora and yum.
I have no idea what is the state of xz support in other package managers.
I think it makes sense to add --xz to Fedora rel-eng scripts now and also
create an upstream ticket, which will probably take years to be resolved.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:6>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
10:38 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by notting):
Well, if mash used the command line (which it doesn't).
In any case, changing the default makes it consistent everywhere, and
allows us to have per-release settings outside of modifying all the tools
& scripts.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:7>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7:45 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
The upstream trac is full of spam and nobody seems to care:
http://createrepo.baseurl.org/report/1
There is no point creating a ticket there. But when looking into git, I
see mostly Red Hatters anyway:
http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=summary
In this light, we can consider
https://bugzilla.redhat.com/show_bug.cgi?id=700020 being an upstream
report. As you can see, not much response there either. I'll try to make
createrepo developers pay more attention to it.
Still, it would be great if we could change Fedora's default independently
of createrepo default. Because my feeling is that that won't be adjusted
in the near future.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:8>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7:48 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
Also, this commit is interesting:
http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=blobdiff;f=docs/c...
It's pretty recent (2012-11-19), so it doesn't seem like they're going to
change the default "compat" setting soon.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:9>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
9:09 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
James Antill responded, please read
https://bugzilla.redhat.com/show_bug.cgi?id=700020#c13
Is it doable to tweak mash, or do you prefer to wait a year or so before
xz becomes the default in createrepo?
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:10>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
9:38 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by notting):
Thinking...
The builders run RHEL 6, and use koji to install from the buildroots
there. Since they use the koji repos, they likely would have the old
metadata compression, so they'd still work. But there may be oppportunity
for breakage there that I'm not aware of.
mash would work, in as much as rawhide is composed in a rawhide mock
chroot.
The releases are composed on a release-ish install, so that would work
there, but we'd need to make sure the createrepo calls in pungi, etc. are
updated to match. But this would mean the release repos are different than
any random createrepo that is run by the a Fedora user. If we're OK with
that, we could do that.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:11>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
2:35 p.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kevin):
I'd be ok with the difference there. (ie, I'd be ok with us just changing
this in mash/pungi for now at least)
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:12>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
3:59 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kparal):
Is there any chance this can get implemented in F20?
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:13>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
4:08 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by elad):
Any chance this can be done for F21 and Rawhide? yum supports xz in all
currently-supported Fedora releases, so the transition would be smooth and
probably un-noticable.
From rel-eng side, it just means adding --compress-type=xz to the
createrepo call in the script.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:14>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
9:56 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by mattdm):
This seems like a pretty good win. The bz2 files in Rawhide right now are
56M, and that goes down to 47M with xz — almost a 20% improvement. And if
we go to xz -9, it's 43M — a 25% "discount".
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:15>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
10:51 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Changes (by ausil):
* keywords: => meeting
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:16>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
11:05 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by till):
Action items:
* Add XZ compression option to
[https://git.fedorahosted.org/cgit/pungi.git pungi] and
[https://git.fedorahosted.org/cgit/mash/ mash]
* pungi uses the createrepo python api, mash executes createrepo
* Patches should be sent to the buildsys list
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:17>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
10:52 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Changes (by ausil):
* cc: releng@… (added)
Comment:
Dennis to look at implementing.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:18>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
3 p.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by ralph):
Mailed in patches:
-
https://lists.fedoraproject.org/pipermail/buildsys/2014-April/004275.html
-
https://lists.fedoraproject.org/pipermail/buildsys/2014-April/004276.html
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:19>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
3:35 p.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: | Component: other
Resolution: | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by elad):
Any status updates? Since the patches have been merged, are we likely to
see xz-compressed metadata in Rawhide anytime soon?
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:20>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
11:28 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: closed
Milestone: | Component: other
Resolution: fixed | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Changes (by kevin):
* status: new => closed
* resolution: => fixed
Comment:
This was enabled in rawhide and epel7beta yesterday. ;)
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:21>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
8:51 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: closed
Milestone: | Component: other
Resolution: fixed | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by vbatts):
and now local `fedpkg mockbuild` for rawhide is broken on rhel6. :-(
```
Error: xz compression not available
```
I can still build for dist f19, f20 and el6.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:22>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
9:42 a.m.
#5362: Use XZ for repository metadata compression
--------------------------+-----------------------
Reporter: elad | Owner: rel-eng@…
Type: enhancement | Status: closed
Milestone: | Component: other
Resolution: fixed | Keywords: meeting
Blocked By: | Blocking:
--------------------------+-----------------------
Comment (by kevin):
'yum install pyliblzma' should fix it.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5362#comment:23>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
3627
days inactive
4212
days old
rel-eng@lists.fedoraproject.org
23 comments
1 participants
participants (1)
-
Fedora Release Engineering