8:19 p.m.
f20:
*filelists.sqlite.bz2 3.5 M
*filelists.xml.gz 3.3 M
*primary.sqlite.bz2 2.7 M
*primary.xml.gz 1.3 M
*other.sqlite.bz2 1.3 M
*other.xml.gz 964 K
*Fedora-20-comps.xml 906 K
*Fedora-20-comps.xml.gz 210 K
repomd.xml 3.6 K
rawhide:
*filelists.xml.gz 28 M <- xz?
*filelists.sqlite.xz 25 M
*primary.sqlite.xz 17 M
*primary.xml.gz 10 M <- xz?
*other.sqlite.xz 7.5 M
*other.xml.gz 5.8 M <- xz?
*prestodelta.xml.xz 2.9 M
*comps-rawhide.xml 1.6 M <- ?
*comps-rawhide.xml.xz 230 K
repomd.xml 4.2 K <- xz?
Why aren't all compressed with the xz, in rawhide?
Why two *comps* with the same original checksums?
poma
8:44 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Fri, 20 Jun 2014 03:19:55 +0200
poma <pomidorabelisima(a)gmail.com> wrote:
f20:
*filelists.sqlite.bz2 3.5 M
*filelists.xml.gz 3.3 M
*primary.sqlite.bz2 2.7 M
*primary.xml.gz 1.3 M
*other.sqlite.bz2 1.3 M
*other.xml.gz 964 K
*Fedora-20-comps.xml 906 K
*Fedora-20-comps.xml.gz 210 K
repomd.xml 3.6 K
rawhide:
*filelists.xml.gz 28 M <- xz?
*filelists.sqlite.xz 25 M
*primary.sqlite.xz 17 M
*primary.xml.gz 10 M <- xz?
*other.sqlite.xz 7.5 M
*other.xml.gz 5.8 M <- xz?
*prestodelta.xml.xz 2.9 M
*comps-rawhide.xml 1.6 M <- ?
*comps-rawhide.xml.xz 230 K
repomd.xml 4.2 K <- xz?
Why aren't all compressed with the xz, in rawhide?
Why two *comps* with the same original checksums?
because thats how createrepo works. the gzipped files are not ones you
download. yum and dnf use the sqlite files.
Dennis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBAgAGBQJTpDrMAAoJEH7ltONmPFDRtsEP/3aRbAhcYMLhHemlAIE/7nRT
1kp6PH4TfWdZxsBBCEK1Ko9lFC254oDr+hpFm7/dFXAXrXPbMHMOTunfI9iO52G2
he2wrJxg8ho6SOpudnI0D7LewGgIQYHV6lGkop+hExRQoaBCGSw7pOIE9b87t4iI
dD4peB7VBUAA7BY/vyz87fLjxgWFM91DTnL8UTnIPoCcbr+BPlE/BQDUtGeRGyLH
hKJTnrbQg8XpDN11xGYAm9tRAtLpAQhY0tm7jgjgRJ+s/3Io4KN2Xz4HCTyTFtSw
rFvOx0T5+ryl7dJ/x737OaPC2aqcQQ0pr7bTwQSvhV2b4cvaT+00sKjJUMgZR/WM
/jrRju2882tR9YeBFql1E7oMHbOV+Yudt5XL72mwc/6GCLHSa5ZgkILdgiN5K36s
3jfkB4P8tRtUkZIxhE+6owNuWwE30eAhEY/EKWo+zK7FhrbOqxZTW/1OXhgHHs8k
q545kyX5sU4oC1o1SAyxkO9ZHPdEa9qlAl62vrkO5MKurzqbYtFuDCy9RhqX2Toh
rSOfbliN27HVd2b8w5/khAu7n071x2F5qzh8B09+WrJqwObO5lV9crKxDmkaaxg7
iA/1pM3sEWbnyPPbggDCkjhBYk1Zvq1zD2bXK5yWf96mQ0+1Xj7YE4JrVf5TpmpR
TW2Gug5u1yY9qNgtw3zs
=cXTi
-----END PGP SIGNATURE-----
1:13 p.m.
On 20.06.2014 15:44, Dennis Gilmore wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Fri, 20 Jun 2014 03:19:55 +0200
poma <pomidorabelisima(a)gmail.com> wrote:
>
> f20:
>
> *filelists.sqlite.bz2 3.5 M
> *filelists.xml.gz 3.3 M
> *primary.sqlite.bz2 2.7 M
> *primary.xml.gz 1.3 M
> *other.sqlite.bz2 1.3 M
> *other.xml.gz 964 K
> *Fedora-20-comps.xml 906 K
> *Fedora-20-comps.xml.gz 210 K
> repomd.xml 3.6 K
>
>
> rawhide:
>
> *filelists.xml.gz 28 M <- xz?
> *filelists.sqlite.xz 25 M
> *primary.sqlite.xz 17 M
> *primary.xml.gz 10 M <- xz?
> *other.sqlite.xz 7.5 M
> *other.xml.gz 5.8 M <- xz?
> *prestodelta.xml.xz 2.9 M
> *comps-rawhide.xml 1.6 M <- ?
> *comps-rawhide.xml.xz 230 K
> repomd.xml 4.2 K <- xz?
>
>
> Why aren't all compressed with the xz, in rawhide?
> Why two *comps* with the same original checksums?
because thats how createrepo works. the gzipped files are not ones you
download. yum and dnf use the sqlite files.
Dennis
"because thats how createrepo works"!? :)
I'm talking in general, why not use the same improved compression for all!
I haven't mentioned an interactive, rpm based, package managers, at all. :)
But when you mention them, it is obvious that we need features and performance of the
delta repodata, which is currently lacking.
Rather than listen to a dandified hullabaloo. :)
poma
2:06 p.m.
"because thats how createrepo works"!? :)
I'm talking in general, why not use the same improved compression for all!
I haven't mentioned an interactive, rpm based, package managers, at all. :)
But when you mention them, it is obvious that we need features and
performance of the delta repodata, which is currently lacking.
Rather than listen to a dandified hullabaloo. :)
I think it would be great to XZ all the things.
For now createrepo still defaults to something else, not XZ.
-Jon
--
-Jon
2:44 p.m.
On Fri, 20 Jun 2014 20:13:00 +0200
poma <pomidorabelisima(a)gmail.com> wrote:
"because thats how createrepo works"!? :)
I'm talking in general, why not use the same improved compression for
all!
Because createrepo wants to keep backward compatibility?
For things that can use xz, they only download the xz files.
For older things that cannot, they can download the gz files.
If your package manager does support xz repodata, you only ever get
those. The mirrors keeping a few trivial sized .gz files doesn't affect
you in any way, and allows people with older tools to still look at the
repos.
The exception in your list is the repomd.xml file. This file is vastly
the most accessed one in repodata. Compressing it with xz (or gz) would
be largely pointless since it's so small already and it would make
everyone have to uncompress it lots and lots of times.
kevin
4:13 p.m.
On 20.06.2014 21:44, Kevin Fenzi wrote:
On Fri, 20 Jun 2014 20:13:00 +0200
poma <pomidorabelisima(a)gmail.com> wrote:
> "because thats how createrepo works"!? :)
> I'm talking in general, why not use the same improved compression for
> all!
Because createrepo wants to keep backward compatibility?
Wait a minute, is that what you're saying someone from EL4 poking around rawhide repo?
:)
Yeah, I see how it is convenient to hold backward compatibility when you need it.
For things that can use xz, they only download the xz files.
For older things that cannot, they can download the gz files.
Old things have to evolve.
Old things should be upgraded.
We are talking about the rawhide repo, right. :)
If your package manager does support xz repodata, you only ever get
those. The mirrors keeping a few trivial sized .gz files doesn't affect
you in any way, and allows people with older tools to still look at the
repos.
Again, these old things! :)
The exception in your list is the repomd.xml file. This file is
vastly
the most accessed one in repodata. Compressing it with xz (or gz) would
be largely pointless since it's so small already and it would make
everyone have to uncompress it lots and lots of times.
Around this we agree. :)
kevin
Although you are not commented delta repodata.
Scary little thing, isn't it. :)
poma
1:28 p.m.
On Sat, Jun 21, 2014 at 11:25 AM, Tim Lauridsen <tim.lauridsen(a)gmail.com> wrote:
On Fri, Jun 20, 2014 at 3:44 PM, Dennis Gilmore <dennis(a)ausil.us> wrote:
>
> because thats how createrepo works. the gzipped files are not ones you
> download. yum and dnf use the sqlite files.
yum is using the sqlite files, dnf uses the .xml.gz files
Would it be too much trouble to use the sqlite data in DNF?
I suppose it would be a step backwards to have our primary (future)
tool using gzip metadata.
--
-Jon
2:54 p.m.
On Sat, 21 Jun 2014 13:28:33 -0500
Jon <jdisnard(a)gmail.com> wrote:
Would it be too much trouble to use the sqlite data in DNF?
I suppose it would be a step backwards to have our primary (future)
tool using gzip metadata.
filed https://bugzilla.redhat.com/show_bug.cgi?id=1111871 on this.
kevin
1:11 a.m.
On Sat, Jun 21, 2014 at 8:28 PM, Jon <jdisnard(a)gmail.com> wrote:
Would it be too much trouble to use the sqlite data in DNF?
I suppose it would be a step backwards to have our primary (future)
tool using gzip metadata.
the information in the sqlite files is orded i a way the yum is using the
data, dnf parses the .xml files into another format used by libsolv.
the sqlite was invented because the yum .xml to sqlite format was slow and
has to happen on every client out there, so it was moved serverside to
speed up things.
Tim
6:26 a.m.
On Sun, Jun 22, 2014 at 8:11 AM, Tim Lauridsen <tim.lauridsen(a)gmail.com> wrote:
On Sat, Jun 21, 2014 at 8:28 PM, Jon <jdisnard(a)gmail.com> wrote:
>
> Would it be too much trouble to use the sqlite data in DNF?
> I suppose it would be a step backwards to have our primary (future)
> tool using gzip metadata.
the information in the sqlite files is orded i a way the yum is using the
data, dnf parses the .xml files into another format used by libsolv.
the sqlite was invented because the yum .xml to sqlite format was slow and
has to happen on every client out there, so it was moved serverside to speed
up things.
Well can't we change create repo to create xml.xz files (in addition
to .gz to preserve backwards compatibility) ?
2:57 p.m.
On Sun, Jun 22, 2014 at 6:26 AM, drago01 <drago01(a)gmail.com> wrote:
On Sun, Jun 22, 2014 at 8:11 AM, Tim Lauridsen
<tim.lauridsen(a)gmail.com> wrote:
>
> On Sat, Jun 21, 2014 at 8:28 PM, Jon <jdisnard(a)gmail.com> wrote:
>>
>> Would it be too much trouble to use the sqlite data in DNF?
>> I suppose it would be a step backwards to have our primary (future)
>> tool using gzip metadata.
>
>
> the information in the sqlite files is orded i a way the yum is using the
> data, dnf parses the .xml files into another format used by libsolv.
> the sqlite was invented because the yum .xml to sqlite format was slow and
> has to happen on every client out there, so it was moved serverside to speed
> up things.
Well can't we change create repo to create xml.xz files (in addition
to .gz to preserve backwards compatibility) ?
--
Perhaps we should make createrepo default to XZ in Fedora rawhide, and
make all the tools do the right thing?
Would be nice if createrepo was influenced by a system-wide config
file, so we could easily implement that idea.
Otherwise we are forced to work around createrepo in our release engineering.
Regardless we should strive to make all repo metadata XZ compressed,
unless there is a good reason not to do that?
--
-Jon
3:29 p.m.
On Sun, Jun 22, 2014 at 9:57 PM, Jon <jdisnard(a)gmail.com> wrote:
On Sun, Jun 22, 2014 at 6:26 AM, drago01 <drago01(a)gmail.com>
wrote:
> On Sun, Jun 22, 2014 at 8:11 AM, Tim Lauridsen <tim.lauridsen(a)gmail.com>
wrote:
>>
>> On Sat, Jun 21, 2014 at 8:28 PM, Jon <jdisnard(a)gmail.com> wrote:
>>>
>>> Would it be too much trouble to use the sqlite data in DNF?
>>> I suppose it would be a step backwards to have our primary (future)
>>> tool using gzip metadata.
>>
>>
>> the information in the sqlite files is orded i a way the yum is using the
>> data, dnf parses the .xml files into another format used by libsolv.
>> the sqlite was invented because the yum .xml to sqlite format was slow and
>> has to happen on every client out there, so it was moved serverside to speed
>> up things.
>
> Well can't we change create repo to create xml.xz files (in addition
> to .gz to preserve backwards compatibility) ?
> --
Perhaps we should make createrepo default to XZ in Fedora rawhide, and
make all the tools do the right thing?
Would be nice if createrepo was influenced by a system-wide config
file, so we could easily implement that idea.
Otherwise we are forced to work around createrepo in our release engineering.
Regardless we should strive to make all repo metadata XZ compressed,
unless there is a good reason not to do that?
Well even if we do that we need to support the old format (i.e fall
back when xz is not available) otherwise old (third party / custom)
repos would stop working.
3:44 p.m.
On Sun, 22 Jun 2014 22:29:30 +0200
drago01 <drago01(a)gmail.com> wrote:
> Perhaps we should make createrepo default to XZ in Fedora
rawhide,
> and make all the tools do the right thing?
We have already done that. rawhide and epel7 repos use
--compress-type xz
> Would be nice if createrepo was influenced by a system-wide
config
> file, so we could easily implement that idea.
> Otherwise we are forced to work around createrepo in our release
> engineering.
>
> Regardless we should strive to make all repo metadata XZ compressed,
> unless there is a good reason not to do that?
Please file a RFE or bug on createrepo if you think it should do
something different with --compress-type xz enabled. ;)
Well even if we do that we need to support the old format (i.e fall
back when xz is not available) otherwise old (third party / custom)
repos would stop working.
Well, it depends on what level of compatibility you want.
Right now with xz repos, RHEL6 instances need to manually install
pyliblzma in order to work with the repos.
In any case, any changes we want in createrepo should funnel to the
maintainer(s) for comment.
kevin
4:20 p.m.
On Sun, Jun 22, 2014 at 10:44 PM, Kevin Fenzi <kevin(a)scrye.com> wrote:
On Sun, 22 Jun 2014 22:29:30 +0200
drago01 <drago01(a)gmail.com> wrote:
> > Perhaps we should make createrepo default to XZ in Fedora rawhide,
> > and make all the tools do the right thing?
We have already done that. rawhide and epel7 repos use
--compress-type xz
> > Would be nice if createrepo was influenced by a system-wide config
> > file, so we could easily implement that idea.
> > Otherwise we are forced to work around createrepo in our release
> > engineering.
> >
> > Regardless we should strive to make all repo metadata XZ compressed,
> > unless there is a good reason not to do that?
Please file a RFE or bug on createrepo if you think it should do
something different with --compress-type xz enabled. ;)
> Well even if we do that we need to support the old format (i.e fall
> back when xz is not available) otherwise old (third party / custom)
> repos would stop working.
Well, it depends on what level of compatibility you want.
Right now with xz repos, RHEL6 instances need to manually install
pyliblzma in order to work with the repos.
In any case, any changes we want in createrepo should funnel to the
maintainer(s) for comment.
Well I mean yum/dnf should still work with repos created by older (non
xz) createrepo.
1:12 p.m.
On 06/21/2014 11:42 AM, Kalev Lember wrote:
On 06/20/2014 03:19 AM, poma wrote:
>
> f20:
> [...]
> *primary.sqlite.bz2 2.7 M
> *primary.xml.gz 1.3 M
>
Is this "Fedora" or "Everything" ?
For f20 "Everything" (comparable to rawhide) I have:
19M *-primary.sqlite.bz2
so looks like xz is helping.
> rawhide:
> [...]
> *primary.sqlite.xz 17 M
> *primary.xml.gz 10 M
Anyone know why the rawhide metadata size is an order of magnitude
bigger than in F20?
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA/CoRA Division FAX: 303-415-9702
3380 Mitchell Lane orion(a)cora.nwra.com
Boulder, CO 80301 http://www.cora.nwra.com
3617
days inactive
3619
days old
16 comments
8 participants
participants (8)
-
Dennis Gilmore -
drago01 -
Jon -
Kalev Lember -
Kevin Fenzi -
Orion Poplawski -
poma -
Tim Lauridsen