F29 System Wide Change: Zchunk Metadata

[Jan Kurik]

= Proposed System Wide Change: Zchunk Metadata =
https://fedoraproject.org/wiki/Changes/Zchunk_Metadata


Owner(s):
  * Jonathan Dieter <jdieter at gmail dot com>
  * Neal Gompa <ngompa13 at gmail dot com>


All dnf repository metadata will be compressed with the zchunk format
rather than xz or gzip.


== Detailed description ==
Currently Fedora's repository metadata is compressed using the xz and gzip
formats.  Zchunk is a new compression format designed to allow for highly
efficient deltas.  When Fedora's metadata is compressed using zchunk, dnf
will download only the differences between any earlier copies of the
metadata and the current version.


== Scope ==
* Proposal owners:
** Package zchunk for Fedora
** Get the pull requests to enable zchunk in dnf, libdnf, librepo, libsolv
and createrepo_c merged upstream
*** https://github.com/rpm-software-management/librepo/pull/127
*** https://github.com/openSUSE/libsolv/pull/270
*** https://github.com/rpm-software-management/dnf/pull/1107
*** https://github.com/rpm-software-management/libdnf/pull/478
*** https://github.com/rpm-software-management/createrepo_c/pull/92
** Create a new package for Fedora's zchunk dictionaries.

* Other developers:
Fedora Infrastructure needs to start creating zchunked metadata

* Release engineering:
#7600 [https://pagure.io/releng/issue/7600]

** List of deliverables:
Zchunk repository metadata

* Policies and guidelines:
Packaging guidelines are not affected by this change.

* Trademark approval:
N/A (not needed for this Change)
-- 
Jan Kuřík
JBoss EAP Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic

[Josh Boyer]

On Mon, Jul 2, 2018 at 1:19 AM Jan Kurik <jkurik(a)redhat.com&gt; wrote:
>  = Proposed System Wide Change: Zchunk Metadata =
>  https://fedoraproject.org/wiki/Changes/Zchunk_Metadata
>
>
>  Owner(s):
>    * Jonathan Dieter <jdieter at gmail dot com>
>    * Neal Gompa <ngompa13 at gmail dot com>
>
>
>  All dnf repository metadata will be compressed with the zchunk format rather than xz or
> gzip.
>
>
>  == Detailed description ==
>  Currently Fedora's repository metadata is compressed using the xz and gzip formats. 
> Zchunk is a new compression format designed to allow for highly efficient deltas.  When
> Fedora's metadata is compressed using zchunk, dnf will download only the differences
> between any earlier copies of the metadata and the current version. 
How does this work with distribution upgrades?  If we only produce
zchunk compressed metadata, but dnf in f28 doesn't even understand it,
that seems wrong.

josh

[Miro Hrončok]

On 9.7.2018 23:52, Josh Boyer wrote:
>  On Mon, Jul 2, 2018 at 1:19 AM Jan Kurik <jkurik(a)redhat.com&gt;
> wrote:
> >
> > = Proposed System Wide Change: Zchunk Metadata =
> > https://fedoraproject.org/wiki/Changes/Zchunk_Metadata
> >
> >
> > Owner(s):
> >    * Jonathan Dieter <jdieter at gmail dot com>
> >    * Neal Gompa <ngompa13 at gmail dot com>
> >
> >
> > All dnf repository metadata will be compressed with the zchunk format rather than xz
> or gzip.
> >
> >
> > == Detailed description ==
> > Currently Fedora's repository metadata is compressed using the xz and gzip
> formats.  Zchunk is a new compression format designed to allow for highly efficient
> deltas.  When Fedora's metadata is compressed using zchunk, dnf will download only the
> differences between any earlier copies of the metadata and the current version.
>  
>  How does this work with distribution upgrades?  If we only produce
>  zchunk compressed metadata, but dnf in f28 doesn't even understand it,
>  that seems wrong.
>   
As I understood it form one fo the previous discussions about the topic, 
zchunk will be only provided together with xz. Older dnf will simply 
request xz and newer dnf will request zchunk but fallback to xz if it is 
not available or checksums don't match or whatever error occurs.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Mon, Jul 9, 2018 at 6:28 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 9.7.2018 23:52, Josh Boyer wrote:
>  > On Mon, Jul 2, 2018 at 1:19 AM Jan Kurik <jkurik(a)redhat.com&gt; wrote:
>  >>
>  >> = Proposed System Wide Change: Zchunk Metadata =
>  >> https://fedoraproject.org/wiki/Changes/Zchunk_Metadata
>  >>
>  >>
>  >> Owner(s):
>  >>    * Jonathan Dieter <jdieter at gmail dot com>
>  >>    * Neal Gompa <ngompa13 at gmail dot com>
>  >>
>  >>
>  >> All dnf repository metadata will be compressed with the zchunk format rather
> than xz or gzip.
>  >>
>  >>
>  >> == Detailed description ==
>  >> Currently Fedora's repository metadata is compressed using the xz and gzip
> formats.  Zchunk is a new compression format designed to allow for highly efficient
> deltas.  When Fedora's metadata is compressed using zchunk, dnf will download only the
> differences between any earlier copies of the metadata and the current version.
>  >
>  > How does this work with distribution upgrades?  If we only produce
>  > zchunk compressed metadata, but dnf in f28 doesn't even understand it,
>  > that seems wrong.
>  >
>
>  As I understood it form one fo the previous discussions about the topic,
>  zchunk will be only provided together with xz. Older dnf will simply
>  request xz and newer dnf will request zchunk but fallback to xz if it is
>  not available or checksums don't match or whatever error occurs. 
Can we update the language on the Change page to clarify that?

josh

[Jonathan Dieter]

On Mon, 2018-07-09 at 18:48 -0400, Josh Boyer wrote:
>  Can we update the language on the Change page to clarify that?
I thought I had explained at:
https://fedoraproject.org/wiki/Changes/Zchunk_Metadata#Upgrade.2Fcompat
ibility_impact

Is there something I should put that would make it clearer?

Jonathan

[Jonathan Dieter]

On Tue, 2018-07-10 at 08:34 +0100, Jonathan Dieter wrote:
>  On Mon, 2018-07-09 at 18:48 -0400, Josh Boyer wrote:
>  > Can we update the language on the Change page to clarify that?
>  
>  I thought I had explained at:
>  https://fedoraproject.org/wiki/Changes/Zchunk_Metadata#Upgrade.2Fcompat
>  ibility_impact
>  
>  Is there something I should put that would make it clearer? 
Ok, I saw your comment on the FESCO ticket, and I see why it's
confusing.

I've updated the summary to say:
"All dnf repository metadata will be compressed with the zchunk format
in addition to xz or gzip."

Jonathan