[PATCH] build-functions: Generate OSTree static deltas for rawhide

[Colin Walters]

Adam Miller pointed out that OSTree downloads are slow.  We've been
working on "static deltas" for a while now, and they're ready for some
real world testing.

For more information, see:

 - https://mail.gnome.org/archives/ostree-list/2015-March/msg00005.html
 - https://bugzilla.gnome.org/show_bug.cgi?id=721799

This is a *basic* implementation - a more sophisticated script is
necessary to generate "spanning deltas" across multiple revisions.

The deltas make more sense for releases that are slower than daily -
for example, there's discussion of doing 2 week stable Fedora Atomic
Host updates.

In that model, it'd be much saner to generate deltas from say the last
update, and possibly the update before it.

For "rolling" repositories, the audience is far smaller, and that's
exactly what the incrementalism of the default OSTree archive-z2
repository is designed for.

The delta size is going to vary based on the input.  Some numbers are
in the bugzilla.  For reference, a static delta based on the the
latest rawhide commit to the previous is 5MB.
---
 scripts/build-functions.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/build-functions.sh b/scripts/build-functions.sh
index 49f6252..2a3bcd5 100644
--- a/scripts/build-functions.sh
+++ b/scripts/build-functions.sh
@@ -146,6 +146,12 @@ $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "if [ ! -d
$ATOMICREPO ]; then os
 $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "git clone
https://git.fedorahosted.org/git/fedora-atomic.git $ATOMIC && pushd $ATOMIC
&& git log -n 1 --pretty='%h: %ci - %s' && git checkout
${GIT_BRANCH}"
 $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "cd $ATOMIC && sed -i -e
's|mirrorlist=.*$|baseurl=http://kojipkgs.fedoraproject.org/mash/${DIST}/x86_64/os/|g'
fedora*repo"
 $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "rpm-ostree compose tree
--repo=$ATOMICREPO $ATOMIC/fedora-atomic-docker-host.json >$logdir/atomic"
+if [ "$BRANCHED" = "rawhide"]; then
+    # We're only generatic static deltas for rawhide.
+    REF="fedora-atomic/$BRANCHED/x86_64/docker-host"
+    $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "ostree --repo=$ATOMICREPO
rev-parse ${REF} && ostree --repo=$ATOMICREPO static-delta generate ${REF}"
+    # TODO: GPG sign it
+fi
 log "finished atomic tree creation"
 }
 
-- 
1.8.3.1

[Dennis Gilmore]

On Wednesday, May 13, 2015 09:10:23 PM Colin Walters wrote:
>  Adam Miller pointed out that OSTree downloads are slow.  We've
> been
>  working on "static deltas" for a while now, and they're ready for some
>  real world testing.
>  
>  For more information, see:
>  
>   - https://mail.gnome.org/archives/ostree-list/2015-March/msg00005.html
>   - https://bugzilla.gnome.org/show_bug.cgi?id=721799
>  
>  This is a *basic* implementation - a more sophisticated script is
>  necessary to generate "spanning deltas" across multiple revisions.
>  
>  The deltas make more sense for releases that are slower than daily -
>  for example, there's discussion of doing 2 week stable Fedora Atomic
>  Host updates.
>  
>  In that model, it'd be much saner to generate deltas from say the last
>  update, and possibly the update before it.
>  
>  For "rolling" repositories, the audience is far smaller, and that's
>  exactly what the incrementalism of the default OSTree archive-z2
>  repository is designed for.
>  
>  The delta size is going to vary based on the input.  Some numbers are
>  in the bugzilla.  For reference, a static delta based on the the
>  latest rawhide commit to the previous is 5MB.
>  ---
>   scripts/build-functions.sh | 6 ++++++
>   1 file changed, 6 insertions(+)
>  
>  diff --git a/scripts/build-functions.sh b/scripts/build-functions.sh
>  index 49f6252..2a3bcd5 100644
>  --- a/scripts/build-functions.sh
>  +++ b/scripts/build-functions.sh
>  @@ -146,6 +146,12 @@ $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "if [ !
>  -d $ATOMICREPO ]; then os $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell
>  "git clone https://git.fedorahosted.org/git/fedora-atomic.git $ATOMIC &&
>  pushd $ATOMIC && git log -n 1 --pretty='%h: %ci - %s' && git
> checkout
>  ${GIT_BRANCH}" $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "cd $ATOMIC
>  && sed -i -e
>  's|mirrorlist=.*$|baseurl=http://kojipkgs.fedoraproject.org/mash/${DIST}/x8
>  6_64/os/|g' fedora*repo" $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell
>  "rpm-ostree compose tree --repo=$ATOMICREPO
>  $ATOMIC/fedora-atomic-docker-host.json >$logdir/atomic" +if [
> "$BRANCHED" =
>  "rawhide"]; then
>  +    # We're only generatic static deltas for rawhide.
>  +    REF="fedora-atomic/$BRANCHED/x86_64/docker-host"
>  +    $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "ostree
>  --repo=$ATOMICREPO rev-parse ${REF} && ostree --repo=$ATOMICREPO
>  static-delta generate ${REF}" +    # TODO: GPG sign it
>  +fi
>   log "finished atomic tree creation"
>   } 
Other than F22 is nearly out the door, any reason we should not do this for 
branched also?

Dennis

[Colin Walters]

On Thu, May 14, 2015, at 12:21 PM, Dennis Gilmore wrote:
>  
>  Other than F22 is nearly out the door, any reason we should not do this for 
>  branched also? 
I'd like a little bit of real world experience with it in rawhide before it's
visible to stable
update clients if possible.

(A variation of this would be to have static deltas for f22->rawhide as well...but
 that blocks on https://fedorahosted.org/rel-eng/ticket/6125 as deltas only work inside
 one repo)