Failed Delta RPMs observed in repo

[John Mellor]

Question for the repo managers,

Why do I sometimes see invalid checksums for drpm downloads? E.g: This 
excerpt from the update this morning:

/var/cache/dnf/updates-7fc4c739b3909d9f/packages/selinux-policy-targeted-3.14.4-46.fc31_3.14.4-47.fc31.noarch.drpm:
    md5 mismatch of resultSome packages were not downloaded.
    Retrying.selinux-policy-targeted-3.14.4-47.fc31.noarch.r  14 MB/s | 
    13 MB     00:00

It almost always causes the total download size to be larger than any 
savings from using the drpm format:

Failed Delta RPMs increased 41.3 MB of updates to 43.3 MB (-4.1% wasted)

Is this a repo bug?

[stan]

On Sun, 9 Feb 2020 09:36:54 -0500
John Mellor  wrote:
> Question for the repo managers,
>
> Why do I sometimes see invalid checksums for drpm downloads? E.g:
> This excerpt from the update this morning:
> /var/cache/dnf/updates-7fc4c739b3909d9f/packages/selinux-policy-targeted-3.14.4-46.fc31_3.14.4-47.fc31.noarch.drpm:
> md5 mismatch of resultSome packages were not downloaded.
> Retrying.selinux-policy-targeted-3.14.4-47.fc31.noarch.r  14 MB/s
>
> | 13 MB     00:00
>
> It almost always causes the total download size to be larger than any 
> savings from using the drpm format:
> Failed Delta RPMs increased 41.3 MB of updates to 43.3 MB (-4.1%
>
> wasted)
>
> Is this a repo bug?

TLDR;  seeing it too, but don't know the cause.

I've started seeing this recently as well.  In the same update that
contains the error though, many other drpms will rebuild just fine.
So, my guess is that the local version is somehow different from the
version the drpm was created against, so when it rebuilds, the md5 is
different.  That is, I don't think it is a bug in either the repo or
the drpm builder.  But the fact that it only started happening recently
argues against that opinion.

[George N. White III]

On Sun, 9 Feb 2020 at 17:14, stan via users 
wrote:
> On Sun, 9 Feb 2020 09:36:54 -0500
> John Mellor  wrote:
> > Question for the repo managers,
> >
> > Why do I sometimes see invalid checksums for drpm downloads? E.g:
> > This excerpt from the update this morning:
>
> /var/cache/dnf/updates-7fc4c739b3909d9f/packages/selinux-policy-targeted-3.14.4-46.fc31_3.14.4-47.fc31.noarch.drpm:
> > md5 mismatch of resultSome packages were not downloaded.
> > Retrying.selinux-policy-targeted-3.14.4-47.fc31.noarch.r  14 MB/s
> >
> > | 13 MB     00:00

I saw the same "mismatch" today:

2020-02-09T23:24:45Z SUBDEBUG drpm: spawned 13721: /usr/bin/applydeltarpm
-a noarch /var/cache/dnf/updates-7fc4c739b3909
d9f/packages/selinux-policy-targeted-3.14.4-46.fc31_3.14.4-47.fc31.noarch.drpm
/var/cache/dnf/updates-7fc4c739b3909d9f/p
ackages/selinux-policy-targeted-3.14.4-47.fc31.noarch.rpm
2020-02-09T23:24:58Z SUBDEBUG drpm: 13721: return code: 1, 0
2020-02-09T23:24:58Z INFO Some packages were not downloaded. Retrying.
2020-02-09T23:25:02Z INFO
------------------------------------------------------------------------------------------------------------------------
2020-02-09T23:25:02Z INFO Total
                                      1.9 MB/s |  43 MB     00:22
2020-02-09T23:25:02Z INFO Failed Delta RPMs increased 41.4 MB of updates to
43.3 MB (-4.1% wasted)
> > It almost always causes the total download size to be larger than any
> > savings from using the drpm format:

I grepped "Failed Delta RPMs" in all the dnf logs on my system and this is
the only instance.
> > Failed Delta RPMs increased 41.3 MB of updates to 43.3 MB (-4.1%
> >
> > wasted)
> >
> > Is this a repo bug?
>
> TLDR;  seeing it too, but don't know the cause.
>
> I've started seeing this recently as well.  In the same update that
> contains the error though, many other drpms will rebuild just fine.
> So, my guess is that the local version is somehow different from the
> version the drpm was created against, so when it rebuilds, the md5 is
> different.  That is, I don't think it is a bug in either the repo or
> the drpm builder.  But the fact that it only started happening recently
> argues against that opinion.

[Jonathan Dieter]

On Sun, 2020-02-09 at 09:36 -0500, John Mellor wrote:
> Question for the repo managers,
>
> Why do I sometimes see invalid checksums for drpm downloads?  E.g:
> This excerpt from the update this morning:
> > /var/cache/dnf/updates-7fc4c739b3909d9f/packages/selinux-policy-
> > targeted-3.14.4-46.fc31_3.14.4-47.fc31.noarch.drpm: md5 mismatch of
> > result
> > Some packages were not downloaded. Retrying.
> > selinux-policy-targeted-3.14.4-47.fc31.noarch.r  14 MB/s |  13
> > MB     00:00
>
> It almost always causes the total download size to be larger than any
> savings from using the drpm format:
> > Failed Delta RPMs increased 41.3 MB of updates to 43.3 MB (-4.1%
> > wasted)
>
> Is this a repo bug?

This is not a repo bug, but rather a (small) bug in how selinux-policy-
targeted is packaged.  A drpm is assembled by taking the parts of the
old installed rpm that haven't changed from the local system and then
only downloading the parts that have changed.

The problem is that a file in selinux-policy-targeted changes *after*
it's installed, but dnf doesn't detect that the old installed rpm has
changed until after downloading and attempting to apply the drpm.

As you can see on my local system, there are four files that have
changed after selinux-policy-targeted was installed:
$ sudo rpm -V selinux-policy-targeted
S.5....T.  c /etc/selinux/targeted/contexts/files/file_contexts.local
..5....T.    /var/lib/selinux/targeted/active/commit_num
S.5....T.    /var/lib/selinux/targeted/active/file_contexts
.......T.    /var/lib/selinux/targeted/active/homedir_template
S.5....T.    /var/lib/selinux/targeted/active/policy.kern
.M.......  g /var/lib/selinux/targeted/active/policy.linked
.......T.    /var/lib/selinux/targeted/active/seusers
.......T.    /var/lib/selinux/targeted/active/users_extra

The file in /etc is marked as a config file, so won't cause the
deltarpm to fail, but the changes to the other three will.  Arguably,
any files that may change on the local system should be marked as
config files.

Jonathan