Many directories without owning packages

[Dridi Boukelmoune]

Hello,

I was surprised to see /usr/share/texlive on my system although I
remembered very well removing it months ago. It turned out to be
caused by two rpmsave files, although some *empty* directories weren't
removed:

$ find /usr/share/texlive/
/usr/share/texlive/
/usr/share/texlive/texmf-dist
/usr/share/texlive/texmf-dist/web2c
/usr/share/texlive/texmf-dist/web2c/fmtutil.cnf.rpmsave
/usr/share/texlive/texmf-dist/web2c/updmap.cfg.rpmsave
/usr/share/texlive/texmf-dist/tex
/usr/share/texlive/texmf-dist/tex/latex
/usr/share/texlive/texmf-dist/tex/latex/misc
/usr/share/texlive/tlpkg
/usr/share/texlive/tlpkg/TeXLive
/usr/share/texlive/tlpkg/tlpostcode
/usr/share/texlive/tlpkg/translations

But then I decided to check my whole /usr tree and found more orphans
than I expected.

My crude data gathering:

$ find /usr/ -type d -exec rpm -qf {} \; |
grep 'not owned' |
tee not_owned.txt

And the results on my current fc24 laptop:

$ wc -l <not_owned.txt
828

$ sort not_owned.txt | awk '{print $2}' | head
/usr/include/at-spi2-atk/2.0
/usr/include/at-spi2-atk
/usr/include/mono-2.0
/usr/include/mono-2.0/mono/cil
/usr/include/mono-2.0/mono
/usr/include/mono-2.0/mono/jit
/usr/include/mono-2.0/mono/metadata
/usr/include/mono-2.0/mono/utils
/usr/include/nss3/templates
/usr/lib64/akonadi/contact/editorpageplugins

Most directories contain files that are owned by packages, although I
could find an interesting number of actual orphan files on my /usr
tree! Upon quick inspection it looks like remains from old packages
after they got upgraded. Some of the remains date fc22. There's an
insignificant bunch of files I installed in /usr/local too.

$ find $(awk '{print $2}' <not_owned.txt) -type f |
grep -v ' ' | # I don't want to deal with spaces in file names
xargs rpm -qf |
grep 'not owned' |
wc -l
find: ‘/usr/lost+found’: Permission denied
find: ‘/usr/share/rhel/secrets’: Permission denied
1191

Sparing you more shell soup, under /usr I currently have 10738 files
owned by a package whose parent directories are *not all* owned,
belonging to 290 packages.

Once my /usr is clean I will try to monitor it after each upgrade. It
should be easy enough to script, although scanning 18GB of /usr
against the whole RPM database takes some time.

I don't know whether we have tools checking this automatically, and if
DNF/RPM tells you about paths that weren't removed along with their
packages during a transaction, I'm interested in having this
information on every `dnf update`.

Cheers,
Dridi

[Dominik 'Rathann' Mierzejewski]

On Thursday, 06 October 2016 at 18:58, Dridi Boukelmoune wrote:
>  Hello,
>  
>  I was surprised to see /usr/share/texlive on my system although I
>  remembered very well removing it months ago. It turned out to be
>  caused by two rpmsave files, although some *empty* directories weren't
>  removed: 
[...]
That's normal rpm behaviour. It only deletes directories if they're
owned by the removed package and empty.

>  But then I decided to check my whole /usr tree and found more
> orphans
>  than I expected. 
Feel free to open bugs against the offending packages. Complete file and
directory ownership is mandated in the packaging guidelines.

Regards,
Dominik
-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
RPMFusion http://rpmfusion.org
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Dridi Boukelmoune]

Hi,

This is mass bug filing material, I have been under the weather
shortly after the first message on this thread but I have made more
progress now. Enough to CC the packaging list too.

On Fri, Oct 7, 2016 at 9:55 AM, Dominik 'Rathann' Mierzejewski
<dominik(a)greysector.net&gt; wrote:
>  On Thursday, 06 October 2016 at 18:58, Dridi Boukelmoune wrote:
> > Hello,
> >
> > I was surprised to see /usr/share/texlive on my system although I
> > remembered very well removing it months ago. It turned out to be
> > caused by two rpmsave files, although some *empty* directories weren't
> > removed:
>  [...]
>  That's normal rpm behaviour. It only deletes directories if they're
>  owned by the removed package and empty. 
It was so obvious after you pointed it out that not figuring it out
myself felt quite embarrassing :)

> > But then I decided to check my whole /usr tree and found more
> orphans
> > than I expected.
>
>  Feel free to open bugs against the offending packages. Complete file and
>  directory ownership is mandated in the packaging guidelines. 
Around the same time I had two ongoing reviews and realized that the
fedora-review tool warns about incomplete ownership inside built
packages, so I figured I could do something with DNF's filenames
databases via repoquery or something else.

However this approach has a high space and time complexity, because
for every single binary package it must check that all directories are
owned by the package or one of its dependencies with the exception of
directories owned by the filesystems package (and maybe others?)

So instead I wrote a proof of concept shell script that relies on installed
RPMs, because unless you purposefully install packages with missing
dependencies, a missing ownership of a directory _with owned contents_
is evidence of a packaging bug.

The script took about 30 minutes to run on my laptop, so I rewrote it in
bash to avoid forking and got below the 10 minutes mark. I then added
some cute reporting capabilities.

Example output for the /boot sub-tree on my daily driver:

    grub2-efi-2.02-0.34.fc24.x86_64
    └── /boot/efi/EFI/fedora
    mactel-boot-0.9-13.fc24.x86_64
    ├── /boot/efi/.VolumeIcon.icns
    ├── /boot/efi/System/Library/CoreServices/SystemVersion.plist
    ├── /boot/efi/System/Library/CoreServices/boot.efi
    └── /boot/efi/mach_kernel
    shim-0.8-9.x86_64
    ├── /boot/efi/EFI/BOOT/BOOTX64.EFI
    └── /boot/efi/EFI/BOOT/fallback.efi

For each package, the files that are listed have at least one parent
directory not owned.

Example's first culprits:

    $ sudo rpm -qf /boot /boot/efi /boot/efi/EFI
    filesystem-3.2-37.fc24.x86_64
    file /boot/efi is not owned by any package
    file /boot/efi/EFI is not owned by any package

For my RPMs sample, results are incomplete because:

- legacy pre-UsrMove paths are ignored
- symbolic links aren't followed
- no dependencies check

I did check dozens of results to refine the exceptions until I could no
longer find a false positive at random. The overall algorithm is dead
simple, so simple that I could implement it in bourne shell. It overlooks
many cases and probably has a high false-negative rate, but even
with such limitations the results are telling.

On my system, I found 2991 files "not fully owned" in 354 packages:

    $ egrep '^\W' not-owned.txt | wc -l
    2991
    $ egrep '^\w' not-owned.txt | wc -l
    354

I don't know how many directories are missing an owner.

The PoC can most likely be pushed further and even be comprehensive,
using a much much faster approach like DNF did for package resolution.
At this point it can only point out what's partially wrong, accurately.

The script is available [1] for anyone to try and review. It would work
best if people using different spins could try it, because I have less than
5000 packages installed.

Thoughts?

Cheers,
Dridi

PS. yes, I enjoy shell scripting, a lot

[1] https://dridi.fedorapeople.org/devel/not-fully-owned.sh

[Zbigniew Jędrzejewski-Szmek]

On Tue, Nov 01, 2016 at 01:05:03PM +0100, Dridi Boukelmoune wrote:
>  Example output for the /boot sub-tree on my daily driver:
>  
>      grub2-efi-2.02-0.34.fc24.x86_64
>      └── /boot/efi/EFI/fedora
>      mactel-boot-0.9-13.fc24.x86_64
>      ├── /boot/efi/.VolumeIcon.icns
>      ├── /boot/efi/System/Library/CoreServices/SystemVersion.plist
>      ├── /boot/efi/System/Library/CoreServices/boot.efi
>      └── /boot/efi/mach_kernel
>      shim-0.8-9.x86_64
>      ├── /boot/efi/EFI/BOOT/BOOTX64.EFI
>      └── /boot/efi/EFI/BOOT/fallback.efi 
It'd be even more useful to print owning package after each file name,
so that people can easily search for their own packages.

>  Thoughts? 
Why not paste the output of the script here?

Zbyszek

[Dridi Boukelmoune]

On Tue, Nov 1, 2016 at 2:00 PM, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Tue, Nov 01, 2016 at 01:05:03PM +0100, Dridi Boukelmoune wrote:
> > Example output for the /boot sub-tree on my daily driver:
> >
> >     grub2-efi-2.02-0.34.fc24.x86_64
> >     └── /boot/efi/EFI/fedora
> >     mactel-boot-0.9-13.fc24.x86_64
> >     ├── /boot/efi/.VolumeIcon.icns
> >     ├── /boot/efi/System/Library/CoreServices/SystemVersion.plist
> >     ├── /boot/efi/System/Library/CoreServices/boot.efi
> >     └── /boot/efi/mach_kernel
> >     shim-0.8-9.x86_64
> >     ├── /boot/efi/EFI/BOOT/BOOTX64.EFI
> >     └── /boot/efi/EFI/BOOT/fallback.efi
>
>  It'd be even more useful to print owning package after each file name,
>  so that people can easily search for their own packages. 
I don't see how that's a problem, I had no issue looking for the
packages I maintain.

If you want this kind of output, terminate the script at the end of
the `while read perm path` loop.

> > Thoughts?
>  Why not paste the output of the script here? 
I can share that, I did some filtering in my report to remove non Fedora
packages. If people maintaining RPM Fusion see this message, I had
openshot, kdenlive and vlc packages or sub-packages in my results.

For Fedora itself, please find my report [1] probably too big to
inline in this email. I have also updated the script to deal with a
corner case and only use ASCII characters.

Cheers,
Dridi

[1] https://paste.fedoraproject.org/467731/14780369/

[Zbigniew Jędrzejewski-Szmek]

On Tue, Nov 01, 2016 at 10:51:18PM +0100, Dridi Boukelmoune wrote:
>  [1] https://paste.fedoraproject.org/467731/14780369/ 
systemd-229-16.fc24.x86_64
|-- /etc/dbus-1/system.d/org.freedesktop.hostname1.conf
|-- /etc/dbus-1/system.d/org.freedesktop.locale1.conf
|-- /etc/dbus-1/system.d/org.freedesktop.login1.conf
|-- /etc/dbus-1/system.d/org.freedesktop.network1.conf
|-- /etc/dbus-1/system.d/org.freedesktop.resolve1.conf
|-- /etc/dbus-1/system.d/org.freedesktop.systemd1.conf
|-- /etc/dbus-1/system.d/org.freedesktop.timedate1.conf
|-- /usr/share/factory/etc/nsswitch.conf
|-- /usr/share/factory/etc/pam.d/other
'-- /usr/share/factory/etc/pam.d/system-auth
systemd-container-229-16.fc24.x86_64
|-- /etc/dbus-1/system.d/org.freedesktop.import1.conf
'-- /etc/dbus-1/system.d/org.freedesktop.machine1.conf

Fixed in systemd-232-2.fc26.

Zbyszek

[Dridi Boukelmoune]

On Fri, Nov 4, 2016 at 2:59 PM, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  Fixed in systemd-232-2.fc26. 
Zbigniew, thanks for the prompt fix, but there goes my dream to file a
bug against systemd :-p

I'm personally waiting for Fedora 25 to be out, and after upgrading I
will start looking at the mass bug filing process. It'd be nice if
people with different sets of packages gave it a try too. On my
machine, more than 7% of the packages installed from Fedora
repositories have this ownership issue. Filing bugs may not be the
best thing to do if 7% of the whole package collection have ownership
issues.

Thoughts?

Cheers,
Dridi

[Matthew Miller]

On Fri, Nov 04, 2016 at 04:00:21PM +0100, Dridi Boukelmoune wrote:
>  I'm personally waiting for Fedora 25 to be out, and after
> upgrading I
>  will start looking at the mass bug filing process. It'd be nice if 
Since F25 is branched off of what-will-become-F26 already, I suggest
looking at Rawhide. (And you wouldn't have to wait.)


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Matthew Miller]

On Fri, Nov 04, 2016 at 04:00:21PM +0100, Dridi Boukelmoune wrote:
>  repositories have this ownership issue. Filing bugs may not be the
>  best thing to do if 7% of the whole package collection have ownership
>  issues. 
It might be something to file a ticket with FESCo about and to work
with a proven packager to address across the distro. Bugs filed or not.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Zbigniew Jędrzejewski-Szmek]

>  Zbigniew, thanks for the prompt fix, but there goes my dream to file
> a
>  bug against systemd :-p 
Don't worry, we already have plenty :)

>  I'm personally waiting for Fedora 25 to be out, and after
> upgrading I
>  will start looking at the mass bug filing process. It'd be nice if
>  people with different sets of packages gave it a try too. On my
>  machine, more than 7% of the packages installed from Fedora
>  repositories have this ownership issue. Filing bugs may not be the
>  best thing to do if 7% of the whole package collection have ownership
>  issues. 
I think you should apply for provenpackager privileges and fix them
yourself.

Justification: lots of those are really simple bugs, with a fix like
  %files
- %{python2_sitelib}/whatever/*
+ %{python2_sitelib}/whatever
Filing a bug is more work than fixing this (even for you, and then you
have to involve the maintainer, so in all the work is quadrupled between
all the parties). Many of those are really simple packaging mistakes and
a good thing to start with for a fresh provenpackager.

I now fixed and built kobo and imagefactory, and pushed a fix without
building for anaconda. On your list, there's still quite a few easy
ones. For the more complicated ones it might be worthwhile to file
bugs.

On Fri, Nov 04, 2016 at 11:32:11AM -0400, Matthew Miller wrote:
>  On Fri, Nov 04, 2016 at 04:00:21PM +0100, Dridi Boukelmoune wrote:
>  > repositories have this ownership issue. Filing bugs may not be the
>  > best thing to do if 7% of the whole package collection have ownership
>  > issues.
>  
>  It might be something to file a ticket with FESCo about and to work
>  with a proven packager to address across the distro. Bugs filed or not. 
I think that in this case cutting straight to the chase
(i.e. committing the fix) is the best option. Those are simple issues
and not really worth the bureaucracy, imo.

Zbyszek

[Fedora-packaging] Re: Many directories without owning packages

[Dominik 'Rathann' Mierzejewski]

On Tuesday, 01 November 2016 at 13:05, Dridi Boukelmoune wrote:
>  Hi,
>  
>  This is mass bug filing material, I have been under the weather
>  shortly after the first message on this thread but I have made more
>  progress now. Enough to CC the packaging list too. 
Actually, a check for this would be useful to have in both
fedora-review and in automated bodhi tests.

Regards,
Dominik
-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
RPMFusion http://rpmfusion.org
"Faith manages."
        -- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"

[Fedora-packaging] Re: Many directories without owning packages

[Alec Leamas]

On 02/11/16 15:55, Dominik 'Rathann' Mierzejewski wrote:
>  On Tuesday, 01 November 2016 at 13:05, Dridi Boukelmoune wrote:
>  Actually, a check for this would be useful to have in both
>  fedora-review 
Actually, f-r is testing this since long. However, the review approach 
is different: does this package use/install directories with bad 
ownerships vs the overall "are all directories sane" approach.

I'm still curious if this elegant shell code could be used to enhance 
the current tests f-r has. As noted, they are extremely expensive, in a 
class of it's own besides the build and install tasks.


Cheers!

--alec

[Fedora-packaging] Re: Many directories without owning packages

[Dridi Boukelmoune]

>  I'm still curious if this elegant shell code could be used to
> enhance the 
current tests f-r has. As noted, they are extremely expensive, in a
class
of it's own besides the build and install tasks.

I don't think so, f-r works on packages built from a source rpm. This on
the other hand is a brute-force search on all installed packages.

I'm sure we'd find more packages if people with different sets of packages
gave it a try. For instance I run the Xfce spin and tend to do software
development on this laptop.

However, a check similar or identical to the one in f-r could be made more
convenient to package maintainers via a fedpkg check-ownership command for
instance. If that's not already available, I'm not very good at looking up
that sort of thing...

Cheers,
Dridi

PS. define elegant, I'd say it's rather hackish :)

[Fedora-packaging] Re: Many directories without owning packages

[Alec Leamas]

On 02/11/16 17:49, Dridi Boukelmoune wrote:
> > I'm still curious if this elegant shell code could be used to
> enhance
>  the current tests f-r has. As noted, they are extremely expensive, in a
>  class of it's own besides the build and install tasks.
>
>  I don't think so, f-r works on packages built from a source rpm. This on
>  the other hand is a brute-force search on all installed packages. 

Actually, f-r installs the built package in a a mock chroot, and this is 
accessible for tests-. E. g., f-r runs rpmlint not only on the result 
packages but also on the installed ones in the chroot.

So, I have this idea to do the same brute-force method on the chroot...


Cheers!

--alec