tracker in F16

[Lennart Poettering]

On Mon, 19.09.11 12:43, Matthias Clasen (mclasen at redhat.com) wrote:

> > a) tracker uses inotify recursively and creates a massive number of
> > watches due to that. That is both ugly and doesn't scale. Tracker
> > apparently tries to not take up the full pool of inotfy handles the
> > system provides, but that won't help if you have more than one user on
> > the system. The solution here should probably be fanotify, which allows
> > proper recursive file system watches. So far fanotify has been
> > accessible to root only, which is presumably why tracker doesn't use
> > it. However, the solution here cannot be to work around that fact by
> > using inotify, but must be to invest the necessary kernel work to make
> > fanotify useful from unprivileged processes. 
> > 
> > b) We still don't have a way to detect offline modification of
> > directories. That means detecting changes to the home directory made
> > offline is very expensive. btrfs now has hooks to improve the situation,
> > but ext4 still hasn't. Does tracker at least use the btrfs hooks? (btrfs
> > provides a log of changes to userspace, which can be used for that. Another
> > solution are recursive directory change timestamps).
> > 
> > I'd really prefer if we could fix these fundamental issues before we
> > enable tracker. To me it appears here as if we are trying to make the
> > second step before the first.
> > 
> > [ And there are acouple of other things I'd like to see changed. For
> > example, I am pretty sure that tracker's open() calls to files should
> > not be considered accesses in regards to access time. O_NOATIME should
> > be used here, which would reduce the amount of disk writes
> > substantially. Also, tracker appears to BSD lock all files it
> > accesses. That looks quite borked. Which other tool is it synchronizing
> > against here? This looks unsecure to do (because the files are often
> > accessible to others), and since these locks are advisory only there
> > needs to be a strict protocol followed by everybody else accessing these
> > files, which I guarantee you there isn't since these are basically all
> > the user's files. Moreover it appears tracker is mixing BSD and POSIX
> > locks, which is dangerous due to ABBA, in particular when used on NFS
> > directories, which will just end up in total chaos since Linux is so
> > stupid to "upgrade" BSD locks on NFS shares to POSIX locks on the
> > way. In any case you should NEVER EVER use POSIX locking, since it is
> > compltely borked anyway. The locking must go. I also see a massive
> > amount of futex calls in strace, i.e. probably some mutexes thrown in
> > the mix to make the locking problems even more interesting, which makes
> > my fingernails roll up, since they apparently are congested all the
> > time? ]
> > 
> 
> Good stuff, Lennart.
> 
> But it would probably have much more effect in 
> https://bugzilla.gnome.org/browse.cgi?product=tracker
> or 
> https://mail.gnome.org/archives/tracker-list/
> 
> Can you put it there ?

I already filed two bugs there today. And the inotify/offline
modification issues the tracker folks are well aware of, because people
(including me) told them that over and over again over the years.

This mail was mostly intended as a summary for fedora, and for pointing
out that these core issues have not been addressed in the last
years. There's no news in all of this. Maybe except for the fact that
Fedora appears willing to ignore the issues that previously
mattered.

Anyway, I will forward this to Jürg an Martyn, maybe they have something
to say something new about this.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.