rename: No such file or directory (errno = 2)

[Vladimir Marek]

Hi,

Since I'm usually using several mail clients (from phone, old mutt,
mutt-kz) it happens quite often that mutt complains.

rename: No such file or directory (errno = 2)

and the only thing I can do about it is to kill whole mutt. What I
believe is happening is that if I view unread mail, mutt remembers that
it has to move the mail once it leaves current inbox. But before that,
someone else touches the mail (probably moving the message from 'new'
directory to 'cur'. After that mutt-kz can't find the original file and
complains with the error message.

This seems to be problem in the mutt maildir handling rather than in the
notmuch addon, still I wonder if you saw that or even better have some
plans on tackling that :)

Cheers

[Suvayu Ali]

Hi Vladimir,

On Mon, May 13, 2013 at 10:36:07AM +0200, Vladimir Marek wrote:
> Hi,
>
> Since I'm usually using several mail clients (from phone, old mutt,
> mutt-kz) it happens quite often that mutt complains.
>
> rename: No such file or directory (errno = 2)
>
> and the only thing I can do about it is to kill whole mutt. What I
> believe is happening is that if I view unread mail, mutt remembers that
> it has to move the mail once it leaves current inbox. But before that,
> someone else touches the mail (probably moving the message from 'new'
> directory to 'cur'. After that mutt-kz can't find the original file and
> complains with the error message.

I think this is a problem with mutt-kz not syncing the notmuch database
as it updates the maildir flags for the email.  I have noticed this
while using emacs-notmuch, notmuch cli, and mutt-kz.

Now consider the following scenario.  Mutt sees new mail in a maildir,
then you read it.  Now until the next notmuch sync (say with `notmuch
new' from a cron job or OfflineIMAP hook) notmuch thinks the file is
<maildir>/new/<some_name>,N.  Whereas mutt has renamed it to
<maildir>/cur/<some_name>,S!  This can happen everytime mutt updates a
maildir flag (IOW renames a message file).  IMO, this is an inherrent
issue with the maildir format.  So you will see this out-of-sync
behaviour when mutt flags an email as "old", read, or adds the replied
flag.

I find this behaviour quite inconvenient at times and planned to take a
look at it.  But then both lack of time, and lack of understanding of
mutt internals have been in the way :-p.
> This seems to be problem in the mutt maildir handling rather than in the
> notmuch addon, still I wonder if you saw that or even better have some
> plans on tackling that :)

I thought one possible solution might be introducing a hook to what ever
function that does the file renaming when maildir flags are updated.
I'm not sure how one can introduce a hook though.  Another option would
be to add a inotify watch on the maildir currently mutt is viewing.  And
follow-up on any inotify events.

Hopefully I understood your question correctly and my comments make
sense to you.

Cheers,

:)

[Vladimir Marek]

> > Since I'm usually using several mail clients (from phone, old mutt,
> > mutt-kz) it happens quite often that mutt complains.
> >
> > rename: No such file or directory (errno = 2)
> >
> > and the only thing I can do about it is to kill whole mutt. What I
> > believe is happening is that if I view unread mail, mutt remembers that
> > it has to move the mail once it leaves current inbox. But before that,
> > someone else touches the mail (probably moving the message from 'new'
> > directory to 'cur'. After that mutt-kz can't find the original file and
> > complains with the error message.
>
> I think this is a problem with mutt-kz not syncing the notmuch database
> as it updates the maildir flags for the email.  I have noticed this
> while using emacs-notmuch, notmuch cli, and mutt-kz.
>
> Now consider the following scenario.  Mutt sees new mail in a maildir,
> then you read it.  Now until the next notmuch sync (say with `notmuch
> new' from a cron job or OfflineIMAP hook) notmuch thinks the file is
> <maildir>/new/<some_name>,N.  Whereas mutt has renamed it to
> <maildir>/cur/<some_name>,S!  This can happen everytime mutt updates a
> maildir flag (IOW renames a message file).  IMO, this is an inherrent
> issue with the maildir format.  So you will see this out-of-sync
> behaviour when mutt flags an email as "old", read, or adds the replied
> flag.

Right, that's also what I think is happening. Although when I try to
provocate the change deliberately, I'm not successful.
> I find this behaviour quite inconvenient at times and planned to take a
> look at it.  But then both lack of time, and lack of understanding of
> mutt internals have been in the way :-p.
> > This seems to be problem in the mutt maildir handling rather than in the
> > notmuch addon, still I wonder if you saw that or even better have some
> > plans on tackling that :)
>
> I thought one possible solution might be introducing a hook to what ever
> function that does the file renaming when maildir flags are updated.
> I'm not sure how one can introduce a hook though.  Another option would
> be to add a inotify watch on the maildir currently mutt is viewing.  And
> follow-up on any inotify events.

That seems to be complicated and no matter what you do you will IMO just
decrease the likehood of such issue. You can't stop it completely, there
will always be a window when you can get two concurrent MUAs out of
sync. What I would like to try is to make mutt ignore any local changes
to the message when it has been changed underneath it. Just accept the
new state which has been force upon it.

in mh.c we have

static int maildir_sync_message (CONTEXT * ctx, int msgno)
{
  ...
  if (rename (oldpath, fullpath) != 0)
  {
    mutt_perror ("rename");
    return (-1);
  }
}

what I believe happens is that rename fails, maildir_sync_message
returns -1, but mutt still remembers that it's supposed to rename the
file so it tries again next time you update the mailbox. Instead it
should forget the rename what was supposed to be written to disk, and
move to write any other outstanding changes. Then it should re-read
whole mailbox to get in sync with real state of emails.

But that's problem of mh.c so I believe mutt discussion alias is more
suited for this.

I wanted to know whether I'm the only one struggling at this, and I'm
glad I'm not (well, sort of :) ).
> Hopefully I understood your question correctly and my comments make
> sense to you.

Yes, definitely.

Thank you

[Vladimir Marek]

It just happened again

$ truss -t rename -p 27999
rename(".../INBOX/cur/1368493840_0.26185.vi64-x3-2e-prg06,U=32846,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,",
       ".../INBOX/cur/1368493840_0.26185.vi64-x3-2e-prg06,U=32846,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S") Err#2 ENOENT

$ sudo dtrace -n 'pid$target::rename:entry{ustack()}' -p 27999
dtrace: description 'pid$target::rename:entry' matched 1 probe
 CPU     ID                    FUNCTION:NAME
  26    109                     rename:entry
              libc.so.1`rename
              mutt`maildir_sync_message+0x1b1

$ ls -l .../INBOX/*/1368493840_0.26185.vi64-x3-2e-prg06,U=32846,*
.../INBOX/cur/1368493840_0.26185.vi64-x3-2e-prg06,U=32846,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S

In this particular case it tries to rename file to exactly the same it
already is.

[Suvayu Ali]

Hey Vlad,

On Tue, May 14, 2013 at 12:08:07AM +0200, Vladimir Marek wrote:
> > > Since I'm usually using several mail clients (from phone, old mutt,
> > > mutt-kz) it happens quite often that mutt complains.
> > >
> > > rename: No such file or directory (errno = 2)
> > >
> > > and the only thing I can do about it is to kill whole mutt. What I
> > > believe is happening is that if I view unread mail, mutt remembers that
> > > it has to move the mail once it leaves current inbox. But before that,
> > > someone else touches the mail (probably moving the message from 'new'
> > > directory to 'cur'. After that mutt-kz can't find the original file and
> > > complains with the error message.
> >
> > I think this is a problem with mutt-kz not syncing the notmuch database
> > as it updates the maildir flags for the email.  I have noticed this
> > while using emacs-notmuch, notmuch cli, and mutt-kz.
> >
> > Now consider the following scenario.  Mutt sees new mail in a maildir,
> > then you read it.  Now until the next notmuch sync (say with `notmuch
> > new' from a cron job or OfflineIMAP hook) notmuch thinks the file is
> > <maildir>/new/<some_name>,N.  Whereas mutt has renamed it to
> > <maildir>/cur/<some_name>,S!  This can happen everytime mutt updates a
> > maildir flag (IOW renames a message file).  IMO, this is an inherrent
> > issue with the maildir format.  So you will see this out-of-sync
> > behaviour when mutt flags an email as "old", read, or adds the replied
> > flag.
>
> Right, that's also what I think is happening. Although when I try to
> provocate the change deliberately, I'm not successful.

Since I read and send emails primarily with mutt, I usually encounter
the problem from other clients.  Given that, I would think you can
replicate it from mutt with the following steps:

1. Open a virtual folder and do something that modifies the maildir
   flags, like reading a message.  Do not write the changes back.

2. Open the same message from another client: emacs-notmuch, alot[1],
   bower[2], notmuch-vim-ruby[3], notmuch-web[4], etc (that is some
   list!).  This time write the changes back to disk.

3. If you try to write the change back to disk from mutt now, you should
   be able trigger the error.

Actually, I just successfully trigger the error with the above recipe
with mutt and alot.  On second thought, you should be replicate it with
two different instances of mutt :-p!
> > > This seems to be problem in the mutt maildir handling rather than in the
> > > notmuch addon, still I wonder if you saw that or even better have some
> > > plans on tackling that :)
> >
> > I thought one possible solution might be introducing a hook to what ever
> > function that does the file renaming when maildir flags are updated.
> > I'm not sure how one can introduce a hook though.  Another option would
> > be to add a inotify watch on the maildir currently mutt is viewing.  And
> > follow-up on any inotify events.
>
> That seems to be complicated and no matter what you do you will IMO just
> decrease the likehood of such issue. You can't stop it completely, there
> will always be a window when you can get two concurrent MUAs out of
> sync.

This I don't agree with completely but for a very different reason :).
IMO if mutt-kz opens a virtual/real folder, modifies message flags, and
writes it to disk.  It is mutt-kz's responsibility to update the notmuch
database as well.  After all that is the claim, mutt-kz is mutt +
notmuch support!  Mutt-kz should not be breaking other tools using the
database by leaving it in an inconsistent state; specially breakage of
the notmuch cli should not be acceptable.
> What I would like to try is to make mutt ignore any local changes
> to the message when it has been changed underneath it. Just accept the
> new state which has been force upon it.

Yes, I think that is the right approach given the motivation behind
maildir design.  Quoting from the Wikipedia article on maildir:

[...] The local filesystem handles file locking as messages are added,
  moved and deleted. A major design goal of Maildir is to eliminate
  program code having to handle locking, which is often difficult.

I really appreciate that you are looking into these things :).
Hopefully my comments help.

Cheers,

Footnotes:

[1] https://github.com/pazz/alot

[2] https://github.com/wangp/bower

[3] https://github.com/felipec/notmuch-vim-ruby

[4] https://bitbucket.org/wuzzeb/notmuch-web

[Karel Zak]

On Mon, May 13, 2013 at 08:46:11PM +0200, Suvayu Ali wrote:
> I thought one possible solution might be introducing a hook to what ever
> function that does the file renaming when maildir flags are updated.
> I'm not sure how one can introduce a hook though.  Another option would
> be to add a inotify watch on the maildir currently mutt is viewing.  And
> follow-up on any inotify events.

Well, with virtual folders you don't have one maildir, there is
 nothing like "the maildir currently mutt is viewing" ;-)

In the function nm_check_database() we check for notmuch DB mtime
 changes, so we're able to detect changes. The question is if this
 function works always correctly together with nm_sync() where we
 write our (in-mutt) changes back to maildirs.

Karel

[Suvayu Ali]

Hi Karel,

On Tue, May 14, 2013 at 11:30:41AM +0200, Karel Zak wrote:
> On Mon, May 13, 2013 at 08:46:11PM +0200, Suvayu Ali wrote:
> > I thought one possible solution might be introducing a hook to what ever
> > function that does the file renaming when maildir flags are updated.
> > I'm not sure how one can introduce a hook though.  Another option would
> > be to add a inotify watch on the maildir currently mutt is viewing.  And
> > follow-up on any inotify events.
>
> Well, with virtual folders you don't have one maildir, there is
>  nothing like "the maildir currently mutt is viewing" ;-)
>
> In the function nm_check_database() we check for notmuch DB mtime
>  changes, so we're able to detect changes. The question is if this
>  function works always correctly together with nm_sync() where we
>  write our (in-mutt) changes back to maildirs.

I have noticed the problem (inconsistent database) when trying to read
messages from regular maildirs too.  So maybe your hunch about nm_sync()
and nm_check_database() not working well together is correct.  Here is
one recipe to reproduce the problem:

Assumptions: `notmuch new' is run every 5 minutes.

Now if I read an email and write the changes to disk from mutt, calling
the notmuch cli fails in the 5 minute window until the next invocation
of `notmuch new'.

$ notmuch show id:[e-mail]
Error opening <maildir>/cur/1367868776_0.31600.kuru.dyndns-at-home.com,U=45683,FMD5=3ba688d3536fd0a0303502d046ba5419:2,: No such file or directory

5 minutes later:

$ notmuch show id:[e-mail] > /dev/null && echo pass
pass

Hopefully I explained the situation adequately.

I just read your other message.  I had noticed your use of notmuch
deliver on one of the examples before, I'll look into it.

Cheers,

PS: I love this fork, and use it full time :).

[Karel Zak]

On Mon, May 13, 2013 at 10:36:07AM +0200, Vladimir Marek wrote:
> Since I'm usually using several mail clients (from phone, old mutt,
> mutt-kz) it happens quite often that mutt complains.
>
> rename: No such file or directory (errno = 2)
>
> and the only thing I can do about it is to kill whole mutt. What I
> believe is happening is that if I view unread mail, mutt remembers that
> it has to move the mail once it leaves current inbox. But before that,
> someone else touches the mail (probably moving the message from 'new'
> directory to 'cur'. After that mutt-kz can't find the original file and
> complains with the error message.

This is reason why I don't use 'notmuch new' or something like this
 from crontab. All I have is notmuch-deliver which cares about one new
 email only and does not try to sync whole DB.

It's really bad idea to modify on-disk data (maildirs) if you're not
 able to sync in-memory cached data (mutt).
> This seems to be problem in the mutt maildir handling rather than in the
> notmuch addon, still I wonder if you saw that or even better have some
> plans on tackling that :)

I see this problem very very rarely (usually when I have more one
 running mutts that modify the same virtual folder).

Maybe it would be possible to improve nm_sync() to read the message 
 path from notmuch DB before it call mh_sync_mailbox_message(). Now
 the code blindly follows cached path (from nm_header_get_fullpath()).

Karel

[Vladimir Marek]

> I see this problem very very rarely (usually when I have more one
>  running mutts that modify the same virtual folder).
>
> Maybe it would be possible to improve nm_sync() to read the message 
>  path from notmuch DB before it call mh_sync_mailbox_message(). Now
>  the code blindly follows cached path (from nm_header_get_fullpath()).

As I saw the issue quite a lot lately, I tried to create a simple
workaround.

--- a/mh.c
+++ b/mh.c
@@ -1721,8 +1721,11 @@ static int maildir_sync_message (CONTEXT * ctx, int msgno)

if (rename (oldpath, fullpath) != 0)
     {
-      mutt_perror ("rename");
-      return (-1);
+      struct stat stat_s;
+      if (errno != 2 || stat (fullpath, &stat_s) != 0) {
+       mutt_perror ("rename");
+       return (-1);
+      }
     }
     mutt_str_replace (&h->path, partpath);
   }

Even if we would read the mail file path from notmuch DB beore calling
mh_sync_mailbox_message(), we would have to check whether we still need to call
rename(2) or not. So in the situation when I know that my source file
disapeared and the destination file is there, I blindly believe that the mails
are the same. That's not the best solution out there, but I think that for me
it will work perfectly.

[Selene Scriven]

* Vladimir Marek  wrote:
> As I saw the issue quite a lot lately, I tried to create a 
> simple workaround.

Awesome.  I see this issue at least a dozen times per day.  The 
reason is because I have 'afew' moving messages around behind the 
scenes, based on tags.  So, if I go through a mailbox, archiving 
some messages and saving others to maildirs or deleting them, the 
archived ones get moved within a few minutes and then mutt won't 
let me delete the ones marked for deletion.  I have to exit and 
re-start mutt before I can delete those.  Or, to work around it, 
archive, re-load the virtual folder, delete, sync mailbox and 
wait a while, archive more, re-load, delete, sync, etc...  I just 
can't do both in one pass without problems.

-- Selene

[Vladimir Marek]

> > As I saw the issue quite a lot lately, I tried to create a 
> > simple workaround.
>
> Awesome.  I see this issue at least a dozen times per day.

Did the change help? If yes, in 100% cases or sometimes you see the
'rename' error again? I could add some more debug to the 'rename' error
message which would help us catch more issues this way.
> The reason is because I have 'afew' moving messages around behind the
> scenes, based on tags.  So, if I go through a mailbox, archiving some
> messages and saving others to maildirs or deleting them, the archived
> ones get moved within a few minutes and then mutt won't let me delete
> the ones marked for deletion.  I have to exit and re-start mutt before
> I can delete those.  Or, to work around it, archive, re-load the
> virtual folder, delete, sync mailbox and wait a while, archive more,
> re-load, delete, sync, etc...  I just can't do both in one pass
> without problems.

That does not sound like an ideal scenario indeed :)