F14: Encfs/procmail problem setting st_atime unexpectedly

[Corinna Vinschen]

Hi,


I'm running into a strange problem using procmail and mutt, with
some of my MBOX mailboxes stored on an encfs encrypted directory.
This is not the known problem with hardlinks, rather it's a problem
with st_atime.

My default mailbox folder is on an unencrypted filesystem (ext4).  Some
mailboxes with more confidential stuff I'm storing in an encfs mounted
directory on the same ext4 FS.  The FS mount options are default, which
includes the relatime option.

When I set my mail delivery up I found that mutt didn't notice when an
encrypted MBOX file received a new message via procmail.  A bit of
experimenting showed that the setting "check_mbox_size=yes" rectified
this problem.  However, "check_mbox_size=yes" is kind of a hack with
annoying side-effects, so that's not really a solution.

Eventually I found the following strange behaviour.

- When I append data manually to a file on the encfs, only st_mtime and
  st_ctime changes, as expected.

- When procmail appends new messages to an unencrypted mailbox, only
  st_mtime and st_ctime changes, as expected.

- However, when procmail appends a message to an encfs encrypted MBOX,
  it doesn't only change st_mtime and st_ctime, but also st_atime!

Naturally this breaks mutt's algorithm to detect mailboxes with new
mail.  For the time being I tweaked mutt to recognize mailboxes with new
mails also if st_atime == st_mtime, but that's almost just a bad hack as
using check_mbox_size=yes.

So the real question is, why does st_atime change here at all?  I tried
the debug option for procmail, but the debug output didn't contain
anything helpful. 

Does anybody have a hunch what happens here?


Thank in advance,
Corinna

[Corinna Vinschen]

On Dec 29 17:14, JB wrote:
>  Corinna Vinschen <vinschen <at> redhat.com> writes:
>  
>  > ...
>  
>  Please give us the display:
>  $ cat /proc/mounts 
>  so we can see all FS's mount options.
>  JB 
Sure.  I just left out the unrelated virtual FSes (proc, sysfs, etc):

/dev/sdd3 / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/sdd1 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
gvfs-fuse-daemon /home/corinna/.gvfs fuse.gvfs-fuse-daemon
rw,nosuid,nodev,relatime,user_id=500,group_id=11125 0 0
encfs /home/corinna/Encrypted fuse.encfs
rw,nosuid,nodev,relatime,user_id=500,group_id=11125,default_permissions 0 0


Corinna

[JB]

Corinna Vinschen <vinschen <at> redhat.com> writes:

>  ... 
For mailx a Bugzilla report 666403 has been filed.

JB

[Corinna Vinschen]

On Dec 30 10:44, JB wrote:
>  Hi,
>  
>  I can confirm similar test results for mailx.
>  I have tested it on Fedora 14.
>  [...]
>  Conclusion:
>  On Linux, with file systems mounted with default mount option 'relatime',
>  on an encrypted file system (encfs) mailx deviates from conventional rules of
>  file times modifications.
>  This will cause some applications (like mutt mail client) that rely on mbox's
>  file times attributes to not work properly. 
Interesting to see that not only procmail is affected.

>  Corinna,
>  you have Red Hat's e-mail address, but I do not know if you are a dev or
>  a user. 
As far as Fedora is concerned I'm just a user.

>  We have an option to file Fedora Bugzilla bugs for procmail and mailx
> and let
>  the package maintainers take care of it. 
Today I built procmail from source to have a look.  There's a
Fedora-specific patch procmail-3.22-rhconfig.patch, which contains this
hunk:

  @@ -91,14 +92,14 @@
	  restriction does not apply to the /etc/procmailrc and
	  /etc/procmailrcs files) */

  -#define NO_NFS_ATIME_HACK    /* uncomment if you're definitely not using
  +/*#define NO_NFS_ATIME_HACK    /* uncomment if you're definitely not using
				     NFS mounted filesystems and can't afford
	  procmail to sleep for 1 sec. before writing to an empty regular
	  mailbox.  This lets programs correctly judge whether there is unread
	  mail present.  procmail automatically suppresses this when it isn't
	  needed or under heavy load. */
  [...]

So what this patch does is to disable the NFS hack for no apparent
reason.  This also breaks MBOX files on encfs.  I built my own procmail
with the NFS atime hack re-enabled and it fixes my problem.

>  What is your suggestion ? 
As for procmail, I generated a bugzilla entry
https://bugzilla.redhat.com/show_bug.cgi?id=666383

As for mailx, it doesn't have any Fedora-specific patch which might
contribute to this problem so I'm not sure.  This may be worth to be
reported upstream, I guess.


Corinna

[JB]

Corinna Vinschen <vinschen <at> redhat.com> writes:

>  ... 
Hi,

I can confirm similar test results for mailx.
I have tested it on Fedora 14.

Heirloom Mail version 12.5 7/5/10.
$ yum list installed mailx
mailx.i686                         12.5-1.fc14                          @updates

$ yum list installed fuse-encfs
fuse-encfs.i686                       1.7.2-1.fc14                       @fedora

The objective is to test time attributes of non-encrypted (ext4 fs) and
encfs-encrypted files depending on who created/acted upon them.

Non-encrypted (ext4 fs) files test.
-----------------------------------

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda7             14420896   6992248   7282168  49% /
tmpfs                  1025992       428   1025564   1% /dev/shm

$ cat /proc/mounts
...
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda7 / ext4 rw,relatime,barrier=1,data=ordered 0 0

Note: relatime - default mount option

/home/jb/file1.txt       - file acted upon by myself
/home/jb/my-mbox         - file acted upon by mailx's 'save' command 

- I created/acted upon a file file1.txt

  Initial file creation:
  $ touch file1.txt
  $ ls -lc file1.txt 
  -rw-rw-r-- 1 jb jb 0 Dec 30 10:31 file1.txt
  $ ls -l file1.txt 
  -rw-rw-r-- 1 jb jb 0 Dec 30 10:31 file1.txt
  $ ls -lu file1.txt 
  -rw-rw-r-- 1 jb jb 0 Dec 30 10:31 file1.txt

  Now file is appended with text:
  $ echo "append" >> file1.txt 
  $ ls -lc file1.txt 
  -rw-rw-r-- 1 jb jb 7 Dec 30 10:34 file1.txt
  $ ls -l file1.txt 
  -rw-rw-r-- 1 jb jb 7 Dec 30 10:34 file1.txt
  $ ls -lu file1.txt 
  -rw-rw-r-- 1 jb jb 7 Dec 30 10:31 file1.txt

  Result: only ctime and mtime changed - as expected.

- mailx's 'save n /home/jb/my-mbox' created/acted upon a file my-mbox

  Initial my-mbox file creation by mail client's save command:
  $ ls -lc my-mbox 
  -rw-rw-r-- 1 jb jb 773 Dec 30 10:39 my-mbox
  $ ls -l my-mbox 
  -rw-rw-r-- 1 jb jb 773 Dec 30 10:39 my-mbox
  $ ls -lu my-mbox 
  -rw-rw-r-- 1 jb jb 773 Dec 30 10:39 my-mbox

  After my-mbox appended with mail message by mail client's save command:
  $ ls -lc my-mbox 
  -rw-rw-r-- 1 jb jb 1556 Dec 30 10:43 my-mbox
  $ ls -l my-mbox 
  -rw-rw-r-- 1 jb jb 1556 Dec 30 10:43 my-mbox
  $ ls -lu my-mbox 
  -rw-rw-r-- 1 jb jb 1556 Dec 30 10:39 my-mbox

  Result: only ctime and mtime changed - as expected.

encfs-encrypted files test.
---------------------------

$ encfs ~/.crypt-raw ~/mbox

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda7             14420896   6992248   7282168  49% /
tmpfs                  1025992       428   1025564   1% /dev/shm
encfs                 14420896   6992248   7282168  49% /home/jb/mbox

$ cat /proc/mounts
...
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda7 / ext4 rw,relatime,barrier=1,data=ordered 0 0
encfs /home/jb/mbox fuse.encfs
rw,nosuid,nodev,relatime,user_id=500,group_id=500,default_permissions 0 0

Note: relatime - default mount option

/home/jb/mbox/             - mountpoint dir
/home/jb/mbox/test1.txt    - file acted upon by myself
/home/jb/mbox/mbox         - file acted upon by mailx's 'save' command 

- I created/acted upon a file test1.txt

  Initial file creation:
  $ touch mbox/test1.txt
  $ ls -lc mbox/test1.txt 
  -rw-rw-r-- 1 jb jb 0 Dec 30 11:00 mbox/test1.txt
  $ ls -l mbox/test1.txt 
  -rw-rw-r-- 1 jb jb 0 Dec 30 11:00 mbox/test1.txt
  $ ls -lu mbox/test1.txt 
  -rw-rw-r-- 1 jb jb 0 Dec 30 11:00 mbox/test1.txt

  Now file is appended with text:
  $ echo "append" >> mbox/test1.txt 
  $ ls -lc mbox/test1.txt 
  -rw-rw-r-- 1 jb jb 7 Dec 30 11:02 mbox/test1.txt
  $ ls -l mbox/test1.txt 
  -rw-rw-r-- 1 jb jb 7 Dec 30 11:02 mbox/test1.txt
  $ ls -lu mbox/test1.txt 
  -rw-rw-r-- 1 jb jb 7 Dec 30 11:00 mbox/test1.txt

  $ ls -al .crypt-raw/
  total 24
  drwx------   2 jb jb  4096 Dec 30 11:00 .
  drwx------. 56 jb jb 12288 Dec 30 11:03 ..
  -rw-rw-r--   1 jb jb  1076 Dec 29 21:58 .encfs6.xml
  -rw-rw-r--   1 jb jb    15 Dec 30 11:02 NN-MKpwsiu50nui4b9XByy5C

  Result: only ctime and mtime changed - as expected.

- mailx's 'save n /home/jb/mbox/mbox' created/acted upon a file mbox

  Initial mbox file creation by mail client's save command:
  $ ls -lc mbox/mbox 
  -rw-rw-r-- 1 jb jb 773 Dec 30 11:07 mbox/mbox
  $ ls -l mbox/mbox 
  -rw-rw-r-- 1 jb jb 773 Dec 30 11:07 mbox/mbox
  $ ls -lu mbox/mbox 
  -rw-rw-r-- 1 jb jb 773 Dec 30 11:07 mbox/mbox

  After mbox appended with mail message by mail client's save command:
  $ ls -lc mbox/mbox 
  -rw-rw-r-- 1 jb jb 1546 Dec 30 11:11 mbox/mbox
  $ ls -l mbox/mbox 
  -rw-rw-r-- 1 jb jb 1546 Dec 30 11:11 mbox/mbox
  $ ls -lu mbox/mbox 
  -rw-rw-r-- 1 jb jb 1546 Dec 30 11:11 mbox/mbox

  $ ls -al .crypt-raw/
  total 28
  drwx------   2 jb jb  4096 Dec 30 11:07 .
  drwx------. 56 jb jb 12288 Dec 30 11:12 ..
  -rw-rw-r--   1 jb jb  1076 Dec 29 21:58 .encfs6.xml
  -rw-rw-r--   1 jb jb  1554 Dec 30 11:11 KvMcK-a1xgrvB36y5ebYEv7q
  -rw-rw-r--   1 jb jb    15 Dec 30 11:02 NN-MKpwsiu50nui4b9XByy5C

  Result: ctime, mtime, and atime changed - NOT as expected.

Conclusion:
On Linux, with file systems mounted with default mount option 'relatime',
on an encrypted file system (encfs) mailx deviates from conventional rules of
file times modifications.
This will cause some applications (like mutt mail client) that rely on mbox's
file times attributes to not work properly.

Corinna,
you have Red Hat's e-mail address, but I do not know if you are a dev or
a user.
We have an option to file Fedora Bugzilla bugs for procmail and mailx and let
the package maintainers take care of it.
What is your suggestion ?
 
JB

Pavarotti - Nessun Dorma 
http://www.youtube.com/watch?v=O0Sx5lbVlQA

[JB]

Corinna Vinschen <vinschen <at> redhat.com> writes:

>  ... 
Please give us the display:
$ cat /proc/mounts 
so we can see all FS's mount options.
JB