an interface for filesystem preallocation that doesn't suck?

[James Ralston]

Now that ext4 is the default filesystem, it would be nice to see more
programs taking advantage of its preallocation features.

But here's my question: what is the best way to actually do this?

In recent glibc, posix_fallocate() will first try to use fallocate(),
and then fall back to writing nulls to fill up the requested range if
fallocate() fails.

The problem is, posix_fallocate()'s fallback behavior effectively
results in writing the file twice.  If we expect that the process of
writing the file's actual content will take a long time (e.g., I'm
using a BitTorrent client to download some Fedora DVD ISOs), then
that's not a big deal.  But if I'm using a utility like rsync or cp,
the performance penalty of writing every single file twice is
unacceptable.

Additionally, the semantics that result from using fallocate() with
FALLOC_FL_KEEP_SIZE (which posix_fallocate() does NOT do) are arguably
more intuitive, because FALLOC_FL_KEEP_SIZE (essentially) creates a
sparse file:

$ fallocate-test foo $[1024*1024*512] && ls -lsa foo
fallocate-test: allocating 536870912 bytes for new file foo
524292 -rw-r--r-- 1 root root 0 2009-04-02 13:59 foo

If I'm (e.g.) downloading a large file via a P2P network, then I can
see the progress of the download by eyeballing the file's actual size
versus its allocated size.

I see that at one point, people were kicking around patches to use
ioctl() and/or fcntl() to perform preallocation.  Do any of those
interfaces actually exist, or were they deprecated in favor of
fallocate()?

If the latter, is there an interface for fallocate() that doesn't have
quite as much FAIL as posix_fallocate()?

[Eric Sandeen]

James Ralston wrote:

>  If the latter, is there an interface for fallocate() that doesn't
> have
>  quite as much FAIL as posix_fallocate()? 
yes, plain ol' fallocate(2), I think:

       #include <linux/falloc.h>

       long fallocate(int fd, int mode, loff_t offset, loff_t len);

but you need a very recent glibc to get it:

http://sourceware.org/bugzilla/show_bug.cgi?id=7083#c1

it'd be great if we could get that into F11...

BTW, ext4, xfs, btrfs, and ocfs2 all support ->fallocate.

-Eric

[Rahul Sundaram]

Eric Sandeen wrote:
>  James Ralston wrote:
>  
> > If the latter, is there an interface for fallocate() that doesn't have
> > quite as much FAIL as posix_fallocate()?
>  
>  yes, plain ol' fallocate(2), I think:
>  
>         #include <linux/falloc.h>
>  
>         long fallocate(int fd, int mode, loff_t offset, loff_t len);
>  
>  but you need a very recent glibc to get it:
>  
>  http://sourceware.org/bugzilla/show_bug.cgi?id=7083#c1
>  
>  it'd be great if we could get that into F11...
>  
>  BTW, ext4, xfs, btrfs, and ocfs2 all support ->fallocate. 
Shouldn't we getting the new glibc and file RFE's against RPM, the
torrent clients etc?

Rahul

[Eric Sandeen]

Rahul Sundaram wrote:
>  Eric Sandeen wrote:
> > James Ralston wrote:
> >
> >> If the latter, is there an interface for fallocate() that doesn't have
> >> quite as much FAIL as posix_fallocate()?
> > yes, plain ol' fallocate(2), I think:
> >
> >        #include <linux/falloc.h>
> >
> >        long fallocate(int fd, int mode, loff_t offset, loff_t len);
> >
> > but you need a very recent glibc to get it:
> >
> > http://sourceware.org/bugzilla/show_bug.cgi?id=7083#c1
> >
> > it'd be great if we could get that into F11...
> >
> > BTW, ext4, xfs, btrfs, and ocfs2 all support ->fallocate.
>  
>  Shouldn't we getting the new glibc and file RFE's against RPM, the
>  torrent clients etc? 
Ah, just checked and it's already in our rawhide glibc, yay :)  didn't
realize that.

http://sandeen.fedorapeople.org/utilities/fallocate.c

is a simple tool to exercise it, FWIW.

One caveat:  ext4 doesn't always coalesce extents, so preallocating for,
say, bittorent won't necessarily get ext4 down to tracking a single
extent; in my testing it can still have thousands of extents tracked by
the filesystem even if they are adjacent/contiguous.  :(  I hope to fix
that.  It's still worth doing though; a thousand adjacent extents is
better than a thousand discontiguous extents.

-Eric

[Eric Sandeen]

Eric Sandeen wrote:
>  Rahul Sundaram wrote:
> > Eric Sandeen wrote:
> >> James Ralston wrote:
> >>
> >>> If the latter, is there an interface for fallocate() that doesn't have
> >>> quite as much FAIL as posix_fallocate()?
> >> yes, plain ol' fallocate(2), I think:
> >>
> >>        #include <linux/falloc.h>
> >>
> >>        long fallocate(int fd, int mode, loff_t offset, loff_t len);
> >>
> >> but you need a very recent glibc to get it:
> >>
> >> http://sourceware.org/bugzilla/show_bug.cgi?id=7083#c1
> >>
> >> it'd be great if we could get that into F11...
> >>
> >> BTW, ext4, xfs, btrfs, and ocfs2 all support ->fallocate.
> > Shouldn't we getting the new glibc and file RFE's against RPM, the
> > torrent clients etc?
>  
>  Ah, just checked and it's already in our rawhide glibc, yay :)  didn't
>  realize that.
>  
>  http://sandeen.fedorapeople.org/utilities/fallocate.c
>  
>  is a simple tool to exercise it, FWIW. 
oh and (sorry for the spam)

http://sandeen.fedorapeople.org/utilities/fiemap-test-new.c

is a tool which can help show you the resulting file layout.

It needs fiemap.h which only recently started getting packaged up in
kernel-headers.

-Eric

[James Ralston]

On 2009-04-02 at 17:00-0500 Eric Sandeen <sandeen(a)redhat.com&gt; wrote:

>  James Ralston wrote:
>  
>  > [I]s there an interface for fallocate() that doesn't have quite as
>  > much FAIL as posix_fallocate()?
>  
>  yes, plain ol' fallocate(2), I think:
>  
>         #include <linux/falloc.h>
>  
>         long fallocate(int fd, int mode, loff_t offset, loff_t len);
>  
>  but you need a very recent glibc to get it:
>  
>  http://sourceware.org/bugzilla/show_bug.cgi?id=7083#c1 
Ah, that's the ticket (no pun intended).

>  it'd be great if we could get that into F11... 
The rawhide glibc has it, but I'm not sure if it made the cut for the
beta.  I'll test later.

>  BTW, ext4, xfs, btrfs, and ocfs2 all support ->fallocate. 
Yup.

On 2009-04-03 at 03:52+0530 Rahul Sundaram <sundaram(a)fedoraproject.org&gt; wrote:

>  Shouldn't we getting the new glibc and file RFE's against
> RPM, the
>  torrent clients etc? 
I intend to start filing bugs against anything I can find that doesn't
use fallocate(), yes.  But I needed a better implementation to suggest
than posix_fallocate().

James

[Eric Sandeen]

James Ralston wrote:

>  I intend to start filing bugs against anything I can find that
> doesn't
>  use fallocate(), yes.  But I needed a better implementation to suggest
>  than posix_fallocate(). 
Cool.  Feel free to cc: me on them?

FWIW transmission already can do it though it requires some
undocumented(?) configuration.

It also gives you the choice of posix_fallocate(), or not.   It'd be
nice if you could say "fallocate if you've got it; but no
posix_fallocate, thanks"

-Eric

[Rahul Sundaram]

Eric Sandeen wrote:

>  
>  http://sandeen.fedorapeople.org/utilities/fiemap-test-new.c
>  
>  is a tool which can help show you the resulting file layout.
>  
>  It needs fiemap.h which only recently started getting packaged up in
>  kernel-headers. 
Thanks Eric. That's very helpful.

Rahul

[Richard W.M. Jones]

On Thu, Apr 02, 2009 at 05:44:07PM -0500, Eric Sandeen wrote:
>  James Ralston wrote:
>  
>  > I intend to start filing bugs against anything I can find that doesn't
>  > use fallocate(), yes.  But I needed a better implementation to suggest
>  > than posix_fallocate().
>  
>  Cool.  Feel free to cc: me on them?
>  
>  FWIW transmission already can do it though it requires some
>  undocumented(?) configuration.
>  
>  It also gives you the choice of posix_fallocate(), or not.   It'd be
>  nice if you could say "fallocate if you've got it; but no
>  posix_fallocate, thanks" 
I just wrote a new program that uses posix_fallocate to allocate large
files (of zeroes).  Should I change it to use fallocate?  What's the
recommended code snippet / autoconf configuration?

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Eric Sandeen]

Richard W.M. Jones wrote:
>  On Thu, Apr 02, 2009 at 05:44:07PM -0500, Eric Sandeen wrote:
> > James Ralston wrote:
> >
> >> I intend to start filing bugs against anything I can find that doesn't
> >> use fallocate(), yes.  But I needed a better implementation to suggest
> >> than posix_fallocate().
> > Cool.  Feel free to cc: me on them?
> >
> > FWIW transmission already can do it though it requires some
> > undocumented(?) configuration.
> >
> > It also gives you the choice of posix_fallocate(), or not.   It'd be
> > nice if you could say "fallocate if you've got it; but no
> > posix_fallocate, thanks"
>  
>  I just wrote a new program that uses posix_fallocate to allocate large
>  files (of zeroes).  Should I change it to use fallocate?  What's the
>  recommended code snippet / autoconf configuration? 
Good question, it'd be nice to have some canned recipe for this... but
TBH I'm no autoconf expert.

The downside of posix_fallocate is that it will fall back to explicit
zero-writing if the filesystem (or the kernel) doesn't support
sys_fallocate - and you won't know the difference, if you care ...

To call the syscall directly when glibc doesn't have fallocate(2) it's
something like:

error = syscall(SYS_fallocate, fd, falloc_mode, offset, length);

(modulo the vagaries of syscalls on some architectures) and for
fallocate(2) it's:

error = fallocate(fd, falloc_mode, offset, length);

but you probably meant a code snippet & autoconf magic for navigating
the possible calls available at any given time... :)

-Eric

>  Rich.
>

[Eric Sandeen]

James Ralston wrote:
>  Now that ext4 is the default filesystem, it would be nice to see
> more
>  programs taking advantage of its preallocation features. 
Just FWIW, I ran a scan of an x86_64 rawhide install to see which
executables & libraries are using these interfaces now; it's not a very
long list:

[root@host /mnt/test/F11]# ../summarize_falloc.pl usr/bin usr/sbin bin
sbin usr/lib usr/lib64
usr/bin/transmission-daemon uses fallocate
usr/bin/transmissioncli uses fallocate
usr/bin/transmission-remote uses fallocate
usr/bin/aria2c uses posix_fallocate
usr/bin/transmission uses fallocate
usr/sbin/nscd uses posix_fallocate
usr/lib64/libvirt.so.0 uses posix_fallocate
usr/lib64/libvirt.so.0.6.2 uses posix_fallocate
usr/lib64/libMonoPosixHelper.so uses posix_fallocate
usr/lib64/libvirt.so uses posix_fallocate

 249105 85.5% are scripts (shell, perl, whatever)
  42149 14.5% don't use any fallocate() family calls at all
      6  0.0% use posix_fallocate() interfaces only
      4  0.0% use fallocate() interfaces only


It might be nice to document how these apps actually use the interfaces
though, so people can take advantage of them.  For example, it's a
fairly obscure config option for transmission, IIRC.

...

>  Additionally, the semantics that result from using fallocate() with
>  FALLOC_FL_KEEP_SIZE (which posix_fallocate() does NOT do) are arguably
>  more intuitive, because FALLOC_FL_KEEP_SIZE (essentially) creates a
>  sparse file:
>  
>  $ fallocate-test foo $[1024*1024*512] && ls -lsa foo
>  fallocate-test: allocating 536870912 bytes for new file foo
>  524292 -rw-r--r-- 1 root root 0 2009-04-02 13:59 foo 
Just a nitpick; this isn't really a sparse file (depending on what your
test does) - the blocks really and truly are allocated to it, there are
no holes.  The blocks just happen to be allocated past EOF.  :)

-Eric