6:20 p.m.
So, it seems to me that our backup software situation is really a
mess.
One particular feature that I absolutely need is xattr support, because
all my machines run SELinux. My personal server also uses ACLs for a
few files.
As far as I can tell, the *only* software we ship that even supports
xattrs at all is "star". However star has the enormous drawback that it
doesn't support incremental dumps. This makes it nearly useless for me
since my server has various ISO images and the like lying around, and
re-archiving them for each backup would quickly eat space. Also, star
is written by someone who has displayed a fundamental misunderstanding
of the GPL.
Now, rsync has an experimental patch for xattr support:
http://lists.samba.org/archive/rsync/2003-June/006413.html
But we don't apply it. We'd also need an additional SELinux-specific
patch to make it use setfscreatecon to avoid a race condition.
It doesn't look to me like librsync supports xattrs either, so none of
the programs based on it like rdiff-backup will.
However, even if we had an xattr+SELinux-capable rsync, I think we'd
still need a tar-like archiver (one better than star). I did a bit of
searching on this and found "dar":
http://dar.linux.free.fr/
Has anyone used this program? It looks to be quite nice; if we can just
add the specific SELinux support I think it would be a good replacement
for star. I particularly like how its incremental backups support
deleting files too. I'm a bit unsure of whether its xattr support
implies ACL support.
Are there any other programs I missed? Particularly ones that people
out there actually use?
7:03 p.m.
Colin Walters wrote:
As far as I can tell, the *only* software we ship that even supports
xattrs at all is "star". However star has the enormous drawback ...
Wouldn't it be more sensible to extend "tar" to support xattrs?
tar file headers are 512 bytes long; "traditional" tar used only 256
of those bytes, and GNU tar added additional fields out to 345 bytes.
That leaves 167 bytes left over, and stuffing extra data in there
would be backward compatible (both older tar implementations would
simply ignore it.)
--
Jamie Zawinski
jwz(a)jwz.org http://www.jwz.org/
jwz(a)dnalounge.com http://www.dnalounge.com/
7:10 p.m.
On Sun, 2004-09-19 at 17:03 -0700, Jamie Zawinski wrote:
Colin Walters wrote:
>
> As far as I can tell, the *only* software we ship that even supports
> xattrs at all is "star". However star has the enormous drawback ...
Wouldn't it be more sensible to extend "tar" to support xattrs?
My impression was that several people have looked at modifying GNU tar
to support xattrs, and ended up running away screaming...
8:50 p.m.
Colin Walters wrote:
My impression was that several people have looked at modifying GNU tar
to support xattrs, and ended up running away screaming...
That doesn't mean it's the wrong thing to do! Deprecating tar doesn't
sound like a winning idea to me. Surely someone will make their sanity
saving throw.
--
Jamie Zawinski
jwz(a)jwz.org http://www.jwz.org/
jwz(a)dnalounge.com http://www.dnalounge.com/
9:06 p.m.
On Sun, 2004-09-19 at 18:50 -0700, Jamie Zawinski wrote:
Colin Walters wrote:
>
> My impression was that several people have looked at modifying GNU tar
> to support xattrs, and ended up running away screaming...
That doesn't mean it's the wrong thing to do! Deprecating tar doesn't
sound like a winning idea to me.
Yeah, I agree. On the other hand though some of the newer formats like
dar's have more advantages besides xattrs, like handling file deletions,
and being indexed. So we might actually want both; tar for people still
clinging to their tape drives, and something like dar for people backing
up to disk.
Surely someone will make their sanity saving throw.
Heh. Maybe, but I think we're talking about a level 50 Cruft here...
7:39 p.m.
--On Sunday, September 19, 2004 7:20 PM -0400 Colin Walters
<walters(a)redhat.com> wrote:
One particular feature that I absolutely need is xattr support,
because
all my machines run SELinux. My personal server also uses ACLs for a
few files.
Are these all features of ext3? If so, wouldn't dump do the job?
<http://dump.sourceforge.net/>
8:48 p.m.
On Sun, 2004-09-19 at 17:39 -0700, Kenneth Porter wrote:
--On Sunday, September 19, 2004 7:20 PM -0400 Colin Walters
<walters(a)redhat.com> wrote:
> One particular feature that I absolutely need is xattr support, because
> all my machines run SELinux. My personal server also uses ACLs for a
> few files.
Are these all features of ext3? If so, wouldn't dump do the job?
Well, it's not quite the same thing. You can't safely dump a live
filesystem, making it unusable for me. Also it's far less granular than
an archiver like tar or dar.
10:09 p.m.
--On Sunday, September 19, 2004 9:48 PM -0400 Colin Walters
<walters(a)redhat.com> wrote:
Well, it's not quite the same thing. You can't safely dump a
live
filesystem, making it unusable for me.
An explanation of the issue and why it not be an issue, depending on the
situation:
<http://dump.sourceforge.net/isdumpdeprecated.html>
Also it's far less granular than an archiver like tar or dar.
It sounded like you were doing filesystem-level backups (including
incrementals), not archives of isolated directory trees. Dump is very good
for that.
Do you use any sparely allocated files? How is tar at handling those?
Dump's restore will restore all the holes as holes, instead of allocating
zero-filled disk blocks.
10:17 p.m.
On Sun, 2004-09-19 at 20:09 -0700, Kenneth Porter wrote:
--On Sunday, September 19, 2004 9:48 PM -0400 Colin Walters
<walters(a)redhat.com> wrote:
> Well, it's not quite the same thing. You can't safely dump a live
> filesystem, making it unusable for me.
An explanation of the issue and why it not be an issue, depending on the
situation:
<http://dump.sourceforge.net/isdumpdeprecated.html>
> Also it's far less granular than an archiver like tar or dar.
It sounded like you were doing filesystem-level backups (including
incrementals), not archives of isolated directory trees. Dump is very good
for that.
Do you use any sparely allocated files? How is tar at handling those?
Dump's restore will restore all the holes as holes, instead of allocating
zero-filled disk blocks.
tar has a flag for handling sparse files.
-sv
10:32 p.m.
On Sun, 2004-09-19 at 20:09 -0700, Kenneth Porter wrote:
--On Sunday, September 19, 2004 9:48 PM -0400 Colin Walters
<walters(a)redhat.com> wrote:
> Well, it's not quite the same thing. You can't safely dump a live
> filesystem, making it unusable for me.
An explanation of the issue and why it not be an issue, depending on the
situation:
<http://dump.sourceforge.net/isdumpdeprecated.html>
If Linus says a program will eat my data and is generally stupid, I tend
to believe him :)
> Also it's far less granular than an archiver like tar or
dar.
It sounded like you were doing filesystem-level backups (including
incrementals), not archives of isolated directory trees. Dump is very good
for that.
Not really - I'm only interested in backing up a few things like home
directories, web sites, and possibly /etc. Right now I just have one
big partition so dump would back up e.g. all of /usr, which isn't needed
for me. Also, dump's filesystem-specific nature is very annoying.
12:14 a.m.
On Sun, 2004-09-19 at 21:32, Colin Walters wrote:
On Sun, 2004-09-19 at 20:09 -0700, Kenneth Porter wrote:
> --On Sunday, September 19, 2004 9:48 PM -0400 Colin Walters
> <walters(a)redhat.com> wrote:
>
> > Well, it's not quite the same thing. You can't safely dump a live
> > filesystem, making it unusable for me.
>
> An explanation of the issue and why it not be an issue, depending on the
> situation:
>
> <http://dump.sourceforge.net/isdumpdeprecated.html>
If Linus says a program will eat my data and is generally stupid, I tend
to believe him :)
"In the future, dump's problems will be solved by snapshots."
The future is now, eh?
> > Also it's far less granular than an archiver like tar
or dar.
>
> It sounded like you were doing filesystem-level backups (including
> incrementals), not archives of isolated directory trees. Dump is very good
> for that.
Not really - I'm only interested in backing up a few things like home
directories, web sites, and possibly /etc. Right now I just have one
big partition so dump would back up e.g. all of /usr, which isn't needed
for me.
This is no longer accurate. You can use dump to backup selected
directories. You don't have to backup the entire fs.
Dax Kelson
11:55 p.m.
On Sun, Sep 19, 2004 at 09:48:04PM -0400, Colin Walters wrote:
Well, it's not quite the same thing. You can't safely dump a
live
filesystem, making it unusable for me. Also it's far less granular than
an archiver like tar or dar.
Have you looked at making lvm snapshots and dumping those?
--
Matthew Miller mattdm(a)mattdm.org <http://www.mattdm.org/>
Boston University Linux ------> <http://linux.bu.edu/>
10:21 a.m.
On Mon, 2004-09-20 at 00:55 -0400, Matthew Miller wrote:
On Sun, Sep 19, 2004 at 09:48:04PM -0400, Colin Walters wrote:
> Well, it's not quite the same thing. You can't safely dump a live
> filesystem, making it unusable for me. Also it's far less granular than
> an archiver like tar or dar.
Have you looked at making lvm snapshots and dumping those?
I haven't, mainly because my server is FC2, and I didn't think to use
LVM when setting it up, and I've heard that snapshotting is flaky in the
FC2 version anyways.
However, since new installs default to being on LVM, this might be a
future option.
The major problem I see with it though is the granularity and
filesystem-specificity. It probably still makes sense to have a
filesystem-independent archiver.
10:24 a.m.
mattdm(a)mattdm.org (Matthew Miller) writes:
> Well, it's not quite the same thing. You can't safely
dump a live
> filesystem, making it unusable for me. Also it's far less granular
> than an archiver like tar or dar.
Have you looked at making lvm snapshots and dumping those?
LVM snapshots are not usably at the moment: for kernel 2.4 they work with
a kernel patch only, and for kernel 2.6 they are marked as experimental
which is confirmed by my experiences (kernel-panics on creation and manual
cleanup required).
Enrico
7:39 a.m.
On Tue, 21 Sep 2004 01:24, Enrico Scholz
<enrico.scholz(a)informatik.tu-chemnitz.de> wrote:
mattdm(a)mattdm.org (Matthew Miller) writes:
>> Well, it's not quite the same thing. You can't safely dump a live
>> filesystem, making it unusable for me. Also it's far less granular
>> than an archiver like tar or dar.
>
> Have you looked at making lvm snapshots and dumping those?
LVM snapshots are not usably at the moment: for kernel 2.4 they work with
a kernel patch only, and for kernel 2.6 they are marked as experimental
which is confirmed by my experiences (kernel-panics on creation and manual
cleanup required).
FC2 has kernel 2.6 but doesn't have snapshot support.
FC3 has snapshot support, I've just been using it and it works for me. Can
you file bug reports about any LVM oops you can get with a Fedora kernel?
--
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/ My home page
1:44 p.m.
russell(a)coker.com.au (Russell Coker) writes:
> LVM snapshots are not usably at the moment: for kernel 2.4 they
work with
> a kernel patch only, and for kernel 2.6 they are marked as experimental
> which is confirmed by my experiences (kernel-panics on creation and manual
> cleanup required).
FC2 has kernel 2.6 but doesn't have snapshot support.
FC3 has snapshot support, I've just been using it and it works for me. Can
you file bug reports about any LVM oops you can get with a Fedora
kernel?
Sorry; the FC3 kernel is missing some features so that I have to use the
vanilla 2.6.8.1 kernel + some patches. FWIW, the stacktrace from the
kernel oops is
| Sep 22 04:44:18 delenn kernel: lvcreate: page allocation failure. order:0, mode:0xd0
| Sep 22 04:44:18 delenn kernel: [dump_stack+30/32] dump_stack+0x1e/0x20
| Sep 22 04:44:18 delenn kernel: [<c0105eee>] dump_stack+0x1e/0x20
| Sep 22 04:44:18 delenn kernel: [__alloc_pages+739/784] __alloc_pages+0x2e3/0x310
| Sep 22 04:44:18 delenn kernel: [<c0147723>] __alloc_pages+0x2e3/0x310
| Sep 22 04:44:18 delenn kernel: [alloc_pl+52/96] alloc_pl+0x34/0x60
| Sep 22 04:44:18 delenn kernel: [<c0280914>] alloc_pl+0x34/0x60
| Sep 22 04:44:18 delenn kernel: [client_alloc_pages+37/112]
client_alloc_pages+0x25/0x70
| Sep 22 04:44:18 delenn kernel: [<c0280aa5>] client_alloc_pages+0x25/0x70
| Sep 22 04:44:18 delenn kernel: [kcopyd_client_create+101/192]
kcopyd_client_create+0x65/0xc0
| Sep 22 04:44:18 delenn kernel: [<c02816d5>] kcopyd_client_create+0x65/0xc0
| Sep 22 04:44:18 delenn kernel: [pg0+140642617/1069535232] snapshot_ctr+0x329/0x3c0
[dm_snapshot]
| Sep 22 04:44:18 delenn kernel: [<c8a21939>] snapshot_ctr+0x329/0x3c0
[dm_snapshot]
| Sep 22 04:44:18 delenn udev[2296]: PROGRAM execution of '/sbin/devmap_name 254
5' failed
| Sep 22 04:44:18 delenn udev[2297]: PROGRAM execution of '/sbin/devmap_name 254
6' failed
| Sep 22 04:44:18 delenn kernel: [dm_table_add_target+315/496]
dm_table_add_target+0x13b/0x1f0
| Sep 22 04:44:19 delenn udev[2296]: creating device node '/dev/dm-5'
| Sep 22 04:44:19 delenn udev[2297]: creating device node '/dev/dm-6'
| Sep 22 04:44:19 delenn kernel: [<c027c2db>] dm_table_add_target+0x13b/0x1f0
| Sep 22 04:44:19 delenn udev[2314]: PROGRAM execution of '/sbin/devmap_name 254
7' failed
| Sep 22 04:44:19 delenn udev[2258]: PROGRAM execution of '/sbin/devmap_name 254
4' failed
| Sep 22 04:44:19 delenn kernel: [populate_table+142/240] populate_table+0x8e/0xf0
| Sep 22 04:44:19 delenn udev[2314]: creating device node '/dev/dm-7'
| Sep 22 04:44:19 delenn udev[2258]: creating device node '/dev/dm-4'
| Sep 22 04:44:19 delenn kernel: [<c027edce>] populate_table+0x8e/0xf0
| Sep 22 04:44:19 delenn kernel: [table_load+102/336] table_load+0x66/0x150
| Sep 22 04:44:19 delenn kernel: [<c027ee96>] table_load+0x66/0x150
| Sep 22 04:44:19 delenn kernel: [ctl_ioctl+222/320] ctl_ioctl+0xde/0x140
| Sep 22 04:44:19 delenn kernel: [<c027f70e>] ctl_ioctl+0xde/0x140
| Sep 22 04:44:19 delenn kernel: [sys_ioctl+530/1088] sys_ioctl+0x212/0x440
| Sep 22 04:44:19 delenn kernel: [<c0176cd2>] sys_ioctl+0x212/0x440
| Sep 22 04:44:19 delenn kernel: [syscall_call+7/11] syscall_call+0x7/0xb
| Sep 22 04:44:19 delenn kernel: [<c010510b>] syscall_call+0x7/0xb
| Sep 22 04:44:19 delenn kernel: device-mapper: : Could not create kcopyd client
| Sep 22 04:44:19 delenn kernel:
| Sep 22 04:44:19 delenn kernel: device-mapper: error adding target to table
It happens perhaps once per week (2 snapshots per day) and is not
reproducible.
Enrico
6:14 p.m.
On Wed, 2004-09-22 at 06:39, Russell Coker wrote:
On Tue, 21 Sep 2004 01:24, Enrico Scholz
<enrico.scholz(a)informatik.tu-chemnitz.de> wrote:
> mattdm(a)mattdm.org (Matthew Miller) writes:
> >> Well, it's not quite the same thing. You can't safely dump a live
> >> filesystem, making it unusable for me. Also it's far less granular
> >> than an archiver like tar or dar.
> >
> > Have you looked at making lvm snapshots and dumping those?
>
> LVM snapshots are not usably at the moment: for kernel 2.4 they work with
> a kernel patch only, and for kernel 2.6 they are marked as experimental
> which is confirmed by my experiences (kernel-panics on creation and manual
> cleanup required).
FC2 has kernel 2.6 but doesn't have snapshot support.
Outdated info. The release kernel and the first few errata kernel didn't
have support. Update to a recent/latest errata kernel and you'll be
set.
find /lib/modules -name dm-snapshot.ko
Dax Kelson
Guru Labs
9:09 a.m.
On Mon, 20 Sep 2004 11:48, Colin Walters <walters(a)redhat.com> wrote:
> Are these all features of ext3? If so, wouldn't dump do the
job?
Well, it's not quite the same thing. You can't safely dump a live
filesystem, making it unusable for me. Also it's far less granular than
an archiver like tar or dar.
LVM snapshots will allow dump to be used on live file systems.
Dump does not work across file systems, so if you want to convert an Ext3 file
system to ReiserFS or XFS then dump will not help.
Dump is tied to Inode numbers and I believe that this has some implications
for restoring deleted files if the machine has continued operation and
created other files with the same Inode numbers.
We do need a better backup solution, a friend investigated backups some time
ago (without regard to SE Linux) and reached the conclusion that extending
"dar" offered the best promise of all the Linux backup options. So dar may
be worth considering.
--
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/ My home page
9:14 a.m.
On Mon, 2004-09-20 at 10:09, Russell Coker wrote:
On Mon, 20 Sep 2004 11:48, Colin Walters <walters(a)redhat.com>
wrote:
> > Are these all features of ext3? If so, wouldn't dump do the job?
>
> Well, it's not quite the same thing. You can't safely dump a live
> filesystem, making it unusable for me. Also it's far less granular than
> an archiver like tar or dar.
LVM snapshots will allow dump to be used on live file systems.
Dump does not work across file systems, so if you want to convert an Ext3 file
system to ReiserFS or XFS then dump will not help.
Dump is tied to Inode numbers and I believe that this has some implications
for restoring deleted files if the machine has continued operation and
created other files with the same Inode numbers.
We do need a better backup solution, a friend investigated backups some time
ago (without regard to SE Linux) and reached the conclusion that extending
"dar" offered the best promise of all the Linux backup options. So dar may
be worth considering.
Has anyone looked into bacula's ability to store extended attributes?
Does it have any?
-sv
7:22 p.m.
On Mon, 2004-09-20 at 10:14 -0400, seth vidal wrote:
Has anyone looked into bacula's ability to store extended
attributes?
Does it have any?
Wow, so I should have googled for bacula when you mentioned it here, but
Michael K. Johnson just mentioned it on IRC and I actually looked: it's
quite impressive!
Someone should totally package this for Fedora.
It claims to have ACL support, so hopefully that implies generic xattr
support. It will still require a small SELinux-specific patch though.
10:26 a.m.
On Tue, 2004-09-21 at 00:09 +1000, Russell Coker wrote:
LVM snapshots will allow dump to be used on live file systems.
Ok, LVM snapshots help a lot, although as I mentioned elsewhere I think
we probably still need an archiver.
We do need a better backup solution, a friend investigated backups
some time
ago (without regard to SE Linux) and reached the conclusion that extending
"dar" offered the best promise of all the Linux backup options. So dar may
be worth considering.
Ok. I actually investigated dar a bit more closely last night and found
that it needs to be patched to support the "security" xattr namespace.
Also we will need a patch to make it use setfscreatecon like the star
SELinux patch does.
7:58 p.m.
Replying to Colin Walters:
It doesn't look to me like librsync supports xattrs either, so
none of
the programs based on it like rdiff-backup will.
rdiff-backup only uses librsync to calculate increments of file data.
By its design stuff like xattr and acl belong to file metadata. It can
store them too but if corresponding python modules (pylibacl and
pyxattr) available
--
Paul P 'Stingray' Komkoff Jr // http://stingr.net/key <- my pgp key
This message represents the official view of the voices in my head
8:22 p.m.
On Mon, 20 Sep 2004 04:58:48 +0400, Paul P Komkoff Jr <i(a)stingr.net> wrote:
rdiff-backup only uses librsync to calculate increments of file
data.
By its design stuff like xattr and acl belong to file metadata. It can
store them too but if corresponding python modules (pylibacl and
pyxattr) available
to be precise... support for pyxattr and pylibacl are in the
development 0.13 branch of rdiff-backup and not in the stable 0.12
branch. Maybe if we all make puppy dog faces in Colin's general
direction we can guilt him into taking a look at the 0.13.4
rdiff-backup release and seeing how good the xattr support is.
-jef"watches as Colin fires a technical bullet at point blank range at
rdiff-backup and watches it bounce harmlessly off."spaleta
9:21 p.m.
On Sun, 2004-09-19 at 21:22 -0400, Jeff Spaleta wrote:
to be precise... support for pyxattr and pylibacl are in the
development 0.13 branch of rdiff-backup and not in the stable 0.12
branch. Maybe if we all make puppy dog faces in Colin's general
direction we can guilt him into taking a look at the 0.13.4
rdiff-backup release and seeing how good the xattr support is.
Testing beta backup software. My favorite thing to do :)
Seriously though, while that is good news, I think we still need a good
archiver. For example, I'd like to be able to encrypt my backups and
drop them onto untrusted media. Or even just back up to a filesystem
without xattr or SELinux support. You can't do that with rdiff-backup.
9:22 p.m.
On Sun, 2004-09-19 at 22:21 -0400, Colin Walters wrote:
On Sun, 2004-09-19 at 21:22 -0400, Jeff Spaleta wrote:
> to be precise... support for pyxattr and pylibacl are in the
> development 0.13 branch of rdiff-backup and not in the stable 0.12
> branch. Maybe if we all make puppy dog faces in Colin's general
> direction we can guilt him into taking a look at the 0.13.4
> rdiff-backup release and seeing how good the xattr support is.
Testing beta backup software. My favorite thing to do :)
Seriously though, while that is good news, I think we still need a good
archiver. For example, I'd like to be able to encrypt my backups and
drop them onto untrusted media. Or even just back up to a filesystem
without xattr or SELinux support. You can't do that with rdiff-backup.
look at duplicity.
-sv
9:34 p.m.
On Sun, 2004-09-19 at 22:22 -0400, seth vidal wrote:
look at duplicity.
duplicity uses (presumably GNU) tar, which brings us full-circle :)
There is a page where the duplicity people talk about tar's flaws
and what a new archive format might look like:
http://www.nongnu.org/duplicity/new_format.html
But it's just a design, not an implementation. dar is an
implementation, which makes it much more interesting :) But so far no
one has commented on dar. I was hoping someone out there would at least
say "yes, i use it, it rocks!" or "yes, i tried it, it sucks!"...
7206
days inactive
7210
days old
25 comments
10 participants
participants (10)
-
Colin Walters -
Dax Kelson -
Enrico Scholz -
Jamie Zawinski -
Jef Spaleta -
Kenneth Porter -
Matthew Miller -
Paul P Komkoff Jr -
Russell Coker -
seth vidal