After publicly bitching about Linux's poor backup infrastructure for the hundredth time, I decided to write a common system for making snapshots. I've written the first iteration in bash. It took one day to do most of the work, and then a few hours of testing and fixing to get things working reasonably well.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
At this point, there's enough working for other people to start looking at. Systems with ext3/4 filesystems on LVM are supported. btrfs will follow. PostgreSQL has a script to make its data consistent, but other common systems like MySQL, OpenLDAP, and 389 DS need similar support. Documentation needs to be written. A few architectural issues need to be ironed out.
If you're interested in improving the state of backups on GNU/Linux, please have a look and contact me if you want to help with code, testing, documentation, packaging, or maintaining packages in Fedora. I would very much like to see this project become a standard feature of GNU/Linux installs.
Gordon Messmer wrote:
After publicly bitching about Linux's poor backup infrastructure for the hundredth time, I decided to write a common system for making snapshots. I've written the first iteration in bash. It took one day to do most of the work, and then a few hours of testing and fixing to get things working reasonably well.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
At this point, there's enough working for other people to start looking at. Systems with ext3/4 filesystems on LVM are supported. btrfs will follow. PostgreSQL has a script to make its data consistent, but other common systems like MySQL, OpenLDAP, and 389 DS need similar support. Documentation needs to be written. A few architectural issues need to be ironed out.
If you're interested in improving the state of backups on GNU/Linux, please have a look and contact me if you want to help with code, testing, documentation, packaging, or maintaining packages in Fedora. I would very much like to see this project become a standard feature of GNU/Linux installs.
check out obnam. I'm very pleased with it.
On 03/19/2013 04:22 AM, Neal Becker wrote:
check out obnam. I'm very pleased with it.
That's not at all the same thing.
You cannot back up files in active use safely. If you have a SQL server, for instance, you need to either shut it down or quiesce the data files for the entire duration of the backup. On production systems, that creates an outage that's unacceptable.
While GNU/Linux systems support making a snapshot of their filesystems in order to reduce the duration of any outage, there is no standard infrastructure to do so, and no integration with the services that need to make their data consistent for backup. That is what I'm hoping to provide.
By including a standard snapshot infrastructure, you can back up a system safely and servers only need to quiesce their data for a moment, while the snapshot is made, rather than the full duration of the backup.
On 03/19/2013 02:29 AM, Gordon Messmer wrote:
After publicly bitching about Linux's poor backup infrastructure for the hundredth time, I decided to write a common system for making snapshots.
Hi,
This is interesting since we have nothing like the VSS infrastructure on Windows.
My experience is with snapshots at the hypervisor level (VMware vSphere) where I've been doing the quiescing you mention by means of the following 2 scripts that the "VMware Tools"(inside the VM) would call (when you request a snapshot):
/usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script
It's just a matter of putting the necessary stuff in those scripts and you're done. BTW, I always thought it would be great if there were some standard directory where every app, when it gets installed, would put its pre-freeze & post-thaw stuff. (Just like Apache puts its logrotate config file in /etc/logrotate.d/ when you install it). That way, whenever you perform a snapshot (no matter the kind of snapshot), all files within those directories would be executed (prior & post snaps).
The thing is to separate the snapshotting & the quiescing activities so that the quiescing works regardless of the snapshot technology being used. Is this more or less what you're doing?
-- Jorge
On 03/19/2013 05:13 PM, Jorge Fábregas wrote:
The thing is to separate the snapshotting & the quiescing activities so that the quiescing works regardless of the snapshot technology being used. Is this more or less what you're doing?
Yes! It might be as simple as:
for x in /usr/lib/snapshot/writers.d/* do test -x "$x" || continue "$x" quiesce done
... in the pre-freeze script, and the same loop with "resume" as an argument in the post-thaw script. I'd also like to see services that need to quiesce their data ship with a script to handle that.
-
Gordon Messmer -
Jorge Fábregas -
Neal Becker