Hi,
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
Something as easy of use as the snapshot feature in VirtualBox would be great.. But near would do as well.
Kind regards,
On 08/07/2015 12:56 PM, Diogene Laerce wrote:
Hi,
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
Something as easy of use as the snapshot feature in VirtualBox would be great.. But near would do as well.
http://snapper.io/ http://rpm-software-management.github.io/dnf-plugins-extras/snapper.html
On Fri, Aug 7, 2015 at 1:56 PM, Diogene Laerce me_buss777@yahoo.fr wrote:
Hi,
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
Something as easy of use as the snapshot feature in VirtualBox would be great.. But near would do as well.
Well, VirtualBox snapshots have an equivalent in virt-manager that are as easy to use. But if you're talking about bare metal, no there is nothing even remotely approaching that level of ease for rollbacks yet.
The closest it comes is choosing Btrfs for installation (you can use something else for /home if you want). The root subvolume on Btrfs can then be snapshot before you do a dnf update, update the snapshot's copy of fstab, and if things go bad you can change the rootflags=subvol=<subvolname> boot parameter to that of the snapshot name. But this assumes some Btrfs knowledge, which at least is not nearly as esoteric and complicated like the rabbit hole that is LVM thin volume snapshots. But that's also an option if you're at least semi-comfortable with LVM.
Another option, which is still maturing and really intended now only as a platform for deploying containers, is Fedora Atomic (rpm-ostree). This is really where rollbacks are at, because updates are atomic, and it deals with all the gory bootloader details. The tea leaves suggests this is the direction for a future Workstation product that can do what you describe.
Not a snapshot, but there are various programs that can do a bare image of the system and it works with Windows and LInux. They can be done at the partition level or the entire disk.
I am the current maintainer of the G4L project, and there is also GNU and Clonezilla that can do similar things.
With my classroom lab that has systems with 500G disks with windows 7 and Fedora. I have a 160G W7 partition, and make an image of it to another partition that is about 24G in size. Takes about 12 minutes to make image, and about 10 minutes to restore. Have an option on the grub menu that can automatically, restore it, so if students mess up windows, it can be quickly restored to the previous image. Use NTFSCLONE option for the windows.
Similar process can be done with Linux, but since it has multiple partitions, one needs to do an image of each one, or one can do a full disk image, but it has to be made to another device like external disk or ftp server.
Another recommendation, unlike NTFSCLONE, which only backs up used data, the raw method will be much more effictive if the unused space on each partition is cleared (Nulls written to sectors). Program has options to do this, and then make images of each partition or the whole disk.Does take time since it has to read every sector, but image is much smaller with compression.
I have gotten even better speeds by using USB3 128G flash. Using the USB 3 flash, the same windows partition can be reimaged in about 4 1/2 minutes using USB 3 port. Takes about 8 minutes in a USB 2 port. Single hard disk takes longer, since it has to read and write from same device. The time to create the image is about the same, since the compression process seems to be the bottle neck there.
I generally always, make images of critical machines, and home machine, so that if something goes wrong, I can quickly get a machine back and running to a known state. One could just backup the /boot, and / and maybe /home partitions depending on setup, and restore them.
The G4L also, has a program calles fsarchiver that is a filelevel backup program that works with Linux, but I included it as a request of a user, and have time limited testing, in which it worked fine, but prefer the bare metal options.
So, not sure if that is the solution you are looking for.
On 7 Aug 2015 at 21:56, Diogene Laerce wrote:
To: users@lists.fedoraproject.org From: Diogene Laerce me_buss777@yahoo.fr Subject: backup snapshot Date sent: Fri, 7 Aug 2015 21:56:09 +0200 Send reply to: me_buss777@yahoo.fr, Community support for Fedora users users@lists.fedoraproject.org
Hi,
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
Something as easy of use as the snapshot feature in VirtualBox would be great.. But near would do as well.
Kind regards,
-- “One original thought is worth a thousand mindless quotings.” “Le vrai n'est pas plus sûr que le probable.”
Diogene Laerce
+----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes@kuentos.guam.net mailto:msetzerii@gmail.com http://www.guam.net/home/mikes Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ +----------------------------------------------------------+
http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489)
BOINC@HOME CREDITS ROSETTA 32682316.626719 | SETI 58701396.531536 ABC 16613838.513356 | EINSTEIN 67783109.164895
Hi,
Thanks for your answers.
Le 07/08/2015 22:12, Michael Morgan a écrit :
Btrfs root, snapper, and python-dnf-plugins-extras-snapper have been a > godsend for me in this regard. I also run snapper-timeline on my
/home > subvol which has saved me from frustration a few times already. btrfs sounds good but I will need to wait for the next installation from scratch as I run on ext4, on multiple partitions /boot (ext2), /etc, /opt, etc.. And conversion from ext4 does not seem worth the risk, for the moment.
http://snapper.io/ http://rpm-software-management.github.io/dnf-plugins-extras/snapper.html
The documentation says that snapper supports ext4 but only experimentally, do you have any feedback on the matter ?
Le 08/08/2015 02:52, Chris Murphy a écrit :
The closest it comes is choosing Btrfs for installation (you can use something else for /home if you want). The root subvolume on Btrfs can then be snapshot before you do a dnf update, update the snapshot's copy of fstab, and if things go bad you can change the rootflags=subvol=<subvolname> boot parameter to that of the snapshot name. But this assumes some Btrfs knowledge, which at least is not nearly as esoteric and complicated like the rabbit hole that is LVM thin volume snapshots. But that's also an option if you're at least semi-comfortable with LVM.
I even saw there :
https://wiki.archlinux.org/index.php/Btrfs_-_Tips_and_tricks#Automatic_snaps...
that one can configure automatic snapshots at boot which is not far from what you propose.
I guess this is reproducible on Fedora and that would definitely be my preference.
Another option, which is still maturing and really intended now only as a platform for deploying containers, is Fedora Atomic (rpm-ostree). This is really where rollbacks are at, because updates are atomic, and it deals with all the gory bootloader details. The tea leaves suggests this is the direction for a future Workstation product that can do what you describe.
That, seems to be way over my head and what I want to achieve as I just have one computer to manage.
Le 08/08/2015 04:58, Michael D. Setzer II a écrit :
Not a snapshot, but there are various programs that can do a bare image of > the system and it works with Windows and Linux. They can be done at
the > partition level or the entire disk. > > I am the current maintainer of the G4L project, and there is also GNU and > Clonezilla that can do similar things. I actually went for redobackup : http://redobackup.org/ It has a web access as it runs on ubuntu, that's what convince me. So I will go for this for now, and I will update my system to btrfs next time I install. Le 08/08/2015 15:57, Heinz Diehl a écrit :
rsync -avxHSAX /source/ /target
Heinz.. Try to catch up please.. Sorry ! I couldn't resist.. :)
Seriously, I'm not sure that rsync, even executed with root privileges, will have access to all files and will be able to copy the entire /dev/sd[n] identically. Will it ?
Thanks again to all ! ;)
On Sun, Aug 9, 2015 at 1:43 PM, Diogene Laerce me_buss777@yahoo.fr wrote:
http://snapper.io/ http://rpm-software-management.github.io/dnf-plugins-extras/snapper.html
The documentation says that snapper supports ext4 but only experimentally, do you have any feedback on the matter ?
It requires LVM thinly provisioned volumes. Proper configuration for snapshots is necessary and I don't know whether the defaults from the installer do that.
Le 08/08/2015 02:52, Chris Murphy a écrit :
The closest it comes is choosing Btrfs for installation (you can use something else for /home if you want). The root subvolume on Btrfs can then be snapshot before you do a dnf update, update the snapshot's copy of fstab, and if things go bad you can change the rootflags=subvol=<subvolname> boot parameter to that of the snapshot name. But this assumes some Btrfs knowledge, which at least is not nearly as esoteric and complicated like the rabbit hole that is LVM thin volume snapshots. But that's also an option if you're at least semi-comfortable with LVM.
I even saw there :
https://wiki.archlinux.org/index.php/Btrfs_-_Tips_and_tricks#Automatic_snaps...
that one can configure automatic snapshots at boot which is not far from what you propose.
I guess this is reproducible on Fedora and that would definitely be my preference.
Seems like it's adaptable for Fedora.
So long as the top level of the file system is not mounted (where all subvolumes and snapshots are located), it's OK to do a conventional backup of the whole system. But if the top level is mounted anywhere not excluded from that backup, the backup will see each snapshot as a completely separate instance of all of that data. So if you have root, and 5 snapshots, a backup will see six root fs's that need to be backed up unless you exclude those snapshots somehow (by not making them available in the mounted hierarchy is easiest).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Am 08.08.2015 um 15:57 schrieb Heinz Diehl:
On 07.08.2015, Diogene Laerce wrote:
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
rsync -avxHSAX /source/ /target
What Heinz Diel is saying.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Am 11.08.2015 um 03:25 schrieb Erik Grun:
Am 08.08.2015 um 15:57 schrieb Heinz Diehl:
On 07.08.2015, Diogene Laerce wrote:
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
rsync -avxHSAX /source/ /target
What Heinz Diel is saying.
What I forgot to add: Have a look at this page
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Le 11/08/2015 03:31, Erik Grun a écrit :
Am 11.08.2015 um 03:25 schrieb Erik Grun:
Am 08.08.2015 um 15:57 schrieb Heinz Diehl:
On 07.08.2015, Diogene Laerce wrote:
After a sad experience with a system update, I would like to ask if there is a software on Fedora or more generally on Linux which would allow me to make a complete snapshot of the system ?
rsync -avxHSAX /source/ /target
What Heinz Diel is saying.
What I forgot to add: Have a look at this page
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
Thank you,
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
On 08/11/2015 09:18 AM, Chris Murphy wrote:
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
I agree with Chris. Snapshots are ways of restoring data that has been perhaps corrupted or deleted or going back in time to some earlier point in the filesystems' life. They aren't backups.
Backup philosophies and techniques vary depending on what you need for your unique situation. I'm not claiming what I do is the best, but this is what works for me:
On the last Friday of a month, I plug in a big ESATA or USB3 drive and use it to store the output of Mondo Rescue. I have the backup in the form of DVD-sized ISO images I can burn DVDs from and there is a recovery DVD you can boot from. That gives me a backup usable to restore to bare metal.
Once a week (or more often if there's been significant changes), I use a _different_ ESATA or USB3 drive and run an rsync that backs up everything except a few things (/proc, /sys, /dev, /media, /var/log/journal, various caches, etc.) to a directory on that external drive based on the hostname and date I ran the backup. That permits me to restore data that's a bit more recent than the MondoRescue stuff.
I'd be happy to share the MondoArchive and rsync scripts if you wish. Tweak to suit your needs. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - Whoever said "Money can't buy happiness" obviously never had any - - money! - ----------------------------------------------------------------------
Le 11/08/2015 18:18, Chris Murphy a écrit :
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
Actually, until now, I didn't get the subtle difference between those.
As they don't serve the same purpose : I will use snapshots with bfrts for the Fedora update system issue, and backups for more general system failure. Kind of the same conclusion I found here :
http://www.esg-global.com/blogs/snapshots-vs-backups-a-great-debate-no-longe...
Thanks again to all of you, kind regards,
Hi Rick,
Le 11/08/2015 19:18, Rick Stevens a écrit :
On 08/11/2015 09:18 AM, Chris Murphy wrote:
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
I agree with Chris. Snapshots are ways of restoring data that has been perhaps corrupted or deleted or going back in time to some earlier point in the filesystems' life. They aren't backups.
Backup philosophies and techniques vary depending on what you need for your unique situation. I'm not claiming what I do is the best, but this is what works for me:
On the last Friday of a month, I plug in a big ESATA or USB3 drive and use it to store the output of Mondo Rescue. I have the backup in the form of DVD-sized ISO images I can burn DVDs from and there is a recovery DVD you can boot from. That gives me a backup usable to restore to bare metal.
Once a week (or more often if there's been significant changes), I use a _different_ ESATA or USB3 drive and run an rsync that backs up everything except a few things (/proc, /sys, /dev, /media, /var/log/journal, various caches, etc.) to a directory on that external drive based on the hostname and date I ran the backup. That permits me to restore data that's a bit more recent than the MondoRescue stuff.
I'd be happy to share the MondoArchive and rsync scripts if you wish. Tweak to suit your needs.
Thank you for the offer.
I'm going to stick with RedoBackup but I'd really like to have a look at your rsync scripts, and maybe at what various caches you think of.
Kind regards,
Hi, this stupid argument rears its head again - snapshots vs "backups". At least now someone advocating time-consuming "backups" has come up with a logical argument - You "usually" put the system into a static state before performing a "backup". Trouble is - the "backup" usually means your system is down (e.g. in single user mode) for as long as it takes to perform the "backup". A snapshot on the other hand can be validly performed (on a mirrorred file- system by shutting apps down (going to single user mode/rebooting) detaching a mirror, bringing your system back online immediately (resyncing to a new mirror) then (if you so wish!) performing a normal "backup" of the detached mirror - or (if you simply don't run database apps or such like) simply stashng the detached mirror away as todays "backup". Expect lots of flack from keen "backup"! fans here.
KInd Regards Andy
On Tuesday 11 August 2015 19:52:02 Diogene Laerce wrote:
Hi Rick,
Le 11/08/2015 19:18, Rick Stevens a écrit :
On 08/11/2015 09:18 AM, Chris Murphy wrote:
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr
wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
I agree with Chris. Snapshots are ways of restoring data that has been perhaps corrupted or deleted or going back in time to some earlier point in the filesystems' life. They aren't backups.
Backup philosophies and techniques vary depending on what you need for your unique situation. I'm not claiming what I do is the best, but this is what works for me:
On the last Friday of a month, I plug in a big ESATA or USB3 drive and use it to store the output of Mondo Rescue. I have the backup in the form of DVD-sized ISO images I can burn DVDs from and there is a recovery DVD you can boot from. That gives me a backup usable to restore to bare metal.
Once a week (or more often if there's been significant changes), I use a _different_ ESATA or USB3 drive and run an rsync that backs up everything except a few things (/proc, /sys, /dev, /media, /var/log/journal, various caches, etc.) to a directory on that external drive based on the hostname and date I ran the backup. That permits me to restore data that's a bit more recent than the MondoRescue stuff.
I'd be happy to share the MondoArchive and rsync scripts if you wish. Tweak to suit your needs.
Thank you for the offer.
I'm going to stick with RedoBackup but I'd really like to have a look at your rsync scripts, and maybe at what various caches you think of.
Kind regards,
On 11.08.2015, Diogene Laerce wrote:
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ?
If you backup all your partitions with rsync, all you have to do in an emergency case is to boot from e.g. CD or a memory stick [1] and reverse the rsync command.
Example:
rsync -avxHSAX /home/ /backup/home --> backup rsync -avxHSAX --delete /backup/home/ /home --> restore
The "--delete" parameter will take care of all the files that are not in the same state as when they were saved.
So in case of a complete disaster, just restore your data as described. If your boot sector is damaged, you'll however have to restore it by hand (which isn't all too difficult).
The other context for snapshots are system rollbacks, which is on a sliding continuum between stateless vs stateful systems. So you can get certain aspects of statelessness with snapshots, with an otherwise stateful system. This is how Windows has done updates for a long time now, and snapper, and Fedora Atomic (rpm-ostree) work. The underlying technical details of how the snapshot is achieved are dissimilar, but the basic idea is the same which is you have multiple trees and can revert to previous states.
So maybe it's better to call these rollbacks in terms of "user selectable stateful states" haha. Whereas statelessness is like a system reset: such as what we find on mobile devices, and since Windows 8.
Is restoring an rsync backup to a currently running system a rollback? It's not atomic, and unless you first backup the current state you can't then do a rollforward after you've done the rollback because you've overwritten the current state with the backup. And since the overwrite happens with in-use files, it's not atomic. Any mistakes and it can easily implode the system in a way that you can't go forward or backward to get to a bootable system and you're in diagnose and repair mode.
NTFS shadow copy, snapper+Btrfs (or LVM thinp), and rpm-ostree are all atomic rollbacks. I think it can be argued that rollbacks imply the expectation of atomicity. Otherwise you'd just say "restoring from backup" or "doing a system restore/rebuild from backup".
On 08/11/2015 01:27 PM, Chris Murphy wrote:
The other context for snapshots are system rollbacks, which is on a sliding continuum between stateless vs stateful systems. So you can get certain aspects of statelessness with snapshots, with an otherwise stateful system. This is how Windows has done updates for a long time now, and snapper, and Fedora Atomic (rpm-ostree) work. The underlying technical details of how the snapshot is achieved are dissimilar, but the basic idea is the same which is you have multiple trees and can revert to previous states.
So maybe it's better to call these rollbacks in terms of "user selectable stateful states" haha. Whereas statelessness is like a system reset: such as what we find on mobile devices, and since Windows 8.
Is restoring an rsync backup to a currently running system a rollback? It's not atomic, and unless you first backup the current state you can't then do a rollforward after you've done the rollback because you've overwritten the current state with the backup. And since the overwrite happens with in-use files, it's not atomic. Any mistakes and it can easily implode the system in a way that you can't go forward or backward to get to a bootable system and you're in diagnose and repair mode.
NTFS shadow copy, snapper+Btrfs (or LVM thinp), and rpm-ostree are all atomic rollbacks. I think it can be argued that rollbacks imply the expectation of atomicity. Otherwise you'd just say "restoring from backup" or "doing a system restore/rebuild from backup".
Thanx Chris for this elucidation. However, it is still not clear how one can do an atomic rollback when the currently booted system and perhaps logged in user has (have) already modified some files. How does one proceed then? Save the files modified since last backup or snapshot, do the restoration/ rollback, and then go through the tedious process of comparing the just saved files against the ones brought by rollback? How does one proceed in such a situation?
On Tue, Aug 11, 2015 at 1:56 PM, jd1008 jd1008@gmail.com wrote:
On 08/11/2015 01:27 PM, Chris Murphy wrote:
The other context for snapshots are system rollbacks, which is on a sliding continuum between stateless vs stateful systems. So you can get certain aspects of statelessness with snapshots, with an otherwise stateful system. This is how Windows has done updates for a long time now, and snapper, and Fedora Atomic (rpm-ostree) work. The underlying technical details of how the snapshot is achieved are dissimilar, but the basic idea is the same which is you have multiple trees and can revert to previous states.
So maybe it's better to call these rollbacks in terms of "user selectable stateful states" haha. Whereas statelessness is like a system reset: such as what we find on mobile devices, and since Windows 8.
Is restoring an rsync backup to a currently running system a rollback? It's not atomic, and unless you first backup the current state you can't then do a rollforward after you've done the rollback because you've overwritten the current state with the backup. And since the overwrite happens with in-use files, it's not atomic. Any mistakes and it can easily implode the system in a way that you can't go forward or backward to get to a bootable system and you're in diagnose and repair mode.
NTFS shadow copy, snapper+Btrfs (or LVM thinp), and rpm-ostree are all atomic rollbacks. I think it can be argued that rollbacks imply the expectation of atomicity. Otherwise you'd just say "restoring from backup" or "doing a system restore/rebuild from backup".
Thanx Chris for this elucidation. However, it is still not clear how one can do an atomic rollback when the currently booted system and perhaps logged in user has (have) already modified some files. How does one proceed then? Save the files modified since last backup or snapshot, do the restoration/ rollback, and then go through the tedious process of comparing the just saved files against the ones brought by rollback? How does one proceed in such a situation?
There's the big picture point of view:
It depends on the snapshot+rollback strategy and underlying storage technology involved.
It's possible that the snapshot taken is used for rollback; or it can be the thing updates are applied to.
For snapper+btrfs+yast (on opensuse 13.2) the behavior is to take a snapshot before any configuration changes including adding or deleting packages. So the current state is snapshot, then the current state is modified. A rollback means reverting to a snapshot, which has two kinds: temporary via GRUB menu, and persistent via a yast or snapper command.
For ostree and rpm-ostree (Fedora Atomic), the behavior isn't a filesystem snapshot in a Btrfs sense because this works on plain ext4 and XFS by using uniquely named directory "trees" and hardlinks. So a new tree "faux-snapshot" is setup with a bunch of hardlinks, and then that new directory tree is what gets the updates. The currently active tree is not modified. There is no need for a rollback if the update fails, because the current tree was never modified.
And then there's the detailed view:
The different strategies above have different layouts to separate out what things are rolled back and not. For example there's never a snapshot or rollback of /home. Your /home files are always progressing unidirectionally through time, no matter what. And the same thing applies to /etc and much of /var. So if you make a mistake with some system settings, those don't get rolled back with the above strategies.
Could you have more granularity so you could snapshot and rollback such things? Sure. But someone would have to do that implementation work and test it. And probably it'd need to be something fairly agnostic file system wise so it works on anything; which could just be a simple config file rename and versioning scheme. If it were Btrfs based you could very cheaply snapshot these things separate from other directories and for selective rollback of one file use cp --reflink behind the scenes.
On 08/11/2015 10:52 AM, Diogene Laerce wrote:
Hi Rick,
Le 11/08/2015 19:18, Rick Stevens a écrit :
On 08/11/2015 09:18 AM, Chris Murphy wrote:
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
I agree with Chris. Snapshots are ways of restoring data that has been perhaps corrupted or deleted or going back in time to some earlier point in the filesystems' life. They aren't backups.
Backup philosophies and techniques vary depending on what you need for your unique situation. I'm not claiming what I do is the best, but this is what works for me:
On the last Friday of a month, I plug in a big ESATA or USB3 drive and use it to store the output of Mondo Rescue. I have the backup in the form of DVD-sized ISO images I can burn DVDs from and there is a recovery DVD you can boot from. That gives me a backup usable to restore to bare metal.
Once a week (or more often if there's been significant changes), I use a _different_ ESATA or USB3 drive and run an rsync that backs up everything except a few things (/proc, /sys, /dev, /media, /var/log/journal, various caches, etc.) to a directory on that external drive based on the hostname and date I ran the backup. That permits me to restore data that's a bit more recent than the MondoRescue stuff.
I'd be happy to share the MondoArchive and rsync scripts if you wish. Tweak to suit your needs.
Thank you for the offer.
I'm going to stick with RedoBackup but I'd really like to have a look at your rsync scripts, and maybe at what various caches you think of.
The rsync backup is done via this script:
---------------------- CUT HERE -------------------------------------- #!/bin/bash # Back up system to a specific directory given on the command line. Excludes # the /proc, /sys, /dev and /media directories MYHOST=`hostname` TODAY=`date +%d-%b-%Y` if [ $# -lt 1 ]; then TGT="/media/500GB-Drive/Backups/$MYHOST-BackUp-$TODAY" OPT="" else TGT="$1/$MYHOST-BackUp-$TODAY" OPT="--exclude=$1/***" fi
nice -n 19 /bin/rsync -avXA --exclude-from=/etc/skipdirs.rsync $OPT / $TGT ---------------------- CUT HERE --------------------------------------
And the "/etc/skipdirs.rsync" file is:
---------------------- CUT HERE -------------------------------------- /proc/* /sys/* /dev/* /media/** /var/log/journal/* /BACKUPS/*** **/.cache/google-chrome/*** **/.ccache/*** /run/media/** var/lib/docker/* ---------------------- CUT HERE --------------------------------------
Obviously, change the "TGT" bit to reflect where your drive is located and modify the skipdirs.rsync file to reflect your needs.
As far as the mondoarchive stuff:
---------------------- CUT HERE -------------------------------------- #!/bin/bash # runmondoarchive.sh - Run mondoarchive with standard options
# Only allow this script to be run as the root user... if [ $EUID -ne 0 ]; then echo "This script must be run as the root user. Exiting." exit 1 fi
# Grab this host's name... THISHOST=`hostname -s`
# Get the date as mm-dd-yyyy... DT=`date +%m-%d-%Y`
# Say where the images are to be saved... DEST=/media/1TB-Drive/Backups/MondoImages
# Scratch directory... SCR=/tmp/mondoscratch if [ ! -d $SCR ]; then mkdir -p $SCR fi
# Temp directory... TMP=/tmp/mondotemp if [ ! -d $TMP ]; then mkdir -p $TMP fi
# Set how big the images can be... #SZ=640m # CD/CD-R SZ=4480m # DVD/DVD-R
# Call MondoArchive. Arguments are: # # -O Back up the host # (you can add "-V" to verify images # against the current filesystem) # -i Create ISO files # -p $THISHOST-$DT Specify the prefix each ISO image will # have (hostname-dd-mm-yyyy) # -I "/" Back up EVERYTHING from the root on down # -E "/media" Do NOT back up anything in "/media" # -d $DEST Destination where ISO images will go # -s $SZ Set the maximum size of each ISO image # -S $SCR Set scratch directory (where ISOs are # built) # -T $TMP Set temporary working directory # -z Archive extended attributes of files CMD="mondoarchive -O -i -p ${THISHOST}-${DT} -I "/" -E "/media" -d ${DEST} -s ${SZ} -z" echo -e "About to run:\n\t$CMD" $CMD ---------------------- CUT HERE --------------------------------------
Again, modify as you see fit. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - I haven't lost my mind. It's backed up on tape somewhere, but - - probably not recoverable. - ----------------------------------------------------------------------
Le 11/08/2015 22:54, Rick Stevens a écrit :
On 08/11/2015 10:52 AM, Diogene Laerce wrote:
Hi Rick,
Le 11/08/2015 19:18, Rick Stevens a écrit :
On 08/11/2015 09:18 AM, Chris Murphy wrote:
On Tue, Aug 11, 2015 at 3:05 AM, Diogene Laerce me_buss777@yahoo.fr wrote:
https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ? As one does not need to reboot and just have to make a script or/and a cron job to make his own snapshot.
I don't consider a backup and snapshot to be the same thing. Copying (or rsyncing) some directories to another volume is a backup. A snapshot is a deduplicated copy of something at a particular moment in time, on the same volume or storage pool. It's not a backup, in that if the pool implodes both the original and snapshot are lost. A snapshot can be used as a source for a backup, since you can make a snapshot that doesn't change while the backup is happening.
I agree with Chris. Snapshots are ways of restoring data that has been perhaps corrupted or deleted or going back in time to some earlier point in the filesystems' life. They aren't backups.
Backup philosophies and techniques vary depending on what you need for your unique situation. I'm not claiming what I do is the best, but this is what works for me:
On the last Friday of a month, I plug in a big ESATA or USB3 drive and use it to store the output of Mondo Rescue. I have the backup in the form of DVD-sized ISO images I can burn DVDs from and there is a recovery DVD you can boot from. That gives me a backup usable to restore to bare metal.
Once a week (or more often if there's been significant changes), I use a _different_ ESATA or USB3 drive and run an rsync that backs up everything except a few things (/proc, /sys, /dev, /media, /var/log/journal, various caches, etc.) to a directory on that external drive based on the hostname and date I ran the backup. That permits me to restore data that's a bit more recent than the MondoRescue stuff.
I'd be happy to share the MondoArchive and rsync scripts if you wish. Tweak to suit your needs.
Thank you for the offer.
I'm going to stick with RedoBackup but I'd really like to have a look at your rsync scripts, and maybe at what various caches you think of.
The rsync backup is done via this script:
---------------------- CUT HERE -------------------------------------- #!/bin/bash # Back up system to a specific directory given on the command line. Excludes # the /proc, /sys, /dev and /media directories MYHOST=`hostname` TODAY=`date +%d-%b-%Y` if [ $# -lt 1 ]; then TGT="/media/500GB-Drive/Backups/$MYHOST-BackUp-$TODAY" OPT="" else TGT="$1/$MYHOST-BackUp-$TODAY" OPT="--exclude=$1/***" fi
nice -n 19 /bin/rsync -avXA --exclude-from=/etc/skipdirs.rsync $OPT / $TGT ---------------------- CUT HERE --------------------------------------
And the "/etc/skipdirs.rsync" file is:
---------------------- CUT HERE -------------------------------------- /proc/* /sys/* /dev/* /media/** /var/log/journal/* /BACKUPS/*** **/.cache/google-chrome/*** **/.ccache/*** /run/media/** var/lib/docker/* ---------------------- CUT HERE --------------------------------------
Thank you for those.
Caches are roughly what I was thinking of but I would not have thought of nice.. Nice.
Kind regards,
Hi Mr Heinz,
Thanks for stopping by. ;)
Le 11/08/2015 20:15, Heinz Diehl a écrit :
On 11.08.2015, Diogene Laerce wrote:
Is there a trick I don't see here ? Because if the backup of those directories is enough for a full restoration of a system state, this method is far more efficient than the others, isn't it ?
If you backup all your partitions with rsync, all you have to do in an emergency case is to boot from e.g. CD or a memory stick [1] and reverse the rsync command.
Example:
rsync -avxHSAX /home/ /backup/home --> backup rsync -avxHSAX --delete /backup/home/ /home --> restore
The "--delete" parameter will take care of all the files that are not in the same state as when they were saved.
So in case of a complete disaster, just restore your data as described. If your boot sector is damaged, you'll however have to restore it by hand (which isn't all too difficult).
I actually modified Rick's script with those rsync parameters you provided which seem more accurate for my system.
Kind regards,
On 08/11/2015 02:15 PM, Heinz Diehl wrote:
So in case of a complete disaster, just restore your data as described. If your boot sector is damaged, you'll however have to restore it by hand (which isn't all too difficult).
whenever I mess up my MBR, I boot the boot rescue CD.. it is automatic, rebuilds the MBR so grub works again..
http://sourceforge.net/p/boot-repair-cd/home/Home/
-
Andy Paterson -
Chris Murphy -
Diogene Laerce -
Erik Grun -
Gordon Messmer -
Heinz Diehl -
JD -
Michael D. Setzer II -
Paul Cartwright -
Rick Stevens