I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
That seems a more direct approach and I could easily check to make certain that the second drive was a usable copy, insurance against loss of data.
Am I going wrong somewhere in my thinking?
Bob
On Mon, Mar 9, 2015 at 12:04 PM, Bob Goodwin bobgoodwin@wildblue.net wrote:
I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
That seems a more direct approach and I could easily check to make certain that the second drive was a usable copy, insurance against loss of data.
Am I going wrong somewhere in my thinking?
Nope, definitely setup the rsync as backup before worrying about raid1 for improved uptime. Once the rsync is in place, should you wish to ensure improved uptime for either your production system or the backup itself, then consider raid1 in addition.
Rsync gives you incremental backup flexibility, and at least the possibility of recovering not yet overwritten files. A common form of data loss is user induced, e.g. accidentally deleting a file. With raid1, that change happens immediately. With rsync, there's a delay. So you actually have a real backup with rsync between two independent file systems; whereas raid1 is really not backup, it's design goal is improving uptime so you can keep working despite a device failure. Another option is using rsync checksum verification, which is a ton slower, but absolutely ensures source and destination are the same independent of drives' ECC.
On 03/09/15 14:58, Chris Murphy wrote:
On Mon, Mar 9, 2015 at 12:04 PM, Bob Goodwin bobgoodwin@wildblue.net wrote:
I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
That seems a more direct approach and I could easily check to make certain that the second drive was a usable copy, insurance against loss of data.
Am I going wrong somewhere in my thinking?
Nope, definitely setup the rsync as backup before worrying about raid1 for improved uptime. Once the rsync is in place, should you wish to ensure improved uptime for either your production system or the backup itself, then consider raid1 in addition.
Rsync gives you incremental backup flexibility, and at least the possibility of recovering not yet overwritten files. A common form of data loss is user induced, e.g. accidentally deleting a file. With raid1, that change happens immediately. With rsync, there's a delay. So you actually have a real backup with rsync between two independent file systems; whereas raid1 is really not backup, it's design goal is improving uptime so you can keep working despite a device failure. Another option is using rsync checksum verification, which is a ton slower, but absolutely ensures source and destination are the same independent of drives' ECC.
I'm primarily interested in the convenience of having my data on a server and preventing it's loss. I don't require extreme performance, etc. Reassured by your comments I will give this a try then.
Thank you,
Bob
Bob Goodwin writes:
I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
That seems a more direct approach and I could easily check to make certain that the second drive was a usable copy, insurance against loss of data.
Am I going wrong somewhere in my thinking?
Raid, and rsyncing, are not equivalent to each other. They solve different problems.
For starters, in your case taking an existing partition and turning it into a RAID volume is going to be a big project. You'll have to back up your existing data somewhere, erase the existing partition, create a raid partition, then restore your data from backups. I'm not aware of a way to turn an existing non-RAID partition into a RAID one.
rsync is easier to set up when you have existing data, and I do have a few laptops where I have a daily job to rsync their data onto a different server.
On the other hand, RAID is generally faster, and runs in realtime. With rsync, you do have some window of vulnerability where you will lose everything since your last rsync run, when you have a failure, where RAID provides up to the second redundancy.
On 03/09/15 20:23, Sam Varshavchik wrote:
Bob Goodwin writes:
I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
That seems a more direct approach and I could easily check to make certain that the second drive was a usable copy, insurance against loss of data.
Am I going wrong somewhere in my thinking?
Raid, and rsyncing, are not equivalent to each other. They solve different problems.
For starters, in your case taking an existing partition and turning it into a RAID volume is going to be a big project. You'll have to back up your existing data somewhere, erase the existing partition, create a raid partition, then restore your data from backups. I'm not aware of a way to turn an existing non-RAID partition into a RAID one.
I had intended to set up the raid server first and then transfer data from the original drive, that seemed like it should would work?
And yes, I think rsync is really what I want anyway since I'm more interested in saving the data in two places to minimize the chance of losing everything with a failure of some kind. I realize there is a chance of losing some data depending on when the failure occurs relative to the last rsync and I'm willing to accept that. My files are important to me but not so critical that I need to cover every possibility.
rsync is easier to set up when you have existing data, and I do have a few laptops where I have a daily job to rsync their data onto a different server.
I have been using rsync to make copies of files between several computers for some time and it works without a hitch for me ...
On the other hand, RAID is generally faster, and runs in realtime. With rsync, you do have some window of vulnerability where you will lose everything since your last rsync run, when you have a failure, where RAID provides up to the second redundancy.
So it looks like that is a reasonable thing to try. I've set up the NFS server and will transfer some data into it tomorrow.
Thanks for the advice,
Bob
On 10.03.2015 01:23, Sam Varshavchik wrote: ...
For starters, in your case taking an existing partition and turning it into a RAID volume is going to be a big project. You'll have to back up your existing data somewhere, erase the existing partition, create a raid partition, then restore your data from backups. I'm not aware of a way to turn an existing non-RAID partition into a RAID one.
:)
man 8 mdadm
CREATE MODE ... To create a "degraded" array in which some devices are missing, simply give the word "missing" in place of a device name. This will cause mdadm to leave the corresponding slot in the array empty. ... For a RAID1 array, only one real device needs to be given. All of the others can be "missing".
"missing" == existing non RAID drive
In brief: 1. Create a "degraded" array with 2nd drive 2. Transfer the contents of the "missing" drive to it 3. Prepare "missing" and add it to the array i.e. RAID-1
Ref. There are plenty howtos out there.
On Mon, Mar 09, 2015 at 08:23:23PM -0400, Sam Varshavchik wrote:
rsync is easier to set up when you have existing data, and I do have a few laptops where I have a daily job to rsync their data onto a different server.
I would recommend rsnapshot. It makes setting up an rsync & cron based incremental backup system very easy. The only downside is recovery is a rather manual process. For a more full-blown solution with rsync, BackupPC might be interesting to the OP.
Cheers,
One assumes that if you are migrating a filesystem to RAID, you are installing a new disk (for e.g. RAID 1) (you need multiple disks - as in the acoronym RAID). So, if you install your new disk alongside your existing disk, create a RAID mirror with only one partition (from the new disk) create a filesystem on the new "RAID with only one disk" - then copy the contents of your existing filesystem to your new filesystem. Then you can remove reference to your old filesystem from fstab and maybe use fdisk to change the partition type associated with it. Add the old disk (partition) to your new raid with mdadm - allow the mirrors to resync and you have your RAID 1 mirror. So i don't think "taking an existing partition and turning it into a RAID volume is going to be a big project".
FYI M$ now allow "snapshots" be taken from their raid 1 setup and used as backups. Its relatively easy to achieve the same thing using LINUX raid but best to use a three way mirror - its just a matter of ensuring the filesystem is ideally unmounted before detaching a mirror) - although even if you don't - fsck can usually sort out any mess. Contrast that with unmounting a filesystem and doing a level 0 backup via dump or bacula and having your system offline for an hour or two! OK if you want to do incremental backups - its faster, but you sitll need media for each of your incrementals. I have even found that it is possible to detach a mirror and force the detached mirror to mount as a non-raid (ext3/4 or whatever) filesystem. (good for getting at your "backups".
(expect smoke and flame here!) Andy
I find it amusing that a few people give comments like "Raid, and rsyncing, are not equivalent to each other. They solve different problems" without actually defining in detail what they mean and why - without waffle!
On Monday 09 March 2015 20:23:23 Sam Varshavchik wrote:
Bob Goodwin writes:
I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
That seems a more direct approach and I could easily check to make certain that the second drive was a usable copy, insurance against loss of data.
Am I going wrong somewhere in my thinking?
Raid, and rsyncing, are not equivalent to each other. They solve different problems.
For starters, in your case taking an existing partition and turning it into a RAID volume is going to be a big project. You'll have to back up your existing data somewhere, erase the existing partition, create a raid partition, then restore your data from backups. I'm not aware of a way to turn an existing non-RAID partition into a RAID one.
rsync is easier to set up when you have existing data, and I do have a few laptops where I have a daily job to rsync their data onto a different server.
On the other hand, RAID is generally faster, and runs in realtime. With rsync, you do have some window of vulnerability where you will lose everything since your last rsync run, when you have a failure, where RAID provides up to the second redundancy.
On Tue, 2015-03-10 at 10:27 +0100, Suvayu Ali wrote:
On Mon, Mar 09, 2015 at 08:23:23PM -0400, Sam Varshavchik wrote:
rsync is easier to set up when you have existing data, and I do have a few laptops where I have a daily job to rsync their data onto a different server.
I would recommend rsnapshot. It makes setting up an rsync & cron based incremental backup system very easy. The only downside is recovery is a rather manual process. For a more full-blown solution with rsync, BackupPC might be interesting to the OP.
+1
rsnapshot is simple and very effective, once you understand that it's meant to be run on the server rather than the client (the docs aren't sufficiently explicit about this and as it will also work the other way you can find yourself with a correct but very slow backup process if you don't pick up on it).
poc
On 03/09/2015 11:04 AM, Bob Goodwin wrote:
However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
I know I'm going to repeat some of what has already been said. My 2c anyway:
No, rsync would not work just as well.
Do you want your system to continue functioning when one of your drives fails? If so, then set up RAID1 and make sure you actually get and read email from cron jobs. In the event of failure, the mdmonitor service will send email to "root" to indicate that a drive needs to be replaced. The down side is that your data will have no backups. If a file is accidentally deleted or corrupted, you probably have no recourse.
Do you, instead, want multiple levels of online backups? In that case, there are a handful of backup applications, including rsnapshot, that handle rotation and rsync to provide efficient backups. If your primary drive fails, you'll deal with the outage while you get a replacement drive, format it, install a system, restore your data, etc, which could be a fairly long process. Instead, you'll gain very coarse file versions and protection from accidental deletions.
Am I going wrong somewhere in my thinking?
Thinking that you have to make a choice between the two may be in error. Depending on the size of your disks and the amount of data on them, you could potentially have both RAID1 and backups.
Build a system with a RAID1 mirror on the two drives that uses half of the available space for your system, and half of the space for a separate backup filesystem. Keeping the backup filesystem separate provides some additional protection against filesystem corruption. It's still possible for some errors to destroy both your system and its backups, but in most cases, you'll get good coverage for the most common failures with this setup.
On 03/10/15 12:29, Gordon Messmer wrote:
On 03/09/2015 11:04 AM, Bob Goodwin wrote:
However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
I know I'm going to repeat some of what has already been said. My 2c anyway:
No, rsync would not work just as well.
Do you want your system to continue functioning when one of your drives fails? If so, then set up RAID1 and make sure you actually get and read email from cron jobs. In the event of failure, the mdmonitor service will send email to "root" to indicate that a drive needs to be replaced. The down side is that your data will have no backups. If a file is accidentally deleted or corrupted, you probably have no recourse.
Do you, instead, want multiple levels of online backups? In that case, there are a handful of backup applications, including rsnapshot, that handle rotation and rsync to provide efficient backups. If your primary drive fails, you'll deal with the outage while you get a replacement drive, format it, install a system, restore your data, etc, which could be a fairly long process. Instead, you'll gain very coarse file versions and protection from accidental deletions.
Am I going wrong somewhere in my thinking?
Thinking that you have to make a choice between the two may be in error. Depending on the size of your disks and the amount of data on them, you could potentially have both RAID1 and backups.
Build a system with a RAID1 mirror on the two drives that uses half of the available space for your system, and half of the space for a separate backup filesystem. Keeping the backup filesystem separate provides some additional protection against filesystem corruption. It's still possible for some errors to destroy both your system and its backups, but in most cases, you'll get good coverage for the most common failures with this setup.
. Well as I said earlier, I mainly want to have files that I consider critical backed up somewhere. I'm not very much concerned about equipment failure and downtime. I've been backing stuff up between computers using rsync so that I'll always have a fairly recent copy of my notes, checking account, genealogy, etc.
Presently most of that data resides on a 1 TB drive in an NFS server running SL-6. I have two new WD/black 1 TB drives, that I am going to use on the computer I'm working on which is presently running Fedora-21, probably not the best choice for the purpose but I thought I'd try it.
I also have a Raid1 samba server [SL-7], mainly to deal with the family Apple users [everyone but me]. I have never received any messages from that. As it stands I might not know if there was a failure, I've been worrying about that!
I am considering everything in the responses I've received ...
Thanks,
Bob
On Tue, Mar 10, 2015 at 09:29:50AM -0700, Gordon Messmer wrote:
Build a system with a RAID1 mirror on the two drives that uses half of the available space for your system, and half of the space for a separate backup filesystem. Keeping the backup filesystem separate provides some additional protection against filesystem corruption. It's still possible for some errors to destroy both your system and its backups, but in most cases, you'll get good coverage for the most common failures with this setup.
In my encouter with data corruption in the past, it has usually been entire drive failures rather than one particular filesystem failing on me, and the other keeps functioning.
I would always encourage separate physical disks as backup partitions. If the OP has flaky power, maybe having them offline when not in use, would also be a good idea.
just my 2¢,
On Mon, Mar 9, 2015 at 11:04 AM, Bob Goodwin bobgoodwin@wildblue.net wrote:
I had a mainboard fail in a box I use as a server, I moved the hard drive into old computer and carried on from there. Now I've replaced the board and intended to set it up using Raid to mirror two drives. However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
For me, it depends on how much data is involved and how safe you want it to be.
I would set up the Raid array and have either a third drive, or better yet, a separate server for rsync.
Ideally the capacity of your rsync server would be many times that of your main server's data so you could make backups daily, weekly and monthly and save enough of them for file recovery in the event of human error.
What usually happens is that a file is corrupted, by either man or machine, and then that corrupted data goes to your backup, and you are screwed.
If you have backups from different periods of time, at least you can go back to an older backup and find a "good" copy of a file.
This might also be a good use case for either ZFS or btrfs, so you can wind back the clock on file changes if there is trouble.
-- Steven Rosenberg http://stevenrosenberg.net/blog http://blogs.dailynews.com/click stevenhrosenberg@gmail.com steven@stevenrosenberg.net
On Tue, Mar 10, 2015 at 1:24 PM, Bob Goodwin bobgoodwin@wildblue.net wrote:
Well as I said earlier, I mainly want to have files that I consider critical backed up somewhere.
I'd say all data needs at least one backup. Critical stuff should also be off-site in addition to the local backups. If you won't or can't do an off-site backup, well maybe it isn't really critical, but then you need at least two local backups.
Because of the high time penalty for cloud backup service restore, you may want a 2nd local backup anyway, just in case.
I'm not very much concerned about equipment failure and downtime.
Discounting downtime is OK. Discounting equipment failure probably isn't.
You need to understand what the consequences are if any two individual pieces of equipment fail at the same time, and mitigate that somehow. e.g. Crashplan explicitly supports Linux, Backblaze explicitly does not.
Amazon Glacier is fairly inexpensive and equivalent or better to the reliability of tape and cheaper for most cases; a small percentage of retrieval per month is free, but if you go over that (i.e. a full blown recovery is needed rather than just an accidentally deleted or corrupted file) it can be expensive. There are 3rd party calculators for computing this, it's not exactly obvious what things will cost just based on their rather convoluted pricing page.
On Tue, Mar 10, 2015 at 1:43 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
In my encouter with data corruption in the past, it has usually been entire drive failures rather than one particular filesystem failing on me, and the other keeps functioning.
I would always encourage separate physical disks as backup partitions. If the OP has flaky power, maybe having them offline when not in use, would also be a good idea.
This is quite good advice.
It's one reason why I do use Btrfs. But I have multiple separate Btrfs backups (one is raid0, one is raid1). One backup is on HFS+ just because if I had to do a quick restore of the Mac, this is push a single button type restore, but of course is subject to SDC concerns. And before I was confident in both Btrfs and my ability to repair it should that be necessary, I kept two additional backups on XFS on separate drives and file systems. And then the 6th is select data in an encrypted image stored in the cloud, which is yet again a separate physical device and filesystem.
The purpose of Btrfs raid1 is not so much avoiding downtime for me, but to take advantage of automatic detection of corruption and autohealing. The raid0 is mainly for size, of course it totally dies if anyone device dies. But in the meantime, I still get notifications of any corrupt files (by full filename path) should that happen.
On 03/10/2015 12:24 PM, Bob Goodwin wrote:
On 03/10/15 12:29, Gordon Messmer wrote:
On 03/09/2015 11:04 AM, Bob Goodwin wrote:
However I have been wondering if it wouldn't work just as well to periodically rsync the drive in use with a second drive?
I know I'm going to repeat some of what has already been said. My 2c anyway:
No, rsync would not work just as well.
Do you want your system to continue functioning when one of your drives fails? If so, then set up RAID1 and make sure you actually get and read email from cron jobs. In the event of failure, the mdmonitor service will send email to "root" to indicate that a drive needs to be replaced. The down side is that your data will have no backups. If a file is accidentally deleted or corrupted, you probably have no recourse.
Do you, instead, want multiple levels of online backups? In that case, there are a handful of backup applications, including rsnapshot, that handle rotation and rsync to provide efficient backups. If your primary drive fails, you'll deal with the outage while you get a replacement drive, format it, install a system, restore your data, etc, which could be a fairly long process. Instead, you'll gain very coarse file versions and protection from accidental deletions.
Am I going wrong somewhere in my thinking?
Thinking that you have to make a choice between the two may be in error. Depending on the size of your disks and the amount of data on them, you could potentially have both RAID1 and backups.
Build a system with a RAID1 mirror on the two drives that uses half of the available space for your system, and half of the space for a separate backup filesystem. Keeping the backup filesystem separate provides some additional protection against filesystem corruption. It's still possible for some errors to destroy both your system and its backups, but in most cases, you'll get good coverage for the most common failures with this setup.
. Well as I said earlier, I mainly want to have files that I consider critical backed up somewhere. I'm not very much concerned about equipment failure and downtime. I've been backing stuff up between computers using rsync so that I'll always have a fairly recent copy of my notes, checking account, genealogy, etc.
Presently most of that data resides on a 1 TB drive in an NFS server running SL-6. I have two new WD/black 1 TB drives, that I am going to use on the computer I'm working on which is presently running Fedora-21, probably not the best choice for the purpose but I thought I'd try it.
I also have a Raid1 samba server [SL-7], mainly to deal with the family Apple users [everyone but me]. I have never received any messages from that. As it stands I might not know if there was a failure, I've been worrying about that!
I am considering everything in the responses I've received ...
As people have remarked, RAID (at least RAID1, RAID5, RAID6 and RAID10) is a way to make sure that one (or more) drive failures doesn't kill your system. A drive can die in one of those and the thing keeps running (albeit at a reduced rate and you lose the reliability until the failed drive has been replaced and the "rebuild" process completes).
Depending on the physical size of any individual disk in the RAID array, these rebuild times can get fairly lengthy, which is why I tend to use RAID6 when drives go above 1TB. A RAID6 with one failed drive behaves like a RAID5, so you can tolerate one more drive failure before you have a data corruption problem:
RAID6 minus 1 drive = RAID5 RAID5 minus 1 drive = Degraded RAID 5 (cannot tolerate another failure)
I replace the failed RAID6 drive right away, but because it might take a day or two to rebuild, it's highly possible a second drive might die in that period. RAID6 reduces that window of vulnerability.
RAID0 is a way to spread data across multiple spindles (disk drives) to improve performance but it does NOT offer redundancy. Use RAID10 for that (RAID10 = a RAID0 with each drive in a RAID1 mirrored pair).
Note that all these RAID things do is make multiple drives appear to the operating system as a single physical drive. It is NOT a backup.
Regardless of WHERE the RAID array is (another physical machine, the machine you're running on, whatever), you have to get data ONTO the RAID array somehow. rsync (and its various permutations such as rsnapshot, UDR, etc.) is one way. Dedicated backup programs such as Bacula and Amanda are another. If the RAID is on the machine you want to back up, you could even use tar, cpio or cp to do it.
We use Bacula because we have to back up a large number of machines (well over 200). Bacula makes that management fairly tolerable and you can define what gets backed up and when (full backups, incrementals, snapshots, etc.).
The backup media where all the stuff goes is a big storage array (one filesystem on an HP StoreAll 9730), which (at its core) is a whole lot of RAID6 arrays.
---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - First Law of Work: - - If you can't get it done in the first 24 hours, work nights. - ----------------------------------------------------------------------
On Tue, Mar 10, 2015 at 2:38 PM, Steven Rosenberg stevenhrosenberg@gmail.com wrote:
What usually happens is that a file is corrupted, by either man or machine, and then that corrupted data goes to your backup, and you are screwed.
SDC and propagation of corruption is a big problem actually especially rotated media backups, or even the restore process, causes derivatives. Each derivative inherits and creates its own SDC over time. A file gets corrupt, and then backed up to one or more backups, eventually replacing all good copies. And there's no notification of this.
This even affects Btrfs and ZFS if it isn't exclusively used in the chain. If corruption has already happened, of course Btrfs and ZFS simply maintain and propagate that corruption. Granted it should stop additional corruptions (or at least notify of them). But we need e.g. /home to be on Btrfs to significant reduce this.
The network is a source of SDC also. It's conceivable to have a Btrfs source and destination, using rsync between them, and for SDC to get introduced without notice by the network. The BER for networks is highly variable, it's not a static thing, it can change just by pinching a wire. If you follow the specs exactly then you get a BER that probably isn't being exceeded value, but that's it, no guarantee.
The best guarantee? Btrfs /home, Btrfs backups, and rsync with checksumming enabled to confirm what's on the source is actually what's on the destination. That's slow. There is a feature idea to get rsync some Btrfs awareness so that this can be optimized, taking advantage of work Btrfs has already done, rather than separately compute additional checksums.
Anyone using RAID of any type: hardware, md, LVM, Btrfs, ZFS, really need to be aware of drive SCT ERC and kernel SCSI command timer mismatches. The mismatch happens by default if you're using consumer hard drives, many of which now either have SCT ERC disabled by default or do not support it. This mismatch will eventually lead to array collapse, even in the face of just a one disk failure for RAID5 or 2 disk failure for RAID6. RAID1 is a bit more tolerant but only because a sector read error on a degraded raid1 only means the array doesn't go offline, other files can still be retrieved in such a case - assuming the sector errors haven't resulted in data corruption.
The linux-raid@ list is full of these kinds of horror stories. Invariably it's a - yep, your raid5 with just one dead drive? It's toast, or at least very tedious to recover data from, because one or more surviving drives have one or more bad sectors and md can't recover from that automatically.
Chris Murphy
On 10.03.2015, Suvayu Ali wrote:
I would always encourage separate physical disks as backup partitions. If the OP has flaky power, maybe having them offline when not in use, would also be a good idea.
I second that, this is a very important advice! In addition, use a good lightning protector both for the power outlet and the network.
On Tue, Mar 10, 2015 at 2:55 PM, Chris Murphy lists@colorremedies.com wrote:
Amazon Glacier is fairly inexpensive
Google announced Nearline option for Google Cloud Storage, priced similar to Amazon Glacier but with faster retrieval. Storage price is the same, but retrieval is more expensive from the looks of it. Amazon has a complicated formula for retrieval. Fast retrievals cost a LOT more than slow ones.
On 03/10/2015 09:38 PM, Steven Rosenberg wrote:
Ideally the capacity of your rsync server would be many times that of your main server's data so you could make backups daily, weekly and monthly and save enough of them for file recovery in the event of human error.
Actually not. You just a need a slightly larger server, if you use hardlinks: rsync with option --link-dest or utilities using the same principle (rsnapshot).
You are perfectly right about human error.
RAID-1 protects from disk failure (transparenty) rsync protects from human error (and disk failure, but with some inconvenience)
TL;DR (at least not the entire thread)
If you go with Rsync you may find incron an useful add-on as well as with it you can monitor a variety of events on any given file or directory, like OPEN_READ, CREATE, CLOSE, CLOSE_WRITE, etc - and this way backup your data in real time whenever a condition applies. Also if you will be rsyncing from one computer to another (in contrast to an attached storage) setting up an Rsync daemon on destination will make the sync to be faster and lighter than syncing over ssh; of course that if you need an encrypted connection between the two nodes you will likely be rsyncing over ssh. But again, if you trust your network, the Rsync daemon option will use much less resources. Also should you go with Rsync, setting up xinetd may be even a better option than having an Rsync daemon constantly running in your target server.
HTH.
On Fri, Mar 13, 2015 at 12:55 PM, Roberto Ragusa mail@robertoragusa.it wrote:
On 03/10/2015 09:38 PM, Steven Rosenberg wrote:
Ideally the capacity of your rsync server would be many times that of your main server's data so you could make backups daily, weekly and monthly and save enough of them for file recovery in the event of human error.
Actually not. You just a need a slightly larger server, if you use hardlinks: rsync with option --link-dest or utilities using the same principle (rsnapshot).
You are perfectly right about human error.
RAID-1 protects from disk failure (transparenty) rsync protects from human error (and disk failure, but with some inconvenience)
-- Roberto Ragusa mail at robertoragusa.it -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
On 03/14/2015 11:02 PM, Martin Cigorraga wrote:
Also if you will be rsyncing from one computer to another (in contrast to an attached storage) setting up an Rsync daemon on destination will make the sync to be faster and lighter than syncing over ssh; of course that if you need an encrypted connection between the two nodes you will likely be rsyncing over ssh. But again, if you trust your network, the Rsync daemon option will use much less resources.
If the network is trusted, there is also "rsync --rsh=rsh" (with a properly configured rsh and .rhosts).
On Tue, Mar 10, 2015 at 10:27:08AM +0100, Suvayu Ali wrote:
I would recommend rsnapshot. It makes setting up an rsync & cron based incremental backup system very easy. The only downside is recovery is a rather manual process. For a more full-blown solution with rsync, BackupPC might be interesting to the OP.
You might also look into rdiff-backup. It has an advantage for remote backups as it stores metadata separately (and so you can write somewhere you don't have root, or even to weird filesystems which don't understand acls (or even permissions).
We also have newer entry in Fedora, ZBackup, which does deduplication, too. I haven't used it, but it looks promising.
-
Andy Paterson -
Bob Goodwin -
Chris Murphy -
Gordon Messmer -
Heinz Diehl -
Martin Cigorraga -
Matthew Miller -
Patrick O'Callaghan -
poma -
Rick Stevens -
Roberto Ragusa -
Sam Varshavchik -
Steven Rosenberg -
Suvayu Ali