[PATCH] Ensure that rsync backups are only running one at a time

[Toshio Kuratomi]

On Wed, Mar 24, 2010 at 10:54:07AM -0500, Mike McGrath wrote:
> ---
>  configs/db/backup-dbs |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/configs/db/backup-dbs b/configs/db/backup-dbs
> index 83ed6c8..7af66fb 100755
> --- a/configs/db/backup-dbs
> +++ b/configs/db/backup-dbs
> @@ -20,6 +20,8 @@ mv $DEST/$HOSTNAME.new $DEST/$HOSTNAME
>  
>  # Sync out
>  for host in db01 db02 db03; do
> +    # Sleep if any other rsyncs are already
> +    while ssh $host "pgrep rsync" | grep -q [0-9]; do sleep 10; done
>      if [ "$host" != $HOSTNAME ]; then
>          su - dbbackup -c "ssh $host mkdir -p $DEST/$HOSTNAME"
>          su - dbbackup -c "rsync -azr --bwlimit=5000 -e ssh $DEST/$HOSTNAME/* $host:$DEST/$HOSTNAME/"
>
So this means that among the three db hosts, db01, db02, and db03 we'll have
at most one rsync running at any one time.  That's going to increase the
time to sync some more.

I think that this code wonn't quite function properly when more than one
backup-dbs script runs on a box at a time... what gets transferred to the
remote host will be a mixture of what was in $DEST/$HOSTNAME when the rsync
starts and what's in there when the rsync ends.. We would get errors if
filenames were removed (ie: a database is removed while the rsync is still
procesing).  Do to copying the directory prior to rsyncing, I don't think
we'll get corruption of the actual dump files -- we'll just get dump files
from two separate runs intermingled on the host being backed up to.

It doesn't look like the scripts are currently stacking on a single host.
They run at every six hours on db01 and db02 (every 12 hours on db03).

So the risk of problems at this time doesn't seem too bad.

+1

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20100324/d39bfc56/attachment.bin