Need help with rsync

[ToddAndMargo]

Hi All,

I am trying to use rsync to

1) copy new files on the source to the destination

2) remove all files on the destination that do not
    reside on the source

Problem: the extra files on the destination are
not being removed.

This is bash code:

Synchronize () {
    # $1 rsync command
    local Cmd="rsync ${1}"
    echo "${Cmd}"
    sync; sync
    eval "${Cmd}"
    sync; sync
    echo "-----------------------------"; echo ""
}

Options="--recursive --verbose --delete --delete-excluded 
--modify-window=5 --times --inplace --copy-links "

ExcludeLinux="--exclude='wine-*'"

Synchronize "${Options} ${ExcludeLinux} ${MyCDsSource}/Linux 
${MyCDsTarget}/."

Synchronize "${Options} ${MyCDsSource}/WindowsInternet ${MyCDsTarget}/."

... [eleven more]

What am I missing?

Many thanks,
-T

[Samuel Sieb]

On 11/12/20 5:45 PM, ToddAndMargo via users wrote:
> I am trying to use rsync to
>
> copy new files on the source to the destination
>
> remove all files on the destination that do not
>
> reside on the source
>
> Problem: the extra files on the destination are
> not being removed.
>
> Options="--recursive --verbose --delete --delete-excluded 
> --modify-window=5 --times --inplace --copy-links "
>
> ExcludeLinux="--exclude='wine-*'"
>
> Synchronize "${Options} ${ExcludeLinux} ${MyCDsSource}/Linux 
> ${MyCDsTarget}/."

That should work.
Is there any output indicating why it's not working?
What is an example file?

[ToddAndMargo]

On 2020-11-12 18:07, Samuel Sieb wrote:
> On 11/12/20 5:45 PM, ToddAndMargo via users wrote:
> > I am trying to use rsync to
> >
> > copy new files on the source to the destination
> >
> > remove all files on the destination that do not
> >
> > reside on the source
> >
> > Problem: the extra files on the destination are
> > not being removed.
> >
> > Options="--recursive --verbose --delete --delete-excluded 
> > --modify-window=5 --times --inplace --copy-links "
> >
> > ExcludeLinux="--exclude='wine-*'"
> >
> > Synchronize "${Options} ${ExcludeLinux} ${MyCDsSource}/Linux 
> > ${MyCDsTarget}/."
>
> That should work.
> Is there any output indicating why it's not working?
> What is an example file?

Source:
$ pwd
/home/CDs/Linux/Perl/6

$ ls
Perl6.WebSite.txt
rakudo-pkg-Fedora29-2019.03.1-01.x86_64.rpm
rakudo-pkg-Fedora30-2019.07.1-03.x86_64.rpm
RakudoPkgFedora31-2020.02.1.05.x86_64.rpm
RakudoPkgFedora32-2020.10.02.x86_64.rpm
RakudoPkgFedora33-2020.10.02.x86_64.rpm

Destination:
$ pwd
/mnt/MyCDs/Linux/Perl/6

$ ls
Perl6.WebSite.txt
rakudo-pkg-Fedora29-2018.11-01.x86_64.rpm
rakudo-pkg-Fedora29-2019.03.1-01.x86_64.rpm
rakudo-pkg-Fedora30-2019.03.1-03.x86_64.rpm
rakudo-pkg-Fedora30-2019.07.1-03.x86_64.rpm
rakudo-pkg-Fedora31-2019.07.1-03.x86_64.rpm
rakudo-pkg-Fedora31-2019.11-01.x86_64.rpm
rakudo-pkg-Fedora31-2020.01-01.x86_64.rpm
RakudoPkgFedora31-2020.02.1.04.x86_64.rpm
RakudoPkgFedora31-2020.02.1.05.x86_64.rpm
RakudoPkgFedora32-2020.02.1.05.x86_64.rpm
RakudoPkgFedora32-2020.02.1.07.x86_64.rpm
RakudoPkgFedora32-2020.05.1.x86_64.rpm
RakudoPkgFedora32-2020.05.x86_64.rpm
RakudoPkgFedora32-2020.06.x86_64.rpm
RakudoPkgFedora32-2020.07.02.x86_64.rpm
RakudoPkgFedora32-2020.07.x86_64.rpm
RakudoPkgFedora32-2020.08.2.x86_64.rpm
RakudoPkgFedora32-2020.09.x86_64.rpm
RakudoPkgFedora32-2020.10.02.x86_64.rpm
RakudoPkgFedora33-2020.10.02.x86_64.rpm

This is exiting after the call to Linux:

# UpdateMyCDs
Resetting permissions
Resetting ownerships

rsync --recursive --verbose --delete --delete-excluded --modify-window=5 
--times --inplace --copy-links  --exclude='wine-*' /home/CDs/Linux 
/mnt/MyCDs/.
sending incremental file list
symlink has no referent: 
"/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd"
IO error encountered -- skipping file deletion

sent 31,338 bytes  received 210 bytes  63,096.00 bytes/sec
total size is 3,207,298,531  speedup is 101,664.08
rsync error: some files/attrs were not transferred (see previous errors) 
(code 23) at main.c(1330) [sender=3.2.3]
-----------------------------

Does "IO error encountered -- skipping file deletion" just
skip the file or does it terminate the command completely?

The destination is being updated, just not pruned.

Here is somethign interesting:

ls -al /home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd
lrwxrwxrwx. 1 todd users 35 Nov 15  2019 
/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd -> 
virtio-win-0.1.171_servers__x86.vfd

"ls" does not sow the double dash showing in the link.

Trying it with the link corrected:
ls -al /home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd

lrwxrwxrwx. 1 todd users 35 Nov 15  2019 
/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd -> 
virtio-win-0.1.171_servers__x86.vfd

Oh and look-e-here:

# UpdateMyCDs
Resetting permissions
Resetting ownerships

rsync --recursive --verbose --delete --delete-excluded --modify-window=5 
--times --inplace --copy-links  --exclude='wine-*' /home/CDs/Linux 
/mnt/MyCDs/.
sending incremental file list
Linux/KVM/virtio-win/RPM/
Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd
deleting 
Linux/PDFStudio.Master-Pdf-Editor/master-pdf-editor-5.4.38-qt5-all.amd64.tar.gz
deleting Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.42-x86_64.rpm
deleting Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.29-x86_64.rpm
deleting Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.20-x86_64.rpm
deleting 
Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.09-qt5-all.x86_64.rpm
deleting 
Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.4.38-qt5.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora31-2020.01-01.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora31-2019.11-01.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora31-2019.07.1-03.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora30-2019.03.1-03.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora29-2018.11-01.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.09.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.08.2.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.07.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.07.02.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.06.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.05.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.05.1.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.02.1.07.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.02.1.05.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora31-2020.02.1.04.x86_64.rpm

sent 2,981,248 bytes  received 1,479 bytes  1,988,484.67 bytes/sec

So it was the bad link.

But the question still arises, why did the command stop
pruning after it found the bad link?  It still updated.

-T

[Cameron Simpson]

On 12Nov2020 18:55, ToddAndMargo  wrote:
> On 2020-11-12 18:07, Samuel Sieb wrote:
> > On 11/12/20 5:45 PM, ToddAndMargo via users wrote:
> > > I am trying to use rsync to
> > > copy new files on the source to the destination

I like -a for this.
> > > remove all files on the destination that do not
> > >
> > > reside on the source

--delete
> > > Problem: the extra files on the destination are
> > > not being removed.
> > >
> > > Options="--recursive --verbose --delete --delete-excluded 
> > > --modify-window=5 --times --inplace --copy-links "

Looks ok. Why --inplace? rsync has a nice make-temp-file-then-rename 
default behaviour which means the target file is the old file or the new 
file and never some partial chimera of the two.

[...]
> # UpdateMyCDs
> Resetting permissions
> Resetting ownerships
>
> rsync --recursive --verbose --delete --delete-excluded 
> --modify-window=5 --times --inplace --copy-links  --exclude='wine-*' 
> /home/CDs/Linux /mnt/MyCDs/.

_Is_ there something odd mounted are /mnt/MyCDs or below? As opposed to 
plain old Linux filesystems?
> sending incremental file list
> symlink has no referent: 
> "/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd"
> IO error encountered -- skipping file deletion

What if you leave our --copy-links? I suspect a symlink pointing at 
something which doesn't exist.

From the manual:

‐‐copy‐links, ‐L         transform symlink into referent file/dir

I imagine this involves repointing an absolute symlink to an absolute 
but adjusted symlink at the far end. Maybe what it refers to would not 
exist, causing trouble for the adjustment. I've never tried this option.  
Leave it out and see if it is the source of the problem.
> Does "IO error encountered -- skipping file deletion" just
> skip the file or does it terminate the command completely?

It omited the --delete phase because there was other trouble.

Cheers,
Cameron Simpson

[ToddAndMargo]

On 2020-11-12 20:29, Cameron Simpson wrote:
> I like -a for this.

--archive, -a  archive mode; equals -rlptgoD (no -H,-A,-X)

--recursive, -r
--links, -l
--perms, -p
--times, -t
--group, -g
--owner, -o
--devices --specials, -D

[ToddAndMargo]

On 2020-11-12 18:07, Samuel Sieb wrote:
> On 11/12/20 5:45 PM, ToddAndMargo via users wrote:
> > I am trying to use rsync to
> >
> > copy new files on the source to the destination
> >
> > remove all files on the destination that do not
> >
> > reside on the source
> >
> > Problem: the extra files on the destination are
> > not being removed.
> >
> > Options="--recursive --verbose --delete --delete-excluded 
> > --modify-window=5 --times --inplace --copy-links "
> >
> > ExcludeLinux="--exclude='wine-*'"
> >
> > Synchronize "${Options} ${ExcludeLinux} ${MyCDsSource}/Linux 
> > ${MyCDsTarget}/."
>
> That should work.
> Is there any output indicating why it's not working?
> What is an example file?

Source:
$ pwd
/home/CDs/Linux/Perl/6

$ ls
Perl6.WebSite.txt
rakudo-pkg-Fedora29-2019.03.1-01.x86_64.rpm
rakudo-pkg-Fedora30-2019.07.1-03.x86_64.rpm
RakudoPkgFedora31-2020.02.1.05.x86_64.rpm
RakudoPkgFedora32-2020.10.02.x86_64.rpm
RakudoPkgFedora33-2020.10.02.x86_64.rpm

Destination:
$ pwd
/mnt/MyCDs/Linux/Perl/6

$ ls
Perl6.WebSite.txt
rakudo-pkg-Fedora29-2018.11-01.x86_64.rpm
rakudo-pkg-Fedora29-2019.03.1-01.x86_64.rpm
rakudo-pkg-Fedora30-2019.03.1-03.x86_64.rpm
rakudo-pkg-Fedora30-2019.07.1-03.x86_64.rpm
rakudo-pkg-Fedora31-2019.07.1-03.x86_64.rpm
rakudo-pkg-Fedora31-2019.11-01.x86_64.rpm
rakudo-pkg-Fedora31-2020.01-01.x86_64.rpm
RakudoPkgFedora31-2020.02.1.04.x86_64.rpm
RakudoPkgFedora31-2020.02.1.05.x86_64.rpm
RakudoPkgFedora32-2020.02.1.05.x86_64.rpm
RakudoPkgFedora32-2020.02.1.07.x86_64.rpm
RakudoPkgFedora32-2020.05.1.x86_64.rpm
RakudoPkgFedora32-2020.05.x86_64.rpm
RakudoPkgFedora32-2020.06.x86_64.rpm
RakudoPkgFedora32-2020.07.02.x86_64.rpm
RakudoPkgFedora32-2020.07.x86_64.rpm
RakudoPkgFedora32-2020.08.2.x86_64.rpm
RakudoPkgFedora32-2020.09.x86_64.rpm
RakudoPkgFedora32-2020.10.02.x86_64.rpm
RakudoPkgFedora33-2020.10.02.x86_64.rpm

This is exiting after the call to Linux:

# UpdateMyCDs
Resetting permissions
Resetting ownerships

rsync --recursive --verbose --delete --delete-excluded --modify-window=5 
--times --inplace --copy-links  --exclude='wine-*' /home/CDs/Linux 
/mnt/MyCDs/.
sending incremental file list
symlink has no referent: 
"/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd"
IO error encountered -- skipping file deletion

sent 31,338 bytes  received 210 bytes  63,096.00 bytes/sec
total size is 3,207,298,531  speedup is 101,664.08
rsync error: some files/attrs were not transferred (see previous errors) 
(code 23) at main.c(1330) [sender=3.2.3]
-----------------------------

Does "IO error encountered -- skipping file deletion" just
skip the file or does it terminate the command completely?

The destination is being updated, just not pruned.

Here is somethign interesting:

ls -al /home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd
lrwxrwxrwx. 1 todd users 35 Nov 15  2019 
/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd -> 
virtio-win-0.1.171_servers__x86.vfd

"ls" does not sow the double dash showing in the link.

Trying it with the link corrected:
ls -al /home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd

lrwxrwxrwx. 1 todd users 35 Nov 15  2019 
/home/CDs/Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd -> 
virtio-win-0.1.171_servers__x86.vfd

Oh and look-e-here:

# UpdateMyCDs
Resetting permissions
Resetting ownerships

rsync --recursive --verbose --delete --delete-excluded --modify-window=5 
--times --inplace --copy-links  --exclude='wine-*' /home/CDs/Linux 
/mnt/MyCDs/.
sending incremental file list
Linux/KVM/virtio-win/RPM/
Linux/KVM/virtio-win/RPM/virtio-win_servers_x86.vfd
deleting 
Linux/PDFStudio.Master-Pdf-Editor/master-pdf-editor-5.4.38-qt5-all.amd64.tar.gz
deleting Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.42-x86_64.rpm
deleting Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.29-x86_64.rpm
deleting Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.20-x86_64.rpm
deleting 
Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.6.09-qt5-all.x86_64.rpm
deleting 
Linux/PDFStudio.Master-Pdf-Editor/MasterPDFEditor-5.4.38-qt5.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora31-2020.01-01.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora31-2019.11-01.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora31-2019.07.1-03.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora30-2019.03.1-03.x86_64.rpm
deleting Linux/Perl/6/rakudo-pkg-Fedora29-2018.11-01.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.09.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.08.2.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.07.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.07.02.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.06.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.05.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.05.1.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.02.1.07.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora32-2020.02.1.05.x86_64.rpm
deleting Linux/Perl/6/RakudoPkgFedora31-2020.02.1.04.x86_64.rpm

sent 2,981,248 bytes  received 1,479 bytes  1,988,484.67 bytes/sec

So it was the bad link.

But the question still arises, why did the command stop
pruning after it found the bad link?  It still updated.

-T

[Samuel Sieb]

On 11/12/20 8:36 PM, ToddAndMargo via users wrote:
> So it was the bad link.
>
> But the question still arises, why did the command stop
> pruning after it found the bad link?  It still updated.

It's being safe.  If something went wrong in the copying, it can't be 
sure that it will delete the right files.  There are many options you 
could use though.  "--delete-before" might have done it, 
"--ignore-errors" definitely would.

[ToddAndMargo]

For some reason, my follow up is getting bounced.
If you want to see it, respond to this and I will
eMail it to you directly.