why does this rsync commands copy again the same files?

[M. Fioretti]

Greetings,

I am running Fedora 16 x86_64. When I connect an external USB drive
and rsync to it my pictures with this command:

rsync -rvt --delete --no-o --no-g /home/z/photo/ /media/Verbatim/photo

because I don't care about ownership and permissions, I just want to
have the same files,with the same names and modification times, in the
same hierarchy of folders.

However, rsync keeps copying certain folders and pictures, and always
the same ones, every time, even if they are not changed at all. If I
rsync, unmount and unplug the drive, then plug it in again and re-run
the command above, rsync copies again hundreds of pictures to the
drive. What am I missing here?

FIY, the source folder is ext4, the destination vfat (see their mount
entries below)

TIA,
Marco

/dev/sda7 on /home type ext4 (rw,relatime,seclabel,data=ordered)
/dev/sdb1 on /media/Verbatim type vfat
(rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=cp437,i

[SOLVED]: why does this rsync commands copy again the same files?

[M. Fioretti]

On Thu, Nov 22, 2012 14:32:56 PM -0800, Gordon Messmer wrote:

>  Use --modify-window=1 
this seems enough to make the problem disappear, without mounting the
file system Harald suggested. (I prefer not to do that because
sometimes I have to do this on boxes on which I have no root password
and/or control on how to mount file systems)

Thanks a lot to Harald and Gordon!!!

While we're at this:

>  vfat's time resolution is only 2 seconds, so about half of
> original
>  files copied to vfat using rsync will have a timestamp that doesn't
>  match.  Since rsync uses size and modified time to decide whether or
>  not to copy files (unless you specify -c), 
my understanding is that -c completely avoids all these headaches, but
is not used normally ONLY because it would be much slower.  if it
isn't so, please let me know, otherwise no need to answer.

Marco

[Gordon Messmer]

On 11/22/2012 01:55 PM, M. Fioretti wrote:
>  However, I do remember that this thing of having the trailing slash
> in
>  the source, but not in the destination, is something I have read in
>  many places as the thing to do, and this is the first time I hear
>  speaking against it. 
The man page for rsync is clear on the topic:
"""
A trailing slash on the source changes this behavior to avoid  creating
an  additional  directory level at the destination.  You can think of a
trailing / on a source as meaning "copy the contents of this directory"
as  opposed  to  "copy  the  directory  by name"
"""
Putting a trailing slash on both directories will produce exactly the 
same behavior as not putting a trailing slash on the destination. 
Suggestions otherwise are mistaken.  Putting a slash on both is very 
common, since it relieves users of needing to understand the more 
specific details of this behavior.

>  Back on topic, why do you think there may be some link between this
>  issue and the problem I see? My problem seems more due to some wrong
>  option, because a) it is the only problem I see, never had any filled
>  partition, and b) it is very, very constant. Sorry if I forgot to
>  specify this earlier, but if I run the command on two different
>  drives, both vfat, both times the folder(s) that are recopied are the
>  same. 
Use --modify-window=1

vfat's time resolution is only 2 seconds, so about half of original 
files copied to vfat using rsync will have a timestamp that doesn't 
match.  Since rsync uses size and modified time to decide whether or not 
to copy files (unless you specify -c), about half of your files will be 
rsynced every time you copy from ext4 to vfat.

[Reindl Harald]

Am 22.11.2012 23:21, schrieb M. Fioretti:
>  On Thu, Nov 22, 2012 23:07:33 PM +0100, Reindl Harald wrote:
>  
> > as said: please compare the timestamp on the target most likely the
> > timestamp on vfat contains daylight-savings which is not the case on
> > unix-filesystems
>  
>  Indeed! Thanks. Here are the outputs of fstat on two copies of the
>  same file, just rsync-ed from the ext4 folder to the vfat ext drive:
>  
>  [marco@avalon LACIE]$ stat  --format "%y"
> /home/z/photo/2011/20110719/P1110546.JPG
>  2011-07-19 07:28:36.000000000 +0200
>  [marco@avalon LACIE]$ stat  --format "%y"
> /media/LACIE/photo/2011/20110719/P1110546.JPG
>  2011-07-19 06:28:36.000000000 +0200
>  
>  OK, thanks you got it exactly right!!! But, now what? Is there any way
>  to keep the drive vfat AND end this problem, without losing file
>  modification time? I'd really like to keep the vfat simply because some of
>  those drives are not mine  
google says:

http://lwn.net/Articles/287428/
http://www.kernel.org/doc/Documentation/filesystems/vfat.txt

tz=UTC        -- Interpret timestamps as UTC rather than local time.
                 This option disables the conversion of timestamps
                 between local time (as used by Windows on FAT) and UTC
                 (which Linux uses internally).  This is particularly
                 useful when mounting devices (like digital cameras)
                 that are set to UTC in order to avoid the pitfalls of
                 local time.

so i guess mount the crappy vfat with the mount-param
tz=UTC should do the job

[M. Fioretti]

On Thu, Nov 22, 2012 23:07:33 PM +0100, Reindl Harald wrote:

>  as said: please compare the timestamp on the target most likely the
>  timestamp on vfat contains daylight-savings which is not the case on
>  unix-filesystems 
Indeed! Thanks. Here are the outputs of fstat on two copies of the
same file, just rsync-ed from the ext4 folder to the vfat ext drive:

[marco@avalon LACIE]$ stat  --format "%y"
/home/z/photo/2011/20110719/P1110546.JPG
2011-07-19 07:28:36.000000000 +0200
[marco@avalon LACIE]$ stat  --format "%y"
/media/LACIE/photo/2011/20110719/P1110546.JPG
2011-07-19 06:28:36.000000000 +0200

OK, thanks you got it exactly right!!! But, now what? Is there any way
to keep the drive vfat AND end this problem, without losing file
modification time? I'd really like to keep the vfat simply because some of
those drives are not mine (*)


Marco

(*) I do photo archiving for the extended family, and telling some of
them to "mess with their computers" to make them read decent file
systems would be... OK, pls tell me if there's another way first...

[Reindl Harald]

Am 22.11.2012 22:55, schrieb M. Fioretti:
>  On Thu, Nov 22, 2012 22:44:25 PM +0100, Reindl Harald wrote:
>  
> >> IIRC, adding a trailing slash /media/Verbatim/photo/ would cause
> >> another "photo" folder to appear inside /media/Verbatim/photo/,
> >> which is not what I want
> >
> > i am using rsync since many years in any environment
> > for backups, for sync folders, for sync TB's of data all over the world
> >
> > adding a trailing slash /media/Verbatim/photo/ would cause
> > NEVER EVER another "photo" folder to appear inside /media/Verbatim/photo/
>  
>  I am sure you have more experience than me. I said "IIRC". If my
>  specific explanation above, which I plucked out from memory in a rush,
>  is the wrong one, OK, no problem. 
i did not feel offended - really!

>  However, I do remember that this thing of having the trailing slash
> in
>  the source, but not in the destination, is something I have read in
>  many places as the thing to do, and this is the first time I hear
>  speaking against it 
my intention was to give the hint that a mssing trailing
slash may some times produce unexpected results while have
both sides with a trailing slash is 100% sure

>  I'm not "attacking" you in any way, of course. Just
> saying that I am
>  puzzled, because I had never heard of such an issue before. 
as said: i did not feel offendned nor was my intention to offend you

>  Back on topic, why do you think there may be some link between this
>  issue and the problem I see? 
hmm - i did not
look at my first reply
the hint with the lsahses was one line and a "yb the way"

>  So it is either something weird in the permissions or other metadata
>  of those folders only, or some wrong option in the command. But what? 
for sure vfat is the problem

as said: please compare the timestamp on the target
most likely the timestamp on vfat contains daylight-savings
which is not the case on unix-filesystems

but it is long ago that i had to deal with anything not ext4

[M. Fioretti]

On Thu, Nov 22, 2012 22:44:25 PM +0100, Reindl Harald wrote:

>  > IIRC, adding a trailing slash /media/Verbatim/photo/ would
> cause
>  > another "photo" folder to appear inside /media/Verbatim/photo/,
>  > which is not what I want
>  
>  i am using rsync since many years in any environment
>  for backups, for sync folders, for sync TB's of data all over the world
>  
>  adding a trailing slash /media/Verbatim/photo/ would cause
>  NEVER EVER another "photo" folder to appear inside /media/Verbatim/photo/
Harald,

I am sure you have more experience than me. I said "IIRC". If my
specific explanation above, which I plucked out from memory in a rush,
is the wrong one, OK, no problem.

However, I do remember that this thing of having the trailing slash in
the source, but not in the destination, is something I have read in
many places as the thing to do, and this is the first time I hear
speaking against it.

I'm not "attacking" you in any way, of course. Just saying that I am
puzzled, because I had never heard of such an issue before.

Back on topic, why do you think there may be some link between this
issue and the problem I see? My problem seems more due to some wrong
option, because a) it is the only problem I see, never had any filled
partition, and b) it is very, very constant. Sorry if I forgot to
specify this earlier, but if I run the command on two different
drives, both vfat, both times the folder(s) that are recopied are the
same.

So it is either something weird in the permissions or other metadata
of those folders only, or some wrong option in the command. But what?

TIA,
Marco

[Reindl Harald]

Am 22.11.2012 22:44, schrieb Reindl Harald:
>  
>  
>  Am 22.11.2012 22:34, schrieb M. Fioretti:
> > On Thu, Nov 22, 2012 22:10:04 PM +0100, Reindl Harald wrote:
> >
> >>>>> rsync -rvt --delete --no-o --no-g /home/z/photo/
> /media/Verbatim/photo
> >>>>
> >>>> this command has a error
> >>>> the last aparm is missing a trailing slash
> >>>
> >>> Why would that be an error?
> >>
> >> because i had enough cases with rsync over different partitions
> >> where the missing trailing slash resulted in copy the whole data to
> >> the source partition and fill it up
> >
> > omitting the trailing slash is the way explained by many tutorials and
> > by the rsync documentation itself if what one wants to do is, as in my
> > case, to copy the whole content of /home/z/photo/ inside
> > /media/Verbatim/photo
> >
> > IIRC, adding a trailing slash /media/Verbatim/photo/ would cause
> > another "photo" folder to appear inside /media/Verbatim/photo/, which
> > is not what I want
>  
>  i am using rsync since many years in any environment
>  for backups, for sync folders, for sync TB's of data all over the world
>  
>  adding a trailing slash /media/Verbatim/photo/ would cause
>  NEVER EVER another "photo" folder to appear inside /media/Verbatim/photo/
http://defindit.com/readme_files/rsync_backup.html

if you are using /source/ /target/ you do not need
to care about any destination folder - /target/
will be a 100% identical copy of the source

[Reindl Harald]

Am 22.11.2012 22:34, schrieb M. Fioretti:
>  On Thu, Nov 22, 2012 22:10:04 PM +0100, Reindl Harald wrote:
>  
> >>>> rsync -rvt --delete --no-o --no-g /home/z/photo/ /media/Verbatim/photo
> >>>
> >>> this command has a error
> >>> the last aparm is missing a trailing slash
> >>
> >> Why would that be an error?
> >
> > because i had enough cases with rsync over different partitions
> > where the missing trailing slash resulted in copy the whole data to
> > the source partition and fill it up
>  
>  omitting the trailing slash is the way explained by many tutorials and
>  by the rsync documentation itself if what one wants to do is, as in my
>  case, to copy the whole content of /home/z/photo/ inside
>  /media/Verbatim/photo
>  
>  IIRC, adding a trailing slash /media/Verbatim/photo/ would cause
>  another "photo" folder to appear inside /media/Verbatim/photo/, which
>  is not what I want 
i am using rsync since many years in any environment
for backups, for sync folders, for sync TB's of data all over the world

adding a trailing slash /media/Verbatim/photo/ would cause
NEVER EVER another "photo" folder to appear inside /media/Verbatim/photo/

[M. Fioretti]

On Thu, Nov 22, 2012 22:10:04 PM +0100, Reindl Harald wrote:

>  >>> rsync -rvt --delete --no-o --no-g /home/z/photo/
> /media/Verbatim/photo
>  >>
>  >> this command has a error
>  >> the last aparm is missing a trailing slash
>  > 
>  > Why would that be an error?
>  
>  because i had enough cases with rsync over different partitions
>  where the missing trailing slash resulted in copy the whole data to
>  the source partition and fill it up 
omitting the trailing slash is the way explained by many tutorials and
by the rsync documentation itself if what one wants to do is, as in my
case, to copy the whole content of /home/z/photo/ inside
/media/Verbatim/photo

IIRC, adding a trailing slash /media/Verbatim/photo/ would cause
another "photo" folder to appear inside /media/Verbatim/photo/, which
is not what I want.

FWIW, I have never seen errors like the ones you mention. Only this
"copy again". The only case in which it does NOT happen is if I run
rsync, and then run it again immediately after. If I unmount the drive
between the two runs, it does the unneeded copies again (and this only
happens for SOME, always the same, files/folders). But in no case I
have seen data loss, partition filled and so on

Marco

[Reindl Harald]

Am 22.11.2012 22:02, schrieb Andras Simon:
>  2012/11/22, Reindl Harald <h.reindl(a)thelounge.net&gt;:
> >
> >
> > Am 22.11.2012 21:41, schrieb M. Fioretti:
> >> Greetings,
> >>
> >> I am running Fedora 16 x86_64. When I connect an external USB drive
> >> and rsync to it my pictures with this command:
> >>
> >> rsync -rvt --delete --no-o --no-g /home/z/photo/ /media/Verbatim/photo
> >
> > this command has a error
> > the last aparm is missing a trailing slash
>  
>  Why would that be an error? 
because i had enough cases with rsync over different partitions
where the missing trailing slash resulted in copy the whole data
to the source partition and fill it up

i did not research the exatc reason because it is and was
enough for me to realize that this NEVER happens if using
a trailing slash EVERYTIME when folders are used

[Reindl Harald]

Am 22.11.2012 21:41, schrieb M. Fioretti:
>  Greetings,
>  
>  I am running Fedora 16 x86_64. When I connect an external USB drive
>  and rsync to it my pictures with this command:
>  
>  rsync -rvt --delete --no-o --no-g /home/z/photo/ /media/Verbatim/photo 
this command has a error
the last aparm is missing a trailing slash

>  FIY, the source folder is ext4, the destination vfat (see their
> mount
>  entries below) 
maybe because the target is vfat

have you compared the timestamps on tke target?
maybe try --update

maybe there are also chars which are not represented properly
on vfat in your filenames

if the filenames are 100% identical rsync will not copy the whole
data in any case, but with USB read-peroformance for verify is
ugly - so look at rsync --help and ignore as many things as
possible with crappy destination filesystems

[Andras Simon]

2012/11/22, Reindl Harald <h.reindl(a)thelounge.net&gt;:
>  Am 22.11.2012 21:41, schrieb M. Fioretti:
> > Greetings,
> >
> > I am running Fedora 16 x86_64. When I connect an external USB drive
> > and rsync to it my pictures with this command:
> >
> > rsync -rvt --delete --no-o --no-g /home/z/photo/ /media/Verbatim/photo
>
>  this command has a error
>  the last aparm is missing a trailing slash 
Why would that be an error?

Andras