rsync not backing up all files

[Robert Moskowitz]

I am getting ready for a new install to f20 (currently still lagging 
behind on f17).  I attached a 2TB USB drive, stopped MOST tasks (firefox 
with 16 open windows, a few terminals some running SSH, and VNC), and 
ran the following:

$ rsync -ah --stats ./ /run/media/rgm/6XW1484R/lx120e.htt/rgm/backup-131218

Number of files: 30174
Number of files transferred: 25251
Total file size: 34.93G bytes
Total transferred file size: 34.93G bytes
Literal data: 34.93G bytes
Matched data: 0 bytes
File list size: 776.54K
File list generation time: 0.121 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 34.94G
Total bytes received: 499.44K

sent 34.94G bytes  received 499.44K bytes  23.53M bytes/sec
total size is 34.93G  speedup is 1.00

==================================================

So the question is why were ~5k files NOT copied?  Properties on 
/home/rgm shows ~42GB disk used.

I am going to do a clean install to a new 256GB SSD drive (hopeing to 
get more inflight operating time), so backing up everything to restore 
seems important.  Unless there are that many hidden files MOST of which 
(need ./.ssh for example) are not needed.

Plus I would like rsync to report how long it actually took to run. I 
can't find an option for that.

thank you for your help.

[Robert Nichols]

On 12/18/2013 06:56 AM, Robert Moskowitz wrote:
> $ rsync -ah --stats ./ /run/media/rgm/6XW1484R/lx120e.htt/rgm/backup-131218
>
> Number of files: 30174
> Number of files transferred: 25251
> Total file size: 34.93G bytes
> Total transferred file size: 34.93G bytes
> Literal data: 34.93G bytes
> Matched data: 0 bytes
> File list size: 776.54K
> File list generation time: 0.121 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 34.94G
> Total bytes received: 499.44K
>
> sent 34.94G bytes  received 499.44K bytes  23.53M bytes/sec
> total size is 34.93G  speedup is 1.00
>
> ==================================================
>
> So the question is why were ~5k files NOT copied?  Properties on /home/rgm shows
> ~42GB disk used.

"Number of files" includes a lot of things (directories, symlinks) that are
not ordinary files.  See if "find . ! -type f | wc -l" gives you a number
that accounts for the difference.

[Robert Moskowitz]

On 12/18/2013 10:05 AM, Robert Nichols wrote:
> On 12/18/2013 06:56 AM, Robert Moskowitz wrote:
> > $ rsync -ah --stats ./ 
> > /run/media/rgm/6XW1484R/lx120e.htt/rgm/backup-131218
> >
> > Number of files: 30174
> > Number of files transferred: 25251
> > Total file size: 34.93G bytes
> > Total transferred file size: 34.93G bytes
> > Literal data: 34.93G bytes
> > Matched data: 0 bytes
> > File list size: 776.54K
> > File list generation time: 0.121 seconds
> > File list transfer time: 0.000 seconds
> > Total bytes sent: 34.94G
> > Total bytes received: 499.44K
> >
> > sent 34.94G bytes  received 499.44K bytes  23.53M bytes/sec
> > total size is 34.93G  speedup is 1.00
> >
> > ==================================================
> >
> > So the question is why were ~5k files NOT copied?  Properties on 
> > /home/rgm shows
> > ~42GB disk used.
>
> "Number of files" includes a lot of things (directories, symlinks) 
> that are
> not ordinary files.  See if "find . ! -type f | wc -l" gives you a number
> that accounts for the difference.

Reponds back with 4962 which is pretty close, given that I have done a 
few things since the backup (like restarting a few copies of Thunderbird).

Thanks for clarifying this for me. I suspected directories, but did not 
think I had THAT many! Obviously it was more than just directories. I 
will save this little find in my notes of things to check.

[Cameron Simpson]

On 18Dec2013 10:17, Robert Moskowitz  wrote:
> Thanks for clarifying this for me. I suspected directories, but did
> not think I had THAT many! Obviously it was more than just
> directories. I will save this little find in my notes of things to
> check.

Symlinks maybe.

Or hardlinks (but you're not using -H).
IIRC, the /lib/modulestree is heavily hardlinked if you keep multiple
kernels around.

Anyway, run a report on that find to count things by file type
(dirs, symlinks, devices etc) and see what you get.

Cheers,

[Patrick O'Callaghan]

On Wed, Dec 18, 2013 at 12:56 PM, Robert Moskowitz wrote:
> Plus I would like rsync to report how long it actually took to run. I
> can't find an option for that.

man time

poc