Most likely OT: rsync to cifs mount

[Ranjan Maitra]

Hi,

I am at my wits end with this, so I thought I would post here to see if anyone had any thoughts. The only connection with F21 (updated in all installed components) is that everything is on a F21 machine.

So, I have a cifs share mounted onto /mnt/space using the usual mount.cifs option with sec=ntlmv2.

I wanted to copy the following directory (kmeans) onto /mnt/space -- because I wanted to preserve permissions and structure, I tried rsync:

rsync -avSz --delete --progress kmeans/. /mnt/space/kmnsinit

So here is the problem:

The original directory has size:

% du -sh kmeans
156G	kmeans

But the copying has continued on and on with no sign of ending (and we are at 378GB).

% du -sh kmnsinit
378G kmnsinit

I also tried without the . after the kmeans/ in the rsync command (though not sure why it would matter) but to no avail. (Same story, and note that it is not ending at 378G but going on, and chances are will go on for some more.)

Is it possible that since all things Windoze are bad, the cifs puts stuff all over the place and represents large files differently. Is it possible to however make it behave?

Does anyone have an explanation and solution for the above? I apologize again for the largely OT nature of this post.

Many thanks and best wishes,
Ranjan

Most likely OT: rsync to cifs mount (additional info)

[Ranjan Maitra]

Sorry I forgot to add that there are no symlinks in the directories being rsync'ed so that is not an issue here.

Best wishes,
Ranjan

On Mon, 30 Mar 2015 11:29:07 -0500 Ranjan Maitra  wrote:
> Hi,
>
> I am at my wits end with this, so I thought I would post here to see if anyone had any thoughts. The only connection with F21 (updated in all installed components) is that everything is on a F21 machine.
>
> So, I have a cifs share mounted onto /mnt/space using the usual mount.cifs option with sec=ntlmv2.
>
> I wanted to copy the following directory (kmeans) onto /mnt/space -- because I wanted to preserve permissions and structure, I tried rsync:
>
> rsync -avSz --delete --progress kmeans/. /mnt/space/kmnsinit
>
> So here is the problem:
>
> The original directory has size:
>
> % du -sh kmeans
> 156G	kmeans
>
> But the copying has continued on and on with no sign of ending (and we are at 378GB).
>
> % du -sh kmnsinit
> 378G kmnsinit
>
> I also tried without the . after the kmeans/ in the rsync command (though not sure why it would matter) but to no avail. (Same story, and note that it is not ending at 378G but going on, and chances are will go on for some more.)
>
> Is it possible that since all things Windoze are bad, the cifs puts stuff all over the place and represents large files differently. Is it possible to however make it behave?
>
> Does anyone have an explanation and solution for the above? I apologize again for the largely OT nature of this post.
>
> Many thanks and best wishes,
> Ranjan
>
> -- 
> Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt. Please respond to the mailing list if appropriate. For those needing to send personal or professional e-mail, please use appropriate addresses.
>
> FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
> Check it out at http://www.inbox.com/earth
>
> -- 
> users mailing list
> [e-mail]
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org

[Cameron Simpson]

On 30Mar2015 11:29, Ranjan Maitra  wrote:
> I am at my wits end with this, so I thought I would post here to see if anyone had any thoughts. The only connection with F21 (updated in all installed components) is that everything is on a F21 machine.
>
> So, I have a cifs share mounted onto /mnt/space using the usual mount.cifs option with sec=ntlmv2.
>
> I wanted to copy the following directory (kmeans) onto /mnt/space -- because I wanted to preserve permissions and structure, I tried rsync:
>
> rsync -avSz --delete --progress kmeans/. /mnt/space/kmnsinit
>
> So here is the problem:
>
> The original directory has size:
>
> % du -sh kmeans
> 156G	kmeans
>
> But the copying has continued on and on with no sign of ending (and we are at 378GB).
>
> % du -sh kmnsinit
> 378G kmnsinit

If you are using a CIFS share, does that mean the far end is not a UNIX 
filesystem? The -S (sparse) option is only useful if the backend can store 
sparse files, otherwise the backend will just store lots of blocks of zeroes 
(presuming you really have sparse files).

Is the source kmeans directory full of hard links (not symlinks)? If so, rsync 
will not preserve hard links without the -H option (even with -a) and 
regardless I do not know if CIFS supports making hard links or if your backend 
supports hard links).
> I also tried without the . after the kmeans/ in the rsync command (though not sure why it would matter) but to no avail.

With rsync, a trailing / is sufficient. A trailing . can have subtle effects on 
naming (top level versus one level down) but should not affect the main problem
unless this is your second run at things, differently invoked (if so, check 
that you don't have an extra "kmeans" subdirectory at the far end, resulting in 
two copies, one at the top and one one level down).
> (Same story, and note that it is not ending at 378G but going on, and chances are will go on for some more.)

Add the --progress option. It will give you a better idea of how far through 
the file list rsync has got.
> Is it possible that since all things Windoze are bad, the cifs puts stuff all over the place and represents large files differently. Is it possible to however make it behave?

If the far end is windows, certainly sparse files will no longer be sparse at 
the far end. This kind of thing is one reason we get so picky when getting 
employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend) 
and they wanted to order Microsoft's storage thingummy, which would have broken 
all sorts of stuff just like what you're encountering.
> Does anyone have an explanation and solution for the above? I apologize again for the largely OT nature of this post.

Seems on topic to me.

Does anything above assist?

Cheers,
Cameron Simpson

"What do you want to reinstall today?"  - Bob O`Bob

[Ranjan Maitra]

Thanks, Cameron!
> If you are using a CIFS share, does that mean the far end is not a UNIX 
> filesystem? The -S (sparse) option is only useful if the backend can store 
> sparse files, otherwise the backend will just store lots of blocks of zeroes 
> (presuming you really have sparse files).

The filesystem is a "high-end" IFS filesystem according to our systems administrator. (I have no idea what an IFS filesystem is.)
> Is the source kmeans directory full of hard links (not symlinks)? If so, rsync 
> will not preserve hard links without the -H option (even with -a) and 
> regardless I do not know if CIFS supports making hard links or if your backend 
> supports hard links).

No, there are no hard links. Only a few files at the top level and about 4 directories with lots of files in them.
> If the far end is windows, certainly sparse files will no longer be sparse at 
> the far end. This kind of thing is one reason we get so picky when getting 
> employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend) 
> and they wanted to order Microsoft's storage thingummy, which would have broken 
> all sorts of stuff just like what you're encountering.

I suspect that that is what has happened here: the stuff is too high-end to be of any use.
> > Does anyone have an explanation and solution for the above? I apologize again for the largely OT nature of this post.
>
> Seems on topic to me.

Thanks!
> Does anything above assist?

Yes, it does, at least providing a possible explanation.

I don't think it would make any difference if we tar'ed the file over to the cifs share and then untarred, would it?

Many thanks again!

Best wishes,
Ranjan

____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth

[Cameron Simpson]

On 30Mar2015 16:39, Ranjan Maitra  wrote:
> Thanks, Cameron!

Any time.
> > If you are using a CIFS share, does that mean the far end is not a UNIX
> > filesystem? The -S (sparse) option is only useful if the backend can store
> > sparse files, otherwise the backend will just store lots of blocks of zeroes
> > (presuming you really have sparse files).
>
> The filesystem is a "high-end" IFS filesystem according to our systems administrator. (I have no idea what an IFS filesystem is.)

It isn't, if The Google and Wikipedia are to be believed. IFS may mean 
"Installable File System":

http://en.wikipedia.org/wiki/Installable_File_System

which is a Windows API for presenting filesystems to the OS, allowing various 
drivers for backend filesystems to be plugged in. Like the VFS layer in a Linux 
kernel I suppose.

At any rate, it says nothing of itself about what filesystem implements the 
storage, but it may place constraints on what you can do with said filesystem 
i.e. if the API does not support symlinks then you do not get symlinks and so 
forth. Under UNIX there's no specific API for sparse files - you just avoid 
writing the blocks full of NULs, instead seeking past them to the next non-NUL 
data. The filesystem remembers the gap and fakes up empty (well, full of NULs) 
blocks if you read from the gap instead of allocating (wasted) storage. The 
point of this is that _if_ the backend supports sparse files they should work 
if your file writing tool knows to behave that way (as rsync with -S does).

So your sysadmin might not know. Or might not be telling you (weird, but some 
people are like that). This page:

http://support.microsoft.com/en-us/kb/100108

suggests that M$ consider NTFS to be "high end"; at any rate that is the only 
place on the above page with the term. So: no hard links I think, no sparse 
files I think, support for symlinks in the filesystem itself but no Windows API 
for making them (IIRC).
> > Is the source kmeans directory full of hard links (not symlinks)? If so, rsync
> > will not preserve hard links without the -H option (even with -a) and
> > regardless I do not know if CIFS supports making hard links or if your backend
> > supports hard links).
>
> No, there are no hard links. Only a few files at the top level and about 4 directories with lots of files in them.

Ok.

Sparse files?

One way to check is to tally the byte sizes of the files, and compare it with 
the result of "du", which counts allocated blocks.

Tallying the file byte lengths can be done like this:

find . -type f -ls | colsum 7

"colsum" is a script of my own which makes and runs an awk script to sum 
particular columns:

https://bitbucket.org/cameron_simpson/css/src/tip/bin/colsum

Nothing very clever, but surprisingly useful to me.
> > If the far end is windows, certainly sparse files will no longer be sparse at
> > the far end. This kind of thing is one reason we get so picky when getting
> > employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> > and they wanted to order Microsoft's storage thingummy, which would have broken
> > all sorts of stuff just like what you're encountering.
>
> I suspect that that is what has happened here: the stuff is too high-end to be of any use.

Ah. Like "enterprise":-)
> > Does anything above assist?
>
> Yes, it does, at least providing a possible explanation.
> I don't think it would make any difference if we tar'ed the file over to the 
> cifs share and then untarred, would it?

If all the access is done via CIFS from your client system, I would not expect 
this to be any better.

Besides, I don't think tar does sparse files. (Or does it? GNU tar might, I 
haven't looked.)

If GNU tar will store sparse files efficiently, and you have sparse files, then 
you could profitably store the tar file on the CIFS share. I would expect 
things to go sour as soon as you untarred it, which makes accessing the 
contents tedious.

Do you have sparsae file?

As an anecdote, the other day I was working on set of web pages with mugshots 
and full images, etc. Wanting to show it to a remote colleague I thought "I'll 
use a zip file, he'll understand it". 133MB of files, 4GB of zip file. Ugh.  
Why? I was still waiting on the real image data so we'd taken a few photos 
(~20) and allocated them to the 500 odd images needed, using hard links. Zip 
does not understand hard links.

Cheers,
Cameron Simpson

(Bashir tells the story of the boy who cried "Wolf")
Bashir: If you lie all the time, no one is going to believe you, even
  when you're telling the truth.
Garak: Are you sure that's the point, Doctor?
Bashir: Of course. What else would it be?
Garak: That you should never tell the same lie twice.

[Robert Nichols]

On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
> Thanks, Cameron!
> > If the far end is windows, certainly sparse files will no longer be sparse at
> > the far end. This kind of thing is one reason we get so picky when getting
> > employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> > and they wanted to order Microsoft's storage thingummy, which would have broken
> > all sorts of stuff just like what you're encountering.
>
> I suspect that that is what has happened here: the stuff is too high-end to be of any use.

Try the "du" command again, but this time with the "--apparent-size" option.

[Ranjan Maitra]

Thanks to both Cameron and you, Bob!

After the transfer, here is what we have, on that filesystem:

$ du -sh kmeans --apparent-size
154G	kmeans

$ du -sh kmeans
628G	kmeans

So, I guess that leaves me (and others) stuck.

Ranjan

On Mon, 30 Mar 2015 21:14:38 -0500 Robert Nichols  wrote:
> On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
> > Thanks, Cameron!
> > > If the far end is windows, certainly sparse files will no longer be sparse at
> > > the far end. This kind of thing is one reason we get so picky when getting
> > > employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> > > and they wanted to order Microsoft's storage thingummy, which would have broken
> > > all sorts of stuff just like what you're encountering.
> >
> > I suspect that that is what has happened here: the stuff is too high-end to be of any use.
>
> Try the "du" command again, but this time with the "--apparent-size" option.
>
> -- 
> Bob Nichols     "NOSPAM" is really part of my email address.
>                  Do NOT delete it.
>
> -- 
> users mailing list
> [e-mail]
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org

[Ranjan Maitra]

Btw, I just found out that the filesystem in question is an Isilon filesystem.

Ranjan

On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra  wrote:
> Thanks to both Cameron and you, Bob!
>
> After the transfer, here is what we have, on that filesystem:
>
> $ du -sh kmeans --apparent-size
> 154G	kmeans
>
> $ du -sh kmeans
> 628G	kmeans
>
> So, I guess that leaves me (and others) stuck.
>
> Ranjan
>
> On Mon, 30 Mar 2015 21:14:38 -0500 Robert Nichols  wrote:
> > On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
> > > Thanks, Cameron!
> > > > If the far end is windows, certainly sparse files will no longer be sparse at
> > > > the far end. This kind of thing is one reason we get so picky when getting
> > > > employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> > > > and they wanted to order Microsoft's storage thingummy, which would have broken
> > > > all sorts of stuff just like what you're encountering.
> > >
> > > I suspect that that is what has happened here: the stuff is too high-end to be of any use.
> >
> > Try the "du" command again, but this time with the "--apparent-size" option.
> >
> > -- 
> > Bob Nichols     "NOSPAM" is really part of my email address.
> >                  Do NOT delete it.
> >
> > -- 
> > users mailing list
> > [e-mail]
> > To unsubscribe or change subscription options:
> > https://admin.fedoraproject.org/mailman/listinfo/users
> > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> > Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> > Have a question? Ask away: http://ask.fedoraproject.org
>
> -- 
> Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt. Please respond to the mailing list if appropriate. For those needing to send personal or professional e-mail, please use appropriate addresses.
>
> Can't remember your password? Do you need a strong and secure password?
> Use Password manager! It stores your passwords & protects your account.
> Check it out at http://mysecurelogon.com/manager
>
> -- 
> users mailing list
> [e-mail]
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org

[Rick Stevens]

On 04/01/2015 07:07 AM, Ranjan Maitra wrote:
> Btw, I just found out that the filesystem in question is an Isilon filesystem.

That's EMC's "OneFS" filesystem (EMC bought out Isilon).
> On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra  wrote:
> > Thanks to both Cameron and you, Bob!
> >
> > After the transfer, here is what we have, on that filesystem:
> >
> > $ du -sh kmeans --apparent-size
> > 154G	kmeans
> >
> > $ du -sh kmeans
> > 628G	kmeans
> >
> > So, I guess that leaves me (and others) stuck.

Is "kmeans" on the target or the source filesystem? If it's the source,
keep in mind that OneFS can do data dedupes (assuming it's enabled),
but it is a NAS device (NFS and/or SMB). I don't believe it's capable
of sparse files (few NAS are). The data dedupe would reduce the actual
storage on disk on the EMC device , but not report it as a sparse
filesystem.
> > On Mon, 30 Mar 2015 21:14:38 -0500 Robert Nichols  wrote:
> > > On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
> > > > Thanks, Cameron!
> > > > > If the far end is windows, certainly sparse files will no longer be sparse at
> > > > > the far end. This kind of thing is one reason we get so picky when getting
> > > > > employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> > > > > and they wanted to order Microsoft's storage thingummy, which would have broken
> > > > > all sorts of stuff just like what you're encountering.
> > > >
> > > > I suspect that that is what has happened here: the stuff is too high-end to be of any use.
> > >
> > > Try the "du" command again, but this time with the "--apparent-size" option.
> > >
> > > --
> > > Bob Nichols     "NOSPAM" is really part of my email address.
> > >                   Do NOT delete it.
> > >
> > > --
> > > users mailing list
> > > [e-mail]
> > > To unsubscribe or change subscription options:
> > > https://admin.fedoraproject.org/mailman/listinfo/users
> > > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> > > Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> > > Have a question? Ask away: http://ask.fedoraproject.org
> >
> > --
> > Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt. Please respond to the mailing list if appropriate. For those needing to send personal or professional e-mail, please use appropriate addresses.
> >
> > Can't remember your password? Do you need a strong and secure password?
> > Use Password manager! It stores your passwords & protects your account.
> > Check it out at http://mysecurelogon.com/manager
> >
> > --
> > users mailing list
> > [e-mail]
> > To unsubscribe or change subscription options:
> > https://admin.fedoraproject.org/mailman/listinfo/users
> > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> > Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> > Have a question? Ask away: http://ask.fedoraproject.org

[Ranjan Maitra]

Thanks!
> That's EMC's "OneFS" filesystem (EMC bought out Isilon).
> > On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra  wrote:
> > > Thanks to both Cameron and you, Bob!
> > >
> > > After the transfer, here is what we have, on that filesystem:
> > >
> > > $ du -sh kmeans --apparent-size
> > > 154G	kmeans
> > >
> > > $ du -sh kmeans
> > > 628G	kmeans
> > >
> > > So, I guess that leaves me (and others) stuck.
>
> Is "kmeans" on the target or the source filesystem?

Sorry, this is on the target (Isilon FS). Locally (on a F21 workstation and ext4 FS) it clocks in at 154G and 159G respectively.

If it's the source,
> keep in mind that OneFS can do data dedupes (assuming it's enabled),
> but it is a NAS device (NFS and/or SMB). I don't believe it's capable
> of sparse files (few NAS are). The data dedupe would reduce the actual
> storage on disk on the EMC device , but not report it as a sparse
> filesystem

Yes, I have been given this explanation, as well as that th block size is turned up on the isilon. This means that the size of a single file is probably 16K, rather than the typical 4K desktop file size. However, I do not have files that are that small where it would make a difference. So, I don't know.

I see: the dedupe is supposed to run over weekends but I am not sure what it does.

Best wishes,
Ranjan

____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth

[Rick Stevens]

On 04/01/2015 10:57 AM, Ranjan Maitra wrote:
> Thanks!
> > That's EMC's "OneFS" filesystem (EMC bought out Isilon).
> > > On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra  wrote:
> > > > Thanks to both Cameron and you, Bob!
> > > >
> > > > After the transfer, here is what we have, on that filesystem:
> > > >
> > > > $ du -sh kmeans --apparent-size
> > > > 154G	kmeans
> > > >
> > > > $ du -sh kmeans
> > > > 628G	kmeans
> > > >
> > > > So, I guess that leaves me (and others) stuck.
> >
> > Is "kmeans" on the target or the source filesystem?
>
> Sorry, this is on the target (Isilon FS). Locally (on a F21 workstation and ext4 FS) it clocks in at 154G and 159G respectively.
>
> If it's the source,
> > keep in mind that OneFS can do data dedupes (assuming it's enabled),
> > but it is a NAS device (NFS and/or SMB). I don't believe it's capable
> > of sparse files (few NAS are). The data dedupe would reduce the actual
> > storage on disk on the EMC device , but not report it as a sparse
> > filesystem
>
> Yes, I have been given this explanation, as well as that th block size is turned up on the isilon. This means that the size of a single file is probably 16K, rather than the typical 4K desktop file size. However, I do not have files that are that small where it would make a difference. So, I don't know.
>
> I see: the dedupe is supposed to run over weekends but I am not sure what it does.

Deduping is a process by which redundant data on a storage device is
removed. You can loosely think of it as "gzip" at the block level on
the storage device itself (although gzip is _compression_, not
deduping). Everything on the device will _appear_ normal, but the
redundancies will have been removed and less physical space used.

Here's a good explanation:

http://www.webopedia.com/TERM/D/data_deduplication.html

----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    [e-mail] -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-       A squeegee, by any other name, wouldn't sound as funny.      -
----------------------------------------------------------------------

SV: Re: Most likely OT: rsync to cifs mount

[Birger Monsen]

Deduplication kind of handles sparse files since all blocks containing only zero will get mapped to the same storage.

As soon as one of those blocks sharing storage get written to it will be written to a new block, and the usage counter of the shared block gets reduced by one. Once usage reaches zero the block is flagged for reuse. At least that is how it seems to work in the netapp wafl file system. Wafl never rewrites a block in place, it always writes to a new location. I don't know about OneFS.

Sendt fra min Sony Xperia™-smarttelefon

---- Rick Stevens skrev ----
> On 04/01/2015 10:57 AM, Ranjan Maitra wrote:
> > Thanks!
> > > That's EMC's "OneFS" filesystem (EMC bought out Isilon).
> > > > On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra  wrote:
> > > > > Thanks to both Cameron and you, Bob!
> > > > >
> > > > > After the transfer, here is what we have, on that filesystem:
> > > > >
> > > > > $ du -sh kmeans --apparent-size
> > > > > 154G	kmeans
> > > > >
> > > > > $ du -sh kmeans
> > > > > 628G	kmeans
> > > > >
> > > > > So, I guess that leaves me (and others) stuck.
> > >
> > > Is "kmeans" on the target or the source filesystem?
> >
> > Sorry, this is on the target (Isilon FS). Locally (on a F21 workstation and ext4 FS) it clocks in at 154G and 159G respectively.
> >
> > If it's the source,
> > > keep in mind that OneFS can do data dedupes (assuming it's enabled),
> > > but it is a NAS device (NFS and/or SMB). I don't believe it's capable
> > > of sparse files (few NAS are). The data dedupe would reduce the actual
> > > storage on disk on the EMC device , but not report it as a sparse
> > > filesystem
> >
> > Yes, I have been given this explanation, as well as that th block size is turned up on the isilon. This means that the size of a single file is probably 16K, rather than the typical 4K desktop file size. However, I do not have files that are that small where it would make a difference. So, I don't know.
> >
> > I see: the dedupe is supposed to run over weekends but I am not sure what it does.
>
> Deduping is a process by which redundant data on a storage device is
> removed. You can loosely think of it as "gzip" at the block level on
> the storage device itself (although gzip is _compression_, not
> deduping). Everything on the device will _appear_ normal, but the
> redundancies will have been removed and less physical space used.
>
> Here's a good explanation:
>
> http://www.webopedia.com/TERM/D/data_deduplication.html
> Rick Stevens, Systems Engineer, AllDigital    [e-mail] -
> AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
>                                                                -
>   A squeegee, by any other name, wouldn't sound as funny.      -
>
> -- 
> users mailing list
> [e-mail]
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org

[Cameron Simpson]

On 01Apr2015 08:07, Ranjan Maitra  wrote:
> Thanks to both Cameron and you, Bob!
>
> After the transfer, here is what we have, on that filesystem:
>
> $ du -sh kmeans --apparent-size
> 154G	kmeans
>
> $ du -sh kmeans
> 628G	kmeans

That looks backwards to me, presuming you have sparse files?
Although "man du" is, frankly, vague about what this means, I'd imagine from 
the description that it tallies the byte sizes of the files; du normally 
tallies "blocks" from the st_blocks stat field, which reports allocated blocks 
(which is lower than st_size/st_blksize if the file is sparse).

So how you get a smaller "du" with --apparent-size escapes me. You are duing 
the same directory, yes? Not the source and copied directories?

I'm still wondering if you do have sparse files; images generally are not.

Cheers,
Cameron Simpson

[Ranjan Maitra]

On Thu, 2 Apr 2015 15:20:52 +1100 Cameron Simpson  wrote:
> On 01Apr2015 08:07, Ranjan Maitra  wrote:
> > Thanks to both Cameron and you, Bob!
> >
> > After the transfer, here is what we have, on that filesystem:
> >
> > $ du -sh kmeans --apparent-size
> > 154G	kmeans
> >
> > $ du -sh kmeans
> > 628G	kmeans
>
> That looks backwards to me, presuming you have sparse files?
> Although "man du" is, frankly, vague about what this means, I'd imagine from 
> the description that it tallies the byte sizes of the files; du normally 
> tallies "blocks" from the st_blocks stat field, which reports allocated blocks 
> (which is lower than st_size/st_blksize if the file is sparse).
>
> So how you get a smaller "du" with --apparent-size escapes me. You are duing 
> the same directory, yes? Not the source and copied directories?

Yes, it is on the same directory of the remote Inselon FS mounted as a cifs mount on a F21 system.

The same command on my local disk (F21 machine, ext4 FS) yields:

$ du -sh kmeans --apparent-size
154G	kmeans

$ du -sh kmeans
156G	kmeans
> I'm still wondering if you do have sparse files; images generally are not.

I don't know but I assume I do. How does one find out if one does.

Ranjan

____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth

[Cameron Simpson]

On 02Apr2015 07:54, Ranjan Maitra  wrote:
> On Thu, 2 Apr 2015 15:20:52 +1100 Cameron Simpson  wrote:
> Yes, it is on the same directory of the remote Inselon FS mounted as a cifs mount on a F21 system.
>
> The same command on my local disk (F21 machine, ext4 FS) yields:
>
> $ du -sh kmeans --apparent-size
> 154G	kmeans
>
> $ du -sh kmeans
> 156G	kmeans
> > I'm still wondering if you do have sparse files; images generally are not.
>
> I don't know but I assume I do. How does one find out if one does.

Only by comparing the byte size (st_size from stat) with the blocks * blocksize 
(st_blocks * st_blksize). If the byte size well exceeds the blocks*blksize then 
the file is presumably sparse.

But images tend not to be. Sparse files tend to be either disk images (eg 
*.img) where not all blocks have been used or data files driven through a hash 
table (where data is spread out over a huge filespace in little chunks).

Cheers,
Cameron Simpson

What the hell is that?
Sounds like Cole Porter to me, sir.     - _Tank Girl_

[Gordon Messmer]

On 04/01/2015 09:20 PM, Cameron Simpson wrote:
> > $ du -sh kmeans --apparent-size
> > 154G    kmeans
> >
> > $ du -sh kmeans
> > 628G    kmeans
>
> That looks backwards to me, presuming you have sparse files?

It makes sense if there's a very large number of files of less than 2k 
each, while Isilon OneFS uses an 8k block size.  Right?

[Cameron Simpson]

On 06Apr2015 10:53, Gordon Messmer  wrote:
> On 04/01/2015 09:20 PM, Cameron Simpson wrote:
> > > $ du -sh kmeans --apparent-size
> > > 154G    kmeans
> > >
> > > $ du -sh kmeans
> > > 628G    kmeans
> >
> > That looks backwards to me, presuming you have sparse files?
>
> It makes sense if there's a very large number of files of less than 2k 
> each, while Isilon OneFS uses an 8k block size.  Right?

Yes, but Ranjan has reported:

This means that the size of a single file is probably 16K, rather
  than the typical 4K desktop file size. However, I do not have
  files that are that small where it would make a difference. So,
  I don't know.

So this may not be the issue.

Cheers,
Cameron Simpson

[Ranjan Maitra]

On Tue, 7 Apr 2015 08:10:28 +1000 Cameron Simpson  wrote:
> On 06Apr2015 10:53, Gordon Messmer  wrote:
> > On 04/01/2015 09:20 PM, Cameron Simpson wrote:
> > > > $ du -sh kmeans --apparent-size
> > > > 154G    kmeans
> > > >
> > > > $ du -sh kmeans
> > > > 628G    kmeans
> > >
> > > That looks backwards to me, presuming you have sparse files?
> >
> > It makes sense if there's a very large number of files of less than 2k 
> > each, while Isilon OneFS uses an 8k block size.  Right?
>
> Yes, but Ranjan has reported:
>
> This means that the size of a single file is probably 16K, rather
>   than the typical 4K desktop file size. However, I do not have
>   files that are that small where it would make a difference. So,
>   I don't know.

Yes, that is correct. Likely, the issue is the inability of OneFS to handle sparse files.

Best wishes,
Ranjan

____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/manager