11:29 a.m.
Hi,
I am at my wits end with this, so I thought I would post here to see if anyone had any
thoughts. The only connection with F21 (updated in all installed components) is that
everything is on a F21 machine.
So, I have a cifs share mounted onto /mnt/space using the usual mount.cifs option with
sec=ntlmv2.
I wanted to copy the following directory (kmeans) onto /mnt/space -- because I wanted to
preserve permissions and structure, I tried rsync:
rsync -avSz --delete --progress kmeans/. /mnt/space/kmnsinit
So here is the problem:
The original directory has size:
% du -sh kmeans
156G kmeans
But the copying has continued on and on with no sign of ending (and we are at 378GB).
% du -sh kmnsinit
378G kmnsinit
I also tried without the . after the kmeans/ in the rsync command (though not sure why it
would matter) but to no avail. (Same story, and note that it is not ending at 378G but
going on, and chances are will go on for some more.)
Is it possible that since all things Windoze are bad, the cifs puts stuff all over the
place and represents large files differently. Is it possible to however make it behave?
Does anyone have an explanation and solution for the above? I apologize again for the
largely OT nature of this post.
Many thanks and best wishes,
Ranjan
--
Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth
5:22 p.m.
On Tue, 7 Apr 2015 08:10:28 +1000 Cameron Simpson <cs(a)zip.com.au> wrote:
On 06Apr2015 10:53, Gordon Messmer <gordon.messmer(a)gmail.com>
wrote:
>On 04/01/2015 09:20 PM, Cameron Simpson wrote:
>>>$ du -sh kmeans --apparent-size
>>>154G kmeans
>>>
>>>$ du -sh kmeans
>>>628G kmeans
>>
>>That looks backwards to me, presuming you have sparse files?
>
>It makes sense if there's a very large number of files of less than 2k
>each, while Isilon OneFS uses an 8k block size. Right?
Yes, but Ranjan has reported:
This means that the size of a single file is probably 16K, rather
than the typical 4K desktop file size. However, I do not have
files that are that small where it would make a difference. So,
I don't know.
Yes, that is correct. Likely, the issue is the inability of OneFS to handle sparse files.
Best wishes,
Ranjan
____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/manager
5:10 p.m.
On 06Apr2015 10:53, Gordon Messmer <gordon.messmer(a)gmail.com> wrote:
On 04/01/2015 09:20 PM, Cameron Simpson wrote:
>>$ du -sh kmeans --apparent-size
>>154G kmeans
>>
>>$ du -sh kmeans
>>628G kmeans
>
>That looks backwards to me, presuming you have sparse files?
It makes sense if there's a very large number of files of less than 2k
each, while Isilon OneFS uses an 8k block size. Right?
Yes, but Ranjan has reported:
This means that the size of a single file is probably 16K, rather
than the typical 4K desktop file size. However, I do not have
files that are that small where it would make a difference. So,
I don't know.
So this may not be the issue.
Cheers,
Cameron Simpson <cs(a)zip.com.au>
12:53 p.m.
On 04/01/2015 09:20 PM, Cameron Simpson wrote:
> $ du -sh kmeans --apparent-size
> 154G kmeans
>
> $ du -sh kmeans
> 628G kmeans
That looks backwards to me, presuming you have sparse files?
It makes sense if there's a very large number of files of less than 2k
each, while Isilon OneFS uses an 8k block size. Right?
7:54 a.m.
On Thu, 2 Apr 2015 15:20:52 +1100 Cameron Simpson <cs(a)zip.com.au> wrote:
On 01Apr2015 08:07, Ranjan Maitra
<maitra.mbox.ignored(a)inbox.com> wrote:
>Thanks to both Cameron and you, Bob!
>
>After the transfer, here is what we have, on that filesystem:
>
>$ du -sh kmeans --apparent-size
>154G kmeans
>
>$ du -sh kmeans
>628G kmeans
That looks backwards to me, presuming you have sparse files?
Although "man du" is, frankly, vague about what this means, I'd imagine
from
the description that it tallies the byte sizes of the files; du normally
tallies "blocks" from the st_blocks stat field, which reports allocated blocks
(which is lower than st_size/st_blksize if the file is sparse).
So how you get a smaller "du" with --apparent-size escapes me. You are duing
the same directory, yes? Not the source and copied directories?
Yes, it is on the same directory of the remote Inselon FS mounted as a cifs mount on a F21
system.
The same command on my local disk (F21 machine, ext4 FS) yields:
$ du -sh kmeans --apparent-size
154G kmeans
$ du -sh kmeans
156G kmeans
I'm still wondering if you do have sparse files; images generally
are not.
I don't know but I assume I do. How does one find out if one does.
Ranjan
____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth
5:47 p.m.
On 02Apr2015 07:54, Ranjan Maitra <maitra.mbox.ignored(a)inbox.com> wrote:
On Thu, 2 Apr 2015 15:20:52 +1100 Cameron Simpson
<cs(a)zip.com.au> wrote:
Yes, it is on the same directory of the remote Inselon FS mounted as a cifs mount on a F21
system.
The same command on my local disk (F21 machine, ext4 FS) yields:
$ du -sh kmeans --apparent-size
154G kmeans
$ du -sh kmeans
156G kmeans
> I'm still wondering if you do have sparse files; images generally are not.
I don't know but I assume I do. How does one find out if one does.
Only by comparing the byte size (st_size from stat) with the blocks * blocksize
(st_blocks * st_blksize). If the byte size well exceeds the blocks*blksize then
the file is presumably sparse.
But images tend not to be. Sparse files tend to be either disk images (eg
*.img) where not all blocks have been used or data files driven through a hash
table (where data is spread out over a huge filespace in little chunks).
Cheers,
Cameron Simpson <cs(a)zip.com.au>
What the hell is that?
Sounds like Cole Porter to me, sir. - _Tank Girl_
noon
New subject: Most likely OT: rsync to cifs mount (additional info)
Sorry I forgot to add that there are no symlinks in the directories being rsync'ed so
that is not an issue here.
Best wishes,
Ranjan
On Mon, 30 Mar 2015 11:29:07 -0500 Ranjan Maitra <maitra.mbox.ignored(a)inbox.com>
wrote:
Hi,
I am at my wits end with this, so I thought I would post here to see if anyone had any
thoughts. The only connection with F21 (updated in all installed components) is that
everything is on a F21 machine.
So, I have a cifs share mounted onto /mnt/space using the usual mount.cifs option with
sec=ntlmv2.
I wanted to copy the following directory (kmeans) onto /mnt/space -- because I wanted to
preserve permissions and structure, I tried rsync:
rsync -avSz --delete --progress kmeans/. /mnt/space/kmnsinit
So here is the problem:
The original directory has size:
% du -sh kmeans
156G kmeans
But the copying has continued on and on with no sign of ending (and we are at 378GB).
% du -sh kmnsinit
378G kmnsinit
I also tried without the . after the kmeans/ in the rsync command (though not sure why it
would matter) but to no avail. (Same story, and note that it is not ending at 378G but
going on, and chances are will go on for some more.)
Is it possible that since all things Windoze are bad, the cifs puts stuff all over the
place and represents large files differently. Is it possible to however make it behave?
Does anyone have an explanation and solution for the above? I apologize again for the
largely OT nature of this post.
Many thanks and best wishes,
Ranjan
--
Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth
--
users mailing list
users(a)lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
--
Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/manager
9:14 p.m.
On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
Thanks, Cameron!
> If the far end is windows, certainly sparse files will no longer be sparse at
> the far end. This kind of thing is one reason we get so picky when getting
> employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> and they wanted to order Microsoft's storage thingummy, which would have broken
> all sorts of stuff just like what you're encountering.
I suspect that that is what has happened here: the stuff is too high-end to be of any
use.
Try the "du" command again, but this time with the "--apparent-size"
option.
--
Bob Nichols "NOSPAM" is really part of my email address.
Do NOT delete it.
4:39 p.m.
Thanks, Cameron!
If you are using a CIFS share, does that mean the far end is not a
UNIX
filesystem? The -S (sparse) option is only useful if the backend can store
sparse files, otherwise the backend will just store lots of blocks of zeroes
(presuming you really have sparse files).
The filesystem is a "high-end" IFS filesystem according to our systems
administrator. (I have no idea what an IFS filesystem is.)
Is the source kmeans directory full of hard links (not symlinks)? If so, rsync
will not preserve hard links without the -H option (even with -a) and
regardless I do not know if CIFS supports making hard links or if your backend
supports hard links).
No, there are no hard links. Only a few files at the top level and about 4 directories
with lots of files in them.
If the far end is windows, certainly sparse files will no longer be
sparse at
the far end. This kind of thing is one reason we get so picky when getting
employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
and they wanted to order Microsoft's storage thingummy, which would have broken
all sorts of stuff just like what you're encountering.
I suspect that that is what has happened here: the stuff is too high-end to be of any
use.
>Does anyone have an explanation and solution for the above? I
apologize again for the largely OT nature of this post.
Seems on topic to me.
Thanks!
Does anything above assist?
Yes, it does, at least providing a possible explanation.
I don't think it would make any difference if we tar'ed the file over to the cifs
share and then untarred, would it?
Many thanks again!
Best wishes,
Ranjan
____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth
3:56 p.m.
On 30Mar2015 11:29, Ranjan Maitra <maitra.mbox.ignored(a)inbox.com> wrote:
I am at my wits end with this, so I thought I would post here to see
if anyone had any thoughts. The only connection with F21 (updated in all installed
components) is that everything is on a F21 machine.
So, I have a cifs share mounted onto /mnt/space using the usual mount.cifs option with
sec=ntlmv2.
I wanted to copy the following directory (kmeans) onto /mnt/space -- because I wanted to
preserve permissions and structure, I tried rsync:
rsync -avSz --delete --progress kmeans/. /mnt/space/kmnsinit
So here is the problem:
The original directory has size:
% du -sh kmeans
156G kmeans
But the copying has continued on and on with no sign of ending (and we are at 378GB).
% du -sh kmnsinit
378G kmnsinit
If you are using a CIFS share, does that mean the far end is not a UNIX
filesystem? The -S (sparse) option is only useful if the backend can store
sparse files, otherwise the backend will just store lots of blocks of zeroes
(presuming you really have sparse files).
Is the source kmeans directory full of hard links (not symlinks)? If so, rsync
will not preserve hard links without the -H option (even with -a) and
regardless I do not know if CIFS supports making hard links or if your backend
supports hard links).
I also tried without the . after the kmeans/ in the rsync command
(though not sure why it would matter) but to no avail.
With rsync, a trailing / is sufficient. A trailing . can have subtle effects on
naming (top level versus one level down) but should not affect the main problem
unless this is your second run at things, differently invoked (if so, check
that you don't have an extra "kmeans" subdirectory at the far end, resulting
in
two copies, one at the top and one one level down).
(Same story, and note that it is not ending at 378G but going on, and
chances are will go on for some more.)
Add the --progress option. It will give you a better idea of how far through
the file list rsync has got.
Is it possible that since all things Windoze are bad, the cifs puts
stuff all over the place and represents large files differently. Is it possible to however
make it behave?
If the far end is windows, certainly sparse files will no longer be sparse at
the far end. This kind of thing is one reason we get so picky when getting
employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
and they wanted to order Microsoft's storage thingummy, which would have broken
all sorts of stuff just like what you're encountering.
Does anyone have an explanation and solution for the above? I
apologize again for the largely OT nature of this post.
Seems on topic to me.
Does anything above assist?
Cheers,
Cameron Simpson <cs(a)zip.com.au>
"What do you want to reinstall today?" - Bob O`Bob
<obob(a)shell3.ba.best.com>
11:20 p.m.
On 01Apr2015 08:07, Ranjan Maitra <maitra.mbox.ignored(a)inbox.com> wrote:
Thanks to both Cameron and you, Bob!
After the transfer, here is what we have, on that filesystem:
$ du -sh kmeans --apparent-size
154G kmeans
$ du -sh kmeans
628G kmeans
That looks backwards to me, presuming you have sparse files?
Although "man du" is, frankly, vague about what this means, I'd imagine from
the description that it tallies the byte sizes of the files; du normally
tallies "blocks" from the st_blocks stat field, which reports allocated blocks
(which is lower than st_size/st_blksize if the file is sparse).
So how you get a smaller "du" with --apparent-size escapes me. You are duing
the same directory, yes? Not the source and copied directories?
I'm still wondering if you do have sparse files; images generally are not.
Cheers,
Cameron Simpson <cs(a)zip.com.au>
2:10 p.m.
New subject: SV: Re: Most likely OT: rsync to cifs mount
Deduplication kind of handles sparse files since all blocks containing only zero will get
mapped to the same storage.
As soon as one of those blocks sharing storage get written to it will be written to a new
block, and the usage counter of the shared block gets reduced by one. Once usage reaches
zero the block is flagged for reuse. At least that is how it seems to work in the netapp
wafl file system. Wafl never rewrites a block in place, it always writes to a new
location. I don't know about OneFS.
Sendt fra min Sony Xperia™-smarttelefon
---- Rick Stevens skrev ----
On 04/01/2015 10:57 AM, Ranjan Maitra wrote:
> Thanks!
>
>
>> That's EMC's "OneFS" filesystem (EMC bought out Isilon).
>>
>>> On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra
<maitra.mbox.ignored(a)inbox.com> wrote:
>>>
>>>> Thanks to both Cameron and you, Bob!
>>>>
>>>> After the transfer, here is what we have, on that filesystem:
>>>>
>>>> $ du -sh kmeans --apparent-size
>>>> 154G kmeans
>>>>
>>>> $ du -sh kmeans
>>>> 628G kmeans
>>>>
>>>> So, I guess that leaves me (and others) stuck.
>>
>> Is "kmeans" on the target or the source filesystem?
>
> Sorry, this is on the target (Isilon FS). Locally (on a F21 workstation and ext4 FS)
it clocks in at 154G and 159G respectively.
>
> If it's the source,
>> keep in mind that OneFS can do data dedupes (assuming it's enabled),
>> but it is a NAS device (NFS and/or SMB). I don't believe it's capable
>> of sparse files (few NAS are). The data dedupe would reduce the actual
>> storage on disk on the EMC device , but not report it as a sparse
>> filesystem
>
>
> Yes, I have been given this explanation, as well as that th block size is turned up
on the isilon. This means that the size of a single file is probably 16K, rather than the
typical 4K desktop file size. However, I do not have files that are that small where it
would make a difference. So, I don't know.
>
> I see: the dedupe is supposed to run over weekends but I am not sure what it does.
Deduping is a process by which redundant data on a storage device is
removed. You can loosely think of it as "gzip" at the block level on
the storage device itself (although gzip is _compression_, not
deduping). Everything on the device will _appear_ normal, but the
redundancies will have been removed and less physical space used.
Here's a good explanation:
http://www.webopedia.com/TERM/D/data_deduplication.html
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ricks(a)alldigital.com -
- AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 -
- -
- A squeegee, by any other name, wouldn't sound as funny. -
----------------------------------------------------------------------
--
users mailing list
users(a)lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
1:45 p.m.
On 04/01/2015 10:57 AM, Ranjan Maitra wrote:
Thanks!
> That's EMC's "OneFS" filesystem (EMC bought out Isilon).
>
>> On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra
<maitra.mbox.ignored(a)inbox.com> wrote:
>>
>>> Thanks to both Cameron and you, Bob!
>>>
>>> After the transfer, here is what we have, on that filesystem:
>>>
>>> $ du -sh kmeans --apparent-size
>>> 154G kmeans
>>>
>>> $ du -sh kmeans
>>> 628G kmeans
>>>
>>> So, I guess that leaves me (and others) stuck.
>
> Is "kmeans" on the target or the source filesystem?
Sorry, this is on the target (Isilon FS). Locally (on a F21 workstation and ext4 FS) it
clocks in at 154G and 159G respectively.
If it's the source,
> keep in mind that OneFS can do data dedupes (assuming it's enabled),
> but it is a NAS device (NFS and/or SMB). I don't believe it's capable
> of sparse files (few NAS are). The data dedupe would reduce the actual
> storage on disk on the EMC device , but not report it as a sparse
> filesystem
Yes, I have been given this explanation, as well as that th block size is turned up on
the isilon. This means that the size of a single file is probably 16K, rather than the
typical 4K desktop file size. However, I do not have files that are that small where it
would make a difference. So, I don't know.
I see: the dedupe is supposed to run over weekends but I am not sure what it does.
Deduping is a process by which redundant data on a storage device is
removed. You can loosely think of it as "gzip" at the block level on
the storage device itself (although gzip is _compression_, not
deduping). Everything on the device will _appear_ normal, but the
redundancies will have been removed and less physical space used.
Here's a good explanation:
http://www.webopedia.com/TERM/D/data_deduplication.html
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ricks(a)alldigital.com -
- AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 -
- -
- A squeegee, by any other name, wouldn't sound as funny. -
----------------------------------------------------------------------
12:57 p.m.
Thanks!
That's EMC's "OneFS" filesystem (EMC bought out
Isilon).
> On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra
<maitra.mbox.ignored(a)inbox.com> wrote:
>
>> Thanks to both Cameron and you, Bob!
>>
>> After the transfer, here is what we have, on that filesystem:
>>
>> $ du -sh kmeans --apparent-size
>> 154G kmeans
>>
>> $ du -sh kmeans
>> 628G kmeans
>>
>> So, I guess that leaves me (and others) stuck.
Is "kmeans" on the target or the source filesystem?
Sorry, this is on the target (Isilon FS). Locally (on a F21 workstation and ext4 FS) it
clocks in at 154G and 159G respectively.
If it's the source,
keep in mind that OneFS can do data dedupes (assuming it's
enabled),
but it is a NAS device (NFS and/or SMB). I don't believe it's capable
of sparse files (few NAS are). The data dedupe would reduce the actual
storage on disk on the EMC device , but not report it as a sparse
filesystem
Yes, I have been given this explanation, as well as that th block size is turned up on the
isilon. This means that the size of a single file is probably 16K, rather than the typical
4K desktop file size. However, I do not have files that are that small where it would make
a difference. So, I don't know.
I see: the dedupe is supposed to run over weekends but I am not sure what it does.
Best wishes,
Ranjan
____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth
12:21 p.m.
On 04/01/2015 07:07 AM, Ranjan Maitra wrote:
Btw, I just found out that the filesystem in question is an Isilon
filesystem.
That's EMC's "OneFS" filesystem (EMC bought out Isilon).
On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra
<maitra.mbox.ignored(a)inbox.com> wrote:
> Thanks to both Cameron and you, Bob!
>
> After the transfer, here is what we have, on that filesystem:
>
> $ du -sh kmeans --apparent-size
> 154G kmeans
>
> $ du -sh kmeans
> 628G kmeans
>
> So, I guess that leaves me (and others) stuck.
Is "kmeans" on the target or the source filesystem? If it's the source,
keep in mind that OneFS can do data dedupes (assuming it's enabled),
but it is a NAS device (NFS and/or SMB). I don't believe it's capable
of sparse files (few NAS are). The data dedupe would reduce the actual
storage on disk on the EMC device , but not report it as a sparse
filesystem.
> On Mon, 30 Mar 2015 21:14:38 -0500 Robert Nichols
<rnicholsNOSPAM(a)comcast.net> wrote:
>
>> On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
>>> Thanks, Cameron!
>>>> If the far end is windows, certainly sparse files will no longer be
sparse at
>>>> the far end. This kind of thing is one reason we get so picky when
getting
>>>> employers to order NASes; we asked for a QNAP (cheap, useful, Linux
backend)
>>>> and they wanted to order Microsoft's storage thingummy, which would
have broken
>>>> all sorts of stuff just like what you're encountering.
>>>
>>> I suspect that that is what has happened here: the stuff is too high-end to
be of any use.
>>
>> Try the "du" command again, but this time with the
"--apparent-size" option.
>>
>> --
>> Bob Nichols "NOSPAM" is really part of my email address.
>> Do NOT delete it.
>>
>> --
>> users mailing list
>> users(a)lists.fedoraproject.org
>> To unsubscribe or change subscription options:
>> https://admin.fedoraproject.org/mailman/listinfo/users
>> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
>> Have a question? Ask away: http://ask.fedoraproject.org
>
>
> --
> Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
>
> ____________________________________________________________
> Can't remember your password? Do you need a strong and secure password?
> Use Password manager! It stores your passwords & protects your account.
> Check it out at http://mysecurelogon.com/manager
>
>
> --
> users mailing list
> users(a)lists.fedoraproject.org
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org
--
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ricks(a)alldigital.com -
- AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 -
- -
- Diplomacy: The art of saying "Nice doggy!" until you can find a -
- big enough rock. -
----------------------------------------------------------------------
8:07 a.m.
Thanks to both Cameron and you, Bob!
After the transfer, here is what we have, on that filesystem:
$ du -sh kmeans --apparent-size
154G kmeans
$ du -sh kmeans
628G kmeans
So, I guess that leaves me (and others) stuck.
Ranjan
On Mon, 30 Mar 2015 21:14:38 -0500 Robert Nichols <rnicholsNOSPAM(a)comcast.net>
wrote:
On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
> Thanks, Cameron!
>> If the far end is windows, certainly sparse files will no longer be sparse at
>> the far end. This kind of thing is one reason we get so picky when getting
>> employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
>> and they wanted to order Microsoft's storage thingummy, which would have
broken
>> all sorts of stuff just like what you're encountering.
>
> I suspect that that is what has happened here: the stuff is too high-end to be of
any use.
Try the "du" command again, but this time with the "--apparent-size"
option.
--
Bob Nichols "NOSPAM" is really part of my email address.
Do NOT delete it.
--
users mailing list
users(a)lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
--
Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/manager
6:02 p.m.
On 30Mar2015 16:39, Ranjan Maitra <maitra.mbox.ignored(a)inbox.com> wrote:
Thanks, Cameron!
Any time.
> If you are using a CIFS share, does that mean the far end is not
a UNIX
> filesystem? The -S (sparse) option is only useful if the backend can store
> sparse files, otherwise the backend will just store lots of blocks of zeroes
> (presuming you really have sparse files).
The filesystem is a "high-end" IFS filesystem according to our systems
administrator. (I have no idea what an IFS filesystem is.)
It isn't, if The Google and Wikipedia are to be believed. IFS may mean
"Installable File System":
http://en.wikipedia.org/wiki/Installable_File_System
which is a Windows API for presenting filesystems to the OS, allowing various
drivers for backend filesystems to be plugged in. Like the VFS layer in a Linux
kernel I suppose.
At any rate, it says nothing of itself about what filesystem implements the
storage, but it may place constraints on what you can do with said filesystem
i.e. if the API does not support symlinks then you do not get symlinks and so
forth. Under UNIX there's no specific API for sparse files - you just avoid
writing the blocks full of NULs, instead seeking past them to the next non-NUL
data. The filesystem remembers the gap and fakes up empty (well, full of NULs)
blocks if you read from the gap instead of allocating (wasted) storage. The
point of this is that _if_ the backend supports sparse files they should work
if your file writing tool knows to behave that way (as rsync with -S does).
So your sysadmin might not know. Or might not be telling you (weird, but some
people are like that). This page:
http://support.microsoft.com/en-us/kb/100108
suggests that M$ consider NTFS to be "high end"; at any rate that is the only
place on the above page with the term. So: no hard links I think, no sparse
files I think, support for symlinks in the filesystem itself but no Windows API
for making them (IIRC).
> Is the source kmeans directory full of hard links (not symlinks)?
If so, rsync
> will not preserve hard links without the -H option (even with -a) and
> regardless I do not know if CIFS supports making hard links or if your backend
> supports hard links).
No, there are no hard links. Only a few files at the top level and about 4 directories
with lots of files in them.
Ok.
Sparse files?
One way to check is to tally the byte sizes of the files, and compare it with
the result of "du", which counts allocated blocks.
Tallying the file byte lengths can be done like this:
find . -type f -ls | colsum 7
"colsum" is a script of my own which makes and runs an awk script to sum
particular columns:
https://bitbucket.org/cameron_simpson/css/src/tip/bin/colsum
Nothing very clever, but surprisingly useful to me.
> If the far end is windows, certainly sparse files will no longer
be sparse at
> the far end. This kind of thing is one reason we get so picky when getting
> employers to order NASes; we asked for a QNAP (cheap, useful, Linux backend)
> and they wanted to order Microsoft's storage thingummy, which would have broken
> all sorts of stuff just like what you're encountering.
I suspect that that is what has happened here: the stuff is too high-end to be of any use.
Ah. Like "enterprise":-)
> Does anything above assist?
Yes, it does, at least providing a possible explanation.
I don't think it would make any difference if we tar'ed the file over to the
cifs share and then untarred, would it?
If all the access is done via CIFS from your client system, I would not expect
this to be any better.
Besides, I don't think tar does sparse files. (Or does it? GNU tar might, I
haven't looked.)
If GNU tar will store sparse files efficiently, and you have sparse files, then
you could profitably store the tar file on the CIFS share. I would expect
things to go sour as soon as you untarred it, which makes accessing the
contents tedious.
Do you have sparsae file?
As an anecdote, the other day I was working on set of web pages with mugshots
and full images, etc. Wanting to show it to a remote colleague I thought "I'll
use a zip file, he'll understand it". 133MB of files, 4GB of zip file. Ugh.
Why? I was still waiting on the real image data so we'd taken a few photos
(~20) and allocated them to the 500 odd images needed, using hard links. Zip
does not understand hard links.
Cheers,
Cameron Simpson <cs(a)zip.com.au>
(Bashir tells the story of the boy who cried "Wolf")
Bashir: If you lie all the time, no one is going to believe you, even
when you're telling the truth.
Garak: Are you sure that's the point, Doctor?
Bashir: Of course. What else would it be?
Garak: That you should never tell the same lie twice.
9:07 a.m.
Btw, I just found out that the filesystem in question is an Isilon filesystem.
Ranjan
On Wed, 1 Apr 2015 08:07:34 -0500 Ranjan Maitra <maitra.mbox.ignored(a)inbox.com>
wrote:
Thanks to both Cameron and you, Bob!
After the transfer, here is what we have, on that filesystem:
$ du -sh kmeans --apparent-size
154G kmeans
$ du -sh kmeans
628G kmeans
So, I guess that leaves me (and others) stuck.
Ranjan
On Mon, 30 Mar 2015 21:14:38 -0500 Robert Nichols <rnicholsNOSPAM(a)comcast.net>
wrote:
> On 03/30/2015 04:39 PM, Ranjan Maitra wrote:
> > Thanks, Cameron!
> >> If the far end is windows, certainly sparse files will no longer be sparse
at
> >> the far end. This kind of thing is one reason we get so picky when getting
> >> employers to order NASes; we asked for a QNAP (cheap, useful, Linux
backend)
> >> and they wanted to order Microsoft's storage thingummy, which would
have broken
> >> all sorts of stuff just like what you're encountering.
> >
> > I suspect that that is what has happened here: the stuff is too high-end to be
of any use.
>
> Try the "du" command again, but this time with the
"--apparent-size" option.
>
> --
> Bob Nichols "NOSPAM" is really part of my email address.
> Do NOT delete it.
>
> --
> users mailing list
> users(a)lists.fedoraproject.org
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org
--
Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/manager
--
users mailing list
users(a)lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
--
Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt.
Please respond to the mailing list if appropriate. For those needing to send personal or
professional e-mail, please use appropriate addresses.
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
3307
days inactive
3314
days old
17 comments
6 participants
participants (6)
-
Birger Monsen -
Cameron Simpson -
Gordon Messmer -
Ranjan Maitra -
Rick Stevens -
Robert Nichols