People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
Thanks,
Phil.
How about checking the permissions of all files/dirs in phil?
Run ls -lR ....phil
and report results.
On 03/11/2018 12:48 PM, Philip Rhoades wrote:
People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
Thanks,
Phil.
On 03/11/2018 01:48 PM, Philip Rhoades wrote:
People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
That "du -s -BG 20180216/phil/*" is ignoring any "dot" files/directories under 20180216/phil .
It's better to leave out the "-s" option and use the "--max-depth" option to limit the depth of the display. du -BG --max-depth=1 20180216/phil
JD, Gordon, Robert,
On 2018-03-12 06:13, Robert Nichols wrote:
On 03/11/2018 01:48 PM, Philip Rhoades wrote:
People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
That "du -s -BG 20180216/phil/*" is ignoring any "dot" files/directories under 20180216/phil .
It's better to leave out the "-s" option and use the "--max-depth" option to limit the depth of the display. du -BG --max-depth=1 20180216/phil
Damn, I should have thought of that - normally, outside of .config, there would not be much in my dot dirs - but I had been experimenting with cryptocurrency nodes . .
Thanks for the useful tips!
Regards,
P.
On 12/3/18 10:48 am, Philip Rhoades wrote:
JD, Gordon, Robert,
On 2018-03-12 06:13, Robert Nichols wrote:
On 03/11/2018 01:48 PM, Philip Rhoades wrote:
People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
That "du -s -BG 20180216/phil/*" is ignoring any "dot" files/directories under 20180216/phil .
It's better to leave out the "-s" option and use the "--max-depth" option to limit the depth of the display. du -BG --max-depth=1 20180216/phil
Damn, I should have thought of that - normally, outside of .config, there would not be much in my dot dirs - but I had been experimenting with cryptocurrency nodes . .
Thanks for the useful tips!
Regards,
P.
Just one question on this, is the scaling that the -BG screwing around with the results of du as shown below?
bash-4.4$ du -s -BG /home/steve 8G /home/steve
bash-4.4$ du -s -BG /home/steve/* 1G /home/steve/andrew 1G /home/steve/config 1G /home/steve/Desktop 1G /home/steve/Documents 1G /home/steve/Downloads 1G /home/steve/gtk.css 1G /home/steve/jxbrowser-browser.log 1G /home/steve/jxbrowser-chromium.log 1G /home/steve/jxbrowser-ipc.log 1G /home/steve/jxbrowser-ipc.log.1 0G /home/steve/jxbrowser-ipc.log.1.lck 0G /home/steve/jxbrowser-ipc.log.lck 1G /home/steve/l10n 1G /home/steve/Music 1G /home/steve/NetBeansProjects 1G /home/steve/Pictures 1G /home/steve/Public 1G /home/steve/R 3G /home/steve/rpmbuild 1G /home/steve/Templates 1G /home/steve/Videos 1G /home/steve/workspace
du -s -BG /home/steve/workspace/* 1G /home/steve/workspace/basics.zip_expanded 1G /home/steve/workspace/jsf-blank.zip_expanded 1G /home/steve/workspace/jsffacletstutorial 1G /home/steve/workspace/libraries 1G /home/steve/workspace/RemoteSystemsTempFiles 1G /home/steve/workspace/Servers 1G /home/steve/workspace/test-app 1G /home/steve/workspace/whirlwind
regards,
Steve
On 03/11/2018 07:19 PM, Stephen Morris wrote:
On 12/3/18 10:48 am, Philip Rhoades wrote:
JD, Gordon, Robert,
On 2018-03-12 06:13, Robert Nichols wrote:
On 03/11/2018 01:48 PM, Philip Rhoades wrote:
People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
That "du -s -BG 20180216/phil/*" is ignoring any "dot" files/directories under 20180216/phil .
It's better to leave out the "-s" option and use the "--max-depth" option to limit the depth of the display. du -BG --max-depth=1 20180216/phil
Damn, I should have thought of that - normally, outside of .config, there would not be much in my dot dirs - but I had been experimenting with cryptocurrency nodes . .
Thanks for the useful tips!
Regards,
P.
Just one question on this, is the scaling that the -BG screwing around with the results of du as shown below?
bash-4.4$ du -s -BG /home/steve 8G /home/steve
bash-4.4$ du -s -BG /home/steve/* 1G /home/steve/andrew 1G /home/steve/config 1G /home/steve/Desktop 1G /home/steve/Documents 1G /home/steve/Downloads 1G /home/steve/gtk.css 1G /home/steve/jxbrowser-browser.log 1G /home/steve/jxbrowser-chromium.log 1G /home/steve/jxbrowser-ipc.log 1G /home/steve/jxbrowser-ipc.log.1 0G /home/steve/jxbrowser-ipc.log.1.lck 0G /home/steve/jxbrowser-ipc.log.lck 1G /home/steve/l10n 1G /home/steve/Music 1G /home/steve/NetBeansProjects 1G /home/steve/Pictures 1G /home/steve/Public 1G /home/steve/R 3G /home/steve/rpmbuild 1G /home/steve/Templates 1G /home/steve/Videos 1G /home/steve/workspace
du -s -BG /home/steve/workspace/* 1G /home/steve/workspace/basics.zip_expanded 1G /home/steve/workspace/jsf-blank.zip_expanded 1G /home/steve/workspace/jsffacletstutorial 1G /home/steve/workspace/libraries 1G /home/steve/workspace/RemoteSystemsTempFiles 1G /home/steve/workspace/Servers 1G /home/steve/workspace/test-app 1G /home/steve/workspace/whirlwind
Of course it is. Any size greater than 0 and not exceeding 1GB will show as 1G. It's answering the question, "How many blocks 1GB in size would it take to hold this?" Anything of non-zero size will take at least 1 such block.
So, your "workspace" directory contains 8 things, none of which is larger than 1G, and the total space for all of them is also less than 1G.
On 12/3/18 11:29 am, Robert Nichols wrote:
On 03/11/2018 07:19 PM, Stephen Morris wrote:
On 12/3/18 10:48 am, Philip Rhoades wrote:
JD, Gordon, Robert,
On 2018-03-12 06:13, Robert Nichols wrote:
On 03/11/2018 01:48 PM, Philip Rhoades wrote:
People,
I started deleting GBs of stuff from:
/dev/sdb1 /backup
but df did not reduce from 95% so I looked more closely and found this weirdness:
# du -s -BG 20180216 43G 20180216
# du -s -BG 20180216/* 1G 20180216/naf_dirs 43G 20180216/phil 1G 20180216/root
# du -s -BG 20180216/phil/* 1G 20180216/phil/0 1G 20180216/phil/0_finance 1G 20180216/phil/0_naf 1G 20180216/phil/Maildir 1G 20180216/phil/txts 1G 20180216/phil/vimwiki
Where has ~37GB disappeared to? There are no files held open and I have successfully umounted and re-mounted the partition - what is going on?
That "du -s -BG 20180216/phil/*" is ignoring any "dot" files/directories under 20180216/phil .
It's better to leave out the "-s" option and use the "--max-depth" option to limit the depth of the display. du -BG --max-depth=1 20180216/phil
Damn, I should have thought of that - normally, outside of .config, there would not be much in my dot dirs - but I had been experimenting with cryptocurrency nodes . .
Thanks for the useful tips!
Regards,
P.
Just one question on this, is the scaling that the -BG screwing around with the results of du as shown below?
bash-4.4$ du -s -BG /home/steve 8G /home/steve
bash-4.4$ du -s -BG /home/steve/* 1G /home/steve/andrew 1G /home/steve/config 1G /home/steve/Desktop 1G /home/steve/Documents 1G /home/steve/Downloads 1G /home/steve/gtk.css 1G /home/steve/jxbrowser-browser.log 1G /home/steve/jxbrowser-chromium.log 1G /home/steve/jxbrowser-ipc.log 1G /home/steve/jxbrowser-ipc.log.1 0G /home/steve/jxbrowser-ipc.log.1.lck 0G /home/steve/jxbrowser-ipc.log.lck 1G /home/steve/l10n 1G /home/steve/Music 1G /home/steve/NetBeansProjects 1G /home/steve/Pictures 1G /home/steve/Public 1G /home/steve/R 3G /home/steve/rpmbuild 1G /home/steve/Templates 1G /home/steve/Videos 1G /home/steve/workspace
du -s -BG /home/steve/workspace/* 1G /home/steve/workspace/basics.zip_expanded 1G /home/steve/workspace/jsf-blank.zip_expanded 1G /home/steve/workspace/jsffacletstutorial 1G /home/steve/workspace/libraries 1G /home/steve/workspace/RemoteSystemsTempFiles 1G /home/steve/workspace/Servers 1G /home/steve/workspace/test-app 1G /home/steve/workspace/whirlwind
Of course it is. Any size greater than 0 and not exceeding 1GB will show as 1G. It's answering the question, "How many blocks 1GB in size would it take to hold this?" Anything of non-zero size will take at least 1 such block.
So, your "workspace" directory contains 8 things, none of which is larger than 1G, and the total space for all of them is also less than 1G.
Thanks Robert, so basically what you are saying is that if you use that parameter, the output is rounded up to the nearest integer representation (in this case Gig) rather than displaying it as a fraction? For example, for a file that is 600 MB in size I would have expected the command to display it as 0.6G rather than 1G.
regards,
Steve
On 03/12/18 09:41, Stephen Morris wrote:
Thanks Robert, so basically what you are saying is that if you use that parameter, the output is rounded up to the nearest integer representation (in this case Gig) rather than displaying it as a fraction? For example, for a file that is 600 MB in size I would have expected the command to display it as 0.6G rather than 1G.
From the man page....
-B, --block-size=SIZE scale sizes by SIZE before printing them; e.g., '-BM' prints sizes in units of 1,048,576 bytes; see SIZE format below
and
The SIZE argument is an integer and optional unit (example: 10K is 10*1024). Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (pow‐ ers of 1000)
Key word in the above definition of "SIZE" is "integer". And, as you've noted, 0.6 isn't an integer.
I just happen to have a computer in front of me. So, I can create a few files and try things.
[egreshko@meimei misty]$ ll test-dir/ total 1110004 -rw-rw-r--. 1 egreshko egreshko 1024000000 Mar 12 10:21 test -rw-rw-r--. 1 egreshko egreshko 102400000 Mar 12 10:27 test1 -rw-rw-r--. 1 egreshko egreshko 10240000 Mar 12 10:29 test2
[egreshko@meimei misty]$ ll -h test-dir/ total 1.1G -rw-rw-r--. 1 egreshko egreshko 977M Mar 12 10:21 test -rw-rw-r--. 1 egreshko egreshko 98M Mar 12 10:27 test1 -rw-rw-r--. 1 egreshko egreshko 9.8M Mar 12 10:29 test2
[egreshko@meimei misty]$ ll --si test-dir/ total 1.2G -rw-rw-r--. 1 egreshko egreshko 1.1G Mar 12 10:21 test -rw-rw-r--. 1 egreshko egreshko 103M Mar 12 10:27 test1 -rw-rw-r--. 1 egreshko egreshko 11M Mar 12 10:29 test2
[egreshko@meimei misty]$ du -s -BG test-dir/* 1G test-dir/test 1G test-dir/test1 1G test-dir/test2
[egreshko@meimei misty]$ du -s -BG test-dir 2G test-dir
[egreshko@meimei misty]$ du -s -BM test-dir/* 977M test-dir/test 98M test-dir/test1 10M test-dir/test2
[egreshko@meimei misty]$ du -s -BM test-dir 1084M test-dir
And, since there is a difference, the man page for "ls" should be consulted to find....
-h, --human-readable with -l and/or -s, print human readable sizes (e.g., 1K 234M 2G)
--si likewise, but use powers of 1000 not 1024
On 12/3/18 1:41 pm, Ed Greshko wrote:
On 03/12/18 09:41, Stephen Morris wrote:
Thanks Robert, so basically what you are saying is that if you use that parameter, the output is rounded up to the nearest integer representation (in this case Gig) rather than displaying it as a fraction? For example, for a file that is 600 MB in size I would have expected the command to display it as 0.6G rather than 1G.
From the man page....
-B, --block-size=SIZE scale sizes by SIZE before printing them; e.g., '-BM' prints sizes in units of 1,048,576 bytes; see SIZE format below
and
The SIZE argument is an integer and optional unit (example: 10K is 10*1024). Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (pow‐ ers of 1000)
Key word in the above definition of "SIZE" is "integer". And, as you've noted, 0.6 isn't an integer.
I just happen to have a computer in front of me. So, I can create a few files and try things.
[egreshko@meimei misty]$ ll test-dir/ total 1110004 -rw-rw-r--. 1 egreshko egreshko 1024000000 Mar 12 10:21 test -rw-rw-r--. 1 egreshko egreshko 102400000 Mar 12 10:27 test1 -rw-rw-r--. 1 egreshko egreshko 10240000 Mar 12 10:29 test2
[egreshko@meimei misty]$ ll -h test-dir/ total 1.1G -rw-rw-r--. 1 egreshko egreshko 977M Mar 12 10:21 test -rw-rw-r--. 1 egreshko egreshko 98M Mar 12 10:27 test1 -rw-rw-r--. 1 egreshko egreshko 9.8M Mar 12 10:29 test2
[egreshko@meimei misty]$ ll --si test-dir/ total 1.2G -rw-rw-r--. 1 egreshko egreshko 1.1G Mar 12 10:21 test -rw-rw-r--. 1 egreshko egreshko 103M Mar 12 10:27 test1 -rw-rw-r--. 1 egreshko egreshko 11M Mar 12 10:29 test2
[egreshko@meimei misty]$ du -s -BG test-dir/* 1G test-dir/test 1G test-dir/test1 1G test-dir/test2
[egreshko@meimei misty]$ du -s -BG test-dir 2G test-dir
[egreshko@meimei misty]$ du -s -BM test-dir/* 977M test-dir/test 98M test-dir/test1 10M test-dir/test2
[egreshko@meimei misty]$ du -s -BM test-dir 1084M test-dir
And, since there is a difference, the man page for "ls" should be consulted to find....
-h, --human-readable with -l and/or -s, print human readable sizes (e.g., 1K 234M 2G)
--si likewise, but use powers of 1000 not 1024
Thanks Ed, this functionality seems to be counter intuitive to me. Given that du shows space used in sub-directories, it seems to me the only useful output is du without any parameters.
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On Mon, 2018-03-12 at 21:41 +1100, Stephen Morris wrote:
Thanks Ed, this functionality seems to be counter intuitive to me. Given that du shows space used in sub-directories, it seems to me the only useful output is du without any parameters.
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
On 12/3/18 10:11 pm, Patrick O'Callaghan wrote:
On Mon, 2018-03-12 at 21:41 +1100, Stephen Morris wrote:
Thanks Ed, this functionality seems to be counter intuitive to me. Given that du shows space used in sub-directories, it seems to me the only useful output is du without any parameters.
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below. I am listing my Desktop directory via ll, du -hs and du -bhs. Just further to this is it a bug with du that the -a parameter which is supposed to list all files not just directories, does not list files prefixed with a '.'?:
bash-4.4$ ll /home/steve/Desktop/* -rwxr-xr-x. 1 steve steve 466 Oct 3 2015 '/home/steve/Desktop/Acrobat Reader.desktop' -rwxrwxrwx. 1 steve steve 233 May 5 2014 /home/steve/Desktop/autoplus.desktop -rwxr-xr-x. 1 steve steve 310 Dec 26 2016 /home/steve/Desktop/chrome-eppojlglocelodeimnohnlnionkobfln-Default.desktop -rwxr-xr-x. 1 steve steve 374 Jan 11 07:54 /home/steve/Desktop/Eclipse.desktop -rwxr-xr-x. 1 steve steve 417 Jan 16 2017 /home/steve/Desktop/Firefox.desktop -rwxr-xr-x. 1 steve steve 358 Sep 8 2013 '/home/steve/Desktop/Google Chrome.desktop' -rwxrw-r--. 1 steve steve 412 Nov 30 2013 /home/steve/Desktop/jws_app_shortcut_1385783507561.desktop -rwxrwxr-x. 1 steve steve 247 Jan 19 07:47 /home/steve/Desktop/netbeans-8.2.desktop -rw-------. 1 steve steve 169 Mar 3 2015 '/home/steve/Desktop/Nova Genesis.url' -rwx------. 1 steve steve 2191 Mar 12 13:10 /home/steve/Desktop/steam.desktop -rwxr-xr-x. 1 steve steve 370 Jan 12 06:57 /home/steve/Desktop/Thunderbird.desktop -rwxr--r--. 1 steve steve 313 Oct 5 2013 /home/steve/Desktop/Vuescan.desktop -rwxrw-r--. 1 steve steve 362 Jan 11 07:10 /home/steve/Desktop/yatta-eclipse-launcher.desktop
bash-4.4$ du -hs /home/steve/Desktop/* 8.0K /home/steve/Desktop/Acrobat Reader.desktop 8.0K /home/steve/Desktop/autoplus.desktop 4.0K /home/steve/Desktop/chrome-eppojlglocelodeimnohnlnionkobfln-Default.desktop 4.0K /home/steve/Desktop/Eclipse.desktop 4.0K /home/steve/Desktop/Firefox.desktop 8.0K /home/steve/Desktop/Google Chrome.desktop 8.0K /home/steve/Desktop/jws_app_shortcut_1385783507561.desktop 4.0K /home/steve/Desktop/netbeans-8.2.desktop 4.0K /home/steve/Desktop/Nova Genesis.url 4.0K /home/steve/Desktop/steam.desktop 4.0K /home/steve/Desktop/Thunderbird.desktop 8.0K /home/steve/Desktop/Vuescan.desktop 4.0K /home/steve/Desktop/yatta-eclipse-launcher.desktop
bash-4.4$ du -bhs /home/steve/Desktop/* 466 /home/steve/Desktop/Acrobat Reader.desktop 233 /home/steve/Desktop/autoplus.desktop 310 /home/steve/Desktop/chrome-eppojlglocelodeimnohnlnionkobfln-Default.desktop 374 /home/steve/Desktop/Eclipse.desktop 417 /home/steve/Desktop/Firefox.desktop 358 /home/steve/Desktop/Google Chrome.desktop 412 /home/steve/Desktop/jws_app_shortcut_1385783507561.desktop 247 /home/steve/Desktop/netbeans-8.2.desktop 169 /home/steve/Desktop/Nova Genesis.url 2.2K /home/steve/Desktop/steam.desktop 370 /home/steve/Desktop/Thunderbird.desktop 313 /home/steve/Desktop/Vuescan.desktop 362 /home/steve/Desktop/yatta-eclipse-launcher.desktop
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/13/18 04:26, Stephen Morris wrote:
Just further to this is it a bug with du that the -a parameter which is supposed to list all files not just directories, does not list files prefixed with a '.'?:
Of course not.
Compare the -a opt of ls with that of du in their man pages.
ls
-a, --all do not ignore entries starting with
du
-a, --all write counts for all files, not just directories
With du it is saying "all files found based on the pattern specified in [FILES]" And like all shell commands [FILES} follows the rules of "globbing". Gordon has already told you a method you can use to have du consider "." files.
This is by turning on the shell option "dotglob".
Run the "shopt" command by itself and notice the setting for "dotglob". Then set it with -s to see how this affects commands when you use "*". And -u will unset options.
You would probably benefit from doing a bit of searching and finding pages such as
https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html#Th...
and more generally
https://www.gnu.org/software/bash/manual/html_node/
On 13/3/18 8:10 am, Ed Greshko wrote:
On 03/13/18 04:26, Stephen Morris wrote:
Just further to this is it a bug with du that the -a parameter which is supposed to list all files not just directories, does not list files prefixed with a '.'?:
Of course not.
Compare the -a opt of ls with that of du in their man pages.
ls
-a, --all do not ignore entries starting with
du
-a, --all write counts for all files, not just directories
With du it is saying "all files found based on the pattern specified in [FILES]" And like all shell commands [FILES} follows the rules of "globbing". Gordon has already told you a method you can use to have du consider "." files.
This is by turning on the shell option "dotglob".
Run the "shopt" command by itself and notice the setting for "dotglob". Then set it with -s to see how this affects commands when you use "*". And -u will unset options.
You would probably benefit from doing a bit of searching and finding pages such as
https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html#Th...
and more generally
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
On 03/13/18 05:47, Ed Greshko wrote:
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
Oh, when it comes to ls, I've been gently reminded about -A
[egreshko@meimei test-dir]$ ls -A test test1 test2 .test3 .test4 .test5
On 03/12/2018 03:26 PM, Stephen Morris wrote:
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below. I am listing my Desktop directory via ll, du -hs and du -bhs. Just further to this is it a bug with du that the -a parameter which is supposed to list all files not just directories, does not list files prefixed with a '.'?:
The du command is doing exactly what it is told to do. It is your _shell_ that is expanding the "*" into a list of names that does not (by default) include "dot" names. The du command is never asked to look at the current directory, and thus will never see the dotfiles, regardless of what flag arguments you use. What you are complaining about is like running "du -a [abc]*" and complaining that names beginning with "d" were not included.
Whenever you have questions like this, you should run "set -x" in the shell to see exactly how commands are being invoked. (Run "set +x" to turn that off again.)
On Tue, 2018-03-13 at 07:26 +1100, Stephen Morris wrote:
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below.
I think you have a misconception here. 'du' does not give file sizes, it gives disk usage. A 1-byte file takes up at least 1 disk block, so that's the size 'du' will give. I seem to remember that it also counts indirect blocks and other housekeeping that corresponds to the file without being included in the file's content, but I could be mistaken (though I'm fairly sure early versions did do that).
I am listing my Desktop directory via ll, du -hs and du -bhs. Just further to this is it a bug with du that the -a parameter which is supposed to list all files not just directories, does not list files prefixed with a '.'?:
It does list files beginning with '.'. The reason you aren't seeing these files is because you're implicitly excluding them when you write '.../*'. The Shell meta-character '*' doesn't match the initial '.'.
poc
On 03/12/2018 03:37 PM, Patrick O'Callaghan wrote:
On Tue, 2018-03-13 at 07:26 +1100, Stephen Morris wrote:
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below.
I think you have a misconception here. 'du' does not give file sizes, it gives disk usage. A 1-byte file takes up at least 1 disk block, so that's the size 'du' will give. I seem to remember that it also counts indirect blocks and other housekeeping that corresponds to the file without being included in the file's content, but I could be mistaken (though I'm fairly sure early versions did do that).
du (with no flags) gives disk usage. As Patrick says, a 1-byte file uses one disk block (which is generally 4KiB) and that's what du is reporting (after all, "du" means "disk usage"). The "-b" flag means "set the block size to 1 byte and show the apparent size", which is what "ls -l" would report (there may be differences between du and ls if sparse files are involved).
Also, du walks down the entire current directory unless you give it arguments to tell it what to look at. Note that the arguments you pass it are interpreted by the _shell_, not "du" (even the man page says "PATTERN is a shell pattern (not a regular expression)").
This is a common confusion point with many people. Unless you enclose shell metacharacters in quotes (and dots and splats are metacharacters), the shell WILL interpret them--sometimes in ways you aren't expecting! By default, shell globbing does NOT expand filenames that start with a dot (to the shell, a dot means "current directory").
If you want to list JUST files/dirs that start with a dot, you can use "du [flags] .* (so the shell interprets the dot as a dot, not "current directory"). Or you can tell the shell to expand files starting with a dot by using "shopt -s dotglob" (as Ed said). "shopt -u dotglob" will restore the default setting. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - The Theory of Rapitivity: E=MC Hammer - - -- Glenn Marcus (via TopFive.com) - ----------------------------------------------------------------------
On Mon, 2018-03-12 at 18:25 -0700, Rick Stevens wrote:
On 03/12/2018 03:37 PM, Patrick O'Callaghan wrote:
On Tue, 2018-03-13 at 07:26 +1100, Stephen Morris wrote:
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below.
I think you have a misconception here. 'du' does not give file sizes, it gives disk usage. A 1-byte file takes up at least 1 disk block, so that's the size 'du' will give. I seem to remember that it also counts indirect blocks and other housekeeping that corresponds to the file without being included in the file's content, but I could be mistaken (though I'm fairly sure early versions did do that).
du (with no flags) gives disk usage. As Patrick says, a 1-byte file uses one disk block (which is generally 4KiB) and that's what du is reporting (after all, "du" means "disk usage"). The "-b" flag means "set the block size to 1 byte and show the apparent size", which is what "ls -l" would report (there may be differences between du and ls if sparse files are involved).
Good point about sparse files, which I'd forgotten in this context. A sparse file can be 10TB in size yet occupy only 1 disk block. 'du' will give the usage as 1 block and 'ls' will say it's 10TB.
Also, du walks down the entire current directory unless you give it arguments to tell it what to look at. Note that the arguments you pass it are interpreted by the _shell_, not "du" (even the man page says "PATTERN is a shell pattern (not a regular expression)").
This is a common confusion point with many people. Unless you enclose shell metacharacters in quotes (and dots and splats are metacharacters), the shell WILL interpret them--sometimes in ways you aren't expecting! By default, shell globbing does NOT expand filenames that start with a dot (to the shell, a dot means "current directory").
Slight nit: '.' (on its own) means 'current directory' to the kernel, not to the Shell, i.e. it's wired into the system and doesn't depend on the command interpreter (in the same way that '/' is the path separator and cannot be changed). The use of dot-files as a way of hiding certain names is IIRC originally just a convention of 'ls' which the Shell adopted for consistency.
poc
On 13/3/18 9:05 am, Ed Greshko wrote:
On 03/13/18 05:47, Ed Greshko wrote:
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
Oh, when it comes to ls, I've been gently reminded about -A
[egreshko@meimei test-dir]$ ls -A test test1 test2 .test3 .test4 .test5
Thanks Ed, I knew about the differences between the -a and -A on ls, and having had a look at the --help for ls and du again, I can see the subtle difference between the -a parameters on both commands. It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/13/2018 01:19 PM, Stephen Morris wrote:
On 13/3/18 9:05 am, Ed Greshko wrote:
On 03/13/18 05:47, Ed Greshko wrote:
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
Oh, when it comes to ls, I've been gently reminded about -A
[egreshko@meimei test-dir]$ ls -A test test1 test2 .test3 .test4 .test5
Thanks Ed, I knew about the differences between the -a and -A on ls, and having had a look at the --help for ls and du again, I can see the subtle difference between the -a parameters on both commands. It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
I COULD drive a dump truck or a sedan to work every day. If I were in the construction business, the dump truck might make sense. I'm not, so I drive a sedan. Both would get me to the job, it's just which is more "appropriate" that's at issue.
"ls" lists the characteristics of files, while "du" displays disk usage. While they are related, they are, none the less, different things. It's up to you, the user, to use the appropriate command for what you want.
The alternative is one program that tries to do EVERYTHING. That's been tried. It's called "Windows Explorer" and we all know what an atrocity that turned out to be! ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - The difference between nerds and geeks? Nerds are socially inept - - and are painfully aware of that. Geeks couldn't care less. - ----------------------------------------------------------------------
On 14/3/18 7:19 am, Stephen Morris wrote:
On 13/3/18 9:05 am, Ed Greshko wrote:
On 03/13/18 05:47, Ed Greshko wrote:
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
Oh, when it comes to ls, I've been gently reminded about -A
[egreshko@meimei test-dir]$ ls -A test test1 test2 .test3 .test4 .test5
Thanks Ed, I knew about the differences between the -a and -A on ls, and having had a look at the --help for ls and du again, I can see the subtle difference between the -a parameters on both commands. It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
I'm now completely confused. The command du -abh /home/steve/workspace is now displaying the information directories beginning with a '.' and at least some files beginning with a '.' without having issued the shopt command.
regards,
Steve
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 13/3/18 9:33 am, Robert Nichols wrote:
On 03/12/2018 03:26 PM, Stephen Morris wrote:
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below. I am listing my Desktop directory via ll, du -hs and du -bhs. Just further to this is it a bug with du that the -a parameter which is supposed to list all files not just directories, does not list files prefixed with a '.'?:
The du command is doing exactly what it is told to do. It is your _shell_ that is expanding the "*" into a list of names that does not (by default) include "dot" names. The du command is never asked to look at the current directory, and thus will never see the dotfiles, regardless of what flag arguments you use. What you are complaining about is like running "du -a [abc]*" and complaining that names beginning with "d" were not included.
Whenever you have questions like this, you should run "set -x" in the shell to see exactly how commands are being invoked. (Run "set +x" to turn that off again.)
I'm not sure what the point of this command is relative to du, it seems to be more relevant when using commands that are aliases.
Having issued the command "set -x", if I issue the command du -abh /home/steve/workspace it just echos that command back, and if I issue the command du -abh /home/steve/workspace/* it echos the command with a list of sub-directories of workspace that du is going to do its work over. If I issue my alias command "la /home/steve/workspace" it echos back the command "ls --color -a /home/steve/workspace", which is much more meaningful.
regards,
Steve
On 14/3/18 7:42 am, Rick Stevens wrote:
On 03/13/2018 01:19 PM, Stephen Morris wrote:
On 13/3/18 9:05 am, Ed Greshko wrote:
On 03/13/18 05:47, Ed Greshko wrote:
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
Oh, when it comes to ls, I've been gently reminded about -A
[egreshko@meimei test-dir]$ ls -A test test1 test2 .test3 .test4 .test5
Thanks Ed, I knew about the differences between the -a and -A on ls, and having had a look at the --help for ls and du again, I can see the subtle difference between the -a parameters on both commands. It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
I COULD drive a dump truck or a sedan to work every day. If I were in the construction business, the dump truck might make sense. I'm not, so I drive a sedan. Both would get me to the job, it's just which is more "appropriate" that's at issue.
"ls" lists the characteristics of files, while "du" displays disk usage. While they are related, they are, none the less, different things. It's up to you, the user, to use the appropriate command for what you want.
The alternative is one program that tries to do EVERYTHING. That's been tried. It's called "Windows Explorer" and we all know what an atrocity that turned out to be!
This is true, but "du" is extremely useful, when it provides a total of the space occupied by directories, to determine where space hogs are and why.
regards,
Steve
- Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com -
- AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 -
-- The difference between nerds and geeks? Nerds are socially inept -
- and are painfully aware of that. Geeks couldn't care less. -
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 13/3/18 8:46 pm, Patrick O'Callaghan wrote:
On Mon, 2018-03-12 at 18:25 -0700, Rick Stevens wrote:
On 03/12/2018 03:37 PM, Patrick O'Callaghan wrote:
On Tue, 2018-03-13 at 07:26 +1100, Stephen Morris wrote:
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below.
I think you have a misconception here. 'du' does not give file sizes, it gives disk usage. A 1-byte file takes up at least 1 disk block, so that's the size 'du' will give. I seem to remember that it also counts indirect blocks and other housekeeping that corresponds to the file without being included in the file's content, but I could be mistaken (though I'm fairly sure early versions did do that).
du (with no flags) gives disk usage. As Patrick says, a 1-byte file uses one disk block (which is generally 4KiB) and that's what du is reporting (after all, "du" means "disk usage"). The "-b" flag means "set the block size to 1 byte and show the apparent size", which is what "ls -l" would report (there may be differences between du and ls if sparse files are involved).
Good point about sparse files, which I'd forgotten in this context. A sparse file can be 10TB in size yet occupy only 1 disk block. 'du' will give the usage as 1 block and 'ls' will say it's 10TB.
Also, du walks down the entire current directory unless you give it arguments to tell it what to look at. Note that the arguments you pass it are interpreted by the _shell_, not "du" (even the man page says "PATTERN is a shell pattern (not a regular expression)").
This is a common confusion point with many people. Unless you enclose shell metacharacters in quotes (and dots and splats are metacharacters), the shell WILL interpret them--sometimes in ways you aren't expecting! By default, shell globbing does NOT expand filenames that start with a dot (to the shell, a dot means "current directory").
Slight nit: '.' (on its own) means 'current directory' to the kernel, not to the Shell, i.e. it's wired into the system and doesn't depend on the command interpreter (in the same way that '/' is the path separator and cannot be changed). The use of dot-files as a way of hiding certain names is IIRC originally just a convention of 'ls' which the Shell adopted for consistency.
poc
Thanks guys, I fully understood the -b parameter was giving an apparent size of the file (which approximately matches what ls -l provides), that sparse files can impact the apparent size, and that the physical size of a file is governed by the sector size, but my usage of du, especially with the -s parameter, is the amount of "apparent" space occupied by files and directories (where possible I usually use a sector size of 512) to identify space hogs. I'm also fully aware of the sector size trade off between efficient space utilization and efficient I/O and the apparent space used by files.
What I hadn't realized until Patrick mentioned it, was the significance of the * in the path specification. I hadn't realized that with using the * not only did it cause files prefixed with a '.' to be ignored but it also causes directories prefixed with a '.' to be ignored also.
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/14/18 04:44, Stephen Morris wrote:
I'm now completely confused. The command du -abh /home/steve/workspace is now displaying the information directories beginning with a '.' and at least some files beginning with a '.' without having issued the shopt command.
The command you're showing DOES NOT do pattern matching!
du -abh /home/steve/workspace
v.s.
du -abh /home/steve/workspace/*
The later command is basically creating a list of files (a shell function) and doing a "du" on each file individually.
Notice that if you use that command you don't get a summary of what is in the workspace directory. Also, note that the later command creates a sorted list.
You are mixing functions of the bash shell and the command being run within the shell!
On Wed, 2018-03-14 at 08:21 +1100, Stephen Morris wrote:
What I hadn't realized until Patrick mentioned it, was the significance of the * in the path specification. I hadn't realized that with using the * not only did it cause files prefixed with a '.' to be ignored but it also causes directories prefixed with a '.' to be ignored also.
Shell name expansion is applied to the contents of a directory irrespective of what those contents mean. Thus '*' matches all directory entries except those prefixed with '.', as the Shell manual states. That applies to files, directories, sockets, devices, ...
poc
On 03/14/18 05:22, Ed Greshko wrote:
The later command is basically creating a list of files (a shell function) and doing a "du" on each file individually.
Here is another example to show you what is happening....
[egreshko@meimei test-dir]$ ls stuff test test1 test2
[egreshko@meimei test-dir]$ cat stuff tes te te*
[egreshko@meimei test-dir]$ grep te stuff tes te te*
[egreshko@meimei test-dir]$ grep te* stuff [egreshko@meimei test-dir]$
But if I change to a directory which contains no files....
[egreshko@meimei test-dir]$ cd .hidden
[egreshko@meimei .hidden]$ ls
[egreshko@meimei .hidden]$ grep te* ../stuff tes te te*
On 03/13/2018 01:44 PM, Stephen Morris wrote:
On 14/3/18 7:19 am, Stephen Morris wrote:
On 13/3/18 9:05 am, Ed Greshko wrote:
On 03/13/18 05:47, Ed Greshko wrote:
On 03/13/18 05:20, Stephen Morris wrote:
Thanks Ed, I'll check the doco out, I was just expecting the command to do exactly what the help info said, output the information for all files, not just a subset.
It *does* do exactly what the man page says. You just have to understand the "context" in which it is saying it.
Let me complete my thought. Even take the "ls" command as an example.
[egreshko@meimei test-dir]$ ls test test1 test2
[egreshko@meimei test-dir]$ ls -a . .. test test1 test2 .test3 .test4 .test5
[egreshko@meimei test-dir]$ ls * test test1 test2
[egreshko@meimei test-dir]$ shopt -s dotglob
[egreshko@meimei test-dir]$ ls * test test1 test2 .test3 .test4 .test5
Oh, when it comes to ls, I've been gently reminded about -A
[egreshko@meimei test-dir]$ ls -A test test1 test2 .test3 .test4 .test5
Thanks Ed, I knew about the differences between the -a and -A on ls, and having had a look at the --help for ls and du again, I can see the subtle difference between the -a parameters on both commands. It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
I'm now completely confused. The command du -abh /home/steve/workspace is now displaying the information directories beginning with a '.' and at least some files beginning with a '.' without having issued the shopt command.
Because you didn't include a shell glob. You told du specifically: "show me the disk usage of /home/steve/workspace". du then walks down THAT directory tree and du does NOT ignore files starting with a dot.
The difference is: before you used "*" which made the shell expand the glob (glob meaning "*") and pass a list of files to du to check, but the important bit is that the _shell_ created the list of files and, by default, the shell does NOT list files starting with a dot.
Does that clarify it a bit? ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Memory is the second thing to go, but I can't remember the first! - ----------------------------------------------------------------------
On 03/13/2018 02:21 PM, Stephen Morris wrote:
On 13/3/18 8:46 pm, Patrick O'Callaghan wrote:
On Mon, 2018-03-12 at 18:25 -0700, Rick Stevens wrote:
On 03/12/2018 03:37 PM, Patrick O'Callaghan wrote:
On Tue, 2018-03-13 at 07:26 +1100, Stephen Morris wrote:
'du' with no parameters recursively lists all the subdirectories and their sizes, along with the grand total. When applied to my home directory, I get over 30,000 lines of output. That's almost never what I want. My usual call is 'du -hs'.
poc
Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below.
I think you have a misconception here. 'du' does not give file sizes, it gives disk usage. A 1-byte file takes up at least 1 disk block, so that's the size 'du' will give. I seem to remember that it also counts indirect blocks and other housekeeping that corresponds to the file without being included in the file's content, but I could be mistaken (though I'm fairly sure early versions did do that).
du (with no flags) gives disk usage. As Patrick says, a 1-byte file uses one disk block (which is generally 4KiB) and that's what du is reporting (after all, "du" means "disk usage"). The "-b" flag means "set the block size to 1 byte and show the apparent size", which is what "ls -l" would report (there may be differences between du and ls if sparse files are involved).
Good point about sparse files, which I'd forgotten in this context. A sparse file can be 10TB in size yet occupy only 1 disk block. 'du' will give the usage as 1 block and 'ls' will say it's 10TB.
Also, du walks down the entire current directory unless you give it arguments to tell it what to look at. Note that the arguments you pass it are interpreted by the _shell_, not "du" (even the man page says "PATTERN is a shell pattern (not a regular expression)").
This is a common confusion point with many people. Unless you enclose shell metacharacters in quotes (and dots and splats are metacharacters), the shell WILL interpret them--sometimes in ways you aren't expecting! By default, shell globbing does NOT expand filenames that start with a dot (to the shell, a dot means "current directory").
Slight nit: '.' (on its own) means 'current directory' to the kernel, not to the Shell, i.e. it's wired into the system and doesn't depend on the command interpreter (in the same way that '/' is the path separator and cannot be changed). The use of dot-files as a way of hiding certain names is IIRC originally just a convention of 'ls' which the Shell adopted for consistency.
poc
Thanks guys, I fully understood the -b parameter was giving an apparent size of the file (which approximately matches what ls -l provides), that sparse files can impact the apparent size, and that the physical size of a file is governed by the sector size, but my usage of du, especially with the -s parameter, is the amount of "apparent" space occupied by files and directories (where possible I usually use a sector size of 512) to identify space hogs. I'm also fully aware of the sector size trade off between efficient space utilization and efficient I/O and the apparent space used by files.
What I hadn't realized until Patrick mentioned it, was the significance of the * in the path specification. I hadn't realized that with using the * not only did it cause files prefixed with a '.' to be ignored but it also causes directories prefixed with a '.' to be ignored also.
No. What you have to get straight is that when you use "*", you're asking the shell to create a list of files and return it, and by default the shell doesn't list files starting with a dot.
To clarify, assume you have a directory containing the files "file1", "file2", "file3" and ".dotfile". If you "cd" to it and do an "ls *" (asking the shell to expand the glob), you'll get a list of "file1", "file2" and "file3". The same is true if the command was "du" and not "ls". So the command:
du -a *
is exactly equivalent to:
du -a file1 file2 file3
If you simply do:
du -a
then du will walk down the _directory_ you're in and display all of the files, including ".dotfile", since du does not ignore files starting with a dot. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Admitting you have a problem is the first step toward getting - - medicated for it. -- Jim Evarts (http://www.TopFive.com) - ----------------------------------------------------------------------
On 03/13/2018 04:59 PM, Rick Stevens wrote:
On 03/13/2018 02:21 PM, Stephen Morris wrote:
On 13/3/18 8:46 pm, Patrick O'Callaghan wrote:
On Mon, 2018-03-12 at 18:25 -0700, Rick Stevens wrote:
On 03/12/2018 03:37 PM, Patrick O'Callaghan wrote:
On Tue, 2018-03-13 at 07:26 +1100, Stephen Morris wrote:
> 'du' with no parameters recursively lists all the subdirectories and > their sizes, along with the grand total. When applied to my home > directory, I get over 30,000 lines of output. That's almost never > what > I want. My usual call is 'du -hs'. > > poc Thanks Patrick, taking this a step further, it seems to me that the only parameter for du that, to me, provides the correct file size is -b as shown below.
I think you have a misconception here. 'du' does not give file sizes, it gives disk usage. A 1-byte file takes up at least 1 disk block, so that's the size 'du' will give. I seem to remember that it also counts indirect blocks and other housekeeping that corresponds to the file without being included in the file's content, but I could be mistaken (though I'm fairly sure early versions did do that).
du (with no flags) gives disk usage. As Patrick says, a 1-byte file uses one disk block (which is generally 4KiB) and that's what du is reporting (after all, "du" means "disk usage"). The "-b" flag means "set the block size to 1 byte and show the apparent size", which is what "ls -l" would report (there may be differences between du and ls if sparse files are involved).
Good point about sparse files, which I'd forgotten in this context. A sparse file can be 10TB in size yet occupy only 1 disk block. 'du' will give the usage as 1 block and 'ls' will say it's 10TB.
Also, du walks down the entire current directory unless you give it arguments to tell it what to look at. Note that the arguments you pass it are interpreted by the _shell_, not "du" (even the man page says "PATTERN is a shell pattern (not a regular expression)").
This is a common confusion point with many people. Unless you enclose shell metacharacters in quotes (and dots and splats are metacharacters), the shell WILL interpret them--sometimes in ways you aren't expecting! By default, shell globbing does NOT expand filenames that start with a dot (to the shell, a dot means "current directory").
Slight nit: '.' (on its own) means 'current directory' to the kernel, not to the Shell, i.e. it's wired into the system and doesn't depend on the command interpreter (in the same way that '/' is the path separator and cannot be changed). The use of dot-files as a way of hiding certain names is IIRC originally just a convention of 'ls' which the Shell adopted for consistency.
poc
Thanks guys, I fully understood the -b parameter was giving an apparent size of the file (which approximately matches what ls -l provides), that sparse files can impact the apparent size, and that the physical size of a file is governed by the sector size, but my usage of du, especially with the -s parameter, is the amount of "apparent" space occupied by files and directories (where possible I usually use a sector size of 512) to identify space hogs. I'm also fully aware of the sector size trade off between efficient space utilization and efficient I/O and the apparent space used by files.
What I hadn't realized until Patrick mentioned it, was the significance of the * in the path specification. I hadn't realized that with using the * not only did it cause files prefixed with a '.' to be ignored but it also causes directories prefixed with a '.' to be ignored also.
No. What you have to get straight is that when you use "*", you're asking the shell to create a list of files and return it, and by default the shell doesn't list files starting with a dot.
To clarify, assume you have a directory containing the files "file1", "file2", "file3" and ".dotfile". If you "cd" to it and do an "ls *" (asking the shell to expand the glob), you'll get a list of "file1", "file2" and "file3". The same is true if the command was "du" and not "ls". So the command:
du -a *
is exactly equivalent to:
du -a file1 file2 file3
If you simply do:
du -a
then du will walk down the _directory_ you're in and display all of the files, including ".dotfile", since du does not ignore files starting with a dot.
Here's an example:
[root@golem4 ~]# mkdir /tmp/testdir [root@golem4 ~]# cd /tmp/testdir [root@golem4 testdir]# touch file1 file2 file3 .dotfile [root@golem4 testdir]# echo "Without shopt:";ls *;echo "With shopt:";shopt -s dotglob;ls *;echo "Without shopt again:";shopt -u dotglob;ls * Without shopt: file1 file2 file3 With shopt: .dotfile file1 file2 file3 Without shopt again: file1 file2 file3 [root@golem4 testdir]# du * 0 file1 0 file2 0 file3 [root@golem4 testdir]# du -a * 0 file1 0 file2 0 file3 [root@golem4 testdir]# du -a 0 ./file2 0 ./file3 0 ./.dotfile 0 ./file1 4 .
Hope that clarifies it a bit.
And when I say "files that start with a dot", that includes directories, devices, whatever (remember, essentially in Unixish systems, EVERYTHING is some type of a file). ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - To err is human, to moo bovine. - ----------------------------------------------------------------------
"ls .?*" is another "ls -A" equivalent.
Hidden files or directories are hidden. What part of that are some people missing. You have to tell the system to expose those "hidden" items for many operations such as listing files, which this abuse of "du" is showing. (Isn't "ls -l" simpler? I alias that to "ll" and "lla" is "ls -lA".
This thread is amusing and bemusing for the willful disregard for standard 'ix behavior since virtually the beginning of 'ix time.
{^_-}
On 20180313 16:28, Ed Greshko wrote:
On 03/14/18 05:22, Ed Greshko wrote:
The later command is basically creating a list of files (a shell function) and doing a "du" on each file individually.
Here is another example to show you what is happening....
[egreshko@meimei test-dir]$ ls stuff test test1 test2
[egreshko@meimei test-dir]$ cat stuff tes te te*
[egreshko@meimei test-dir]$ grep te stuff tes te te*
[egreshko@meimei test-dir]$ grep te* stuff [egreshko@meimei test-dir]$
But if I change to a directory which contains no files....
[egreshko@meimei test-dir]$ cd .hidden
[egreshko@meimei .hidden]$ ls
[egreshko@meimei .hidden]$ grep te* ../stuff tes te te*
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/14/18 08:20, jdow wrote:
"ls .?*" is another "ls -A" equivalent.
Hummm..... Not really? Since .. is matched and depending on where you are in the file system you may encounter this....
[egreshko@meimei test-dir]$ ls -A .hidden stuff test test1 test2
[egreshko@meimei test-dir]$ ls .?* ..: 20180309_213210.mp4 GPS m_g-large.jpg Pictures-silly US-Taxes CDC_Retirement.odt home.ovpn NYE Rachel wall DSCOVR ironsocket personal Taiwan-Tax Weather DS-Reports keys Pictures-meimei test-dir Garmin manuals Pictures-misty Trips
.hidden:
Hidden files or directories are hidden. What part of that are some people missing. You have to tell the system to expose those "hidden" items for many operations such as listing files, which this abuse of "du" is showing. (Isn't "ls -l" simpler? I alias that to "ll" and "lla" is "ls -lA".
This thread is amusing and bemusing for the willful disregard for standard 'ix behavior since virtually the beginning of 'ix time.
Could it be that some of the posters haven't been around as long as others? :-) :-)
On 03/13/2018 01:19 PM, Stephen Morris wrote:
It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
Having read this thread, I think you may still not grasp what the shell does and what command (du, in this case) does.
So, let's illustrate with the standard interview question: "How do you remove a file called '-r'?"
Let's say that you're perusing your filesystem, and you find that someone has created a file called "-r". You want to remove it. How? You might start by running "rm -r".
When you enter a command into a shell, it copies your input into a buffer. First, it breaks your input into words. It uses the value of the IFS variable to determine what characters separate words. In the above example, it will produce an array like ["rm", "-r"]. It will then do expansion of shell variables and globs (the details are in the EXPANSION section of "man bash"), though in this case there are neither in the command. Finally, it will search the directories in the PATH variable (and also its aliases, shell functions, and built-ins) for a command that matches the first word in the list. It will execute that program with its list as arguments. Argument 0 will be "rm" and argument 1 will be "-r".
That's what the shell does.
Now the command runs and parses its arguments. "rm" sees "-r" as an option argument. It doesn't find any non-option arguments (files to remove), so it tells you that an operand is missing.
That's what the command does.
You haven't removed the "-r" file, so you try again. Many people will try quoting the filename. They'll run "rm '-r'".
The shell breaks the input into words again. This time, it consumes the quotes, because they are meaningful to the shell. The word list is ["rm", "-r"], same as the last time. The outcome is the same.
Some people will try using a glob. They'll run "rm ?r" or "rm *r".
The shell breaks the input into words. This time, the list is ["rm, "?r"]. There are glob characters this time, so the shell expands those. "?r" matches a filename, so it gets replaced with "-r" and the word list is now ["rm", "-r"]. The list is the same as the last one, and the outcome is the same.
So, how does that apply to your situation?
When you run "du /path/*", the shell breaks that into words: ["du", "/path/*"]. Since there are globs present, the shell replaces those before it runs the command. It expands into ["du", "/path/a", "/path/b", ...]. The "*" isn't passed to the command, it's expanded by the shell.
When you enter "shopt -s dotglob", you're not modifying the behavior of "du", you're modifying the behavior of the shell. You're changing the way the shell handles globs, so that files with a dot prefix will be matched by globs. When dotglob is enabled, the shell expands the same word set differently. This time it might be ["du", "/path/.config", "/path/.local", "/path/a", ...]. "du" behaves exactly the same. It knows nothing of the dotglob option. It only reports the use for the files and directories that are given to it as arguments. In the standard configuration, the shell won't give dot-files as arguments when it expands "*", but it will when you enable dotglob.
(And if you want to remove a file named "-r" you need to use a complete path ("/path/-r"), or a relative path ("./-r") or tell rm to stop parsing option arguments by giving it the "--" argument, as in "rm -- -r".)
On Tue, 2018-03-13 at 17:20 -0700, jdow wrote:
Hidden files or directories are hidden. What part of that are some people missing. You have to tell the system to expose those "hidden" items for many operations such as listing files, which this abuse of "du" is showing.
The system has no concept of a hidden file. There is simply a convention that 'ls' and some other commands ignore names beginning with '.' by default, and this is incorporated into the Shell's interpretation of '*'. That's all.
poc
On 03/13/2018 03:54 PM, Stephen Morris wrote:
On 13/3/18 9:33 am, Robert Nichols wrote:
Whenever you have questions like this, you should run "set -x" in the shell to see exactly how commands are being invoked. (Run "set +x" to turn that off again.)
I'm not sure what the point of this command is relative to du, it seems to be more relevant when using commands that are aliases.
Having issued the command "set -x", if I issue the command du -abh /home/steve/workspace it just echos that command back, and if I issue the command du -abh /home/steve/workspace/* it echos the command with a list of sub-directories of workspace that du is going to do its work over.
Yes. And does that list include the parent directory or any names that begin with "."? How could you expect "du" to process any names not in that list? Perhaps you tried that in a directory that does not contain any dotfiles. Try it in one that _does_.
On 14/3/18 4:50 pm, Gordon Messmer wrote:
On 03/13/2018 01:19 PM, Stephen Morris wrote:
It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
Having read this thread, I think you may still not grasp what the shell does and what command (du, in this case) does.
So, let's illustrate with the standard interview question: "How do you remove a file called '-r'?"
Let's say that you're perusing your filesystem, and you find that someone has created a file called "-r". You want to remove it. How? You might start by running "rm -r".
When you enter a command into a shell, it copies your input into a buffer. First, it breaks your input into words. It uses the value of the IFS variable to determine what characters separate words. In the above example, it will produce an array like ["rm", "-r"]. It will then do expansion of shell variables and globs (the details are in the EXPANSION section of "man bash"), though in this case there are neither in the command. Finally, it will search the directories in the PATH variable (and also its aliases, shell functions, and built-ins) for a command that matches the first word in the list. It will execute that program with its list as arguments. Argument 0 will be "rm" and argument 1 will be "-r".
That's what the shell does.
Now the command runs and parses its arguments. "rm" sees "-r" as an option argument. It doesn't find any non-option arguments (files to remove), so it tells you that an operand is missing.
That's what the command does.
You haven't removed the "-r" file, so you try again. Many people will try quoting the filename. They'll run "rm '-r'".
The shell breaks the input into words again. This time, it consumes the quotes, because they are meaningful to the shell. The word list is ["rm", "-r"], same as the last time. The outcome is the same.
Some people will try using a glob. They'll run "rm ?r" or "rm *r".
The shell breaks the input into words. This time, the list is ["rm, "?r"]. There are glob characters this time, so the shell expands those. "?r" matches a filename, so it gets replaced with "-r" and the word list is now ["rm", "-r"]. The list is the same as the last one, and the outcome is the same.
So, how does that apply to your situation?
When you run "du /path/*", the shell breaks that into words: ["du", "/path/*"]. Since there are globs present, the shell replaces those before it runs the command. It expands into ["du", "/path/a", "/path/b", ...]. The "*" isn't passed to the command, it's expanded by the shell.
When you enter "shopt -s dotglob", you're not modifying the behavior of "du", you're modifying the behavior of the shell. You're changing the way the shell handles globs, so that files with a dot prefix will be matched by globs. When dotglob is enabled, the shell expands the same word set differently. This time it might be ["du", "/path/.config", "/path/.local", "/path/a", ...]. "du" behaves exactly the same. It knows nothing of the dotglob option. It only reports the use for the files and directories that are given to it as arguments. In the standard configuration, the shell won't give dot-files as arguments when it expands "*", but it will when you enable dotglob.
(And if you want to remove a file named "-r" you need to use a complete path ("/path/-r"), or a relative path ("./-r") or tell rm to stop parsing option arguments by giving it the "--" argument, as in "rm -- -r".)
I think the issue is my lack of knowledge. I wasn't aware the "*" was a shell specific function, I thought /home/steve/workspace and /home/steve/workspace/* were just different ways of specifying the same thing, which was why I was confused by the output I was getting.
regards,
Steve
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 03/14/2018 01:49 PM, Stephen Morris wrote:
On 14/3/18 4:50 pm, Gordon Messmer wrote:
On 03/13/2018 01:19 PM, Stephen Morris wrote:
It just seems counter intuitive to me to have to issue another command (even if one knows of its existence) to get a command to function "properly".
Having read this thread, I think you may still not grasp what the shell does and what command (du, in this case) does.
So, let's illustrate with the standard interview question: "How do you remove a file called '-r'?"
Let's say that you're perusing your filesystem, and you find that someone has created a file called "-r". You want to remove it. How? You might start by running "rm -r".
When you enter a command into a shell, it copies your input into a buffer. First, it breaks your input into words. It uses the value of the IFS variable to determine what characters separate words. In the above example, it will produce an array like ["rm", "-r"]. It will then do expansion of shell variables and globs (the details are in the EXPANSION section of "man bash"), though in this case there are neither in the command. Finally, it will search the directories in the PATH variable (and also its aliases, shell functions, and built-ins) for a command that matches the first word in the list. It will execute that program with its list as arguments. Argument 0 will be "rm" and argument 1 will be "-r".
That's what the shell does.
Now the command runs and parses its arguments. "rm" sees "-r" as an option argument. It doesn't find any non-option arguments (files to remove), so it tells you that an operand is missing.
That's what the command does.
You haven't removed the "-r" file, so you try again. Many people will try quoting the filename. They'll run "rm '-r'".
The shell breaks the input into words again. This time, it consumes the quotes, because they are meaningful to the shell. The word list is ["rm", "-r"], same as the last time. The outcome is the same.
Some people will try using a glob. They'll run "rm ?r" or "rm *r".
The shell breaks the input into words. This time, the list is ["rm, "?r"]. There are glob characters this time, so the shell expands those. "?r" matches a filename, so it gets replaced with "-r" and the word list is now ["rm", "-r"]. The list is the same as the last one, and the outcome is the same.
So, how does that apply to your situation?
When you run "du /path/*", the shell breaks that into words: ["du", "/path/*"]. Since there are globs present, the shell replaces those before it runs the command. It expands into ["du", "/path/a", "/path/b", ...]. The "*" isn't passed to the command, it's expanded by the shell.
When you enter "shopt -s dotglob", you're not modifying the behavior of "du", you're modifying the behavior of the shell. You're changing the way the shell handles globs, so that files with a dot prefix will be matched by globs. When dotglob is enabled, the shell expands the same word set differently. This time it might be ["du", "/path/.config", "/path/.local", "/path/a", ...]. "du" behaves exactly the same. It knows nothing of the dotglob option. It only reports the use for the files and directories that are given to it as arguments. In the standard configuration, the shell won't give dot-files as arguments when it expands "*", but it will when you enable dotglob.
(And if you want to remove a file named "-r" you need to use a complete path ("/path/-r"), or a relative path ("./-r") or tell rm to stop parsing option arguments by giving it the "--" argument, as in "rm -- -r".)
I think the issue is my lack of knowledge. I wasn't aware the "*" was a shell specific function, I thought /home/steve/workspace and /home/steve/workspace/* were just different ways of specifying the same thing, which was why I was confused by the output I was getting.
That's sort of it. In the first ("/home/steve/workspace"), the shell doesn't do any expansion or pattern matching since there's no metacharacters in it. In the second ("/home/steve/workspace/*"), you DO have a metacharacter (the "*") so the shell expands it, does the match and passes the list to du. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - 500: Internal Fortune Cookie Error - ----------------------------------------------------------------------
-
Ed Greshko -
Ed Greshko -
Gordon Messmer -
JD -
jdow -
Patrick O'Callaghan -
Philip Rhoades -
Rick Stevens -
Robert Nichols -
Stephen Morris