Disk usage error

[William John Murray]

Hello all,
         Can anyone help me with a disk usage problem? I have a disk
partition of 65GB in LVM; df says:

/dev/dm-0             65570580  60494828   1744888  98% /

However, if I use either du or Baobab they reckon the directories in it
add up to 30Gb or so. As it is my root directory, and there are various
others it is a little difficult to get the total, so I booted under
liveUSB and saw exactly the same - 30Gb used, but 98% full.
   So something is stealing half my disk. If I try to write more it is
out of space.
   Any ideas how I get my space back? fsck reports the disk is clean.
   Thanks,
         Bill

[Mogens Kjaer]

On 01/20/2010 07:49 AM, William John Murray wrote:
> Hello all,
>           Can anyone help me with a disk usage problem? I have a disk
> partition of 65GB in LVM; df says:
>
> /dev/dm-0             65570580  60494828   1744888  98% /
>
> However, if I use either du or Baobab they reckon the directories in it
> add up to 30Gb or so.

How do you give the du command?

Any /.hidden folders in root?

Mogens

[Rick Stevens]

On 01/19/2010 10:49 PM, William John Murray wrote:
> Hello all,
>           Can anyone help me with a disk usage problem? I have a disk
> partition of 65GB in LVM; df says:
>
> /dev/dm-0             65570580  60494828   1744888  98% /
>
> However, if I use either du or Baobab they reckon the directories in it
> add up to 30Gb or so. As it is my root directory, and there are various
> others it is a little difficult to get the total, so I booted under
> liveUSB and saw exactly the same - 30Gb used, but 98% full.
>     So something is stealing half my disk. If I try to write more it is
> out of space.
>     Any ideas how I get my space back? fsck reports the disk is clean.

As someone else said, have a look for ".*" files (files whose names
start with a dot).  Also keep in mind that du and df look at "sparse"
files differently. df shows what's been allocated and du shows what's
actually used.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer                      [e-mail] -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-                 All generalizations are false.                     -
----------------------------------------------------------------------

[Tony Nelson]

On 10-01-20 12:51:30, Rick Stevens wrote:
 ...
> ... Also keep in mind that du and df look at "sparse"
> files differently. df shows what's been allocated and du shows what's
> actually used.

Are you sure that those two things are different?  See du's --apparant-
size option (not the default), and note that df reports the free blocks 
on the disk (parse files don't have blocks allocated for their empty 
parts).

The OP has some other problem.

[Tony Nelson]

On 10-01-20 01:49:37, William John Murray wrote:
> Hello all,
>          Can anyone help me with a disk usage problem? I have a disk
> partition of 65GB in LVM; df says:
>
> /dev/dm-0             65570580  60494828   1744888  98% /
>
> However, if I use either du or Baobab they reckon the directories in
> it add up to 30Gb or so. As it is my root directory, and there are
> various others it is a little difficult to get the total, so I booted 
> under liveUSB and saw exactly the same - 30Gb used, but 98% full.
>    So something is stealing half my disk. If I try to write more it 
> is out of space.
>    Any ideas how I get my space back? fsck reports the disk is clean.

If the filesystem seems clean you need to use the -f option to force 
the check to actually happen.

If that doesn't help, you could copy everything to someplace else, make 
a new filesystem, and copy back again, assuming you have a "someplace 
else".  cpio or tar would work, booted from liveUSB.

[William John Murray]

On Wed, 2010-01-20 at 07:49 +0100, William John Murray wrote:
> Hello all,
>          Can anyone help me with a disk usage problem? I have a disk
> partition of 65GB in LVM; df says:
>
> /dev/dm-0             65570580  60494828   1744888  98% /
>
> However, if I use either du or Baobab they reckon the directories in it
> add up to 30Gb or so. As it is my root directory, and there are various
> others it is a little difficult to get the total, so I booted under
> liveUSB and saw exactly the same - 30Gb used, but 98% full.
>    So something is stealing half my disk. If I try to write more it is
> out of space.
>    Any ideas how I get my space back? fsck reports the disk is clean.
>    Thanks,
>          Bill

Thanks for all the suggestions. The dead links (lsof | grep deleted) is
very interesting - I have 49 of them, which seems bad, mostly /tmp files
from a
 "gnome-terminal -ssh XXX.YYY"
which are all 3Mb. However, this is only 150MB, I am hunting for 30GB.
Rebooting should remove all such, and makes no difference.

Chris Smart asked what 'du' I had tried. 'It was 'du -sh'. Note that it
agreed well with baobob so I thought that made it trustworthy.

Any more ideas? I guess I could copy the filesystem contents to
another disk and back, I have the space for that, but it seems a little
over-the-top. And it may well come back...
    Bill

[Mogens Kjaer]

On 01/21/2010 02:01 PM, William John Murray wrote:
...
> Any more ideas?

What happens if you read the whole file system:

cd /
tar c --one-file-system -f - .|dd of=/dev/null bs=1M

dd should tell you the size of the generated tar
archive.

It may take a while...

Mogens

[Mogens Kjaer]

On 01/21/2010 02:23 PM, Mogens Kjaer wrote:
> tar c --one-file-system -f - .|dd of=/dev/null bs=1M

Hm, I forgot the -S option:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_mk-lv_root
                       222G   48G  163G  23% /
# tar c --one-file-system -S -f - .|dd of=/dev/null bs=1M
...
50122147840 bytes (50 GB) copied, 1737.65 s, 28.8 MB/s

Without the -S I get:
134616043520 bytes (135 GB) copied, 1394.3 s, 96.5 MB/s

as I have some large sparse files.

48G and 50G are pretty close...

Mogens

[Robert Nichols]

William John Murray wrote:
> On Wed, 2010-01-20 at 07:49 +0100, William John Murray wrote:
> > Hello all,
> >          Can anyone help me with a disk usage problem? I have a disk
> > partition of 65GB in LVM; df says:
> >
> > /dev/dm-0             65570580  60494828   1744888  98% /
> >
> > However, if I use either du or Baobab they reckon the directories in it
> > add up to 30Gb or so. As it is my root directory, and there are various
> > others it is a little difficult to get the total, so I booted under
> > liveUSB and saw exactly the same - 30Gb used, but 98% full.
> >    So something is stealing half my disk. If I try to write more it is
> > out of space.
> >    Any ideas how I get my space back? fsck reports the disk is clean.
> >    Thanks,
> >          Bill
>
> Thanks for all the suggestions. The dead links (lsof | grep deleted) is
> very interesting - I have 49 of them, which seems bad, mostly /tmp files
> from a
>  "gnome-terminal -ssh XXX.YYY"
> which are all 3Mb. However, this is only 150MB, I am hunting for 30GB.
> Rebooting should remove all such, and makes no difference.
>
> Chris Smart asked what 'du' I had tried. 'It was 'du -sh'. Note that it
> agreed well with baobob so I thought that made it trustworthy.
>
> Any more ideas? I guess I could copy the filesystem contents to
> another disk and back, I have the space for that, but it seems a little
> over-the-top. And it may well come back...

Anything that is hidden below an active mount point in that file system
will not be seen by 'du'.  If you boot with "init=/bin/sh" as a kernel
parameter you can look at the file system before anything (not even /dev
or /proc) has been mounted.  Caution: You will be working in a very
restricted and unfamiliar environment.  I've never tried dealing with
LVM that way.  You might be better off booting from a rescue disk and
examining your file system that way.

[Don Quixote de la Manvha]

On Thu, Jan 21, 2010 at 5:01 AM, William John Murray
 wrote:
> Any more ideas? I guess I could copy the filesystem contents to
> another disk and back, I have the space for that, but it seems a little
> over-the-top. And it may well come back...

That would have the added benefit of completely defragmenting your filesystem.

I know it's off-topic, but it's really the best way to defragment;
defragmentation tools are generally unable to defragment everything.

Don Quixote

[Florian Gerstenberger]

On 01/20/2010 07:49 AM, William John Murray wrote:
> Hello all,
>          Can anyone help me with a disk usage problem? I have a disk
> partition of 65GB in LVM; df says:
>
> /dev/dm-0             65570580  60494828   1744888  98% /
>
> However, if I use either du or Baobab they reckon the directories in it
> add up to 30Gb or so. As it is my root directory, and there are various
> others it is a little difficult to get the total, so I booted under
> liveUSB and saw exactly the same - 30Gb used, but 98% full.
>    So something is stealing half my disk. If I try to write more it is
> out of space.
>    Any ideas how I get my space back? fsck reports the disk is clean.
>    Thanks,
>          Bill

Hi Bill,

could it be that the "reserved Blocks percent" Option of an ext2/3/4 FS
is the matter for that behaviour?

Find out how it is set by using dumpe2fs /dev/ice

You can set it to zero by using tune2fs -m 0 /dev/ice.

Regards, Florian

[William John Murray]

On Wed, 2010-01-20 at 07:49 +0100, William John Murray wrote:
> Hello all,
>          Can anyone help me with a disk usage problem? I have a disk
> partition of 65GB in LVM; df says:
>
> /dev/dm-0             65570580  60494828   1744888  98% /
>
> However, if I use either du or Baobab they reckon the directories in it
> add up to 30Gb or so. As it is my root directory, and there are various
> others it is a little difficult to get the total, so I booted under
> liveUSB and saw exactly the same - 30Gb used, but 98% full.
>    So something is stealing half my disk. If I try to write more it is
> out of space.
>    Any ideas how I get my space back? fsck reports the disk is clean.
>    Thanks,
>          Bill

Thanks to everyone,
          I found it, using the idea of making a tar file - thanks
Mogens. [I don't know why booting of a USB system didn't have the same
affect. Maybe I was careless]
   The tar file was 60Gb, so I actually made it and inspected it and
found that the problem was:
   * I have a big USB disk
   * I Make a nightly backup to this disk of /home /etc for 2 machines.
   * As it is USB. gnome mounts it under /media WHEN I LOG IN
   * If the machine reboots without my log in the USB is not mounted
   * The autobackup rsync looks for and fails to find /media/backup  
          - so it makes a new one WHICH IS in /
   * I did not regard and of the /media/XXXX directories as part of / so
missed it.

So now I better fix the mounting of USB drives with /etc/fstab,
   Bill

[Robert Nichols]

William John Murray wrote:
> On Wed, 2010-01-20 at 07:49 +0100, William John Murray wrote:
> Thanks to everyone,
>           I found it, using the idea of making a tar file - thanks
> Mogens. [I don't know why booting of a USB system didn't have the same
> affect. Maybe I was careless]
>    The tar file was 60Gb, so I actually made it and inspected it and
> found that the problem was:
> I have a big USB disk
> I Make a nightly backup to this disk of /home /etc for 2 machines.
> As it is USB. gnome mounts it under /media WHEN I LOG IN
> If the machine reboots without my log in the USB is not mounted
> The autobackup rsync looks for and fails to find /media/backup  
>      - so it makes a new one WHICH IS in /

The way I protect against that happening is to give my backup prog a
destination that is actually a symlink to a subdirectory on the
external drive.  If the drive is not mounted the backup process
complains about the broken symlink and fails.  Use of a subdirectory
on the external drive is to protect against the case where the drive
is not mounted but the mount point directory in /media somehow did
not get deleted.

[Will Yonker]

> I did not regard and of the /media/XXXX directories as part of / so
>
> missed it.
>
> So now I better fix the mounting of USB drives with /etc/fstab,

Hi Bill,

I created a label for my backup device/partition.  Then I wrote this into
my backup script:

mount -L Backup /home/data/backup
RETURN=`mount|grep /home/data/backup|wc -l`
if [ $RETURN -ne 1 ]
then
    logger "Backup failed due to unmounted backup device."
    exit 1
fi

---
Will Y.

[William John Murray]

Thanks Will,
       I have fixed this now to automount the USB drive, which is better
because it means the backup is made.
  Thanks anyway,
       Bill

On Wed, 2010-02-10 at 15:17 -0500, [e-mail] wrote:
> > I did not regard and of the /media/XXXX directories as part of / so
> >
> > missed it.
> >
> > So now I better fix the mounting of USB drives with /etc/fstab,
>
> Hi Bill,
>
> I created a label for my backup device/partition.  Then I wrote this into
> my backup script:
>
> mount -L Backup /home/data/backup
> RETURN=`mount|grep /home/data/backup|wc -l`
> if [ $RETURN -ne 1 ]
> then
>     logger "Backup failed due to unmounted backup device."
>     exit 1
> fi
>
> Will Y.

[Will Yonker]

> Thanks Will,
>        I have fixed this now to automount the USB drive, which is better
> because it means the backup is made.
>   Thanks anyway,
>        Bill
>
> On Wed, 2010-02-10 at 15:17 -0500, [e-mail] wrote:
> > > I did not regard and of the /media/XXXX directories as part of /
> >
> > so
> > > missed it.
> > >
> > > So now I better fix the mounting of USB drives with /etc/fstab,
> >
> > Hi Bill,
> >
> > I created a label for my backup device/partition.  Then I wrote this
> > into
> > my backup script:
> >
> > mount -L Backup /home/data/backup
> > RETURN=`mount|grep /home/data/backup|wc -l`
> > if [ $RETURN -ne 1 ]
> > then
> >     logger "Backup failed due to unmounted backup device."
> >     exit 1
> > fi

Hi Bill,

Maybe I'm paranoid but I don't want to have my backup device mounted all
the time.  So I only mount it when I need to write to it.  In that way, if
I do something stupid like run an unlimited search or delete, the backup
device will not be affected.

My next thought was to put the drive to sleep right after I unmount it. 
Has anyone done that with something like:

hdparm -Y /dev/sd{x}

I'm hoping to be a little green, reduce wear on the drive (extend the
life) and protect my data all at the same time.

Is this a good or bad idea?

---
Will Y.

[Roberto Ragusa]

William John Murray wrote:
> The autobackup rsync looks for and fails to find /media/backup  
>      - so it makes a new one WHICH IS in /

See the positive side.
You unknowingly had an extra backup for some time. :-)

(but, being on the same disk as the original data, it was a poor one)