Filesystem problems

[Dan Track]

Hi

I've got a server with 59G /opt partiition. Currently df -h reports
that I'm using 59G, however if I run "du -hs" on /opt the size of the
indidivaual dir's sitting in /opt are much smaller and don't sum up to
59G. So my question is what is making the server think that more disk
space is being used?

Any help would be appreciated.

Kind Regards
Dan

[Steve Siegfried]

Dan Track wrote:
>  
>  Hi
>  
>  I've got a server with 59G /opt partiition. Currently df -h reports
>  that I'm using 59G, however if I run "du -hs" on /opt the size of the
>  indidivaual dir's sitting in /opt are much smaller and don't sum up to
>  59G. So my question is what is making the server think that more disk
>  space is being used?
>  
>  Any help would be appreciated.
>   
It might be useful if you showed the du & df output, and that you should
remember to run du as root in order to see all the directories and not
just the ones you have permission to read as a normal user.

Suggest also that you read the fedora archives for the topic "df cmd
calculates wrong percent usage (FC6)".

-S

[Phil Meyer]

Steve Siegfried wrote:
>  Dan Track wrote:
>    
> > Hi
> >
> > I've got a server with 59G /opt partiition. Currently df -h reports
> > that I'm using 59G, however if I run "du -hs" on /opt the size of the
> > indidivaual dir's sitting in /opt are much smaller and don't sum up to
> > 59G. So my question is what is making the server think that more disk
> > space is being used?
> >
> > Any help would be appreciated.
> >
> >     
>
>  It might be useful if you showed the du & df output, and that you should
>  remember to run du as root in order to see all the directories and not
>  just the ones you have permission to read as a normal user.
>
>  Suggest also that you read the fedora archives for the topic "df cmd
>  calculates wrong percent usage (FC6)".
>
>  -S
>
>     
Also remember that space is not freed when a rm command is issued.  It 
is freed when the file is afterwards closed.

It is quite possible to have files open that do not show in the 
directory listing.

Imagine a large file such as a database file that is being used by a 
mysql process.  Lets also assume that this mysql process is involved in 
heavy access to that very file doing a report that takes hours.

It is possible to rm that file out from under the process.  However, as 
long as that process holds the file open, nothing has happened except 
the inode has been removed from the directory listing.  At that point, 
df does not show any more space available than before the rm command.  
That is because the kernel driver for the file system does not need to 
free up the space until the file is closed.

Different types of file systems behave differently, but most of them 
will behave this way.

Never trust df on a busy file system except at boot up. :)


Good luck!

[Dan Track]

On 2/27/07, Steve Siegfried <sos(a)zjod.net&gt; wrote:
>  Dan Track wrote:
>  >
>  > Hi
>  >
>  > I've got a server with 59G /opt partiition. Currently df -h reports
>  > that I'm using 59G, however if I run "du -hs" on /opt the size of the
>  > indidivaual dir's sitting in /opt are much smaller and don't sum up to
>  > 59G. So my question is what is making the server think that more disk
>  > space is being used?
>  >
>  > Any help would be appreciated.
>  >
>
>  It might be useful if you showed the du & df output, and that you should
>  remember to run du as root in order to see all the directories and not
>  just the ones you have permission to read as a normal user.
>
>  Suggest also that you read the fedora archives for the topic "df cmd
>  calculates wrong percent usage (FC6)".
>
>  -S
>
>  --
>  fedora-list mailing list
>  fedora-list(a)redhat.com
>  To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list
>  
Hi

Thanks everyone for your replies. Here's the relevant output. Is there
some tests I can run to see what is going on?

 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     3.0G  1.9G  967M  67% /
/dev/cciss/c0d0p1     147M   15M  125M  11% /boot
/dev/cciss/c0d0p7      59G   53G  3.3G  95% /opt
none                 1007M     0 1007M   0% /dev/shm
/dev/cciss/c0d0p2     3.0G  2.5G  358M  88% /var

 du -hs /opt/
27G     /opt

Hope this helps
Dan

[Alexander Apprich]

Hi Dan,

Dan Track wrote:
>  On 2/27/07, Steve Siegfried <sos(a)zjod.net&gt; wrote: 
[snip]
> >
>  Hi
>  
>  Thanks everyone for your replies. Here's the relevant output. Is there
>  some tests I can run to see what is going on?
>  
>  df -h
>  Filesystem            Size  Used Avail Use% Mounted on
>  /dev/cciss/c0d0p3     3.0G  1.9G  967M  67% /
>  /dev/cciss/c0d0p1     147M   15M  125M  11% /boot
>  /dev/cciss/c0d0p7      59G   53G  3.3G  95% /opt
>  none                 1007M     0 1007M   0% /dev/shm
>  /dev/cciss/c0d0p2     3.0G  2.5G  358M  88% /var
>  
>  du -hs /opt/
>  27G     /opt
>   
I've seen this before on a suse box where the user had couple downloads
running that he canceled but the browser didn't let go of the files.
His filesystem was filled 100%. After he killed his browser df -hl
showed the correct information. Maybe you can see what's going on by
running

    /usr/sbin/lsof /opt | less

on your box.

Alex

[shhgs]

Hi, Dan

Maybe someone has removed a file while a process is still writing to
it. Try fcsk.

G.S. Huang

On 2/28/07, Alexander Apprich <a.apprich(a)science-computing.de&gt; wrote:
>  Hi Dan,
>
>  Dan Track wrote:
>  > On 2/27/07, Steve Siegfried <sos(a)zjod.net&gt; wrote:
>  [snip]
>  >>
>  > Hi
>  >
>  > Thanks everyone for your replies. Here's the relevant output. Is there
>  > some tests I can run to see what is going on?
>  >
>  > df -h
>  > Filesystem            Size  Used Avail Use% Mounted on
>  > /dev/cciss/c0d0p3     3.0G  1.9G  967M  67% /
>  > /dev/cciss/c0d0p1     147M   15M  125M  11% /boot
>  > /dev/cciss/c0d0p7      59G   53G  3.3G  95% /opt
>  > none                 1007M     0 1007M   0% /dev/shm
>  > /dev/cciss/c0d0p2     3.0G  2.5G  358M  88% /var
>  >
>  > du -hs /opt/
>  > 27G     /opt
>  >
>
>  I've seen this before on a suse box where the user had couple downloads
>  running that he canceled but the browser didn't let go of the files.
>  His filesystem was filled 100%. After he killed his browser df -hl
>  showed the correct information. Maybe you can see what's going on by
>  running
>
>     /usr/sbin/lsof /opt | less
>
>  on your box.
>
>  Alex
>
>  --
>  fedora-list mailing list
>  fedora-list(a)redhat.com
>  To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list
>

[Steve Siegfried]

This topic was handled off-list, which was probably a mistake.  Thus,
for you edification and enjoyment....


Basically, files can temporarily "disappear" from the file-system when a
program:
        - opens a file with fopen/open/etc, and then
        - unlinks the file (i.e.: removes the file, see unlink(2)) while
          keeping the the file open.

Note that when this happens, the disk space isn't really available
for reallocation until the program ends and/or closes the file
(fclose/close/etc).  However, because of the way df and du calculate used
and free file-system space, they can disagree under these conditions.

Also note that USED + AVAIL != SIZE for df because some of the space on
a disk file-system is reserved for use by the system (for superblocks,
"reserved", etc).

So how did you get here?
        - downloads that got canceled
        - some program that's messed up and poops all over its log or
          output file (that's been unlinked but is still open)
        - a log rotation that went awry
        - processes listed as "zombie" in ps/top output
        - a make where the user hit cntl-C (unlikely, make's pretty
          good about this kind of thing)
	- a daemon or long running program that poorly manages files
        - etc

If you can find and zap the PID that's tying up the disk space, you'll
probably see the disk space reappear.  If not, you can reboot the server.
I'd suggest rebooting and then repeating the df/du immediately afterward.

The quickest way to find PIDs that have deleted (i.e.: NLINK == 0) but
still open files is
	$ su
	# lsof +L 1 /opt | grep deleted

BTW: lsof is in the lsof rpm.

If this _still_ doesn't solve the issue (to say within 5%), then you'll
need to unmount the drive and run "fsck -f /dev/cciss/c0d0p7".  Note that
while fsck will fix damaged filesystems, it won't find/fix files that lsof
reports as deleted but still open.

-S


shhgs wrote:
>  
>  Hi, Dan
>  
>  Maybe someone has removed a file while a process is still writing to
>  it. Try fcsk.
>  
>  G.S. Huang
>  
>  On 2/28/07, Alexander Apprich <a.apprich(a)science-computing.de&gt; wrote:
>  > Hi Dan,
>  >
>  > Dan Track wrote:
>  > > On 2/27/07, Steve Siegfried <sos(a)zjod.net&gt; wrote:
>  > [snip]
>  > >>
>  > > Hi
>  > >
>  > > Thanks everyone for your replies. Here's the relevant output. Is there
>  > > some tests I can run to see what is going on?
>  > >
>  > > df -h
>  > > Filesystem            Size  Used Avail Use% Mounted on
>  > > /dev/cciss/c0d0p3     3.0G  1.9G  967M  67% /
>  > > /dev/cciss/c0d0p1     147M   15M  125M  11% /boot
>  > > /dev/cciss/c0d0p7      59G   53G  3.3G  95% /opt
>  > > none                 1007M     0 1007M   0% /dev/shm
>  > > /dev/cciss/c0d0p2     3.0G  2.5G  358M  88% /var
>  > >
>  > > du -hs /opt/
>  > > 27G     /opt
>  > >
>  >
>  > I've seen this before on a suse box where the user had couple downloads
>  > running that he canceled but the browser didn't let go of the files.
>  > His filesystem was filled 100%. After he killed his browser df -hl
>  > showed the correct information. Maybe you can see what's going on by
>  > running
>  >
>  >    /usr/sbin/lsof /opt | less
>  >
>  > on your box.
>  >
>  > Alex