Hi all, A today's mail from one of the new comer brings me this question again in my mind. Linux disk defragmenter. Does it really not needed?
I've been googling around and find that this matter has been discussed as early as 1998. And it seems that the only distro that provides a defragmenter program is debian.
There are several way of fixing a heavy defragmented disk in Linux, but the easiest way is to copy all of the content of the partition into another place, completely erase that partition, and copy back the content.
My own experience shows me just that. My /home partition was almost full with only 2% freespace. During that time, my Kmail became very slow such as when downloading email or when I moved between mail folders. The harddisk was just spinning all the time.
Then I copy all my files and mails from the /home partition and move them all to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
However, still the question remains. If Linux ext3 doesn't need defragmenter, and able to defrag itself, what is the process name? And when does it run? Can I see it in action? Is there an utility to see on what percentage my current defragmentation? I tried fschk but no luck.
Thank you,
On Fri, 2005-12-23 at 14:32 +0700, Fajar Priyanto wrote:
My own experience shows me just that. My /home partition was almost full with only 2% freespace. During that time, my Kmail became very slow such as when downloading email or when I moved between mail folders. The harddisk was just spinning all the time.
Most hard drives spin all the time, anyway... ;-)
Then I copy all my files and mails from the /home partition and move them all to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
A too full drive is a problem in itself, defragging a drive (on any system) in that condition is only rescheduling the inevitable.
From my experience at least, defragmenters are not necessary. What I think
was happening in your case is that your system was probably using alout of virtual memory, and so when it was running low on disk space, it couldn't do it paging to virtual memory as efficiently as before (just a theory)
On 12/23/05, Fajar Priyanto fajarpri@cbn.net.id wrote:
Hi all, A today's mail from one of the new comer brings me this question again in my mind. Linux disk defragmenter. Does it really not needed?
I've been googling around and find that this matter has been discussed as early as 1998. And it seems that the only distro that provides a defragmenter program is debian.
There are several way of fixing a heavy defragmented disk in Linux, but the easiest way is to copy all of the content of the partition into another place, completely erase that partition, and copy back the content.
My own experience shows me just that. My /home partition was almost full with only 2% freespace. During that time, my Kmail became very slow such as when downloading email or when I moved between mail folders. The harddisk was just spinning all the time.
Then I copy all my files and mails from the /home partition and move them all to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
However, still the question remains. If Linux ext3 doesn't need defragmenter, and able to defrag itself, what is the process name? And when does it run? Can I see it in action? Is there an utility to see on what percentage my current defragmentation? I tried fschk but no luck.
Thank you,
Fajar Priyanto | Reg'd Linux User #327841 | Linux tutorial http://linux2.arinet.org 14:32:29 up 1:32, 2.6.14-1.1653_FC4 GNU/Linux Let's use OpenOffice. http://www.openoffice.org
-- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list
-- As a boy I jumped through Windows, as a man I play with Penguins.
Fajar Priyanto wrote:
Hi all, A today's mail from one of the new comer brings me this question again in my mind. Linux disk defragmenter. Does it really not needed?
Very good question. Opinions vary. Personally, I think that part of the reason some people believe that disc defragging is not necessary with Linux is the myth that Linux is just better than the other OS in every way.
I've been googling around and find that this matter has been discussed as early as 1998. And it seems that the only distro that provides a defragmenter program is debian.
"Hotly debated" might be a better phrase than the word "discussed".
There are several way of fixing a heavy defragmented disk in Linux, but the
I guess you meant "heavily fragmented".
[snip]
to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
Just freeing up disc is a way to speed up ext3, so I am told.
However, still the question remains. If Linux ext3 doesn't need defragmenter, and able to defrag itself, what is the process name? And when does it run?
Umm, I believe the argument is not that it defrags itself, just that the type of fragmentation it enjoys does not affect performance. Some sort of fertilizer[*], if you ask me.
Can I see it in action? Is there an utility to see on what percentage my current defragmentation? I tried fschk but no luck.
Nah.
[TONGUE IN CHEEK MODE ON]
The ext3 file system is so much superior to other file systems, that it runs as slowly as a fully fragmented disc ALL the time, even when everything is fully contiguous. It's equally slow.
[TONGUE IN CHEEK MODE OFF]
[*] Bull shit and horse shit come to mind.
Mike
On Fri, 2005-12-23 at 10:24 -0600, Mike McCarty wrote:
Umm, I believe the argument is not that it defrags itself, just that the type of fragmentation it enjoys does not affect performance. Some sort of fertilizer[*], if you ask me.
...[snip]...
The ext3 file system is so much superior to other file systems, that it runs as slowly as a fully fragmented disc ALL the time, even when everything is fully contiguous. It's equally slow.
I suppose there could be some truth in that. If it was equally slow, you wouldn't notice the difference. ;-) And if you were comparing it to an OS that was really really bad with fragmented drives...
Since hard drives act in a non-linear fashion (data scattered about the drive, where the drive always has to seek around the hardware), unlike a tape (where data access has to work in a sequential manner for physical reasons), and modern hard drives are quite nippy, it may well be that you don't notice a small bit of hunting around.
Looking around at various explanations, some reasons why you might not *usually* notice fragmentation seem quite straight forward:
e.g. "The ext2 and ext3 file systems most often used on Linux systems also attempt to keep fragmentation at a minimum. These file systems keep all blocks in a file close together. How they do this is by preallocating disk data blocks to regular files before they are actually used. Because of this, when a file increases in size, several adjacent blocks are already reserved, reducing file fragmentation. It is, therefore, seldom necessary to analyze the amount of fragmentation on a Linux system, never mind actually run a defragment command. An exception exists for files that are constantly appended to as the reserved blocks will only last so long."
Snipped from http://www.itworld.com/Comp/3380/nls_unixfrag040929/
At 2:32 PM +0700 12/23/05, Fajar Priyanto wrote:
Hi all, A today's mail from one of the new comer brings me this question again in my mind. Linux disk defragmenter. Does it really not needed?
I've been googling around and find that this matter has been discussed as early as 1998. And it seems that the only distro that provides a defragmenter program is debian.
There are several way of fixing a heavy defragmented disk in Linux, but the easiest way is to copy all of the content of the partition into another place, completely erase that partition, and copy back the content.
My own experience shows me just that. My /home partition was almost full with only 2% freespace. During that time, my Kmail became very slow such as when downloading email or when I moved between mail folders. The harddisk was just spinning all the time.
Then I copy all my files and mails from the /home partition and move them all to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
However, still the question remains. If Linux ext3 doesn't need defragmenter, and able to defrag itself, what is the process name? And when does it run? Can I see it in action? Is there an utility to see on what percentage my current defragmentation? I tried fschk but no luck.
The opinion that EXT2 doesn't need defragmenting is based on only a filesystem-level view of the problem, and doesn't consider data read and write performance. EXT2 does make an effort to keep data only a short seek away ("clustered"). With this clustering, the filesystem operations of adding, removing, extending, and shortening files are not much affected by fragmentation.
With EXT3 (journalling), which always writes data to a new place, updates the filesystem info, and then frees the old data (roughly speaking), fragmentation is a way of life, and there isn't much to be done about it. Clustering helps by keeping the seeks relatively short, if there is space nearby.
When you have only 2% free, it's just about certain that the free space is a long way away from the rest of the data in a file. Just deleting to get 20% free would probably have fixed your problem. ____________________________________________________________________ TonyN.:' mailto:tonynelson@georgeanelson.com ' http://www.georgeanelson.com/
Tony Nelson wrote:
At 2:32 PM +0700 12/23/05, Fajar Priyanto wrote:
Hi all, A today's mail from one of the new comer brings me this question again in my mind. Linux disk defragmenter. Does it really not needed?
I've been googling around and find that this matter has been discussed as early as 1998. And it seems that the only distro that provides a defragmenter program is debian.
There are several way of fixing a heavy defragmented disk in Linux, but the easiest way is to copy all of the content of the partition into another place, completely erase that partition, and copy back the content.
My own experience shows me just that. My /home partition was almost full with only 2% freespace. During that time, my Kmail became very slow such as when downloading email or when I moved between mail folders. The harddisk was just spinning all the time.
Then I copy all my files and mails from the /home partition and move them all to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
However, still the question remains. If Linux ext3 doesn't need defragmenter, and able to defrag itself, what is the process name? And when does it run? Can I see it in action? Is there an utility to see on what percentage my current defragmentation? I tried fschk but no luck.
The opinion that EXT2 doesn't need defragmenting is based on only a filesystem-level view of the problem, and doesn't consider data read and write performance. EXT2 does make an effort to keep data only a short seek away ("clustered"). With this clustering, the filesystem operations of adding, removing, extending, and shortening files are not much affected by fragmentation.
With EXT3 (journalling), which always writes data to a new place, updates the filesystem info, and then frees the old data (roughly speaking), fragmentation is a way of life, and there isn't much to be done about it. Clustering helps by keeping the seeks relatively short, if there is space nearby.
I've heard this argument before, quite a few times in fact. It ignores one big fact of life with regards to discs. Almost all the data on my disc are *static*. They don't change. And so having files (like /bin/ls, for example) be contiguous saves enormously when they are read. Even if most of the data on one's disc is not static, still quite a bit of it is, or should be. (I'll omit to discuss the abominable prelink for the moment.)
When you have only 2% free, it's just about certain that the free space is a long way away from the rest of the data in a file. Just deleting to get 20% free would probably have fixed your problem.
Absolutely. Running with just that much free on a Linux system is insanity, anyway. I'd be afraid of a system lockup.
Mike
Tony Nelson:
The opinion that EXT2 doesn't need defragmenting is based on only a filesystem-level view of the problem, and doesn't consider data read and write performance. EXT2 does make an effort to keep data only a short seek away ("clustered"). With this clustering, the filesystem operations of adding, removing, extending, and shortening files are not much affected by fragmentation.
With EXT3 (journalling), which always writes data to a new place, updates the filesystem info, and then frees the old data (roughly speaking), fragmentation is a way of life, and there isn't much to be done about it. Clustering helps by keeping the seeks relatively short, if there is space nearby.
Mike McCarty:
I've heard this argument before, quite a few times in fact. It ignores one big fact of life with regards to discs. Almost all the data on my disc are *static*. They don't change. And so having files (like /bin/ls, for example) be contiguous saves enormously when they are read.
But such (static) data doesn't get fragmented, it stays as it was original written. It's changing files that become fragmented, and newly written ones that have to fit into the holes of a fragmented drive. And since the general advice is to keep /home on a separate partition, for many good reasons, user files should't affect system and application files.
Tim wrote:
Tony Nelson:
The opinion that EXT2 doesn't need defragmenting is based on only a filesystem-level view of the problem, and doesn't consider data read and write performance. EXT2 does make an effort to keep data only a short seek away ("clustered"). With this clustering, the filesystem operations of adding, removing, extending, and shortening files are not much affected by fragmentation.
With EXT3 (journalling), which always writes data to a new place, updates the filesystem info, and then frees the old data (roughly speaking), fragmentation is a way of life, and there isn't much to be done about it. Clustering helps by keeping the seeks relatively short, if there is space nearby.
Mike McCarty:
I've heard this argument before, quite a few times in fact. It ignores one big fact of life with regards to discs. Almost all the data on my disc are *static*. They don't change. And so having files (like /bin/ls, for example) be contiguous saves enormously when they are read.
But such (static) data doesn't get fragmented, it stays as it was original written. It's changing files that become fragmented, and newly
Er? Perhaps what Tony wrote was in error, but his understanding is the same as mine. The ext3 tends to fragment files as it writes them. And what you wrote doesn't address the directories, which get appended to, and presumably fragmented, at the time they are created.
written ones that have to fit into the holes of a fragmented drive. And since the general advice is to keep /home on a separate partition, for many good reasons, user files should't affect system and application files.
Mike
Tim:
But such (static) data doesn't get fragmented, it stays as it was original written. It's changing files that become fragmented, and newly created ones
Mike McCarty:
Er? Perhaps what Tony wrote was in error, but his understanding is the same as mine. The ext3 tends to fragment files as it writes them.
It would only be fragmenting the files that it writes to, not the ones already on the disk. Sure, a fragmented word processor document might take a bit longer to open (though it'd have to be a large file for you to notice), but the word processor is going to take just as long to start up as it ever did. Likewise with all the other unmodified files on the drive (most of the OS and applications). Writing a fragmented file doesn't shuffle everything else around.
Things like large mail spool files have been about the only thing that strike me as a fragmentation issue. Most other files are rather small.
And what you wrote doesn't address the directories, which get appended to, and presumably fragmented, at the time they are creat
I was under the impression that the directory structure was recorded in manner that's different from how the files are stored.
Tim wrote:
Tim:
But such (static) data doesn't get fragmented, it stays as it was original written. It's changing files that become fragmented, and newly created ones
Mike McCarty:
Er? Perhaps what Tony wrote was in error, but his understanding is the same as mine. The ext3 tends to fragment files as it writes them.
It would only be fragmenting the files that it writes to, not the ones already on the disk. Sure, a fragmented word processor document might take a bit longer to open (though it'd have to be a large file for you to notice), but the word processor is going to take just as long to start up as it ever did. Likewise with all the other unmodified files on the drive (most of the OS and applications). Writing a fragmented file doesn't shuffle everything else around.
I wasn't saying that. But writing to the files *during installation* might result in fragmentation.
Things like large mail spool files have been about the only thing that strike me as a fragmentation issue. Most other files are rather small.
True.
And what you wrote doesn't address the directories, which get appended to, and presumably fragmented, at the time they are creat
I was under the impression that the directory structure was recorded in manner that's different from how the files are stored.
You may know something that I do not. I thought everything was inodes.
Mike
Tim wrote:
Tim:
<snip>
Mike McCarty:
<snip>
I was under the impression that the directory structure was recorded in manner that's different from how the files are stored.
You may know something that I do not. I thought everything was inodes.
If you look in the Linux header files you can see the structure of a directory entry in a directory. A directory is a structured file but one treated differently by the OS because the directory bit is set in its inode. But like other files, in some sense, it is just bits on the platter, albeit with some structure, as such a directory IS subject to possible fragmentation.
dlg
<snip>
Tim:
I was under the impression that the directory structure was recorded in manner that's different from how the files are stored.
Mike McCarty:
You may know something that I do not. I thought everything was inodes.
David L. Gehrt:
If you look in the Linux header files you can see the structure of a directory entry in a directory. A directory is a structured file but one treated differently by the OS because the directory bit is set in its inode. But like other files, in some sense, it is just bits on the platter, albeit with some structure, as such a directory IS subject to possible fragmentation.
I was thinking of the following, though I may be mixing up this file system with another type: The directory structure being stored some distance away from the files, so that one doesn't tend to fragment the other.
It's been a long time since I read about the nitty-gritty of filing systems, it's not the sort of thing I do for fun. ;-\ It tends to be the sort of thing you read as you're diagnosing some condition (somewhere along the line, someone explains how it works behind the scenes).
On Fri, 2005-12-23 at 14:18 -0600, Mike McCarty wrote:
But writing to the files *during installation* might result in fragmentation.
Initial installation, no. Installations of applications after a system has been used for some time, perhaps.
This is another area where the Unix idea of keeping applications and data (user data, variable data, temporary files, etc.) partitioned away from brings you big benefits.
Temporary files, in their many guises, are a pain for fragmenting drives on the Windows world, because they're generally all on the same partition. Put them on their own, and they only affect each other. Which doesn't really matter, as they can be periodically removed, easily and without harm (none of them need keeping once they've been used).
User data doesn't usually suffer too much from fragmentation unless the user handles very large data files (mail spools, they do audio or video editing, etc.). Who cares if a 20kB file takes two or four head seeks to read, just the once while being opened up? It's not noticeable.
Variable data, such as log files, are another cause for it. Again, partitioning is a good solution, for a variety of reasons--not just fragmentation, but from drive filling up by run-away or hijacked processes.
At the very least, I like separate partitions for all of them (/home, /tmp, /var) from the main partition/s.
On Sat, 2005-12-24 at 03:41, Tim wrote:
On Fri, 2005-12-23 at 14:18 -0600, Mike McCarty wrote:
But writing to the files *during installation* might result in fragmentation.
Initial installation, no. Installations of applications after a system has been used for some time, perhaps.
This is fedora we are talking about. If you stay up to date you've replaced almost everything several times and the files are likely scattered all over the disk.
This is another area where the Unix idea of keeping applications and data (user data, variable data, temporary files, etc.) partitioned away from brings you big benefits.
It is the disk head movement that that takes thousands of times longer than any other computer operation. Using separate partitions only helps in this respect to the extent that they are on different drives so the heads are independent. The thing that really saves you is the RAM cache so frequently used items don't need to repeat the seek.
Les Mikesell wrote:
On Sat, 2005-12-24 at 03:41, Tim wrote:
On Fri, 2005-12-23 at 14:18 -0600, Mike McCarty wrote:
But writing to the files *during installation* might result in fragmentation.
Initial installation, no. Installations of applications after a system has been used for some time, perhaps.
This is fedora we are talking about. If you stay up to date you've replaced almost everything several times and the files are likely scattered all over the disk.
This assertion really needs to be tested with a typical setup of two partitions: / /boot
Just apply all of the updates in the order they were released, then measure the actual fragmentation.
You could also test disk performance at various points, maybe using Bonnie, to see what the changes are.
Les Mikesell wrote:
On Sat, 2005-12-24 at 03:41, Tim wrote:
On Fri, 2005-12-23 at 14:18 -0600, Mike McCarty wrote:
But writing to the files *during installation* might result in fragmentation.
Initial installation, no. Installations of applications after a system has been used for some time, perhaps.
This is fedora we are talking about. If you stay up to date you've replaced almost everything several times and the files are likely scattered all over the disk.
[snip]
Thank you, that was my point (perhaps I didn't make myself clear).
Mike
<<I wasn't saying that. But writing to the files *during installation* might result in fragmentation.>>
Just my two cents, if you install to a freshly formatted partition and the install has to write 20,000 files during package installation I cannot see the fs code scattering the files all of creation.
For example, when the mozilla package is installed I cannot see the mozilla binary being written to random sectors of the disc, I see the binary being written sequentially from start to finish.
Would this be a correct assumption?
John
-- Registered Linux User 263680, get counted at http://counter.li.org
On Wed, 2005-12-28 at 12:34 -0600, John Pierce wrote:
<<I wasn't saying that. But writing to the files *during installation* might result in fragmentation.>>
Just my two cents, if you install to a freshly formatted partition and the install has to write 20,000 files during package installation I cannot see the fs code scattering the files all of creation.
For example, when the mozilla package is installed I cannot see the mozilla binary being written to random sectors of the disc, I see the binary being written sequentially from start to finish.
Would this be a correct assumption?
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
In general, the only fragmentation that occurs is when files are dynamically growing with use, such as log files and the like. I (as others also have noted) have seen systems that have been in use for years and only had a minimal fragmentation. Generally that remains in the low single digit range.
John
-- Registered Linux User 263680, get counted at http://counter.li.org
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45. /usr/bin/postgres: 45 extents found, perfection would be 1 extent I outputted the findings to a file which ended up being around 75 kb The log files didn't seems to be as fragmented as I expected. Another pecularity was that some of the files did not seem to be things that I use much. Another excerpt from the query. /usr/bin/php: 52 extents found, perfection would be 1 extent /usr/bin/php-cgi: 62 extents found, perfection would be 1 extent
My system is running development. This of course means frequently updated programs. Checking usr/lib showed fragmentation also. /usr/lib/libXm.so.4: 48 extents found, perfection would be 1 extent /usr/lib/libXm.so.4.0.0: 48 extents found, perfection would be 1 extent .. /usr/lib/libxvidcore.so.4: 49 extents found, perfection would be 1 extent
I guess fragmentation is present on the system and probably needs addressed or a less fragmented filesystem type needs implemented.
Jim
Jim Cornette wrote:
John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45.
On my system I did this...
# filefrag /usr/bin/* | sort -k2 -nr | grep 'would be'
Here're the first few entries...
/usr/bin/emacs: 248 extents found, perfection would be 1 extent /usr/bin/emacs-21.3: 248 extents found, perfection would be 1 extent /usr/bin/kermit: 80 extents found, perfection would be 1 extent /usr/bin/kbabel: 45 extents found, perfection would be 1 extent /usr/bin/ddd: 45 extents found, perfection would be 1 extent /usr/bin/gthumb: 41 extents found, perfection would be 1 extent /usr/bin/gdbtui: 36 extents found, perfection would be 1 extent /usr/bin/elinks: 30 extents found, perfection would be 1 extent /usr/bin/iniomega: 22 extents found, perfection would be 1 extent /usr/bin/kpersonalizer: 21 extents found, perfection would be 1 extent /usr/bin/artsd: 21 extents found, perfection would be 1 extent /usr/bin/artscat: 20 extents found, perfection would be 1 extent /usr/bin/kiconedit: 19 extents found, perfection would be 1 extent /usr/bin/glade-2: 19 extents found, perfection would be 1 extent /usr/bin/karm: 18 extents found, perfection would be 1 extent /usr/bin/dia: 18 extents found, perfection would be 1 extent /usr/bin/designer3: 18 extents found, perfection would be 1 extent /usr/bin/designer: 18 extents found, perfection would be 1 extent /usr/bin/kppplogview: 16 extents found, perfection would be 1 extent /usr/bin/kfontinst: 16 extents found, perfection would be 1 extent /usr/bin/civclient-xaw: 15 extents found, perfection would be 1 extent /usr/bin/cdrecord: 15 extents found, perfection would be 1 extent /usr/bin/knewstickerstub: 14 extents found, perfection would be 1 ext
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
I assure you that I have done nothing on my system to try to fragment emacs.
Mike
Mike McCarty wrote:
Jim Cornette wrote:
John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45.
On my system I did this...
# filefrag /usr/bin/* | sort -k2 -nr | grep 'would be'
Here're the first few entries...
/usr/bin/emacs: 248 extents found, perfection would be 1 extent /usr/bin/emacs-21.3: 248 extents found, perfection would be 1 extent /usr/bin/kermit: 80 extents found, perfection would be 1 extent /usr/bin/kbabel: 45 extents found, perfection would be 1 extent /usr/bin/ddd: 45 extents found, perfection would be 1 extent /usr/bin/gthumb: 41 extents found, perfection would be 1 extent /usr/bin/gdbtui: 36 extents found, perfection would be 1 extent /usr/bin/elinks: 30 extents found, perfection would be 1 extent /usr/bin/iniomega: 22 extents found, perfection would be 1 extent /usr/bin/kpersonalizer: 21 extents found, perfection would be 1 extent /usr/bin/artsd: 21 extents found, perfection would be 1 extent /usr/bin/artscat: 20 extents found, perfection would be 1 extent /usr/bin/kiconedit: 19 extents found, perfection would be 1 extent /usr/bin/glade-2: 19 extents found, perfection would be 1 extent /usr/bin/karm: 18 extents found, perfection would be 1 extent /usr/bin/dia: 18 extents found, perfection would be 1 extent /usr/bin/designer3: 18 extents found, perfection would be 1 extent /usr/bin/designer: 18 extents found, perfection would be 1 extent /usr/bin/kppplogview: 16 extents found, perfection would be 1 extent /usr/bin/kfontinst: 16 extents found, perfection would be 1 extent /usr/bin/civclient-xaw: 15 extents found, perfection would be 1 extent /usr/bin/cdrecord: 15 extents found, perfection would be 1 extent /usr/bin/knewstickerstub: 14 extents found, perfection would be 1 ext
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
I assure you that I have done nothing on my system to try to fragment emacs.
Mike
The fragmentation for your emacs is unbelievably high. I did not find anything yet fragmented in the hundreds, let alone several hundred extents. Are you using LVM? My system is setup in traditional partitions. LVM usage "seemed" slower in responsiveness, so I assumed it was more in fragments
Jim
Jim Cornette wrote:
Mike McCarty wrote:
Jim Cornette wrote:
John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45.
On my system I did this...
# filefrag /usr/bin/* | sort -k2 -nr | grep 'would be'
Here're the first few entries...
/usr/bin/emacs: 248 extents found, perfection would be 1 extent /usr/bin/emacs-21.3: 248 extents found, perfection would be 1 extent /usr/bin/kermit: 80 extents found, perfection would be 1 extent /usr/bin/kbabel: 45 extents found, perfection would be 1 extent /usr/bin/ddd: 45 extents found, perfection would be 1 extent /usr/bin/gthumb: 41 extents found, perfection would be 1 extent /usr/bin/gdbtui: 36 extents found, perfection would be 1 extent /usr/bin/elinks: 30 extents found, perfection would be 1 extent /usr/bin/iniomega: 22 extents found, perfection would be 1 extent /usr/bin/kpersonalizer: 21 extents found, perfection would be 1 extent /usr/bin/artsd: 21 extents found, perfection would be 1 extent /usr/bin/artscat: 20 extents found, perfection would be 1 extent /usr/bin/kiconedit: 19 extents found, perfection would be 1 extent /usr/bin/glade-2: 19 extents found, perfection would be 1 extent /usr/bin/karm: 18 extents found, perfection would be 1 extent /usr/bin/dia: 18 extents found, perfection would be 1 extent /usr/bin/designer3: 18 extents found, perfection would be 1 extent /usr/bin/designer: 18 extents found, perfection would be 1 extent /usr/bin/kppplogview: 16 extents found, perfection would be 1 extent /usr/bin/kfontinst: 16 extents found, perfection would be 1 extent /usr/bin/civclient-xaw: 15 extents found, perfection would be 1 extent /usr/bin/cdrecord: 15 extents found, perfection would be 1 extent /usr/bin/knewstickerstub: 14 extents found, perfection would be 1 ext
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
I assure you that I have done nothing on my system to try to fragment emacs.
Mike
The fragmentation for your emacs is unbelievably high. I did not find anything yet fragmented in the hundreds, let alone several hundred extents. Are you using LVM? My system is setup in traditional partitions. LVM usage "seemed" slower in responsiveness, so I assumed it was more in fragments
Why "unvelievably"? Do you mean that you do not believe what my system says? Or that you do not believe my e-mail? Or that you find that it stretches your imagination? Or what?
To answer your question, I use FC2.
Mike
From: "Mike McCarty" mike.mccarty@sbcglobal.net
The fragmentation for your emacs is unbelievably high. I did not find anything yet fragmented in the hundreds, let alone several hundred extents. Are you using LVM? My system is setup in traditional partitions. LVM usage "seemed" slower in responsiveness, so I assumed it was more in fragments
Why "unvelievably"? Do you mean that you do not believe what my system says? Or that you do not believe my e-mail? Or that you find that it stretches your imagination? Or what?
Perhaps this is an Americanism. It is higher than one would believe in a "fantasy story" but perforce must believe when seen in real life. It does seem rather ridiculously or unbelievably or unrealistically or fish story high. But you have the numbers right in front of you. So unbelievable or not it can happen. It certainly is fascinating that it has.
How full is that file system? {^_^}
jdow wrote:
From: "Mike McCarty" mike.mccarty@sbcglobal.net
The fragmentation for your emacs is unbelievably high. I did not find anything yet fragmented in the hundreds, let alone several hundred extents. Are you using LVM? My system is setup in traditional partitions. LVM usage "seemed" slower in responsiveness, so I assumed it was more in fragments
Why "unvelievably"? Do you mean that you do not believe what my system says? Or that you do not believe my e-mail? Or that you find that it stretches your imagination? Or what?
Perhaps this is an Americanism. It is higher than one would believe
Well, though my first language is not English, it is my preferred language, which I've been speaking since I was about 4 or so. And I was born in and grew up in the USA. Native Texican.
in a "fantasy story" but perforce must believe when seen in real life.
This is what I meant by "stretches your imagination".
It does seem rather ridiculously or unbelievably or unrealistically or fish story high. But you have the numbers right in front of you. So unbelievable or not it can happen. It certainly is fascinating that it has.
I was, well, shocked. OTOH, someone stated that extents and fragmentation are not the same, as different extents may well be contiguous on disc.
How full is that file system? {^_^}
$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda5 7633264 4681896 2563620 65% / /dev/hda3 99075 24602 69358 27% /boot none 124044 0 124044 0% /dev/shm
Hmm, anyone can explain what /dev/shm is? And why it's 62MB? This is a dual-boot WinXP/Linux.
# fdisk -l
Disk /dev/hda: 40.0 GB, 40020664320 bytes 16 heads, 63 sectors/track, 77545 cylinders Units = cylinders of 1008 * 512 = 516096 bytes
Device Boot Start End Blocks Id System /dev/hda1 1 8625 4346968+ b W95 FAT32 Partition 1 does not end on cylinder boundary. /dev/hda2 * 8626 60915 26354160 7 HPFS/NTFS Partition 2 does not end on cylinder boundary. /dev/hda3 60916 61118 102312 83 Linux Partition 3 does not end on cylinder boundary. /dev/hda4 61119 77545 8279208 f W95 Ext'd (LBA) Partition 4 does not end on cylinder boundary. /dev/hda5 61119 76505 7755016+ 83 Linux /dev/hda6 76506 77545 524128+ 82 Linux swap
Mike
On Fri, 2005-12-30 at 10:01 -0600, Mike McCarty wrote:
Hmm, anyone can explain what /dev/shm is? And why it's 62MB? This is a dual-boot WinXP/Linux.
It's used for POSIX shared memory support in applications. Essentially it's just a ramdisk that takes up almost no space if there's nothing using it. You may want to check the following mailing list post[1] as well as a quick Google search for more information.
[1] http://www.luci.org/luci-discuss/200204/msg00015.html
Hope that helps!
Peter Gordon wrote:
On Fri, 2005-12-30 at 10:01 -0600, Mike McCarty wrote:
Hmm, anyone can explain what /dev/shm is? And why it's 62MB? This is a dual-boot WinXP/Linux.
It's used for POSIX shared memory support in applications. Essentially it's just a ramdisk that takes up almost no space if there's nothing using it. You may want to check the following mailing list post[1] as well as a quick Google search for more information.
[1] http://www.luci.org/luci-discuss/200204/msg00015.html
Hope that helps!
Thank you very much. I had thought perhaps SHM was shared memory, but then it shows up as a file system...
Anyway, thanks!
Mike
Mike McCarty wrote:
Jim Cornette wrote:
Mike McCarty wrote:
Jim Cornette wrote:
John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45.
On my system I did this...
# filefrag /usr/bin/* | sort -k2 -nr | grep 'would be'
Here're the first few entries...
/usr/bin/emacs: 248 extents found, perfection would be 1 extent /usr/bin/emacs-21.3: 248 extents found, perfection would be 1 extent /usr/bin/kermit: 80 extents found, perfection would be 1 extent /usr/bin/kbabel: 45 extents found, perfection would be 1 extent /usr/bin/ddd: 45 extents found, perfection would be 1 extent /usr/bin/gthumb: 41 extents found, perfection would be 1 extent /usr/bin/gdbtui: 36 extents found, perfection would be 1 extent /usr/bin/elinks: 30 extents found, perfection would be 1 extent /usr/bin/iniomega: 22 extents found, perfection would be 1 extent /usr/bin/kpersonalizer: 21 extents found, perfection would be 1 extent /usr/bin/artsd: 21 extents found, perfection would be 1 extent /usr/bin/artscat: 20 extents found, perfection would be 1 extent /usr/bin/kiconedit: 19 extents found, perfection would be 1 extent /usr/bin/glade-2: 19 extents found, perfection would be 1 extent /usr/bin/karm: 18 extents found, perfection would be 1 extent /usr/bin/dia: 18 extents found, perfection would be 1 extent /usr/bin/designer3: 18 extents found, perfection would be 1 extent /usr/bin/designer: 18 extents found, perfection would be 1 extent /usr/bin/kppplogview: 16 extents found, perfection would be 1 extent /usr/bin/kfontinst: 16 extents found, perfection would be 1 extent /usr/bin/civclient-xaw: 15 extents found, perfection would be 1 extent /usr/bin/cdrecord: 15 extents found, perfection would be 1 extent /usr/bin/knewstickerstub: 14 extents found, perfection would be 1 ext
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
I assure you that I have done nothing on my system to try to fragment emacs.
Mike
The fragmentation for your emacs is unbelievably high. I did not find anything yet fragmented in the hundreds, let alone several hundred extents. Are you using LVM? My system is setup in traditional partitions. LVM usage "seemed" slower in responsiveness, so I assumed it was more in fragments
Why "unvelievably"? Do you mean that you do not believe what my system says? Or that you do not believe my e-mail? Or that you find that it stretches your imagination? Or what?
To answer your question, I use FC2.
Mike
Unbelievable simply refers to this fragmentation number sounds like it should not happen. I have no doubt that you are seeing this on your system. Since you are running FC2, I assume the system is using regular partitions and that the system has been in operation for a long time. Sorry if the response sounded otherwise.
Jim
Jim Cornette wrote:
Mike McCarty wrote:
[snip]
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
I assure you that I have done nothing on my system to try to fragment emacs.
Mike
The fragmentation for your emacs is unbelievably high. I did not find anything yet fragmented in the hundreds, let alone several hundred extents. Are you using LVM? My system is setup in traditional partitions. LVM usage "seemed" slower in responsiveness, so I assumed it was more in fragments
Why "unvelievably"? Do you mean that you do not believe what my system says? Or that you do not believe my e-mail? Or that you find that it stretches your imagination? Or what?
To answer your question, I use FC2.
Mike
Unbelievable simply refers to this fragmentation number sounds like it
Well, then I agree with you. This seems extreme.
should not happen. I have no doubt that you are seeing this on your system. Since you are running FC2, I assume the system is using regular partitions and that the system has been in operation for a long time. Sorry if the response sounded otherwise.
It was installed in late October 2004. Although, as one pointed out, multiple extents does not *necessarily* imply fragmentation, it is extremely suggestive, and indicates that fragmentation is *likely*.
Mike
Mike McCarty wrote:
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
While that almost certainly constitutes "significant fragmentation" it probably has very little effect on performance. It _would_ affect performance if that file were being read sequentially, but ELF binaries don't get loaded that way. Individual pages are read in via demand paging, and the next page needed could be from anywhere in the file.
Robert Nichols wrote:
Mike McCarty wrote:
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
While that almost certainly constitutes "significant fragmentation" it probably has very little effect on performance. It _would_ affect performance if that file were being read sequentially, but ELF binaries don't get loaded that way. Individual pages are read in via demand paging, and the next page needed could be from anywhere in the file.
I thought we were discussing ext3 and what could happen with it. Someone was claiming that ext3 was inherently immune or nearly so to fragmentation.
So, back to the OP's question: Where is there/why isn't there a defragmenter for ext3?
Mike
Mike McCarty wrote:
Robert Nichols wrote:
Mike McCarty wrote:
Surely those who argue that ext3 does not get fragmented during install don't think that 248 extents is "not significant fragmentation".
While that almost certainly constitutes "significant fragmentation" it probably has very little effect on performance. It _would_ affect performance if that file were being read sequentially, but ELF binaries don't get loaded that way. Individual pages are read in via demand paging, and the next page needed could be from anywhere in the file.
I thought we were discussing ext3 and what could happen with it. Someone was claiming that ext3 was inherently immune or nearly so to fragmentation.
So, back to the OP's question: Where is there/why isn't there a defragmenter for ext3?
With a file system that typically minimizes fragmentation and an OS that isn't very sensitive to such fragmentation as is likely to occur, there is just insufficient perceived need to justify the effort for writing one.
Jim Cornette wrote:
John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45. /usr/bin/postgres: 45 extents found, perfection would be 1 extent I outputted the findings to a file which ended up being around 75 kb The log files didn't seems to be as fragmented as I expected. Another pecularity was that some of the files did not seem to be things that I use much. Another excerpt from the query. /usr/bin/php: 52 extents found, perfection would be 1 extent /usr/bin/php-cgi: 62 extents found, perfection would be 1 extent
My system is running development. This of course means frequently updated programs. Checking usr/lib showed fragmentation also. /usr/lib/libXm.so.4: 48 extents found, perfection would be 1 extent /usr/lib/libXm.so.4.0.0: 48 extents found, perfection would be 1 extent .. /usr/lib/libxvidcore.so.4: 49 extents found, perfection would be 1 extent
I guess fragmentation is present on the system and probably needs addressed or a less fragmented filesystem type needs implemented.
extent ^= fragmentation. On filesystems I know, extents can be on consecutive areas (sectors, blocks, tracks) of disk. You need better info than that to see whether you have a fragmentation problem. filefrag will give it to you, but it needs a wrapper to make sense to Mere Mortals.
fwiw summer@bilby ~]$ sudo filefrag /usr/sbin/postfix /usr/X11R6/lib/libXm* /usr/sbin/postfix: 1 extent found /usr/X11R6/lib/libXm.a: 1 extent found /usr/X11R6/lib/libXm.so: 1 extent found /usr/X11R6/lib/libXm.so.2: 1 extent found /usr/X11R6/lib/libXm.so.2.1: 1 extent found /usr/X11R6/lib/libXm.so.3: 1 extent found /usr/X11R6/lib/libXm.so.3.0.2: 1 extent found /usr/X11R6/lib/libXmu.a: 1 extent found /usr/X11R6/lib/libXmu.so: 2 extents found, perfection would be 1 extent /usr/X11R6/lib/libXmu.so.6: 2 extents found, perfection would be 1 extent /usr/X11R6/lib/libXmu.so.6.2: 2 extents found, perfection would be 1 extent /usr/X11R6/lib/libXmuu.a: 1 extent found /usr/X11R6/lib/libXmuu.so: 1 extent found /usr/X11R6/lib/libXmuu.so.1: 1 extent found /usr/X11R6/lib/libXmuu.so.1.0: 1 extent found
This system was installed as Fedora Core 3, and all these package subsequently replaced (with force where necessary) with packages I built myself from RHEL 4 source. That is, they have all been updated after installation.
Checking lib and bin directories under /usr, I see no more than three extents.
I note my file placement's different from yours.
John Summerfied wrote:
Since the discussions regarding fragmentation on ext3 filesystems was pretty long running. I decided to try filefrag /usr/bin/* |sort |grep 'would be' and the output showed a lot of fragmentation. One of the files was up to 45. /usr/bin/postgres: 45 extents found, perfection would be 1 extent I outputted the findings to a file which ended up being around 75 kb The log files didn't seems to be as fragmented as I expected. Another pecularity was that some of the files did not seem to be things that I use much. Another excerpt from the query. /usr/bin/php: 52 extents found, perfection would be 1 extent /usr/bin/php-cgi: 62 extents found, perfection would be 1 extent
My system is running development. This of course means frequently updated programs. Checking usr/lib showed fragmentation also. /usr/lib/libXm.so.4: 48 extents found, perfection would be 1 extent /usr/lib/libXm.so.4.0.0: 48 extents found, perfection would be 1 extent .. /usr/lib/libxvidcore.so.4: 49 extents found, perfection would be 1 extent
I guess fragmentation is present on the system and probably needs addressed or a less fragmented filesystem type needs implemented.
extent ^= fragmentation. On filesystems I know, extents can be on consecutive areas (sectors, blocks, tracks) of disk. You need better info than that to see whether you have a fragmentation problem. filefrag will give it to you, but it needs a wrapper to make sense to Mere Mortals.
Thanks, I did not take into account the possibility the file would be in series and read in order. Fragmentation is not of major importance to me at present. Thanks for pointing out the additional factors that need consideration.
fwiw summer@bilby ~]$ sudo filefrag /usr/sbin/postfix /usr/X11R6/lib/libXm* /usr/sbin/postfix: 1 extent found /usr/X11R6/lib/libXm.a: 1 extent found /usr/X11R6/lib/libXm.so: 1 extent found /usr/X11R6/lib/libXm.so.2: 1 extent found /usr/X11R6/lib/libXm.so.2.1: 1 extent found /usr/X11R6/lib/libXm.so.3: 1 extent found /usr/X11R6/lib/libXm.so.3.0.2: 1 extent found /usr/X11R6/lib/libXmu.a: 1 extent found /usr/X11R6/lib/libXmu.so: 2 extents found, perfection would be 1 extent /usr/X11R6/lib/libXmu.so.6: 2 extents found, perfection would be 1 extent /usr/X11R6/lib/libXmu.so.6.2: 2 extents found, perfection would be 1 extent /usr/X11R6/lib/libXmuu.a: 1 extent found /usr/X11R6/lib/libXmuu.so: 1 extent found /usr/X11R6/lib/libXmuu.so.1: 1 extent found /usr/X11R6/lib/libXmuu.so.1.0: 1 extent found
This system was installed as Fedora Core 3, and all these package subsequently replaced (with force where necessary) with packages I built myself from RHEL 4 source. That is, they have all been updated after installation.
Thanks for the comparison. The fact that you brought your system up from FC3 to RHEL home brewed edition is interesting also.
Checking lib and bin directories under /usr, I see no more than three extents.
I note my file placement's different from yours.
Running rawhide vs. RHEL 4 might be the difference in comparison. Jim
On Fri, 2005-12-30 at 11:07 +0800, John Summerfied wrote:
Jeff Vian wrote:
Exactly, and IIRC the filesystem knows that if it needs X amount of space for a file, then Y number of inodes are marked for use for that file at the beginning. Thus space allocated is as contiguous as is efficient for read/write on the disk.
If "the filesystem knows that if it needs X amount of space for a file," that implies there's a way of telling it that.
How's that done? I don't recall any system call for *x (there is one for OS/2), and one could do it in JCL in IBM's OS in the 60s), but in the *x world I've never seen a way to do it.
I believe many apps know the size of the file being transferred/copied/extracted and reserve space for it.
I know that several ftp clients know as soon as they start the transfer exactly how many bytes they should receive. Several other things similarly know the expected file size being handled.
It would surprise me if rpm did not know that and be able to reserve space for the file to be placed as soon as each file begins to be extracted. Especially since rpm (some versions at least) does a check of the file it extracts to verify it is intact.
--
Cheers John
-- spambait 1aaaaaaa@computerdatasafe.com.au Z1aaaaaaa@computerdatasafe.com.au Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/
do not reply off-list
Jeff Vian wrote:
It would surprise me if rpm did not know that and be able to reserve space for the file to be placed as soon as each file begins to be extracted. Especially since rpm (some versions at least) does a check of the file it extracts to verify it is intact.
There is no mechanism for requesting that space be reserved for a file. The ext2/ext3 file system code does pre-allocate some number of blocks when new space is needed to accommodate a write(), and that reduces the liklihood of tiny fragments. Beyond that, there is simply no way to do what you suggest.
Tim wrote:
Tim:
But such (static) data doesn't get fragmented, it stays as it was original written. It's changing files that become fragmented, and newly created ones
Mike McCarty:
Er? Perhaps what Tony wrote was in error, but his understanding is the same as mine. The ext3 tends to fragment files as it writes them.
It would only be fragmenting the files that it writes to, not the ones already on the disk. Sure, a fragmented word processor document might take a bit longer to open (though it'd have to be a large file for you to notice), but the word processor is going to take just as long to start up as it ever did. Likewise with all the other unmodified files on the drive (most of the OS and applications). Writing a fragmented file doesn't shuffle everything else around.
Things like large mail spool files have been about the only thing that strike me as a fragmentation issue. Most other files are rather small.
And what you wrote doesn't address the directories, which get appended to, and presumably fragmented, at the time they are creat
I was under the impression that the directory structure was recorded in manner that's different from how the files are stored.
Might I observe that the many-partitions layout so often recommended gives you all the disadvantages of a fragmented drive from day one?
Two busy partitions is one too many. In these times of cheap disks and USB2 enclosures, I'd rather have one partition for everything (except maybe /boot and maybe other low-use stuff), and if an upgrade is contemplated, back up /home to a USB drive. At worst, almost anything can be backed up overnight. According to dd, I can backup /dev/hda (80 Gb) to a USB2 disk at 14 Mbytes/sec on my laptop.
Arguably, I should be doing something of the sort regardless. As should anyone with many computers in their care.
fwiw I used to use OS/2 and it was IBM's recommendation that one should not defrag HPFS (which, btw, predates NTFS) partitions because HPFS allocates space in bands (take your pad, divide it into eight columns and you have eight bands) (and so takes an initial performance hit). File expansions are done within the same band where possible, so reducing the impact of further fragmentation. Performance was pretty uniform up to, I think, about 95% full.
Defragging an HPFS drive would involve putting all the files together into a single block, and the chances were good that you'd soon find files occupying extents both inside and outside that block and consequent bad performance.
I've always assumed that, since the algorithms have existed for a long time, that Linux filesystems are also good in that respect. The fact that no defragger is included in populare distros supports my (underinformed) view.
Journalling introduces a complication, but its effect depends on where the journal is. Also, journalling only has an effect when files are written.
Finally, the ultimate defag tool is backup and restore. It might not be necessary, but it won't do any harm either.
On Wed, 2005-28-12 at 07:57 +0800, John Summerfied wrote: ...snip...
Might I observe that the many-partitions layout so often recommended gives you all the disadvantages of a fragmented drive from day one?
Just plain wrong. Keeping relatively static files separated from relatively dynamic files can keep the "static" files less fragmented. And since spool files are very dynamic and command files are usually very static, it makes a great deal of sense to keep /usr separate from /var. There are many other security and performance reasons to keep other directories in separate partitions, not necessarily related to fragmentation. There is also a good reason for leaving adequate "head room" on your partitions to alleviate fragmentation.
As mentioned mail spools are notorious for fragmenting. There are lots of files many small but some are often very large. Using hashed mail directories can help, by keeping the size of the directories from adding to the problem. Discouraging large mailboxes is also effective in minimizing fragmentation.
Two busy partitions is one too many. In these times of cheap disks and USB2 enclosures, I'd rather have one partition for everything (except maybe /boot and maybe other low-use stuff), and if an upgrade is contemplated, back up /home to a USB drive. At worst, almost anything can be backed up overnight. According to dd, I can backup /dev/hda (80 Gb) to a USB2 disk at 14 Mbytes/sec on my laptop.
Arguably, I should be doing something of the sort regardless. As should anyone with many computers in their care.
fwiw I used to use OS/2 and it was IBM's recommendation that one should not defrag HPFS (which, btw, predates NTFS) partitions because HPFS allocates space in bands (take your pad, divide it into eight columns and you have eight bands) (and so takes an initial performance hit). File expansions are done within the same band where possible, so reducing the impact of further fragmentation. Performance was pretty uniform up to, I think, about 95% full.
Defragging an HPFS drive would involve putting all the files together into a single block, and the chances were good that you'd soon find files occupying extents both inside and outside that block and consequent bad performance.
I've always assumed that, since the algorithms have existed for a long time, that Linux filesystems are also good in that respect. The fact that no defragger is included in populare distros supports my (underinformed) view.
Not sure what point your making.
Journalling introduces a complication, but its effect depends on where the journal is. Also, journalling only has an effect when files are written.
Finally, the ultimate defag tool is backup and restore. It might not be necessary, but it won't do any harm either.
That is likely true.
Stuffing a lot of files into a directory is a bad practice, and Red Hat is well known for it. Check /usr/bin, /etc and a few others. Many of the files would normaly be located under /usr/local . If you are short of space on a partition or are writing a huge file after creating and deleting lots of small files, then you can get considerable fragmentation. Copying or archiving the files to a different partition, then deleting them and copying or restoring them back will "defrag" a partitiion.
On Tue, 2005-12-27 at 17:27 -0700, Guy Fraser wrote:
Stuffing a lot of files into a directory is a bad practice, and Red Hat is well known for it. Check /usr/bin, /etc and a few others. Many of the files would [normally] be located under /usr/local.
Pardon my probable misunderstanding, but isn't this the whole idea of the standard filesystem layout for Unix/Unix-like operating systems?
I.e., if it's not installed through the system package manager, it has its tree of /lib, /sbin, /bin, /man, /share, etc under /usr/local; whereas if it *is* installed through the system packaging, it has its own /lib, /sbin, /bin, /share, /man, etc layout tree under / or /usr, depending on various factors like partitioning or network-mounting /usr, etc.
On Tue, 2005-27-12 at 19:46 -0800, Peter Gordon wrote:
On Tue, 2005-12-27 at 17:27 -0700, Guy Fraser wrote:
Stuffing a lot of files into a directory is a bad practice, and Red Hat is well known for it. Check /usr/bin, /etc and a few others. Many of the files would [normally] be located under /usr/local.
Pardon my probable misunderstanding, but isn't this the whole idea of the standard filesystem layout for Unix/Unix-like operating systems?
I.e., if it's not installed through the system package manager, it has its tree of /lib, /sbin, /bin, /man, /share, etc under /usr/local; whereas if it *is* installed through the system packaging, it has its own /lib, /sbin, /bin, /share, /man, etc layout tree under / or /usr, depending on various factors like partitioning or network-mounting /usr, etc.
That is how Red Hat and it's offshoots work.
It can be a contentious issue, exactly where to put non base system files. Sun Microsystems used to prefer /opt but almost everyone else uses /usr/local for most add on software. But it is well understood that :
/bin, /sbin and /lib are for single user base system commands and their required libraries.
/usr/bin, /usr/sbin and /usr/lib are for multiuser base system commands and their required libraries.
/etc is for base system configuration.
The real theological battle is what is considered part of the base system. Most Unix variants and Linux distributions consider only essential software to be part of the base. In order to be as Newbie friendly as possible, Red Hat decided to include all the software they package as part of the base system, that way new comers could find stuff more easily. From the start Red Hat was the leader in making Linux user friendly, and has had many other distributions use their system as a base for their distribution. In the Linux world, since many distributions are based on early versions of Red Hat it has become common practise to put all "packaged" software in system areas, but common practise is not a standard, and really there is no standard, only suggested good practises. Coming from a Unix Administration background I prefer to use /usr/local but am not about to repackage every RPM just to suite my preferences. I however do design all my software to be relocatable with the preferred location of /usr/local, as do most others.
Guy Fraser wrote:
On Tue, 2005-27-12 at 19:46 -0800, Peter Gordon wrote:
On Tue, 2005-12-27 at 17:27 -0700, Guy Fraser wrote:
Stuffing a lot of files into a directory is a bad practice, and Red Hat is well known for it. Check /usr/bin, /etc and a few others. Many of the files would [normally] be located under /usr/local.
Pardon my probable misunderstanding, but isn't this the whole idea of the standard filesystem layout for Unix/Unix-like operating systems?
I.e., if it's not installed through the system package manager, it has its tree of /lib, /sbin, /bin, /man, /share, etc under /usr/local; whereas if it *is* installed through the system packaging, it has its own /lib, /sbin, /bin, /share, /man, etc layout tree under / or /usr, depending on various factors like partitioning or network-mounting /usr, etc.
That is how Red Hat and it's offshoots work.
It can be a contentious issue, exactly where to put non base system files. Sun Microsystems used to prefer /opt but almost everyone else uses /usr/local for most add on software. But it is well understood that :
/bin, /sbin and /lib are for single user base system commands and their required libraries.
That's not quite the case. Check the facts at http://www.pathname.com/ (and fall in love with Enya while you're there).
/usr/bin, /usr/sbin and /usr/lib are for multiuser base system commands and their required libraries.
/etc is for base system configuration.
On Thu, 2005-29-12 at 01:21 +0800, John Summerfied wrote:
Guy Fraser wrote:
On Tue, 2005-27-12 at 19:46 -0800, Peter Gordon wrote:
On Tue, 2005-12-27 at 17:27 -0700, Guy Fraser wrote:
Stuffing a lot of files into a directory is a bad practice, and Red Hat is well known for it. Check /usr/bin, /etc and a few others. Many of the files would [normally] be located under /usr/local.
Pardon my probable misunderstanding, but isn't this the whole idea of the standard filesystem layout for Unix/Unix-like operating systems?
I.e., if it's not installed through the system package manager, it has its tree of /lib, /sbin, /bin, /man, /share, etc under /usr/local; whereas if it *is* installed through the system packaging, it has its own /lib, /sbin, /bin, /share, /man, etc layout tree under / or /usr, depending on various factors like partitioning or network-mounting /usr, etc.
That is how Red Hat and it's offshoots work.
It can be a contentious issue, exactly where to put non base system files. Sun Microsystems used to prefer /opt but almost everyone else uses /usr/local for most add on software. But it is well understood that :
/bin, /sbin and /lib are for single user base system commands and their required libraries.
That's not quite the case. Check the facts at http://www.pathname.com/ (and fall in love with Enya while you're there).
That is from January 2004, and appears to be Red Hat centric. I learnt about file system hierarchy in the 1980's. Just because someone claims to be in charge of some kind of official web page on file system hierarchy does mean it is the official recommendations for all distributions and Unix variants.
See: http://www.debian.org/doc/packaging-manuals/fhs/ http://www.isu.edu/departments/comcom/unix/workshop/fstour.html http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dirstructure.html
Recommended best practises have been around much longer than Linux. The first distributions of Linux favoured the classic best practises, and many distributions still do.
There are many good reasons for the classic hierarchy and a lot of study and research went into it's development. Many of the reasons still hold, but most are too complex to deal with in this forum. Some things to consider are frequency of modification, accessability, ownership and type or purpose of the file. Other things to remember are that large numbers of files in a directory and large variations in the file sizes on a partition can have detrimental effects on performance and more specifically on this topic, can be the cause of fragmentation.
If you can take one fact from this conversation, it is that the system you use depends on how you prefer to manage your system. There may be a standard that Red Hat and it's adopters prefer, but there is no official standard that must be followed as is demonstrated by differences in Unix variants and Linux distributions.
Guy Fraser wrote:
That's not quite the case. Check the facts at http://www.pathname.com/ (and fall in love with Enya while you're there).
That is from January 2004, and appears to be Red Hat centric.
It's not RH-centric.It applies equally to all distros.
I learnt about file system hierarchy in the 1980's. Just because
So? Linux isn't Unix, it's only very similar.
someone claims to be in charge of some kind of official web page on file system hierarchy does mean it is the official recommendations for all distributions and Unix variants.
But it is. Well, originally it was Linux-only. I don't know the status of Unix compliance (not relevant here, this is Linux), but I note one of the maintainers, Rusty Russell, is a well-known Linux guru with particular credentials in the kernel.
Copyright (C) 1994-2000 Daniel Quinlan Same chap I pointed you at.
http://www.isu.edu/departments/comcom/unix/workshop/fstour.html
So? Linux isn't Unix, it's only very similar.
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dirstructure.html
So? Linux isn't Unix, it's only very similar.
Read up on LSB 3.0 ch 16. It happens to refer to the link I gave earlier.
On Wed, 2005-12-28 at 10:27, Guy Fraser wrote:
It can be a contentious issue, exactly where to put non base system files. Sun Microsystems used to prefer /opt but almost everyone else uses /usr/local for most add on software. But it is well understood that :
/opt is for third-party things packaged with installers. /usr/local is your own. With appropriate PATH settings you can have the system, pre-packaged, and your own compiled versions installed and execute the one of your choice.
/bin, /sbin and /lib are for single user base system commands and their required libraries.
In case no one has mentioned it yet, these are the things that may be needed before /usr is mounted, given that /usr can be a separate partition.
The real theological battle is what is considered part of the base system. Most Unix variants and Linux distributions consider only essential software to be part of the base.
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict. Sometimes that's what you want (say for an update to a stock program). Sometimes it isn't. When it isn't, your system may not work any more. That means you have to be very careful about installing any packages that aren't part of the distribution.
Les Mikesell wrote:
On Wed, 2005-12-28 at 10:27, Guy Fraser wrote:
It can be a contentious issue, exactly where to put non base system files. Sun Microsystems used to prefer /opt but almost everyone else uses /usr/local for most add on software. But it is well understood that :
/opt is for third-party things packaged with installers. /usr/local is your own. With appropriate PATH settings you can have the system, pre-packaged, and your own compiled versions installed and execute the one of your choice.
/bin, /sbin and /lib are for single user base system commands and their required libraries.
In case no one has mentioned it yet, these are the things that may be needed before /usr is mounted, given that /usr can be a separate partition.
More to the point, on another machine. And/or shared with other systems.
I know of a chap who ran 40,000+ Linux virtual machines on one zBox a few years ago. (Note, I didn't say he had tens of thousands of useful Linux systems, the underlying zBoz was very busy running Linux, but not very busy doing work for people. Hundreds, however, is relatively common). He'd not have wanted to give each its own /usr (8.5 Gb here).
That it where the FHS and its requirement that /usr be ro (so don't go building kernels there) are important.
The real theological battle is what is considered part of the base system. Most Unix variants and Linux distributions consider only essential software to be part of the base.
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict. Sometimes that's what you
Never clobbering vendor-provided files unless, you the system administrator, make them do so. And it's perfectly possible to make relocatable rpms that will install almost anywhere.
I used Debian for a couple of years, and early on went mad getting extra packages from here, there, everywhere. All installed into the dpkg-managed vendor space just as most rpms do on FC.
On Wed, 2005-12-28 at 17:43, John Summerfied wrote:
/bin, /sbin and /lib are for single user base system commands and their required libraries.
In case no one has mentioned it yet, these are the things that may be needed before /usr is mounted, given that /usr can be a separate partition.
More to the point, on another machine. And/or shared with other systems.
Regardless, the big issue is that all the tools needed for booting and mounting it must be in root partition - and in the shared case that means bringing up the network too.
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict. Sometimes that's what you
Never clobbering vendor-provided files unless, you the system administrator, make them do so.
I'm not sure I understand. How do you install updated or modified libraries from a third-party yum repository without clobbering the system files?
And it's perfectly possible to make relocatable rpms that will install almost anywhere.
Where do you find them?
Les Mikesell wrote:
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict. Sometimes that's what you
Never clobbering vendor-provided files unless, you the system administrator, make them do so.
I'm not sure I understand. How do you install updated or modified libraries from a third-party yum repository without clobbering the system files?
If you install a third-party glibc designed to update your Fedora or RH system, that's one thing. Think the legacy project for RHL 7.3 and FC < FC2.
If you install wottawidget and it includes (eg) /usr/bin/ldd then rpm will not do it unless you (the sysadmin) apply a little force*
And it's perfectly possible to make relocatable rpms that will install almost anywhere.
Where do you find them?
?? Anywhere the packager distributes them.
Read the rpm documentation.
* It's possible for wottawidget to obsolete glibc, but I'd regard that as a malicious act, little different from including Evil Acts in the rpm scripts where they will be run as root.
Hi
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict.
It shouldnt do that. RPM packages are supposed to create a .rpmnew file if the preexisting configuration files are modifed. The rules governing such changes are documented in detail in several guides. See http://fedoraproject.org/wiki/Tools/RPM. If packages dont honor such rules then it can be consider a bug with those specific packages rather than RPM itself. Do file bug reports as appropriate.
On Wed, 2005-12-28 at 17:52, Rahul Sundaram wrote:
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict.
It shouldnt do that. RPM packages are supposed to create a .rpmnew file if the preexisting configuration files are modifed. The rules governing such changes are documented in detail in several guides. See http://fedoraproject.org/wiki/Tools/RPM. If packages dont honor such rules then it can be consider a bug with those specific packages rather than RPM itself. Do file bug reports as appropriate.
I'm talking about things that depend on updated/modified versions of stock libraries or other tools like you find in some third-party repositories.
Hi
I'm talking about things that depend on updated/modified versions of stock libraries or other tools like you find in some third-party repositories.
Well thats not a RPM issue either. Thats a effect of packaging that the package maintainers provide by design.
On Wed, 2005-12-28 at 18:30, Rahul Sundaram wrote:
I'm talking about things that depend on updated/modified versions of stock libraries or other tools like you find in some third-party repositories.
Well thats not a RPM issue either. Thats a effect of packaging that the package maintainers provide by design.
Yes, but it falls out of the scheme that third party packages live in the same space as the distribution. If, for example, they were built to reside under /opt instead of replacing system files there would be a lot less chance of breakage.
Hi
Yes, but it falls out of the scheme that third party packages live in the same space as the distribution. If, for example, they were built to reside under /opt instead of replacing system files there would be a lot less chance of breakage.
There isnt really much stopping the repositories from making use of /opt for such purposes. If there are I havent heard of it yet. There are probably some issues over how path order is set to take use of whatever libraries that the packages are modifying that should take precedence for them.
Les Mikesell wrote:
On Wed, 2005-12-28 at 17:52, Rahul Sundaram wrote:
And more to the point, where third-party RPMs fit into the picture. In RedHat-land they are almost always made to fit into the vendor-provided scheme, clobbering system files if there is a conflict.
It shouldnt do that. RPM packages are supposed to create a .rpmnew file if the preexisting configuration files are modifed. The rules governing such changes are documented in detail in several guides. See http://fedoraproject.org/wiki/Tools/RPM. If packages dont honor such rules then it can be consider a bug with those specific packages rather than RPM itself. Do file bug reports as appropriate.
I'm talking about things that depend on updated/modified versions of stock libraries or other tools like you find in some third-party repositories.
If they're well designed, they should just fit in along with official packages and in every respect be suitable replacements for them.
I'd make exceptions for betaware and the like, such as (maybe) KDE 3.5 or 4.0 before their official release. In such cases, one would wish to have both and choose between them cleanly.
On Thu, 2005-12-29 at 19:37, John Summerfied wrote:
I'm talking about things that depend on updated/modified versions of stock libraries or other tools like you find in some third-party repositories.
If they're well designed, they should just fit in along with official packages and in every respect be suitable replacements for them.
That would mean there would be a simple piecemeal upgrade path from any version of a distribution to any other. That doesn't seem to be the case and it's even less likely when the reason you are installing a different version is to get features not included in the stock distribution.
On Wed, 2005-12-28 at 07:57 +0800, John Summerfied wrote:
Might I observe that the many-partitions layout so often recommended gives you all the disadvantages of a fragmented drive from day one?
Though, that would be in situation where the system is trying to read several different files at once. And even with one partition, it can still act that way, because your commands and your data files (for instance) might be nowhere near each other on the disk structure.
If you were doing something that needed very little interruptions of data traffic, like high resolution video capture straight to disk, I'd be recommending that you had a separate drive for such things, on a different host controller, too.
But I doubt, that for most people, the differences in timing between one partition or many are going to be that noticeable. I've run systems using various permutations of partitions, none was really noticeably faster than others. My main like for partitioning is other reasons.
e.g. Ever had to sit through an 80 meg drive being checked, because one file that you probably don't care about caused the system to do so? On a multi-partition system, if I see a warning about a error in /tmp, I won't fsck the drive.
And, as you say, backing up home, or updating a system leaving home untouched, and that sort of thing. Not that I've noticed a way to get Linux to leave one partition alone during the installation routine, and use it afterwards.
There's one quite serious potential problem if you don't partition: Something that would otherwise have filled up a /tmp partition that's easy for you to work around to fix up, can fill up the entire drive space, and that's much harder to deal with.
Tim wrote:
On Wed, 2005-12-28 at 07:57 +0800, John Summerfied wrote:
Might I observe that the many-partitions layout so often recommended gives you all the disadvantages of a fragmented drive from day one?
Though, that would be in situation where the system is trying to read several different files at once.
In a multi-tasking multi-user system that's quite probable. SUSE, for reasons I don't understand, tries to run the daily housekeeping every 24 hours, timed from when the system last booted: if you boot at 8:00 then the housekeeping's done then and at 8:00 daily until the next boot.
I'm often using the system at 06:00 (Debian's time) and even sometimes at 04:00.
Then there's my tendency to run programs to read files and stuff their contents into a postgresql database on the same system, do download details of homes for sale and process the HTML to remove ads and other noise, or to point Mozilla at theregister.co.uk and then middle-click (opens links in a new tab) every unread interesting-looking story.
And even with one partition, it can still act that way, because your commands and your data files (for instance) might be nowhere near each other on the disk structure.
"can act that way" is quite different from "forcing it to act that way."
If you were doing something that needed very little interruptions of data traffic, like high resolution video capture straight to disk, I'd be recommending that you had a separate drive for such things, on a different host controller, too.
But I doubt, that for most people, the differences in timing between one partition or many are going to be that noticeable. I've run systems using various permutations of partitions, none was really noticeably faster than others. My main like for partitioning is other reasons.
OTOH years ago, when I used OS/2, I had two drives and thought my data drive was the right one for swap. I was very quickly disabused of that notion: my data got hit way harder than programs.
e.g. Ever had to sit through an 80 meg drive being checked, because one file that you probably don't care about caused the system to do so? On a multi-partition system, if I see a warning about a error in /tmp, I won't fsck the drive.
On OS/2, before IBM fixed chkdsk, I used to sit through 45-minute checks at least one a day. I don't recall ever waiting that long for Linux, and it rarely needs to do more than a cursory check anyway.
And, as you say, backing up home, or updating a system leaving home untouched, and that sort of thing. Not that I've noticed a way to get Linux to leave one partition alone during the installation routine, and use it afterwards.
There's one quite serious potential problem if you don't partition: Something that would otherwise have filled up a /tmp partition that's easy for you to work around to fix up, can fill up the entire drive space, and that's much harder to deal with.
Umm. I find the more partitions I have the more likely I am to run out of space somewhere. Regarding /tmp, I rather like the Debian approach: delete everything in it on reboot. (Actually, I'd like to leave it alone when booting to single-user mode as important diagnostic info can get lost, but that's a complaint for another forum).
I don't think I've ever found multiple partitions advantageous, except for multibooting. One case comes vividly to mind: /dev/sda died, and as it was in its final throes I backed it uo. However, of the several gigglebyte tarball, only the root partition was readable, and that of course was the least useful. /var, containing email and such, was a writeoff.
Of course, this is wandering a little from defraggers. I don't see a use for them:-)
<snip>
However, still the question remains. If Linux ext3 doesn't need defragmenter, and able to defrag itself, what is the process name? And when does it run? Can I see it in action? Is there an utility to see on what percentage my current defragmentation? I tried fschk but no luck
Linux does NOT need a defrager. There is an argument to be me that on a LINUX system with many processes accessing a pretty dynamic file system a bit of fragmentation can help throughput. Take squint at:
http://www.salmar.com/pipermail/wftl-lug/2002-March/000603.html
I have run large active servers with dynamic file systems that ran Linux for years rebooting just to up date the OS, with never doing more file system maintenance beyond removing file or moving them to other partitions. In my early UNIX days the BSD Fast File System used 10% free space on its file systems. I note that modern Linux file systems are created with 5% of free space. I think that is a bit tight even with improvements in design. You can check free space in the values from a df of your file systems. The file system free space is
free space = total blocks - (used + available)
That is why you can see a file system at more than a 100% utilization.
The opinion that EXT2 doesn't need defragmenting is based on only a filesystem-level view of the problem, and doesn't consider data read and write performance. EXT2 does make an effort to keep data only a short seek away ("clustered"). With this clustering, the filesystem operations of adding, removing, extending, and shortening files are not much affected by fragmentation.
I do not understand this comment as disk I/O is largely about what file system designs are about. Integrity, and reliability are important considerations true enough, but if a Linux system is servicing numerous client processes it is receiving requests to read or write data randomly around the disk. With disk read-ahead,and disk address sorting a well designed driver accessing a Linux file system does not notice file fragmentation. The fragmentation is really an issue higher up in the OS. That part of the file system code which deals with the allocations of inodes and data blocks (and in file systems that support them, file system fragments). Things can get really ugly here while at a lower lever things tend to perk along.
File systems that support "file system fragments" use large "file system blocks" say 4K which is the basic disk I/O size, but allocate data by the fragment 1K or 2k. These are design issues one of the benefits of this design is an aid to clustering of data with in a single physical disk I/O operation and makes an accommodation to file systems with large files measured in file system blocks and smaller files for which a small number of fragments is appropriate. It does complicate the file system code, some.
Actually the Linux EXT[23] file system structures contain a fragment size, but the fragment size is the same as the block size.
<snip>\
When you have only 2% free, it's just about certain that the free space is a long way away from the rest of the data in a file. Just deleting to get 20% free would probably have fixed your problem.
Absolutely. Running with just that much free on a Linux system is insanity, anyway. I'd be afraid of a system lockup.
As I said above I am not sure that even 5% is enough free space, 2% is clearly too little.
dlg
Tony Nelson <tonynelson <at> georgeanelson.com> writes:
The opinion that EXT2 doesn't need defragmenting is based on only a filesystem-level view of the problem, and doesn't consider data read and write performance. EXT2 does make an effort to keep data only a short seek away ("clustered"). With this clustering, the filesystem operations of adding, removing, extending, and shortening files are not much affected by fragmentation.
With EXT3 (journalling), which always writes data to a new place, updates the filesystem info, and then frees the old data (roughly speaking), fragmentation is a way of life, and there isn't much to be done about it. Clustering helps by keeping the seeks relatively short, if there is space nearby.
When you have only 2% free, it's just about certain that the free space is a long way away from the rest of the data in a file. Just deleting to get 20% free would probably have fixed your problem.
You are confused about how journalling works. The new blocks are written to the journal, then when the change is comitted, the blocks are written to their normal place and the journal reused. For a recovery, the journal is replayed, writing changes to the normal location on the disk. Since the journal file is usually already allocated, the new blocks are pretty much in the same place as they would have been without it. Most people run ext3 in the default mode where only the metadata is journalled.
Log-structured filesystems are where the entire filesystem is the journal. The changes are written to the end and the old blocks left in the place. This results in better write performance but bad fragmentation.
- Ian
On Fri, 2005-12-23 at 14:32 +0700, Fajar Priyanto wrote:
My own experience shows me just that. My /home partition was almost full with only 2% freespace. During that time, my Kmail became very slow such as when downloading email or when I moved between mail folders. The harddisk was just spinning all the time. Then I copy all my files and mails from the /home partition and move them all to another partition. Then delete them from /home. After that, I copied some of the files and mail back to /home in order to keep 20% of /home free. So far the performance is ok.
It is a good policy to never exceed 80% usage on any partition, precisely for performance reasons. I am not certain how much each filesystem type (Ext2/3, XFS, FAT32) is affected, but 80...85% is a good number to keep in mind. Exceed it, and the performance usually starts to drop dramatically.
On Sun, 2006-01-01 at 12:35 -0800, Florin Andrei wrote:
It is a good policy to never exceed 80% usage on any partition, precisely for performance reasons. I am not certain how much each filesystem type (Ext2/3, XFS, FAT32) is affected, but 80...85% is a good number to keep in mind. Exceed it, and the performance usually starts to drop dramatically.
Agreed in full. I generally make my partitions a lot larger than I expect them to be used, for this very reason. Granted, it could be easier for me, as disk space really *is* inexpensive where I live; whereas it may not be in other areas...
On Sun, 2006-01-01 at 12:44 -0800, Peter Gordon wrote:
On Sun, 2006-01-01 at 12:35 -0800, Florin Andrei wrote:
It is a good policy to never exceed 80% usage on any partition, precisely for performance reasons. I am not certain how much each filesystem type (Ext2/3, XFS, FAT32) is affected, but 80...85% is a good number to keep in mind. Exceed it, and the performance usually starts to drop dramatically.
Agreed in full. I generally make my partitions a lot larger than I expect them to be used, for this very reason. Granted, it could be easier for me, as disk space really *is* inexpensive where I live; whereas it may not be in other areas...
This is all good, but sometimes you do run out of space. I've seen several servers run out of space at night due to runaway processes or something similarly uncontrollable situation.
Another possibility is large mailservers, databases, news spools or similar files that grow/shrinks/removes data in the middle etc.
So IF a partition has gone over 90% you should expect a great deal of fragmentation, but no matter how bad it gets you cant just do a quick fixer-up since we have no online defragmentation program.
I too see 250+ extents on binaries like /usr/sbin/mplayer, but that potential problem has been dismissed since ELF doesn't load the file up sequentially. (but readahead would have helped)
Now, what I see as a bigger problem is my home directory.. A little excerpt:
./.evolution/mail/local/Inbox.sbd/Devel.sbd/linux-kernel: 1718 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/Wine-devel: 1472 extents ./.mozilla/firefox/jajiw9vd.default/Cache/_CACHE_003_: 1150 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/fedora-devel: 1139 extents ./.spamassassin/bayes_toks: 970 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/linux-raid: 863 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/fedora-users: 855 extents ./.mozilla/default/lxaxhxet.slt/Cache/_CACHE_003_: 777 extents ./.spamassassin/bayes_seen: 691 extents ./.evolution/mail/local/Inbox: 537 extents
Now, I hope no one actually thinks this kind of fragmentation is just how it should be. True, I will survive even with this but I'd rather it wasn't.
I've used this non-opensource util at a few occations, and had good results. Problems are: Not opensource, beta, offline-defrag. http://www.oo-software.com/en/products/oodlinux/index.html
I would like to see an open-source defragmenter for ext3, if just for those worst-case scenarios. Fragmentation CAN happen, so there is a need for some.
I'm posting this mostly to encourage those who might think of making an open source defragmenter.
For the record, this system was installed using FC4 isos from scratch and has followed the stream of updates closely.
-HK
On Mon, 2006-01-02 at 04:02, Hans Kristian Rosbach wrote:
I would like to see an open-source defragmenter for ext3, if just for those worst-case scenarios. Fragmentation CAN happen, so there is a need for some.
There's always backup/mkfs/restore. If you aren't prepared to restore you should probably take care of that before worrying about efficiency.
On Mon, 2006-01-02 at 12:17 -0600, Les Mikesell wrote:
On Mon, 2006-01-02 at 04:02, Hans Kristian Rosbach wrote:
I would like to see an open-source defragmenter for ext3, if just for those worst-case scenarios. Fragmentation CAN happen, so there is a need for some.
There's always backup/mkfs/restore. If you aren't prepared to restore you should probably take care of that before worrying about efficiency.
Do you know how long time it would take to put back 800GB of mailboxes on our mailserver from the tape backup? No customers in the world would allow us to actually take such a time-off from serving their mail.
Granted we COULD set up another box with a 1TB raid5 array to hold the data temporarily, but it would still take time since we would still have to shut down mail services before taking a copy. It would still be a bad workaround for a real problem. (Yes I know that ~99% of us don't ever need it)
Thus we would need an on-line defragmenter that we could set up for a low-prio night run once or twice a year.
-HK
On Tue, 2006-01-03 at 02:35, Hans Kristian Rosbach wrote:
There's always backup/mkfs/restore. If you aren't prepared to restore you should probably take care of that before worrying about efficiency.
Do you know how long time it would take to put back 800GB of mailboxes on our mailserver from the tape backup? No customers in the world would allow us to actually take such a time-off from serving their mail.
How do you plan to deal with an operator or software error that erases your current files?
Granted we COULD set up another box with a 1TB raid5 array to hold the data temporarily, but it would still take time since we would still have to shut down mail services before taking a copy. It would still be a bad workaround for a real problem. (Yes I know that ~99% of us don't ever need it)
You can do a swap like this with very little downtime if you do an rsync while the system is live to get most of the data copied, the repeat the rsync with the --delete option to catch any subsequent changes with the system offline, then swap mountpoints.
Thus we would need an on-line defragmenter that we could set up for a low-prio night run once or twice a year.
If you use maildir or other 1-message-per-file format, a defragmenter isn't going to help much because it won't know to move the directory contents together. If you have standard mbox format you just have to re-write them periodically which will happen anyway if the users ever delete anything.
On Tue, 2006-01-03 at 07:52 -0600, Les Mikesell wrote:
On Tue, 2006-01-03 at 02:35, Hans Kristian Rosbach wrote:
There's always backup/mkfs/restore. If you aren't prepared to restore you should probably take care of that before worrying about efficiency.
Do you know how long time it would take to put back 800GB of mailboxes on our mailserver from the tape backup? No customers in the world would allow us to actually take such a time-off from serving their mail.
How do you plan to deal with an operator or software error that erases your current files?
That has nothing to do with the current discussion.
But that would of course result in data loss and downtime while data is recovered from backup. Both covered by the contract agreement.
Granted we COULD set up another box with a 1TB raid5 array to hold the data temporarily, but it would still take time since we would still have to shut down mail services before taking a copy. It would still be a bad workaround for a real problem. (Yes I know that ~99% of us don't ever need it)
You can do a swap like this with very little downtime if you do an rsync while the system is live to get most of the data copied, the repeat the rsync with the --delete option to catch any subsequent changes with the system offline, then swap mountpoints.
Yes, and this is soo much easier than running a program at night.
Thus we would need an on-line defragmenter that we could set up for a low-prio night run once or twice a year.
If you use maildir or other 1-message-per-file format, a defragmenter isn't going to help much because it won't know to move the directory contents together. If you have standard mbox format you just have to re-write them periodically which will happen anyway if the users ever delete anything.
mbox.
pop3 users are usually ok, imap on the other hand =/
And the database holding userdata and statistics still won't often get such relief.
-HK
PS: I am fully aware of the possible workarounds, and do exercise them when needed. The point here is that there is indeed a need for an online defragmenter.
On Tue, 2006-01-03 at 08:05, Hans Kristian Rosbach wrote:
You can do a swap like this with very little downtime if you do an rsync while the system is live to get most of the data copied, the repeat the rsync with the --delete option to catch any subsequent changes with the system offline, then swap mountpoints.
Yes, and this is soo much easier than running a program at night.
The program that exists only in your imagination? Let us know how that works out for you...
On Tue, 2006-01-03 at 15:05 +0100, Hans Kristian Rosbach wrote:
On Tue, 2006-01-03 at 07:52 -0600, Les Mikesell wrote:
On Tue, 2006-01-03 at 02:35, Hans Kristian Rosbach wrote:
There's always backup/mkfs/restore. If you aren't prepared to restore you should probably take care of that before worrying about efficiency.
Do you know how long time it would take to put back 800GB of mailboxes on our mailserver from the tape backup? No customers in the world would allow us to actually take such a time-off from serving their mail.
How do you plan to deal with an operator or software error that erases your current files?
That has nothing to do with the current discussion.
But that would of course result in data loss and downtime while data is recovered from backup. Both covered by the contract agreement.
And you are saying that routine maintenance is not covered by the contract agreement? In every situation I have been in routine planned maintenance is more often accepted than an outage. After all, routine maintenance usually has no data loss (only an inconvenience) and an outage often does have data loss as well as inconvenience.
Granted we COULD set up another box with a 1TB raid5 array to hold the data temporarily, but it would still take time since we would still have to shut down mail services before taking a copy. It would still be a bad workaround for a real problem. (Yes I know that ~99% of us don't ever need it)
You can do a swap like this with very little downtime if you do an rsync while the system is live to get most of the data copied, the repeat the rsync with the --delete option to catch any subsequent changes with the system offline, then swap mountpoints.
Yes, and this is soo much easier than running a program at night.
Thus we would need an on-line defragmenter that we could set up for a low-prio night run once or twice a year.
If you use maildir or other 1-message-per-file format, a defragmenter isn't going to help much because it won't know to move the directory contents together. If you have standard mbox format you just have to re-write them periodically which will happen anyway if the users ever delete anything.
mbox.
pop3 users are usually ok, imap on the other hand =/
And the database holding userdata and statistics still won't often get such relief.
-HK
PS: I am fully aware of the possible workarounds, and do exercise them when needed. The point here is that there is indeed a need for an online defragmenter.
Agreed
I have held off making didactic comments on this topic but de-fragmenting a Unix or Linux file system just is not the way the filesystem was supposed to work. 5%-10% of the disk is set aside so that files are re-written in this buffer before writing to the disk,
No program is ever going to able to guess whether two different files are supposed to near each other on the disk.
When you do a fsck on a disk you discover how much fragmenting there is and it usually rather minimal.
Now I expect to hear from the de-fragmenting people but such is life.
Hans Kristian Rosbach wrote:
On Mon, 2006-01-02 at 12:17 -0600, Les Mikesell wrote:
On Mon, 2006-01-02 at 04:02, Hans Kristian Rosbach wrote:
I would like to see an open-source defragmenter for ext3, if just for those worst-case scenarios. Fragmentation CAN happen, so there is a need for some.
There's always backup/mkfs/restore. If you aren't prepared to restore you should probably take care of that before worrying about efficiency.
Do you know how long time it would take to put back 800GB of mailboxes on our mailserver from the tape backup? No customers in the world would allow us to actually take such a time-off from serving their mail.
You weren't expecting to defragment a mounted, active filesystem, were you? The kernel isn't going to take kindly to having a mounted filesystem get restructured on disk. How long would the mail server have to be offline while you unmounted the filesystem and defragmented it? OK, probably less than the time to backup and restore from tape, but it's still going to be a while.
If all you want is a way to fix some badly fragmented files, that can be done with a shell script build around 'filefrag', 'sort', and 'sed'.
On Mon, 2006-01-02 at 11:02 +0100, Hans Kristian Rosbach wrote:
This is all good, but sometimes you do run out of space. I've seen several servers run out of space at night due to runaway processes or something similarly uncontrollable situation.
Yeah, S*%t Happens (TM).
Now, what I see as a bigger problem is my home directory.
/home usually gets very fragmented, indeed. That's usually taken care of when you're going to the next Fedora release. One more reason to backup/format/reinstall instead of upgrade.
On Mon, 2006-01-02 at 10:48 -0800, Florin Andrei wrote:
On Mon, 2006-01-02 at 11:02 +0100, Hans Kristian Rosbach wrote:
This is all good, but sometimes you do run out of space. I've seen several servers run out of space at night due to runaway processes or something similarly uncontrollable situation.
Yeah, S*%t Happens (TM).
Now, what I see as a bigger problem is my home directory.
/home usually gets very fragmented, indeed. That's usually taken care of when you're going to the next Fedora release. One more reason to backup/format/reinstall instead of upgrade.
If that is the schedule I suggest we start pumping out FC releases every 3 months instead so we can catch up with the fragmentation issue. Oh, and state this as a reason not to use the FC upgrade feature.
Still a bad workaround.
-HK
On Tue, 2006-01-03 at 09:39 +0100, Hans Kristian Rosbach wrote:
On Mon, 2006-01-02 at 10:48 -0800, Florin Andrei wrote:
/home usually gets very fragmented, indeed. That's usually taken care of when you're going to the next Fedora release. One more reason to backup/format/reinstall instead of upgrade.
If that is the schedule I suggest we start pumping out FC releases every 3 months instead so we can catch up with the fragmentation issue. Oh, and state this as a reason not to use the FC upgrade feature.
What I implied was the opposite.
I did not say that releasing new FC versions is the solution to the fragmentation problem. I said that an OS upgrade is a good opportunity to get rid of the entropy that accumulates on the disk, fragmentation being just one aspect. But it's not so bad as to require an aggressive OS upgrade policy.
opportunity versus requirement - see the difference?
On Mon, 2006-01-02 at 11:02 +0100, Hans Kristian Rosbach wrote:
On Sun, 2006-01-01 at 12:44 -0800, Peter Gordon wrote:
On Sun, 2006-01-01 at 12:35 -0800, Florin Andrei wrote:
It is a good policy to never exceed 80% usage on any partition, precisely for performance reasons. I am not certain how much each filesystem type (Ext2/3, XFS, FAT32) is affected, but 80...85% is a good number to keep in mind. Exceed it, and the performance usually starts to drop dramatically.
Agreed in full. I generally make my partitions a lot larger than I expect them to be used, for this very reason. Granted, it could be easier for me, as disk space really *is* inexpensive where I live; whereas it may not be in other areas...
This is all good, but sometimes you do run out of space. I've seen several servers run out of space at night due to runaway processes or something similarly uncontrollable situation.
Another possibility is large mailservers, databases, news spools or similar files that grow/shrinks/removes data in the middle etc.
So IF a partition has gone over 90% you should expect a great deal of fragmentation, but no matter how bad it gets you cant just do a quick fixer-up since we have no online defragmentation program.
I too see 250+ extents on binaries like /usr/sbin/mplayer, but that potential problem has been dismissed since ELF doesn't load the file up sequentially. (but readahead would have helped)
Now, what I see as a bigger problem is my home directory.. A little excerpt:
./.evolution/mail/local/Inbox.sbd/Devel.sbd/linux-kernel: 1718 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/Wine-devel: 1472 extents ./.mozilla/firefox/jajiw9vd.default/Cache/_CACHE_003_: 1150 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/fedora-devel: 1139 extents ./.spamassassin/bayes_toks: 970 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/linux-raid: 863 extents ./.evolution/mail/local/Inbox.sbd/Devel.sbd/fedora-users: 855 extents ./.mozilla/default/lxaxhxet.slt/Cache/_CACHE_003_: 777 extents ./.spamassassin/bayes_seen: 691 extents ./.evolution/mail/local/Inbox: 537 extents
Now, I hope no one actually thinks this kind of fragmentation is just how it should be. True, I will survive even with this but I'd rather it wasn't.
I agree fragmentation is irritating, but the examples you give are not duplicated on my system. I checked the same directory ./.evolution/mail/ and found the largest # of extents on ./.evolution/mail/sql-ledger: 97 extents found This is a mailbox that I do not routinely delete messages from so it has been growing for many months. In contrast, my fedora mailbox has ./.evolution/mail/fedora: 30 extents found even though the traffic is over 100x the volume, but this one I regularly delete messages from.
The only assumption I can make is that as files grow additional extents are added and when data is deleted the extents may remain assigned so they are reused (or the assigning/releasing maintains a mostly static quantity).
I've used this non-opensource util at a few occations, and had good results. Problems are: Not opensource, beta, offline-defrag. http://www.oo-software.com/en/products/oodlinux/index.html
I would like to see an open-source defragmenter for ext3, if just for those worst-case scenarios. Fragmentation CAN happen, so there is a need for some.
I'm posting this mostly to encourage those who might think of making an open source defragmenter.
There are defragmenting tools for some *nix type systems. IBMs AIX has defragfs for this purpose.
For the record, this system was installed using FC4 isos from scratch and has followed the stream of updates closely.
-HK
-
akonstam＠trinity.edu -
Arthur Pemberton -
David L. Gehrt -
Fajar Priyanto -
Florin Andrei -
Guy Fraser -
Hans Kristian Rosbach -
Ian Burrell -
jdow -
Jeff Vian -
Jim Cornette -
John Pierce -
John Summerfield -
Les Mikesell -
Mike McCarty -
Peter Gordon -
Rahul Sundaram -
Robert Nichols -
Tim -
Tony Nelson