I have two very large directories.
One an rsync of all the RFCs and that is up to 6456 files. But all the Internet Drafts (including all expired ones) is 72,509 files. Then I have a number of large directories with IEEE 802 documents, but rarely over 3,000 per WG/year (eg ..../802.15/11/ for .15 document numbers assigned in 2011).
So anyway, Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
2012/3/16, Robert Moskowitz rgm@htt-consult.com:
I have two very large directories.
One an rsync of all the RFCs and that is up to 6456 files. But all the Internet Drafts (including all expired ones) is 72,509 files. Then I have a number of large directories with IEEE 802 documents, but rarely over 3,000 per WG/year (eg ..../802.15/11/ for .15 document numbers assigned in 2011).
So anyway, Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
I don't know how fast it is on such big directories, but you could try emacs dired. Once you have the directory listed, searching (normal or incremental and/or regexp search) will be blazingly fast. And perhaps view the files from within emacs with w3m (although I'm pretty sure that Firefox can be arranged).
Andras
On Thu, 2012-03-15 at 23:01 -0700, Robert Moskowitz wrote:
Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
Nautilus seems to sniff the files to discover their types, and a plug-in tries to generate a thumbnail image for the file. Both these features are painfully slow with moderately largish directories. You can turn off the inclusion of the filetype in the list details, but not the image generation. Perhaps the plug-in could be manually replaced with something which worked faster, or instantly "did nothing."
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
On Fri, Mar 16, 2012 at 13:31, Tim ignored_mailbox@yahoo.com.au wrote:
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
Another suggestion would be use Thunar. Thunar lets you turn off the thumbnails (Edit >> Preferences >> Display), but I'm not sure whether it is the thumbnail generation or just the display.
Another option would be to try a find/locate gui front end. I used to use one which came with XFCE, but I forgot the name now that I use the command line most of the time.
On Fri, Mar 16, 2012 at 11:01:03PM +1030, Tim wrote:
On Thu, 2012-03-15 at 23:01 -0700, Robert Moskowitz wrote:
Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
Nautilus seems to sniff the files to discover their types, and a plug-in tries to generate a thumbnail image for the file. Both these features are painfully slow with moderately largish directories. You can turn off the inclusion of the filetype in the list details, but not the image generation. Perhaps the plug-in could be manually replaced with something which worked faster, or instantly "did nothing."
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal. A "problem" of long-standing on pretty much all Unix(-ish) file systems (and I'm not saying it won't happen on other systems, I don't really know, but suspect it does.)
Am 17.03.2012 00:25, schrieb fred smith:
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal. A "problem" of long-standing on pretty much all Unix(-ish) file systems (and I'm not saying it won't happen on other systems, I don't really know, but suspect it does.)
it is ALWAYS a problem of the filemanager and NEVER of the FS a filemanager counting subitems, generating previews and in the worst case waits until he has the whole folder enumerated sucks with many files, the second worst is konqueror currently with his "count items"-behavior even on sshfs-mounts
the really worst case is Apple Finder, open a folder with 30000 files and it waits until he can display the whole content
on the same fileserver (fedora with smb/netatalk) konqueror over SMB starts to display the folder long before it has the whole listing-informations and while you scroll down the list is growing
the filesystem itself is not interested if there are 30000 files in one folder or 10 folders with 1000 files
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal. A "problem" of long-standing on pretty much all Unix(-ish) file systems (and I'm not saying it won't happen on other systems, I don't really know, but suspect it does.)
Ext3, ext4 and recent file systems have hashed directory indexing so this is no longer true. Plus a window of files isn't searching, its simply scanning the directory which is fast.
If you have spinning rust however and nautilus is trying to read content from each file the cost of the disk seeks to each file will kill your performance pretty effectively. Disk seek time hasn't really improved in years as its down the limits of physics.
The cure ultimately is probably to use more subdirectories 8)
Alan
On 03/16/2012 04:25 PM, fred smith wrote:
On Fri, Mar 16, 2012 at 11:01:03PM +1030, Tim wrote:
On Thu, 2012-03-15 at 23:01 -0700, Robert Moskowitz wrote:
Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
Nautilus seems to sniff the files to discover their types, and a plug-in tries to generate a thumbnail image for the file. Both these features are painfully slow with moderately largish directories. You can turn off the inclusion of the filetype in the list details, but not the image generation. Perhaps the plug-in could be manually replaced with something which worked faster, or instantly "did nothing."
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal. A "problem" of long-standing on pretty much all Unix(-ish) file systems (and I'm not saying it won't happen on other systems, I don't really know, but suspect it does.)
Many file managers not only generate thumbs, stats and other things, but if the directory is quite large there's a significant time spent sorting the filenames into alphabetical order. For example, from the command line go to a large directory and time the difference between an "ls" and an "ls -f". It can be frightening (e.g. checking the mail spool of a busy mail server). ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Is that a buffer overflow or are you just happy to see me? - ----------------------------------------------------------------------
On Fri, 2012-03-16 at 19:25 -0400, fred smith wrote:
On Fri, Mar 16, 2012 at 11:01:03PM +1030, Tim wrote:
On Thu, 2012-03-15 at 23:01 -0700, Robert Moskowitz wrote:
Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
Nautilus seems to sniff the files to discover their types, and a plug-in tries to generate a thumbnail image for the file. Both these features are painfully slow with moderately largish directories. You can turn off the inclusion of the filetype in the list details, but not the image generation. Perhaps the plug-in could be manually replaced with something which worked faster, or instantly "did nothing."
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal. A "problem" of long-standing on pretty much all Unix(-ish) file systems (and I'm not saying it won't happen on other systems, I don't really know, but suspect it does.)
If that were so, 'ls' would take as long as Nautilus (or Dolphin or whatever) to list a large directory. I don't have any huge directories to test, but I'm sceptical.
poc
On Fri, Mar 16, 2012 at 09:39:03PM -0430, Patrick O'Callaghan wrote:
On Fri, 2012-03-16 at 19:25 -0400, fred smith wrote:
On Fri, Mar 16, 2012 at 11:01:03PM +1030, Tim wrote:
On Thu, 2012-03-15 at 23:01 -0700, Robert Moskowitz wrote:
Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
Nautilus seems to sniff the files to discover their types, and a plug-in tries to generate a thumbnail image for the file. Both these features are painfully slow with moderately largish directories. You can turn off the inclusion of the filetype in the list details, but not the image generation. Perhaps the plug-in could be manually replaced with something which worked faster, or instantly "did nothing."
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal. A "problem" of long-standing on pretty much all Unix(-ish) file systems (and I'm not saying it won't happen on other systems, I don't really know, but suspect it does.)
If that were so, 'ls' would take as long as Nautilus (or Dolphin or whatever) to list a large directory. I don't have any huge directories to test, but I'm sceptical.
Hmm. you do seem to be correct:
time ls | wc -l 105612
real 0m2.582s user 0m2.429s sys 0m0.163s
I know that on (much) older systems, large directories were inherently slow to traverse. I guess I shouldn't assume that is still the case.
On Fri, 2012-03-16 at 23:49 -0400, fred smith wrote:
If that were so, 'ls' would take as long as Nautilus (or Dolphin or whatever) to list a large directory. I don't have any huge
directories
to test, but I'm sceptical.
Hmm. you do seem to be correct:
time ls | wc -l 105612 real 0m2.582s user 0m2.429s sys 0m0.163s
Note that almost all the time is in "user" (presumably doing the lexical sort) and very little in "sys" which is the actual directory traversal.
poc
Once upon a time, fred smith fredex@fcshome.stoneham.ma.us said:
I know that on (much) older systems, large directories were inherently slow to traverse. I guess I shouldn't assume that is still the case.
Old systems also much less RAM. I made a directory with a similar number of files, and the space on disk for the directory is 8MB. When system RAM is measure in MB, you're using a large portion of it just to sort a list that size (possibly swapping or tossing cached buffers). When you have GB of RAM, what's 8MB?
On 16Mar2012 23:49, fred smith fredex@fcshome.stoneham.ma.us wrote: | On Fri, Mar 16, 2012 at 09:39:03PM -0430, Patrick O'Callaghan wrote: | > If that were so, 'ls' would take as long as Nautilus (or Dolphin or | > whatever) to list a large directory. I don't have any huge directories | > to test, but I'm sceptical. | | Hmm. you do seem to be correct: | | time ls | wc -l | 105612 | | real 0m2.582s | user 0m2.429s | sys 0m0.163s | | I know that on (much) older systems, large directories were inherently | slow to traverse. I guess I shouldn't assume that is still the case.
Gah. Please compare apple with apples.
Run the same test with "ls -l" and compare, then think about the difference.
The consider that many GUI browsers try to give cues that a directory has contents - that needs even more work.
Finally, run strace against Nautilus and other tools and see how much real work they're doing. It can be illuminating.
Cheers,
Tim:
Nautilus seems to sniff the files to discover their types, and a plug-in tries to generate a thumbnail image for the file. Both these features are painfully slow with moderately largish directories.
I've used the emelFM2 file manager as an alternative, it doesn't show thumbnails of files. And it does let you do some wildcarding to show/hide files in the lister gadget.
(And it's much faster...)
fred smith:
I think it's most likely not that nautilus, etc., are slow in large directories, it is that large directories take a long time to search for files, making any action on those directories much slower than normal.
I didn't say searching, just listing. But the *noticeable* problem with Nautilus, that's not noticeable with other file managers, seems to resolve around sniffing file contents to work out file types.
Get Nautilus to show a directory with say a smallish number of large video files (it only takes around 200 files), and it takes an age to do so. Get it to show a similarly populated directory of (much smaller) plain text files, and it's much quicker. Even more quicker if you turn off generating preview images for text files.
The generating of the thumbnails is timeconsuming, also the rendering of showing them in the lister. You can see the slowness of the latter, by itself, by going back and browsing a directory that has had thumbnails already generated and cached, previously.
Even generating thumbnails for JPEGs is timeconsuming. But, at least, you could set a filesize threshold so that it'd ignore the huge image files, even if you couldn't easily set it to not show any preview images for certain file types (old Nautilus gave you options for text, sound, and /other/ file types).
i.e. I'd, long ago, done enough comparative testing to confidently point the finger at the reason that I did.
It's not just Nautilus. You could try double-clicking on a JPEG file to open it with a simple program like eog (eye of gnome), which would take an age chugging through all the files in the directory, to work out which it could work with, before even showing you the file that you wanted to look at (if designed better, it'd show me my chosen file, first, then start directory browsing, afterwards). In a large directory, the wait could easily outlast my patience.
Yet, you can simply "ls -l" the same directory, or browse it with emelfm2 or midnight commander, or any other simple file browser, and you get the results in a snap.
File sniffing is timeconsuming, and CPU-consuming. You've got to identify the file type, which means burrowing in to some degree, on each and every file, and comparing them against your list of file types (which seems to work in a slow manner). Then, for more delays, generate a thumbnail of said file.
It's, also, timeconsuming for it to check through its cache, to see if it's already got a generated preview image, when you try to browse a directory. So, periodically, I'd delete the cached images to speed things up (and to reclaim oodles of wasted drive space).
e.g. rm -f .thumbnails/fail/gnome-thumbnail-factory/* .thumbnails/large/* .thumbnails/normal/*
Nautilus, pretty much, sucks as a file manager. About it's only use, for me, is to give me an (albeit slow) previewing of files in a directory. If I actually want to *manage* files and directories, something else is better.
On 03/16/2012 01:01 AM, Robert Moskowitz wrote:
I have two very large directories.
One an rsync of all the RFCs and that is up to 6456 files. But all the Internet Drafts (including all expired ones) is 72,509 files. Then I have a number of large directories with IEEE 802 documents, but rarely over 3,000 per WG/year (eg ..../802.15/11/ for .15 document numbers assigned in 2011).
So anyway, Nautilus takes forever to list these directories and at times I just want to look for a particular file by string in the name. Is there a fast graphic tool for this? Then when I find the desired file, I typically open it in Firefox.
How about Midnight Commander (yum install mc)
-
Alan Cox -
Andras Simon -
Cameron Simpson -
Chris Adams -
Fred Smith -
Patrick O'Callaghan -
Reindl Harald -
Rick Stevens -
Robert Moskowitz -
SternData -
Suvayu Ali -
Tim