Suppose I have a "green" USB hard drive that frequently spins down the drive in order to save energy, then spins back up when a file system mounted on it is accessed. It is mounted at /var/backups.
I'm logged in as a non-root user with my home directory as my current working directory. The file system containing my home directory is mounted at /home. I'm using a shell prompt via a graphical terminal emulator (xfce4-terminal, in my case). Now, I enter an 'ls' command at a bash prompt. The output doesn't appear until after my USB hard drive spins up. Note that neither /var/backups nor any directory under it is in my shell's PATH, nor is there any symlink to /var/backups in my current working directory. Thus, there should be no need to read /var/backups, yet evidently this exactly what happens.
Question: why does a sleeping hard drive that has no connection with the file system I'm working with have to spin up? The behavior I would expect is that the hard drive would spin down after its file system(s) have not been accessed for a certain amount of time and would spin up if/when a file system on it is accessed. If I'd entered 'ls /var/backups' or 'cd /var/backups' or even a command that accesses all file systems (e.g., 'df') I'd totally expect the hard drive to spin up, but I don't get at all why it should spin up if I'm not accessing any data on it.
I don't like this behavior for two reasons: (1) unnecessary I/O delays while waiting for a hard drive that I'm not explicitly accessing to spin up, plus (2) unnecessary wear and tear on the "green" hard drive.
Other than keeping all of my hard drives spinning all the time--i.e., by using a non-green hard drive in my USB enclosure--is there any way to prevent this behavior? What might be causing it?
Thanks, Dave
On Tue, 2020-02-11 at 19:53 -0600, Dave Ulrick wrote:
I'm logged in as a non-root user with my home directory as my current working directory. The file system containing my home directory is mounted at /home. I'm using a shell prompt via a graphical terminal emulator (xfce4-terminal, in my case). Now, I enter an 'ls' command at a bash prompt. The output doesn't appear until after my USB hard drive spins up. Note that neither /var/backups nor any directory under it is in my shell's PATH, nor is there any symlink to /var/backups in my current working directory. Thus, there should be no need to read /var/backups, yet evidently this exactly what happens.
Just a stab in the dark: Is something poking about /var/run or /var/cache? Perhaps that's enough to look through /var. I wonder if you could try another terminal program, just to see if it's the terminal, itself.
Maybe strace ls, to see what it's up to.
I know with GUI programs, I had to move mountable thing to be inside a sub-directory of my homespace. Otherwise, anything that listed ~/ would wake up the drives to count the number of files in them. So, I feel your pain.
Interesting thought. I can envision how a lookup for /var/xyz could cause everything under /var to be looked up, and I can see how /var/cache or /var/run would be frequently read. I'll try mounting a green USB drive's file system at a third-level directory (e.g., /var/backups/0) or under a less popular directory (e.g., /mnt/backups) and see if that behaves any differently.
Thanks, Dave
On 2/12/20 7:11 AM, Tim via users wrote:
On Tue, 2020-02-11 at 19:53 -0600, Dave Ulrick wrote:
I'm logged in as a non-root user with my home directory as my current working directory. The file system containing my home directory is mounted at /home. I'm using a shell prompt via a graphical terminal emulator (xfce4-terminal, in my case). Now, I enter an 'ls' command at a bash prompt. The output doesn't appear until after my USB hard drive spins up. Note that neither /var/backups nor any directory under it is in my shell's PATH, nor is there any symlink to /var/backups in my current working directory. Thus, there should be no need to read /var/backups, yet evidently this exactly what happens.
Just a stab in the dark: Is something poking about /var/run or /var/cache? Perhaps that's enough to look through /var. I wonder if you could try another terminal program, just to see if it's the terminal, itself.
Maybe strace ls, to see what it's up to.
I know with GUI programs, I had to move mountable thing to be inside a sub-directory of my homespace. Otherwise, anything that listed ~/ would wake up the drives to count the number of files in them. So, I feel your pain.
On 2/12/20 7:53 AM, Dave Ulrick wrote:
Interesting thought. I can envision how a lookup for /var/xyz could cause everything under /var to be looked up, and I can see how /var/cache or /var/run would be frequently read. I'll try mounting a green USB drive's file system at a third-level directory (e.g., /var/backups/0) or under a less popular directory (e.g., /mnt/backups) and see if that behaves any differently.
I ran 'strace' on 'ls' but nothing interesting showed up. Then, I ran 'strace' on 'bash'. I ran 'ls' from 'bash' and then exited. The strace log shows two connect()s to a socket file under /var/run:
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOEN T (No such file or directory) close(3)
/var/run/nscd/socket appears to be related to the 'nscd' DNS cache which I am not running on my PCs.
So, it looks likely that reading /var/run caused the contents of /var to be read. This would have triggered a wakeup of the device hosting /var/backups which would be a cause of its hard drive spinning up.
In addition to one PC that mounted a green USB drive under /var I had several other PCs that mounted a NAS under /var. That NAS is intended to store backup files so its hard drive is configured to spin down after 10 idle minutes.
In view of these findings, I've reconfigured my PCs to mount the backups directory under /mnt instead of /var. So far since doing so I've not noticed any spin-up delays related to a USB hard drive or the backup NAS.
Thanks, Tim!
Dave
The most irritating spin-up for me occurs on a reboot. I have a really slow USB drive that isn't even mounted, just plugged in (in case I want to mount it). The reboot always hangs for several seconds, and I can hear the drive spin up before the reboot proceeds.
It may be the pwd command doing it. It works like this:
if something runs pwd when its cwd is under say /var/log then pwd goes through all files in /var/log until it finds .. then it goes up a directory and repeats, until it gets to /.
Assuming that is the case your solution would be expected to work, if you put it under /mnt/backups then any other pwd anywhere under mnt may also cause the spinup. On nfs mounts an nfs mount that is hanging of say /mnt/host1 can hang everything else in /mnt even coming from other responding hosts. the trick there is to /mnt/host1/host1mntpoint and put each separate host in a separate top level directory to isolate them from each other. You may not need to do that so long as you don't have other things in /mnt being used that may cause a pwd.
if you run ls -l /proc/*/cwd | more it will show you everything running's cwd. I see /var/spool/at (atd process) with that as a home dir, so atd doing a pwd would cause a spinup.
I don't actively use atd for anything and strace does not show atd doing anything on my machine. If you use atd then it may be what is doing it. nfs's statd also its cwd under var and is used on nfs servers.
On Wed, Feb 12, 2020 at 1:11 PM Dave Ulrick d-ulrick@comcast.net wrote:
On 2/12/20 7:53 AM, Dave Ulrick wrote:
Interesting thought. I can envision how a lookup for /var/xyz could cause everything under /var to be looked up, and I can see how /var/cache or /var/run would be frequently read. I'll try mounting a green USB drive's file system at a third-level directory (e.g., /var/backups/0) or under a less popular directory (e.g., /mnt/backups) and see if that behaves any differently.
I ran 'strace' on 'ls' but nothing interesting showed up. Then, I ran 'strace' on 'bash'. I ran 'ls' from 'bash' and then exited. The strace log shows two connect()s to a socket file under /var/run:
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOEN T (No such file or directory) close(3)
/var/run/nscd/socket appears to be related to the 'nscd' DNS cache which I am not running on my PCs.
So, it looks likely that reading /var/run caused the contents of /var to be read. This would have triggered a wakeup of the device hosting /var/backups which would be a cause of its hard drive spinning up.
In addition to one PC that mounted a green USB drive under /var I had several other PCs that mounted a NAS under /var. That NAS is intended to store backup files so its hard drive is configured to spin down after 10 idle minutes.
In view of these findings, I've reconfigured my PCs to mount the backups directory under /mnt instead of /var. So far since doing so I've not noticed any spin-up delays related to a USB hard drive or the backup NAS.
Thanks, Tim!
Dave
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Wed, 2020-02-12 at 13:10 -0600, Dave Ulrick wrote:
In addition to one PC that mounted a green USB drive under /var I had several other PCs that mounted a NAS under /var. That NAS is intended to store backup files so its hard drive is configured to spin down after 10 idle minutes.
In view of these findings, I've reconfigured my PCs to mount the backups directory under /mnt instead of /var. So far since doing so I've not noticed any spin-up delays related to a USB hard drive or the backup NAS.
Thanks, Tim!
Glad to help, even if I'm not quite sure how I steered things in the right way. ;-) It was one of those "in the back of my mind" ideas.
Gnome (or Gnome-based) things, will put temporary auto-mounting things (flash drives, etc) inside of /var/run/, then give you an apparently separate mount point. You'll get a desktop icon for it, and no immediate indication that it's accessed through /var/run.
I'm not sure if the same applies to KDE.
On 2/12/20 7:09 PM, Roger Heflin wrote:
It may be the pwd command doing it. It works like this:
if something runs pwd when its cwd is under say /var/log then pwd goes through all files in /var/log until it finds .. then it goes up a directory and repeats, until it gets to /.
That makes sense. Just occurred to me: a typical shell prompt displays the cwd, so 'pwd' or something like it must get run just before a shell prompt is displayed. This likely explains why I'd sometimes experience the spin-up delay after running a command: it wasn't the command itself necessarily that triggered the spin-up but the pwd afterwards.
Assuming that is the case your solution would be expected to work,
It seemed to work earlier today when I fired up the green USB hard drive in that I experienced no spin-up delays, but...
if you put it under /mnt/backups then any other pwd anywhere under mnt may also cause the spinup. On nfs mounts an nfs mount that is hanging of say /mnt/host1 can hang everything else in /mnt even coming from other responding hosts. the trick there is to /mnt/host1/host1mntpoint and put each separate host in a separate top level directory to isolate them from each other. You may not need to do that so long as you don't have other things in /mnt being used that may cause a pwd.
That's a great idea. I've gone ahead and implemented it on my PCs.
if you run ls -l /proc/*/cwd | more it will show you everything running's cwd. I see /var/spool/at (atd process) with that as a home dir, so atd doing a pwd would cause a spinup.
I don't actively use atd for anything and strace does not show atd doing anything on my machine. If you use atd then it may be what is doing it. nfs's statd also its cwd under var and is used on nfs servers.
There certainly are a lot of processes on my PC running cwd. Interesting...
Thanks! Dave
On 12Feb2020 19:09, Roger Heflin rogerheflin@gmail.com wrote:
It may be the pwd command doing it. It works like this:
if something runs pwd when its cwd is under say /var/log then pwd goes through all files in /var/log until it finds .. then it goes up a directory and repeats, until it gets to /.
getcwd() is a system call on Linux since 2.1.92. It doesn't need to perform the expensive ".. then stat everything for a match" loop.
[... snip arranging mounts to be in more out-of-the-way places ...]
On Wed, Feb 12, 2020 at 1:11 PM Dave Ulrick d-ulrick@comcast.net wrote:
I ran 'strace' on 'ls' but nothing interesting showed up. Then, I ran 'strace' on 'bash'. I ran 'ls' from 'bash' and then exited. The strace log shows two connect()s to a socket file under /var/run:
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOEN T (No such file or directory) close(3)
/var/run/nscd/socket appears to be related to the 'nscd' DNS cache which I am not running on my PCs.
Aye.
So, it looks likely that reading /var/run caused the contents of /var to be read. This would have triggered a wakeup of the device hosting /var/backups which would be a cause of its hard drive spinning up.
This would surprise me; accessing a direct name doesn't require stating everything adjacent.
You can test this:
- wait for the drive to spin down - type ">>var/run/nscd/socket" and preess <enter> (do NOT filecomplete that path, your shell will do unwanted filesystem access) - see if/when the drive spins up
Cheers, Cameron Simpson cs@cskk.id.au
On 2/12/20 7:00 PM, Tim via users wrote:
Gnome (or Gnome-based) things, will put temporary auto-mounting things (flash drives, etc) inside of /var/run/, then give you an apparently separate mount point. You'll get a desktop icon for it, and no immediate indication that it's accessed through /var/run.
Actually, the mount point is /run/media/<userid>/<label>. /var/run is a symlink to /run.
On Thursday, 13 February 2020 01:09:20 GMT Roger Heflin wrote:
It may be the pwd command doing it. It works like this:
if something runs pwd when its cwd is under say /var/log then pwd goes through all files in /var/log until it finds .. then it goes up a directory and repeats, until it gets to /.
Assuming that is the case your solution would be expected to work, if you put it under /mnt/backups then any other pwd anywhere under mnt may also cause the spinup. On nfs mounts an nfs mount that is hanging of say /mnt/host1 can hang everything else in /mnt even coming from other responding hosts. the trick there is to /mnt/host1/host1mntpoint and put each separate host in a separate top level directory to isolate them from each other. You may not need to do that so long as you don't have other things in /mnt being used that may cause a pwd.
if you run ls -l /proc/*/cwd | more it will show you everything running's cwd. I see /var/spool/at (atd process) with that as a home dir, so atd doing a pwd would cause a spinup.
I don't actively use atd for anything and strace does not show atd doing anything on my machine. If you use atd then it may be what is doing it. nfs's statd also its cwd under var and is used on nfs servers.
On Wed, Feb 12, 2020 at 1:11 PM Dave Ulrick d-ulrick@comcast.net wrote:
On 2/12/20 7:53 AM, Dave Ulrick wrote:
Interesting thought. I can envision how a lookup for /var/xyz could cause everything under /var to be looked up, and I can see how /var/cache or /var/run would be frequently read. I'll try mounting a green USB drive's file system at a third-level directory (e.g., /var/backups/0) or under a less popular directory (e.g., /mnt/backups) and see if that behaves any differently.
I ran 'strace' on 'ls' but nothing interesting showed up. Then, I ran 'strace' on 'bash'. I ran 'ls' from 'bash' and then exited. The strace log shows two connect()s to a socket file under /var/run:
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOEN T (No such file or directory) close(3)
/var/run/nscd/socket appears to be related to the 'nscd' DNS cache which I am not running on my PCs.
So, it looks likely that reading /var/run caused the contents of /var to be read. This would have triggered a wakeup of the device hosting /var/backups which would be a cause of its hard drive spinning up.
In addition to one PC that mounted a green USB drive under /var I had several other PCs that mounted a NAS under /var. That NAS is intended to store backup files so its hard drive is configured to spin down after 10 idle minutes.
In view of these findings, I've reconfigured my PCs to mount the backups directory under /mnt instead of /var. So far since doing so I've not noticed any spin-up delays related to a USB hard drive or the backup NAS.
Thanks, Tim!
Dave
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List
Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.o rg
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
I am a little confused by getcwd(), on linux it is supposed to be a system call .. I expect to find a manual page for it in section 2 (man 2 getcwd) instead it gives me the library function manual page from section 3 (man 3 getcwd) I would fully expect a libc version of getcwd to make many system calls to traverse the current path but here getcwd() is a system call with all work done internally in the kernel, so I wouldnt expect anything to probe the path elements. When I knock up a simple program to do a getcwd() and strace it I find that getcwd (on FC31 anyway) IS a system call. Why does "man getcwd" give me a page from section 3?
On Thu, 13 Feb 2020 at 06:40, Andy Paterson via users < users@lists.fedoraproject.org> wrote:
[...]
I am a little confused by getcwd(), on linux it is supposed to be a system call .. I expect to find a manual page for it in section 2 (man 2 getcwd)
instead it gives me the library function manual page from section 3 (man 3 getcwd)
$ man -k getcwd getcwd (2) - get current working directory getcwd (3) - get current working directory getcwd (3p) - get the pathname of the current working directory $ gzip -dc /usr/share/man/man2/getcwd.2.gz .so man3/getcwd.3
I would fully expect a libc version of getcwd to make many system calls to traverse the current path but here getcwd() is a system call with all work done internally in the kernel, so I wouldnt expect anything to probe the path elements.
When I knock up a simple program to do a getcwd() and strace it I find that getcwd (on FC31 anyway) IS a system call.
Why does "man getcwd" give me a page from section 3?
The man page explains the differences, so it makes sense that "man 2" and "man 3" give the same text:
C library/kernel differences On Linux, the kernel provides a getcwd() system call, which the func‐ tions described in this page will use if possible. The system call takes the same arguments as the library function of the same name, but is limited to returning at most PATH_MAX bytes.
The getcwd() system call saves overhead but should make the same storage accesses as the library version.
Except the system call version doesn't need you to include any header files! Which is NOT clear from the man page!
On 13 Feb 2020, at 13:53, George N. White III gnwiii@gmail.com wrote:
On Thu, 13 Feb 2020 at 06:40, Andy Paterson via users users@lists.fedoraproject.org wrote:
[...] I am a little confused by getcwd(), on linux it is supposed to be a system call .. I expect to find a manual page for it in section 2 (man 2 getcwd) instead it gives me the library function manual page from section 3 (man 3 getcwd)
$ man -k getcwd getcwd (2) - get current working directory getcwd (3) - get current working directory getcwd (3p) - get the pathname of the current working directory $ gzip -dc /usr/share/man/man2/getcwd.2.gz .so man3/getcwd.3
I would fully expect a libc version of getcwd to make many system calls to traverse the current path but here getcwd() is a system call with all work done internally in the kernel, so I wouldnt expect anything to probe the path elements. When I knock up a simple program to do a getcwd() and strace it I find that getcwd (on FC31 anyway) IS a system call. Why does "man getcwd" give me a page from section 3?
The man page explains the differences, so it makes sense that "man 2" and "man 3" give the same text:
C library/kernel differences On Linux, the kernel provides a getcwd() system call, which the func‐ tions described in this page will use if possible. The system call takes the same arguments as the library function of the same name, but is limited to returning at most PATH_MAX bytes.
The getcwd() system call saves overhead but should make the same storage accesses as the library version.
-- George N. White III
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Thu, 13 Feb 2020 at 11:35, Andy Paterson via users < users@lists.fedoraproject.org> wrote:
Except the system call version doesn't need you to include any header files! Which is NOT clear from the man page!
The man page says "On Linux, the kernel provides a getcwd() system call, which the functions described in this page will use if possible." so it seems applications are meant to use the header files and let the library do the system call "if possible".
On 13 Feb 2020, at 13:53, George N. White III gnwiii@gmail.com wrote:
On Thu, 13 Feb 2020 at 06:40, Andy Paterson via users < users@lists.fedoraproject.org> wrote:
[...]
I am a little confused by getcwd(), on linux it is supposed to be a system call .. I expect to find a manual page for it in section 2 (man 2 getcwd)
instead it gives me the library function manual page from section 3 (man 3 getcwd)
$ man -k getcwd getcwd (2) - get current working directory getcwd (3) - get current working directory getcwd (3p) - get the pathname of the current working directory $ gzip -dc /usr/share/man/man2/getcwd.2.gz .so man3/getcwd.3
I would fully expect a libc version of getcwd to make many system calls to traverse the current path but here getcwd() is a system call with all work done internally in the kernel, so I wouldnt expect anything to probe the path elements.
When I knock up a simple program to do a getcwd() and strace it I find that getcwd (on FC31 anyway) IS a system call.
Why does "man getcwd" give me a page from section 3?
The man page explains the differences, so it makes sense that "man 2" and "man 3" give the same text:
C library/kernel differences On Linux, the kernel provides a getcwd() system call, which the func‐ tions described in this page will use if possible. The system call takes the same arguments as the library function of the same name, but is limited to returning at most PATH_MAX bytes.
The getcwd() system call saves overhead but should make the same storage accesses as the library version.
-- George N. White III
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
-
Andy Paterson -
Cameron Simpson -
Dave Ulrick -
George N. White III -
Roger Heflin -
Samuel Sieb -
Tim -
Tom Horsley