Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice; but not the console. Firefox and Thunderbird just stop; Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Has anyone seen anything like this? Any ideas on how to prevent it? Any experience with other virtual environments?
System info: Operating System: Fedora Linux 37 KDE Plasma Version: 5.27.4 KDE Frameworks Version: 5.104.0 Qt Version: 5.15.8 Kernel Version: 6.2.9-200.fc37.x86_64 (64-bit) Graphics Platform: X11 Processors: 8 × Intel® Core™ i7-4790K CPU @ 4.00GHz Memory: 15.5 GiB of RAM Graphics Processor: Mesa Intel® HD Graphics 4600 Manufacturer: ASUS Product Name: All Series
On Mon, Apr 10, 2023 at 8:53 PM Jonathan Ryshpan jonrysh@pacbell.net wrote:
Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice; but not the console. Firefox and Thunderbird just stop; Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Guessing - OOMD is killing things to free up RAM?
On Mon, 2023-04-10 at 22:21 -0400, Go Canes wrote:
On Mon, Apr 10, 2023 at 8:53 PM Jonathan Ryshpan jonrysh@pacbell.net wrote:
Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice; but not the console. Firefox and Thunderbird just stop; Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Guessing - OOMD is killing things to free up RAM?
Certainly correct. In future I'll have to take corrective action -- maybe buy more RAM. From the system log:
Apr 10 13:35:44 amito systemd-oomd[948]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app- libreoffice\x2dcalc-d5d271b128784076915db377a70340bc.scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 59.60% > 50.00% for > 20s with reclaim activity Apr 10 13:35:44 amito systemd[1440]: app-libreoffice\x2dcalc- d5d271b128784076915db377a70340bc.scope: systemd-oomd killed 5 process(es) in this unit. Apr 10 13:35:44 amito systemd[1440]: app-libreoffice\x2dcalc- d5d271b128784076915db377a70340bc.scope: Consumed 4min 26.480s CPU time. Apr 10 13:35:59 amito systemd-oomd[948]: Considered 113 cgroups for killing, top candidates were: Apr 10 13:35:59 amito systemd-oomd[948]: Path: /user.slice/user-1000.slice/user@1000.service/app.slice/app- mozilla\x2dthunderbird-c2bba51b2b8346e1bbe7647080a15d7b. scope Apr 10 13:35:59 amito systemd-oomd[948]: Memory Pressure Limit: 0.00% Apr 10 13:35:59 amito systemd-oomd[948]: Pressure: Avg10: 0.45 Avg60: 2.44 Avg300: 1.02 Total: 1min 35s Apr 10 13:35:59 amito systemd-oomd[948]: Current Memory Usage: 163.0M Apr 10 13:35:59 amito systemd-oomd[948]: Memory Min: 0B Apr 10 13:35:59 amito systemd-oomd[948]: Memory Low: 0B Apr 10 13:35:59 amito systemd-oomd[948]: Pgscan: 577674 Apr 10 13:35:59 amito systemd-oomd[948]: Last Pgscan: 577354
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan jonrysh@pacbell.net wrote:
On Mon, 2023-04-10 at 22:21 -0400, Go Canes wrote:
On Mon, Apr 10, 2023 at 8:53 PM Jonathan Ryshpan jonrysh@pacbell.net wrote:
Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice; but not the console. Firefox and Thunderbird just stop; Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Guessing - OOMD is killing things to free up RAM?
Certainly correct. In future I'll have to take corrective action -- maybe buy more RAM. From the system log:
I masked systemd-oomd.service because I have real swap, and when something starts using too much memory, it starts using that swap and I notice the slowdown in the system. Then I can do something about it.
Might or might not work for your use case.
This is unlikely to be the problem, but it is a fun story:
Many years ago there was a bug in the KVM kernel code that failed to correctly context switch all the registers in virtual machines. In this case, the somewhat obscure debug registers. So if I was running debugger tests inside a virtual machine, and the test was using a address trap (where the debug registers can be set to generate an interrupt when you reference a specific memory location), then sometimes programs running on the host machine would get the address trap if they referenced that memory address :-). After much head scratching I managed to produce a test case that could trigger the bug nearly instantly and reported the bug, which was actually fixed.
I suppose the virtual box kernel code might have introduced such a bug, but that's probably unlikely.
On Tue, 2023-04-11 at 08:30 -0700, stan via users wrote:
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan jonrysh@pacbell.net wrote:
On Mon, 2023-04-10 at 22:21 -0400, Go Canes wrote:
On Mon, Apr 10, 2023 at 8:53 PM Jonathan Ryshpan jonrysh@pacbell.net wrote:
Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice; but not the console. Firefox and Thunderbird just stop; Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Guessing - OOMD is killing things to free up RAM?
Certainly correct. In future I'll have to take corrective action -- maybe buy more RAM. From the system log:
I masked systemd-oomd.service because I have real swap, and when something starts using too much memory, it starts using that swap and I notice the slowdown in the system. Then I can do something about it.
If I understand the situation correctly (which I may not} in my system this happens: All real and virtual memory is allocated (The system has 16 Gb RAM + 32 Gb Swap.), but more memory is needed, at which point oomd starts killing processes. The system often becomes quite slow, with a lot of disk activity, before processes start to die.
Cures might be more RAM, more swap, or cutting the system load. More swap is probably a bad idea, since the system already appears to be running slow on account of a lot of swapping.
On Tue, 2023-04-11 at 09:49 -0700, Jonathan Ryshpan wrote:
On Tue, 2023-04-11 at 08:30 -0700, stan via users wrote:
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan jonrysh@pacbell.net wrote:
On Mon, 2023-04-10 at 22:21 -0400, Go Canes wrote:
On Mon, Apr 10, 2023 at 8:53 PM Jonathan Ryshpan jonrysh@pacbell.net wrote:
Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice; but not the console. Firefox and Thunderbird just stop; Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Guessing - OOMD is killing things to free up RAM?
Certainly correct. In future I'll have to take corrective action
maybe buy more RAM. From the system log:
I masked systemd-oomd.service because I have real swap, and when something starts using too much memory, it starts using that swap and I notice the slowdown in the system. Then I can do something about it.
If I understand the situation correctly (which I may not} in my system this happens: All real and virtual memory is allocated (The system has 16 Gb RAM + 32 Gb Swap.), but more memory is needed, at which point oomd starts killing processes. The system often becomes quite slow, with a lot of disk activity, before processes start to die.
Cures might be more RAM, more swap, or cutting the system load. More swap is probably a bad idea, since the system already appears to be running slow on account of a lot of swapping.
You might try monitoring usage via oomctl to see what is actually happening.
poc
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan <jonrysh(a)pacbell.net> wrote:
I masked systemd-oomd.service because I have real swap,
Masking it is unreliable because it might start running again if system-oomd-defaults is updated or reinstalled. (To check this, try reinstalling it.) It's better to just remove the system-oomd-defaults package. That way, systemd-oomd.service continues to run, but it's not monitoring anything (as verified with oomctl).
I just tried masking/reinstalling just now and it didn't restart. I experienced it before with a systemd update so it might require that. But masking is definitely not 100% effective.
On 4/14/23 22:06, Andre Robatino wrote:
I just tried masking/reinstalling just now and it didn't restart. I experienced it before with a systemd update so it might require that. But masking is definitely not 100% effective.
Masking should be 100% effective. It changes files in /etc which should not be touched by any install or update.
On F37, masking was the first thing I tried. Then after a systemd update, I noticed it was running again, even though it was still masked (verified by "systemctl status systemd-oomd" which showed it both running and masked at the same time).
On Sat, Apr 15, 2023 at 1:05 PM Andre Robatino robatino@fedoraproject.org wrote:
On F37, masking was the first thing I tried. Then after a systemd update, I noticed it was running again, even though it was still masked (verified by "systemctl status systemd-oomd" which showed it both running and masked at the same time).
Fedora's bug reporter is at https://bugzilla.redhat.com/ .
Jeff
I don't know enough about systemd to think that's even a bug - I had assumed it was normal behavior. In any case the bug that was forcing me to disable the OOM killer ( https://bugzilla.redhat.com/show_bug.cgi?id=2177722 ) has been fixed so I'm not disabling it anymore.
On Sat, Apr 15, 2023 at 3:48 PM Andre Robatino robatino@fedoraproject.org wrote:
I don't know enough about systemd to think that's even a bug - I had assumed it was normal behavior. In any case the bug that was forcing me to disable the OOM killer ( https://bugzilla.redhat.com/show_bug.cgi?id=2177722 ) has been fixed so I'm not disabling it anymore.
As far as I know, a masked service cannot be started. (A disabled service can be manually started).
See https://www.techrepublic.com/article/masked-services-linux-how-manage/ .
Jeff
On Sat, 15 Apr 2023 04:59:48 -0000 "Andre Robatino" robatino@fedoraproject.org wrote:
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan <jonrysh(a)pacbell.net> wrote:
I masked systemd-oomd.service because I have real swap,
Masking it is unreliable because it might start running again if system-oomd-defaults is updated or reinstalled. (To check this, try reinstalling it.) It's better to just remove the system-oomd-defaults package. That way, systemd-oomd.service continues to run, but it's not monitoring anything (as verified with oomctl).
So, I have it masked, and this shows from systemctl, systemd-oomd.service masked inactive dead systemd-oomd.service However, I don't have the systemd-oomd-defaults package installed so I can't say whether that would cause it to start while being masked or not. Perhaps I will give that a try at some point. If it is being started despite being masked, that is definitely a bug. Masking is supposed to be inviolable. Since systemd-oomd is part of the main systemd package, that would be the package to file the bug against. It is possible that it is not checking the service status if the systemd-oomd-defaults package is installed.
On 16 Apr 2023, at 15:39, stan via users users@lists.fedoraproject.org wrote:
On Sat, 15 Apr 2023 04:59:48 -0000 "Andre Robatino" robatino@fedoraproject.org wrote:
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan <jonrysh(a)pacbell.net> wrote:
I masked systemd-oomd.service because I have real swap,
Masking it is unreliable because it might start running again if system-oomd-defaults is updated or reinstalled. (To check this, try reinstalling it.) It's better to just remove the system-oomd-defaults package. That way, systemd-oomd.service continues to run, but it's not monitoring anything (as verified with oomctl).
So, I have it masked, and this shows from systemctl, systemd-oomd.service masked inactive dead systemd-oomd.service However, I don't have the systemd-oomd-defaults package installed so I can't say whether that would cause it to start while being masked or not. Perhaps I will give that a try at some point. If it is being started despite being masked, that is definitely a bug. Masking is supposed to be inviolable. Since systemd-oomd is part of the main systemd package, that would be the package to file the bug against. It is possible that it is not checking the service status if the systemd-oomd-defaults package is installed.
Its not a check as such. Masking means the service is defined as /dev/null, hard to start when there is nothing defined.
Barry
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Sun, 16 Apr 2023 16:45:09 +0100 Barry barry@barrys-emacs.org wrote:
Its not a check as such. Masking means the service is defined as /dev/null, hard to start when there is nothing defined.
True that. :-)
On Sun, 16 Apr 2023 07:38:56 -0700 stan upaitag@zoho.com wrote:
On Sat, 15 Apr 2023 04:59:48 -0000 "Andre Robatino" robatino@fedoraproject.org wrote:
On Tue, 11 Apr 2023 00:59:59 -0700 Jonathan Ryshpan <jonrysh(a)pacbell.net> wrote:
I masked systemd-oomd.service because I have real swap,
Masking it is unreliable because it might start running again if system-oomd-defaults is updated or reinstalled. (To check this, try reinstalling it.) It's better to just remove the system-oomd-defaults package. That way, systemd-oomd.service continues to run, but it's not monitoring anything (as verified with oomctl).
So, I have it masked, and this shows from systemctl, systemd-oomd.service masked inactive dead systemd-oomd.service However, I don't have the systemd-oomd-defaults package installed so I can't say whether that would cause it to start while being masked or not. Perhaps I will give that a try at some point. If it is being started despite being masked, that is definitely a bug. Masking is supposed to be inviolable. Since systemd-oomd is part of the main systemd package, that would be the package to file the bug against. It is possible that it is not checking the service status if the systemd-oomd-defaults package is installed.
I installed systemd-oomd-defaults, rebooted, and the service remained masked. I checked with ps and top, and it is not running. On F37, fully up to date.
Name : systemd Version : 251.14 Release : 2.fc37 Architecture: x86_64
On Mon, 2023-04-10 at 17:52 -0700, Jonathan Ryshpan wrote:
Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Some thirty-plus years ago I learnt to regularly press "CTRL S" to save the current status of whatever I was working on, because crashes, power outages, cats, and nearby twits with a bad sense of humour can so easily wreck a lot of hard work in a fraction of a second.
These days, that's any time my train of thought has changed, or after finishing a long paragraph. Yep, my trust in technology is that poor, and that includes any faith in auto-recovery.
On Tue, 2023-04-11 at 15:46 +0930, Tim via users wrote:
On Mon, 2023-04-10 at 17:52 -0700, Jonathan Ryshpan wrote:
Libreoffice attempts to restore files being worked on (but it failed to restore about 2 hours' worth of work).
Some thirty-plus years ago I learnt to regularly press "CTRL S" to save the current status of whatever I was working on, because crashes, power outages, cats, and nearby twits with a bad sense of humour can so easily wreck a lot of hard work in a fraction of a second.
These days, that's any time my train of thought has changed, or after finishing a long paragraph. Yep, my trust in technology is that poor, and that includes any faith in auto-recovery.
Always good advice. But I am a luck fellow. Linux has been good to me; over 30 years of work I have lost an appreciable amount of work only this once.
On Mon, Apr 10, 2023, at 5:52 PM, Jonathan Ryshpan wrote:
Starting Oracle's VirtualBox has started to crash other apps, in particular Thunderbird, Firefox, and Libreoffice;
[snip]
I assume it is not just starting the VirtualBox Manager that is doing this.
It might help to know how much memory is assigned to the VB clients that are running.
-
Andre Robatino -
Barry -
Doug Herr -
Go Canes -
Jeffrey Walton -
Jonathan Ryshpan -
Patrick O'Callaghan -
Samuel Sieb -
stan -
Tim -
Tom Horsley