6:33 a.m.
On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org> wrote:
This seems like a distraction from the real goal here, which is to
ensure Fedora remains responsive under heavy memory pressure,
I think this is an overwhelmingly important point, and as somebody
regularly working with ARM machines with tiny amounts of RAM, it is of
considerable interest to me.
I typically use CentOS because stability is important to me, but most
worthwhile things filter to there, so I hope what I'm about to say is not
_too_ deprecated.
1) Compile options
From what I can tell from rpm macro options, default on C7 seems to be -O2.
-Os seems to help in most cases.
Adding -ffunction-sections -fdata-sections to defaults can help
considerably in producing smaller binaries, and is not the default.
Linking with -Wl,--gc-sections helps a lot and is not the default
Extensive stripping seems to already be the default (--strip-unneeded,
removal of .comment and .note sections)
2) Runtime condiguration
Default stack size is 8192 (ulimit -s). This unnecessarily eats a
considerably amount of memory. I have yet to see anything that actually
experiences problems with 1M.
3) zram
This was mentioned earlier in the thread, and on most of my systems, memory
constrained or otherwise, unless I have an overwhelming reason not to, I
run with zram swap equal in size to RAM with lz4 compression and
vm.swappiness=100. I typically see compression ratios between 2:1 and 3:1
in zram, so on a system with, say, 10GB of RAM, it would provide 10GB of
very fast swap at a cost of 3-5GB of RAM. This seems like a favourable
trade off, especially on systems with extremely constrained RAM (e.g. ARM
devices with 512MB of RAM).
I'm sure there is more that can be done, but this seems like a good start
as far as the cost / benefit is concerned.
8:05 a.m.
New subject: Better interactivity in low-memory situations
Hi,
On 18-08-19 13:33, Gordan Bobic wrote:
On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org
<http://gnome.org/>> wrote:
> This seems like a distraction from the real goal here, which is to
> ensure Fedora remains responsive under heavy memory pressure,
I think this is an overwhelmingly important point, and as somebody regularly working with
ARM machines with tiny amounts of RAM, it is of considerable interest to me.
I typically use CentOS because stability is important to me, but most worthwhile things
filter to there, so I hope what I'm about to say is not _too_ deprecated.
1) Compile options
From what I can tell from rpm macro options, default on C7 seems to be -O2. -Os seems to
help in most cases.
I don't think it is likely that Fedora will switch to -Os
Adding -ffunction-sections -fdata-sections to defaults can help
considerably in producing smaller binaries, and is not the default.
Linking with -Wl,--gc-sections helps a lot and is not the default
These OTOH are interesting I know that e.g. uboot combines these and it helps a lot to get
smaller binaries,
and this should help with RAM size too, since if a page of a binary contains mostly unused
things and 1 symbol
which is actually used it will still get paged in.
Can you perhaps start a new devel list thread about just this ? Maybe with some binary
size numbers for
some apps / libs build with and without these options?
Extensive stripping seems to already be the default
(--strip-unneeded, removal of .comment and .note sections)
2) Runtime condiguration
Default stack size is 8192 (ulimit -s). This unnecessarily eats a considerably amount of
memory. I have yet to see anything that actually experiences problems with 1M.
Actually ulimit -s is the *maximum* stack size, I'm pretty sure the stack will start
much smaller and
grow dynamically. So changing this is not saving any RAM and it will makes apps which do
have high
stack usage crash when they hit the new lower limit.
Regards,
Hans
8:25 a.m.
New subject: Better interactivity in low-memory situations
On Sun, Aug 18, 2019 at 2:06 PM Hans de Goede <hdegoede(a)redhat.com> wrote:
Hi,
On 18-08-19 13:33, Gordan Bobic wrote:
> On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org <
http://gnome.org/>> wrote:
> > This seems like a distraction from the real goal here, which is to
> > ensure Fedora remains responsive under heavy memory pressure,
>
> I think this is an overwhelmingly important point, and as somebody
regularly working with ARM machines with tiny amounts of RAM, it is of
considerable interest to me.
> I typically use CentOS because stability is important to me, but most
worthwhile things filter to there, so I hope what I'm about to say is not
_too_ deprecated.
>
> 1) Compile options
> From what I can tell from rpm macro options, default on C7 seems to be
-O2. -Os seems to help in most cases.
I don't think it is likely that Fedora will switch to -Os
It is not my place to argue about whether it will. The thread was asking
for things that might contribute toward alleviating the memory pressure
problem. This can make a fairly dramatic difference and it would contribute
toward alleviating the problem because smaller binaries mean less to mmap().
> Adding -ffunction-sections -fdata-sections to defaults can help
considerably in producing smaller binaries, and is not the default.
> Linking with -Wl,--gc-sections helps a lot and is not the default
These OTOH are interesting I know that e.g. uboot combines these and it
helps a lot to get smaller binaries,
and this should help with RAM size too, since if a page of a binary
contains mostly unused things and 1 symbol
which is actually used it will still get paged in.
Can you perhaps start a new devel list thread about just this ? Maybe with
some binary size numbers for
some apps / libs build with and without these options?
It's pretty well documented in various articles, e.g.:
https://wiki.wxwidgets.org/Reducing_Executable_Size
It also covers how much difference -Os can make.
> Extensive stripping seems to already be the default
(--strip-unneeded,
removal of .comment and .note sections)
>
> 2) Runtime condiguration
> Default stack size is 8192 (ulimit -s). This unnecessarily eats a
considerably amount of memory. I have yet to see anything that actually
experiences problems with 1M.
Actually ulimit -s is the *maximum* stack size, I'm pretty sure the stack
will start much smaller and
grow dynamically. So changing this is not saving any RAM and it will makes
apps which do have high
stack usage crash when they hit the new lower limit.
Either way, it makes a noticeable difference to memory consumption on a
very memory constrained system without any other obvious adverse effects.
8:33 a.m.
New subject: Better interactivity in low-memory situations
Hi,
On 18-08-19 15:25, Gordan Bobic wrote:
On Sun, Aug 18, 2019 at 2:06 PM Hans de Goede <hdegoede(a)redhat.com
<mailto:hdegoede@redhat.com>> wrote:
Hi,
On 18-08-19 13:33, Gordan Bobic wrote:
> On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org
<http://gnome.org> <http://gnome.org/>> wrote:
> > This seems like a distraction from the real goal here, which is to
> > ensure Fedora remains responsive under heavy memory pressure,
>
> I think this is an overwhelmingly important point, and as somebody regularly
working with ARM machines with tiny amounts of RAM, it is of considerable interest to me.
> I typically use CentOS because stability is important to me, but most
worthwhile things filter to there, so I hope what I'm about to say is not _too_
deprecated.
>
> 1) Compile options
> From what I can tell from rpm macro options, default on C7 seems to be -O2.
-Os seems to help in most cases.
I don't think it is likely that Fedora will switch to -Os
It is not my place to argue about whether it will. The thread was asking for things that
might contribute toward alleviating the memory pressure problem. This can make a fairly
dramatic difference and it would contribute toward alleviating the problem because smaller
binaries mean less to mmap().
> Adding -ffunction-sections -fdata-sections to defaults can help considerably in
producing smaller binaries, and is not the default.
> Linking with -Wl,--gc-sections helps a lot and is not the default
These OTOH are interesting I know that e.g. uboot combines these and it helps a lot
to get smaller binaries,
and this should help with RAM size too, since if a page of a binary contains mostly
unused things and 1 symbol
which is actually used it will still get paged in.
Can you perhaps start a new devel list thread about just this ? Maybe with some
binary size numbers for
some apps / libs build with and without these options?
It's pretty well documented in various articles, e.g.:
https://wiki.wxwidgets.org/Reducing_Executable_Size
It also covers how much difference -Os can make.
Interesting, thank you for that link.
> Extensive stripping seems to already be the default
(--strip-unneeded, removal of .comment and .note sections)
>
> 2) Runtime condiguration
> Default stack size is 8192 (ulimit -s). This unnecessarily eats a considerably
amount of memory. I have yet to see anything that actually experiences problems with 1M.
Actually ulimit -s is the *maximum* stack size, I'm pretty sure the stack will
start much smaller and
grow dynamically. So changing this is not saving any RAM and it will makes apps which
do have high
stack usage crash when they hit the new lower limit.
Either way, it makes a noticeable difference to memory consumption on a very memory
constrained system without any other obvious adverse effects.
Interesting unless I'm reading the manpage wrong, "ulimit -s" sets the
maximum stack-size.
Maybe that also influences the initial sizing of the stack ?
Can someone who knows more about this shed some light on this? Is there a way to go with
a smaller initial stack-size without changing the maximum size?
Regards,
Hans
8:38 a.m.
New subject: Better interactivity in low-memory situations
On Sun, Aug 18, 2019 at 2:33 PM Hans de Goede <hdegoede(a)redhat.com> wrote:
> > Adding -ffunction-sections -fdata-sections to defaults can help
considerably in producing smaller binaries, and is not the default.
> > Linking with -Wl,--gc-sections helps a lot and is not the default
>
> These OTOH are interesting I know that e.g. uboot combines these and
it helps a lot to get smaller binaries,
> and this should help with RAM size too, since if a page of a binary
contains mostly unused things and 1 symbol
> which is actually used it will still get paged in.
>
> Can you perhaps start a new devel list thread about just this ?
Maybe with some binary size numbers for
> some apps / libs build with and without these options?
>
>
> It's pretty well documented in various articles, e.g.:
> https://wiki.wxwidgets.org/Reducing_Executable_Size
> It also covers how much difference -Os can make.
Interesting, thank you for that link.
> > Extensive stripping seems to already be the default
(--strip-unneeded, removal of .comment and .note sections)
> >
> > 2) Runtime condiguration
> > Default stack size is 8192 (ulimit -s). This unnecessarily eats a
considerably amount of memory. I have yet to see anything that actually
experiences problems with 1M.
>
> Actually ulimit -s is the *maximum* stack size, I'm pretty sure the
stack will start much smaller and
> grow dynamically. So changing this is not saving any RAM and it will
makes apps which do have high
> stack usage crash when they hit the new lower limit.
>
>
> Either way, it makes a noticeable difference to memory consumption on a
very memory constrained system without any other obvious adverse effects.
Interesting unless I'm reading the manpage wrong, "ulimit -s" sets the
maximum stack-size.
Maybe that also influences the initial sizing of the stack ?
I believe it does. Or at least that is the only explanation I can come up
with for the observation.
Can someone who knows more about this shed some light on this? Is there a
way to go with
a smaller initial stack-size without changing the maximum size?
It may be simpler to approach the question from the other side, i.e. is
there anything that actually ever needs more than 1MB of stack space? If
there is, I haven't seen it in the decade since I've been using this tweak
with various Fedora derived distributions.
10:15 a.m.
New subject: Better interactivity in low-memory situations
It may be simpler to approach the question from the other side, i.e.
is there anything that actually ever needs more than 1MB of stack space? If there is, I
haven't seen it in the decade since I've been using this tweak with various Fedora
derived distributions.
Any application allowing arbitrary regular expressions and/or regex
input using libpcre...
Dridi
2:50 p.m.
New subject: Better interactivity in low-memory situations
Gordan Bobic wrote:
It may be simpler to approach the question from the other side, i.e.
is
there anything that actually ever needs more than 1MB of stack space? If
there is, I haven't seen it in the decade since I've been using this tweak
with various Fedora derived distributions.
I've more than once had Java applications crash with a StackOverflowError
because Java has such a retarded 1 MiB default stack size independently of
the amount of available RAM. (You have to explicitly use the -Xss parameter
to get more.) It happened at least once in Java code and at least once in
C++ code interfaced through JNI.
So I don't think 1 MiB is a reasonable default stack size for
general-purpose computers, though it might make sense on ARM.
Kevin Kofler
2:56 p.m.
New subject: Better interactivity in low-memory situations
Gordan Bobic wrote:
Adding -ffunction-sections -fdata-sections to defaults can help
considerably in producing smaller binaries, and is not the default.
Linking with -Wl,--gc-sections helps a lot and is not the default
Well, -ffunction-sections -fdata-sections -Wl,--gc-sections mostly helps if
the binary contains a lot of unused code. This is not often the case in
dynamically linked executables or shared libraries. Where it typically
happens is statically linked binaries (where the static libraries contain
APIs not used by the statically linked executable), and even there only if
the library is not actually designed for static linking (in which case it
would put separate APIs into separate compilation units to begin with). And
static linking is a bad idea for RAM consumption anyway, at least on
GNU/Linux where dynamically linked executables actually share the code
sections of shared libraries in RAM too, not just on disk.
So I'm afraid you will probably find those options to be less useful in
practice than you expect.
Kevin Kofler
2:59 p.m.
New subject: Better interactivity in low-memory situations
On Sun, Aug 18, 2019 at 8:51 PM Kevin Kofler <kevin.kofler(a)chello.at> wrote:
Gordan Bobic wrote:
> It may be simpler to approach the question from the other side, i.e. is
> there anything that actually ever needs more than 1MB of stack space? If
> there is, I haven't seen it in the decade since I've been using this
tweak
> with various Fedora derived distributions.
I've more than once had Java applications crash with a StackOverflowError
because Java has such a retarded 1 MiB default stack size independently of
the amount of available RAM. (You have to explicitly use the -Xss
parameter
to get more.) It happened at least once in Java code and at least once in
C++ code interfaced through JNI.
So I don't think 1 MiB is a reasonable default stack size for
general-purpose computers, though it might make sense on ARM.
Right, but is it better that _everything_ else suffers with more memory
pressure for the handful of relatively infrequent use cases for which
ulimit can be used to explicitly raise the limit?
3:06 p.m.
New subject: Better interactivity in low-memory situations
Gordan Bobic wrote:
Right, but is it better that _everything_ else suffers with more
memory
pressure for the handful of relatively infrequent use cases for which
ulimit can be used to explicitly raise the limit?
Well, as I wrote, a lower limit might actually make sense on ARM. But modern
x86 computers have gigabytes of RAM, so 1 MiB is ridiculously small there.
So this would have to be an architecture-specific setting for ARM.
Kevin Kofler
3:54 p.m.
New subject: Better interactivity in low-memory situations
On Sun, Aug 18, 2019 at 9:07 PM Kevin Kofler <kevin.kofler(a)chello.at> wrote:
Gordan Bobic wrote:
> Right, but is it better that _everything_ else suffers with more memory
> pressure for the handful of relatively infrequent use cases for which
> ulimit can be used to explicitly raise the limit?
Well, as I wrote, a lower limit might actually make sense on ARM. But
modern
x86 computers have gigabytes of RAM, so 1 MiB is ridiculously small there.
So this would have to be an architecture-specific setting for ARM.
That may be so, but this thread started off with memory pressure also being
an issue for regular desktop x86 use.
4:51 p.m.
New subject: Better interactivity in low-memory situations
On Sun, Aug 18, 2019 at 2:55 PM Gordan Bobic <gordan(a)redsleeve.org> wrote:
On Sun, Aug 18, 2019 at 9:07 PM Kevin Kofler <kevin.kofler(a)chello.at> wrote:
>
> Gordan Bobic wrote:
> > Right, but is it better that _everything_ else suffers with more memory
> > pressure for the handful of relatively infrequent use cases for which
> > ulimit can be used to explicitly raise the limit?
>
> Well, as I wrote, a lower limit might actually make sense on ARM. But modern
> x86 computers have gigabytes of RAM, so 1 MiB is ridiculously small there.
> So this would have to be an architecture-specific setting for ARM.
That may be so, but this thread started off with memory pressure also being an issue for
regular desktop x86 use.
I think optimizations like this, and including compile time defaults
should get smarter to do such optimizations and have a lot of
intrinsic value. But in any case, I think it's fair to say that we're
in very broad agreement that no matter what options get used or what
optimization do or don't happen, unprivileged processes should not be
able to effectively take down the system. That to me is really
incredible to discover.
Everything else: no swap at all and tolerate abrupt and random
oom-killer killoffs, double the swap or use /dev/zram, or use 1/4 RAM
for swap, or throw a metric f ton of RAM at it, all of those are
different ways of dodging a cannon ball. Dodging the problem doesn't
actually fix the problem.Iff your dodge doesn't work out, you get hit
by a cannon ball. Not OK. It's an unprivileged task! I'm aghast.
--
Chris Murphy
9:59 p.m.
New subject: Better interactivity in low-memory situations
For what it's worth, my research group attacked basically exactly this
problem quite some time ago. We built a modified Linux kernel that we
called Redline that was impervious to fork bombs, malloc bombs, and so on.
No process could take down the system, much less unprivileged ones. I think
some of the ideas we described back then would be worth adopting / adapting
today (the code is of course hopelessly out of date: we published our paper
on this at OSDI 2008).
We had a demo where we would run two identical systems, side by side, with
the same workloads (a number of videos playing simultaneously), but with
one running Redline, and the other running stock Linux. We would launch a
fork/malloc bomb on both. The Redline system barely hiccuped. The stock
Linux kernel would freeze and become totally unresponsive (or panic). It
was a great demo, but also a pain, since we invariably had to restart the
stock Linux box :).
Redline: first class support for interactivity in commodity operating
systems
While modern workloads are increasingly interactive and resource-intensive
(e.g., graphical user interfaces, browsers, and multimedia players),
current operating systems have not kept up. These operating systems, which
evolved from core designs that date to the 1970s and 1980s, provide good
support for batch and command-line applications, but their ad hoc attempts
to handle interactive workloads are poor. Their best-effort, priority-based
schedulers provide no bounds on delays, and their resource managers (e.g.,
memory managers and disk I/O schedulers) are mostly oblivious to response
time requirements. Pressure on any one of these resources can significantly
degrade application responsiveness.
We present Redline, a system that brings first-class support for
interactive applications to commodity operating systems. Redline works with
unaltered applications and standard APIs. It uses lightweight
specifications to orchestrate memory and disk I/O management so that they
serve the needs of interactive applications. Unlike realtime systems that
treat specifications as strict requirements and thus pessimistically limit
system utilization, Redline dynamically adapts to recent load, maximizing
responsiveness and system utilization. We show that Redline delivers
responsiveness to interactive applications even in the face of extreme
workloads including fork bombs, memory bombs and bursty, large disk I/O
requests, reducing application pauses by up to two orders of magnitude.
Paper here:
https://www.usenix.org/legacy/events/osdi08/tech/full_papers/yang/yang.pdf
And links to code here:
https://emeryberger.com/research/redline/
There has been some recent follow-on work in this direction: see this work
out of Remzi and Andrea's lab at Wisconsin:
http://pages.cs.wisc.edu/~remzi/Classes/739/Fall2016/Papers/splitio-sosp1...
-- emery
--
Professor Emery Berger
College of Information and Computer Sciences
University of Massachusetts Amherst
www.emeryberger.org, @emeryberger
On Sun, Aug 18, 2019 at 2:53 PM Chris Murphy <lists(a)colorremedies.com>
wrote:
On Sun, Aug 18, 2019 at 2:55 PM Gordan Bobic
<gordan(a)redsleeve.org> wrote:
>
> On Sun, Aug 18, 2019 at 9:07 PM Kevin Kofler <kevin.kofler(a)chello.at>
wrote:
>>
>> Gordan Bobic wrote:
>> > Right, but is it better that _everything_ else suffers with more
memory
>> > pressure for the handful of relatively infrequent use cases for which
>> > ulimit can be used to explicitly raise the limit?
>>
>> Well, as I wrote, a lower limit might actually make sense on ARM. But
modern
>> x86 computers have gigabytes of RAM, so 1 MiB is ridiculously small
there.
>> So this would have to be an architecture-specific setting for ARM.
>
>
> That may be so, but this thread started off with memory pressure also
being an issue for regular desktop x86 use.
>
I think optimizations like this, and including compile time defaults
should get smarter to do such optimizations and have a lot of
intrinsic value. But in any case, I think it's fair to say that we're
in very broad agreement that no matter what options get used or what
optimization do or don't happen, unprivileged processes should not be
able to effectively take down the system. That to me is really
incredible to discover.
Everything else: no swap at all and tolerate abrupt and random
oom-killer killoffs, double the swap or use /dev/zram, or use 1/4 RAM
for swap, or throw a metric f ton of RAM at it, all of those are
different ways of dodging a cannon ball. Dodging the problem doesn't
actually fix the problem.Iff your dodge doesn't work out, you get hit
by a cannon ball. Not OK. It's an unprivileged task! I'm aghast.
--
Chris Murphy
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
3:55 a.m.
New subject: Better interactivity in low-memory situations
* Gordan Bobic:
That may be so, but this thread started off with memory pressure
also
being an issue for regular desktop x86 use.
I think the problem there is that the system has sufficient reclaimable
memory, but cannot make that memory available to applications in a
timely fashion.
Reducing compiled application footprint will only increase the amount of
reclaimable memory, probably not changing anything as far as the actual
problem is concerned.
Thanks,
Florian
4:20 a.m.
New subject: Better interactivity in low-memory situations
This seems very reminiscent of what is often referred to as "swapping
insanity", often in the context of MySQL.
The root cause there is NUMA with memory allocation only happening on the
node that requests the memory, resulting in the possibility of there being
plenty of free memory on one node but the allocating node deciding to swap
out instead of allocating far memory. I don't think there was ever a
solution fielded, but the workaround was to set the memory heavy process up
with numa round-robin allocation. This results in even allocation across
the nodes, thus avoiding swapping, but it capitulates to the fact that half
of the memory is always going to be twice as far away in terms of latency.
I don't know if the underlying root cause is shared in this case (something
less than clever in the way memory allocation is handled).
On Mon, Aug 19, 2019 at 9:55 AM Florian Weimer <fweimer(a)redhat.com> wrote:
* Gordan Bobic:
> That may be so, but this thread started off with memory pressure also
> being an issue for regular desktop x86 use.
I think the problem there is that the system has sufficient reclaimable
memory, but cannot make that memory available to applications in a
timely fashion.
Reducing compiled application footprint will only increase the amount of
reclaimable memory, probably not changing anything as far as the actual
problem is concerned.
Thanks,
Florian
12:48 a.m.
New subject: Better interactivity in low-memory situations
On Sunday, August 18, 2019 4:33:47 AM MST Gordan Bobic wrote:
On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org>
wrote:
> This seems like a distraction from the real goal here, which is to
> ensure Fedora remains responsive under heavy memory pressure,
I think this is an overwhelmingly important point, and as somebody
regularly working with ARM machines with tiny amounts of RAM, it is of
considerable interest to me.
I typically use CentOS because stability is important to me, but most
worthwhile things filter to there, so I hope what I'm about to say is not
_too_ deprecated.
1) Compile options
From what I can tell from rpm macro options, default on C7 seems to be -O2.
-Os seems to help in most cases.
Adding -ffunction-sections -fdata-sections to defaults can help
considerably in producing smaller binaries, and is not the default.
Linking with -Wl,--gc-sections helps a lot and is not the default
Extensive stripping seems to already be the default (--strip-unneeded,
removal of .comment and .note sections)
2) Runtime condiguration
Default stack size is 8192 (ulimit -s). This unnecessarily eats a
considerably amount of memory. I have yet to see anything that actually
experiences problems with 1M.
3) zram
This was mentioned earlier in the thread, and on most of my systems, memory
constrained or otherwise, unless I have an overwhelming reason not to, I
run with zram swap equal in size to RAM with lz4 compression and
vm.swappiness=100. I typically see compression ratios between 2:1 and 3:1
in zram, so on a system with, say, 10GB of RAM, it would provide 10GB of
very fast swap at a cost of 3-5GB of RAM. This seems like a favourable
trade off, especially on systems with extremely constrained RAM (e.g. ARM
devices with 512MB of RAM).
I'm sure there is more that can be done, but this seems like a good start
as far as the cost / benefit is concerned.
Python, Lua and a few other common programs can have issues with a stack size
of 1MiB.
--
John M. Harris, Jr. <johnmh(a)splentity.com>
Splentity
https://splentity.com/
11:30 p.m.
New subject: Better interactivity in low-memory situations
On Tuesday, August 20, 2019 10:48:06 PM MST John Harris wrote:
On Sunday, August 18, 2019 4:33:47 AM MST Gordan Bobic wrote:
> On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org> wrote:
>
> > This seems like a distraction from the real goal here, which is to
> > ensure Fedora remains responsive under heavy memory pressure,
>
>
> I think this is an overwhelmingly important point, and as somebody
> regularly working with ARM machines with tiny amounts of RAM, it is of
> considerable interest to me.
> I typically use CentOS because stability is important to me, but most
> worthwhile things filter to there, so I hope what I'm about to say is not
> _too_ deprecated.
>
> 1) Compile options
> From what I can tell from rpm macro options, default on C7 seems to be
> -O2.
-Os seems to help in most cases.
> Adding -ffunction-sections -fdata-sections to defaults can help
> considerably in producing smaller binaries, and is not the default.
> Linking with -Wl,--gc-sections helps a lot and is not the default
> Extensive stripping seems to already be the default (--strip-unneeded,
> removal of .comment and .note sections)
>
> 2) Runtime condiguration
> Default stack size is 8192 (ulimit -s). This unnecessarily eats a
> considerably amount of memory. I have yet to see anything that actually
> experiences problems with 1M.
>
> 3) zram
> This was mentioned earlier in the thread, and on most of my systems,
> memory constrained or otherwise, unless I have an overwhelming reason
> not to, I run with zram swap equal in size to RAM with lz4 compression
> and
> vm.swappiness=100. I typically see compression ratios between 2:1 and 3:1
> in zram, so on a system with, say, 10GB of RAM, it would provide 10GB of
> very fast swap at a cost of 3-5GB of RAM. This seems like a favourable
> trade off, especially on systems with extremely constrained RAM (e.g. ARM
> devices with 512MB of RAM).
>
> I'm sure there is more that can be done, but this seems like a good start
> as far as the cost / benefit is concerned.
Python, Lua and a few other common programs can have issues with a stack
size of 1MiB.
--
John M. Harris, Jr. <johnmh(a)splentity.com>
Splentity
https://splentity.com/
I would also like to add that I don't see how it's even possible to run into a
low-memory scenario on a system with 10 GiB (That's a *lot* of memory! I run
on a Core 2 Duo based system that can support a max of 8 GiB as my daily
driver.) often enough to have a problem with oom_killer.
--
John M. Harris, Jr. <johnmh(a)splentity.com>
Splentity
https://splentity.com/
1:01 p.m.
New subject: Better interactivity in low-memory situations
On 9/3/19 12:30 AM, John Harris wrote:
On Tuesday, August 20, 2019 10:48:06 PM MST John Harris wrote:
> On Sunday, August 18, 2019 4:33:47 AM MST Gordan Bobic wrote:
>
>> On Sun, Aug 11, 2019 at 10:36 AM <mcatanzaro(a)gnome.org> wrote:
>>
>>> This seems like a distraction from the real goal here, which is to
>>> ensure Fedora remains responsive under heavy memory pressure,
>>
>> I think this is an overwhelmingly important point, and as somebody
>> regularly working with ARM machines with tiny amounts of RAM, it is of
>> considerable interest to me.
>> I typically use CentOS because stability is important to me, but most
>> worthwhile things filter to there, so I hope what I'm about to say is not
>> _too_ deprecated.
>>
>> 1) Compile options
>> From what I can tell from rpm macro options, default on C7 seems to be
>> -O2.
-Os seems to help in most cases.
>> Adding -ffunction-sections -fdata-sections to defaults can help
>> considerably in producing smaller binaries, and is not the default.
>> Linking with -Wl,--gc-sections helps a lot and is not the default
>> Extensive stripping seems to already be the default (--strip-unneeded,
>> removal of .comment and .note sections)
>>
>> 2) Runtime condiguration
>> Default stack size is 8192 (ulimit -s). This unnecessarily eats a
>> considerably amount of memory. I have yet to see anything that actually
>> experiences problems with 1M.
>>
>> 3) zram
>> This was mentioned earlier in the thread, and on most of my systems,
>> memory constrained or otherwise, unless I have an overwhelming reason
>> not to, I run with zram swap equal in size to RAM with lz4 compression
>> and
>> vm.swappiness=100. I typically see compression ratios between 2:1 and 3:1
>> in zram, so on a system with, say, 10GB of RAM, it would provide 10GB of
>> very fast swap at a cost of 3-5GB of RAM. This seems like a favourable
>> trade off, especially on systems with extremely constrained RAM (e.g. ARM
>> devices with 512MB of RAM).
>>
>> I'm sure there is more that can be done, but this seems like a good start
>> as far as the cost / benefit is concerned.
>
> Python, Lua and a few other common programs can have issues with a stack
> size of 1MiB.
>
> --
> John M. Harris, Jr. <johnmh(a)splentity.com>
> Splentity
> https://splentity.com/
I would also like to add that I don't see how it's even possible to run into a
low-memory scenario on a system with 10 GiB (That's a *lot* of memory! I run
on a Core 2 Duo based system that can support a max of 8 GiB as my daily
driver.) often enough to have a problem with oom_killer.
I'm finding that 32G is not necessarily sufficient for compiling clang
itself. Similarly I've had a hard time compiling UnrealEngine from
source. I usually see ld using up to 12G of memory to link each
artifact. Using -j$(nproc) on a 16 VCPU system amplifies the issue. I
rely on adding another 32G swap file to complete the job.
I'm now using an NVME SSD for my swap file. No more hangs for the most
part! Usually if I lock up, it's for maybe a minute before OOM steps in.
However, I'd definitely like to see a non-zram "solution" for use cases
like this. Ultimately I'd like to see the use of traditional swap files
to not hang a system even if it is placed on a md RAID array. I'm
guessing that the long term fix is for OOM to happen sooner, and/or
kernel schedulers to be improved, as mentioned elsewhere in this thread.
Ideally this wouldn't require the use of systemd or cgroups to make it
possible.
2:03 p.m.
New subject: Better interactivity in low-memory situations
On Wed, Sep 4, 2019 at 12:01 PM Kyle Marek <psppsn96(a)gmail.com> wrote:
I'm finding that 32G is not necessarily sufficient for compiling clang
itself. Similarly I've had a hard time compiling UnrealEngine from
source. I usually see ld using up to 12G of memory to link each
artifact. Using -j$(nproc) on a 16 VCPU system amplifies the issue. I
rely on adding another 32G swap file to complete the job.
I setup swap on ZRAM device at 1.5X RAM. That was just as
intermittently unresponsive (seconds, and periods above 10 minutes)
GUI, to the degree using the laptop as a laptop was pointless. But,
webkitgtk build did complete with the default command, and in a
shorter time than with a swap partition on SSD (which became
sufficiently unresponsive even via ssh that I gave up and forced power
off).
But, I don't think I can recommend such a setup as a default
installation of Fedora Workstation. On a system with 32G RAM, that
would mean a 48G /dev/zram device for swap. If it completely filled
up, it would use ~24G of RAM, leaving 8GiB for compiling + OS +
workstation stuff. You are getting a huge swap device that's almost as
fast as RAM, but it's so fast that it means burning a huge amount of
CPU cycles on compression/decompression. And kswapd is not threaded,
it maxes out at 100% of a single core, and once it's there, that's it,
you can't scale beyond it.
I'm now using an NVME SSD for my swap file. No more hangs for the
most
part! Usually if I lock up, it's for maybe a minute before OOM steps in.
However, I'd definitely like to see a non-zram "solution" for use cases
like this. Ultimately I'd like to see the use of traditional swap files
to not hang a system even if it is placed on a md RAID array. I'm
guessing that the long term fix is for OOM to happen sooner, and/or
kernel schedulers to be improved, as mentioned elsewhere in this thread.
Ideally this wouldn't require the use of systemd or cgroups to make it
possible.
I'm skeptical of swapfiles for this use case, because it likely
increases the chance for file system disk contention, see section
11.17
https://www.kernel.org/doc/gorman/html/understand/understand014.html
And in heavy swap use and memory pressure, I already see evidence of
applications evicting their own executable code and constantly
re-reading it from disk. That's increasing the chance of IO pressure
resulting in stalls.
Incidental swap is fine, and I think it's possible to use either zram
or zswap to moderate the transition to a swap partition. But once this
grows to an aggressive swap dependency, the desktop is responsiveness
is lost.
--
Chris Murphy
2:16 p.m.
New subject: Better interactivity in low-memory situations
* Kyle Marek:
I'm finding that 32G is not necessarily sufficient for compiling
clang
itself. Similarly I've had a hard time compiling UnrealEngine from
source. I usually see ld using up to 12G of memory to link each
artifact. Using -j$(nproc) on a 16 VCPU system amplifies the issue. I
rely on adding another 32G swap file to complete the job.
Out of curiosity, which ld do you use? BFD ld, gold, or LLVM's?
Thanks,
Florian
2:17 p.m.
New subject: Better interactivity in low-memory situations
per cgroup swap file support
https://lwn.net/Articles/591495/
This might be interesting. What happens if swap is only assigned to
new unprivileged tasks? Your compile might take three days but it
doesn't make the GUI fall over? *shrug*
---
Chris Murphy
2:38 p.m.
New subject: Better interactivity in low-memory situations
On Wed, Sep 4, 2019 at 1:03 PM Chris Murphy <lists(a)colorremedies.com> wrote:
I'm skeptical of swapfiles for this use case, because it likely
increases the chance for file system disk contention, see section
11.17
https://www.kernel.org/doc/gorman/html/understand/understand014.html
Actually that looks outdated now.
https://patchwork.kernel.org/patch/10347293/
Btrfs since 5.0 also supports swapfiles by avoiding bmap, using
btrfs_swap_activate, the btrfs equivalent of iomap_swap_activate
--
Chris Murphy
5:08 p.m.
New subject: Better interactivity in low-memory situations
On 9/4/19 3:38 PM, Chris Murphy wrote:
On Wed, Sep 4, 2019 at 1:03 PM Chris Murphy
<lists(a)colorremedies.com> wrote:
> I'm skeptical of swapfiles for this use case, because it likely
> increases the chance for file system disk contention, see section
> 11.17
> https://www.kernel.org/doc/gorman/html/understand/understand014.html
Actually that looks outdated now.
https://patchwork.kernel.org/patch/10347293/
Btrfs since 5.0 also supports swapfiles by avoiding bmap, using
btrfs_swap_activate, the btrfs equivalent of iomap_swap_activate
I misspoke when I said "file". It's an LVM logical volume.
5:12 p.m.
New subject: Better interactivity in low-memory situations
On 9/4/19 3:16 PM, Florian Weimer wrote:
* Kyle Marek:
> I'm finding that 32G is not necessarily sufficient for compiling clang
> itself. Similarly I've had a hard time compiling UnrealEngine from
> source. I usually see ld using up to 12G of memory to link each
> artifact. Using -j$(nproc) on a 16 VCPU system amplifies the issue. I
> rely on adding another 32G swap file to complete the job.
Out of curiosity, which ld do you use? BFD ld, gold, or LLVM's?
Thanks,
Florian
In both cases, I believe it was GNU ld.
I know it was GNU ld when building clang/llvm tools itself. I only had
GCC and GNU binutils installed at the time.
UnrealEngine requires clang to compile, but I can't remember for sure if
they finally linked with ld (GNU on my $PATH) or lld.
1689
days inactive
1706
days old
23 comments
9 participants
participants (9)
-
Chris Murphy -
Dridi Boukelmoune -
Emery Berger -
Florian Weimer -
Gordan Bobic -
Hans de Goede -
John Harris -
Kevin Kofler -
Kyle Marek