Out of virtual memory on ARM builder

[Jerry James]

What do I do about this?

http://koji.fedoraproject.org/koji/taskinfo?taskID=6526911

[While building and running tests]:

    CC   ../build/flintxx/test/t-traits
    CC   ../build/flintxx/test/t-fmpzxx
virtual memory exhausted: Cannot allocate memory
make[1]: *** [../build/flintxx/test/t-fmpzxx] Error 1
make[1]: Leaving directory `/builddir/build/BUILD/flint-2.4.1/flintxx'
make: *** [check] Error 2

The i686 and x86_64 builds were successful.  What can I do to increase the
likelihood that the ARM build will also complete successfully?  I would
rather not disable tests if that is not absolutely necessary.
-- 
Jerry James
http://www.jamezone.org/

[Dennis Gilmore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 13 Feb 2014 13:47:51 -0700
Jerry James <loganjerry(a)gmail.com&gt; wrote:

>  What do I do about this?
>  
>  http://koji.fedoraproject.org/koji/taskinfo?taskID=6526911
>  
>  [While building and running tests]:
>  
>      CC   ../build/flintxx/test/t-traits
>      CC   ../build/flintxx/test/t-fmpzxx
>  virtual memory exhausted: Cannot allocate memory
>  make[1]: *** [../build/flintxx/test/t-fmpzxx] Error 1
>  make[1]: Leaving directory `/builddir/build/BUILD/flint-2.4.1/flintxx'
>  make: *** [check] Error 2
>  
>  The i686 and x86_64 builds were successful.  What can I do to
>  increase the likelihood that the ARM build will also complete
>  successfully?  I would rather not disable tests if that is not
>  absolutely necessary.
>  
The arm builders all have 4gb of ram. how much ram should the tests
need?

Dennis
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBAgAGBQJS/dkZAAoJEH7ltONmPFDRw+kP/Aldw1w08M2v0/Wrnf5rkJb3
> jtny2XE4UCLV1zx/g4Fy7SN/f2DIbiJrWrVFul9qHrvTqe0hGHi5tGy50w8XuBtr
> 01n1NMhuAiXWRXvvPGMGUkxJnDCJWMk39sfuwFzn2Cut5WkkprFzUInjstwtGRkP
> k8xySLkUrh+9dhO+Gke72e4LSMbphhD5IlmukLnFt3Z/tBsR2pn2TN5QrBfZ1PSH
> AoJzluVLsnNql7r6tCK/YhBIyb3B9TP1sv32KcKgPWv+9l+KbBGuZYuUIwTaBvqw
> ZowB93sJiAlaFimbfEbjESHRHtWL2BjhHxHXSMcJf9tcbhejNL4RcUYunZ0d7tTq
> xdAL7RXPv98rDAuoADU9ibsLWa4TckCh63DUcxzmmrWOHANQQxHeNi994iMQB5VA
> i+bqjkrkeinDhBm3XvYjZgfAnYXEYMDaNbKYqUpz/Wfg5slgLl4QsH4oeXWFRIbP
> qn4MZTfwt6VVtMrloYtXSOcvcSYkq/AjoeZ8iB4WzYlTBUG03OpxKG3gK+DyuvXd
> D++lPy5DYjncQIC/h0XaNfzinWIiKNDWQZKarhC90Zgf4SrHjoLaMKIWlZOA+yhN
> +NBjw3oOYm5NkKe7S0YquHgNJS+hdt7IF+2p3tsKEQcIwW9rJXrhs2HabNfTYQlv
> qEi9tDTS157U1GhREHJ1
> =FteW
> -----END PGP SIGNATURE-----
>

[Dan Horák]

On Fri, 14 Feb 2014 02:51:36 -0600
Dennis Gilmore <dennis(a)ausil.us&gt; wrote:

>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  On Thu, 13 Feb 2014 13:47:51 -0700
>  Jerry James <loganjerry(a)gmail.com&gt; wrote:
>  
>  > What do I do about this?
>  > 
>  > http://koji.fedoraproject.org/koji/taskinfo?taskID=6526911
>  > 
>  > [While building and running tests]:
>  > 
>  >     CC   ../build/flintxx/test/t-traits
>  >     CC   ../build/flintxx/test/t-fmpzxx
>  > virtual memory exhausted: Cannot allocate memory
>  > make[1]: *** [../build/flintxx/test/t-fmpzxx] Error 1
>  > make[1]: Leaving directory
>  > `/builddir/build/BUILD/flint-2.4.1/flintxx' make: *** [check] Error
>  > 2
>  > 
>  > The i686 and x86_64 builds were successful.  What can I do to
>  > increase the likelihood that the ARM build will also complete
>  > successfully?  I would rather not disable tests if that is not
>  > absolutely necessary.
>  
>  The arm builders all have 4gb of ram. how much ram should the tests
>  need? 
I think it was g++  (or the ld linker) what went out of memory, not
unexpected with 4GB memory plus some swap, 4 CPUs and parallel make.
Jerry, can you retry with parallel make disabled?


		Dan

[Jerry James]

On Fri, Feb 14, 2014 at 1:59 AM, Dan Horák <dan(a)danny.cz&gt; wrote:

>  I think it was g++  (or the ld linker) what went out of memory, not
>  unexpected with 4GB memory plus some swap, 4 CPUs and parallel make.
>  Jerry, can you retry with parallel make disabled?
>  
Actually, the failure occurred while running %check, which is this:

%check
make check

There's no parallel make involved.  Drat.  Well, I'll figure out which
test(s) are eating up the memory and disable it/them on ARM, I guess.
 Thanks for the replies, everybody.
-- 
Jerry James
http://www.jamezone.org/

[Jerry James]

On Fri, Feb 14, 2014 at 8:54 AM, Jerry James <loganjerry(a)gmail.com&gt; wrote:

>  There's no parallel make involved.  Drat.  Well, I'll figure
> out which
>  test(s) are eating up the memory and disable it/them on ARM, I guess.
>   Thanks for the replies, everybody.
>  
A little bit of digging into the sources shows that I just need to add
-DHAVE_FAST_COMPILER=0 to the build flags to turn off the tests that
require large amounts of memory and CPU cycles to compile.  I will do this
for ARM.  Are there any secondary arches that are likely to have the same
problem?
-- 
Jerry James
http://www.jamezone.org/

[Richard W.M. Jones]

On Fri, Feb 14, 2014 at 09:10:23AM -0700, Jerry James wrote:
>  On Fri, Feb 14, 2014 at 8:54 AM, Jerry James
> <loganjerry(a)gmail.com&gt; wrote:
>  
>  > There's no parallel make involved.  Drat.  Well, I'll figure out which
>  > test(s) are eating up the memory and disable it/them on ARM, I guess.
>  >  Thanks for the replies, everybody.
>  >
>  
>  A little bit of digging into the sources shows that I just need to add
>  -DHAVE_FAST_COMPILER=0 to the build flags to turn off the tests that
>  require large amounts of memory and CPU cycles to compile.  I will do this
>  for ARM.  Are there any secondary arches that are likely to have the same
>  problem? 
It sounds rather ill-defined :-)  What counts as large amounts of
memory or CPU?

Can you choose it based on something like the output of `free -m`
and/or `grep -i bogomips /proc/cpuinfo` ?  Note the second command
might not produce any output (no output on ARM for sure) so don't rely
on that.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Jerry James]

On Fri, Feb 14, 2014 at 9:16 AM, Richard W.M. Jones <rjones(a)redhat.com&gt;wrote:

>  It sounds rather ill-defined :-)  What counts as large amounts of
>  memory or CPU?
>  
Yes, I really don't know.  CPU isn't so much the concern, anyway.  Let
those builder churn away for long periods of time.  Bwahahahaha!

But memory, now, that's an issue.  All I know for sure is that the x86_64
and i686 builds succeeded, so those boxes had "enough" memory; the ARM
build failed, so that box did not have "enough" memory.  Since memory is
exhausted while compiling the test, not while running the test, the
required amount of memory isn't necessarily fixed.  It may depend on
architecture, version of the compiler, and possibly other factors that I
don't know about.  I really don't know what "enough" is, therefore ...

Can you choose it based on something like the output of `free -m`
>  and/or `grep -i bogomips /proc/cpuinfo` ?  Note the second command
>  might not produce any output (no output on ARM for sure) so don't rely
>  on that. 

... I don't really see how I can do something like this.  For now I have
turned the troublesome tests off for ARM only, and left behind a comment in
the spec that if other architectures fail to compile the tests due to
memory exhaustion, the same should be done for them.
-- 
Jerry James
http://www.jamezone.org/

[Florian Weimer]

On 02/14/2014 05:30 PM, Jerry James wrote:

>  But memory, now, that's an issue.  All I know for sure is that
> the
>  x86_64 and i686 builds succeeded, so those boxes had "enough" memory;
>  the ARM build failed, so that box did not have "enough" memory. 
Based on the data presented so far, it could also be an ARM-specific GCC 
bug, so no amount of memory would be sufficient there.

-- 
Florian Weimer / Red Hat Product Security Team

[Jan Kratochvil]

On Fri, 14 Feb 2014 17:16:48 +0100, Richard W.M. Jones wrote:
>  Can you choose it based on something like the output of `free -m`
>  and/or `grep -i bogomips /proc/cpuinfo` ? 
That would not be great, it would make build results unreproducible
(=the testsuite results) on different build hosts.


Jan

[Dmitry Butskoy]

Dennis Gilmore wrote:
>  The arm builders all have 4gb of ram. how much ram should the tests
>  need? 
BTW, some "big" application -- seamonkey (former mozilla/netscape suite) 
-- fails to build on arm due to the same reason -- not enough memory on 
the build host.


~buc

[Mátyás Selmeci]

On 02/14/2014 02:53 PM, Dmitry Butskoy wrote:
>  Dennis Gilmore wrote:
> > The arm builders all have 4gb of ram. how much ram should the tests
> > need?
>
>  BTW, some "big" application -- seamonkey (former mozilla/netscape 
>  suite) -- fails to build on arm due to the same reason -- not enough 
>  memory on the build host.
>
>
>  ~buc
>  
This may be a stupid question, but can you solve this by putting more 
swap on those builders?
-Mat

[Kevin Fenzi]

On Fri, 14 Feb 2014 16:00:09 -0600
Mátyás Selmeci <matyas(a)cs.wisc.edu&gt; wrote:

>  This may be a stupid question, but can you solve this by putting more
>
>  swap on those builders? 
Possibly so yeah. Currently they have 4GB mem and 4GB swap. 

Ideally it would be good to fix in the package or tools though, so
people with smallish machines could actually build things too. 

kevin

[Matthew Garrett]

On Fri, Feb 14, 2014 at 04:00:09PM -0600, Mátyás Selmeci wrote:

>  This may be a stupid question, but can you solve this by putting
>  more swap on those builders? 
It depends. If the system is sufficiently resource constrained that 
malloc() is actually telling you that you're not going to the moon, more 
RAM will help. But what's more likely is that it's running out of 
process address space - a 32 bit process can only address 3GB of address 
space (which isn't necessarily all RAM), no matter how much RAM is 
available[1]. Adding more RAM isn't going to help there. Getting rid of 
32-bit build systems is.

[1] On x86, anyway. I don't know what the ARM VM split is.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Richard W.M. Jones]

On Fri, Feb 14, 2014 at 10:29:38PM +0000, Matthew Garrett wrote:
>  On Fri, Feb 14, 2014 at 04:00:09PM -0600, Mátyás Selmeci wrote:
>  
>  > This may be a stupid question, but can you solve this by putting
>  > more swap on those builders?
>  
>  It depends. If the system is sufficiently resource constrained that 
>  malloc() is actually telling you that you're not going to the moon, more 
>  RAM will help. But what's more likely is that it's running out of 
>  process address space - a 32 bit process can only address 3GB of address 
>  space (which isn't necessarily all RAM), no matter how much RAM is 
>  available[1]. Adding more RAM isn't going to help there. Getting rid of 
>  32-bit build systems is.
>  
>  [1] On x86, anyway. I don't know what the ARM VM split is. 
3G/1G on my ARM 32 bit test system, same as x86.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

On Thu, Feb 13, 2014 at 01:47:51PM -0700, Jerry James wrote:
>  What do I do about this?
>  
>  http://koji.fedoraproject.org/koji/taskinfo?taskID=6526911
>  
>  [While building and running tests]:
>  
>      CC   ../build/flintxx/test/t-traits
>      CC   ../build/flintxx/test/t-fmpzxx
>  virtual memory exhausted: Cannot allocate memory
>  make[1]: *** [../build/flintxx/test/t-fmpzxx] Error 1
>  make[1]: Leaving directory `/builddir/build/BUILD/flint-2.4.1/flintxx'
>  make: *** [check] Error 2
>  
>  The i686 and x86_64 builds were successful.  What can I do to increase the
>  likelihood that the ARM build will also complete successfully?  I would
>  rather not disable tests if that is not absolutely necessary. 
As a general comment, ARM 32 bit has quite limited physical RAM.

Commonly available hardware maxes out at 2 GB.  1 GB or even 512 MB is
not uncommon.  This might be a factor if you expect people to rebuild
your package on their Cubies and Olimexs.

The builders apparently have 4 GB according to a comment in this
thread.  The Calxeda machines I used had 8 GB, the largest I've seen
on ARM.  These use LPAE (which is like PAE on x86), so all that memory
is not available to a single process.

ARM 64 bit will be much better.

Also the ratio of cores to memory can be unusual.  I have an 8 core
ARM 32 bit machine that has 2 GB of RAM, with swap on MMC (very slow),
so you wouldn't want to do a 'make -j8' on it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/