armv7hl builds running out of memory

[Kaleb S. KEITHLEY]

trying to build ceph-12 on f27 armv7hl.

It builds on everything x86_64, aarch64, s390x, and i686 (w/o java), but
on armv7hl the build fails, reporting out of memory.

...
[100%] Built target ceph-osd
cc1plus: out of memory allocating 11284160 bytes after a total of
58859520 bytes
make[2]: *** [src/CMakeFiles/ceph-dencoder.dir/build.make:64:
src/CMakeFiles/ceph-dencoder.dir/test/encoding/ceph_dencoder.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1626:
src/CMakeFiles/ceph-dencoder.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
...

full log at
https://kojipkgs.fedoraproject.org//work/tasks/4038/20724038/build.log

Is there any way to bump up swap on the builders? Or any trick to get
more memory or run on a particular machine that has more memory?

Thanks

[Peter Robinson]

On Wed, Jul 26, 2017 at 12:53 PM, Kaleb S. KEITHLEY  wrote:
> trying to build ceph-12 on f27 armv7hl.
>
> It builds on everything x86_64, aarch64, s390x, and i686 (w/o java), but
> on armv7hl the build fails, reporting out of memory.
>
> ...
> [100%] Built target ceph-osd
> cc1plus: out of memory allocating 11284160 bytes after a total of
> 58859520 bytes
> make[2]: *** [src/CMakeFiles/ceph-dencoder.dir/build.make:64:
> src/CMakeFiles/ceph-dencoder.dir/test/encoding/ceph_dencoder.cc.o] Error 1
> make[1]: *** [CMakeFiles/Makefile2:1626:
> src/CMakeFiles/ceph-dencoder.dir/all] Error 2
> make[1]: *** Waiting for unfinished jobs....
> ...
>
> full log at
> https://kojipkgs.fedoraproject.org//work/tasks/4038/20724038/build.log
>
> Is there any way to bump up swap on the builders? Or any trick to get
> more memory or run on a particular machine that has more memory?

The ARM builders (both ARMv7 and aarch64) have 24 Gb of memory, which
is more than all other arches which have 10Gb, so I suspect the issue
is not in the build hardware but an issue with ceph itself using (or
leaking) too much memory.

Peter

[Daniel P. Berrange]

On Wed, Jul 26, 2017 at 12:57:32PM +0100, Peter Robinson wrote:
> On Wed, Jul 26, 2017 at 12:53 PM, Kaleb S. KEITHLEY  wrote:
> > trying to build ceph-12 on f27 armv7hl.
> >
> > It builds on everything x86_64, aarch64, s390x, and i686 (w/o java), but
> > on armv7hl the build fails, reporting out of memory.
> >
> > ...
> > [100%] Built target ceph-osd
> > cc1plus: out of memory allocating 11284160 bytes after a total of
> > 58859520 bytes
> > make[2]: *** [src/CMakeFiles/ceph-dencoder.dir/build.make:64:
> > src/CMakeFiles/ceph-dencoder.dir/test/encoding/ceph_dencoder.cc.o] Error 1
> > make[1]: *** [CMakeFiles/Makefile2:1626:
> > src/CMakeFiles/ceph-dencoder.dir/all] Error 2
> > make[1]: *** Waiting for unfinished jobs....
> > ...
> >
> > full log at
> > https://kojipkgs.fedoraproject.org//work/tasks/4038/20724038/build.log
> >
> > Is there any way to bump up swap on the builders? Or any trick to get
> > more memory or run on a particular machine that has more memory?
>
> The ARM builders (both ARMv7 and aarch64) have 24 Gb of memory, which
> is more than all other arches which have 10Gb, so I suspect the issue
> is not in the build hardware but an issue with ceph itself using (or
> leaking) too much memory.

Strangely the error message is complaining about being unable to
allocate 10 MB, with a current total usage of 55 MB, so no where
near the GB of memory, unless there is some other process running
in parallel which is consuming it all.

Regards,
Daniel

[Björn 'besser82' Esser]

Am 26.07.2017 um 14:10 schrieb Daniel P. Berrange:
> On Wed, Jul 26, 2017 at 12:57:32PM +0100, Peter Robinson wrote:
> > On Wed, Jul 26, 2017 at 12:53 PM, Kaleb S. KEITHLEY  wrote:
> > > trying to build ceph-12 on f27 armv7hl.
> > >
> > > It builds on everything x86_64, aarch64, s390x, and i686 (w/o java), but
> > > on armv7hl the build fails, reporting out of memory.
> > >
> > > ...
> > > [100%] Built target ceph-osd
> > > cc1plus: out of memory allocating 11284160 bytes after a total of
> > > 58859520 bytes
> > > make[2]: *** [src/CMakeFiles/ceph-dencoder.dir/build.make:64:
> > > src/CMakeFiles/ceph-dencoder.dir/test/encoding/ceph_dencoder.cc.o] Error 1
> > > make[1]: *** [CMakeFiles/Makefile2:1626:
> > > src/CMakeFiles/ceph-dencoder.dir/all] Error 2
> > > make[1]: *** Waiting for unfinished jobs....
> > > ...
> > >
> > > full log at
> > > https://kojipkgs.fedoraproject.org//work/tasks/4038/20724038/build.log
> > >
> > > Is there any way to bump up swap on the builders? Or any trick to get
> > > more memory or run on a particular machine that has more memory?
> >
> > The ARM builders (both ARMv7 and aarch64) have 24 Gb of memory, which
> > is more than all other arches which have 10Gb, so I suspect the issue
> > is not in the build hardware but an issue with ceph itself using (or
> > leaking) too much memory.
>
> Strangely the error message is complaining about being unable to
> allocate 10 MB, with a current total usage of 55 MB, so no where
> near the GB of memory, unless there is some other process running
> in parallel which is consuming it all.
>
> Regards,
> Daniel

It looks like the whole parallelized make-process is hitting the 4 
Gbytes limit per process / task on 32 Bit arches…

Have you tried this?

%build
export CFLAGS="echo %{optflags} | sed -e 's!-pipe!!g'"
export CXXFLAGS="$CFLAGS"
…

Sometimes piping from cpp to gcc to as to ld takes too much memory on 32 
Bit arches…

Cheers,
   Björn

[Al Stone]

I've been experimenting in a slightly different environment (RHEL vs Fedora) but have been seeing oddly similar results.  The use or not of the "-pipe" in GCC didn't seem to help.  If I forced the make in the %build step to be just "make" (aka, "make -j1"), I could always get a build to work, albeit slowly.

It turns out there is a typo in the spec file; look for the string "WTIH_BABELTRACE" -- that should be "WITH_BABELTRACE".  In the environment I'm using, "make -j32" is the default state.  If I leave the typo alone and do not change the "make -j32", I can pretty consistently get the ceph build to fail; the failure moves around a bit but generally seems to hang around with where the babeltrace headers are being used (somewhere in RBD code, usually).  If I fix the typo -- and change nothing else -- the build succeeds.

Would you mind trying this one change -- fixing the typo *only* -- and see if you get the same results?

[Kaleb S. KEITHLEY]

On 07/26/2017 06:25 PM, Al Stone wrote:
> I've been experimenting in a slightly different environment (RHEL vs Fedora) but have been seeing oddly similar results.  The use or not of the "-pipe" in GCC didn't seem to help.  If I forced the make in the %build step to be just "make" (aka, "make -j1"), I could always get a build to work, albeit slowly.
>
> It turns out there is a typo in the spec file; look for the string "WTIH_BABELTRACE" -- that should be "WITH_BABELTRACE".  In the environment I'm using, "make -j32" is the default state.  If I leave the typo alone and do not change the "make -j32", I can pretty consistently get the ceph build to fail; the failure moves around a bit but generally seems to hang around with where the babeltrace headers are being used (somewhere in RBD code, usually).  If I fix the typo -- and change nothing else -- the build succeeds.
>
> Would you mind trying this one change -- fixing the typo *only* -- and see if you get the same results?

If by same result you mean the build still fails, then yes. I get the 
same result.

It's still running out of memory. Not the same way as the prior builds 
though.

...
[100%] Building CXX object src/rgw/CMakeFiles/radosgw.dir/rgw_main.cc.o
/usr/include/c++/7/bits/stl_map.h: In static member function 'static 
void 
pg_missing_set<TrackChanges>::generate_test_instances(std::__cxx11::list<pg_missing_set<TrackChanges>*>&) 
[with bool TrackChanges = false]':
/usr/include/c++/7/bits/stl_map.h:493:4: note: parameter passing for 
argument of type 'std::_Rb_tree<hobject_t, std::pair<const hobject_t, 
pg_missing_item>, std::_Select1st<std::pair<const hobject_t, 
pg_missing_item> >, std::less<hobject_t>, std::allocator<std::pair<const 
hobject_t, pg_missing_item> > >::const_iterator {aka 
std::_Rb_tree_const_iterator<std::pair<const hobject_t, pg_missing_item>
> }' changed in GCC 7.1

__i = _M_t._M_emplace_hint_unique(__i, std::piecewise_construct,
     ^~~
virtual memory exhausted: Operation not permitted
...
make: *** [Makefile:141: all] Error 2
RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.RgosXb (%build)
     Bad exit status from /var/tmp/rpm-tmp.RgosXb (%build)
Child return code was: 1
...

See https://koji.fedoraproject.org/koji/taskinfo?taskID=20797264 for 
full logs.

--

Kaleb

[Al Stone]

On 07/27/2017 09:16 AM, Kaleb S. KEITHLEY wrote:
> On 07/26/2017 06:25 PM, Al Stone wrote:
> > I've been experimenting in a slightly different environment (RHEL vs Fedora)
> > but have been seeing oddly similar results.  The use or not of the "-pipe" in
> > GCC didn't seem to help.  If I forced the make in the %build step to be just
> > "make" (aka, "make -j1"), I could always get a build to work, albeit slowly.
> >
> > It turns out there is a typo in the spec file; look for the string
> > "WTIH_BABELTRACE" -- that should be "WITH_BABELTRACE".  In the environment I'm
> > using, "make -j32" is the default state.  If I leave the typo alone and do not
> > change the "make -j32", I can pretty consistently get the ceph build to fail;
> > the failure moves around a bit but generally seems to hang around with where
> > the babeltrace headers are being used (somewhere in RBD code, usually).  If I
> > fix the typo -- and change nothing else -- the build succeeds.
> >
> > Would you mind trying this one change -- fixing the typo *only* -- and see if
> > you get the same results?
>
> If by same result you mean the build still fails, then yes. I get the same result.
>
> It's still running out of memory. Not the same way as the prior builds though.
>
> ...
> [100%] Building CXX object src/rgw/CMakeFiles/radosgw.dir/rgw_main.cc.o
> /usr/include/c++/7/bits/stl_map.h: In static member function 'static void
> pg_missing_set<TrackChanges>::generate_test_instances(std::__cxx11::list<pg_missing_set<TrackChanges>*>&)
> [with bool TrackChanges = false]':
> /usr/include/c++/7/bits/stl_map.h:493:4: note: parameter passing for argument of
> type 'std::_Rb_tree<hobject_t, std::pair<const hobject_t, pg_missing_item>,
> std::_Select1st<std::pair<const hobject_t, pg_missing_item> >,
> std::less<hobject_t>, std::allocator<std::pair<const hobject_t, pg_missing_item>
> > > ::const_iterator {aka std::_Rb_tree_const_iterator<std::pair<const hobject_t,
>
> pg_missing_item> >}' changed in GCC 7.1
>     __i = _M_t._M_emplace_hint_unique(__i, std::piecewise_construct,
>     ^~~
> virtual memory exhausted: Operation not permitted
> ...
> make: *** [Makefile:141: all] Error 2
> RPM build errors:
> error: Bad exit status from /var/tmp/rpm-tmp.RgosXb (%build)
>     Bad exit status from /var/tmp/rpm-tmp.RgosXb (%build)
> Child return code was: 1
> ...
>
> See https://koji.fedoraproject.org/koji/taskinfo?taskID=20797264 for full logs.
>
> --
>
> Kaleb

Rats.  Thought I had a clue or a pointer; thanks for trying, though.  When a
colleague double checked my results, we discovered that what I was seeing was
not a change in behavior from fixing the typo, but just a fluke, pure random
chance -- we've probably got a race condition in the build somehow where if
exactly the right combination of compiles tries to occur in parallel, the OOM
killer gets invoked and compilation fails.

You may have to force the %build section to use "make -j1", as well as remove
the use of "-pipe" as you have done.

In the meantime, I'm following up on a g++ bug that may or may not be relevant;
it definitely stresses memory horribly but I need to prove that something in the
compile is actually hitting the bug.  As a workaround, something like this might
help:

diff --git a/ceph.spec b/ceph.spec
index b321a1b..f15171e 100644
--- a/ceph.spec
+++ b/ceph.spec
@@ -811,8 +811,11 @@ cmake .. \
 %endif
     -DBOOST_J=%{_smp_ncpus}

+%ifarch aarch64
+make
+%else
 make %{?_smp_mflags}
-
+%endif

%if 0%{with make_check}
 %check

[Kaleb S. KEITHLEY]

On 07/26/2017 08:22 AM, Björn 'besser82' Esser wrote:
> It looks like the whole parallelized make-process is hitting the 4 
> Gbytes limit per process / task on 32 Bit arches…
>
> Have you tried this?
>
> %build
> export CFLAGS="echo %{optflags} | sed -e 's!-pipe!!g'"
> export CXXFLAGS="$CFLAGS"
> …
>
> Sometimes piping from cpp to gcc to as to ld takes too much memory on 32 
> Bit arches…

Unfortunately that didn't help.

FWIW the actual edits to the ceph.spec that I used (since ceph uses 
cmake and other factors):

...
export RPM_OPT_FLAGS=`echo $RPM_OPT_FLAGS | sed -e 's/i386/i486/' -e 
's/-pipe//g'`

export CPPFLAGS="$java_inc"
export CFLAGS="$RPM_OPT_FLAGS"
export CXXFLAGS="$RPM_OPT_FLAGS"
export LDFLAGS=`echo $LDFLAGS | sed -e 's/-pipe//g'`
...

--

Kaleb