glibc: Do not use parallel make for building locales (#1652228)

[Florian Weimer]

I would like to see a quick review for this.

I think have to do this, which is unfortunate.  It increases build time
by about ten minutes on x86-64 (with lots of CPU cores).

Proposed patch:

diff --git a/glibc.spec b/glibc.spec
index 73ecd93..54c96b2 100644
--- a/glibc.spec
+++ b/glibc.spec
@@ -908,7 +908,9 @@ make -j1 install_root=%{glibc_sysroot} install -C build-%{target}
 # locales.
 %ifnarch %{auxarches}
 pushd build-%{target}
-make %{?_smp_mflags} -O install_root=%{glibc_sysroot} \
+# Do not use a parallel make here because the hardlink optimization in
+# localedef is not fully reproducible when running concurrently.
+make install_root=%{glibc_sysroot} \
 	install-locales -C ../localedata objdir=`pwd`
 popd
 %endif

I think this could also explain the x86_64/i686 multilib differences for
the locale archive that we saw downstream occasionally.

Thanks,
Florian

[Carlos O'Donell]

On 11/21/18 1:12 PM, Florian Weimer wrote:
>  I would like to see a quick review for this.
>  
>  I think have to do this, which is unfortunate.  It increases build time
>  by about ten minutes on x86-64 (with lots of CPU cores).
>  
>  Proposed patch:
>  
>  diff --git a/glibc.spec b/glibc.spec
>  index 73ecd93..54c96b2 100644
>  --- a/glibc.spec
>  +++ b/glibc.spec
>  @@ -908,7 +908,9 @@ make -j1 install_root=%{glibc_sysroot} install -C build-%{target}
>   # locales.
>   %ifnarch %{auxarches}
>   pushd build-%{target}
>  -make %{?_smp_mflags} -O install_root=%{glibc_sysroot} \
>  +# Do not use a parallel make here because the hardlink optimization in
>  +# localedef is not fully reproducible when running concurrently.
>  +make install_root=%{glibc_sysroot} \
>   	install-locales -C ../localedata objdir=`pwd`
>   popd
>   %endif
>  
>  I think this could also explain the x86_64/i686 multilib differences for
>  the locale archive that we saw downstream occasionally. 
This is fine with me.

My *preference* would be to do a parallel install, followed by hardlink
de-dup pass that we write and execute as part of the build. I think those
two things can use multiple threads and save on the time.

-- 
Cheers,
Carlos.

[Florian Weimer]

* Carlos O'Donell:

>  On 11/21/18 1:12 PM, Florian Weimer wrote:
> > I would like to see a quick review for this.
> > 
> > I think have to do this, which is unfortunate.  It increases build time
> > by about ten minutes on x86-64 (with lots of CPU cores).
> > 
> > Proposed patch:
> > 
> > diff --git a/glibc.spec b/glibc.spec
> > index 73ecd93..54c96b2 100644
> > --- a/glibc.spec
> > +++ b/glibc.spec
> > @@ -908,7 +908,9 @@ make -j1 install_root=%{glibc_sysroot} install -C
> build-%{target}
> >  # locales.
> >  %ifnarch %{auxarches}
> >  pushd build-%{target}
> > -make %{?_smp_mflags} -O install_root=%{glibc_sysroot} \
> > +# Do not use a parallel make here because the hardlink optimization in
> > +# localedef is not fully reproducible when running concurrently.
> > +make install_root=%{glibc_sysroot} \
> >  	install-locales -C ../localedata objdir=`pwd`
> >  popd
> >  %endif
> > 
> > I think this could also explain the x86_64/i686 multilib differences for
> > the locale archive that we saw downstream occasionally.
>
>  This is fine with me.
>
>  My *preference* would be to do a parallel install, followed by hardlink
>  de-dup pass that we write and execute as part of the build. I think those
>  two things can use multiple threads and save on the time. 
Right, but that needs some sort of --no-hard-link option for localedef,
I think.

Thanks,
Florian

[Carlos O'Donell]

On 11/26/18 8:54 AM, Florian Weimer wrote:
>  * Carlos O'Donell:
>  
> > On 11/21/18 1:12 PM, Florian Weimer wrote:
> >> I would like to see a quick review for this.
> >>
> >> I think have to do this, which is unfortunate.  It increases build time
> >> by about ten minutes on x86-64 (with lots of CPU cores).
> >>
> >> Proposed patch:
> >>
> >> diff --git a/glibc.spec b/glibc.spec
> >> index 73ecd93..54c96b2 100644
> >> --- a/glibc.spec
> >> +++ b/glibc.spec
> >> @@ -908,7 +908,9 @@ make -j1 install_root=%{glibc_sysroot} install -C
> build-%{target}
> >>  # locales.
> >>  %ifnarch %{auxarches}
> >>  pushd build-%{target}
> >> -make %{?_smp_mflags} -O install_root=%{glibc_sysroot} \
> >> +# Do not use a parallel make here because the hardlink optimization in
> >> +# localedef is not fully reproducible when running concurrently.
> >> +make install_root=%{glibc_sysroot} \
> >>  	install-locales -C ../localedata objdir=`pwd`
> >>  popd
> >>  %endif
> >>
> >> I think this could also explain the x86_64/i686 multilib differences for
> >> the locale archive that we saw downstream occasionally.
> >
> > This is fine with me.
> >
> > My *preference* would be to do a parallel install, followed by hardlink
> > de-dup pass that we write and execute as part of the build. I think those
> > two things can use multiple threads and save on the time.
>  
>  Right, but that needs some sort of --no-hard-link option for localedef,
>  I think. 
For the record I proposed a patch for that upstream:
https://www.sourceware.org/ml/libc-alpha/2018-11/msg00681.html

Once we have this option we can then use --no-hard-links to avoid any
hard link usage, and then have a deterministic build result that we can
post-process with hardlink?

-- 
Cheers,
Carlos.