Please test: patch that makes the build script work in parallel, improves build time

[Elad]

These patches[1][2] makes it possible to run our build script with the
-j parameter, which makes it use more then one process and thus
improves build time on multi-core systems.
to test, run make -j2 make -j3 or make -j4 (the number is the number
of allowed jobs, shouldn't be higher then your number of cores). don't
run it with -j without a number, because it will run as many jobs as
it can, resulting your machine to become un-responsive until the build
is complete (that's was the result on my machine, might differ on
others).
Also it won't affect if you are building single language, it only
improves speed if you build two or more.

The second patch fixes a bug that Sijis discovered yesterday with the
first patch, related to RSS caching.
Don't forget to run make clean before you test.

I won't push it before it gets tested properly. One of the reason it
requires a lot of testing is that I don't know python and this is the
first time I'm editing Makefiles, so I probably did some mistakes.

During my attempts to fix that bug, I discovered that there is a
forgotten test file, fedoraproject.org/data/content/rss.html that
pulls the redhat magazine feed, which takes long to download (longer
than the fedora feeds). As we don't really use this feed, I suggest
deleting this file for more speed improvements.

[1]
http://elad.fedorapeople.org/0001-Buildscript-can-now-work-parallel-impro...
[2] http://elad.fedorapeople.org/0002-Fix-RSS-caching-bug-in-parallel.patch
-- 
-Elad.

[Kévin Raymond]

On Fri, Apr 29, 2011 at 10:03 AM, Elad <el.il(a)doom.co.il&gt; wrote:
>  These patches[1][2] makes it possible to run our build script with
> the
>  -j parameter, which makes it use more then one process and thus
>  improves build time on multi-core systems.
>  to test, run make -j2 make -j3 or make -j4 (the number is the number
>  of allowed jobs, shouldn't be higher then your number of cores). don't
>  run it with -j without a number, because it will run as many jobs as
>  it can, resulting your machine to become un-responsive until the build
>  is complete (that's was the result on my machine, might differ on
>  others).
>  Also it won't affect if you are building single language, it only
>  improves speed if you build two or more.
>
>  The second patch fixes a bug that Sijis discovered yesterday with the
>  first patch, related to RSS caching.
>  Don't forget to run make clean before you test.
>
>  I won't push it before it gets tested properly. One of the reason it
>  requires a lot of testing is that I don't know python and this is the
>  first time I'm editing Makefiles, so I probably did some mistakes.
>
>  During my attempts to fix that bug, I discovered that there is a
>  forgotten test file, fedoraproject.org/data/content/rss.html that
>  pulls the redhat magazine feed, which takes long to download (longer
>  than the fedora feeds). As we don't really use this feed, I suggest
>  deleting this file for more speed improvements.
>
>  [1]
> http://elad.fedorapeople.org/0001-Buildscript-can-now-work-parallel-impro...
>  [2] http://elad.fedorapeople.org/0002-Fix-RSS-caching-bug-in-parallel.patch
>  --
>  -Elad.
>  --
>  websites mailing list
>  websites(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/websites
>  
Hi guys,

Thanks Elad for improving the build time.
Applying both patches works great for me (in the master branch...):

make clean && make all
real	3m6.863s

make clean && make all
real 0m54.836s
(AMD PhenomII X4 955)

+1 to apply

+1 to remove the data/content/rss.html RH feed parsing.

msgfmt in concurrency also would be nice (just to know that our build
process is perfect:) )


-- 
Kévin Raymond
User:shaiton
GPG-Key: A5BCB3A2

[Kévin Raymond]

On Fri, Apr 29, 2011 at 11:04 AM, Kévin Raymond
<shaiton(a)fedoraproject.org&gt; wrote:
>  make clean && make all
>  real    3m6.863s
>
>  make clean && make all 
Oops,
make clean && make -j4 all
of course

[Elad]

On Fri, Apr 29, 2011 at 11:03 AM, Elad <el.il(a)doom.co.il&gt; wrote:
>  These patches[1][2] makes it possible to run our build script with
> the
>  -j parameter, which makes it use more then one process and thus
>  improves build time on multi-core systems.
>  to test, run make -j2 make -j3 or make -j4 (the number is the number
>  of allowed jobs, shouldn't be higher then your number of cores). don't
>  run it with -j without a number, because it will run as many jobs as
>  it can, resulting your machine to become un-responsive until the build
>  is complete (that's was the result on my machine, might differ on
>  others).
>  Also it won't affect if you are building single language, it only
>  improves speed if you build two or more.
>
>  The second patch fixes a bug that Sijis discovered yesterday with the
>  first patch, related to RSS caching.
>  Don't forget to run make clean before you test.
>
>  I won't push it before it gets tested properly. One of the reason it
>  requires a lot of testing is that I don't know python and this is the
>  first time I'm editing Makefiles, so I probably did some mistakes.
>
>  During my attempts to fix that bug, I discovered that there is a
>  forgotten test file, fedoraproject.org/data/content/rss.html that
>  pulls the redhat magazine feed, which takes long to download (longer
>  than the fedora feeds). As we don't really use this feed, I suggest
>  deleting this file for more speed improvements.
>
>  [1]
> http://elad.fedorapeople.org/0001-Buildscript-can-now-work-parallel-impro...
>  [2] http://elad.fedorapeople.org/0002-Fix-RSS-caching-bug-in-parallel.patch
>  --
>  -Elad.
>  
Okay, I've finished fixing all build scripts for this, 10 commits total.
Tarball for easy downloading:
http://elad.fedorapeople.org/fedora-web-prallel-patches.tar.xz
A folder in my fedorapeople space, contains all patches for easy
reading: http://elad.fedorapeople.org/parallel/

Please test.
-- 
-Elad.