Updating waf to 1.6

[Simo Sorce]

On Mon, 24 Jan 2011 02:49:04 +0100
Kevin Kofler <kevin.kofler at chello.at> wrote:

> Simo Sorce wrote:
> > A build system is not a a shared library, and it is not code that
> > runs on the built system.
> 
> But it can affect the built code, i.e. the one "that runs on the
> built system", in several ways, e.g. it can mishandle our compiler
> flags (ending up e.g. with broken or missing debugging information),
> pollute binaries with rpaths etc.

We can easily test for rpath, as for build flags Makefiles or even
cmake gives you no guarantees, it is easy to override whatever is
passed on the command line, so you always have to check.

> > I would require you to stop making stupid requirements to please so
> > bogus sense of aesthetic you think you have.
> > You may want to have guidelines to help new developers with their
> > projects, but those are nothing more than guidelines. Unless there
> > are solid reasons why a build system/convention or another is
> > good/bad, please stay away from dictating your preferences to
> > upstream.
> > 
> > Avoid sharing libraries have very good reasons in the realm of
> > tracking security issues, better code sharing and smaller system
> > image, so some stronger guidelines on libraries are acceptable.
> > 
> > What are your solid reasons for build systems ?
> > It's a honest question, there may very well be good solid reasons,
> > although even if there are then we will have to proceed in determine
> > how hard it is to comply.
> 
> * Generated (à la autoconf/automake) or copied (à la waf, libtool
> etc.) build system snippets can have bugs, which end up in ALL
> packages which copy the buggy code and which can affect the built
> packages or cause security issues for the people building them. See
> e.g. libtool thinking that /usr/lib64 needs an rpath, or the recent
> automake security vulnerability (CVE-2009-4029) etc. Regenerating the
> generated files resp. getting rid of or replacing the copied ones
> ensures that fixing the issue once will fix it for all packages at
> the same time, exactly as for libraries.

These kind of bugs are quite rare, I see how it could help in some
cases, but build scripts go beyond the common parts even with the
hateful autoconf/automake/libtool trio. You can catch bugs in the
common part but not in the tons of copied stuff (identical in many
projects) that goes in Makefile.am or other auxiliary custom scripts.

> * One of our goals in Fedora is to be fully self-hosting, i.e. to
> build everything from its source code. This also ensures that the
> source code you get is the complete source code, i.e. that you're
> really using Free Software with no hidden hard-to-modify blobs. This
> is the main reason we care about FTBFS bugs as much as we do. Using
> generated files (configure shell blobs, generated makefile blobs
> etc.) as "source code" short-circuits this, and you can end up with
> unreadable generated blobs which you cannot rebuild from source with
> anything in Fedora. (In fact not being able to rebuild the blob with
> the versions of the tools in Fedora is one of the arguments from the
> "build from generated files" camp. But this should never happen, it
> defeats the point of Free Software!) In this case, the true source
> code needs to be fixed to build with what we ship in order to allow
> our users to exercise their rights under the Free Software licenses.

Then you should love waf, as it ships only source code, no generated
code at all. In this it is much better than autoconf/automake/libtool
I guess this is a point in favor then.

> > Even developers in general are not build experts and just *use* the
> > build systems, asking packagers to patch the build system is crazy
> > for anything but trivial projects.
> 
> One of the main problems starts there: Developers use build systems
> without understanding them. (A disease particularly widespread with
> the autotools. Very few people truly understand those.) The result is
> copy&paste programming which just works by accident and often has
> subtle latent bugs which come up when you least expect it. The build
> system language is a programming language which the developer has to
> understand just as well as his primary programming language (be it C,
> C++ etc.).

Again, aside for adoption, then you should like waf more than
autotools, as waf entails understanding only 1 language: python
Autotools require you to know m4, shell, ac macros, etc... much more
difficult to grasp.

> >> Generated shell scripts are NOT source code, we should require the
> >> stuff to be built from its true source code just as we do for Java
> >> JARs, target binaries in cross-compilation tools etc.
> > 
> > I have projects that have the Makefile as the shell script, and
> > that's it, no generation of anything. Are you going to force the
> > packager to build a Makefile.am and patch it in ? Good Luck with
> > that.
> 
> A handwritten shell script is fine. A generated one is not (and
> neither is a copied one).

So you are adding "not a copied one" just to exclude things like
keeping a copy of waf in the tree ?
Even hand written Makefiles are almost always modified copies of pre
existing ones from other projects. You can't escape that.

> The difference between handwritted and
> generated code is very easy to tell, just look at a generated
> configure script and at a handwritted shell script side by side, the
> difference in legibility is flagrant! As for copied code, it is
> easily detected simply by being identical to a file provided by the
> build system package.

So by this I guess you will soon argue that all projects in fedora will
have to be patch to remove autoconf/autotool/libtool and replaced with
something you like more ?

You may not like it, but that's how things are with autotools, not
really useful to whine about it.
 
> The GPL defines source code as the "preferred form for modification".
> So if you write the shell script by hand, it is source code. If you
> write configure.ac and run autoconf on it, the resulting shell script
> is NOT source code, because it's not the form you're editing.

So ?

> Generating that code from the actual source code should be done as
> part of the build process.

Who said that ?
A license is just a legal tool, has no bearing on the discussion on the
merits of how builds are built. At most the GPL can tell you that you
cannot accept source code without configure.ac, that's all.

Let's not start confusing technical issues with legal issues.
I'll consider your last point just hand waiving, smoke and mirrors to
try to add more meat to your argument, that can be briefly summarized
in:

1) I don't like waf
2) In some rare cases it is possible to fix a "potential", (and often
build time only) security issue if part of the build system is common
code in the system.

I haven't seen any other reasonable technical argument.

Simo.



-- 
Simo Sorce * Red Hat, Inc * New York