Rawhide debug options in 3.14 and future

[Josh Boyer]

We've known for a while that the current policy of shipping Rawhide
kernels with debug enabled leads to a pretty big performance hit for a
large number of people.  A while ago Justin, Adam Williamson, Kevin
Fenzi, and I (with the help of others) dug in and figured out that
most of this impact comes from having SLUB debugging turned on and
enabled by default.  We took two measures to help alleviate some of
this.  The first was that we ship the first -rcX build, and each final
kernel release build, with debug disabled.  The second was that Justin
got the Rawhide NoDebug repo up and running.

Both of those steps seemed to help, and the NoDebug repo has been a
success as far as we can tell.  Which is to say that people tell us
they use it, and they complain when it isn't up to date.  However,
even the most ardent Rawhide testers seem to have realized that
booting with slub_debug=- gets them what they need and made it their
default way of running kernels.  Basically, that negates the value of
having it enabled for the majority of the people actually bothering to
run Rawhide kernels in the first place.

After discussing it with the rest of the team and realizing that the
runtime enable/disable function works both ways, we're going to start
disabling SLUB debug by default in Rawhide kernels starting with 3.14
for all kernel builds after the corresponding -rc1 release.  I spent
some time looking through bugzilla and realized that most of the SLUB
debugging stuff that gets caught happens in the merge window kernels
(pre-rc1), so we'll leave it enabled for the merge window to catch the
stuff happening there.  After -rc1 is released, it will remain
disabled by default.

So to summarize:

- SLUB debugging will only be enabled by default on merge window
snapshot kernels

- No other debug options are changing, as we're doing this to address
a specific problem.  This is not a flag day, and things like lockdep
debugging and other options are still very helpful.  That means for
those wanting all debug options disabled, the NoDebug repo is where
you want to go.

Hopefully this makes Rawhide a bit more usable for people, which will
correlate with an uptick in it being used.  If you have any questions,
let us know.

As an aside, Dave brought up that it might be nice to have a script
that we could run to sort of "debugize" a non-debug kernel.  Things
like switching on slub debugging, disabling 'rhgb' and 'quiet' from
the default command line, or any other runtime tunables that we can
think of.  Rather than asking for things piecemeal, we could just have
the user run such a script to help them get us debugging information.
If someone is interested in scripting something like that, please
speak up!

josh