revisit: turning some of the "always used" modules to built-in

[Arjan van de Ven]

A few weeks ago the discussion about considering making some of the things
in the fedora kernel not-a-module came up, but then died out. I'd like
to revisit that discussion with a very specific proposal of things
to turn into built-in.

For me, there are 3 key reasons to make certain things not-a-module:
1) Built-in code is easier to debug/diagnose. This may sound weird, but
   really, things inside the vmlinux allow for much better automated
   bug diagnostics/analysis.
   ( and the www.kerneloops.org site uses these, but can only do the more
     advanced automatic analysis on the built-in oopses)
2) Built-in code tends to be smaller and faster; there is a space and TLB
   overhead associated with modules (several datastructures get rounded up
   to 4K boundaries, and modules are vmalloc'd which has TLB overhead)
3) A system with key functionality not-in-the-vmlinux is more fragile
   if something goes wrong 

With this in mind, I came up with the list below, I've split it into 4
categories.

I would like to ask feedback on this list, with the goal of getting
these changes made to fedora 10 asap to then get a wider audience input
to see if anything breaks.


Category 1: Always loaded anyway 
--------------------------------
Rationale: Since we load these always anyway, why bother making it modules
- ata_generic, pata_acpi
- libata
- sg, sd_mod, scsi_mod
- ext3, jbd, mbcache


Category 2: Always loaded in default install
--------------------------------------------
Rationale: yes you can turn off your firewall.. but nobody should do that.
           The others are default-loaded as well
- ip_tables, iptable_filter
- ip6_tables, ip6table_filter
- dm_mod
- ipv6
- battery, ac, button, video, output

Category 3: popular/very common and makes the system more robust
----------------------------------------------------------------
Rationale: having these built in makes the system more robust, also
           in case of failure
- ahci (default storage for all new systems; means that there the system 
        always has the / device driver)
- ehci_hcd (means you have a USB keyboard early)
- cpufreq_ondemand (means the cpu can slow down for power/thermal)
- acpi_cpufreq (means the cpu can slow down for power/thermal)

Category 4: VERY popular 
------------------------
Rationale: pretty much always loaded in default installs
- snd_seq_dummy, snd_seq, snd_seq_device, 
  snd_pcm, snd_timer, snd_page_alloc, snd

[Roland McGrath]

>  1) Built-in code is easier to debug/diagnose. This may sound weird,
> but
>     really, things inside the vmlinux allow for much better automated
>     bug diagnostics/analysis.
>     ( and the www.kerneloops.org site uses these, but can only do the more
>       advanced automatic analysis on the built-in oopses) 
Can you point me to the details of this issue?

Thanks,
Roland

[Eric Sandeen]

Arjan van de Ven wrote:

>  I would like to ask feedback on this list, with the goal of getting
>  these changes made to fedora 10 asap to then get a wider audience input
>  to see if anything breaks.
>  
>  
>  Category 1: Always loaded anyway 
>  --------------------------------
>  Rationale: Since we load these always anyway, why bother making it modules
>  - ata_generic, pata_acpi
>  - libata
>  - sg, sd_mod, scsi_mod
>  - ext3, jbd, mbcache 
One minor thing, when I'm debugging ext3 in a fedora kernel it's awfully
nice to boot on a non-ext3 root and then be able to load/unload test
ext3 modules.  That's a little whiny of me; of course I can just rebuild
the kernel with ext3 modular for testing but sometimes it's a very quick
way to be able to troubleshoot on a particular kernel if necessary.

As an even more minor thing not *everyone* uses ext3, but I guess 99.9%
of the Fedora universe is close enough.  :)

-Eric

[Arjan van de Ven]

On Sun, 22 Jun 2008 15:57:21 -0500
Eric Sandeen <sandeen(a)redhat.com&gt; wrote:

>  Arjan van de Ven wrote:
>  
>  > I would like to ask feedback on this list, with the goal of getting
>  > these changes made to fedora 10 asap to then get a wider audience
>  > input to see if anything breaks.
>  > 
>  > 
>  > Category 1: Always loaded anyway 
>  > --------------------------------
>  > Rationale: Since we load these always anyway, why bother making it
>  > modules
>  > - ata_generic, pata_acpi
>  > - libata
>  > - sg, sd_mod, scsi_mod
>  > - ext3, jbd, mbcache
>  
>  One minor thing, when I'm debugging ext3 in a fedora kernel it's
>  awfully nice to boot on a non-ext3 root and then be able to
>  load/unload test ext3 modules.  That's a little whiny of me; of
>  course I can just rebuild the kernel with ext3 modular for testing
>  but sometimes it's a very quick way to be able to troubleshoot on a
>  particular kernel if necessary. 
I understand your point. the contrast to this is the easy of debugging
(being able to turn oopses into source+asm crash views etcc) and the
reliability thing of being able to still boot and mount / when you end
up with a b0rked initrd etc.
>  
>  As an even more minor thing not *everyone* uses ext3, but I guess
>  99.9% of the Fedora universe is close enough.  :) 
those who use xfs and the like often have a /boot that still is ext3 ;)
but yeah even the 99.9% is enough..

-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Matt Domsch]

There is likely a difference between what should be built-in in Fedora
vs what should be in RHEL.  For RHEL, I really like the flexibility
that having something be modular allows.  I have, on many occasions,
needed to replace modules using DKMS, including libata, sd_mod,
scsi_mod, ahci, etc.  Basically anything that touches hardware
directly or one layer up.  Not often, but enough that I like
modularity for this possibility.  It has meant the difference between
being able to ship hardware, or not for another 6 months until new CDs
are spun...

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

[Arjan van de Ven]

On Sun, 22 Jun 2008 18:04:46 -0500
Matt Domsch <Matt_Domsch(a)dell.com&gt; wrote:

>  There is likely a difference between what should be built-in in
> Fedora
>  vs what should be in RHEL.  For RHEL, I really like the flexibility
>  that having something be modular allows.  I have, on many occasions,
>  needed to replace modules using DKMS, including libata, sd_mod,
>  scsi_mod, ahci, etc.  Basically anything that touches hardware
>  directly or one layer up.  Not often, but enough that I like
>  modularity for this possibility.  It has meant the difference between
>  being able to ship hardware, or not for another 6 months until new CDs
>  are spun... 
yeah I sent it to the Fedora list, not the RHEL list ;)

I entirely assume that RHEL has it's own config options (SLUB instead
of the slower-for-database SLUB, support for 4096 CPUs etc etc)...

I'd hate to see a less optimal fedora for something rhel-ish like this
that's different anyway.

>   

-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Matthew Garrett]

On Sun, Jun 22, 2008 at 11:42:18AM -0700, Arjan van de Ven wrote:
>  Rationale: Since we load these always anyway, why bother making it
> modules
>  - ata_generic, pata_acpi 
Won't this cause ata_generic or pata_acpi to claim the devices before a 
hardware specific driver can be loaded?

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Arjan van de Ven]

On Mon, 23 Jun 2008 12:07:34 +0100
Matthew Garrett <mjg(a)redhat.com&gt; wrote:

>  On Sun, Jun 22, 2008 at 11:42:18AM -0700, Arjan van de Ven wrote:
>  > Rationale: Since we load these always anyway, why bother making it
>  > modules
>  > - ata_generic, pata_acpi
>  
>  Won't this cause ata_generic 
not without a specific kernel command line option

>  or pata_acpi to claim the devices before
>  a hardware specific driver can be loaded? 
if this one does we need to add a command
line option for it too


>   

-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Joshua Baker-LePain]

On Sun, 22 Jun 2008 at 11:42am, Arjan van de Ven wrote

>  Category 3: popular/very common and makes the system more robust
>  ----------------------------------------------------------------
>  Rationale: having these built in makes the system more robust, also
>            in case of failure
>  - ahci (default storage for all new systems; means that there the system
>         always has the / device driver) 
This isn't the default for all new systems -- lots of folks boot off 
SCSI/RAID.

>  - cpufreq_ondemand (means the cpu can slow down for power/thermal)
>  - acpi_cpufreq (means the cpu can slow down for power/thermal) 
Not everybody wants this -- think HPC.

>  Category 4: VERY popular
>  ------------------------
>  Rationale: pretty much always loaded in default installs
>  - snd_seq_dummy, snd_seq, snd_seq_device,
>   snd_pcm, snd_timer, snd_page_alloc, snd 
Completely useless and unwanted on servers/HPC nodes.

It seems that there's an assumption here that Fedora = desktop.  While 
there's obviously a lot of that, I've seen it on a fair number of servers 
and it is a great fit for HPC nodes.

-- 
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF

[Arjan van de Ven]

On Mon, 23 Jun 2008 10:48:43 -0400 (EDT)
Joshua Baker-LePain <jlb17(a)duke.edu&gt; wrote:

>  On Sun, 22 Jun 2008 at 11:42am, Arjan van de Ven wrote
>  
>  > Category 3: popular/very common and makes the system more robust
>  > ----------------------------------------------------------------
>  > Rationale: having these built in makes the system more robust, also
>  >           in case of failure
>  > - ahci (default storage for all new systems; means that there the
>  > system always has the / device driver)
>  
>  This isn't the default for all new systems -- lots of folks boot off 
>  SCSI/RAID. 
... and then have the cdrom hang of ahci.

>  
>  > - cpufreq_ondemand (means the cpu can slow down for power/thermal)
>  > - acpi_cpufreq (means the cpu can slow down for power/thermal)
>  
>  Not everybody wants this -- think HPC. 
not everyone wants the policy on by default.. that's a whole different
discussion. But even HPC wants thermal protection etc enabled.

>  
>  > Category 4: VERY popular
>  > ------------------------
>  > Rationale: pretty much always loaded in default installs
>  > - snd_seq_dummy, snd_seq, snd_seq_device,
>  >  snd_pcm, snd_timer, snd_page_alloc, snd
>  
>  Completely useless and unwanted on servers/HPC nodes. 
yet tends to get loaded ;)

>  
>  It seems that there's an assumption here that Fedora = desktop.
>  While there's obviously a lot of that, I've seen it on a fair number
>  of servers and it is a great fit for HPC nodes. 
No that wasn't my assumption actually. My assumption was that those
things are either always loaded, extremely likely loaded by default or
for robustness (say AHCI). I realize it won't hit 100% but... the worst
case downside is a tiny bit of wasted memory. Yawn.
With the upsides I mentioned in my mail.. for me it's worth the
tradeoff for at least almost all of these.

>   


-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Chris Ricker]

On Sun, 22 Jun 2008, Arjan van de Ven wrote:

>  Category 2: Always loaded in default install
>  --------------------------------------------
>  Rationale: yes you can turn off your firewall.. but nobody should do that.
>             The others are default-loaded as well
>  - ip_tables, iptable_filter
>  - ip6_tables, ip6table_filter
>  - dm_mod
>  - ipv6
>  - battery, ac, button, video, output 
I still block loading of ipv6 and friends on lots of machines -- typical 
reasons are "memory's limited and IPv6 will never be used" or "IPv6
will 
never be used so remove the attack surface"

later,
chris

[Joshua Baker-LePain]

On Mon, 23 Jun 2008 at 8:20am, Arjan van de Ven wrote

>  On Mon, 23 Jun 2008 10:48:43 -0400 (EDT)
>  Joshua Baker-LePain <jlb17(a)duke.edu&gt; wrote:
>
> > On Sun, 22 Jun 2008 at 11:42am, Arjan van de Ven wrote
> >
> >> Category 3: popular/very common and makes the system more robust
> >> ----------------------------------------------------------------
> >> Rationale: having these built in makes the system more robust, also
> >>           in case of failure
> >> - ahci (default storage for all new systems; means that there the
> >> system always has the / device driver)
> >
> > This isn't the default for all new systems -- lots of folks boot off
> > SCSI/RAID.
>
>  ... and then have the cdrom hang of ahci. 
What cdrom?  ;)

> >> Category 4: VERY popular
> >> ------------------------
> >> Rationale: pretty much always loaded in default installs
> >> - snd_seq_dummy, snd_seq, snd_seq_device,
> >>  snd_pcm, snd_timer, snd_page_alloc, snd
> >
> > Completely useless and unwanted on servers/HPC nodes.
>
>  yet tends to get loaded ;) 
Not on my systems!

> > It seems that there's an assumption here that Fedora =
> desktop.
> > While there's obviously a lot of that, I've seen it on a fair number
> > of servers and it is a great fit for HPC nodes.
>
>  No that wasn't my assumption actually. My assumption was that those
>  things are either always loaded, extremely likely loaded by default or
>  for robustness (say AHCI). I realize it won't hit 100% but... the worst
>  case downside is a tiny bit of wasted memory. Yawn.
>  With the upsides I mentioned in my mail.. for me it's worth the
>  tradeoff for at least almost all of these. 
I can definitely see that point of view -- just throwing a few things out 
there.

Thanks.

-- 
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF

[Dave Jones]

On Sun, Jun 22, 2008 at 11:42:18AM -0700, Arjan van de Ven wrote:

>  Category 1: Always loaded anyway 
>  --------------------------------
>  Rationale: Since we load these always anyway, why bother making it modules
>  - ata_generic, pata_acpi 
These ones make me hrmm a bit. I'd like to know that our initrd isn't
going to do something silly before we change these. Peter ?

>  - libata 
Sure. Done in CVS.

>  - sg, sd_mod 
Ditto.

>  , scsi_mod 
This one is tricky. It seems to become modular because it's dependant upon
the bool SCSI_NETLINK. A bunch of other scsi modules enable SCSI_NETLINK,
like the fibrechannel stuff.  I'm not convinced we want to build all that
stuff in.

>  - ext3, jbd, mbcache 
I'm torn on these. Mostly for the reasons both Eric and Matt suggested.
The flipside being that Eric knows how to build his own kernels, and can do so,
but at the same time, if it means he spends less time fixing ext bugs each
day and more time staring at compiler output, I'm not sure it's a net win.
Matts argument, whilst RHEL specific, does have an element of validity
for Fedora too.  We always push the 'Fedora isn't a test vehicle for RHEL'
angle, but at the same time, there's no denying that doing something completely
different does take away a certain amount of 'in the field' testing.
(In this case, I'm primarily thinking about the knock-on effects changes
like this have on the kernel periphery packages like mkinitrd/anaconda
which now have to support two separate cases).
 
>  Category 2: Always loaded in default install
>  --------------------------------------------
>  Rationale: yes you can turn off your firewall.. but nobody should do that.
>             The others are default-loaded as well
>  - ip_tables, iptable_filter
>  - ip6_tables, ip6table_filter 
Hm. Could be swayed either way on these ones.

>  - dm_mod 
Yeah, I guess.

>  - ipv6 
A lot of people alias this to off if they don't use it. I think we'd get
quite a few complaints if we broke that.

>  - battery, ac, button, video, output 
Some of these are a bit crap in the case of 'the hardware/bios doesnt support
this'.
Ie, ac will register proc entries even if there aren't any ACPI objects to
register underneath it.  Somewhat benign, but on the whole, not really a problem
per-se. I suppose we could flip these on.

>  - ahci (default storage for all new systems; means that there the
> system 
>          always has the / device driver) 
Same thing as above. The mkinitrd logic surrounding ordering of storage
modules is fragile at best..

>  - ehci_hcd (means you have a USB keyboard early) 
ISTR there was some issue with this. I'm sure we even tried it once
in the FC2 era.  Lets give it a shot, and see what happens.

>  - cpufreq_ondemand (means the cpu can slow down for power/thermal)
>  - acpi_cpufreq (means the cpu can slow down for power/thermal) 
On Intel this is a good idea. On other CPUs, acpi-cpufreq is considered
a fallback 'last resort' driver. 
 
>  Rationale: pretty much always loaded in default installs
>  - snd_seq_dummy, snd_seq, snd_seq_device, 
>    snd_pcm, snd_timer, snd_page_alloc, snd 
 
The only concern here seems to be the one of memory bloat.
ALSA isn't the smallest part of the kernel, and if some folks are
currently aliasing that off in their modules.conf, we'll probably
upset them with such a change.

	Dave

-- 
http://www.codemonkey.org.uk

[Peter Jones]

Dave Jones wrote:
>  On Sun, Jun 22, 2008 at 11:42:18AM -0700, Arjan van de Ven wrote:
>  
>   > Category 1: Always loaded anyway 
>   > --------------------------------
>   > Rationale: Since we load these always anyway, why bother making it modules
>   > - ata_generic, pata_acpi
>  
>  These ones make me hrmm a bit. I'd like to know that our initrd isn't
>  going to do something silly before we change these. Peter ? 
We probably need to do a little bit of mkinitrd work to make sure this 
works, but not much.

That being said, don't we need to install e.g. ahci _before_ ata_generic 
on most systems to get the right behavior?  Also, how does ata_generic 
interact with ata_piix?  I know changing the install order of ahci vs 
ata_piix causes major flux in behavior...

>   > - libata
>  
>  Sure. Done in CVS.
>  
>   > - sg, sd_mod
>  
>  Ditto. 
sr_mod and cdrom?  They're not strictly required, but it's really, 
really common.  It's 60k of bloat, which isn't great, but I think it 
also makes booting incrementally quicker (because hal won't need to load 
the modules.)

>   > , scsi_mod
>  
>  This one is tricky. It seems to become modular because it's dependant upon
>  the bool SCSI_NETLINK. A bunch of other scsi modules enable SCSI_NETLINK,
>  like the fibrechannel stuff.  I'm not convinced we want to build all that
>  stuff in. 
Then we need to make sure that there's a dep chain that shows up with 
"modprobe --show-depends".

>   > - ext3, jbd, mbcache
>  
>  I'm torn on these. Mostly for the reasons both Eric and Matt suggested.
>  The flipside being that Eric knows how to build his own kernels, and can do so,
>  but at the same time, if it means he spends less time fixing ext bugs each
>  day and more time staring at compiler output, I'm not sure it's a net win.
>  Matts argument, whilst RHEL specific, does have an element of validity
>  for Fedora too.  We always push the 'Fedora isn't a test vehicle for RHEL'
>  angle, but at the same time, there's no denying that doing something completely
>  different does take away a certain amount of 'in the field' testing.
>  (In this case, I'm primarily thinking about the knock-on effects changes
>  like this have on the kernel periphery packages like mkinitrd/anaconda
>  which now have to support two separate cases). 
Actually, I don't think these three change mkinitrd/anaconda much at 
all; we always pull these three modules in as dependencies, never manually.

>   > Category 2: Always loaded in default install 
[...]

>   > - dm_mod
>  
>  Yeah, I guess. 
This will require minor changes to mkinitrd, but I'm for it.  Also, 
dm_snapshot, since we also always load it (there's no way to tell if 
it's necessary ahead of time :/ )

Also, though not strictly _required_,  dm_zero, and dm_mirror would be 
nice -- you always get them loaded, and there aren't any module  deps on 
them.  (Though arguably this is a deficiency in lvm2)

>   > - ahci (default storage for all new systems; means that there
> the system 
>   >         always has the / device driver)
>  
>  Same thing as above. The mkinitrd logic surrounding ordering of storage
>  modules is fragile at best.. 
I think this will break some ICH[789] boxes, won't it?  I've seen this 
cause ata_piix badly misbehave if this is already loaded when it loads 
on systems configured in "legacy" mode in the BIOS (which is often not 
exposed as an option to the user AT ALL)

>   > - ehci_hcd (means you have a USB keyboard early)
>  
>  ISTR there was some issue with this. I'm sure we even tried it once
>  in the FC2 era.  Lets give it a shot, and see what happens. 
I think this should work these days... if not, we should really just sit 
down and fix it.  There's some mkinitrd work here, but it's mostly just 
code removal, and you should get a working initrd even before we do it.

Also this is required to do usb-serial console (though that's still 
pretty broken itself ATM), which could eventually be really useful for 
debugging.

I think we also want uhci_hcd here; if you've got EHCI you'll wind up 
loading it anyway, and if you don't, then you'll probably want it loaded.

-- 
   Peter

[Arjan van de Ven]

On Tue, 24 Jun 2008 12:09:32 -0400
Peter Jones <pjones(a)redhat.com&gt; wrote:

>  That being said, don't we need to install e.g. ahci _before_
>  ata_generic on most systems to get the right behavior?  Also, how
>  does ata_generic interact with ata_piix?  
ata_generic is inactive unless you force it on.


-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Arjan van de Ven]

On Sun, 22 Jun 2008 11:44:31 -0700 (PDT)
Roland McGrath <roland(a)redhat.com&gt; wrote:

>  > 1) Built-in code is easier to debug/diagnose. This may sound
> weird,
>  > but really, things inside the vmlinux allow for much better
>  > automated bug diagnostics/analysis.
>  >    ( and the www.kerneloops.org site uses these, but can only do
>  > the more advanced automatic analysis on the built-in oopses)
>  
>  Can you point me to the details of this issue?
>   
very simply put:

it's between getting this level of information:

http://www.kerneloops.org/raw.php?rawid=32356

with pointing at the exact code, and only getting this

http://www.kerneloops.org/raw.php?rawid=32287



-- 
If you want to reach me at my work email, use arjan(a)linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[Roland McGrath]

>  very simply put:
>  
>  it's between getting this level of information:
>  
>  http://www.kerneloops.org/raw.php?rawid=32356
>  
>  with pointing at the exact code, and only getting this
>  
>  http://www.kerneloops.org/raw.php?rawid=32287 
Thanks.  I'd like to help improve the tools so that we can close this gap
in the future.  (I'm really just concerned with making some tools better.
I am not saying anything about the desireability of building in modules.)
I think one of my back-burner items would cover this.  Maybe we can kibitz
offline about the details.  In brief poking I didn't find the code you use
to generate the web reports.


Thanks,
Roland