[WIP] Create kernel-core and kernel-drivers subpackages

[Don Zickus]

On Thu, Mar 20, 2014 at 03:47:19PM -0400, Josh Boyer wrote:
> I believe anaconda does install "kernel" today.  So since the kernel
> metapackage Requires:kernel-drivers, it will bring that in
> automatically.  Updates from existing installs would pick up kernel,
> kernel-core, and kernel-drivers based on the Requires: present.
> 
> It looks like 3 packages instead of one, but the overall installed size
> is literally the same as the existing kernel package today.  It's a
> split to keep things working, while allowing new approaches at the same
> time.
> 
> > So in this case, I suppose cloud only installs kernel-core whereas desktop
> > installs kernel (which installs both -core and -driver)?
> 
> Yes, exactly.  That would be done via kickstart (and is extendable to
> anyone that wants to avoid kernel-drivers but Cloud is the main user.)


Thanks!

> 
> > > > Second, the whole mv .. restore/ and back again dance made me cry.  Can we
> > > > just hack the scripts/Makefile.modinst file to pass a filter argument such
> > > > that the installs look like
> > > > 
> > > > #kernel core
> > > > make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list
> > > > #kernel module extras
> > > > make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
> > > > 
> > > > then you have two directories core/ and suppl/ that you can run depmod on
> > > > individually and for filelist purposes.  Then you can combine it later for
> > > > packaging purposes I think.  Something like that.  Heck you might be able
> > > > to run a depmod check on the filtered files and fail if a dependency isn't
> > > > copied over during install (or just do it automagically to keep
> > > > modules.list smaller).
> > > 
> > > Er... no?  I mean, we could install to core and suppl directories if we
> > > had modules.list.  But if we had modules.list already then we wouldn't
> > > need to generate the %files list, which is what this hunk is doing.
> > > 
> > > Essentially this is avoiding hardcoded, one line per .ko lists by going
> > > through and removing entire directories of things and saving off the
> > > list as the contents of kernel-core and kernel-drivers.  Then we fix it
> > > up since we actually want kernel-drivers to install to
> > > /lib/modules/`uname -r`/kernel/ also.
> > > 
> > > It's probably clearer if you have filter-modules.sh to look at.  I've
> > > attached it below.  Apologies for not including it to begin with.
> > 
> > Ah, yes, now things are clearer.  I had assumed you were listing
> > everything one by one (sorta akin to what is done with the config
> > options).
> > 
> > I could argue you would have better control if you mimic'd the config
> > options and created a modules.list.  But I am not maintaining this, so I
> > won't. :-)
> 
> I might be able to generate a list of modules (.ko files) based on the
> config settings.  That would basically be the equivalent of the first
> find we do in the spec file change.  Not sure it would simplify anything
> else, but I'll think about it a bit and see if there's further
> optimizations to be done.

It was just a different point of view.  Not saying it is any better.  Just
thought it might simplify the spec file changes and control the
kernel-core bloat over a number of releases.

The thought was if you the average desktop user was install kenrel-core
and kernel-driver, then it doesn't matter where the .kos are because in
the end, all the .kos end up on the machine.  So the kernel-core
module.list would really be for cloud.  And someone could keep trimming
that list until the cloud folks had something super small that made them
excited.  If the list was too small it didn't impact the desktop folks
(not that your approach causes any negative impact either).

Again it was just solving a problem from a different perspective.  If you
are happy with your approach continue with it.  I'll have to see how it
works in RHEL in a couple of years. :-)

> 
> > > > Most of this is just moving and renaming so that isn't too bad.  I didn't
> > > > focus on the small subtle changes on the %post stuff.
> > > 
> > > Yeah, that's fairly trivial.
> > > 
> > > > I keep telling myself I would love to rework the spec file to simplify
> > > > again by moving some of the junk into rpm properly.  The number of people
> > > > who can fully understand the spec file is becoming a dying breed. :-(
> > > > 
> > > > So all this churn doesn't make it any better.  Hence why I was looking for
> > > > kernel Makefile solutions or other rpm solutions to simplify the quirks.
> > > 
> > > Except the complexities don't come from the kernel makefiles.  They come
> > > from RPM :\.  It needs a file with a list of files to include for the
> > > %files -f directive of both kernel-core and kernel-drivers to work.  If
> > > we don't use %files -f, we're stuck listing out individual directories
> > > and .ko files in the %files section for each.  That would be a nightmare
> > > to maintain, so I've automated it with %files -f.
> > 
> > I wasn't referring to the %files stuff.  More like all the shell scripting
> > prep work for restore/ and friends.  Just trying to think of ways to
> > simplify things.
> 
> Yeah, I understand.  I tried to keep it as simple as possible and that's
> what I wound up with.  Should have seen it before I realized it was
> easier to just restore things.
> 
> Could move it into a script we call instead of listing it out in the
> spec file ;).

Ugh.  Like rpmmacros and its brethren? I prefer seeing the mess.  :-)

> 
> > > Some of this could be avoided (not a whole lot) if we invented
> > > /lib/modules/`uname -r`/suppl and installed the kernel-drivers content
> > > there, but that's yet-another location for things.  I'm not overly
> > > thrilled with doing that.  All it would really avoid is the restore we
> > > do.
> > > 
> > > > > 2) A per-arch filter list, because the existing one that works on
> > > > > x86_64 leaves modules in kernel-core on ARM that lack their
> > > > > dependencies.  Bad.
> > > > 
> > > > Much like the config files?
> > > 
> > > Eh, not really.  It's a list of directories and individual .ko files to
> > > filter out.  Example filter-x86_64.sh file also below.
> > > 
> > > > > 3) Further testing on all arches.
> > > > > 
> > > > > 4) To make the depmod call fatal to the build.  Without making this
> > > > > fatal we run the risk of pushing kernel-core packages that aren't
> > > > > self-contained and that seems pretty bad to me.  It shouldn't be a huge
> > > > > issue as I would think most of the breakage is going to come from merge
> > > > > window stuff and settle down after that.
> > > > 
> > > > I was trying to envision a 'make depcheck' or something similar to a 'make
> > > > configs' that fails early as opposed to an hour after the build completes.
> > > > But everytime I dive to deep into the Makefile I get lost, try a quick
> > > > hack and make things worse (I still dream of a 'make kabi' like command
> > > > for rhel reasons...).
> > > 
> > > That... wouldn't help here.  The configs are untouched, and Kconfig
> > > should already gripe if you have a config set that doesn't have a dep
> > > set.  You actually need the built modules in the locations they're going
> > > to be installed at in order to run depmod and ensure the required .ko
> > > _files_ are present.  I really don't see a way to avoid building the
> > > drivers here.
> > 
> > I didn't explain myself clearly.  The idea was if you had a list, you
> > could somehow run it through Kconfig (or Makefile) and check that all the
> > deps were on the list and fail at pre-build time with a broken dependency
> > on your module.list.  However, with the way you are doing it, this
> > obviously can't work as you don't know what is on your list until after
> > you build.
> 
> Even ignoring what I'm doing here, I don't see how that works.  You can
> have a pre-made list of drivers, but the depmod check depends on what is
> installed and when.  If you build both foo.ko and bar.ko, but you only
> install foo.ko and not it's dep bar.ko, then you're broken.  Doing it
> before you build and install the drivers doesn't seem feasible.
> 
> It is quite possible we are talking about entirely different kinds of
> checks and I'm just not realizing it.

The thought had nothing to do with the 'depmod' binary.  It was more to do
with the idea that Kconfig knew some deps and perhaps that could be
leveraged in a pre-build way.  But in the end, it is really the
EXPORT_SYMBOLS that create the deps and those are gathered in the .ko
post-build (which is what depmod utilizes anyway).  So my idea might not have
been successful.  Some things sound interesting until you type them out
and realize they don't work.

Cheers,
Don