kmod changes

[Yauheni Kaliuta]

Hi!

I should have sent the proposals first here, but forgot about the
list, sorry.

I've submitted the version update  (25 -> 26) already.

The next step will be enabling openssl for PKCS#7 reading.

Then I would like to put stub instead of weak-modules script to
avoid confusion (there is no point to keep the full RHEL version
in Fedora).

I hope, it is fine with everybody.

[Josh Boyer]

On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
 wrote:
> Hi!
>
> I should have sent the proposals first here, but forgot about the
> list, sorry.
>
> I've submitted the version update  (25 -> 26) already.

Great!
> The next step will be enabling openssl for PKCS#7 reading.

Yes please.
> Then I would like to put stub instead of weak-modules script to
> avoid confusion (there is no point to keep the full RHEL version
> in Fedora).

I don't agree with this part...  Why wouldn't we keep it there?  I
understand Fedora doesn't support weak-modules, but there is nothing
wrong with the script living upstream in Fedora and I see no
disadvantage to keeping the RHEL version in-sync.  It's less work when
we have to pull kmod back into a RHEL release in the future.

josh

[Yauheni Kaliuta]

Hi, Josh!
> > > > > On Mon, 25 Feb 2019 07:50:08 -0500, Josh Boyer  wrote:
> On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
>  wrote:
> > Hi!
> >
> > I should have sent the proposals first here, but forgot about the
> > list, sorry.
> >
> > I've submitted the version update  (25 -> 26) already.
> Great!
> > The next step will be enabling openssl for PKCS#7 reading.
> Yes please.
> > Then I would like to put stub instead of weak-modules script to
> > avoid confusion (there is no point to keep the full RHEL version
> > in Fedora).
> I don't agree with this part...  Why wouldn't we keep it
> there?  I understand Fedora doesn't support weak-modules, but
> there is nothing wrong with the script living upstream in
> Fedora and I see no disadvantage to keeping the RHEL version
> in-sync.  It's less work when we have to pull kmod back into a
> RHEL release in the future.

It's ok for me. I just did not see a lot of advantage. Leaving
the outdated version (like now) is definitely the worst.

[Yauheni Kaliuta]

Hi, Josh!
> > > > > On Mon, 25 Feb 2019 07:50:08 -0500, Josh Boyer  wrote:
>
> On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
>  wrote:
> > The next step will be enabling openssl for PKCS#7 reading.
> Yes please.

Here is the spec diff (already in master):

commit 434a6d5d65bc6f70a6557fa9e4b131c91ee37fa0
Author: Yauheni Kaliuta 
Date:   Sun Feb 24 19:48:49 2019 +0200

add PKCS7/openssl support (rhbz 1320921)

Signed-off-by: Yauheni Kaliuta

diff --git a/kmod.spec b/kmod.spec
index ec3d436f7f65..5c6d4de22a34 100644
--- a/kmod.spec
+++ b/kmod.spec
@@ -1,6 +1,6 @@
 Name:		kmod
 Version:	26
-Release:	1%{?dist}
+Release:	2%{?dist}
 Summary:	Linux kernel module management utilities

License:	GPLv2+
@@ -14,6 +14,7 @@ BuildRequires:	chrpath
 BuildRequires:	zlib-devel
 BuildRequires:	xz-devel
 BuildRequires:  libxslt
+BuildRequires:  openssl-devel

Provides:	module-init-tools = 4.0-1
 Obsoletes:	module-init-tools < 4.0-1
@@ -47,6 +48,7 @@ applications that wish to load or unload Linux kernel modules.
 %build
 export V=1
 %configure \
+  --with-openssl \
   --with-zlib \
   --with-xz
 make %{?_smp_mflags}
@@ -101,6 +103,9 @@ install -m 755 %{SOURCE1} $RPM_BUILD_ROOT%{_sbindir}/weak-modules
 %{_libdir}/libkmod.so

%changelog
+* Sun Feb 24 2019 Yauheni Kaliuta  - 26-2
+- add PKCS7/openssl support (rhbz 1320921)
+
 * Sun Feb 24 2019 Yauheni Kaliuta  - 26-1
 - Update to version 26 (rhbz 1673749)

[Yauheni Kaliuta]

Hi, Josh!
> > > > > On Mon, 25 Feb 2019 07:50:08 -0500, Josh Boyer  wrote:
>
> On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
>  wrote:

[...]
> > Then I would like to put stub instead of weak-modules script to
> > avoid confusion (there is no point to keep the full RHEL version
> > in Fedora).
> I don't agree with this part...  Why wouldn't we keep it
> there?  I understand Fedora doesn't support weak-modules, but
> there is nothing wrong with the script living upstream in
> Fedora and I see no disadvantage to keeping the RHEL version
> in-sync.  It's less work when we have to pull kmod back into a
> RHEL release in the future.

That will be the diff (I hope, it's ok to copy the script as is,
there were really a lot of changes there):

From 559b043f513e17ca89cd062f6f56f48209ccb390 Mon Sep 17 00:00:00 2001
From: Yauheni Kaliuta 
Date: Mon, 25 Feb 2019 15:10:47 +0200
Subject: [PATCH] weak-modules: sync with RHEL

Signed-off-by: Yauheni Kaliuta 
---
 weak-modules | 1105 +++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 876 insertions(+), 229 deletions(-)

diff --git a/weak-modules b/weak-modules
index 4b621e96fe43..db1839cf393c 100644
--- a/weak-modules
+++ b/weak-modules
@@ -3,10 +3,14 @@
 # weak-modules - determine which modules are kABI compatible with installed
 #                kernels and set up the symlinks in /lib/*/weak-updates.
 #
-# Changelog:
-#
-# 2010/01/10 - Further updates for dracut use on Fedora/RHEL (jcm).
-# 2009/09/16 - Rebase and add a bunch of updates for dracut (jcm).
+# This is an updated version of the script which doesn't support
+# multiple installation of the same out-of-tree module (stored in the
+# 'extra' subdirectory) for multiple kernels. This assumption is
+# supposed to be verified at the rpm level of the packages delivering
+# these modules.  There are some checks for this assumption, however we
+# really don't solve this situation. This limitation allows for a much
+# simpler version of the script. Previous version tried to work in this
+# case but was incorrect in some cases.

unset LANG LC_ALL LC_COLLATE

@@ -14,14 +18,19 @@ tmpdir=$(mktemp -td ${0##*/}.XXXXXX)
 trap "rm -rf $tmpdir" EXIT
 unset ${!changed_modules_*} ${!changed_initramfs_*}

-initramfs_prefix="/boot" # can customize here
+unset BASEDIR
+unset CHECK_INITRAMFS
+weak_updates_dir_override=""
+default_initramfs_prefix="/boot" # will be combined with BASEDIR
 dracut="/usr/bin/dracut"
+depmod="/sbin/depmod"
+depmod_orig="$depmod"
+declare -a modules
+declare -A module_krels
+declare -A weak_modules_before

-if [ ! -x "$dracut" ]
-then
-        echo "weak-modules: this tool requires a dracut-enabled kernel"
-	exit 1
-fi
+declare -A groups
+declare -A grouped_modules

# doit:
 # A wrapper used whenever we're going to perform a real operation.
@@ -30,29 +39,82 @@ doit() {
     [ -n "$dry_run" ] || "$@"
 }

+# pr_verbose:
+# print verbose -- wrapper used to print extra messages if required
+pr_verbose() {
+    [ -n "$verbose" ] && echo "$@"
+}
+
+# pr_warning:
+# print warning
+pr_warning() {
+    echo "WARNING: $*"
+}
+
 # rpmsort: The sort in coreutils can't sort the RPM list how we want it so we
 # instead transform the list into a form it will sort correctly, then sort.
 rpmsort() {
-	local IFS=$' '
-	REVERSE=""
-	rpmlist=($(cat))
+    local IFS=$' '
+    REVERSE=""
+    rpmlist=($(cat))
+
+    if [ "-r" == "$1" ];
+    then
+        REVERSE="-r"
+    fi
+
+    echo ${rpmlist[@]} | \
+        sed -e 's/-/../g' | \
+        sort ${REVERSE} -n -t"." -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -k7,7 \
+             -k8,8 -k9,9 -k10,10 | \
+        sed -e 's/../-/g'
+}
+
+# krel_of_module:
+# Compute the kernel release of a module.
+krel_of_module() {
+    local module="$1"
+
+    if [ x"${module_krels[$module]+set}" = x"set" ]; then
+        # version cached in the array already
+        echo "${module_krels[$module]}"
+    elif [ -f "$module" ]; then
+        krel_of_module_modinfo "$module"
+    else
+        # Try to extract the kernel release from the path
+        # delete case, the .ko already deleted
+        set -- "${module#*/lib/modules/}"
+        echo "${1%%/*}"
+    fi
+}
+
+# krel_of_module_modinfo:
+# Fetches module version from internal module info
+krel_of_module_modinfo() {
+    local module="$1"
+    /sbin/modinfo -F vermagic "$module" | awk '{print $1}'
+}

-	if [ "-r" == "$1" ];
-	then
-		REVERSE="-r"
-	fi
+# weak_updates_dir:
+# gives the root directory for the weak-updates
+# We need some flexibility here because of dry-run.
+weak_updates_dir() {
+    local krel="$1"

-	echo ${rpmlist[@]} | \
-	sed -e 's/-/../g' | \
-	sort ${REVERSE} -n -t"." -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -k7,7 \
-				 -k8,8 -k9,9 -k10,10 | \
-	sed -e 's/../-/g'
+    if [[ -z "$weak_updates_dir_override" ]]; then
+        echo "$BASEDIR/lib/modules/$krel/weak-updates"
+    else
+        echo "$weak_updates_dir_override"
+    fi
 }

# read_modules_list:
 # Read in a list of modules from standard input. Convert the filenames into
 # absolute paths and compute the kernel release for each module (either using
 # the modinfo section or through the absolute path.
+# If used with input redirect, should be used as read_module_list < input,
+# not input | read_modules_list, the latter spawns a subshell
+# and the arrays are not seen in the caller
 read_modules_list() {
     local IFS=$'\n'
     modules=($(cat))
@@ -61,16 +123,52 @@ read_modules_list() {
         if [ ${modules[n]:0:1} != '/' ]; then
             modules[n]="$PWD/${modules[n]}"
         fi
-        if [ -f "${modules[n]}" ]; then
-            module_krels[n]=$(krel_of_module ${modules[n]})
-        else
-            # Try to extract the kernel release from the path
-            set -- "${modules[n]#/lib/modules/}"
-            module_krels[n]=${1%%/*}
-        fi
+        module_krels["${modules[n]}"]=$(krel_of_module ${modules[n]})
     done
 }

+decompress_initramfs() {
+    local input=$1
+    local output=$2
+
+    # First, check if this is compressed at all
+    if cpio -i -t < "$input" > /dev/null 2>/dev/null; then
+        # If this archive contains a file early_cpio, it's a trick. Strip off
+        # the early cpio archive and try again.
+        if cpio -i -t < "$input" 2>/dev/null | grep -q '^early_cpio$' ; then
+            /usr/lib/dracut/skipcpio "$input" > "${tmpdir}/post_early_cpio.img"
+            decompress_initramfs "${tmpdir}/post_early_cpio.img" "$output"
+            retval="$?"
+            rm -f "${tmpdir}/post_early_cpio.img"
+            return $retval
+        fi
+
+        cp "$input" "$output"
+        return 0
+    fi
+
+    # Try gzip
+    if gzip -cd < "$input" > "$output" 2>/dev/null ; then
+        return 0
+    fi
+
+    # Next try xz
+    if xz -cd < "$input" > "$output" 2>/dev/null ; then
+        return 0
+    fi
+
+    echo "Unable to decompress $input: Unknown format" >&2
+    return 1
+}
+
+# List all module files and modprobe configuration that could require a new
+# initramfs. The current directory must be the root of the uncompressed
+# initramfs. The unsorted list of files is output to stdout.
+list_module_files() {
+    find . -iname *.ko -o -iname '*.ko.xz' -o -iname '*.ko.gz' 2>/dev/null
+    find etc/modprobe.d usr/lib/modprobe.d -name *.conf 2>/dev/null
+}
+
 # read_old_initramfs:
 compare_initramfs_modules() {
     local old_initramfs=$1
@@ -81,28 +179,35 @@ compare_initramfs_modules() {
     mkdir "$tmpdir/old_initramfs"
     mkdir "$tmpdir/new_initramfs"

+    decompress_initramfs "$old_initramfs" "$tmpdir/old_initramfs.img"
     pushd "$tmpdir/old_initramfs" >/dev/null
-    zcat "$old_initramfs" | cpio -i 2>/dev/null
-    n=0; for i in `find . -iname *.ko|sort`; do
+    cpio -i < "$tmpdir/old_initramfs.img" 2>/dev/null
+    rm "$tmpdir/old_initramfs.img"
+    n=0; for i in `list_module_files|sort`; do
         old_initramfs_modules[n]="$i"
         n=$((n+1))
     done
     popd >/dev/null

+    decompress_initramfs "$new_initramfs" "$tmpdir/new_initramfs.img"
     pushd "$tmpdir/new_initramfs" >/dev/null
-    zcat "$new_initramfs" | cpio -i 2>/dev/null
-    n=0; for i in `find . -iname *.ko|sort`; do
+    cpio -i < "$tmpdir/new_initramfs.img" 2>/dev/null
+    rm "$tmpdir/new_initramfs.img"
+    n=0; for i in `list_module_files|sort`; do
         new_initramfs_modules[n]="$i"
         n=$((n+1))
     done
     popd >/dev/null

-    if [ "${#old_initramfs_modules[@]}" == "${#new_initramfs_modules[@]}" ];
+    # Compare the length and contents of the arrays
+    if [ "${#old_initramfs_modules[@]}" == "${#new_initramfs_modules[@]}" -a \
+         "${old_initramfs_modules[*]}" == "${new_initramfs_modules[*]}" ];
     then
+        # If the file lists are the same, compare each file to find any that changed
         for ((n = 0; n < ${#old_initramfs_modules[@]}; n++)); do
-            old_md5=`md5sum $tmpdir/old_initramfs/${old_initramfs_modules[n]}|sed -nre 's:(^\ )* .*:\1:p'`
-            new_md5=`md5sum $tmpdir/new_initramfs/${new_initramfs_modules[n]}|sed -nre 's:(^\ )* .*:\1:p'`
-            if [ ! "$old_md5" == "$new_md5" ];
+            if ! cmp "$tmpdir/old_initramfs/${old_initramfs_modules[n]}" \
+                     "$tmpdir/new_initramfs/${new_initramfs_modules[n]}" \
+                     >/dev/null 2>&1
             then
                 return 1
             fi
@@ -128,7 +233,7 @@ check_initramfs() {

$dracut -f "$tmp_initramfs" "$kernel"

-        if ! $(compare_initramfs_modules "$old_initramfs" "$tmp_initramfs");
+        if ! compare_initramfs_modules "$old_initramfs" "$tmp_initramfs";
         then
             doit mv "$tmp_initramfs" "$new_initramfs"
         else
@@ -137,71 +242,6 @@ check_initramfs() {
     fi
 }

-# krel_of_module:
-# Compute the kernel release of a module.
-krel_of_module() {
-    declare module=$1
-    /sbin/modinfo -F vermagic "$module" | awk '{print $1}'
-}
-
-# module_is_compatible:
-# Determine if a module is compatible with a particular kernel release. Also
-# include any symbol deps that might be introduced by other external kmods.
-module_is_compatible() {
-    declare module=$1 krel=$2 module_krel=$(krel_of_module "$module")
-
-    if [ ! -e "$tmpdir/all-symvers-$krel-$module_krel" ]; then
-        # Symbols exported by the "new" kernel
-        if [ ! -e $tmpdir/symvers-$krel ]; then
-            if [ -e /boot/symvers-$krel.gz ]; then
-                zcat /boot/symvers-$krel.gz \
-                | sed -r -ne 's:^(0x[0]*[0-9a-f]{8}\t[0-9a-zA-Z_]+)\t.*:\1:p'
-            fi > $tmpdir/symvers-$krel
-        fi
-
-        # Symbols that other add-on modules of the "old" kernel export
-        # (and that this module may require)
-        if [ ! -e "$tmpdir/extra-symvers-$module_krel" ]; then
-            if [ -e /lib/modules/$module_krel/extra ] && \
-	       [ -n "`find /lib/modules/$module_krel/extra -type f`" ]; then
-                find /lib/modules/$module_krel/extra -name '*.ko' \
-                | xargs nm \
-                | sed -nre 's:^[0]*([0-9a-f]{8}) A __crc_(.*):0x\1 \2:p'
-            fi > $tmpdir/extra-symvers-$module_krel
-        fi
-
-        sort -u $tmpdir/symvers-$krel $tmpdir/extra-symvers-$module_krel \
-        > "$tmpdir/all-symvers-$krel-$module_krel"
-    fi
-
-    # If the module does not have modversions enabled, $tmpdir/modvers
-    # will be empty.
-    /sbin/modprobe --dump-modversions "$module" \
-    | sed -r -e 's:^(0x[0]*[0-9a-f]{8}\t.*):\1:' \
-    | sort -u \
-    > $tmpdir/modvers
-
-    # Only include lines of the second file in the output that don't
-    # match lines in the first file. (The default separator is
-    # <space>, so we are matching the whole line.)
-    join -j 1 -v 2 $tmpdir/all-symvers-$krel-$module_krel \
-                   $tmpdir/modvers > $tmpdir/join
-
-    if [ ! -s $tmpdir/modvers ]; then
-        echo "Warning: Module ${module##*/} from kernel $module_krel has no" \
-             "modversions, so it cannot be reused for kernel $krel" >&2
-    elif [ -s $tmpdir/join ]; then
-        [ -n "$verbose" ] &&
-        echo "Module ${module##*/} from kernel $module_krel is not compatible" \             "with kernel $krel in symbols:" $(sed -e 's:.* ::' $tmpdir/join)
-    else
-        [ -n "$verbose" ] &&
-        echo "Module ${module##*/} from kernel $module_krel is compatible" \
-             "with kernel $krel"
-        return 0
-    fi
-    return 1
-}
-
 usage() {
     echo "Usage: ${0##*/} [options] {--add-modules|--remove-modules}"
     echo "${0##*/} [options] {--add-kernel|--remove-kernel} {kernel-release}"
@@ -214,8 +254,7 @@ usage() {
 --remove-modules
         Remove compatibility symlinks from weak-updates/ directories
         for a list of modules.  The list of modules is read from
-        standard input. Optionally specify --delete-modules to
-        prevent weak-modules from attempting to locate any
+        standard input. Note: it doesn't attempt to locate any
         compatible modules to replace those being removed.

--add-kernel
@@ -227,7 +266,7 @@ usage() {
         kernel.

--no-initramfs
-	Do not generate an initramfs.
+        Do not generate an initramfs.

--verbose
         Print the commands executed.
@@ -244,8 +283,11 @@ EOF
 module_has_changed() {

declare module=$1 krel=$2
+    declare orig_module=$module

module=${module%.ko}
+    [[ $module == $orig_module ]] && module=${module%.ko.xz}
+    [[ $module == $orig_module ]] && module=${module%.ko.gz}
     module=${module##*/}

eval "changed_modules_${krel//[^a-zA-Z0-9]/_}=$krel"
@@ -253,134 +295,713 @@ module_has_changed() {

}

-# add_modules:
-# Read in a list of modules from stdinput and process them for compatibility
-# with installed kernels under /lib/modules.
-add_modules() {
+# module_weak_link:
+# Generate a weak link path for the module.
+# Takes module file name and the target kernel release as arguments
+# The way of generation intentionally left from the initial version
+module_weak_link() {
+    local module="$1"
+    local krel="$2"
+    local module_krel
+    local subpath
+    local module_krel_escaped
+
+    module_krel="$(krel_of_module "$module")"
+    module_krel_escaped=$(echo "$module_krel" | \
+                              sed 's/([.+?^$/\|()[]|])/\\0/g')
+    subpath=$(echo $module | sed -nre "s:$BASEDIR(/usr)?/lib/modules/$module_krel_escaped/([^/]*)/(.*):\3:p")
+
+    if [[ -z $subpath ]]; then
+        # module is not in /lib/modules/$krel?
+        # It's possible for example for Oracle ACFS compatibility check
+        # Install it with its full path as a /lib/modules subpath
+        subpath="$module"
+    fi
+
+    echo "$(weak_updates_dir $krel)/${subpath#/}"
+}
+
+# module_short_name:
+# 'basename' version purely in bash, cuts off path from the filename
+module_short_name() {
+    echo "${1##*/}"
+}
+
+#### Helper predicates
+
+# is_weak_for_module_valid:
+# Takes real module filename and target kernel as arguments.
+# Calculates weak symlink filename for the corresponding module
+# for the target kernel,
+# returns 'true' if the symlink filename is a symlink
+# and the symlink points to a readable file
+# EVEN if it points to a different filename
+is_weak_for_module_valid() {
+    local module="$1"
+    local krel="$2"
+    local weak_link
+
+    weak_link="$(module_weak_link $module $krel)"
+    [[ -L "$weak_link" ]] && [[ -r "$weak_link" ]]
+}
+
+# is_weak_link:
+# Takes a filename and a kernel release.
+# 'true' if the filename is symlink under weak-updates/ for the kernel.
+# It doesn't matter, if it's a valid symlink (points to a real file) or not.
+is_weak_link() {
+    local link="$1"
+    local krel="$2"
+
+    echo $link | grep -q "$(weak_updates_dir $krel)" || return 1
+    [[ -L $link ]]
+}
+
+# is_extra_exists:
+# Takes a module filename, the module's kernel release and target kernel release.
+# The module filename should be a real, not a symlink, filename (i.e. in extra/).
+# Returns 'true' if the same module exists for the target kernel.
+is_extra_exists() {
+    local module="$1"
+    local module_krel="$2"
+    local krel="$3"
+    local subpath="${module#*/lib/modules/$module_krel/extra/}"
+
+    [[ -f $BASEDIR/lib/modules/$krel/extra/$subpath ]]
+}
+
+is_kernel_installed() {
+    local krel="$1"
+
+    find_symvers_file "$krel" > /dev/null &&
+        find_systemmap_file "$krel" > /dev/null
+}
+
+is_empty_file() {
+    local file="$1"
+
+    [[ "$(wc -l "$file" | cut -f 1 -d ' ')" == 0 ]]
+}
+
+#### Helpers
+
+# find_modules:
+# Takes kernel release and a list of subdirectories.
+# Produces list of module files in the subdirectories for the kernel
+find_modules() {
+    local krel="$1"
+    shift
+    local dirs="$*"
+
+    for dir in $dirs; do
+        find $BASEDIR/lib/modules/$krel/$dir \
+             -name '*.ko' -o -name '*.ko.xz' -o -name '*.ko.gz' \
+             2>/dev/null
+    done
+}
+
+# find_modules_dirs:
+# Takes a list of directories.
+# Produces list of module files in the subdirectories
+find_modules_dirs() {
+    local dirs="$*"
+
+    for dir in $dirs; do
+        find $dir -name '*.ko' -o -name '*.ko.xz' -o -name '*.ko.gz' \
+             2>/dev/null
+    done
+}
+
+# find_installed_kernels:
+# Produces list of kernels, which modules are still installed
+find_installed_kernels() {
+    ls $BASEDIR/lib/modules/
+}
+
+# find_kernels_with_extra:
+# Produces list of kernels, where exists extra/ directory
+find_kernels_with_extra() {
+    local krel
+    local extra_dir
+
+    for krel in $(find_installed_kernels); do
+        extra_dir="$BASEDIR/lib/modules/$krel/extra"
+        [[ -d "$extra_dir" ]] || continue
+        echo "$krel"
+    done
+}
+
+# remove_weak_link_quiet:
+# Takes symlink filename and target kernel release.
+# Removes the symlink and the directory tree
+# if it was the last file in the tree
+remove_weak_link_quiet() {
+    local link="$1"
+    local krel="$2"
+    local subpath="${link#*$(weak_updates_dir $krel)}"
+
+    rm -f $link
+    ( cd "$(weak_updates_dir $krel)" && \
+          rmdir --parents --ignore-fail-on-non-empty "$(dirname "${subpath#/}")" 2>/dev/null )
+}
+
+# prepare_sandbox:
+# Takes kernel release, creates temporary weak-modules directory for it
+# and depmod config to operate on it.
+# Sets the global state accordingly
+
+prepare_sandbox() {
+    local krel="$1"
+    local orig_dir
+    local dir
+    local conf="$tmpdir/depmod.conf"
+
+    #directory
+    orig_dir=$(weak_updates_dir $krel)
+    dir="$tmpdir/$krel/weak-updates"
+
+    mkdir -p "$dir"
+    # the orig_dir can be empty
+    cp -R "$orig_dir"/* "$dir" 2>/dev/null
+
+    weak_updates_dir_override="$dir"
+
+    #config
+    echo "search external extra built-in weak-updates" >"$conf"
+    echo "external $krel $dir" >>"$conf"
+
+    depmod="$depmod_orig -C $conf"
+}
+
+
+# finish_sandbox:
+# restore global state after sandboxing
+# copy configuration to the kernel directory if not dry run
+finish_sandbox() {
+    local krel="$1"
+    local override="$weak_updates_dir_override"
+    local wa_dir
+
+    weak_updates_dir_override=""
+    depmod="$depmod_orig"
+
+    [[ -n "$dry_run" ]] && return
+
+    wa_dir="$(weak_updates_dir $krel)"
+
+    rm -rf "$wa_dir"
+    mkdir -p "$wa_dir"
+
+    cp -R "${override}"/* "$wa_dir" 2>/dev/null
+}
+
+# Auxiliary functions to find symvers file
+make_kernel_file_names() {
+    local krel="$1"
+    local file="$2"
+    local suffix="$3"
+
+    echo "${BASEDIR}/boot/${file}-${krel}${suffix}"
+    echo "${BASEDIR}/lib/modules/${krel}/${file}${suffix}"
+}
+
+find_kernel_file() {
+    local krel="$1"
+    local file="$2"
+    local suffix="$3"
+    local print="$4"
+    local i
+
+    if [[ "$print" != "" ]]; then
+        make_kernel_file_names "$krel" "$file" "$suffix"
+        return 0
+    fi
+
+    for i in  $(make_kernel_file_names "$krel" "$file" "$suffix"); do
+        if [[ -r "$i" ]]; then
+            echo "$i"
+            return 0
+        fi
+    done
+
+    return 1
+}
+
+# find_symvers_file:
+# Since /boot/ files population process is now controlled by systemd's
+# kernel-install bash script and its plug-ins, it might be the case
+# that, while present, symvers file is not populated in /boot.
+# Let's also check for /lib/modules/$kver/symvers.gz, since that's where
+# it is populated from.
+#
+# $1 - krel
+# return - 0 if symvers file is found, 1 otherwise.
+# Prints symvers path if found, empty string otherwise.
+find_symvers_file() {
+    local krel="$1"
+    local print="$2"
+
+    find_kernel_file "$krel" symvers .gz "$print"
+}
+
+# find_systemmap_file:
+# Same as above but for System.map
+find_systemmap_file() {
+    local krel="$1"
+    local print="$2"
+    local no_suffix=""
+
+    find_kernel_file "$krel" System.map "$no_suffix" "$print"
+}
+
+#### Main logic
+
+# update_modules_for_krel:
+# Takes kernel release and "action" function name.
+# Skips kernel without symvers,
+# otherwise triggers the main logic of modules installing/removing
+# for the given kernel, which is:
+# - save current state of weak modules symlinks
+# - install/remove the symlinks for the given (via stdin) list of modules
+# - validate the state and remove invalid symlinks
+#   (for the modules, which are not compatible (became incompatible) for
+#   the given kernel)
+# - check the state after validation to produce needed messages
+#   and trigger initrd regeneration if the list changed.
+update_modules_for_krel() {
+    local krel="$1"
+    local func="$2"
+    local force_update="$3"
+
+    is_kernel_installed "$krel" || return
+
+    prepare_sandbox $krel
+
+    global_link_state_save $krel
+
+    $func $krel
+
+    if ! validate_weak_links $krel && [[ -z "$force_update" ]]; then
+        global_link_state_restore $krel
+    fi
+
+    global_link_state_announce_changes $krel
+
+    finish_sandbox $krel
+}
+
+# update_modules:
+# Common entry point for add/remove modules command
+# Takes the "action" function, the module list is supplied via stdin.
+# Reads the module list and triggers modules update for all installed
+# kernels.
+# Triggers initrd rebuild for the kernels, which modules are installed.
+update_modules() {
+    local func="$1"
+    local force_update="$2"
+    local module_krel
+
     read_modules_list || exit 1
-    if [ ${#modules[@]} -gt 0 ]; then
-        for krel in $(ls /lib/modules/); do
-            [ -e "/boot/symvers-$krel.gz" ] || continue
-            for ((n = 0; n < ${#modules[@]}; n++)); do
-                module="${modules[n]}"
-                module_krel="${module_krels[n]}"
-                case "$module" in
-                /lib/modules/$krel/*)
-                    # Module was built against this kernel, update initramfs.
-                    module_has_changed $module $krel
-                    continue ;;
-                esac
-
-		# Module my also serve as a weak-update built against another
-		# kernel. We need to create symlinks for compatible kernels
-		# under /lib/modules and rerun depmod/dracut for those.
-
-		subpath=`echo $module | sed -nre "s:/lib/modules/$module_krel/([^/]*)/(.*):\2:p"`
-                weak_module="/lib/modules/$krel/weak-updates/${subpath#/}"
-                if [ -r "$weak_module" ]; then
-                    weak_krel=$(krel_of_module "$weak_module")
-                    if [ "$weak_krel" != "$module_krel" ] &&
-                       [ "$(printf "%s\n" "$weak_krel" "$module_krel" \
-                            | rpmsort | (read input; echo "$input"; \
-                                         while read input; do true; done))" = \
-                         "$module_krel" ]; then
-                        # Keep modules from more recent kernels.
-                        [ -n "$verbose" ] && echo \
-"Keeping module ${module##*/} from kernel $weak_krel for kernel $krel"
-                        continue
-                    fi
-                fi
-                if module_is_compatible $module $krel; then
-                    doit mkdir -p $(dirname $weak_module)
-                    doit ln -sf $module $weak_module
-                    # Module was built against another kernel, update initramfs.
-                    module_has_changed $module $krel
-                fi
-            done
-        done
+    [[ ${#modules[@]} -gt 0 ]] || return
+
+    for krel in $(find_installed_kernels); do
+        update_modules_for_krel $krel $func $force_update
+    done
+
+    for module in "${modules[@]}"; do
+        # Module was built against this kernel, update initramfs.
+        module_krel="${module_krels[$module]}"
+        module_has_changed $module $module_krel
+    done
+}
+
+# add_weak_links:
+# Action function for the "add-modules" command
+# Takes the kernel release, where the modules are added
+# and the modules[] and module_krels[] global arrays.
+# Install symlinks for the kernel with minimal checks
+# (just filename checks, no symbol checks)
+add_weak_links() {
+    local krel="$1"
+    local module_krel
+    local weak_link
+
+    for module in "${modules[@]}"; do
+        module_krel="$(krel_of_module $module)"
+
+        case "$module" in
+            /lib/modules/$krel/*)
+                # Module already installed to the current kernel
+                continue ;;
+        esac
+
+        if is_extra_exists $module $module_krel $krel; then
+            pr_verbose "found $(module_short_name $module) for $krel while installing for $module_krel, update case?"
+        fi
+
+        if is_weak_for_module_valid $module $krel; then
+            pr_verbose "weak module for $(module_short_name $module) already exists for kernel $krel, update case?"
+            # we should update initrd in update case,
+            # the change is not seen by the symlink detector
+            # (global_link_state_announce_changes())
+            module_has_changed $module $krel
+        fi
+
+        weak_link="$(module_weak_link $module $krel)"
+
+        mkdir -p "$(dirname $weak_link)"
+        ln -sf $module $weak_link
+
+    done
+}
+
+# remove_weak_links:
+# Action function for the "remove-modules" command
+# Takes the kernel release, where the modules are removed
+# and the modules[] and module_krels[] global arrays.
+# Removes symlinks from the given kernel if they are installed
+# for the modules in the list.
+remove_weak_links() {
+    local krel="$1"
+    local weak_link
+    local target
+    local module_krel
+
+    for module in "${modules[@]}"; do
+        module_krel="$(krel_of_module $module)"
+
+        weak_link="$(module_weak_link $module $krel)"
+        target="$(readlink $weak_link)"
+
+        if [[ "$module" != "$target" ]]; then
+            pr_verbose "Skipping symlink $weak_link"
+            continue
+        fi
+        # In update case the --remove-modules call is performed
+        # after --add-modules (from postuninstall).
+        # So, we shouldn't really remove the symlink in this case.
+        # But in the remove case the actual target already removed.
+        if ! is_weak_for_module_valid "$module" "$krel"; then
+            remove_weak_link_quiet "$weak_link" "$krel"
+        fi
+    done
+}
+
+# validate_weak_links:
+# Takes kernel release.
+# Checks if all the weak symlinks are suitable for the given kernel.
+# Uses depmod to perform the actual symbol checks and parses the output.
+# Since depmod internally creates the module list in the beginning of its work
+# accroding to the priority list in its configuration, but without symbol
+# check and doesn't amend the list during the check, the function runs it
+# in a loop in which it removes discovered incompatible symlinks
+#
+# Returns 0 (success) if proposal is fine or
+#         1 (false) if some incompatible symlinks were removed
+validate_weak_links() {
+    local krel="$1"
+    local basedir=${BASEDIR:+-b $BASEDIR}
+    local tmp
+    declare -A symbols
+    local is_updates_changed=1
+    local module
+    local module_krel
+    local target
+    local modpath
+    local symbol
+    local weak_link
+    # to return to caller that original proposal is not valid
+    # here 0 is true, 1 is false, since it will be the return code
+    local is_configuration_valid=0
+
+    tmp=$(mktemp -p $tmpdir)
+
+    if ! [[ -e $tmpdir/symvers-$krel ]]; then
+        local symvers_path=$(find_symvers_file "$krel")
+
+        [[ -n "$symvers_path" ]] || return
+        zcat "$symvers_path" > $tmpdir/symvers-$krel
     fi
+
+    while ((is_updates_changed)); do
+        is_updates_changed=0
+
+        # again $tmp because of subshell, see read_modules_list() comment
+        # create incompatibility report by depmod
+        # Shorcut if depmod finds a lot of incompatible modules elsewhere,
+        # we care only about weak-updates
+        $depmod $basedir -naeE $tmpdir/symvers-$krel $krel 2>&1 1>/dev/null | \
+            grep "$(weak_updates_dir $krel)" 2>/dev/null >$tmp
+        # parse it into symbols[] associative array in form a-la
+        #   symbols["/path/to/the/module"]="list of bad symbols"
+        while read line; do
+            set -- $(echo $line | awk '/needs unknown symbol/{print $3 " " $NF}')
+            modpath=$1
+            symbol=$2
+            if [[ -n "$modpath" ]]; then
+                symbols[$modpath]="${symbols[$modpath]} $symbol"
+                continue
+            fi
+
+            set -- $(echo $line | awk '/disagrees about version of symbol/{print $3 " " $NF}')
+            modpath=$1
+            symbol=$2
+            if [[ -n "$modpath" ]]; then
+                symbols[$modpath]="${symbols[$modpath]} $symbol"
+                continue
+            fi
+        done < $tmp
+
+        # loop through all the weak links from the list of incompatible
+        # modules and remove them. Skips non-weak incompatibilities
+        for modpath in "${!symbols[@]}"; do
+            is_weak_link $modpath $krel || continue
+
+            target=$(readlink $modpath)
+            module_krel=$(krel_of_module $target)
+
+            remove_weak_link_quiet "$modpath" "$krel"
+
+            pr_verbose "Module $(module_short_name $modpath) from kernel $module_krel is not compatible with kernel $krel in symbols: ${symbols[$modpath]}"
+            is_updates_changed=1
+            is_configuration_valid=1 # inversed value
+        done
+    done
+    rm -f $tmp
+
+    # this loop is just to produce verbose compatibility messages
+    # for the compatible modules
+    for module in "${modules[@]}"; do
+        is_weak_for_module_valid $module $krel || continue
+
+        weak_link="$(module_weak_link $module $krel)"
+        target="$(readlink $weak_link)"
+        module_krel=$(krel_of_module $target)
+
+        if [[ "$module" == "$target" ]]; then
+            pr_verbose "Module ${module##*/} from kernel $module_krel is compatible with kernel $krel"
+        fi
+    done
+    return $is_configuration_valid
+}
+
+# global_link_state_save:
+# Takes kernel release
+# Saves the given kernel's weak symlinks state into the global array
+# weak_modules_before[] for later processing
+global_link_state_save() {
+    local krel="$1"
+    local link
+    local target
+
+    weak_modules_before=()
+    for link in $(find_modules_dirs $(weak_updates_dir $krel) | xargs); do
+        target=$(readlink $link)
+        weak_modules_before[$link]=$target
+    done
+}
+
+# global_link_state_restore:
+# Takes kernel release
+# Restores the previous weak links state
+# (for example, if incompatible modules were installed)
+global_link_state_restore() {
+    local krel="$1"
+    local link
+    local target
+
+    pr_verbose "Falling back weak-modules state for kernel $krel"
+
+    ( cd "$(weak_updates_dir $krel)" 2>/dev/null && rm -rf * )
+
+    for link in "${!weak_modules_before[@]}"; do
+        target=${weak_modules_before[$link]}
+
+        mkdir -p "$(dirname $link)"
+        ln -sf $target $link
+    done
+}
+
+# global_link_state_announce_changes:
+# Takes kernel release
+# Reads the given kernel's weak symlinks state, compares to the saved,
+# triggers initrd rebuild if there were changes
+# and produces message on symlink removal
+global_link_state_announce_changes() {
+    local krel="$1"
+    local link
+    local target
+    local new_target
+    declare -A weak_modules_after
+
+    for link in $(find_modules_dirs $(weak_updates_dir $krel) | xargs); do
+        target=${weak_modules_before[$link]}
+        new_target=$(readlink $link)
+        weak_modules_after[$link]=$new_target
+
+        # report change of existing link and appearing of a new link
+        [[ "$target" == "$new_target" ]] || module_has_changed $new_target $krel
+    done
+
+    for link in "${!weak_modules_before[@]}"; do
+        target=${weak_modules_before[$link]}
+        new_target=${weak_modules_after[$link]}
+
+        # report change of existing link and disappearing of an old link
+        [[ "$target" == "$new_target" ]] && continue
+        module_has_changed $target $krel
+        [[ -n "$new_target" ]] ||
+            pr_verbose "Removing compatible module $(module_short_name $target) from kernel $krel"
+    done
 }

# remove_modules:
 # Read in a list of modules from stdinput and process them for removal.
-# Parameter is noreplace to delete modules, otherwise link compat.
+# Parameter (noreplace) is deprecated, acts always as "noreplace".
+# There is no sense in the "replace" functionality since according
+# to the current requirements RPM will track existing of only one version
+# of extra/ module (no same extra/ modules for different kernels).
 remove_modules() {
-    delete_modules=${1:-replace}
+    update_modules remove_weak_links force_update
+}

-    read_modules_list || exit 1
-    if [ ${#modules[@]} -gt 0 ]; then
-
-	# Hunt for all known users of this module in /lib/modules, remove them
-	# and create symlinks to other compatible modules (downgrade) if
-	# possible, update initramfs for each modified kernel too.
-
-        krels=($(ls /lib/modules/ | rpmsort -r))
-        for krel in "${krels[@]}"; do
-            [ -e "/boot/symvers-$krel.gz" ] || continue
-            for ((n = 0; n < ${#modules[@]}; n++)); do
-                module="${modules[n]}"
-                module_krel="${module_krels[n]}"
-
-		# Module is going to be removed, update initramfs.
-		module_has_changed $module $krel
-
-                subpath="${module#/lib/modules/$module_krel/extra}"
-                weak_module="/lib/modules/$krel/weak-updates/${subpath#/}"
-                if [ "$module" == "`readlink $weak_module`" ]; then
-                    [ -n "$verbose" ] && echo \
-"Removing compatible module ${module##*/} from kernel $krel"
-                    doit rm -f "$weak_module"
-                    if [ "replace" == "$delete_modules" ]; then
-                        for krel2 in "${krels[@]}"; do
-                            if [ $krel2 != $krel ]; then
-                                module="/lib/modules/$krel2/extra/${subpath#/}"
-                                [ -e "$module" ] || continue
-                                if module_is_compatible "$module" "$krel"; then
-                                    [ -n "$verbose" ] && echo \
-"Adding compatible module ${module##*/} from kernel $krel2 instead"
-                                    doit ln -s "$module" "$weak_module"
-			            module_has_changed $module $krel
-                                    break
-                                fi
-                            fi
-                        done
-                    fi
-                    doit rmdir --parents --ignore-fail-on-non-empty \
-                               "$(dirname "$weak_module")"
-                fi
-            done
+# add_modules:
+# Read in a list of modules from stdinput and process them for compatibility
+# with installed kernels under /lib/modules.
+add_modules() {
+    no_force_update=""
+
+    update_modules add_weak_links $no_force_update
+}
+
+# do_make_groups:
+# Takes tmp file which contains preprocessed modules.dep
+# output (or modules.dep)
+# reads modules.dep format information from stdin
+# produces groups associative array
+# the group is a maximum subset of modules having at least a link
+do_make_groups()
+{
+    local tmp="$1"
+    local group_name
+    local mod
+    declare -a mods
+
+    while read i; do
+        mods=($i)
+
+        # if the module already met, then its dependencies already counted
+        module_group="${grouped_modules[${mods[0]}]}"
+        [[ -n $module_group ]] && continue
+
+        # new group
+        group_name="${mods[0]}"
+
+        for mod in "${mods[@]}"; do
+            # if there is already such group,
+            # it is a subset of the one being created
+            # due to depmod output
+            unset groups[$mod]
+
+            # extra space doesn't matter, since later (in add_kernel())
+            # it is expanded without quotes
+            groups[$group_name]+=" $mod"
+            grouped_modules[$mod]=$group_name
         done
-    fi
+    done < $tmp # avoid subshell
+}
+
+# filter_depmod_deps:
+# preprocess output for make_groups
+# depmod -n produces also aliases, so it cuts them off
+# also it removes colon after the first module
+filter_depmod_deps()
+{
+    awk 'BEGIN { pr = 1 } /^#/{ pr = 0 } pr == 1 {sub(":",""); print $0}'
+}
+
+# make_abs_path:
+# Takes kernel version
+# makes full path from the relative module path
+# (produced by depmod for in-kernel-dir modules)
+make_abs_path()
+{
+    local kver="$1"
+    local mod
+    declare -a mods
+
+    while read i; do
+        mods=($i)
+        for j in "${!mods[@]}"; do
+            mod="${mods[$j]}"
+            [[ ${mod:0:1} == "/" ]] || mod="/lib/modules/$kver/$mod"
+            mods[$j]="$mod"
+        done
+        echo "${mods[@]}"
+    done
+}
+
+# make_groups:
+# takes krel and a file with the list of modules,
+# prepares and feeds to do_make_groups
+# to create the module groups (global)
+make_groups()
+{
+    local krel="$1"
+    local tmp1="$2"
+    local tmp2=$(mktemp -p $tmpdir)
+
+    groups=()
+    grouped_modules=()
+
+    $depmod -n $krel $(cat $tmp1) 2>/dev/null |
+        filter_depmod_deps | make_abs_path $krel > $tmp2
+
+    do_make_groups $tmp2
+
+    rm -f $tmp2
 }

add_kernel() {
-    add_krel=${1:-$(uname -r)}
-    if [ ! -e "/boot/symvers-$add_krel.gz" ]; then
-        echo "Symvers dump file /boot/symvers-$add_krel.gz" \
-             "not found" >&2
+    local krel=${1:-$(uname -r)}
+    local tmp
+    local no_force_update=""
+
+    tmp=$(mktemp -p $tmpdir)
+
+    if ! find_symvers_file "$krel" > /dev/null; then
+        echo "Symvers dump file is not found in" \
+             $(find_symvers_file "$krel" print) >&2
         exit 1
     fi
-    for krel in $(ls /lib/modules/ | rpmsort -r); do
-        [ "$add_krel" = "$krel" ] && continue
-        [ -d /lib/modules/$krel/extra ] || continue
-        for module in $(find /lib/modules/$krel/extra -name '*.ko'); do
-            subpath="${module#/lib/modules/$krel/extra}"
-            weak_module="/lib/modules/$add_krel/weak-updates/${subpath#/}"
-            [ -e "$weak_module" ] && continue
-            if module_is_compatible $module $add_krel; then
-		module_has_changed $module $add_krel
-                doit mkdir -p $(dirname $weak_module)
-                doit ln -sf $module $weak_module
-            fi
+
+    for k in $(find_kernels_with_extra | rpmsort); do
+        [[ "$krel" == "$k" ]] && continue
+        find_modules $k extra > $tmp
+
+        is_empty_file "$tmp" || make_groups $krel $tmp
+
+        # reuse tmp
+
+        for g in "${groups[@]}"; do
+            printf '%s\n' $g > $tmp
+            # to avoid subshell, see the read_modules_list comment
+            read_modules_list < $tmp
+            update_modules_for_krel $krel add_weak_links $no_force_update
         done
     done
+
+    rm -f $tmp
+
 }

remove_kernel() {
     remove_krel=${1:-$(uname -r)}
-    weak_modules="/lib/modules/$remove_krel/weak-updates"
+    weak_modules="$(weak_updates_dir $remove_krel)"
     module_has_changed $weak_modules $remove_krel
-    doit rm -rf "$weak_modules"
+
+    # Remove everything beneath the weak-updates directory
+    ( cd "$weak_modules" && doit rm -rf * )
 }

################################################################################
@@ -389,7 +1010,8 @@ remove_kernel() {

options=`getopt -o h --long help,add-modules,remove-modules \
                      --long add-kernel,remove-kernel \
-                     --long dry-run,no-initramfs,verbose,delete-modules -- "$@"`
+                     --long dry-run,no-initramfs,verbose,delete-modules \
+                     --long basedir:,dracut:,check-initramfs-prog: -- "$@"`

[ $? -eq 0 ] || usage 1

@@ -411,6 +1033,10 @@ while :; do
         ;;
     --dry-run)
         dry_run=1
+        # --dry-run option is not pure dry run anymore,
+        # because of depmod used internally.
+        # For add/remove modules we have to add/remove the symlinks
+        # and just restore the original configuration afterwards.
         ;;
     --no-initramfs)
         no_initramfs=1
@@ -419,7 +1045,19 @@ while :; do
         verbose=1
         ;;
     --delete-modules)
-        do_delete_modules=1
+        pr_warning "--delete-modules is deprecated, no effect"
+        ;;
+    --basedir)
+        BASEDIR="$2"
+        shift
+        ;;
+    --dracut)
+        dracut="$2"
+        shift
+        ;;
+    --check-initramfs-prog)
+        CHECK_INITRAMFS="$2"
+        shift
         ;;
     -h|--help)
         usage 0
@@ -432,27 +1070,31 @@ while :; do
     shift
 done

+if [ ! -x "$dracut" ]
+then
+    echo "weak-modules: could not find dracut at $dracut"
+    exit 1
+fi
+
+initramfs_prefix="$BASEDIR/${default_initramfs_prefix#/}"
+
 if [ -n "$do_add_modules" ]; then
-	add_modules
+    add_modules

elif [ -n "$do_remove_modules" ]; then
-        if [ -n "$do_delete_modules" ]; then
-            remove_modules "noreplace"
-        else
-	    remove_modules
-        fi
+    remove_modules

elif [ -n "$do_add_kernel" ]; then
-	kernel=${1:-$(uname -r)}
-	add_kernel $kernel
+    kernel=${1:-$(uname -r)}
+    add_kernel $kernel

elif [ -n "$do_remove_kernel" ]; then
-	kernel=${1:-$(uname -r)}
-	remove_kernel $kernel
+    kernel=${1:-$(uname -r)}
+    remove_kernel $kernel

-	exit 0
+    exit 0
 else
-	usage 1
+    usage 1
 fi

################################################################################
@@ -462,14 +1104,19 @@ fi
 # run depmod and dracut as needed
 for krel in ${!changed_modules_*}; do
     krel=${!krel}
+    basedir=${BASEDIR:+-b $BASEDIR}

-    doit /sbin/depmod -ae -F /boot/System.map-$krel $krel
+    if is_kernel_installed $krel; then
+        doit $depmod $basedir -ae -F $(find_systemmap_file $krel) $krel
+    else
+        pr_verbose "Skipping depmod for non-installed kernel $krel"
+    fi
 done

for krel in ${!changed_initramfs_*}; do
     krel=${!krel}

if [ ! -n "$no_initramfs" ]; then
-        check_initramfs $krel
+        ${CHECK_INITRAMFS:-check_initramfs} $krel
     fi
 done

[Josh Boyer]

On Mon, Feb 25, 2019 at 8:13 AM Yauheni Kaliuta
 wrote:
> Hi, Josh!
> > > > > > On Mon, 25 Feb 2019 07:50:08 -0500, Josh Boyer  wrote:
> >
> > On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
> >  wrote:
>
> [...]
> > > Then I would like to put stub instead of weak-modules script to
> > > avoid confusion (there is no point to keep the full RHEL version
> > > in Fedora).
> > I don't agree with this part...  Why wouldn't we keep it
> > there?  I understand Fedora doesn't support weak-modules, but
> > there is nothing wrong with the script living upstream in
> > Fedora and I see no disadvantage to keeping the RHEL version
> > in-sync.  It's less work when we have to pull kmod back into a
> > RHEL release in the future.
>
> That will be the diff (I hope, it's ok to copy the script as is,
> there were really a lot of changes there):

Ack.  That's why we should keep it updated :)

josh
> From 559b043f513e17ca89cd062f6f56f48209ccb390 Mon Sep 17 00:00:00 2001
> From: Yauheni Kaliuta 
> Date: Mon, 25 Feb 2019 15:10:47 +0200
> Subject: [PATCH] weak-modules: sync with RHEL
> Signed-off-by: Yauheni Kaliuta
>
> weak-modules | 1105 +++++++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 876 insertions(+), 229 deletions(-)
>
> diff --git a/weak-modules b/weak-modules
> index 4b621e96fe43..db1839cf393c 100644
> --- a/weak-modules
> +++ b/weak-modules
> @@ -3,10 +3,14 @@
>  # weak-modules - determine which modules are kABI compatible with installed
>  #                kernels and set up the symlinks in /lib/*/weak-updates.
>  #
> -# Changelog:
> -#
> -# 2010/01/10 - Further updates for dracut use on Fedora/RHEL (jcm).
> -# 2009/09/16 - Rebase and add a bunch of updates for dracut (jcm).
> +# This is an updated version of the script which doesn't support
> +# multiple installation of the same out-of-tree module (stored in the
> +# 'extra' subdirectory) for multiple kernels. This assumption is
> +# supposed to be verified at the rpm level of the packages delivering
> +# these modules.  There are some checks for this assumption, however we
> +# really don't solve this situation. This limitation allows for a much
> +# simpler version of the script. Previous version tried to work in this
> +# case but was incorrect in some cases.
>
> unset LANG LC_ALL LC_COLLATE
>
> @@ -14,14 +18,19 @@ tmpdir=$(mktemp -td ${0##*/}.XXXXXX)
>  trap "rm -rf $tmpdir" EXIT
>  unset ${!changed_modules_*} ${!changed_initramfs_*}
>
> -initramfs_prefix="/boot" # can customize here
> +unset BASEDIR
> +unset CHECK_INITRAMFS
> +weak_updates_dir_override=""
> +default_initramfs_prefix="/boot" # will be combined with BASEDIR
>  dracut="/usr/bin/dracut"
> +depmod="/sbin/depmod"
> +depmod_orig="$depmod"
> +declare -a modules
> +declare -A module_krels
> +declare -A weak_modules_before
>
> -if [ ! -x "$dracut" ]
> -then
>    echo "weak-modules: this tool requires a dracut-enabled kernel"
>   exit 1
>
> -fi
> +declare -A groups
> +declare -A grouped_modules
>
> # doit:
>  # A wrapper used whenever we're going to perform a real operation.
> @@ -30,29 +39,82 @@ doit() {
>      [ -n "$dry_run" ] || "$@"
>  }
>
> +# pr_verbose:
> +# print verbose -- wrapper used to print extra messages if required
> +pr_verbose() {
> [ -n "$verbose" ] && echo "$@"
>
> +}
>
> +# pr_warning:
> +# print warning
> +pr_warning() {
> echo "WARNING: $*"
>
> +}
>
> # rpmsort: The sort in coreutils can't sort the RPM list how we want it so we
>  # instead transform the list into a form it will sort correctly, then sort.
>  rpmsort() {
>   local IFS=$' '
>   REVERSE=""
>   rpmlist=($(cat))
> local IFS=$' '
> REVERSE=""
> rpmlist=($(cat))
> if [ "-r" == "$1" ];
> then
>    REVERSE="-r"
> fi
> echo ${rpmlist[@]} | \
>    sed -e 's/-/../g' | \
>    sort ${REVERSE} -n -t"." -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -k7,7 \
>         -k8,8 -k9,9 -k10,10 | \
>    sed -e 's/\.\./-/g'
>
> +}
>
> +# krel_of_module:
> +# Compute the kernel release of a module.
> +krel_of_module() {
> local module="$1"
> if [ x"${module_krels[$module]+set}" = x"set" ]; then
>    # version cached in the array already
>    echo "${module_krels[$module]}"
> elif [ -f "$module" ]; then
>    krel_of_module_modinfo "$module"
> else
>    # Try to extract the kernel release from the path
>    # delete case, the .ko already deleted
>    set -- "${module#*/lib/modules/}"
>    echo "${1%%/*}"
> fi
>
> +}
>
> +# krel_of_module_modinfo:
> +# Fetches module version from internal module info
> +krel_of_module_modinfo() {
> local module="$1"
> /sbin/modinfo -F vermagic "$module" | awk '{print $1}'
>
> +}
>   if [ "-r" == "$1" ];
>   then
>           REVERSE="-r"
>   fi
>
> +# weak_updates_dir:
> +# gives the root directory for the weak-updates
> +# We need some flexibility here because of dry-run.
> +weak_updates_dir() {
> local krel="$1"
>   echo ${rpmlist[@]} | \
>   sed -e 's/-/../g' | \
>   sort ${REVERSE} -n -t"." -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -k7,7 \
>                            -k8,8 -k9,9 -k10,10 | \
>   sed -e 's/\.\./-/g'
> if [[ -z "$weak_updates_dir_override" ]]; then
>    echo "$BASEDIR/lib/modules/$krel/weak-updates"
> else
>    echo "$weak_updates_dir_override"
> fi
>
> }
>
> # read_modules_list:
>  # Read in a list of modules from standard input. Convert the filenames into
>  # absolute paths and compute the kernel release for each module (either using
>  # the modinfo section or through the absolute path.
> +# If used with input redirect, should be used as read_module_list < input,
> +# not input | read_modules_list, the latter spawns a subshell
> +# and the arrays are not seen in the caller
>  read_modules_list() {
>      local IFS=$'\n'
>      modules=($(cat))
> @@ -61,16 +123,52 @@ read_modules_list() {
>          if [ ${modules[n]:0:1} != '/' ]; then
>              modules[n]="$PWD/${modules[n]}"
>          fi
>    if [ -f "${modules[n]}" ]; then
>        module_krels[n]=$(krel_of_module ${modules[n]})
>    else
>        # Try to extract the kernel release from the path
>        set -- "${modules[n]#/lib/modules/}"
>        module_krels[n]=${1%%/*}
>    fi
>    module_krels["${modules[n]}"]=$(krel_of_module ${modules[n]})
> done
>
> }
>
> +decompress_initramfs() {
> local input=$1
> local output=$2
> # First, check if this is compressed at all
> if cpio -i -t < "$input" > /dev/null 2>/dev/null; then
>    # If this archive contains a file early_cpio, it's a trick. Strip off
>    # the early cpio archive and try again.
>    if cpio -i -t < "$input" 2>/dev/null | grep -q '^early_cpio$' ; then
>        /usr/lib/dracut/skipcpio "$input" > "${tmpdir}/post_early_cpio.img"
>        decompress_initramfs "${tmpdir}/post_early_cpio.img" "$output"
>        retval="$?"
>        rm -f "${tmpdir}/post_early_cpio.img"
>        return $retval
>    fi
>    cp "$input" "$output"
>    return 0
> fi
> # Try gzip
> if gzip -cd < "$input" > "$output" 2>/dev/null ; then
>    return 0
> fi
> # Next try xz
> if xz -cd < "$input" > "$output" 2>/dev/null ; then
>    return 0
> fi
> echo "Unable to decompress $input: Unknown format" >&2
> return 1
>
> +}
>
> +# List all module files and modprobe configuration that could require a new
> +# initramfs. The current directory must be the root of the uncompressed
> +# initramfs. The unsorted list of files is output to stdout.
> +list_module_files() {
> find . -iname *.ko -o -iname '*.ko.xz' -o -iname '*.ko.gz' 2>/dev/null
> find etc/modprobe.d usr/lib/modprobe.d -name *.conf 2>/dev/null
>
> +}
>
> # read_old_initramfs:
>  compare_initramfs_modules() {
>      local old_initramfs=$1
> @@ -81,28 +179,35 @@ compare_initramfs_modules() {
>      mkdir "$tmpdir/old_initramfs"
>      mkdir "$tmpdir/new_initramfs"
> decompress_initramfs "$old_initramfs" "$tmpdir/old_initramfs.img"
> pushd "$tmpdir/old_initramfs" >/dev/null
> zcat "$old_initramfs" | cpio -i 2>/dev/null
> n=0; for i in `find . -iname *.ko|sort`; do
>
> cpio -i < "$tmpdir/old_initramfs.img" 2>/dev/null
>
> rm "$tmpdir/old_initramfs.img"
>
> n=0; for i in `list_module_files|sort`; do
>     old_initramfs_modules[n]="$i"
>     n=$((n+1))
> done
> popd >/dev/null
>
> decompress_initramfs "$new_initramfs" "$tmpdir/new_initramfs.img"
> pushd "$tmpdir/new_initramfs" >/dev/null
> zcat "$new_initramfs" | cpio -i 2>/dev/null
> n=0; for i in `find . -iname *.ko|sort`; do
> cpio -i < "$tmpdir/new_initramfs.img" 2>/dev/null
> rm "$tmpdir/new_initramfs.img"
> n=0; for i in `list_module_files|sort`; do
>     new_initramfs_modules[n]="$i"
>     n=$((n+1))
> done
> popd >/dev/null
> if [ "${#old_initramfs_modules[@]}" == "${#new_initramfs_modules[@]}" ];
> # Compare the length and contents of the arrays
> if [ "${#old_initramfs_modules[@]}" == "${#new_initramfs_modules[@]}" -a \
>     "${old_initramfs_modules[*]}" == "${new_initramfs_modules[*]}" ];
> then
>    # If the file lists are the same, compare each file to find any that changed
>    for ((n = 0; n < ${#old_initramfs_modules[@]}; n++)); do
>        old_md5=`md5sum $tmpdir/old_initramfs/${old_initramfs_modules[n]}|sed -nre 's:(^\ )* .*:\1:p'`
>        new_md5=`md5sum $tmpdir/new_initramfs/${new_initramfs_modules[n]}|sed -nre 's:(^\ )* .*:\1:p'`
>        if [ ! "$old_md5" == "$new_md5" ];
>        if ! cmp "$tmpdir/old_initramfs/${old_initramfs_modules[n]}" \
>                 "$tmpdir/new_initramfs/${new_initramfs_modules[n]}" \
>                 >/dev/null 2>&1
>        then
>            return 1
>        fi
>
> @@ -128,7 +233,7 @@ check_initramfs() {
>      $dracut -f "$tmp_initramfs" "$kernel"
>
>    if ! $(compare_initramfs_modules "$old_initramfs" "$tmp_initramfs");
>    if ! compare_initramfs_modules "$old_initramfs" "$tmp_initramfs";
>    then
>        doit mv "$tmp_initramfs" "$new_initramfs"
>    else
>
> @@ -137,71 +242,6 @@ check_initramfs() {
>      fi
>  }
>
> -# krel_of_module:
> -# Compute the kernel release of a module.
> -krel_of_module() {
> declare module=$1
> /sbin/modinfo -F vermagic "$module" | awk '{print $1}'
>
> -}
>
> -# module_is_compatible:
> -# Determine if a module is compatible with a particular kernel release. Also
> -# include any symbol deps that might be introduced by other external kmods.
> -module_is_compatible() {
> declare module=$1 krel=$2 module_krel=$(krel_of_module "$module")
> if [ ! -e "$tmpdir/all-symvers-$krel-$module_krel" ]; then
>    # Symbols exported by the "new" kernel
>    if [ ! -e $tmpdir/symvers-$krel ]; then
>        if [ -e /boot/symvers-$krel.gz ]; then
>            zcat /boot/symvers-$krel.gz \
>            | sed -r -ne 's:^(0x[0]*[0-9a-f]{8}\t[0-9a-zA-Z_]+)\t.*:\1:p'
>        fi > $tmpdir/symvers-$krel
>    fi
>    # Symbols that other add-on modules of the "old" kernel export
>    # (and that this module may require)
>    if [ ! -e "$tmpdir/extra-symvers-$module_krel" ]; then
>        if [ -e /lib/modules/$module_krel/extra ] && \
>          [ -n "`find /lib/modules/$module_krel/extra -type f`" ]; then
>            find /lib/modules/$module_krel/extra -name '*.ko' \
>            | xargs nm \
>            | sed -nre 's:^[0]*([0-9a-f]{8}) A __crc_(.*):0x\1 \2:p'
>        fi > $tmpdir/extra-symvers-$module_krel
>    fi
>    sort -u $tmpdir/symvers-$krel $tmpdir/extra-symvers-$module_krel \
>    > "$tmpdir/all-symvers-$krel-$module_krel"
> fi
> # If the module does not have modversions enabled, $tmpdir/modvers
> # will be empty.
> /sbin/modprobe --dump-modversions "$module" \
> | sed -r -e 's:^(0x[0]*[0-9a-f]{8}\t.*):\1:' \
> | sort -u \
> > $tmpdir/modvers
> # Only include lines of the second file in the output that don't
> # match lines in the first file. (The default separator is
> # <space>, so we are matching the whole line.)
> join -j 1 -v 2 $tmpdir/all-symvers-$krel-$module_krel \
>               $tmpdir/modvers > $tmpdir/join
> if [ ! -s $tmpdir/modvers ]; then
>    echo "Warning: Module ${module##*/} from kernel $module_krel has no" \
>         "modversions, so it cannot be reused for kernel $krel" >&2
> elif [ -s $tmpdir/join ]; then
>    [ -n "$verbose" ] &&
>    echo "Module ${module##*/} from kernel $module_krel is not compatible" \             "with kernel $krel in symbols:" $(sed -e 's:.* ::' $tmpdir/join)
> else
>    [ -n "$verbose" ] &&
>    echo "Module ${module##*/} from kernel $module_krel is compatible" \
>         "with kernel $krel"
>    return 0
> fi
> return 1
>
> -}
>
> usage() {
>      echo "Usage: ${0##*/} [options] {--add-modules|--remove-modules}"
>      echo "${0##*/} [options] {--add-kernel|--remove-kernel} {kernel-release}"
> @@ -214,8 +254,7 @@ usage() {
>  --remove-modules
>          Remove compatibility symlinks from weak-updates/ directories
>          for a list of modules.  The list of modules is read from
>    standard input. Optionally specify --delete-modules to
>    prevent weak-modules from attempting to locate any
>    standard input. Note: it doesn't attempt to locate any
>    compatible modules to replace those being removed.
>
> --add-kernel
> @@ -227,7 +266,7 @@ usage() {
>          kernel.
>
> --no-initramfs
>   Do not generate an initramfs.
>    Do not generate an initramfs.
>
> --verbose
>          Print the commands executed.
> @@ -244,8 +283,11 @@ EOF
>  module_has_changed() {
>  declare module=$1 krel=$2
>
> declare orig_module=$module
>
> module=${module%.ko}
>
> [[ $module == $orig_module ]] && module=${module%.ko.xz}
>
> [[ $module == $orig_module ]] && module=${module%.ko.gz}
> module=${module##*/}
>
> eval "changed_modules_${krel//[^a-zA-Z0-9]/_}=$krel"
>
> @@ -253,134 +295,713 @@ module_has_changed() {
>
> }
>
> -# add_modules:
> -# Read in a list of modules from stdinput and process them for compatibility
> -# with installed kernels under /lib/modules.
> -add_modules() {
> +# module_weak_link:
> +# Generate a weak link path for the module.
> +# Takes module file name and the target kernel release as arguments
> +# The way of generation intentionally left from the initial version
> +module_weak_link() {
> local module="$1"
> local krel="$2"
> local module_krel
> local subpath
> local module_krel_escaped
> module_krel="$(krel_of_module "$module")"
> module_krel_escaped=$(echo "$module_krel" | \
>                          sed 's/\([.+?^$\/\\|()\[]\|\]\)/\\\0/g')
> subpath=$(echo $module | sed -nre "s:$BASEDIR(/usr)?/lib/modules/$module_krel_escaped/([^/]*)/(.*):\3:p")
> if [[ -z $subpath ]]; then
>    # module is not in /lib/modules/$krel?
>    # It's possible for example for Oracle ACFS compatibility check
>    # Install it with its full path as a /lib/modules subpath
>    subpath="$module"
> fi
> echo "$(weak_updates_dir $krel)/${subpath#/}"
>
> +}
>
> +# module_short_name:
> +# 'basename' version purely in bash, cuts off path from the filename
> +module_short_name() {
> echo "${1##*/}"
>
> +}
>
> +#### Helper predicates
>
> +# is_weak_for_module_valid:
> +# Takes real module filename and target kernel as arguments.
> +# Calculates weak symlink filename for the corresponding module
> +# for the target kernel,
> +# returns 'true' if the symlink filename is a symlink
> +# and the symlink points to a readable file
> +# EVEN if it points to a different filename
> +is_weak_for_module_valid() {
> local module="$1"
> local krel="$2"
> local weak_link
> weak_link="$(module_weak_link $module $krel)"
> [[ -L "$weak_link" ]] && [[ -r "$weak_link" ]]
>
> +}
>
> +# is_weak_link:
> +# Takes a filename and a kernel release.
> +# 'true' if the filename is symlink under weak-updates/ for the kernel.
> +# It doesn't matter, if it's a valid symlink (points to a real file) or not.
> +is_weak_link() {
> local link="$1"
> local krel="$2"
> echo $link | grep -q "$(weak_updates_dir $krel)" || return 1
> [[ -L $link ]]
>
> +}
>
> +# is_extra_exists:
> +# Takes a module filename, the module's kernel release and target kernel release.
> +# The module filename should be a real, not a symlink, filename (i.e. in extra/).
> +# Returns 'true' if the same module exists for the target kernel.
> +is_extra_exists() {
> local module="$1"
> local module_krel="$2"
> local krel="$3"
> local subpath="${module#*/lib/modules/$module_krel/extra/}"
> [[ -f $BASEDIR/lib/modules/$krel/extra/$subpath ]]
>
> +}
>
> +is_kernel_installed() {
> local krel="$1"
> find_symvers_file "$krel" > /dev/null &&
>    find_systemmap_file "$krel" > /dev/null
>
> +}
>
> +is_empty_file() {
> local file="$1"
> [[ "$(wc -l "$file" | cut -f 1 -d ' ')" == 0 ]]
>
> +}
>
> +#### Helpers
>
> +# find_modules:
> +# Takes kernel release and a list of subdirectories.
> +# Produces list of module files in the subdirectories for the kernel
> +find_modules() {
> local krel="$1"
> shift
> local dirs="$*"
> for dir in $dirs; do
>    find $BASEDIR/lib/modules/$krel/$dir \
>         -name '*.ko' -o -name '*.ko.xz' -o -name '*.ko.gz' \
>         2>/dev/null
> done
>
> +}
>
> +# find_modules_dirs:
> +# Takes a list of directories.
> +# Produces list of module files in the subdirectories
> +find_modules_dirs() {
> local dirs="$*"
> for dir in $dirs; do
>    find $dir -name '*.ko' -o -name '*.ko.xz' -o -name '*.ko.gz' \
>         2>/dev/null
> done
>
> +}
>
> +# find_installed_kernels:
> +# Produces list of kernels, which modules are still installed
> +find_installed_kernels() {
> ls $BASEDIR/lib/modules/
>
> +}
>
> +# find_kernels_with_extra:
> +# Produces list of kernels, where exists extra/ directory
> +find_kernels_with_extra() {
> local krel
> local extra_dir
> for krel in $(find_installed_kernels); do
>    extra_dir="$BASEDIR/lib/modules/$krel/extra"
>    [[ -d "$extra_dir" ]] || continue
>    echo "$krel"
> done
>
> +}
>
> +# remove_weak_link_quiet:
> +# Takes symlink filename and target kernel release.
> +# Removes the symlink and the directory tree
> +# if it was the last file in the tree
> +remove_weak_link_quiet() {
> local link="$1"
> local krel="$2"
> local subpath="${link#*$(weak_updates_dir $krel)}"
> rm -f $link
> ( cd "$(weak_updates_dir $krel)" && \
>      rmdir --parents --ignore-fail-on-non-empty "$(dirname "${subpath#/}")" 2>/dev/null )
>
> +}
>
> +# prepare_sandbox:
> +# Takes kernel release, creates temporary weak-modules directory for it
> +# and depmod config to operate on it.
> +# Sets the global state accordingly
>
> +prepare_sandbox() {
> local krel="$1"
> local orig_dir
> local dir
> local conf="$tmpdir/depmod.conf"
> #directory
> orig_dir=$(weak_updates_dir $krel)
> dir="$tmpdir/$krel/weak-updates"
> mkdir -p "$dir"
> # the orig_dir can be empty
> cp -R "$orig_dir"/* "$dir" 2>/dev/null
> weak_updates_dir_override="$dir"
> #config
> echo "search external extra built-in weak-updates" >"$conf"
> echo "external $krel $dir" >>"$conf"
> depmod="$depmod_orig -C $conf"
>
> +}
>
> +# finish_sandbox:
> +# restore global state after sandboxing
> +# copy configuration to the kernel directory if not dry run
> +finish_sandbox() {
> local krel="$1"
> local override="$weak_updates_dir_override"
> local wa_dir
> weak_updates_dir_override=""
> depmod="$depmod_orig"
> [[ -n "$dry_run" ]] && return
> wa_dir="$(weak_updates_dir $krel)"
> rm -rf "$wa_dir"
> mkdir -p "$wa_dir"
> cp -R "${override}"/* "$wa_dir" 2>/dev/null
>
> +}
>
> +# Auxiliary functions to find symvers file
> +make_kernel_file_names() {
> local krel="$1"
> local file="$2"
> local suffix="$3"
> echo "${BASEDIR}/boot/${file}-${krel}${suffix}"
> echo "${BASEDIR}/lib/modules/${krel}/${file}${suffix}"
>
> +}
>
> +find_kernel_file() {
> local krel="$1"
> local file="$2"
> local suffix="$3"
> local print="$4"
> local i
> if [[ "$print" != "" ]]; then
>    make_kernel_file_names "$krel" "$file" "$suffix"
>    return 0
> fi
> for i in  $(make_kernel_file_names "$krel" "$file" "$suffix"); do
>    if [[ -r "$i" ]]; then
>        echo "$i"
>        return 0
>    fi
> done
> return 1
>
> +}
>
> +# find_symvers_file:
> +# Since /boot/ files population process is now controlled by systemd's
> +# kernel-install bash script and its plug-ins, it might be the case
> +# that, while present, symvers file is not populated in /boot.
> +# Let's also check for /lib/modules/$kver/symvers.gz, since that's where
> +# it is populated from.
> +#
> +# $1 - krel
> +# return - 0 if symvers file is found, 1 otherwise.
> +# Prints symvers path if found, empty string otherwise.
> +find_symvers_file() {
> local krel="$1"
> local print="$2"
> find_kernel_file "$krel" symvers .gz "$print"
>
> +}
>
> +# find_systemmap_file:
> +# Same as above but for System.map
> +find_systemmap_file() {
> local krel="$1"
> local print="$2"
> local no_suffix=""
> find_kernel_file "$krel" System.map "$no_suffix" "$print"
>
> +}
>
> +#### Main logic
>
> +# update_modules_for_krel:
> +# Takes kernel release and "action" function name.
> +# Skips kernel without symvers,
> +# otherwise triggers the main logic of modules installing/removing
> +# for the given kernel, which is:
> +# - save current state of weak modules symlinks
> +# - install/remove the symlinks for the given (via stdin) list of modules
> +# - validate the state and remove invalid symlinks
> +#   (for the modules, which are not compatible (became incompatible) for
> +#   the given kernel)
> +# - check the state after validation to produce needed messages
> +#   and trigger initrd regeneration if the list changed.
> +update_modules_for_krel() {
> local krel="$1"
> local func="$2"
> local force_update="$3"
> is_kernel_installed "$krel" || return
> prepare_sandbox $krel
> global_link_state_save $krel
> $func $krel
> if ! validate_weak_links $krel && [[ -z "$force_update" ]]; then
>    global_link_state_restore $krel
> fi
> global_link_state_announce_changes $krel
> finish_sandbox $krel
>
> +}
>
> +# update_modules:
> +# Common entry point for add/remove modules command
> +# Takes the "action" function, the module list is supplied via stdin.
> +# Reads the module list and triggers modules update for all installed
> +# kernels.
> +# Triggers initrd rebuild for the kernels, which modules are installed.
> +update_modules() {
> local func="$1"
> local force_update="$2"
> local module_krel
> read_modules_list || exit 1
> if [ ${#modules[@]} -gt 0 ]; then
>    for krel in $(ls /lib/modules/); do
>        [ -e "/boot/symvers-$krel.gz" ] || continue
>        for ((n = 0; n < ${#modules[@]}; n++)); do
>            module="${modules[n]}"
>            module_krel="${module_krels[n]}"
>            case "$module" in
>            /lib/modules/$krel/*)
>                # Module was built against this kernel, update initramfs.
>                module_has_changed $module $krel
>                continue ;;
>            esac
>           # Module my also serve as a weak-update built against another
>           # kernel. We need to create symlinks for compatible kernels
>           # under /lib/modules and rerun depmod/dracut for those.
>           subpath=`echo $module | sed -nre "s:/lib/modules/$module_krel/([^/]*)/(.*):\2:p"`
>            weak_module="/lib/modules/$krel/weak-updates/${subpath#/}"
>            if [ -r "$weak_module" ]; then
>                weak_krel=$(krel_of_module "$weak_module")
>                if [ "$weak_krel" != "$module_krel" ] &&
>                   [ "$(printf "%s\n" "$weak_krel" "$module_krel" \
>                        | rpmsort | (read input; echo "$input"; \
>                                     while read input; do true; done))" = \
>                     "$module_krel" ]; then
>                    # Keep modules from more recent kernels.
>                    [ -n "$verbose" ] && echo \
>
> -"Keeping module ${module##*/} from kernel $weak_krel for kernel $krel"
>                    continue
>                fi
>            fi
>            if module_is_compatible $module $krel; then
>                doit mkdir -p $(dirname $weak_module)
>                doit ln -sf $module $weak_module
>                # Module was built against another kernel, update initramfs.
>                module_has_changed $module $krel
>            fi
>        done
>    done
> [[ ${#modules[@]} -gt 0 ]] || return
> for krel in $(find_installed_kernels); do
>    update_modules_for_krel $krel $func $force_update
> done
> for module in "${modules[@]}"; do
>    # Module was built against this kernel, update initramfs.
>    module_krel="${module_krels[$module]}"
>    module_has_changed $module $module_krel
> done
>
> +}
>
> +# add_weak_links:
> +# Action function for the "add-modules" command
> +# Takes the kernel release, where the modules are added
> +# and the modules[] and module_krels[] global arrays.
> +# Install symlinks for the kernel with minimal checks
> +# (just filename checks, no symbol checks)
> +add_weak_links() {
> local krel="$1"
> local module_krel
> local weak_link
> for module in "${modules[@]}"; do
>    module_krel="$(krel_of_module $module)"
>    case "$module" in
>        /lib/modules/$krel/*)
>            # Module already installed to the current kernel
>            continue ;;
>    esac
>    if is_extra_exists $module $module_krel $krel; then
>        pr_verbose "found $(module_short_name $module) for $krel while installing for $module_krel, update case?"
>    fi
>    if is_weak_for_module_valid $module $krel; then
>        pr_verbose "weak module for $(module_short_name $module) already exists for kernel $krel, update case?"
>        # we should update initrd in update case,
>        # the change is not seen by the symlink detector
>        # (global_link_state_announce_changes())
>        module_has_changed $module $krel
>    fi
>    weak_link="$(module_weak_link $module $krel)"
>    mkdir -p "$(dirname $weak_link)"
>    ln -sf $module $weak_link
> done
>
> +}
>
> +# remove_weak_links:
> +# Action function for the "remove-modules" command
> +# Takes the kernel release, where the modules are removed
> +# and the modules[] and module_krels[] global arrays.
> +# Removes symlinks from the given kernel if they are installed
> +# for the modules in the list.
> +remove_weak_links() {
> local krel="$1"
> local weak_link
> local target
> local module_krel
> for module in "${modules[@]}"; do
>    module_krel="$(krel_of_module $module)"
>    weak_link="$(module_weak_link $module $krel)"
>    target="$(readlink $weak_link)"
>    if [[ "$module" != "$target" ]]; then
>        pr_verbose "Skipping symlink $weak_link"
>        continue
>    fi
>    # In update case the --remove-modules call is performed
>    # after --add-modules (from postuninstall).
>    # So, we shouldn't really remove the symlink in this case.
>    # But in the remove case the actual target already removed.
>    if ! is_weak_for_module_valid "$module" "$krel"; then
>        remove_weak_link_quiet "$weak_link" "$krel"
>    fi
> done
>
> +}
>
> +# validate_weak_links:
> +# Takes kernel release.
> +# Checks if all the weak symlinks are suitable for the given kernel.
> +# Uses depmod to perform the actual symbol checks and parses the output.
> +# Since depmod internally creates the module list in the beginning of its work
> +# accroding to the priority list in its configuration, but without symbol
> +# check and doesn't amend the list during the check, the function runs it
> +# in a loop in which it removes discovered incompatible symlinks
> +#
> +# Returns 0 (success) if proposal is fine or
> +#         1 (false) if some incompatible symlinks were removed
> +validate_weak_links() {
> local krel="$1"
> local basedir=${BASEDIR:+-b $BASEDIR}
> local tmp
> declare -A symbols
> local is_updates_changed=1
> local module
> local module_krel
> local target
> local modpath
> local symbol
> local weak_link
> # to return to caller that original proposal is not valid
> # here 0 is true, 1 is false, since it will be the return code
> local is_configuration_valid=0
> tmp=$(mktemp -p $tmpdir)
> if ! [[ -e $tmpdir/symvers-$krel ]]; then
>    local symvers_path=$(find_symvers_file "$krel")
>    [[ -n "$symvers_path" ]] || return
>    zcat "$symvers_path" > $tmpdir/symvers-$krel
> fi
> while ((is_updates_changed)); do
>    is_updates_changed=0
>    # again $tmp because of subshell, see read_modules_list() comment
>    # create incompatibility report by depmod
>    # Shorcut if depmod finds a lot of incompatible modules elsewhere,
>    # we care only about weak-updates
>    $depmod $basedir -naeE $tmpdir/symvers-$krel $krel 2>&1 1>/dev/null | \
>        grep "$(weak_updates_dir $krel)" 2>/dev/null >$tmp
>    # parse it into symbols[] associative array in form a-la
>    #   symbols["/path/to/the/module"]="list of bad symbols"
>    while read line; do
>        set -- $(echo $line | awk '/needs unknown symbol/{print $3 " " $NF}')
>        modpath=$1
>        symbol=$2
>        if [[ -n "$modpath" ]]; then
>            symbols[$modpath]="${symbols[$modpath]} $symbol"
>            continue
>        fi
>        set -- $(echo $line | awk '/disagrees about version of symbol/{print $3 " " $NF}')
>        modpath=$1
>        symbol=$2
>        if [[ -n "$modpath" ]]; then
>            symbols[$modpath]="${symbols[$modpath]} $symbol"
>            continue
>        fi
>    done < $tmp
>    # loop through all the weak links from the list of incompatible
>    # modules and remove them. Skips non-weak incompatibilities
>    for modpath in "${!symbols[@]}"; do
>        is_weak_link $modpath $krel || continue
>        target=$(readlink $modpath)
>        module_krel=$(krel_of_module $target)
>        remove_weak_link_quiet "$modpath" "$krel"
>        pr_verbose "Module $(module_short_name $modpath) from kernel $module_krel is not compatible with kernel $krel in symbols: ${symbols[$modpath]}"
>        is_updates_changed=1
>        is_configuration_valid=1 # inversed value
>    done
> done
> rm -f $tmp
> # this loop is just to produce verbose compatibility messages
> # for the compatible modules
> for module in "${modules[@]}"; do
>    is_weak_for_module_valid $module $krel || continue
>    weak_link="$(module_weak_link $module $krel)"
>    target="$(readlink $weak_link)"
>    module_krel=$(krel_of_module $target)
>    if [[ "$module" == "$target" ]]; then
>        pr_verbose "Module ${module##*/} from kernel $module_krel is compatible with kernel $krel"
>    fi
> done
> return $is_configuration_valid
>
> +}
>
> +# global_link_state_save:
> +# Takes kernel release
> +# Saves the given kernel's weak symlinks state into the global array
> +# weak_modules_before[] for later processing
> +global_link_state_save() {
> local krel="$1"
> local link
> local target
> weak_modules_before=()
> for link in $(find_modules_dirs $(weak_updates_dir $krel) | xargs); do
>    target=$(readlink $link)
>    weak_modules_before[$link]=$target
> done
>
> +}
>
> +# global_link_state_restore:
> +# Takes kernel release
> +# Restores the previous weak links state
> +# (for example, if incompatible modules were installed)
> +global_link_state_restore() {
> local krel="$1"
> local link
> local target
> pr_verbose "Falling back weak-modules state for kernel $krel"
> ( cd "$(weak_updates_dir $krel)" 2>/dev/null && rm -rf * )
> for link in "${!weak_modules_before[@]}"; do
>    target=${weak_modules_before[$link]}
>    mkdir -p "$(dirname $link)"
>    ln -sf $target $link
> done
>
> +}
>
> +# global_link_state_announce_changes:
> +# Takes kernel release
> +# Reads the given kernel's weak symlinks state, compares to the saved,
> +# triggers initrd rebuild if there were changes
> +# and produces message on symlink removal
> +global_link_state_announce_changes() {
> local krel="$1"
> local link
> local target
> local new_target
> declare -A weak_modules_after
> for link in $(find_modules_dirs $(weak_updates_dir $krel) | xargs); do
>    target=${weak_modules_before[$link]}
>    new_target=$(readlink $link)
>    weak_modules_after[$link]=$new_target
>    # report change of existing link and appearing of a new link
>    [[ "$target" == "$new_target" ]] || module_has_changed $new_target $krel
> done
> for link in "${!weak_modules_before[@]}"; do
>    target=${weak_modules_before[$link]}
>    new_target=${weak_modules_after[$link]}
>    # report change of existing link and disappearing of an old link
>    [[ "$target" == "$new_target" ]] && continue
>    module_has_changed $target $krel
>    [[ -n "$new_target" ]] ||
>        pr_verbose "Removing compatible module $(module_short_name $target) from kernel $krel"
> done
>
> }
>
> # remove_modules:
>  # Read in a list of modules from stdinput and process them for removal.
> -# Parameter is noreplace to delete modules, otherwise link compat.
> +# Parameter (noreplace) is deprecated, acts always as "noreplace".
> +# There is no sense in the "replace" functionality since according
> +# to the current requirements RPM will track existing of only one version
> +# of extra/ module (no same extra/ modules for different kernels).
>  remove_modules() {
> delete_modules=${1:-replace}
> update_modules remove_weak_links force_update
>
> +}
> read_modules_list || exit 1
> if [ ${#modules[@]} -gt 0 ]; then
>   # Hunt for all known users of this module in /lib/modules, remove them
>   # and create symlinks to other compatible modules (downgrade) if
>   # possible, update initramfs for each modified kernel too.
>    krels=($(ls /lib/modules/ | rpmsort -r))
>    for krel in "${krels[@]}"; do
>        [ -e "/boot/symvers-$krel.gz" ] || continue
>        for ((n = 0; n < ${#modules[@]}; n++)); do
>            module="${modules[n]}"
>            module_krel="${module_krels[n]}"
>           # Module is going to be removed, update initramfs.
>           module_has_changed $module $krel
>            subpath="${module#/lib/modules/$module_krel/extra}"
>            weak_module="/lib/modules/$krel/weak-updates/${subpath#/}"
>            if [ "$module" == "`readlink $weak_module`" ]; then
>                [ -n "$verbose" ] && echo \
>
> -"Removing compatible module ${module##*/} from kernel $krel"
>                doit rm -f "$weak_module"
>                if [ "replace" == "$delete_modules" ]; then
>                    for krel2 in "${krels[@]}"; do
>                        if [ $krel2 != $krel ]; then
>                            module="/lib/modules/$krel2/extra/${subpath#/}"
>                            [ -e "$module" ] || continue
>                            if module_is_compatible "$module" "$krel"; then
>                                [ -n "$verbose" ] && echo \
>
> -"Adding compatible module ${module##*/} from kernel $krel2 instead"
>                                doit ln -s "$module" "$weak_module"
>                               module_has_changed $module $krel
>                                break
>                            fi
>                        fi
>                    done
>                fi
>                doit rmdir --parents --ignore-fail-on-non-empty \
>                           "$(dirname "$weak_module")"
>            fi
>        done
>
> +# add_modules:
> +# Read in a list of modules from stdinput and process them for compatibility
> +# with installed kernels under /lib/modules.
> +add_modules() {
> no_force_update=""
> update_modules add_weak_links $no_force_update
>
> +}
>
> +# do_make_groups:
> +# Takes tmp file which contains preprocessed modules.dep
> +# output (or modules.dep)
> +# reads modules.dep format information from stdin
> +# produces groups associative array
> +# the group is a maximum subset of modules having at least a link
> +do_make_groups()
> +{
> local tmp="$1"
> local group_name
> local mod
> declare -a mods
> while read i; do
>    mods=($i)
>    # if the module already met, then its dependencies already counted
>    module_group="${grouped_modules[${mods[0]}]}"
>    [[ -n $module_group ]] && continue
>    # new group
>    group_name="${mods[0]}"
>    for mod in "${mods[@]}"; do
>        # if there is already such group,
>        # it is a subset of the one being created
>        # due to depmod output
>        unset groups[$mod]
>        # extra space doesn't matter, since later (in add_kernel())
>        # it is expanded without quotes
>        groups[$group_name]+=" $mod"
>        grouped_modules[$mod]=$group_name
>    done
> fi
> done < $tmp # avoid subshell
>
> +}
>
> +# filter_depmod_deps:
> +# preprocess output for make_groups
> +# depmod -n produces also aliases, so it cuts them off
> +# also it removes colon after the first module
> +filter_depmod_deps()
> +{
> awk 'BEGIN { pr = 1 } /^#/{ pr = 0 } pr == 1 {sub(":",""); print $0}'
>
> +}
>
> +# make_abs_path:
> +# Takes kernel version
> +# makes full path from the relative module path
> +# (produced by depmod for in-kernel-dir modules)
> +make_abs_path()
> +{
> local kver="$1"
> local mod
> declare -a mods
> while read i; do
>    mods=($i)
>    for j in "${!mods[@]}"; do
>        mod="${mods[$j]}"
>        [[ ${mod:0:1} == "/" ]] || mod="/lib/modules/$kver/$mod"
>        mods[$j]="$mod"
>    done
>    echo "${mods[@]}"
> done
>
> +}
>
> +# make_groups:
> +# takes krel and a file with the list of modules,
> +# prepares and feeds to do_make_groups
> +# to create the module groups (global)
> +make_groups()
> +{
> local krel="$1"
> local tmp1="$2"
> local tmp2=$(mktemp -p $tmpdir)
> groups=()
> grouped_modules=()
> $depmod -n $krel $(cat $tmp1) 2>/dev/null |
>    filter_depmod_deps | make_abs_path $krel > $tmp2
> do_make_groups $tmp2
> rm -f $tmp2
>
> }
>
> add_kernel() {
> add_krel=${1:-$(uname -r)}
> if [ ! -e "/boot/symvers-$add_krel.gz" ]; then
>    echo "Symvers dump file /boot/symvers-$add_krel.gz" \
>         "not found" >&2
> local krel=${1:-$(uname -r)}
> local tmp
> local no_force_update=""
> tmp=$(mktemp -p $tmpdir)
> if ! find_symvers_file "$krel" > /dev/null; then
>    echo "Symvers dump file is not found in" \
>         $(find_symvers_file "$krel" print) >&2
>    exit 1
> fi
> for krel in $(ls /lib/modules/ | rpmsort -r); do
>    [ "$add_krel" = "$krel" ] && continue
>    [ -d /lib/modules/$krel/extra ] || continue
>    for module in $(find /lib/modules/$krel/extra -name '*.ko'); do
>        subpath="${module#/lib/modules/$krel/extra}"
>        weak_module="/lib/modules/$add_krel/weak-updates/${subpath#/}"
>        [ -e "$weak_module" ] && continue
>        if module_is_compatible $module $add_krel; then
>           module_has_changed $module $add_krel
>            doit mkdir -p $(dirname $weak_module)
>            doit ln -sf $module $weak_module
>        fi
> for k in $(find_kernels_with_extra | rpmsort); do
>    [[ "$krel" == "$k" ]] && continue
>    find_modules $k extra > $tmp
>    is_empty_file "$tmp" || make_groups $krel $tmp
>    # reuse tmp
>    for g in "${groups[@]}"; do
>        printf '%s\n' $g > $tmp
>        # to avoid subshell, see the read_modules_list comment
>        read_modules_list < $tmp
>        update_modules_for_krel $krel add_weak_links $no_force_update
>    done
> done
> rm -f $tmp
>
> }
>
> remove_kernel() {
>      remove_krel=${1:-$(uname -r)}
> weak_modules="/lib/modules/$remove_krel/weak-updates"
> weak_modules="$(weak_updates_dir $remove_krel)"
> module_has_changed $weak_modules $remove_krel
> doit rm -rf "$weak_modules"
> # Remove everything beneath the weak-updates directory
> ( cd "$weak_modules" && doit rm -rf * )
>
> }
>
> ################################################################################
> @@ -389,7 +1010,8 @@ remove_kernel() {
>
> options=`getopt -o h --long help,add-modules,remove-modules \
>                       --long add-kernel,remove-kernel \
>                 --long dry-run,no-initramfs,verbose,delete-modules -- "$@"`
>                 --long dry-run,no-initramfs,verbose,delete-modules \
>                 --long basedir:,dracut:,check-initramfs-prog: -- "$@"`
>
> [ $? -eq 0 ] || usage 1
>
> @@ -411,6 +1033,10 @@ while :; do
>          ;;
>      --dry-run)
>          dry_run=1
>    # --dry-run option is not pure dry run anymore,
>    # because of depmod used internally.
>    # For add/remove modules we have to add/remove the symlinks
>    # and just restore the original configuration afterwards.
>    ;;
> --no-initramfs)
>        no_initramfs=1
>
> @@ -419,7 +1045,19 @@ while :; do
>          verbose=1
>          ;;
>      --delete-modules)
>    do_delete_modules=1
>    pr_warning "--delete-modules is deprecated, no effect"
>    ;;
> --basedir)
>    BASEDIR="$2"
>    shift
>    ;;
> --dracut)
>    dracut="$2"
>    shift
>    ;;
> --check-initramfs-prog)
>    CHECK_INITRAMFS="$2"
>    shift
>    ;;
> -h|--help)
>        usage 0
>
> @@ -432,27 +1070,31 @@ while :; do
>      shift
>  done
>
> +if [ ! -x "$dracut" ]
> +then
> echo "weak-modules: could not find dracut at $dracut"
> exit 1
>
> +fi
>
> +initramfs_prefix="$BASEDIR/${default_initramfs_prefix#/}"
>
> if [ -n "$do_add_modules" ]; then
>   add_modules
> add_modules
>
> elif [ -n "$do_remove_modules" ]; then
>    if [ -n "$do_delete_modules" ]; then
>        remove_modules "noreplace"
>    else
>       remove_modules
>    fi
> remove_modules
>
> elif [ -n "$do_add_kernel" ]; then
>   kernel=${1:-$(uname -r)}
>   add_kernel $kernel
> kernel=${1:-$(uname -r)}
> add_kernel $kernel
>
> elif [ -n "$do_remove_kernel" ]; then
>   kernel=${1:-$(uname -r)}
>   remove_kernel $kernel
> kernel=${1:-$(uname -r)}
> remove_kernel $kernel
>   exit 0
> exit 0
>
> else
>   usage 1
> usage 1
>
> fi
>
> ################################################################################
> @@ -462,14 +1104,19 @@ fi
>  # run depmod and dracut as needed
>  for krel in ${!changed_modules_*}; do
>      krel=${!krel}
> basedir=${BASEDIR:+-b $BASEDIR}
> doit /sbin/depmod -ae -F /boot/System.map-$krel $krel
> if is_kernel_installed $krel; then
>    doit $depmod $basedir -ae -F $(find_systemmap_file $krel) $krel
> else
>    pr_verbose "Skipping depmod for non-installed kernel $krel"
> fi
>
> done
>
> for krel in ${!changed_initramfs_*}; do
>      krel=${!krel}
>  if [ ! -n "$no_initramfs" ]; then
>    check_initramfs $krel
>    ${CHECK_INITRAMFS:-check_initramfs} $krel
> fi
> done
>
> 2.20.1
>
> --
> WBR,
> Yauheni Kaliuta

[Stanislav Kozina]

On Mon, 2019-02-25 at 07:50 -0500, Josh Boyer wrote:
> On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
>  wrote:
> > Hi!
> >
> > I should have sent the proposals first here, but forgot about the
> > list, sorry.
> >
> > I've submitted the version update  (25 -> 26) already.
>
> Great!
> > The next step will be enabling openssl for PKCS#7 reading.
>
> Yes please.
> > Then I would like to put stub instead of weak-modules script to
> > avoid confusion (there is no point to keep the full RHEL version
> > in Fedora).
>
> I don't agree with this part...  Why wouldn't we keep it there?  I
> understand Fedora doesn't support weak-modules, but there is nothing
> wrong with the script living upstream in Fedora and I see no
> disadvantage to keeping the RHEL version in-sync.  It's less work
> when
> we have to pull kmod back into a RHEL release in the future.

I agree with Josh. I also remember from the past that someone was
actually using Fedora as test bed for weak-modules usage. Makes lot of
sense to keep it up-to-date.

Thanks,
-Stanislav
> josh
> _______________________________________________
> kmod-maint mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]...

[Yauheni Kaliuta]

Hi, Stanislav!
> > > > > On Mon, 04 Mar 2019 11:58:31 +0100, Stanislav Kozina  wrote:
> On Mon, 2019-02-25 at 07:50 -0500, Josh Boyer wrote:
> > On Mon, Feb 25, 2019 at 7:42 AM Yauheni Kaliuta
> >  wrote:
> > > Then I would like to put stub instead of weak-modules script to
> > > avoid confusion (there is no point to keep the full RHEL version
> > > in Fedora).
> >
> > I don't agree with this part...  Why wouldn't we keep it
> > there?  I understand Fedora doesn't support weak-modules, but
> > there is nothing wrong with the script living upstream in
> > Fedora and I see no disadvantage to keeping the RHEL version
> > in-sync.  It's less work when we have to pull kmod back into
> > a RHEL release in the future.
> I agree with Josh. I also remember from the past that someone
> was actually using Fedora as test bed for weak-modules
> usage. Makes lot of sense to keep it up-to-date.

It must be there already.