11:16 a.m.
From: Clark Williams on gitlab.com
Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
Change merge.py to handle input config files in similar manner to
how merge.pl did it.
Signed-off-by: Clark Williams <williams(a)redhat.com>
---
redhat/configs/merge.pl | 72 -----------------------------
redhat/configs/merge.py | 109 +++++++++++++++++++++++++++----------------
redhat/Makefile | 2 +-
redhat/kernel.spec.template | 10 ++--
4 files changed, 73 insertions(+), 120 deletions(-)
11:16 a.m.
New subject: [OS-BUILD PATCHv2 1/3] redhat: update merge.py to handle merge.pl
corner cases
From: Clark Williams <williams(a)redhat.com>
redhat: update merge.py to handle merge.pl corner cases
Change merge.py to handle input config files in similar manner to
how merge.pl did it.
Signed-off-by: Clark Williams <williams(a)redhat.com>
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py
index blahblah..blahblah 100755
--- a/redhat/configs/merge.py
+++ b/redhat/configs/merge.py
@@ -5,54 +5,77 @@
#
# merge.py - a direct replacement for merge.pl in the redhat/configs directory
#
-# invocation: python merge.py overrides baseconfig
+# invocation: python merge.py overrides baseconfig [arch]
#
-# Both input files are kernel config files, where overides is config overides
-# to the baseconfig file. Both are read into python dictionaries with the
-# keys being the config name and the values being the config file text
-
-# The script iterates through the overrides keys adding/replacing the contents
-# of the baseconfig values and then outputs the new baseconfig to stdout.
+# This script merges two kernel configuration files, an override file and a
+# base config file and writes the results to stdout.
#
+# The script reads the overrides into a dictionary, then reads the baseconfig
+# file, looking for overrides and replacing any found, then printing the result
+# to stdout. Finally any remaining (new) configs in the override are appended to the
+# end of the output
import sys
+import re
import os.path
def usage(msg):
- print(msg)
- print("usage: merge.py overrides baseconfig")
+ sys.stderr.write(msg + "\n")
+ sys.stderr.write("usage: merge.py overrides baseconfig [arch]\n")
sys.exit(1)
-# read a config file and return a dictionary of the contents
-def read_config_file(cfgfile):
- configs = {}
- with open(cfgfile) as f:
- for l in [n.strip() for n in f.readlines()]:
- if not l: continue
- if l.startswith("# CONFIG_"):
- configs[l.split()[1]] = l
- continue
- if l.startswith("CONFIG_"):
- configs[l.split('=')[0]] = l
- continue
- return configs
-
-
-if len(sys.argv) < 3: usage("must have two input files")
-
-# read in the overides file
-if not os.path.exists(sys.argv[1]):
- usage("overrides config file %s does not exist!" % sys.argv[1])
-overrides = read_config_file(sys.argv[1])
-
-# read in the base config file
-if not os.path.exists(sys.argv[2]):
- usage("base config file %s does not exist!" % sys.argv[2])
-baseconfigs = read_config_file(sys.argv[2])
-
-# update baseconfigs with the overrides values
-baseconfigs.update(overrides)
-
-# print the new config to stdout
-for v in baseconfigs.values():
- print(v)
+isset = re.compile(r'^(CONFIG_\w+)=')
+notset = re.compile(r'^#\s+(CONFIG_\w+)\s+is not set')
+
+# search an input line for a config (set or notset) pattern
+# if we get a match return the config that is being changed
+def find_config(line):
+ if m := isset .match(line):
+ return m.group(1)
+ if m := notset.match(line):
+ return m.group(1)
+ return None
+
+#########################################################
+
+if len(sys.argv) < 3:
+ usage("must have two input files")
+
+override_file = sys.argv[1]
+baseconfig_file = sys.argv[2]
+
+if not os.path.exists(override_file):
+ usage("overrides config file %s does not exist!" % override_file)
+
+if not os.path.exists(baseconfig_file):
+ usage("base configs file %s does not exist" % baseconfig_file)
+
+if len(sys.argv) == 4:
+ print("# %s" % sys.argv[3])
+
+# read each line of the override file and store any configuration values
+# in the overrides dictionary, keyed by the configuration name.
+overrides = {}
+with open(override_file, "rt") as f:
+ for l in [n.strip() for n in f.readlines()]:
+ if c := find_config(l):
+ overrides[c] = l
+
+# now read and print the base config, checking each line
+# that defines a config value and printing the override if
+# it exists
+with open(baseconfig_file, "rt") as f:
+ for line in [ l.strip() for l in f.readlines() ]:
+ c = find_config(line)
+ if c and c in overrides:
+ print(overrides[c])
+ del overrides[c]
+ else:
+ print(line)
+
+# print out the remaining configs (new values)
+# from the overrides file
+for c in overrides:
+ print (overrides[c])
+
+sys.exit(0)
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
2:43 p.m.
New subject: [OS-BUILD PATCHv2 1/3] redhat: update merge.py to handle merge.pl
corner
cases
From: Clark Williams on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1254...
Updated to address review suggestions and pylint complaints.
11:16 a.m.
New subject: [OS-BUILD PATCHv2 2/3] redhat: remove merge.pl and references to it
From: Clark Williams <williams(a)redhat.com>
redhat: remove merge.pl and references to it
get rid of redhat/configs/merge.pl and any references to it
in redhat/kernel.spec.template and redhat/Makefile
Signed-off-by: Clark Williams <williams(a)redhat.com>
diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -634,7 +634,7 @@ sources-rh: $(TARBALL) generate-testpatch-tmp setup-source
dist-configs-check
scripts/mod/mod-sign.sh \
configs/flavors \
configs/generate_all_configs.sh \
- configs/merge.pl \
+ configs/merge.py \
configs/process_configs.sh \
../Makefile.rhelver \
README.rst \
diff --git a/redhat/configs/merge.pl b/redhat/configs/merge.pl
deleted file mode 100755
index blahblah..blahblah 0
--- a/redhat/configs/merge.pl
+++ /dev/null
@@ -1,72 +0,0 @@
-#! /usr/bin/perl
-
-my @args=@ARGV;
-my %configvalues;
-my @configoptions;
-my $configcounter = 0;
-
-# optionally print out the architecture as the first line of our output
-my $arch = $args[2];
-if (defined $arch) {
- print "# $arch\n";
-}
-
-# first, read the override file
-
-open (FILE,"$args[0]") || die "Could not open $args[0]";
-while (<FILE>) {
- my $str = $_;
- my $configname;
-
- if (/\# ([\w]+) is not set/) {
- $configname = $1;
- } elsif (/^\#/) {
- # fall through on comments like 'avoid CONFIG_FOO=y'
- ;
- } elsif (/([\w]+)=/) {
- $configname = $1;
- }
-
- if (defined($configname) && !exists($configvalues{$configname})) {
- $configvalues{$configname} = $str;
- $configoptions[$configcounter] = $configname;
- $configcounter ++;
- }
-};
-
-# now, read and output the entire configfile, except for the overridden
-# parts... for those the new value is printed.
-
-open (FILE2,"$args[1]") || die "Could not open $args[1]";
-while (<FILE2>) {
- my $configname;
-
- if (/\# ([\w]+) is not set/) {
- $configname = $1;
- } elsif (/^\#/) {
- # fall through on comments like 'avoid CONFIG_FOO=y'
- ;
- } elsif (/([\w]+)=/) {
- $configname = $1;
- }
-
- if (defined($configname) && exists($configvalues{$configname})) {
- print "$configvalues{$configname}";
- delete($configvalues{$configname});
- } else {
- print "$_";
- }
-}
-
-# now print the new values from the overridden configfile
-my $counter = 0;
-
-while ($counter < $configcounter) {
- my $configname = $configoptions[$counter];
- if (exists($configvalues{$configname})) {
- print "$configvalues{$configname}";
- }
- $counter++;
-}
-
-1;
diff --git a/redhat/kernel.spec.template b/redhat/kernel.spec.template
index blahblah..blahblah 100755
--- a/redhat/kernel.spec.template
+++ b/redhat/kernel.spec.template
@@ -850,7 +850,7 @@ Source2002: kvm_stat.logrotate
# Some people enjoy building customized kernels from the dist-git in Fedora and
# use this to override configuration options. One day they may all use the
# source tree, but in the mean time we carry this to support the legacy workflow
-Source3000: merge.pl
+Source3000: merge.py
Source3001: kernel-local
%if %{patchlist_changelog}
Source3002: Patchlist.changelog
@@ -1042,7 +1042,7 @@ This package provides debug information for package kernel-tools.
%{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} -p
'.*%%{_bindir}/centrino-decode(\.debug)?|.*%%{_bindir}/powernow-k8-decode(\.debug)?|.*%%{_bindir}/cpupower(\.debug)?|.*%%{_libdir}/libcpupower.*|.*%%{_bindir}/turbostat(\.debug)?|.*%%{_bindir}/x86_energy_perf_policy(\.debug)?|.*%%{_bindir}/tmon(\.debug)?|.*%%{_bindir}/lsgpio(\.debug)?|.*%%{_bindir}/gpio-hammer(\.debug)?|.*%%{_bindir}/gpio-event-mon(\.debug)?|.*%%{_bindir}/gpio-watch(\.debug)?|.*%%{_bindir}/iio_event_monitor(\.debug)?|.*%%{_bindir}/iio_generic_buffer(\.debug)?|.*%%{_bindir}/lsiio(\.debug)?|.*%%{_bindir}/intel-speed-select(\.debug)?|.*%%{_bindir}/page_owner_sort(\.debug)?|.*%%{_bindir}/slabinfo(\.debug)?|.*%%{_sbindir}/intel_sdsi(\.debug)?|XXX'
-o kernel-tools-debuginfo.list}
%package -n rtla
-Summary: RTLA: Real-Time Linux Analysis tools
+Summary: RTLA: Real-Time Linux Analysis tools
%description -n rtla
The rtla tool is a meta-tool that includes a set of commands that
aims to analyze the real-time properties of Linux. But, instead of
@@ -1512,7 +1512,7 @@ cd configs
# Drop some necessary files from the source dir into the buildroot
cp $RPM_SOURCE_DIR/kernel-*.config .
cp %{SOURCE80} .
-# merge.pl
+# merge.py
cp %{SOURCE3000} .
# kernel-local
cp %{SOURCE3001} .
@@ -1523,12 +1523,12 @@ FLAVOR=%{primary_target} SPECVERSION=%{version}
./generate_all_configs.sh %{debu
for i in %{all_arch_configs}
do
mv $i $i.tmp
- ./merge.pl %{SOURCE3001} $i.tmp > $i
+ ./merge.py %{SOURCE3001} $i.tmp > $i
%if %{with_gcov}
echo "Merging with gcov options"
cat %{SOURCE75}
mv $i $i.tmp
- ./merge.pl %{SOURCE75} $i.tmp > $i
+ ./merge.py %{SOURCE75} $i.tmp > $i
%endif
rm $i.tmp
done
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
11:16 a.m.
New subject: [OS-BUILD PATCHv2 3/3] redhat: fixup pylint complaints
From: Clark Williams <williams(a)redhat.com>
redhat: fixup pylint complaints
Added function doc strings
Use f-strings for output
Exlicitly use utf-8 encoding when processing config files
Signed-off-by: Clark Williams <williams(a)redhat.com>
diff --git a/redhat/configs/merge.py b/redhat/configs/merge.py
index blahblah..blahblah 100755
--- a/redhat/configs/merge.py
+++ b/redhat/configs/merge.py
@@ -20,6 +20,7 @@ import re
import os.path
def usage(msg):
+ '''print a usage message and exit'''
sys.stderr.write(msg + "\n")
sys.stderr.write("usage: merge.py overrides baseconfig [arch]\n")
sys.exit(1)
@@ -30,7 +31,8 @@ notset = re.compile(r'^#\s+(CONFIG_\w+)\s+is not set')
# search an input line for a config (set or notset) pattern
# if we get a match return the config that is being changed
def find_config(line):
- if m := isset .match(line):
+ '''find a configuration line in the input and return the config
name'''
+ if m := isset.match(line):
return m.group(1)
if m := notset.match(line):
return m.group(1)
@@ -45,26 +47,26 @@ override_file = sys.argv[1]
baseconfig_file = sys.argv[2]
if not os.path.exists(override_file):
- usage("overrides config file %s does not exist!" % override_file)
+ usage(f"overrides config file {override_file: s} does not exist!")
if not os.path.exists(baseconfig_file):
- usage("base configs file %s does not exist" % baseconfig_file)
+ usage(f"base configs file {baseconfig_file: s} does not exist")
if len(sys.argv) == 4:
- print("# %s" % sys.argv[3])
+ print(f"# {sys.argv[3]:s}")
# read each line of the override file and store any configuration values
# in the overrides dictionary, keyed by the configuration name.
overrides = {}
-with open(override_file, "rt") as f:
- for l in [n.strip() for n in f.readlines()]:
- if c := find_config(l):
- overrides[c] = l
+with open(override_file, "rt", encoding="utf-8") as f:
+ for line in [l.strip() for l in f.readlines()]:
+ if c := find_config(line):
+ overrides[c] = line
# now read and print the base config, checking each line
# that defines a config value and printing the override if
# it exists
-with open(baseconfig_file, "rt") as f:
+with open(baseconfig_file, "rt", encoding="utf-8") as f:
for line in [ l.strip() for l in f.readlines() ]:
c = find_config(line)
if c and c in overrides:
@@ -75,7 +77,7 @@ with open(baseconfig_file, "rt") as f:
# print out the remaining configs (new values)
# from the overrides file
-for c in overrides:
- print (overrides[c])
+for v in overrides.values():
+ print (v)
sys.exit(0)
--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262
12:43 p.m.
New subject: [OS-BUILD PATCHv2 0/3] redhat: update merge.py to handle merge.pl
corner
cases
From: Herton R. Krzesinski on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1258...
Looks like this part doesn't translate exactly what the perl part implemented:
```
if (defined($configname) && !exists($configvalues{$configname})) {
$configvalues{$configname} = $str;
$configoptions[$configcounter] = $configname;
$configcounter ++;
}
```
If the overrides file has the same config option listed twice, the perl seemed
to ignore it. Eg.
CONFIG_FOO = y
CONFIG_FOO = n
configvalues gets CONFIG_FOO = y
next iteration configvalues is not assigned because CONFIG_FOO is already
assigned previously
The python script is not checking this, always assign what it reads
(```overrides[c] = line```)
Not sure if this is a real problem or not, but just pointing out. We can
resolve this thread if it's not an issue.
3:01 p.m.
New subject: [OS-BUILD PATCHv2 0/3] redhat: update merge.py to handle merge.pl
corner
cases
From: Justin M. Forbes on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2262#note_1258...
This actually seems like the more desirable behavior, but in either case,
having the same config item listed multiple times in the file is a bug. It
does appear that this patch changes the config output, however it seems like
logical changes:
```
--- a/kernel-x86_64-rhel.config
+++ b/kernel-x86_64-rhel.config
@@ -868,7 +868,6 @@ CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_CPUMASK_KUNIT_TEST=m
CONFIG_CPUMASK_OFFSTACK=y
CONFIG_CPUSETS=y
-# CONFIG_CPU_SUP_CENTAUR ignored for predicate CONFIG_EXPERT
# CONFIG_CPU_THERMAL is not set
CONFIG_CPU_UNRET_ENTRY=y
# CONFIG_CRAMFS is not set
@@ -5864,44 +5863,31 @@ CONFIG_SND_SOC_RT715_SDW=m
# CONFIG_SND_SOC_SMDK_WM8994_PCM is not set
# CONFIG_SND_SOC_SNOW is not set
CONFIG_SND_SOC_SOF_ACPI=m
-# CONFIG_SND_SOC_SOF_ALDERLAKE:
CONFIG_SND_SOC_SOF_ALDERLAKE=m
# CONFIG_SND_SOC_SOF_AMD_TOPLEVEL is not set
-# CONFIG_SND_SOC_SOF_APOLLOLAKE:
CONFIG_SND_SOC_SOF_APOLLOLAKE=m
-# CONFIG_SND_SOC_SOF_BAYTRAIL:
CONFIG_SND_SOC_SOF_BAYTRAIL=m
-# CONFIG_SND_SOC_SOF_BROADWELL:
CONFIG_SND_SOC_SOF_BROADWELL=m
-# CONFIG_SND_SOC_SOF_CANNONLAKE:
CONFIG_SND_SOC_SOF_CANNONLAKE=m
-# CONFIG_SND_SOC_SOF_COFFEELAKE:
CONFIG_SND_SOC_SOF_COFFEELAKE=m
-# CONFIG_SND_SOC_SOF_COMETLAKE:
CONFIG_SND_SOC_SOF_COMETLAKE=m
-# CONFIG_SND_SOC_SOF_ELKHARTLAKE:
CONFIG_SND_SOC_SOF_ELKHARTLAKE=m
CONFIG_SND_SOC_SOF_GEMINILAKE=m
CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC=y
CONFIG_SND_SOC_SOF_HDA_LINK=y
-# CONFIG_SND_SOC_SOF_ICELAKE:
CONFIG_SND_SOC_SOF_ICELAKE=m
# CONFIG_SND_SOC_SOF_IMX8M_SUPPORT is not set
# CONFIG_SND_SOC_SOF_IMX8_SUPPORT is not set
# CONFIG_SND_SOC_SOF_IMX_TOPLEVEL is not set
-# CONFIG_SND_SOC_SOF_INTEL_SOUNDWIRE:
CONFIG_SND_SOC_SOF_INTEL_SOUNDWIRE=m
CONFIG_SND_SOC_SOF_INTEL_TOPLEVEL=y
-# CONFIG_SND_SOC_SOF_JASPERLAKE:
CONFIG_SND_SOC_SOF_JASPERLAKE=m
CONFIG_SND_SOC_SOF_KABYLAKE=m
-# CONFIG_SND_SOC_SOF_MERRIFIELD:
CONFIG_SND_SOC_SOF_MERRIFIELD=m
CONFIG_SND_SOC_SOF_METEORLAKE=m
# CONFIG_SND_SOC_SOF_OF is not set
CONFIG_SND_SOC_SOF_PCI=m
CONFIG_SND_SOC_SOF_SKYLAKE=m
-# CONFIG_SND_SOC_SOF_TIGERLAKE:
CONFIG_SND_SOC_SOF_TIGERLAKE=m
CONFIG_SND_SOC_SOF_TOPLEVEL=y
# CONFIG_SND_SOC_SPDIF is not set
@@ -7280,12 +7266,8 @@ CONFIG_XILINX_GMII2RGMII=m
# CONFIG_XILLYBUS is not set
# CONFIG_XILLYUSB is not set
CONFIG_XMON_DEFAULT_RO_MODE=y
-# CONFIG_XZ_DEC_ARM ignored for predicate CONFIG_EXPERT
-# CONFIG_XZ_DEC_ARMTHUMB ignored for predicate CONFIG_EXPERT
-# CONFIG_XZ_DEC_IA64 ignored for predicate CONFIG_EXPERT
# CONFIG_XZ_DEC_MICROLZMA is not set
CONFIG_XZ_DEC_POWERPC=y
-# CONFIG_XZ_DEC_SPARC ignored for predicate CONFIG_EXPERT
# CONFIG_XZ_DEC_TEST is not set
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC=y
```
Other arches included the same or just a subset as the SND_SOC bits were
lacking deps. If you look closely, we are losing comments that shouldn't be
in the file, and do not have an impact on the built kernel. Behavior of the
built kernels should be identical. For future changes, after merge, things
would work just as before the merge in that anyone making a config change
should verify that the config change they are making has the desired effect.
466
days inactive
470
days old
kernel@lists.fedoraproject.org
6 comments
3 participants
participants (3)
-
Clark Williams (via Email Bridge) -
Herton R. Krzesinski (via Email Bridge)
-
Justin M. Forbes (via Email Bridge)