[OS-BUILD PATCHv2 0/8] process_configs.sh: Add parallel processing of configs in dist-configs

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava on gitlab.com
Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

The dist-configs target takes an annoying long time.

Add parallel processing of configs in dist-configs.

Testing Info
------------

make dist-configs               time 1m22.909s
make -j20 dist-configs          time   11.801s

Removed redhat/configs/common/generic/x86/x86_64/CONFIG_ACPI_HOTPLUG_MEMORY
to test unset error.

> Found unset config items in x86_64-debug, please set them to an
> appropriate 
value
> CONFIG_ACPI_HOTPLUG_MEMORY=n
> Found unset config items in x86_64-kgcov, please set them to an appropriate 
value
> CONFIG_ACPI_HOTPLUG_MEMORY=n
> Found unset config items in x86_64, please set them to an appropriate value
> CONFIG_ACPI_HOTPLUG_MEMORY=n 
Changed redhat/configs/common/generic/s390x/CONFIG_HID from 'is not set'
to '=n' to test mismatch error.

> Error: Mismatches found in configuration files for s390x
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x-kgcov
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x-debug
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x-zfcpdump
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x-debug
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x-kgcov
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x-zfcpdump
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree
> Error: Mismatches found in configuration files for s390x
> Found # CONFIG_HID is not set, after generation, had CONFIG_HID n in Source 
tree

Changed redhat/configs/common/generic/CONFIG_PREEMPT to '=y' to test
misconfigured error.

> Found misconfigured config items in aarch64-debug, please set them to
> an 
appropriate value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-aarch64-debug.config.orig:4245:warning:
override: PREEMPT changes choice state
> Found misconfigured config items in aarch64-kgcov, please set them to
> an 
appropriate value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-aarch64-kgcov.config.orig:4224:warning:
override: PREEMPT changes choice state
> Found misconfigured config items in ppc64le-kgcov, please set them to
> an 
appropriate value
> /home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le-
kgcov.config.orig:4028:warning: override: PREEMPT changes choice state
> Found misconfigured config items in ppc64le, please set them to an
appropriate value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-ppc64le.config.orig:4026:warning: override:
PREEMPT changes choice state
> Found misconfigured config items in x86_64-debug, please set them to
> an 
appropriate value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-x86_64-debug.config.orig:4212:warning:
override: PREEMPT changes choice state
> Found misconfigured config items in x86_64-kgcov, please set them to
> an 
appropriate value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-x86_64-kgcov.config.orig:4192:warning:
override: PREEMPT changes choice state
> Found misconfigured config items in x86_64, please set them to an
> appropriate 
value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-x86_64.config.orig:4190:warning: override:
PREEMPT changes choice state
> Found misconfigured config items in aarch64, please set them to an
appropriate value
> /home/prarit/git-kernel/kernel-
ark/redhat/configs/kernel-5.17.0-aarch64.config.orig:4222:warning: override:
PREEMPT changes choice state
> Found misconfigured config items in ppc64le-debug, please set them to
> an 
appropriate value
> /home/prarit/git-kernel/kernel-ark/redhat/configs/kernel-5.17.0-ppc64le-
debug.config.orig:4045:warning: override: PREEMPT changes choice state

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

---
 redhat/configs/build_configs.sh   |   32 +++++-
 redhat/configs/process_configs.sh |  159 +++++++++++++++++++++++--------------
 redhat/Makefile                   |   10 +-
 3 files changed, 128 insertions(+), 73 deletions(-)

[OS-BUILD PATCHv2 1/8] redhat/configs/process_configs.sh: Add processing config function

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/process_configs.sh: Add processing config function

Move the code to process a config into its own function.  This will make
future changes easier.

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -218,68 +218,79 @@ function commit_new_configs()
 	git commit -m "[redhat] AUTOMATIC: New configs"
 }
 
-function process_configs()
+function process_config()
 {
-	# assume we are in $source_tree/configs, need to get to top level
-	pushd "$(switch_to_toplevel)" &>/dev/null
+	local cfg
+	local arch
+	local cfgtmp
+	local cfgorig
 
-	for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
-	do
-		arch=$(head -1 "$cfg" | cut -b 3-)
+	cfg=$1
+	arch=$(head -1 "$cfg" | cut -b 3-)
 
-		if [ "$arch" = "EMPTY" ]
-		then
-			# This arch is intentionally left blank
-			continue
-		fi
+	if [ "$arch" = "EMPTY" ]
+	then
+		# This arch is intentionally left blank
+		return
+	fi
 
-		cfgtmp="${cfg}.tmp"
-		cfgorig="${cfg}.orig"
-		cat "$cfg" > "$cfgorig"
+	cfgtmp="${cfg}.tmp"
+	cfgorig="${cfg}.orig"
+	cat "$cfg" > "$cfgorig"
 
-		echo -n "Processing $cfg ... "
+	echo -n "Processing $cfg ... "
 
-		make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
-		grep -E 'CONFIG_' .listnewconfig > .newoptions
-		if test -n "$NEWOPTIONS" && test -s .newoptions
-		then
-			echo "Found unset config items, please set them to an appropriate value"
-			cat .newoptions
-			rm .newoptions
-			RETURNCODE=1
-			[ "$CONTINUEONERROR" ] || exit 1
-		fi
+	make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
+	grep -E 'CONFIG_' .listnewconfig > .newoptions
+	if test -n "$NEWOPTIONS" && test -s .newoptions
+	then
+		echo "Found unset config items, please set them to an appropriate value"
+		cat .newoptions
 		rm .newoptions
+		RETURNCODE=1
+		[ "$CONTINUEONERROR" ] || exit 1
+	fi
+	rm .newoptions
 
-		grep -E 'config.*warning' .listnewconfig > .warnings
-		if test -n "$CHECKWARNINGS" && test -s .warnings
-		then
-			echo "Found misconfigured config items, please set them to an appropriate
value"
-			cat .warnings
-			rm .warnings
-			RETURNCODE=1
-			[ "$CONTINUEONERROR" ] || exit 1
-		fi
+	grep -E 'config.*warning' .listnewconfig > .warnings
+	if test -n "$CHECKWARNINGS" && test -s .warnings
+	then
+		echo "Found misconfigured config items, please set them to an appropriate
value"
+		cat .warnings
 		rm .warnings
+		RETURNCODE=1
+		[ "$CONTINUEONERROR" ] || exit 1
+	fi
+	rm .warnings
 
-		rm .listnewconfig
+	rm .listnewconfig
 
-		make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
-		echo "# $arch" > "$cfgtmp"
-		cat "$cfgorig" >> "$cfgtmp"
-		if test -n "$CHECKOPTIONS"
-		then
-			checkoptions "$cfg" "$cfgtmp"
-		fi
-		# if test run, don't overwrite original
-		if test -n "$TESTRUN"
-		then
-			rm -f "$cfgtmp"
-		else
-			mv "$cfgtmp" "$cfg"
-		fi
-		rm -f "$cfgorig"
-		echo "done"
+	make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
+	echo "# $arch" > "$cfgtmp"
+	cat "$cfgorig" >> "$cfgtmp"
+	if test -n "$CHECKOPTIONS"
+	then
+		checkoptions "$cfg" "$cfgtmp"
+	fi
+	# if test run, don't overwrite original
+	if test -n "$TESTRUN"
+	then
+		rm -f "$cfgtmp"
+	else
+		mv "$cfgtmp" "$cfg"
+	fi
+	rm -f "$cfgorig"
+	echo "done"
+}
+
+function process_configs()
+{
+	# assume we are in $source_tree/configs, need to get to top level
+	pushd "$(switch_to_toplevel)" &>/dev/null
+
+	for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
+	do
+		process_config "$cfg"
 	done
 	rm "$SCRIPT_DIR"/*.config*.old
 	popd > /dev/null

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 2/8] redhat/configs/process_configs.sh: Create unique output files

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/process_configs.sh: Create unique output files

Create unique output files from process_config().

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -60,6 +60,8 @@ switch_to_toplevel()
 
 checkoptions()
 {
+	count=$3
+
 	/usr/bin/awk '
 
 		/is not set/ {
@@ -82,10 +84,10 @@ checkoptions()
 					 print "Found "a[1]"="a[2]" after generation, had "
a[1]"="configs[a[1]]" in Source tree";
 			}
 		}
-	' "$1" "$2" > .mismatches
+	' "$1" "$2" > .mismatches${count}
 
 	checkoptions_error=false
-	if test -s .mismatches
+	if test -s .mismatches${count}
 	then
 		while read -r LINE
 		do
@@ -97,12 +99,12 @@ checkoptions()
 				checkoptions_error=true
 				break
 			fi
-		done < .mismatches
+		done < .mismatches${count}
 
 		! $checkoptions_error && return
 
 		echo "Error: Mismatches found in configuration files"
-		cat .mismatches
+		cat .mismatches${count}
 		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
 	fi
@@ -224,8 +226,11 @@ function process_config()
 	local arch
 	local cfgtmp
 	local cfgorig
+	local count
 
 	cfg=$1
+	count=$2
+
 	arch=$(head -1 "$cfg" | cut -b 3-)
 
 	if [ "$arch" = "EMPTY" ]
@@ -240,37 +245,37 @@ function process_config()
 
 	echo -n "Processing $cfg ... "
 
-	make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig
-	grep -E 'CONFIG_' .listnewconfig > .newoptions
-	if test -n "$NEWOPTIONS" && test -s .newoptions
+	make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig${count}
+	grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
+	if test -n "$NEWOPTIONS" && test -s .newoptions${count}
 	then
 		echo "Found unset config items, please set them to an appropriate value"
-		cat .newoptions
-		rm .newoptions
+		cat .newoptions${count}
+		rm .newoptions${count}
 		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
 	fi
-	rm .newoptions
+	rm .newoptions${count}
 
-	grep -E 'config.*warning' .listnewconfig > .warnings
-	if test -n "$CHECKWARNINGS" && test -s .warnings
+	grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
+	if test -n "$CHECKWARNINGS" && test -s .warnings${count}
 	then
 		echo "Found misconfigured config items, please set them to an appropriate
value"
-		cat .warnings
-		rm .warnings
+		cat .warnings${count}
+		rm .warnings${count}
 		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
 	fi
-	rm .warnings
+	rm .warnings${count}
 
-	rm .listnewconfig
+	rm .listnewconfig${count}
 
 	make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" olddefconfig > /dev/null || exit 1
 	echo "# $arch" > "$cfgtmp"
 	cat "$cfgorig" >> "$cfgtmp"
 	if test -n "$CHECKOPTIONS"
 	then
-		checkoptions "$cfg" "$cfgtmp"
+		checkoptions "$cfg" "$cfgtmp" "$count"
 	fi
 	# if test run, don't overwrite original
 	if test -n "$TESTRUN"
@@ -288,9 +293,11 @@ function process_configs()
 	# assume we are in $source_tree/configs, need to get to top level
 	pushd "$(switch_to_toplevel)" &>/dev/null
 
+	count=0
 	for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
 	do
-		process_config "$cfg"
+		process_config "$cfg" "$count"
+		((count++))
 	done
 	rm "$SCRIPT_DIR"/*.config*.old
 	popd > /dev/null

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 3/8] redhat/configs/process_configs.sh: Provide better messages

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/process_configs.sh: Provide better messages

Provide better error messages.

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -61,6 +61,7 @@ switch_to_toplevel()
 checkoptions()
 {
 	count=$3
+	variant=$4
 
 	/usr/bin/awk '
 
@@ -103,10 +104,12 @@ checkoptions()
 
 		! $checkoptions_error && return
 
-		echo "Error: Mismatches found in configuration files"
+		sed -i "1s/^/Error: Mismatches found in configuration files for
${variant}\n/" .mismatches${count}
 		cat .mismatches${count}
 		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
+	else
+		rm -f .mismatches${count}
 	fi
 }
 
@@ -227,6 +230,7 @@ function process_config()
 	local cfgtmp
 	local cfgorig
 	local count
+	local variant
 
 	cfg=$1
 	count=$2
@@ -239,17 +243,19 @@ function process_config()
 		return
 	fi
 
+	variant=$(basename "$cfg" | cut -d"-" -f3- | cut -d"."
-f1)
+
 	cfgtmp="${cfg}.tmp"
 	cfgorig="${cfg}.orig"
 	cat "$cfg" > "$cfgorig"
 
-	echo -n "Processing $cfg ... "
+	echo "Processing $cfg ... "
 
 	make ${MAKEOPTS} ARCH="$arch" CROSS_COMPILE=$(get_cross_compile $arch)
KCONFIG_CONFIG="$cfgorig" listnewconfig >& .listnewconfig${count}
 	grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
 	if test -n "$NEWOPTIONS" && test -s .newoptions${count}
 	then
-		echo "Found unset config items, please set them to an appropriate value"
+		echo "Found unset config items in ${variant}, please set them to an appropriate
value"
 		cat .newoptions${count}
 		rm .newoptions${count}
 		RETURNCODE=1
@@ -260,7 +266,7 @@ function process_config()
 	grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
 	if test -n "$CHECKWARNINGS" && test -s .warnings${count}
 	then
-		echo "Found misconfigured config items, please set them to an appropriate
value"
+		echo "Found misconfigured config items in ${variant}, please set them to an
appropriate value"
 		cat .warnings${count}
 		rm .warnings${count}
 		RETURNCODE=1
@@ -275,7 +281,7 @@ function process_config()
 	cat "$cfgorig" >> "$cfgtmp"
 	if test -n "$CHECKOPTIONS"
 	then
-		checkoptions "$cfg" "$cfgtmp" "$count"
+		checkoptions "$cfg" "$cfgtmp" "$count"
"$variant"
 	fi
 	# if test run, don't overwrite original
 	if test -n "$TESTRUN"
@@ -285,7 +291,7 @@ function process_config()
 		mv "$cfgtmp" "$cfg"
 	fi
 	rm -f "$cfgorig"
-	echo "done"
+	echo "Processing $cfg complete"
 }
 
 function process_configs()

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 4/8] redhat/configs/process_configs.sh: Parallelize execution

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/process_configs.sh: Parallelize execution

Parallelize the execution of each config processing.

The '+' adds job processing to 'non-make' commands [1].

[1] https://www.gnu.org/software/make/manual/html_node/Error-Messages.html

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -121,11 +121,11 @@ dist-kabi-dw-check: dist-kabi
 	@rm -rf $(KABIDW)/base/$(CURARCH).tmp
 
 dist-configs-commit: dist-configs-prep
-	@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
+	+@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
 	./process_configs.sh -z $(PACKAGE_NAME) $(KVERSION) "" $(FLAVOR)
 
 dist-configs: dist-configs-prep
-	@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
+	+@cd $(REDHAT)/configs; VERSION=$(KVERSION) ./generate_all_configs.sh
"$(FLAVOR)" 1; \
 	./process_configs.sh $(PROCESS_CONFIGS_OPTS) $(PACKAGE_NAME) $(KVERSION)
 
 dist-fedora-configs: FLAVOR = fedora
@@ -137,7 +137,7 @@ dist-rhel-configs: dist-configs
 rh-configs: dist-rhel-configs
 
 dist-configs-check: dist-configs-prep
-	cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS)
$(PACKAGE_NAME)
+	+cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS)
$(PACKAGE_NAME)
 
 dist-configs-prep: dist-clean-configs dist-buildreq-check
 	cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)"
"snip"
diff --git a/redhat/configs/process_configs.sh b/redhat/configs/process_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/process_configs.sh
+++ b/redhat/configs/process_configs.sh
@@ -106,7 +106,6 @@ checkoptions()
 
 		sed -i "1s/^/Error: Mismatches found in configuration files for
${variant}\n/" .mismatches${count}
 		cat .mismatches${count}
-		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
 	else
 		rm -f .mismatches${count}
@@ -255,8 +254,8 @@ function process_config()
 	grep -E 'CONFIG_' .listnewconfig${count} > .newoptions${count}
 	if test -n "$NEWOPTIONS" && test -s .newoptions${count}
 	then
-		echo "Found unset config items in ${variant}, please set them to an appropriate
value"
-		cat .newoptions${count}
+		echo "Found unset config items in ${variant}, please set them to an appropriate
value" >> .errors${count}
+		cat .newoptions${count} >> .errors${count}
 		rm .newoptions${count}
 		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
@@ -266,10 +265,9 @@ function process_config()
 	grep -E 'config.*warning' .listnewconfig${count} > .warnings${count}
 	if test -n "$CHECKWARNINGS" && test -s .warnings${count}
 	then
-		echo "Found misconfigured config items in ${variant}, please set them to an
appropriate value"
-		cat .warnings${count}
+		echo "Found misconfigured config items in ${variant}, please set them to an
appropriate value" >> .errors${count}
+		cat .warnings${count} >> .errors${count}
 		rm .warnings${count}
-		RETURNCODE=1
 		[ "$CONTINUEONERROR" ] || exit 1
 	fi
 	rm .warnings${count}
@@ -302,13 +300,30 @@ function process_configs()
 	count=0
 	for cfg in "$SCRIPT_DIR/${PACKAGE_NAME}${KVERREL}${SUBARCH}"*.config
 	do
-		process_config "$cfg" "$count"
+		process_config "$cfg" "$count" &
+		waitpids[${count}]=$!
 		((count++))
 	done
+	for pid in ${waitpids[*]}; do
+		wait ${pid}
+	done
+
 	rm "$SCRIPT_DIR"/*.config*.old
+
+	if ls .errors* 1> /dev/null 2>&1; then
+		RETURNCODE=1
+		cat .errors*
+		rm .errors* -f
+	fi
+	if ls .mismatches* 1> /dev/null 2>&1; then
+		RETURNCODE=1
+		cat .mismatches*
+		rm .mismatches* -f
+	fi
+
 	popd > /dev/null
 
-	echo "Processed config files are in $SCRIPT_DIR"
+	[ $RETURNCODE -eq 0 ] && echo "Processed config files are in
$SCRIPT_DIR"
 }
 
 CHECKOPTIONS=""

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 5/8] redhat/configs/build_configs.sh: Add local variables

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/build_configs.sh: Add local variables

Add local variables to the merge_configs function.  This will make future
changes for parallel processing easier.

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -53,6 +53,14 @@ function combine_config_layer()
 
 function merge_configs()
 {
+	local archvar
+	local arch
+	local configs
+	local order
+	local flavor
+	local name
+	local skip_if_missing
+
 	archvar=$1
 	arch=$(echo "$archvar" | cut -f1 -d"-")
 	configs=$2

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 6/8] redhat/configs/build_configs.sh: Create unique output files

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/build_configs.sh: Create unique output files

Create unique output files from merge_configs().

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -58,6 +58,7 @@ function merge_configs()
 	local configs
 	local order
 	local flavor
+	local count
 	local name
 	local skip_if_missing
 
@@ -66,10 +67,11 @@ function merge_configs()
 	configs=$2
 	order=$3
 	flavor=$4
+	count=$5
 
 	name=$OUTPUT_DIR/$PACKAGE_NAME-$archvar-$flavor.config
 	echo -n "Building $name ... "
-	touch config-merging config-merged
+	touch config-merging.$count config-merged.$count
 
 	# apply based on order
 	skip_if_missing=""
@@ -81,10 +83,10 @@ function merge_configs()
 
 			test -n "$skip_if_missing" && test ! -e "$cfile"
&& continue
 
-			if ! perl merge.pl "$cfile" config-merging > config-merged; then
+			if ! perl merge.pl "$cfile" config-merging.$count > config-merged.$count;
then
 				die "Failed to merge $cfile"
 			fi
-			mv config-merged config-merging
+			mv config-merged.$count config-merging.$count
 		done
 
 		# first configs in $order is baseline, all files should be
@@ -104,8 +106,8 @@ function merge_configs()
 	else
 		echo "# $arch" > "$name"
 	fi
-	sort config-merging >> "$name"
-	rm -f config-merged config-merging
+	sort config-merging.$count >> "$name"
+	rm -f config-merged.$count config-merging.$count
 	echo "done"
 }
 
@@ -149,7 +151,7 @@ function build_flavor()
 				esac
 			fi
 
-			merge_configs "$arch" "$configs" "$order"
"$flavor"
+			merge_configs "$arch" "$configs" "$order"
"$flavor" "$count"
 		fi
 	done < "$control_file"
 }

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 7/8] redhat/configs/build_configs.sh: Provide better messages

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/build_configs.sh: Provide better messages

Provide better error messages.

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -70,7 +70,7 @@ function merge_configs()
 	count=$5
 
 	name=$OUTPUT_DIR/$PACKAGE_NAME-$archvar-$flavor.config
-	echo -n "Building $name ... "
+	echo "Building $name ... "
 	touch config-merging.$count config-merged.$count
 
 	# apply based on order
@@ -108,7 +108,7 @@ function merge_configs()
 	fi
 	sort config-merging.$count >> "$name"
 	rm -f config-merged.$count config-merging.$count
-	echo "done"
+	echo "Building $name complete"
 }
 
 function build_flavor()

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[OS-BUILD PATCHv2 8/8] redhat/configs/build_configs.sh: Parallelize execution

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava <prarit(a)redhat.com&gt;

redhat/configs/build_configs.sh: Parallelize execution

Parallelize the execution of each config building.

The '+' adds job processing to 'non-make' commands [1].

[1] https://www.gnu.org/software/make/manual/html_node/Error-Messages.html

Signed-off-by: Prarit Bhargava <prarit(a)redhat.com&gt;

diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -140,8 +140,8 @@ dist-configs-check: dist-configs-prep
 	+cd $(REDHAT)/configs; ./process_configs.sh $(PROCESS_CONFIGS_CHECK_OPTS)
$(PACKAGE_NAME)
 
 dist-configs-prep: dist-clean-configs dist-buildreq-check
-	cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)"
"snip"
-	cd $(REDHAT)/configs; ./build_configs.sh "$(PACKAGE_NAME)"
"$(ARCH_MACH)" "$(FLAVOR)"
+	+cd $(REDHAT)/configs; ./build_configs.sh "partial" "$(ARCH_MACH)"
"snip"
+	+cd $(REDHAT)/configs; ./build_configs.sh "$(PACKAGE_NAME)"
"$(ARCH_MACH)" "$(FLAVOR)"
 
 dist-configs-arch: ARCH_MACH = $(MACH)
 dist-configs-arch: dist-configs
diff --git a/redhat/configs/build_configs.sh b/redhat/configs/build_configs.sh
index blahblah..blahblah 100755
--- a/redhat/configs/build_configs.sh
+++ b/redhat/configs/build_configs.sh
@@ -151,9 +151,15 @@ function build_flavor()
 				esac
 			fi
 
-			merge_configs "$arch" "$configs" "$order"
"$flavor" "$count"
+			merge_configs "$arch" "$configs" "$order"
"$flavor" "$count" &
+			waitpids[$count]=$!
+			((count++))
 		fi
 	done < "$control_file"
+
+	for pid in ${waitpids[*]}; do
+		wait $pid
+	done
 }
 
 while read -r line

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635

[Don Zickus (via Email Bridge)]

From: Don Zickus on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635#note_8578...

@prarit - thanks for splitting it out!  It mostly seems fine.  Though I feel
the need to run this locally to review the output.  Instead of failing inline,
it now fails at the end, right?  Also in your CONFIG_HID example, is it
intended to print the messages twice but in a different order?

[Prarit Bhargava (via Email Bridge)]

From: Prarit Bhargava on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1635#note_8578...

> @prarit - thanks for splitting it out!  It mostly seems fine.  Though
> I feel 
the need to run this locally to review the output.

Cool.  You no longer have time to make a cup of coffee while executing
though ;)  It's nice and fast now :)

>  Instead of failing inline, it now fails at the end, right? 
Yes, it fails at the end.  It's slightly different than before but I think
the "full failure" output is more helpful than the inline output.

>  Also in your CONFIG_HID example, is it intended to print the messages
> twice 
but in a different order?

Oh good catch.  Let me look into that.