[OS-BUILD PATCH 0/2] Add new os-build targets: rt-devel and automotive-devel

[Don Zickus (via Email Bridge)]

From: Don Zickus on gitlab.com
Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2732

This is an attempt to automate the rt and automotive devel branches
using the upstream linux-rt-devel tree as the base combined with
os-build.

The overall process isn't too complicated but there are enough steps to
make it compilicated.

Steps:
* map upstream linux-rt-devel to master-rt branch
* update os-build
* grab linux version from master-rt and os-build
* if version the same then merge os-build and master-rt to
  os-build-rt-automated
* else merge linux-stable and master-rt to os-build-rt-automated until
  master-rt is update to os-build
* merge os-build-rt-automated into os-build-rt-devel
* merge os-build-rt-automated into os-build-automotive-devel
* run the generate pending-rhel config scripts on rt-devel and
  automotive-devel

The script has beginning logic to handle rebasing if necessary when the
rt-devel branch transitions from linux-stable to linux master again.

The script has no effect on the day-to-day operations of os-build.  They
are designed to be run from a gitlab cron job and update in the
background.  Once they are deemed stable, adding ARK MRs that target
either os-build-rt-devel or os-build-automotive-devel will be possible
and those branches can start proper parallel developement with os-build.

Signed-off-by: Don Zickus

---
 redhat/scripts/ci/ark-ci-env.sh              |  151 +++++++++++++++++++++++++++
 redhat/scripts/ci/ark-merge-rt.sh            |   96 +++++++++++++++++
 redhat/scripts/ci/ark-update-configs.sh      |    4 +-
 redhat/scripts/genspec/gen_config_patches.sh |   12 +-
 4 files changed, 258 insertions(+), 5 deletions(-)

[OS-BUILD PATCH 1/2] Add target_branch for gen_config_patches.sh

[Don Zickus (via Email Bridge)]

From: Don Zickus

Add target_branch for gen_config_patches.sh

To support new config updates in other brances besides os-build, we need
to be more flexible with the staging branch.  Currently new configs get
added to configs/$date/<new CONFIG>.  This won't work when new branches
like os-build-rt-devel and os-build-automotive-devel get implemented as
they will have duplicate entries here and cause script problems.  Add a
$branch name into the configs path to create separation.

configs/$date/<new CONFIG>  -->
configs/$target_branch/$date/<new CONFIG>

This is not expected to impact anything else.

Signed-off-by: Don Zickus

diff --git a/redhat/scripts/ci/ark-update-configs.sh b/redhat/scripts/ci/ark-update-configs.sh
index blahblah..blahblah 100755
--- a/redhat/scripts/ci/ark-update-configs.sh
+++ b/redhat/scripts/ci/ark-update-configs.sh
@@ -73,7 +73,7 @@ new_head="$(git rev-parse HEAD)"
 [ "$old_head" != "$new_head" ] && CONFIGS_ADDED="1" || CONFIGS_ADDED=""

if test "$CONFIGS_ADDED"; then
-	./redhat/scripts/genspec/gen_config_patches.sh
+	./redhat/scripts/genspec/gen_config_patches.sh "${BRANCH}"
         PUSH_VERB="Pushing"
 else
    printf "No new configuration values exposed from merging %s into $BRANCH\n" "$UPSTREAM_REF"
@@ -84,7 +84,7 @@ echo
 PUSH_STR="branch ${BRANCH} to ${GITLAB_PROJECT_URL}"
 PUSH_CMD="git push gitlab ${BRANCH}"
 PUSH_CONFIG_STR="config update branches"
-PUSH_CONFIG_CMD="for branch in $(git branch | grep configs/"$(date +%F)"); do
+PUSH_CONFIG_CMD="for branch in $(git branch | grep configs/${BRANCH}/"$(date +%F)"); do
    git push \
    	-o merge_request.create \
    	-o merge_request.target="$BRANCH" \
diff --git a/redhat/scripts/genspec/gen_config_patches.sh b/redhat/scripts/genspec/gen_config_patches.sh
index blahblah..blahblah 100755
--- a/redhat/scripts/genspec/gen_config_patches.sh
+++ b/redhat/scripts/genspec/gen_config_patches.sh
@@ -8,6 +8,10 @@
 # information for the relevant maintainers using get_maintainers.pl. This
 # requires that you have $RHMAINTAINERS pointing to a valid maintainer file.

+target_branch=$1
+prev_branch="$(git rev-parse --abbrev-ref HEAD)"
+git checkout "$target_branch"
+
 if ! git show -s --oneline HEAD | grep -q "AUTOMATIC: New configs"; then
    echo "The git HEAD doesn't look like the correct commit"
    exit 1
@@ -116,12 +120,14 @@ for f in "$config_bundles_dir"/*; do
    	rm redhat/configs/pending-rhel/generic/"$line"
    done < "$f"
    # We do a separate branch per config commit
-	if ! git checkout -b "configs/$(date +%F)/$_f"; then
-		printf "Unable to check out configs/%s/%s branch!\n" "$(date +%F)" "$_f"
+	if ! git checkout -b "configs/$target_branch/$(date +%F)/$_f"; then
+		printf "Unable to check out configs/%s/%s/%s branch!\n" "$target_branch" "$(date +%F)" "$_f"
    	exit 1
    fi
    # One file path is done, time to commit!
    git add redhat/configs
    git commit -s -F "$tmpdir"/commit
-	git checkout os-build
+	git checkout "$target_branch"
 done
+
+git checkout "$prev_branch"

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2732

[OS-BUILD PATCH 2/2] Add new os-build targets: rt-devel and automotive-devel

[Don Zickus (via Email Bridge)]

From: Don Zickus

Add new os-build targets: rt-devel and automotive-devel

This is an attempt to automate the rt and automotive devel branches
using the upstream linux-rt-devel tree as the base combined with
os-build.

The overall process isn't too complicated but there are enough steps to
make it compilicated.

Steps:
* map upstream linux-rt-devel to master-rt branch
* update os-build
* grab linux version from master-rt and os-build
* if version the same then merge os-build and master-rt to
  os-build-rt-automated
* else merge linux-stable and master-rt to os-build-rt-automated until
  master-rt is update to os-build
* merge os-build-rt-automated into os-build-rt-devel
* merge os-build-rt-automated into os-build-automotive-devel
* run the generate pending-rhel config scripts on rt-devel and
  automotive-devel

The script has beginning logic to handle rebasing if necessary when the
rt-devel branch transitions from linux-stable to linux master again.

The script has no effect on the day-to-day operations of os-build.  They
are designed to be run from a gitlab cron job and update in the
background.  Once they are deemed stable, adding ARK MRs that target
either os-build-rt-devel or os-build-automotive-devel will be possible
and those branches can start proper parallel developement with os-build.

Signed-off-by: Don Zickus

diff --git a/redhat/scripts/ci/ark-ci-env.sh b/redhat/scripts/ci/ark-ci-env.sh
index blahblah..blahblah 100644
--- a/redhat/scripts/ci/ark-ci-env.sh
+++ b/redhat/scripts/ci/ark-ci-env.sh
@@ -19,6 +19,157 @@ ci_pre_check()
         git diff-index --quiet HEAD || die "Dirty tree, please clean before merging."
 }

+# GitLab can only mirror one project at a time.  This wrapper function does
+# the mirroring for any other branches.
+ark_git_mirror()
+{
+	target_branch=$1
+	upstream_tree=$2
+	source_branch=$3
+	reset_branch=$4
+
+	prev_branch="$(git rev-parse --abbrev-ref HEAD)"
+	remote_branch="$upstream_tree/$source_branch"
+	git checkout "$target_branch"
+	git fetch "$upstream_tree" "$source_branch"
+	if test -z "$reset_branch"; then
+		git merge "$remote_branch" || die "git merge $remote_branch failed"
+	else
+		git reset --hard "$remote_branch" || die "git reset $remote_branch failed"
+	fi
+	git checkout "$prev_branch"
+}
+
+# Merge wrapper in case issues arise
+ark_git_merge()
+{
+	source_branch=$1
+	target_branch=$2
+	reset_branch=$3
+
+	prev_branch="$(git rev-parse --abbrev-ref HEAD)"
+	git checkout "${target_branch}"
+	if test -n "$reset_branch"; then
+		# there are cases when the initial merge is a reset
+		git reset --hard "$source_branch"  || die "git reset $source_branch failed"
+	elif ! git merge -m "Merge '$source_branch' into '$target_branch'" "$source_branch"; then
+		git merge --abort
+		printf "Merge conflict; halting!\n"
+		printf "To reproduce:\n"
+		printf "* git checkout %s\n" "${target_branch}"
+		printf "* git merge %s\n" "${source_branch}"
+		die "Merge conflicts"
+	fi
+
+	git checkout "$prev_branch"
+	return 0
+}
+
+ark_git_rebase()
+{
+	branch=$1
+	upstream=$2
+	base=$3
+
+	prev_branch="$(git rev-parse --abbrev-ref HEAD)"
+	git checkout "${branch}"
+	if ! git rebase --onto "$base" "$upstream"; then
+		git rebase --abort
+		printf "Rebase conflict; halting!\n"
+		printf "To reproduce:\n"
+		printf "* git checkout %s\n" "${branch}"
+		printf "* git rebase --onto %s %s\n" "${base}" "${upstream}"
+		die "Rebase conflicts"
+	fi
+	git checkout "$prev_branch"
+	return 0
+}
+
+ark_update_configs()
+{
+	branch=$1
+	skip_configs=$2
+
+	prev_branch="$(git rev-parse --abbrev-ref HEAD)"
+	git checkout "${branch}"
+
+	# Generates and commits all the pending configs
+	make -j FLAVOR=fedora dist-configs-commit
+	# Skip executing gen_config_patches.sh for new Fedora configs
+
+	old_head="$(git rev-parse HEAD)"
+	make -j FLAVOR=rhel dist-configs-commit
+	new_head="$(git rev-parse HEAD)"
+
+
+	# Converts each new pending config from above into its finalized git
+	# configs/<date>/<config> branch.  These commits are used for Merge
+	# Requests.
+	[ "$old_head" != "$new_head" ] && CONFIGS_ADDED="1" || CONFIGS_ADDED=""
+
+	if test "$CONFIGS_ADDED"; then
+		if test -z "$skip_configs"; then
+			git checkout "$prev_branch"
+			./redhat/scripts/genspec/gen_config_patches.sh "$branch"
+		fi
+	else
+		printf "No new configuration values exposed from "
+		printf "merging %s into $BRANCH\n" "$UPSTREAM_REF"
+	fi
+
+	git checkout "$prev_branch"
+	test -z "$CONFIGS_ADDED" && return 0 || return 1
+}
+
+ark_push_changes()
+{
+	branch=$1
+	skip_configs=$2
+
+	prev_branch="$(git rev-parse --abbrev-ref HEAD)"
+	git checkout "${branch}"
+
+	TMPFILE=".push-warnings"
+	touch "$TMPFILE"
+
+	test "$TO_PUSH" && PUSH_VERB="Pushing" || PUSH_VERB="To push"
+	PUSH_STR="branch ${branch} to ${GITLAB_URL}"
+	PUSH_CMD="git push gitlab ${branch}"
+	PUSH_CONFIG_STR="config update branches"
+	PUSH_CONFIG_CMD="for conf_branch in $(git branch | grep configs/${branch}/"$(date +%F)"); do
+				git push \
+				-o merge_request.create \
+				-o merge_request.target="$branch" \
+				-o merge_request.remove_source_branch \
+				gitlab "$conf_branch" 2>&1 | tee -a $TMPFILE
+				done
+			 "
+
+	#Push branch
+	echo "# $PUSH_VERB $PUSH_STR"
+	echo "$PUSH_CMD"
+	test "$TO_PUSH" && eval "$PUSH_CMD"
+
+	#Push config branches if created
+	if test -z "$skip_configs"; then
+		echo
+		echo "# $PUSH_VERB $PUSH_CONFIG_STR"
+		echo "$PUSH_CONFIG_CMD"
+		test "$TO_PUSH" && eval "$PUSH_CONFIG_CMD"
+	fi
+
+	# GitLab server side warnings do not fail git-push but leave verbose
+	# WARNING messages.  Grep for those and consider it a script
+	# failure.  Make sure all branches are pushed first as follow up
+	# git-pushes may succeed.
+	grep -q "remote:[ ]* WARNINGS" "$TMPFILE" && die "Server side warnings"
+
+	rm "$TMPFILE"
+	git checkout "$prev_branch"
+	return 0
+>>>>>>> 7aaf04024b2f (Add new os-build targets: rt-devel and automotive-devel)
+}
+
 # Common variables for all CI scripts
 UPSTREAM_REF=${1:-"master"}
 BRANCH=${2:-"os-build"}
diff --git a/redhat/scripts/ci/ark-merge-rt.sh b/redhat/scripts/ci/ark-merge-rt.sh
new file mode 100755
index blahblah..blahblah 100755
--- /dev/null
+++ b/redhat/scripts/ci/ark-merge-rt.sh
@@ -0,0 +1,96 @@
+#!/bin/bash
+#
+# This script is intended to sync up the RT and automotive branch (derivative
+# of RT).  It adds the extra twist of detecting the right upstream rt branch
+# to sync with depending on the existance of the next branch.  Sometimes the
+# rt-devel branch waits until -rc1/2 to create new branches.
+# Finally the code handles the rebases in those cases where newer branches
+# are available.
+#
+# Why the complexity?
+# Development branches will need to be periodically rebased unfortunately.
+# Using 'git rebase --onto <new head> <old_head>' only works with one common
+# branch to rebase from not two.  Meanwhile, the -devel branches are formed
+# from two upstream branches, os-build and linux-rt-devel.  The idea is
+# to merge the two branches into a single throwaway branch that can be
+# recreated from scratch anytime then use that as the base for -devel.
+
+set -e
+
+# source common CI functions and variables
+# shellcheck disable=SC1091
+. "$(dirname "$0")"/ark-ci-env.sh
+
+#Upstream RT tree git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git
+UPSTREAM_RT_TREE_NAME="linux-rt-devel"
+DOWNSTREAM_RT_BRANCH="master-rt-devel"
+RT_PREP_BRANCH="os-build-rt-automated"
+RT_DEVEL_BRANCH="os-build-rt-devel"
+AUTOMOTIVE_DEVEL_BRANCH="os-build-automotive-devel"
+
+# grab the os-build base branches
+ark_git_mirror "os-build" "origin" "os-build"
+ark_git_mirror "master" "origin" "master"
+#ark_git_mirror "os-build-stable" "origin" "os-build-stable"
+
+# upstream -rt devel branches are aligned with version numbers and are not
+# always up to date with master.  Figure out which branch to mirror based on
+# version number and existance.  We may have to trigger a rebase.
+
+# what are the current versions of rt-devel and os-build (use 'master' to
+# avoid fedora tagging of kernel-X.Y.0.0.....)
+# use git tags which are always 'vX.Y-rcZ-aaa-gbbbbb' or 'vX.Y-aaa-gbbbbb'
+# where X.Y is the version number that maps to linux-rt's branch of
+# 'linux-X.Y.y'
+RT_DEVEL_VER="$(git describe "$DOWNSTREAM_RT_BRANCH" | cut -c2-4)"
+OS_BUILD_VER="$(git describe master | cut -c2-4)"
+
+# Grab possible rt branches to choose from
+RT_FETCH_BRANCH="linux-${RT_DEVEL_VER}.y-rt"
+RT_REBASE_FETCH_BRANCH="linux-${OS_BUILD_VER}.y-rt"
+RT_MERGE_BRANCH="$RT_FETCH_BRANCH"
+OS_BUILD_BASE_BRANCH="os-build"
+
+if test "$RT_DEVEL_VER" != "$OS_BUILD_VER"; then
+	# does the newer version branch exist in linux-rt-devel yet?
+	if ! git fetch "$UPSTREAM_RT_TREE_NAME" "$RT_REBASE_FETCH_BRANCH"; then
+		# no newer upstream branch to rebase onto, continue with linux-stable branch
+		OS_BUILD_BASE_BRANCH="os-build-stable"
+		RT_REBASE=""
+	else
+		# rebase time
+		RT_MERGE_BRANCH="$RT_REBASE_FETCH_BRANCH"
+		RT_REBASE="yes"
+	fi
+fi
+
+
+## PREP the upstream branches
+# on a rebase, propogate all the git resets
+# fetch the determined rt-devel branch
+ark_git_mirror "$DOWNSTREAM_RT_BRANCH" "$UPSTREAM_RT_TREE_NAME" "$RT_MERGE_BRANCH" "$RT_REBASE"
+# finally merge the two correct branches
+old_RT_PREP_BRANCH="$(git rev-parse $RT_PREP_BRANCH)"  # for ark_git_rebase
+ark_git_merge "$OS_BUILD_BASE_BRANCH" "$RT_PREP_BRANCH" "$RT_REBASE"
+ark_git_merge "$DOWNSTREAM_RT_BRANCH" "$RT_PREP_BRANCH"
+
+## MERGE the upstream branches to the development branches
+if test -n "$RT_REBASE"; then
+	# handle the rebase
+	ark_git_rebase "$RT_DEVEL_BRANCH" "$old_RT_PREP_BRANCH" "$RT_PREP_BRANCH"
+	ark_git_rebase "$AUTOMOTIVE_DEVEL_BRANCH" "$old_RT_PREP_BRANCH" "$RT_PREP_BRANCH"
+fi
+
+## Build -rt-devel branch, generate pending-rhel configs
+ark_git_merge "$RT_PREP_BRANCH" "$RT_DEVEL_BRANCH"
+# don't care if configs were added or not hence '|| true'
+ark_update_configs "$RT_DEVEL_BRANCH" || true
+# skip pushing config update MRs, keep them in pending-rhel
+ark_push_changes "$RT_DEVEL_BRANCH" "skip"
+
+## Build -automotive-devel branch, generate pending-rhel configs
+ark_git_merge "$RT_PREP_BRANCH" "$AUTOMOTIVE_DEVEL_BRANCH"
+# don't care if configs were added or not hence '|| true'
+ark_update_configs "$AUTOMOTIVE_DEVEL_BRANCH" || true
+# skip pushing config update MRs, keep them in pending-rhel
+ark_push_changes "$AUTOMOTIVE_DEVEL_BRANCH" "skip"

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2732