#6313: atomic host: reorienting ostree commits to match 2 week cadence -----------------------------+------------------------ Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Keywords: | Blocked By: Blocking: | -----------------------------+------------------------ Current as I understand it, we have a "two week" process for images (cloud, ISO etc.), but the OSTree commits are made at the default Fedora at-most-once-a-day cadence.
I would like to propose doing only one released OSTree commit matching the two week release. Meaning `atomic host upgrade` would exactly what the images give. The advantage of this is predictability.
NOTE: we need an exception for critical async security errata.
The /testing/ ref though would continue at its current rate - and ideally go even faster. I would be a lot happier if we skipped the manual once-a-day RPM signing and did direct Koji -> /testing/.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
One thing we can do with change is for coherent versioning of the images and released tree commits. Right now the tree commits are versioned e.g. `23.X` where X simply increments, but if we did one-commit-per-release we could have the OSTree commits match the image versioning. Where "image versioning" is currently just a date stamp.
The high level rationale I'm filing this now is that `rpm-ostree deploy` is introduced in F23, which allows a user to deploy a specific (old or new) commit. With that functionality, effort put into server side versioning becomes significantly more useful.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
This makes sense to me.
The /testing/ ref though would continue at its current rate - and
ideally go even faster. I would be a lot happier if we skipped the manual once-a-day RPM signing and did direct Koji -> /testing/.
What if we had a /devel/ ref that behaved that way, in addition to the signed-RPMs /testing/?
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Replying to [comment:2 mattdm]:
What if we had a /devel/ ref that behaved that way, in addition to the
signed-RPMs /testing/?
That'd be OK by me.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Another reason to do this is that it would make static delta management easier - we'd have fewer deltas, and people would be more likely to hit them.
We could still of course implement deltas for N-2 -> N and such, but it is subject to combinatorics the more variants one tries to support, which hurts more for once-a-day releases.
Whereas for development streams, the "archive-z2" model is okay enough.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by lmacken):
Right now bodhi is responsible for these regular commits, which is decoupled from the 2-week-atomic workflow entirely. Until we have a central releng orchestration framework that will handle $EVERYTHING at whatever interval we desire, we'll have to tweak what we have in place now.
So, during an updates push we could have bodhi query to see if there is a new 2-week-atomic release, and then kick off the stable ostree compose. The testing composes would continue to churn along as they do, and if we want a branch with unsigned testing stuff from koji, then it doesn't really make sense to do that in bodhi. The downside here being there will be a slight lag between the images and the trees, since I'm assuming different people kickoff the twoweek & updates pushes on their own schedules.
For bodhi to query the latest 2-week-atomic release, it could potentially: * Parse https://pagure.io/mark-atomic-bad/raw/master/f/good-builds.json * Query datagrepper for the last `atomic.twoweek.complete` fedmsg. https://apps.fedoraproject.org/datagrepper/raw%5C?topic%5C=org.fedoraproject...
Alternatively, we could move the stable ostree composes out of bodhi and into the `push-two-week-atomic.py` script (https://pagure.io/releng/blob/master/f/scripts/push-two-week-atomic.py), and have bodhi only do `updates-testing` composes.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Your last paragraph seems like the easiest to implement to me - we will just have to remember to watch out for critical security updates.
One additional reason to slow down the ostree commits to 2 weeks - it would allow us to put human-written content in the OSTree commit message, which so far we haven't used. But I could imagine doing so, and then we could display it on the client.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Colin's executive summary: Create a new additional ref:
fedora-atomic/f23/x86_64/batch/docker-host
Allow users to rebase to this who want it.
- Change the two-week-atomic.py script to generate this (new rpm-ostree run, need to sed the json for the new ref) - then sed the kickstarts to pull from it (but still use a *target* ref of fedora-atomic/f23/x86_64/docker-host ?) - Decide on a version numbering scheme...e.g. also use a datestamp? So ostree.version=23.20160106, need to pass this to rpm-ostree
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Let's track the devel/ discussion in https://fedorahosted.org/rel- eng/ticket/6350
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+----------------------- Changes (by dustymabe):
* cc: dusty@… (added)
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
From an Atomic Host perspective, this is the highest priority ticket for us. Is this likely to be looked at in e.g. the next month?
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by maxamillion):
Apologies for the massive delay on a lot of this, the Docker Layered Image Build has taken priority over my ability to work on this and it's been quite the journey (but initial phase is coming to a close and I'll be able to work on other things).
I will be revisiting this very soon (likely very shortly after Red Hat Summit 2016). I have plans to rewrite the current 2-week release as an Ansible module, this will allow for the release to be more flexible (at least that's the end goal). This is part of the [https://fedoraproject.org/wiki/Changes/ReleaseEngineeringAutomationWorkflowE... | RelEng Automation Change]. If it becomes apparent very early on (I'm going to give myself a 2 day time limit once I start working on it) that this is going to not be feasible as a near-term goal (within the next month), then I will make the modifications as requested by Walters in Comment #7 to the two-week-release script to satisfy the need while the larger plan gets worked on. Long term the goal is to have this be a fully automated process that we could trigger based on any event that comes across the fedmsg bus that we see fit to trigger this action (CI or otherwise).
I'm open to feedback and look forward to others thoughts on the plan.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by imcleod):
A quick update from a conversation Colin and I had last week. To fully clarify and summarize what we'd like to see:
* The delivered/highlighted output is a combination ostree compose and image build(s) based on that compose, done every two weeks
* Mirrors should contain the current image, the current compose and 3 prior composes (1 image and 4 ostree composes in total) - older composes may be pruned at will
* Anything done more frequently should be for dev/test only, need not be mirrored and can be pruned at will
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
I am willing to give $1000 (one thousand dollars) of my own personal money to someone who demonstrates commitment to implementing this ticket, and following through on related infrastructure in Fedora. Someone who posts or contributes to the upstream ostree list (or atomic-devel@), and codebase, and helps push forward things related to the OSTree side of Atomic Host release engineering, such as static deltas between these updates.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Example bug report type I'd like to see: https://bugzilla.gnome.org/show_bug.cgi?id=765701
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by walters):
Example implementation:
https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel
See the Jenkins jobs for some examples of how the alpha/continuous branching flow is separated including generating separate images.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by maxamillion):
I forgot to update this, but I have a [https://lists.fedoraproject.org/archives/list/rel- eng@lists.fedoraproject.org/thread/B73P6EDFVKZCMHJ75LBOWUMOJTI34N7N/ proposal out to the rel-eng mailing list] that we're currently hashing out now.
#6313: atomic host: reorienting ostree commits to match 2 week cadence ------------------------------+----------------------- Reporter: walters | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 23 Final | Component: other Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
Hey Adam, what is the status of this? I remember discussing the proposal on the mailing list, but then it seems like it trailed off.
rel-eng@lists.fedoraproject.org
-
Fedora Release Engineering