Re: A modest proposal: Pungi 4 compose process (what we call composes, when we do them, what information we need about them)
On Mon, 2016-02-22 at 10:04 -0800, Adam Williamson wrote:
Also not visible in the mockup: "compose override" packages are *always included in all types of compose*. This is the concept Dennis and I came up with for handling blocker / freeze exception fixes; it's just a more formal version of the current process, really, whereby we mark packages that should be pulled into composes. At present these are only pulled into TCs and RCs, they never appear in the old-style "nightly composes". I believe we should *always* pull them in; it makes the system a good deal simpler.
I'm a bit confused here. The override packages were used for TCs and RCs, but not for Live nightlies done in Koji. Pungi4 will now do all of that as part of a single compose, daily, right? So are you just saying those override packages will end up used in all compose artifacts produced, and we no longer need to care about the TC+RC vs Live nightly difference? In that case that's great.
Yes, that is what I'm suggesting. If we still have the difference, it makes things quite messy primarily for constructing an "order" of composes, because say we do a snapshot the morning after an RC is blessed; if we haven't done the stable push by then, the snapshot will inarguably have been built *later* than the RC, but will contain older packages than it. So what's the order? This is how things work ATM, and I kinda hate it. And it just makes sense to me that "compose override" packages should wind up in all the composes, anyhow. There's no reason to leave them out of snapshots.
So Dennis explained that, unfortunately, we can't do this at least for now.
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
In theory we could consider putting the "compose override" packages into the images but not the repositories, but we might still think that was a mistake, and in any case, Dennis says, the tools just don't work like that at present. The images are built from the packages in the repos; any "override" packages for a compose *must* be in the repositories for that compose, practically speaking.
So for the medium term I think we're stuck with including "override" packages in "candidates" but not "snapshots", and things that do stuff with composes are going to have to understand that distinction.
It also means that we're going to have keep doing "snapshots" at the same time as "candidates", as that's how we sync out the stuff that gets pushed stable during freezes to the mirrors.
So we're kinda gonna have to treat the "snapshots" and the "candidates" as two separate streams of things, and accept that we can't entirely reliably construct a sequence of them mixed together in some senses (because a "snapshot" built after a "candidate" may have older packages, just as is the case now).
There are various ways we can think about dealing with this in the long term, but for now at least we'll just have to live with it.
On Mon, Feb 22, 2016 at 02:11:10PM -0800, Adam Williamson wrote:
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
On Thu, 2016-02-25 at 10:45 -0500, Matthew Miller wrote:
On Mon, Feb 22, 2016 at 02:11:10PM -0800, Adam Williamson wrote:
When I think about 'composes' I tend to just think about a sort of isolated thing with a bunch of images in it, but of course that's not (all) a "snapshot compose" is. The snapshot composes are what will ultimately be staged out to the public mirrors as the 'official' tree for the release. We cannot put the "compose override" packages into the repositories in those trees, because to do so would be effectively to push them out as stable updates.
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
Overrides are just the new name for what we've been doing so far with the /bleed repo; it's the problem where we can't wait for an update to go through the whole review and stable push process before we include it in a TC/RC, so we shove it in a side repo and include that side repo in the compose.
In some cases we're already pretty sure the package is good and we just want to short-circuit the inherent delays in the Bodhi process; if we had a magic button which instantly made the package show up in the 'stable' repo that'd be fine. So for those cases, sure, it wouldn't be a problem if the overrides got pulled into the 'stable' repo as part of the compose.
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
On Thu, Feb 25, 2016 at 11:34:57AM -0800, Adam Williamson wrote:
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
[...]
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
Well, ugh. Now I understand but am not any happier. :)
What about having _this_ type of compose be a special outside the normal system case, and after testing those separately, go back to the other situation (the short circuit)?
Instead of including overrides in "candidates", include them only in... "override tests", which would only be used to test that the particular override package was not a terrible idea.
Then, the _next_ snapshot could be evaluated / validated in full.
On Thu, 2016-02-25 at 14:57 -0500, Matthew Miller wrote:
On Thu, Feb 25, 2016 at 11:34:57AM -0800, Adam Williamson wrote:
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
[...]
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
Well, ugh. Now I understand but am not any happier. :)
What about having _this_ type of compose be a special outside the normal system case, and after testing those separately, go back to the other situation (the short circuit)?
Sure, we could do that. We'd need a name for the "special outside the normal system" cases, of course. Something something "compose", maybe? Well, we need to "test" it, so we could call it a "test compose"?
waaaaaaitaminute...
Instead of including overrides in "candidates", include them only in... "override tests", which would only be used to test that the particular override package was not a terrible idea.
Then, the _next_ snapshot could be evaluated / validated in full.
OK, I think I see what you're driving at, but then you kinda run into the mechanics of the process. Pungi 4 composes are *faster* but they're still not *fast*, they take a couple of hours at minimum. We do not yet have sufficient automated testing for everything we might want to do an "override test" for, so there needs to be a manual step in there somewhere. And we really don't want to be sneak-pushing things into the "stable" repo behind Bodhi's back, the whole design of the system is that anything in the official /development/(release)/ repos went through Bodhi, so we'd still ultimately need to stuff the things we approved via "override tests" through the Bodhi system, with its delays.
Plus I'm not sure the mechanics of doing one "override test" per blocker/FE-fixing update quite work...I think the way we can actually implement this "override" mechanism is via Koji tags, but Koji tags aren't free, so we can't auto-create a new one for every blocker/FE- fixing update that shows up, I don't think. We'd have to have one tag that we'd be continually applying to the packages from a single update, doing a compose from, scrubbing, applying to the packages from the *next* blocker/FE-fixing update, rinse and repeat...well, I mean, I guess that could work?
So, hum, I guess it's possible. I'm not entirely sure it'd work better than the current system of building candidates with all "override" packages and pushing the overridden packages stable as fast as we can, though. I can see a few advantages and disadvantages...we don't test *interaction* between the overridden updates in this system, and it involves producing and shipping around a *lot* of bits...not sure what Dennis would think.
On Thursday, February 25, 2016 02:57:04 PM Matthew Miller wrote:
On Thu, Feb 25, 2016 at 11:34:57AM -0800, Adam Williamson wrote:
Maybe I'm Dumb (™), but I'm not following the problem with the "because". *Shouldn't* the override packages be the equivalent of stable updates?
No.
[...]
In other cases, though, we need to try out a potential fix for an issue in a compose, but it might not be good. We might spin up the compose and find it broke everything. We definitely *don't* want to be short- circuiting Bodhi for those cases. So long as the 'overridden' package only shows up in the images, we can fairly safely drop it again if it turns out to be bad. If the 'overridden' package also went out to the repos, people following Branched would get it as a regular update, and we then can't just silently pull it because it leaves them out on a limb (we have always resisted doing this very hard in the past; it very occasionally happens, but we always hate doing it, we certainly don't want to make it a part of common practice).
Well, ugh. Now I understand but am not any happier. :)
In a world where we only push out bits that have passed openqa we should detect the types of breakages that we wouldn't want to ship. It really boils down to if we put the compose on /pub/fedora or not.
Dennis
What about having _this_ type of compose be a special outside the normal system case, and after testing those separately, go back to the other situation (the short circuit)?
Instead of including overrides in "candidates", include them only in... "override tests", which would only be used to test that the particular override package was not a terrible idea.
Then, the _next_ snapshot could be evaluated / validated in full.
rel-eng@lists.fedoraproject.org
-
Adam Williamson -
Dennis Gilmore -
Matthew Miller