Atomic Updates - do we follow traditional model or a new one?

[Joe Brockmeier]

Hey all,

One of the things that came out of the weekly meeting with infra/releng
and folks working on Atomic is what I think may be a mis-match in
expectations on upgrades/release process for the Atomic host.

As called out in the host definition[1] Atomic is planned as a rolling
stream of updates - and users are expected to move to the next release
in the stream rather than staying on a specific version or having to
carry an overlapping stream.

That is: If you're on Fedora 21 Atomic, when Fedora 22 Atomic is
released then that would be what you switch to - not a Fedora 21 tree.

I know for CentOS Atomic we won't be maintaining a set of overlapping
releases, and I don't think RHEL will either. That sort of defeats the
model, really. (Also, there's not exactly an upgrade for Atomic
something like F21->F22 if there are multiple trees, eventually you
would have to manually switch trees if we were producing overlapping sets.)

In discussing this in today's meeting[2], Dennis suggested we'd need to
go to FESCo to get agreement that we can pursue the non-overlapping
model. Before I do that, I wanted to make sure we were all in agreement
that is the way to go.

Thoughts, comments, flames?

[1] https://gist.github.com/jzb/0f336c6f23a0ba145b0a
[2]
http://meetbot.fedoraproject.org/atomic/2014-10-07/atomic.2014-10-07-18.0...
-- 
Joe Brockmeier | Principal Cloud & Storage Analyst
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/

[James Antill]

On Tue, 2014-10-07 at 17:33 -0500, Joe Brockmeier wrote:
>  Hey all,
>  
>  One of the things that came out of the weekly meeting with infra/releng
>  and folks working on Atomic is what I think may be a mis-match in
>  expectations on upgrades/release process for the Atomic host.
>  
>  As called out in the host definition[1] Atomic is planned as a rolling
>  stream of updates - and users are expected to move to the next release
>  in the stream rather than staying on a specific version or having to
>  carry an overlapping stream.
>  
>  That is: If you're on Fedora 21 Atomic, when Fedora 22 Atomic is
>  released then that would be what you switch to - not a Fedora 21 tree.
>  
>  I know for CentOS Atomic we won't be maintaining a set of overlapping
>  releases, and I don't think RHEL will either. That sort of defeats the
>  model, really. 
 tl;dr I mostly agree with dgilmore here, and I think FESCO should too
at least in the short term ... but the users will be the final arbiters.

 At least one view of the model is that you have atomic upgrades, and
thus. rollback/downgrades. This fits perfectly with the f21/f21/f23
release model (although "rpm-ostree rebase" is very surprising when it
deletes your refs, you can still atomic downgrade).
 Certainly one of the benefits of ostree, to me, is that it should be
possible to freely move between N stable releases.

 I would also disagree strongly that RHEL will ever follow this model. I
would bet a huge amount of money that if customers are using the
official trees at all then enough of them will be willing to pay for
rhel8 after rhel9 goes live that it'll just happen.
 I'd also be willing to bet more than a few beers that if/when you
propose to PM that "rpm-ostree upgrade" magically changes the client
from rhel8 to rhel9 ... you will not be getting smiles and nods.

 As far as Fedora goes, it could be argued that it'd work better than
RHEL ... but again you have the change date problem. More than a few
people will want to move to Atomic 22 ASAP, the vast majority would
probably be happy moving as soon as F22 is released and then there are
some who'd prefer to wait N "seconds" after that.
 The only way to accommodate everyone here is going to be overlapping
releases. However maybe the breakdown is that 90% of users are happy
with it changing on F22 release day, so you can get away with just doing
that. Ironically though, I think the easiest way to get these stats. is
to have overlapping releases for a bit and see when the users move via.
the download stats. ... the less easy way being just doing it and see
how much it blows up :).

 As a minor note, if we do this we'll want some autoprune type
configuration added on the ostree layer.

>   (Also, there's not exactly an upgrade for Atomic
>  something like F21->F22 if there are multiple trees, eventually you
>  would have to manually switch trees if we were producing overlapping sets.) 
 I don't see how this would be different at all, we are just changing
which commit we use ... why would it matter if it came from a different
ref or not? 

>  In discussing this in today's meeting[2], Dennis suggested
> we'd need to
>  go to FESCo to get agreement that we can pursue the non-overlapping
>  model. Before I do that, I wanted to make sure we were all in agreement
>  that is the way to go.
>  
>  Thoughts, comments, flames?
>  
>  [1] https://gist.github.com/jzb/0f336c6f23a0ba145b0a
>  [2]
>  http://meetbot.fedoraproject.org/atomic/2014-10-07/atomic.2014-10-07-18.0...

[Michael Hampton]

On 10/08/2014 12:26 AM, James Antill wrote:
>   I would also disagree strongly that RHEL will ever follow this
> model. I
>  would bet a huge amount of money that if customers are using the
>  official trees at all then enough of them will be willing to pay for
>  rhel8 after rhel9 goes live that it'll just happen.
>   I'd also be willing to bet more than a few beers that if/when you
>  propose to PM that "rpm-ostree upgrade" magically changes the client
>  from rhel8 to rhel9 ... you will not be getting smiles and nods. 
Certainly not from 8 to 9. But I'd love to be able to do this from 8.1 to 8.2 to 8.3
etc.

[James Antill]

On Wed, 2014-10-08 at 00:41 -0400, Michael Hampton wrote:
>  On 10/08/2014 12:26 AM, James Antill wrote:
>  >  I would also disagree strongly that RHEL will ever follow this model. I
>  > would bet a huge amount of money that if customers are using the
>  > official trees at all then enough of them will be willing to pay for
>  > rhel8 after rhel9 goes live that it'll just happen.
>  >  I'd also be willing to bet more than a few beers that if/when you
>  > propose to PM that "rpm-ostree upgrade" magically changes the client
>  > from rhel8 to rhel9 ... you will not be getting smiles and nods.
>  
>  Certainly not from 8 to 9. But I'd love to be able to do this from 8.1 to 8.2 to 8.3
> etc. 
 That's true, it's much more likely we'll be able to do just a single
rhel8 release. Although, even then, while I've never wanted/needed to
use a minor release personally, they presumably exist because of demand.

[Matthew Miller]

On Wed, Oct 08, 2014 at 12:26:43AM -0400, James Antill wrote:
>   At least one view of the model is that you have atomic upgrades,
> and
>  thus. rollback/downgrades. This fits perfectly with the f21/f21/f23
>  release model (although "rpm-ostree rebase" is very surprising when it
>  deletes your refs, you can still atomic downgrade).
>   Certainly one of the benefits of ostree, to me, is that it should be
>  possible to freely move between N stable releases. 
I agree. And since we're making the packages from which the Atomic versions
will be composed, what's the _downside_ of making releases available as
distinct releases? Since it's produced from RPMs that we're making
automatically, isn't it basically something we can offer to users with very
little effort?

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Joe Brockmeier]

On 10/08/2014 08:55 AM, Matthew Miller wrote:
>  I agree. And since we're making the packages from which the
> Atomic versions
>  will be composed, what's the _downside_ of making releases available as
>  distinct releases? Since it's produced from RPMs that we're making
>  automatically, isn't it basically something we can offer to users with very
>  little effort? 
I see a few downsides:

* Having to test multiple releases. Not sure that quite fits under "very
little effort." (Maybe someday when we have automated testing, that will
be different.)

* Undermining the model that Atomic is supposed to support - which is
"you only care about the host as a way to run containers". As long as
Fedora 22 - 23 - 24 don't break anything within the set of functionality
they're supposed to offer (running containers, offering orchestration,
etc.) users *should not care* that it's Fedora 21 or 22.

I realize rpm-ostree makes some interesting things possible around
switching between trees - and that is something to explore for other
products, I think. But the use case here is supposed to be "my apps live
in containers. I need an OS that runs containers and maybe offers some
orchestration features. I'm not going to be tinkering with the host
itself, it's a pre-set unit and offers X,Y, and Z features. As long as
those continue to work, the details don't matter."

Telling users to worry about whether it's Fedora 21 or 22 or whatever
kind of reinforces old habits that this is supposed to get us away from.

Best,

jzb
-- 
Joe Brockmeier | Principal Cloud & Storage Analyst
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/

[Matthew Miller]

On Wed, Oct 08, 2014 at 09:19:41AM -0500, Joe Brockmeier wrote:
>  I see a few downsides:
>  * Having to test multiple releases. Not sure that quite fits under "very
>  little effort." (Maybe someday when we have automated testing, that will
>  be different.) 
I'm not sure that follows... we'll have to test the new release no matter
what, and the old release is just the previous one we tested.

>  * Undermining the model that Atomic is supposed to support - which
> is
>  "you only care about the host as a way to run containers". As long as
>  Fedora 22 - 23 - 24 don't break anything within the set of functionality
>  they're supposed to offer (running containers, offering orchestration,
>  etc.) users *should not care* that it's Fedora 21 or 22. 
Yeah, but... reality isn't going to be that pretty. Also, if we're looking
to bring Atomic to more of Fedora than just the cloud guest, I think people
will find this less scary.

>  Telling users to worry about whether it's Fedora 21 or 22 or
> whatever
>  kind of reinforces old habits that this is supposed to get us away from. 
Especially as this develops, if we want to get people actually using it but
still be able to make big changes to how it works (especially wrt
orchestration and higher levels), having six month release points where
users can transition at their own pace seems appealing. We can maybe look at
discontinuing older versions sooner than the normal 7 month overlap.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Joe Brockmeier]

On 10/08/2014 09:36 AM, Matthew Miller wrote:
>  On Wed, Oct 08, 2014 at 09:19:41AM -0500, Joe Brockmeier wrote:
> > I see a few downsides:
> > * Having to test multiple releases. Not sure that quite fits under "very
> > little effort." (Maybe someday when we have automated testing, that will
> > be different.)
>  
>  I'm not sure that follows... we'll have to test the new release no matter
>  what, and the old release is just the previous one we tested. 
Right, but we need to keep regenerating the tree/image, right? And
actually testing it?

So, when we have a thing like shellshocked, we would be testing Atomic
20 and 21, instead of just 21.

> > * Undermining the model that Atomic is supposed to support -
> which is
> > "you only care about the host as a way to run containers". As long as
> > Fedora 22 - 23 - 24 don't break anything within the set of functionality
> > they're supposed to offer (running containers, offering orchestration,
> > etc.) users *should not care* that it's Fedora 21 or 22.
>  
>  Yeah, but... reality isn't going to be that pretty. Also, if we're looking
>  to bring Atomic to more of Fedora than just the cloud guest, I think people
>  will find this less scary.
>  
> > Telling users to worry about whether it's Fedora 21 or 22 or whatever
> > kind of reinforces old habits that this is supposed to get us away from.
>  
>  Especially as this develops, if we want to get people actually using it but
>  still be able to make big changes to how it works (especially wrt
>  orchestration and higher levels), having six month release points where
>  users can transition at their own pace seems appealing. We can maybe look at
>  discontinuing older versions sooner than the normal 7 month overlap. 
Kinda agree, but OTOH, we'll be "taking something away" if we change the
model later.

But, I'm getting the feeling I'm the lone voice here, so I'll go with
the flow.

Best,

jzb
-- 
Joe Brockmeier | Principal Cloud & Storage Analyst
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/

[Matthew Miller]

On Wed, Oct 08, 2014 at 09:48:19AM -0500, Joe Brockmeier wrote:
>  > I'm not sure that follows... we'll have to test the new
> release no matter
>  > what, and the old release is just the previous one we tested.
>  Right, but we need to keep regenerating the tree/image, right? And
>  actually testing it? 
It should be regenerated from RPMs that went through testing already,
though. We ideally should (but do not!) do automatic integration testing
anywhere else.

>  So, when we have a thing like shellshocked, we would be testing
> Atomic
>  20 and 21, instead of just 21. 
Yeah, but in the scope of what all that effort entails, that's relatively
small.



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Colin Walters]

On Wed, Oct 8, 2014, at 12:26 AM, James Antill wrote:
>   At least one view of the model is that you have atomic upgrades, and
>  thus. rollback/downgrades. This fits perfectly with the f21/f21/f23
>  release model (although "rpm-ostree rebase" is very surprising when it
>  deletes your refs, you can still atomic downgrade). 
If it *didn't* do that, the storage space from the old tree would
continue
to be taken up over time unless you took explicit action.

We could add an option to keep it though?  I'll admit that some of
these choices in the command line tooling are a bit arbitrary - the
underlying library and model is extremely flexible though.

>   Certainly one of the benefits of ostree, to me, is that it should
> be
>  possible to freely move between N stable releases. 
Right.

At the moment, we're actually in a model where updates are
initiated manually.  A feature of doing automatic updates would
be a good place to also implement this behavior of automatically
tracking the latest.

(If we *do* have multiple branches, then an interesting question
 arises as to how the client learns about new releases)