Need owner of Atomic 2 Week releases

[Amanda Carter]

Hey folks, creating a separate thread for this longer term discussion. We're getting
ready to release our first 2 week atomic update on Tuesday and Dusty Mabe has raised 2
potential release blockers that were not part of automated testing. It's good that he
caught them, but it's also a bit of a stroke of luck. Since there is no official QE
for this release, who should own verifying that there are no release blocking bugs prior
to every automated release and escalating if there are? If no one raises the blocker,
we'll have no way to block the release.

This is something that we need an answer to fairly quickly since we don't even have
confidence that the current release is good other than the current reports. And we'll
be taking this plunge again in just 2 weeks.

Thanks for your attention to this,

-- 
Amanda Carter

[Dusty Mabe]

On 11/13/2015 10:41 AM, Amanda Carter wrote:
>  Hey folks, creating a separate thread for this longer term
> discussion. We're getting ready to release our first 2 week atomic update on Tuesday
> and Dusty Mabe has raised 2 potential release blockers that were not part of automated
> testing. It's good that he caught them, but it's also a bit of a stroke of luck.
> Since there is no official QE for this release, who should own verifying that there are no
> release blocking bugs prior to every automated release and escalating if there are? If no
> one raises the blocker, we'll have no way to block the release.
>
>  This is something that we need an answer to fairly quickly since we don't even have
> confidence that the current release is good other than the current reports. And we'll
> be taking this plunge again in just 2 weeks.
>
>  Thanks for your attention to this,
>  
We should really have a blocker review process for these similar to the 
ones we have for normal Fedora releases. Primary items that we should be 
concerned with in the review are rpms like the following:

kernel:
docker:
etcd:
kubernetes:
cloud-init:
ostree:
atomic:
cockpit:


Dusty

[Josh Boyer]

On Fri, Nov 13, 2015 at 11:35 AM, Dusty Mabe <dusty(a)dustymabe.com&gt; wrote:
>  On 11/13/2015 10:41 AM, Amanda Carter wrote:
> >
> > Hey folks, creating a separate thread for this longer term discussion.
> > We're getting ready to release our first 2 week atomic update on Tuesday and
> > Dusty Mabe has raised 2 potential release blockers that were not part of
> > automated testing. It's good that he caught them, but it's also a bit of a
> > stroke of luck. Since there is no official QE for this release, who should
> > own verifying that there are no release blocking bugs prior to every
> > automated release and escalating if there are? If no one raises the blocker,
> > we'll have no way to block the release.
> >
> > This is something that we need an answer to fairly quickly since we don't
> > even have confidence that the current release is good other than the current
> > reports. And we'll be taking this plunge again in just 2 weeks.
> >
> > Thanks for your attention to this,
> >
>
>  We should really have a blocker review process for these similar to the ones
>  we have for normal Fedora releases. Primary items that we should be 
Are these 2 week images built from updates that are currently in
stable, or are they built from updates-testing as well?

I ask because it matters for things outside of atomic.  The release
blocker review process works because it catches things _before_ they
are released for general availability.  If the 2 week atomic images
are only composed from already stable updates, then the packages are
already out there in the project otherwise.

So if you have something "blocker" in an atomic image, you are now
forced to wait for an update to make it through the entire fedora
updates process before you can ship your 2 week image.  For something
like the kernel, that might very well mean you don't ship a 2 week
image because the fix is not in stable in sufficient time.

The atomic images might be better served by doing tests on
updates-testing packages that are included to ensure that blockers
don't otherwise show up as a surprise.  I would also recommend
reaching out to the package owners for each important package in
advance so that the atomic sig is aware of what is planned for updates
and such.

josh

[Mike McGrath]

On Fri, Nov 13, 2015 at 10:50 AM, Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  On Fri, Nov 13, 2015 at 11:35 AM, Dusty Mabe
> <dusty(a)dustymabe.com&gt; wrote:
> >
> >
> > On 11/13/2015 10:41 AM, Amanda Carter wrote:
> >>
> >> Hey folks, creating a separate thread for this longer term discussion.
> >> We're getting ready to release our first 2 week atomic update on Tuesday and
> >> Dusty Mabe has raised 2 potential release blockers that were not part of
> >> automated testing. It's good that he caught them, but it's also a bit of
> a
> >> stroke of luck. Since there is no official QE for this release, who should
> >> own verifying that there are no release blocking bugs prior to every
> >> automated release and escalating if there are? If no one raises the blocker,
> >> we'll have no way to block the release.
> >>
> >> This is something that we need an answer to fairly quickly since we don't
> >> even have confidence that the current release is good other than the current
> >> reports. And we'll be taking this plunge again in just 2 weeks.
> >>
> >> Thanks for your attention to this,
> >>
> >
> > We should really have a blocker review process for these similar to the ones
> > we have for normal Fedora releases. Primary items that we should be
>
>  Are these 2 week images built from updates that are currently in
>  stable, or are they built from updates-testing as well?
>
>  I ask because it matters for things outside of atomic.  The release
>  blocker review process works because it catches things _before_ they
>  are released for general availability.  If the 2 week atomic images
>  are only composed from already stable updates, then the packages are
>  already out there in the project otherwise.
>
>  So if you have something "blocker" in an atomic image, you are now
>  forced to wait for an update to make it through the entire fedora
>  updates process before you can ship your 2 week image.  For something
>  like the kernel, that might very well mean you don't ship a 2 week
>  image because the fix is not in stable in sufficient time.
>
>  The atomic images might be better served by doing tests on
>  updates-testing packages that are included to ensure that blockers
>  don't otherwise show up as a surprise.  I would also recommend
>  reaching out to the package owners for each important package in
>  advance so that the atomic sig is aware of what is planned for updates
>  and such.
>
>  josh 

I'd also point out that the nature of Atomic's rollback features make
it better suited to less testing in that they can always roll back.
The problem that I see comes if there's a bug that lasts through
several releases causing a fairly bad trust scenario w/ upgrades.

-- 
Mike McGrath | mmcgrath(a)redhat.com | [phone]
Atomic | Red Hat Chicago | http://projectatomic.io/

[Mike McGrath]

On Fri, Nov 13, 2015 at 12:56 PM, Michael McGrath <mmcgrath(a)redhat.com&gt; wrote:
>  On Fri, Nov 13, 2015 at 10:50 AM, Josh Boyer
> <jwboyer(a)fedoraproject.org&gt; wrote:
> > On Fri, Nov 13, 2015 at 11:35 AM, Dusty Mabe <dusty(a)dustymabe.com&gt; wrote:
> >>
> >>
> >> On 11/13/2015 10:41 AM, Amanda Carter wrote:
> >>>
> >>> Hey folks, creating a separate thread for this longer term discussion.
> >>> We're getting ready to release our first 2 week atomic update on Tuesday
> and
> >>> Dusty Mabe has raised 2 potential release blockers that were not part of
> >>> automated testing. It's good that he caught them, but it's also a bit
> of a
> >>> stroke of luck. Since there is no official QE for this release, who should
> >>> own verifying that there are no release blocking bugs prior to every
> >>> automated release and escalating if there are? If no one raises the blocker,
> >>> we'll have no way to block the release.
> >>>
> >>> This is something that we need an answer to fairly quickly since we
> don't
> >>> even have confidence that the current release is good other than the current
> >>> reports. And we'll be taking this plunge again in just 2 weeks.
> >>>
> >>> Thanks for your attention to this,
> >>>
> >>
> >> We should really have a blocker review process for these similar to the ones
> >> we have for normal Fedora releases. Primary items that we should be
> >
> > Are these 2 week images built from updates that are currently in
> > stable, or are they built from updates-testing as well?
> >
> > I ask because it matters for things outside of atomic.  The release
> > blocker review process works because it catches things _before_ they
> > are released for general availability.  If the 2 week atomic images
> > are only composed from already stable updates, then the packages are
> > already out there in the project otherwise.
> >
> > So if you have something "blocker" in an atomic image, you are now
> > forced to wait for an update to make it through the entire fedora
> > updates process before you can ship your 2 week image.  For something
> > like the kernel, that might very well mean you don't ship a 2 week
> > image because the fix is not in stable in sufficient time.
> >
> > The atomic images might be better served by doing tests on
> > updates-testing packages that are included to ensure that blockers
> > don't otherwise show up as a surprise.  I would also recommend
> > reaching out to the package owners for each important package in
> > advance so that the atomic sig is aware of what is planned for updates
> > and such.
> >
> > josh
>
>
>  I'd also point out that the nature of Atomic's rollback features make
>  it better suited to less testing in that they can always roll back.
>  The problem that I see comes if there's a bug that lasts through
>  several releases causing a fairly bad trust scenario w/ upgrades.
>
>  --
>  Mike McGrath | mmcgrath(a)redhat.com | [phone]
>  Atomic | Red Hat Chicago | http://projectatomic.io/ 
One other thing I forgot to mention, Dennis requested some additional
QA for this to do some basic smoke tests.  We're working on setting
something up that can be used for a basic smoke test until something
more permanent can be setup.  I've cc'd Ari who's familiar.

Ari, can you give a quick update on what your team is thinking at this
point?  Something automated would help greatly for at least one part
of the 2 week release (though bugs would still need to be handled).

-- 
Mike McGrath | mmcgrath(a)redhat.com | [phone]
Atomic | Red Hat Chicago | http://projectatomic.io/

[Matthew Miller]

On Fri, Nov 13, 2015 at 10:41 AM, Amanda Carter <acarter(a)redhat.com&gt; wrote:
>  Hey folks, creating a separate thread for this longer term
> discussion. We're getting ready to release our first 2 week atomic update on Tuesday
> and Dusty Mabe has raised 2 potential release blockers that were not part of automated
> testing. It's good that he caught them, but it's also a bit of a stroke of luck.
> Since there is no official QE for this release, who should own verifying that there are no
> release blocking bugs prior to every automated release and escalating if there are? If no
> one raises the blocker, we'll have no way to block the release. 
Other than adding "making sure that anything found manually gets tests
added so it doesn't reoccur", I don't have much to add to this except
for saying that I also recognize the need.

Paul Frields tells me that no one on his team (Fedora Infrastructure
Engineering) is in the right position to do this overall. That team --
along with Fedora Release Engineering -- has been working on making
sure that the release artifacts get constructed, but aren't experts in
what goes _in_ them. (It's nice that we're at the point where the
problems are on the inside, I guess!)

I think it needs to be someone who is deeply connected to the teams
working on Atomic development, but who is also given this is a core
responsibility and guaranteed the time for making sure things work in
Fedora specifically.

I hope that over time we can grow community around this, but that's in
a chicken and egg situation. (I can elaborate on that if it's not
obvious to everyone...)



-- 
Matthew Miller • Fedora Project Leader

[atomic-devel] Need owner of Atomic 2 Week releases

[Brian Exelbierd]

>  On Nov 13, 2015, at 5:47 PM, Matthew Miller <mattdm(a)redhat.com&gt;
> wrote:
>  
>  On Fri, Nov 13, 2015 at 10:41 AM, Amanda Carter <acarter(a)redhat.com&gt; wrote:
> > Hey folks, creating a separate thread for this longer term discussion. We're
> getting ready to release our first 2 week atomic update on Tuesday and Dusty Mabe has
> raised 2 potential release blockers that were not part of automated testing. It's good
> that he caught them, but it's also a bit of a stroke of luck. Since there is no
> official QE for this release, who should own verifying that there are no release blocking
> bugs prior to every automated release and escalating if there are? If no one raises the
> blocker, we'll have no way to block the release.
>  
>  Other than adding "making sure that anything found manually gets tests
>  added so it doesn't reoccur", I don't have much to add to this except
>  for saying that I also recognize the need.
>  
>  Paul Frields tells me that no one on his team (Fedora Infrastructure
>  Engineering) is in the right position to do this overall. That team --
>  along with Fedora Release Engineering -- has been working on making
>  sure that the release artifacts get constructed, but aren't experts in
>  what goes _in_ them. (It's nice that we're at the point where the
>  problems are on the inside, I guess!)
>  
>  I think it needs to be someone who is deeply connected to the teams
>  working on Atomic development, but who is also given this is a core
>  responsibility and guaranteed the time for making sure things work in
>  Fedora specifically. 
This is a very small community of people to choose from.  Can we figure out a way to skew
this so that we can ramp someone up?  For example, I would consider offering to help, and
I think my manager would give me some time to do the work, but I am not connected to all
of these Fedora teams.  If we find someone who is deep in 1-2 teams have some committed
folks to ramp them up we can both solve this problem and grow our number of people able to
do this kind of work.

regards,

bex

[atomic-devel] Need owner of Atomic 2 Week releases

[Adam Miller]

On Sat, Nov 21, 2015 at 3:26 AM, Brian (bex) Exelbierd <bex(a)pobox.com&gt; wrote:
> > On Nov 13, 2015, at 5:47 PM, Matthew Miller <mattdm(a)redhat.com&gt; wrote:
> >
> > On Fri, Nov 13, 2015 at 10:41 AM, Amanda Carter <acarter(a)redhat.com&gt; wrote:
> >> Hey folks, creating a separate thread for this longer term discussion. We're
> getting ready to release our first 2 week atomic update on Tuesday and Dusty Mabe has
> raised 2 potential release blockers that were not part of automated testing. It's good
> that he caught them, but it's also a bit of a stroke of luck. Since there is no
> official QE for this release, who should own verifying that there are no release blocking
> bugs prior to every automated release and escalating if there are? If no one raises the
> blocker, we'll have no way to block the release.
> >
> > Other than adding "making sure that anything found manually gets tests
> > added so it doesn't reoccur", I don't have much to add to this except
> > for saying that I also recognize the need.
> >
> > Paul Frields tells me that no one on his team (Fedora Infrastructure
> > Engineering) is in the right position to do this overall. That team --
> > along with Fedora Release Engineering -- has been working on making
> > sure that the release artifacts get constructed, but aren't experts in
> > what goes _in_ them. (It's nice that we're at the point where the
> > problems are on the inside, I guess!)
> >
> > I think it needs to be someone who is deeply connected to the teams
> > working on Atomic development, but who is also given this is a core
> > responsibility and guaranteed the time for making sure things work in
> > Fedora specifically.
>
>  This is a very small community of people to choose from.  Can we figure out a way to skew
> this so that we can ramp someone up?  For example, I would consider offering to help, and
> I think my manager would give me some time to do the work, but I am not connected to all
> of these Fedora teams.  If we find someone who is deep in 1-2 teams have some committed
> folks to ramp them up we can both solve this problem and grow our number of people able to
> do this kind of work. 
I'd be happy to fill the gaps from the Fedora side to the best of my
abilities. I also look forward to using the interaction as an
opportunity to learn more about the finer points of Atomic.

-AdamM

>
> regards,
>
> bex
> _______________________________________________
> cloud mailing list
> cloud(a)lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/[e-mail]