basic plan for two-week atomic images

[Matthew Miller]

This is a strawman based on my understanding of current abilities,
available software/systems, and of what the developers want. Please
poke holes, reorder, add and subtract, etc.

Phase I

1. F22 trees built nightly (already happening)
2. Images built from those nightly (I think right now only rawhide?)
3. Images go through automated tests, automatically (tunir; not fully
   automatic yet, right?)
4. Every two weeks, latest image to pass those tests gets linked 
   on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
    - probably need a manual override
    - page will present these images with appropriate setting of
      expectations ("fedora qa not to blame for anything here").
    - need something to happen if there aren't any that pass for 
      two weeks, which should not happen I hope, but, y'know)
        * automatic message about image being out of date
        * next successful image automatically posted? Or do we
          skip two weeks?
    - also needs: commitment to follow and fix if things break

Phase II

5. Ability to include packages from a side tag, repo, something.
   All full, legit Fedora packages destined for main repo but maybe at
   a different speed. (Implementation needs work)
   - possibly initially just selected packages from updates-testing?

Phase III

6. Something to only expose Atomic users to _update sets_ that pass the
   tests (because, hey, we have tests!)
7. Presumably, there will be a switch to F23 as base at F23 release;
   will there be overlap (beta images) to make that smooth?
8. What else?

Phase IV

9. Expand tests beyond tunir — test installs on hardware, etc.
10. Maybe create a new, special Fedora repository for packages
    with version skew mainline Fedora — e.g. newer systemd
    or hold older Docker version for some reasons. (Up for
    discussion — that's why this is in phase 3!)
11. More what else?


Does this seem basically sane and in line with what people are
expecting? What's missing? What's extra? What's just wrong? :)



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[M. Edward (Ed) Borasky]

A diagram of the artifacts (images, RPMs, issues, source repositories,
etc.) and how they interact would be very helpful to those of us who
are mostly users.

On Thu, May 21, 2015 at 1:25 PM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  This is a strawman based on my understanding of current abilities,
>  available software/systems, and of what the developers want. Please
>  poke holes, reorder, add and subtract, etc.
>
>  Phase I
>
>  1. F22 trees built nightly (already happening)
>  2. Images built from those nightly (I think right now only rawhide?)
>  3. Images go through automated tests, automatically (tunir; not fully
>     automatic yet, right?)
>  4. Every two weeks, latest image to pass those tests gets linked
>     on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
>      - probably need a manual override
>      - page will present these images with appropriate setting of
>        expectations ("fedora qa not to blame for anything here").
>      - need something to happen if there aren't any that pass for
>        two weeks, which should not happen I hope, but, y'know)
>          * automatic message about image being out of date
>          * next successful image automatically posted? Or do we
>            skip two weeks?
>      - also needs: commitment to follow and fix if things break
>
>  Phase II
>
>  5. Ability to include packages from a side tag, repo, something.
>     All full, legit Fedora packages destined for main repo but maybe at
>     a different speed. (Implementation needs work)
>     - possibly initially just selected packages from updates-testing?
>
>  Phase III
>
>  6. Something to only expose Atomic users to _update sets_ that pass the
>     tests (because, hey, we have tests!)
>  7. Presumably, there will be a switch to F23 as base at F23 release;
>     will there be overlap (beta images) to make that smooth?
>  8. What else?
>
>  Phase IV
>
>  9. Expand tests beyond tunir — test installs on hardware, etc.
>  10. Maybe create a new, special Fedora repository for packages
>      with version skew mainline Fedora — e.g. newer systemd
>      or hold older Docker version for some reasons. (Up for
>      discussion — that's why this is in phase 3!)
>  11. More what else?
>
>
>  Does this seem basically sane and in line with what people are
>  expecting? What's missing? What's extra? What's just wrong? :)
>
>
>
>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct 


-- 
OSJourno: Robust Power Tools for Digital Journalists
http://www.znmeb.mobi/stories/osjourno-robust-power-tools-for-digital-jou...

Remember, if you're traveling to Bactria, Hump Day is Tuesday and Thursday.

[Matthew Miller]

On Thu, May 21, 2015 at 02:14:02PM -0700, M. Edward (Ed) Borasky wrote:
>  A diagram of the artifacts (images, RPMs, issues, source
> repositories,
>  etc.) and how they interact would be very helpful to those of us who
>  are mostly users. 
I agree, that'd be helpful. I'll let this conversation go a couple of
rounds and then (after the F22 release and associated everything calms
down) drawing some diagrams seems like a great idea.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Dennis Gilmore]

On Thursday, May 21, 2015 04:25:42 PM Matthew Miller wrote:
>  This is a strawman based on my understanding of current abilities,
>  available software/systems, and of what the developers want. Please
>  poke holes, reorder, add and subtract, etc.
>  
>  Phase I
>  
>  1. F22 trees built nightly (already happening) 
there is new atomic trees built as
part of the package updates process managed 
by bodhi.

>  2. Images built from those nightly (I think right now only rawhide?)
once f22 is final only rawhide.

>  3. Images go through automated tests, automatically (tunir; not
> fully
>     automatic yet, right?) 
there is zero automated testing available, but we really
need it, afaik it is 
all manual

>  4. Every two weeks, latest image to pass those tests gets linked
>     on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
>      - probably need a manual override
>      - page will present these images with appropriate setting of
>        expectations ("fedora qa not to blame for anything here").
>      - need something to happen if there aren't any that pass for
>        two weeks, which should not happen I hope, but, y'know)
>          * automatic message about image being out of date
>          * next successful image automatically posted? Or do we
>            skip two weeks?
>      - also needs: commitment to follow and fix if things break 
We need people to
test, and a process to move things through stages. build all 
the images, etc. a location to put things, i,e lots of things

>  Phase II
>  
>  5. Ability to include packages from a side tag, repo, something.
>     All full, legit Fedora packages destined for main repo but maybe at
>     a different speed. (Implementation needs work)
>     - possibly initially just selected packages from updates-testing? 
really we just
need to push things stable. if we do updates-testing composes. 
we should be able to get the testing done and pushed through. there there is 
no need to go off and do special things
    
>  Phase III
>  
>  6. Something to only expose Atomic users to _update sets_ that pass the
>     tests (because, hey, we have tests!) 
it would be most ideal to have things
signed off as good. we kinda have it for 
updates with bodhi, we likely need something new or to extend something 
existing.

>  7. Presumably, there will be a switch to F23 as base at F23 release;
>     will there be overlap (beta images) to make that smooth? 
we probably should do
all supported releases and rawhide
>  8. What else?
>  
>  Phase IV
>  
>  9. Expand tests beyond tunir — test installs on hardware, etc. 
test everything :)
including other parts of fedora. automated tests for cloud 
images would be good also. I fear we currently do not do anywhere near the 
testing that we should.

>  10. Maybe create a new, special Fedora repository for packages
>      with version skew mainline Fedora — e.g. newer systemd
>      or hold older Docker version for some reasons. (Up for
>      discussion — that's why this is in phase 3!) 
We could do so, I would want
to see a really good reason to do so. I would 
hope that what is done for one part of fedora does not break or have negative 
impacts on other parts of fedora, so just pushing the updates in should be 
good.  making rawhide more of a place to be, for me should be our goal. get 
the new awesome things in there and used by people, leave the stable releases 
stable

>  11. More what else?
>  
>  
>  Does this seem basically sane and in line with what people are
>  expecting? What's missing? What's extra? What's just wrong? :)

[Matthew Miller]

On Thu, May 21, 2015 at 04:14:51PM -0500, Dennis Gilmore wrote:
>  > 1. F22 trees built nightly (already happening)
>  there is new atomic trees built as part of the package updates
>  process managed by bodhi. 
Cool.

>  > 2. Images built from those nightly (I think right now only
> rawhide?)
>  once f22 is final only rawhide. 
What would it take to make those happen post-release?

>  > 3. Images go through automated tests, automatically (tunir; not
> fully
>  >    automatic yet, right?)
>  there is zero automated testing available, but we really need it, afaik it is 
>  all manual 
See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
Taskotron future, I'm sure. :)

>  > 4. Every two weeks, latest image to pass those tests gets
> linked
>  >    on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
>  >     - probably need a manual override
>  >     - page will present these images with appropriate setting of
>  >       expectations ("fedora qa not to blame for anything here").
>  >     - need something to happen if there aren't any that pass for
>  >       two weeks, which should not happen I hope, but, y'know)
>  >         * automatic message about image being out of date
>  >         * next successful image automatically posted? Or do we
>  >           skip two weeks?
>  >     - also needs: commitment to follow and fix if things break
>  We need people to test, and a process to move things through stages.
>  build all the images, etc. a location to put things, i,e lots of
>  things 
Let's work at building the full list of lots of things. If we want to
have people testing, I'd say that'd go something like:

  - every two weeks, latest image to pass is flagged as needing human
    testing, a la a package update in bodhi
  - some sort of karma system — maybe it _is_ bodhi, but don't want to
    block on that
  - enough +1s, it goes to the atomic.fpo page
  - maybe expectation that new image is flagged on Monday, passed by...
    Wednesday? Something like that.

Alternately, we could skip that, or slot it into a later phase, and
just say "these images are auto-generated".

What process is missing for building the images?

For location, I guess
<https://dl.fedoraproject.org/pub/alt/fedora-atomic/images/>? Maybe
with separate subdirs for "all", "passed autotest", "passed
human
tests" — and/or a subdir for the two-week "releases" separate from the
nightlies.

>  
>  > Phase II
>  > 
>  > 5. Ability to include packages from a side tag, repo, something.
>  >    All full, legit Fedora packages destined for main repo but maybe at
>  >    a different speed. (Implementation needs work)
>  >    - possibly initially just selected packages from updates-testing?
>  really we just need to push things stable. if we do updates-testing composes. 
>  we should be able to get the testing done and pushed through. there there is 
>  no need to go off and do special things 
Atomic devs have expressed a strong worry about the lack of something
like this, unless I'm misunderstanding. Possibly an alternate image
including updates-testing would suffice (or, maybe, is actually
separately necessary, since atomic doesn't let individual testers just
pull in individual RPMs to test).

>  > Phase III
>  > 6. Something to only expose Atomic users to _update sets_ that pass the
>  >    tests (because, hey, we have tests!)
>  it would be most ideal to have things signed off as good. we kinda
>  have it for updates with bodhi, we likely need something new or to
>  extend something existing. 
It's kind of different from the Bodhi, in that these will be
integration tests rather than per-package tests.

I'm also worried about sending updates through a human-intervention
gate _twice_ will be a big bottleneck.

>  > 7. Presumably, there will be a switch to F23 as base at F23
> release;
>  >    will there be overlap (beta images) to make that smooth?
>  we probably should do all supported releases and rawhide 
I know Atomic devs have previously suggested wanting to encourage
people to stay on $current, at least while the thing is under such
rapid development. Possibly the right thing here is to generate
nightlies for all but point the phase I two-week-selection process at
only $current.


>  > 8. What else?
>  > 
>  > Phase IV
>  > 
>  > 9. Expand tests beyond tunir — test installs on hardware, etc.
>  test everything :) including other parts of fedora. automated tests for cloud 
>  images would be good also. I fear we currently do not do anywhere near the 
>  testing that we should. 
Absolutely. 

>  
>  > 10. Maybe create a new, special Fedora repository for packages
>  >     with version skew mainline Fedora — e.g. newer systemd
>  >     or hold older Docker version for some reasons. (Up for
>  >     discussion — that's why this is in phase 3!)
>  We could do so, I would want to see a really good reason to do so. I
>  would hope that what is done for one part of fedora does not break or
>  have negative impacts on other parts of fedora, so just pushing the
>  updates in should be good. making rawhide more of a place to be, for
>  me should be our goal. get the new awesome things in there and used
>  by people, leave the stable releases stable 
The concern I'm hearing is that rawhide is hard to develop on, because
parts other than what you're working on are may change from underneath
you too quicky. So, there's a desire to develop on top of the stable
release — but since Atomic isn't really an "on top of" thing entirely,
it might need new systemd or something that the mainline isn't ready
for. This area _definitely_ needs clear details, I agree.


Thanks Dennis!

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Adam Miller]

On Thu, May 21, 2015 at 4:46 PM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Thu, May 21, 2015 at 04:14:51PM -0500, Dennis Gilmore wrote:
> > > 1. F22 trees built nightly (already happening)
> > there is new atomic trees built as part of the package updates
> > process managed by bodhi.
>
>  Cool.
>
> > > 2. Images built from those nightly (I think right now only rawhide?)
> > once f22 is final only rawhide.
>
>  What would it take to make those happen post-release? 
If we could get a clearly defined list of what these items are I would
be happy to work towards getting them done. I just don't know how that
all needs to fit into current processes right now. This is definitely
something I'd like to at least have outlined before the Rel-Eng FAD[0]
so maybe some progress could be made before then and/or during.

I do have an open pull request to enable two week ISO composes with
the rel-eng scripts but it's still being reviewed and iterated on[1].

I also submitted a patch to the pungi 4 dev branch to enable atomic
ISOs[2] be built as a koji task once we migrate to pungi4 and can take
advantage of that capability.

> > > 3. Images go through automated tests, automatically (tunir; not fully
> > >    automatic yet, right?)
> > there is zero automated testing available, but we really need it, afaik it is
> > all manual
>
>  See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
>  Taskotron future, I'm sure. :) 
I'm not sure where in the phases this should fit, it seems like a
giant forklift of work to me given that the base Fedora distro that
Atomic is going to be pulling it's bits from doesn't even have this
functionality. Am I mistaken or possibly missing something? (Please
correct me if I am)

> > > 4. Every two weeks, latest image to pass those tests gets linked
> > >    on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
> > >     - probably need a manual override
> > >     - page will present these images with appropriate setting of
> > >       expectations ("fedora qa not to blame for anything here").
> > >     - need something to happen if there aren't any that pass for
> > >       two weeks, which should not happen I hope, but, y'know)
> > >         * automatic message about image being out of date
> > >         * next successful image automatically posted? Or do we
> > >           skip two weeks?
> > >     - also needs: commitment to follow and fix if things break
> > We need people to test, and a process to move things through stages.
> > build all the images, etc. a location to put things, i,e lots of
> > things
>  
This largely piggy backs off my previous point. Automatic "graduation"
will require a decent amount of work and probably commitment from the
QA group. I won't speak to their capacity to cater to this but I think
what is ultimately being asked of them should be brought to their
attention as early as possible to get their input.

>  Let's work at building the full list of lots of things. If we
> want to
>  have people testing, I'd say that'd go something like:
>
>    - every two weeks, latest image to pass is flagged as needing human
>      testing, a la a package update in bodhi
>    - some sort of karma system — maybe it _is_ bodhi, but don't want to
>      block on that 
If it were to be Bodhi, would the Atomic tree (or image) then be just
another build artifact pushed through and sync'd like an rpm? So for
at least the near term, it would require human intervention of
submitting an update?

>    - enough +1s, it goes to the atomic.fpo page
>    - maybe expectation that new image is flagged on Monday, passed by...
>      Wednesday? Something like that.
>
>  Alternately, we could skip that, or slot it into a later phase, and
>  just say "these images are auto-generated".
>
>  What process is missing for building the images?
>
>  For location, I guess
>  <https://dl.fedoraproject.org/pub/alt/fedora-atomic/images/>? Maybe
>  with separate subdirs for "all", "passed autotest", "passed
> human
>  tests" — and/or a subdir for the two-week "releases" separate from the
>  nightlies.
>
> >
> > > Phase II
> > >
> > > 5. Ability to include packages from a side tag, repo, something.
> > >    All full, legit Fedora packages destined for main repo but maybe at
> > >    a different speed. (Implementation needs work)
> > >    - possibly initially just selected packages from updates-testing?
> > really we just need to push things stable. if we do updates-testing composes.
> > we should be able to get the testing done and pushed through. there there is
> > no need to go off and do special things
>
>  Atomic devs have expressed a strong worry about the lack of something
>  like this, unless I'm misunderstanding. Possibly an alternate image
>  including updates-testing would suffice (or, maybe, is actually
>  separately necessary, since atomic doesn't let individual testers just
>  pull in individual RPMs to test). 
If I remember correctly, there are also concerns about things ending
up in Atomic trees that actually never see the light of day in Fedora
because they might never make it to stable. What's the policy on that
sort of thing? (Or is there one? I suspect this could be uncharted
territory because Atomic is effectively a new Operating System product
with it's own lifecycle aiming to operate within the Fedora umbrella)
> > > Phase III
> > > 6. Something to only expose Atomic users to _update sets_ that pass the
> > >    tests (because, hey, we have tests!)
> > it would be most ideal to have things signed off as good. we kinda
> > have it for updates with bodhi, we likely need something new or to
> > extend something existing.
>
>  It's kind of different from the Bodhi, in that these will be
>  integration tests rather than per-package tests.
>
>  I'm also worried about sending updates through a human-intervention
>  gate _twice_ will be a big bottleneck. 
I suspect that if/when the testing automation is in place the double
gate wouldn't be needed, especially if this is just considered a
development release. But I think that's where part of the problem lies
is that the Atomic dev teams wants to use a stable OS as something
that is re-rev'd with components updated and replaced but still call
it "Fedora $VERSION" when in fact, it is not the same as what others
will get from Fedora $VERSION unless they install from either the
Atomic image or this dev tree. (depending on what the decisions are
there)

> > > 7. Presumably, there will be a switch to F23 as base at F23 release;
> > >    will there be overlap (beta images) to make that smooth?
> > we probably should do all supported releases and rawhide
>
>  I know Atomic devs have previously suggested wanting to encourage
>  people to stay on $current, at least while the thing is under such
>  rapid development. Possibly the right thing here is to generate
>  nightlies for all but point the phase I two-week-selection process at
>  only $current.
>  
I basically covered this already, but short version is that $current
is no longer $current as soon as you ship things that aren't actually
a part of $current as the distro is currently defined. If this is
something that lands within the scope of the Rings concept such that
what the Atomic devs want to rip/replace is outside of the central
Ring and the outer rings have flexibility then I suspect that would be
fine, but that's for the Council to decide. However given that
Anaconda and systemd might need a rebase that might not be an option.

The Fedora Rel-Eng process needs a solid process definition documented
for the scope of a full release somewhere so that we can then look at
how Atomic can work within that scope. I fear there are going to be a
lot of places they will collide since they are effectively two
different distros, composed of almost the same content, but with
completely different release cycles and cadences. I know Rawhide has
been listed as something that poses challenges but it honestly feels
more well suited for something that wants to move this fast, I just
think it's a hard sell to say "that thing that is cut every 6 months,
I want to re-rev it every 2 weeks but it still be the 'same'"

I might be off base and if I am please feel free to explain to me why I am.

> > > 8. What else?
> > >
> > > Phase IV
> > >
> > > 9. Expand tests beyond tunir — test installs on hardware, etc.
> > test everything :) including other parts of fedora. automated tests for cloud
> > images would be good also. I fear we currently do not do anywhere near the
> > testing that we should.
>
>  Absolutely. 
+1

> >
> > > 10. Maybe create a new, special Fedora repository for packages
> > >     with version skew mainline Fedora — e.g. newer systemd
> > >     or hold older Docker version for some reasons. (Up for
> > >     discussion — that's why this is in phase 3!)
> > We could do so, I would want to see a really good reason to do so. I
> > would hope that what is done for one part of fedora does not break or
> > have negative impacts on other parts of fedora, so just pushing the
> > updates in should be good. making rawhide more of a place to be, for
> > me should be our goal. get the new awesome things in there and used
> > by people, leave the stable releases stable
>
>  The concern I'm hearing is that rawhide is hard to develop on, because
>  parts other than what you're working on are may change from underneath
>  you too quicky. So, there's a desire to develop on top of the stable
>  release — but since Atomic isn't really an "on top of" thing entirely,
>  it might need new systemd or something that the mainline isn't ready
>  for. This area _definitely_ needs clear details, I agree. 
Basically this again. If anything I think if Fedora Atomic wants to be
it's own product then it should exist slightly differently such that
it has it's own set of koji tags, inherits from $current, but
subplants the components it needs. This would however be a massive
shift because as it turns out that is something that's never been done
before and I think this would require a whole new Bodhi stream since
it's effectively a new product//release. If it's not Rawhide but it's
not Fedora $current, then It's basically it's own Distro that's just
based on Fedora and if it were to exist within the Fedora umbrella
then I think there need to be policy changes and that's going to be a
giant pile of other work to sort out how that's all going to fit.

Again, I might be off in the weeds but that's just how it plays out in my head.

Looking forward to comments.

-AdamM

[0] - https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
[1] - https://lists.fedoraproject.org/pipermail/rel-eng/2015-April/019822.html

>  Thanks Dennis!
>
>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Adam Miller]

On Wed, May 27, 2015 at 11:37 AM, Adam Miller
<maxamillion(a)fedoraproject.org&gt; wrote:
>  On Thu, May 21, 2015 at 4:46 PM, Matthew Miller
>  <mattdm(a)fedoraproject.org&gt; wrote:
> > On Thu, May 21, 2015 at 04:14:51PM -0500, Dennis Gilmore wrote:
> >> > 1. F22 trees built nightly (already happening)
> >> there is new atomic trees built as part of the package updates
> >> process managed by bodhi.
> >
> > Cool.
> >
> >> > 2. Images built from those nightly (I think right now only rawhide?)
> >> once f22 is final only rawhide.
> >
> > What would it take to make those happen post-release?
>
>  If we could get a clearly defined list of what these items are I would
>  be happy to work towards getting them done. I just don't know how that
>  all needs to fit into current processes right now. This is definitely
>  something I'd like to at least have outlined before the Rel-Eng FAD[0]
>  so maybe some progress could be made before then and/or during.
>
>  I do have an open pull request to enable two week ISO composes with
>  the rel-eng scripts but it's still being reviewed and iterated on[1].
>
>  I also submitted a patch to the pungi 4 dev branch to enable atomic
>  ISOs[2] be built as a koji task once we migrate to pungi4 and can take
>  advantage of that capability.
>
> >
> >> > 3. Images go through automated tests, automatically (tunir; not fully
> >> >    automatic yet, right?)
> >> there is zero automated testing available, but we really need it, afaik it is
> >> all manual
> >
> > See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
> > Taskotron future, I'm sure. :)
>
>  I'm not sure where in the phases this should fit, it seems like a
>  giant forklift of work to me given that the base Fedora distro that
>  Atomic is going to be pulling it's bits from doesn't even have this
>  functionality. Am I mistaken or possibly missing something? (Please
>  correct me if I am)
>
> >
> >> > 4. Every two weeks, latest image to pass those tests gets linked
> >> >    on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
> >> >     - probably need a manual override
> >> >     - page will present these images with appropriate setting of
> >> >       expectations ("fedora qa not to blame for anything here").
> >> >     - need something to happen if there aren't any that pass for
> >> >       two weeks, which should not happen I hope, but, y'know)
> >> >         * automatic message about image being out of date
> >> >         * next successful image automatically posted? Or do we
> >> >           skip two weeks?
> >> >     - also needs: commitment to follow and fix if things break
> >> We need people to test, and a process to move things through stages.
> >> build all the images, etc. a location to put things, i,e lots of
> >> things
> >
>  
tl;dr - Two week Atomic builds is currently blocked but we're working on it.

I wanted to follow up on this particular point. Recently at the Fedora
Activity Day for Release Engineering[0] we started to dig into the
Pungi4 code that was recently released into the wild. The unfortunate
data point is that nothing works right now, we are chewing through it
the best we can but this is all new code to almost everyone on the
team (myself included). The test cases currently pass entirely on the
set of dummy rpms that are fed to the tests but as soon as we open the
flood gates of real Fedora content this fails in new and exciting
ways. These runs take over an hour per test so I'm going to focus on
how to trim that down so we're iterating on a few hundred packages in
a compose instead of 10s of thousands. It's high on my personal
priority list to have this resolved so that we can enable Fedora
Atomic as well as be more flexible with other things. On a similar
note, the koji run-root plugin is making it's way to production soon
(already in stage and testing is looking good) so I have high hopes
there. These two things combined will enable the Atomic builds as well
as eventually enabling the pxe-to-live stuff.

I'm not sure how much people actually care about this since I never
got a response to my previous email. However, I did at least want to
follow up with a status update because this is something I have a
personal interest in as I'm a big fan of Atomic and really want to
make sure it succeeds.

-AdamM

[0] - https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
>  This largely piggy backs off my previous point. Automatic
> "graduation"
>  will require a decent amount of work and probably commitment from the
>  QA group. I won't speak to their capacity to cater to this but I think
>  what is ultimately being asked of them should be brought to their
>  attention as early as possible to get their input.
>
> > Let's work at building the full list of lots of things. If we want to
> > have people testing, I'd say that'd go something like:
> >
> >   - every two weeks, latest image to pass is flagged as needing human
> >     testing, a la a package update in bodhi
> >   - some sort of karma system — maybe it _is_ bodhi, but don't want to
> >     block on that
>
>  If it were to be Bodhi, would the Atomic tree (or image) then be just
>  another build artifact pushed through and sync'd like an rpm? So for
>  at least the near term, it would require human intervention of
>  submitting an update?
>
> >   - enough +1s, it goes to the atomic.fpo page
> >   - maybe expectation that new image is flagged on Monday, passed by...
> >     Wednesday? Something like that.
> >
> > Alternately, we could skip that, or slot it into a later phase, and
> > just say "these images are auto-generated".
> >
> > What process is missing for building the images?
> >
> > For location, I guess
> > <https://dl.fedoraproject.org/pub/alt/fedora-atomic/images/>? Maybe
> > with separate subdirs for "all", "passed autotest", "passed
> human
> > tests" — and/or a subdir for the two-week "releases" separate from
> the
> > nightlies.
> >
> >>
> >> > Phase II
> >> >
> >> > 5. Ability to include packages from a side tag, repo, something.
> >> >    All full, legit Fedora packages destined for main repo but maybe at
> >> >    a different speed. (Implementation needs work)
> >> >    - possibly initially just selected packages from updates-testing?
> >> really we just need to push things stable. if we do updates-testing composes.
> >> we should be able to get the testing done and pushed through. there there is
> >> no need to go off and do special things
> >
> > Atomic devs have expressed a strong worry about the lack of something
> > like this, unless I'm misunderstanding. Possibly an alternate image
> > including updates-testing would suffice (or, maybe, is actually
> > separately necessary, since atomic doesn't let individual testers just
> > pull in individual RPMs to test).
>
>  If I remember correctly, there are also concerns about things ending
>  up in Atomic trees that actually never see the light of day in Fedora
>  because they might never make it to stable. What's the policy on that
>  sort of thing? (Or is there one? I suspect this could be uncharted
>  territory because Atomic is effectively a new Operating System product
>  with it's own lifecycle aiming to operate within the Fedora umbrella)
> >
> >> > Phase III
> >> > 6. Something to only expose Atomic users to _update sets_ that pass the
> >> >    tests (because, hey, we have tests!)
> >> it would be most ideal to have things signed off as good. we kinda
> >> have it for updates with bodhi, we likely need something new or to
> >> extend something existing.
> >
> > It's kind of different from the Bodhi, in that these will be
> > integration tests rather than per-package tests.
> >
> > I'm also worried about sending updates through a human-intervention
> > gate _twice_ will be a big bottleneck.
>
>  I suspect that if/when the testing automation is in place the double
>  gate wouldn't be needed, especially if this is just considered a
>  development release. But I think that's where part of the problem lies
>  is that the Atomic dev teams wants to use a stable OS as something
>  that is re-rev'd with components updated and replaced but still call
>  it "Fedora $VERSION" when in fact, it is not the same as what others
>  will get from Fedora $VERSION unless they install from either the
>  Atomic image or this dev tree. (depending on what the decisions are
>  there)
>
> >
> >> > 7. Presumably, there will be a switch to F23 as base at F23 release;
> >> >    will there be overlap (beta images) to make that smooth?
> >> we probably should do all supported releases and rawhide
> >
> > I know Atomic devs have previously suggested wanting to encourage
> > people to stay on $current, at least while the thing is under such
> > rapid development. Possibly the right thing here is to generate
> > nightlies for all but point the phase I two-week-selection process at
> > only $current.
> >
>
>  I basically covered this already, but short version is that $current
>  is no longer $current as soon as you ship things that aren't actually
>  a part of $current as the distro is currently defined. If this is
>  something that lands within the scope of the Rings concept such that
>  what the Atomic devs want to rip/replace is outside of the central
>  Ring and the outer rings have flexibility then I suspect that would be
>  fine, but that's for the Council to decide. However given that
>  Anaconda and systemd might need a rebase that might not be an option.
>
>  The Fedora Rel-Eng process needs a solid process definition documented
>  for the scope of a full release somewhere so that we can then look at
>  how Atomic can work within that scope. I fear there are going to be a
>  lot of places they will collide since they are effectively two
>  different distros, composed of almost the same content, but with
>  completely different release cycles and cadences. I know Rawhide has
>  been listed as something that poses challenges but it honestly feels
>  more well suited for something that wants to move this fast, I just
>  think it's a hard sell to say "that thing that is cut every 6 months,
>  I want to re-rev it every 2 weeks but it still be the 'same'"
>
>  I might be off base and if I am please feel free to explain to me why I am.
>
> >
> >> > 8. What else?
> >> >
> >> > Phase IV
> >> >
> >> > 9. Expand tests beyond tunir — test installs on hardware, etc.
> >> test everything :) including other parts of fedora. automated tests for cloud
> >> images would be good also. I fear we currently do not do anywhere near the
> >> testing that we should.
> >
> > Absolutely.
>
>  +1
>
> >
> >>
> >> > 10. Maybe create a new, special Fedora repository for packages
> >> >     with version skew mainline Fedora — e.g. newer systemd
> >> >     or hold older Docker version for some reasons. (Up for
> >> >     discussion — that's why this is in phase 3!)
> >> We could do so, I would want to see a really good reason to do so. I
> >> would hope that what is done for one part of fedora does not break or
> >> have negative impacts on other parts of fedora, so just pushing the
> >> updates in should be good. making rawhide more of a place to be, for
> >> me should be our goal. get the new awesome things in there and used
> >> by people, leave the stable releases stable
> >
> > The concern I'm hearing is that rawhide is hard to develop on, because
> > parts other than what you're working on are may change from underneath
> > you too quicky. So, there's a desire to develop on top of the stable
> > release — but since Atomic isn't really an "on top of" thing entirely,
> > it might need new systemd or something that the mainline isn't ready
> > for. This area _definitely_ needs clear details, I agree.
>
>  Basically this again. If anything I think if Fedora Atomic wants to be
>  it's own product then it should exist slightly differently such that
>  it has it's own set of koji tags, inherits from $current, but
>  subplants the components it needs. This would however be a massive
>  shift because as it turns out that is something that's never been done
>  before and I think this would require a whole new Bodhi stream since
>  it's effectively a new product//release. If it's not Rawhide but it's
>  not Fedora $current, then It's basically it's own Distro that's just
>  based on Fedora and if it were to exist within the Fedora umbrella
>  then I think there need to be policy changes and that's going to be a
>  giant pile of other work to sort out how that's all going to fit.
>
>  Again, I might be off in the weeds but that's just how it plays out in my head.
>
>  Looking forward to comments.
>
>  -AdamM
>
>  [0] - https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
>  [1] - https://lists.fedoraproject.org/pipermail/rel-eng/2015-April/019822.html
>
> >
> >
> > Thanks Dennis!
> >
> > --
> > Matthew Miller
> > <mattdm(a)fedoraproject.org&gt;
> > Fedora Project Leader
> > _______________________________________________
> > cloud mailing list
> > cloud(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/cloud
> > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Joe Brockmeier]

On 06/12/2015 11:00 AM, Adam Miller wrote:
>  
>  I'm not sure how much people actually care about this since I never
>  got a response to my previous email. However, I did at least want to
>  follow up with a status update because this is something I have a
>  personal interest in as I'm a big fan of Atomic and really want to
>  make sure it succeeds. 
I, for one, care a lot - but have been buried. Sorry.

Thanks much for the update!

Best,

jzb
-- 
Joe Brockmeier | Open Source and Standards (OSAS)
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://projectatomic.io/

[Matthew Miller]

On Fri, Jun 12, 2015 at 11:11:51AM -0400, Joe Brockmeier wrote:
>  I, for one, care a lot - but have been buried. Sorry. 
Yeah, ditto. There was this whole F22 release thing that happened. :)


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Adam Miller]

On Fri, Jun 12, 2015 at 11:54 AM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Fri, Jun 12, 2015 at 11:11:51AM -0400, Joe Brockmeier wrote:
> > I, for one, care a lot - but have been buried. Sorry.
>
>  Yeah, ditto. There was this whole F22 release thing that happened. :)
>  
Totally understandable, it's just difficult to speculate what's going
on when there's radio silence.

-AdamM

>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Dennis Gilmore]

On Friday, June 12, 2015 10:00:43 AM Adam Miller wrote:
>  On Wed, May 27, 2015 at 11:37 AM, Adam Miller
>  
>  <maxamillion(a)fedoraproject.org&gt; wrote:
>  > On Thu, May 21, 2015 at 4:46 PM, Matthew Miller
>  > 
>  > <mattdm(a)fedoraproject.org&gt; wrote:
>  >> On Thu, May 21, 2015 at 04:14:51PM -0500, Dennis Gilmore wrote:
>  >>> > 1. F22 trees built nightly (already happening)
>  >>> 
>  >>> there is new atomic trees built as part of the package updates
>  >>> process managed by bodhi.
>  >> 
>  >> Cool.
>  >> 
>  >>> > 2. Images built from those nightly (I think right now only rawhide?)
>  >>> 
>  >>> once f22 is final only rawhide.
>  >> 
>  >> What would it take to make those happen post-release?
>  > 
>  > If we could get a clearly defined list of what these items are I would
>  > be happy to work towards getting them done. I just don't know how that
>  > all needs to fit into current processes right now. This is definitely
>  > something I'd like to at least have outlined before the Rel-Eng FAD[0]
>  > so maybe some progress could be made before then and/or during.
>  > 
>  > I do have an open pull request to enable two week ISO composes with
>  > the rel-eng scripts but it's still being reviewed and iterated on[1].
>  > 
>  > I also submitted a patch to the pungi 4 dev branch to enable atomic
>  > ISOs[2] be built as a koji task once we migrate to pungi4 and can take
>  > advantage of that capability.
>  > 
>  >>> > 3. Images go through automated tests, automatically (tunir; not fully
>  >>> > 
>  >>> >    automatic yet, right?)
>  >>> 
>  >>> there is zero automated testing available, but we really need it, afaik
>  >>> it is all manual
>  >> 
>  >> See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
>  >> Taskotron future, I'm sure. :)
>  > 
>  > I'm not sure where in the phases this should fit, it seems like a
>  > giant forklift of work to me given that the base Fedora distro that
>  > Atomic is going to be pulling it's bits from doesn't even have this
>  > functionality. Am I mistaken or possibly missing something? (Please
>  > correct me if I am)
>  > 
>  >>> > 4. Every two weeks, latest image to pass those tests gets linked
>  >>> > 
>  >>> >    on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
>  >>> >    
>  >>> >     - probably need a manual override
>  >>> >     - page will present these images with appropriate setting of
>  >>> >     
>  >>> >       expectations ("fedora qa not to blame for anything
> here").
>  >>> >     
>  >>> >     - need something to happen if there aren't any that pass for
>  >>> >     
>  >>> >       two weeks, which should not happen I hope, but, y'know)
>  >>> >       
>  >>> >         * automatic message about image being out of date
>  >>> >         * next successful image automatically posted? Or do we
>  >>> >         
>  >>> >           skip two weeks?
>  >>> >     
>  >>> >     - also needs: commitment to follow and fix if things break
>  >>> 
>  >>> We need people to test, and a process to move things through stages.
>  >>> build all the images, etc. a location to put things, i,e lots of
>  >>> things
>  
>  tl;dr - Two week Atomic builds is currently blocked but we're working on it.
>  
>  I wanted to follow up on this particular point. Recently at the Fedora
>  Activity Day for Release Engineering[0] we started to dig into the
>  Pungi4 code that was recently released into the wild. The unfortunate
>  data point is that nothing works right now, we are chewing through it
>  the best we can but this is all new code to almost everyone on the
>  team (myself included). The test cases currently pass entirely on the
>  set of dummy rpms that are fed to the tests but as soon as we open the
>  flood gates of real Fedora content this fails in new and exciting
>  ways. These runs take over an hour per test so I'm going to focus on
>  how to trim that down so we're iterating on a few hundred packages in
>  a compose instead of 10s of thousands. It's high on my personal
>  priority list to have this resolved so that we can enable Fedora
>  Atomic as well as be more flexible with other things. On a similar
>  note, the koji run-root plugin is making it's way to production soon
>  (already in stage and testing is looking good) so I have high hopes
>  there. These two things combined will enable the Atomic builds as well
>  as eventually enabling the pxe-to-live stuff.
>  
>  I'm not sure how much people actually care about this since I never
>  got a response to my previous email. However, I did at least want to
>  follow up with a status update because this is something I have a
>  personal interest in as I'm a big fan of Atomic and really want to
>  make sure it succeeds.
>   
My plan was to extend the current updates script that makes docker base images 
and cloud images to make the atomic installer also

we will only use the pungi4 code in f23 on

Dennis

[Matthew Miller]

On Wed, May 27, 2015 at 11:37:18AM -0500, Adam Miller wrote:
>  >> > 2. Images built from those nightly (I think right now
> only rawhide?)
>  >> once f22 is final only rawhide.
>  > What would it take to make those happen post-release?
>  If we could get a clearly defined list of what these items are I would
>  be happy to work towards getting them done. I just don't know how that 
Going back to this old thread. :) What's the list you need here? I
assume it's:

- downloadable Atomic qcow2 (and raw.xz)
- Same image uploaded to EC2 (and hopefully others)
- installer ISO?


>  >> > 3. Images go through automated tests, automatically
> (tunir; not fully
>  >> >    automatic yet, right?)
>  >> there is zero automated testing available, but we really need it, afaik it is
>  >> all manual
>  > See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
>  > Taskotron future, I'm sure. :)
>  I'm not sure where in the phases this should fit, it seems like a
>  giant forklift of work to me given that the base Fedora distro that
>  Atomic is going to be pulling it's bits from doesn't even have this
>  functionality. Am I mistaken or possibly missing something? (Please
>  correct me if I am) 
I think initially, this will be basic smoketests. Currently, I believe
those are https://github.com/kushaldas/tunirtests — Kushal, can you
correct me if I'm wrong. So, we don't need to forklift the whole
thing... just get these running automatically.



>  >> We need people to test, and a process to move things through
> stages.
>  >> build all the images, etc. a location to put things, i,e lots of
>  >> things
>  This largely piggy backs off my previous point. Automatic "graduation"
>  will require a decent amount of work and probably commitment from the
>  QA group. I won't speak to their capacity to cater to this but I think
>  what is ultimately being asked of them should be brought to their
>  attention as early as possible to get their input. 
I don't think we can ask a lot more of the QA group — we may need more,
new bodies.


>  > Let's work at building the full list of lots of things. If
> we want to
>  > have people testing, I'd say that'd go something like:
>  >   - every two weeks, latest image to pass is flagged as needing human
>  >     testing, a la a package update in bodhi
>  >   - some sort of karma system — maybe it _is_ bodhi, but don't want to
>  >     block on that
>  If it were to be Bodhi, would the Atomic tree (or image) then be just
>  another build artifact pushed through and sync'd like an rpm? So for
>  at least the near term, it would require human intervention of
>  submitting an update? 
Yeah. Although maybe that update could be summitted by a bot of some
sort? Presumably the same bot running Tunir. This would have the logic
for "two weeks mark has passed; find the latest image to pass tests
with this period, submit as update (or raise alarm if nothing has
succeeded)".

>  > Atomic devs have expressed a strong worry about the lack of
>  > something like this, unless I'm misunderstanding. Possibly an
>  > alternate image including updates-testing would suffice (or, maybe,
>  > is actually separately necessary, since atomic doesn't let
>  > individual testers just pull in individual RPMs to test).
>  If I remember correctly, there are also concerns about things ending
>  up in Atomic trees that actually never see the light of day in Fedora
>  because they might never make it to stable. What's the policy on that
>  sort of thing? (Or is there one? I suspect this could be uncharted
>  territory because Atomic is effectively a new Operating System product
>  with it's own lifecycle aiming to operate within the Fedora umbrella) 
It's a little bit uncharted territory if we're using F22 as a base
rather than Rawhide. For Rawhide, it's happened before (and
theoretically should be SOP for a Change proposal which doesn't work
out). But even with F22 base, as long as name-epoch-version-release
marches forward, and the packages which are subject to such churn are
clearly described as volatile, I don't see a problem.


>  I suspect that if/when the testing automation is in place the double
>  gate wouldn't be needed, especially if this is just considered a
>  development release. But I think that's where part of the problem lies
>  is that the Atomic dev teams wants to use a stable OS as something
>  that is re-rev'd with components updated and replaced but still call
>  it "Fedora $VERSION" when in fact, it is not the same as what others
>  will get from Fedora $VERSION unless they install from either the
>  Atomic image or this dev tree. (depending on what the decisions are
>  there) 
I think mostly the components that will get updated and replaced are
under the general Atomic umbrella, right? (Kubernetes, Atomic,
Docker...) Do we have concrete examples of things outside of that? I
know there was a theoretical example of systemd, but maybe that can be
coordinated with a mainline update. And the kernel is on an aggressive
update schedule anyway.

My suggestion, at least, is to try this approach with an eye towards
building the thing you talk about in this next paragraph for post-F23,
if it turns out to be necessary.

>  Basically this again. If anything I think if Fedora Atomic wants to
> be
>  it's own product then it should exist slightly differently such that
>  it has it's own set of koji tags, inherits from $current, but
>  subplants the components it needs. This would however be a massive
>  shift because as it turns out that is something that's never been done
>  before and I think this would require a whole new Bodhi stream since
>  it's effectively a new product//release. If it's not Rawhide but it's
>  not Fedora $current, then It's basically it's own Distro that's just
>  based on Fedora and if it were to exist within the Fedora umbrella
>  then I think there need to be policy changes and that's going to be a
>  giant pile of other work to sort out how that's all going to fit. 
-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Mike McGrath]

----- Original Message -----
>  From: "Matthew Miller" <mattdm(a)fedoraproject.org&gt;
>  To: "Fedora Cloud SIG" <cloud(a)lists.fedoraproject.org&gt;
>  Sent: Monday, June 15, 2015 10:17:31 AM
>  Subject: Re: basic plan for two-week atomic images
>  
>  On Wed, May 27, 2015 at 11:37:18AM -0500, Adam Miller wrote:
>  > >> > 2. Images built from those nightly (I think right now only rawhide?)
>  > >> once f22 is final only rawhide.
>  > > What would it take to make those happen post-release?
>  > If we could get a clearly defined list of what these items are I would
>  > be happy to work towards getting them done. I just don't know how that
>  
>  Going back to this old thread. :) What's the list you need here? I
>  assume it's:
>  
>  - downloadable Atomic qcow2 (and raw.xz)
>  - Same image uploaded to EC2 (and hopefully others)
>  - installer ISO?
>   
At this point I would settle for anything I could put here:

http://dl.fedoraproject.org/pub/alt/fedora-atomic/images/testing/

and blog about.

>  
>  > >> > 3. Images go through automated tests, automatically (tunir; not fully
>  > >> >    automatic yet, right?)
>  > >> there is zero automated testing available, but we really need it, afaik
>  > >> it is
>  > >> all manual
>  > > See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
>  > > Taskotron future, I'm sure. :)
>  > I'm not sure where in the phases this should fit, it seems like a
>  > giant forklift of work to me given that the base Fedora distro that
>  > Atomic is going to be pulling it's bits from doesn't even have this
>  > functionality. Am I mistaken or possibly missing something? (Please
>  > correct me if I am)
>  
>  I think initially, this will be basic smoketests. Currently, I believe
>  those are https://github.com/kushaldas/tunirtests — Kushal, can you
>  correct me if I'm wrong. So, we don't need to forklift the whole
>  thing... just get these running automatically.
>  
>  
>  
>  > >> We need people to test, and a process to move things through stages.
>  > >> build all the images, etc. a location to put things, i,e lots of
>  > >> things
>  > This largely piggy backs off my previous point. Automatic "graduation"
>  > will require a decent amount of work and probably commitment from the
>  > QA group. I won't speak to their capacity to cater to this but I think
>  > what is ultimately being asked of them should be brought to their
>  > attention as early as possible to get their input.
>  
>  I don't think we can ask a lot more of the QA group — we may need more,
>  new bodies.
>  
>  
>  > > Let's work at building the full list of lots of things. If we want to
>  > > have people testing, I'd say that'd go something like:
>  > >   - every two weeks, latest image to pass is flagged as needing human
>  > >     testing, a la a package update in bodhi
>  > >   - some sort of karma system — maybe it _is_ bodhi, but don't want to
>  > >     block on that
>  > If it were to be Bodhi, would the Atomic tree (or image) then be just
>  > another build artifact pushed through and sync'd like an rpm? So for
>  > at least the near term, it would require human intervention of
>  > submitting an update?
>  
>  Yeah. Although maybe that update could be summitted by a bot of some
>  sort? Presumably the same bot running Tunir. This would have the logic
>  for "two weeks mark has passed; find the latest image to pass tests
>  with this period, submit as update (or raise alarm if nothing has
>  succeeded)".
>  
>  > > Atomic devs have expressed a strong worry about the lack of
>  > > something like this, unless I'm misunderstanding. Possibly an
>  > > alternate image including updates-testing would suffice (or, maybe,
>  > > is actually separately necessary, since atomic doesn't let
>  > > individual testers just pull in individual RPMs to test).
>  > If I remember correctly, there are also concerns about things ending
>  > up in Atomic trees that actually never see the light of day in Fedora
>  > because they might never make it to stable. What's the policy on that
>  > sort of thing? (Or is there one? I suspect this could be uncharted
>  > territory because Atomic is effectively a new Operating System product
>  > with it's own lifecycle aiming to operate within the Fedora umbrella)
>  
>  It's a little bit uncharted territory if we're using F22 as a base
>  rather than Rawhide. For Rawhide, it's happened before (and
>  theoretically should be SOP for a Change proposal which doesn't work
>  out). But even with F22 base, as long as name-epoch-version-release
>  marches forward, and the packages which are subject to such churn are
>  clearly described as volatile, I don't see a problem.
>  
>  
>  > I suspect that if/when the testing automation is in place the double
>  > gate wouldn't be needed, especially if this is just considered a
>  > development release. But I think that's where part of the problem lies
>  > is that the Atomic dev teams wants to use a stable OS as something
>  > that is re-rev'd with components updated and replaced but still call
>  > it "Fedora $VERSION" when in fact, it is not the same as what others
>  > will get from Fedora $VERSION unless they install from either the
>  > Atomic image or this dev tree. (depending on what the decisions are
>  > there)
>  
>  I think mostly the components that will get updated and replaced are
>  under the general Atomic umbrella, right? (Kubernetes, Atomic,
>  Docker...) Do we have concrete examples of things outside of that? I
>  know there was a theoretical example of systemd, but maybe that can be
>  coordinated with a mainline update. And the kernel is on an aggressive
>  update schedule anyway.
>   
Kubernetes might be removed from the list soon if we can containerize it,
there's extra work to do so but that might simplify the atomic builds a bit.

   -Mike

>  My suggestion, at least, is to try this approach with an eye towards
>  building the thing you talk about in this next paragraph for post-F23,
>  if it turns out to be necessary.
>  
>  > Basically this again. If anything I think if Fedora Atomic wants to be
>  > it's own product then it should exist slightly differently such that
>  > it has it's own set of koji tags, inherits from $current, but
>  > subplants the components it needs. This would however be a massive
>  > shift because as it turns out that is something that's never been done
>  > before and I think this would require a whole new Bodhi stream since
>  > it's effectively a new product//release. If it's not Rawhide but it's
>  > not Fedora $current, then It's basically it's own Distro that's just
>  > based on Fedora and if it were to exist within the Fedora umbrella
>  > then I think there need to be policy changes and that's going to be a
>  > giant pile of other work to sort out how that's all going to fit.
>  
>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>

[Matthew Miller]

On Mon, Jun 15, 2015 at 03:23:39PM -0400, Michael P. McGrath wrote:
>  Kubernetes might be removed from the list soon if we can containerize
> it,
>  there's extra work to do so but that might simplify the atomic builds a bit. 
Which in turn brings up the "we need to be able to deliver layered
images" issue — that wasn't on _this_ list but perhaps needs to be.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Adam Miller]

On Mon, Jun 15, 2015 at 2:23 PM, Michael P. McGrath <mmcgrath(a)redhat.com&gt; wrote:
>  ----- Original Message -----
> > From: "Matthew Miller" <mattdm(a)fedoraproject.org&gt;
> > To: "Fedora Cloud SIG" <cloud(a)lists.fedoraproject.org&gt;
> > Sent: Monday, June 15, 2015 10:17:31 AM
> > Subject: Re: basic plan for two-week atomic images
> >
> > On Wed, May 27, 2015 at 11:37:18AM -0500, Adam Miller wrote:
> > > >> > 2. Images built from those nightly (I think right now only
> rawhide?)
> > > >> once f22 is final only rawhide.
> > > > What would it take to make those happen post-release?
> > > If we could get a clearly defined list of what these items are I would
> > > be happy to work towards getting them done. I just don't know how that
> >
> > Going back to this old thread. :) What's the list you need here? I
> > assume it's:
> >
> > - downloadable Atomic qcow2 (and raw.xz)
> > - Same image uploaded to EC2 (and hopefully others)
> > - installer ISO?
> >
>
>  At this point I would settle for anything I could put here:
>
>  http://dl.fedoraproject.org/pub/alt/fedora-atomic/images/testing/
>
>  and blog about. 
To satisfy this in the near term as we sort out a more appropriate
long term solution, we now have F22 + Upddates nightly ISO builds[0]
as well as re-enabled vagrant and qcow builds[1].

For anyone interested in the "secret sauce" here:

- the ISOs are built in a mock chroot with this script:
https://github.com/maxamillion/fedora-atomic-nightly/blob/master/build-at...

- vagrant/qcow images were re-enabled in the releng script here:
https://pagure.io/releng/blob/master/f/scripts/build-cloud-images

Once we have pungi4 in shape and/or the run-root koji plugin live in
production, we will likely be able to move these sort of tasks into a
more well defined process better suited for the long term but my hope
is that we'll no longer block the projectatomic team.

-AdamM

[0] - http://atomic-nightly.cloud.fedoraproject.org/composes
[1] -
http://koji.fedoraproject.org/koji/tasks?start=0&state=all&view=t...

> >
> > > >> > 3. Images go through automated tests, automatically (tunir; not
> fully
> > > >> >    automatic yet, right?)
> > > >> there is zero automated testing available, but we really need it,
> afaik
> > > >> it is
> > > >> all manual
> > > > See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus glorious
> > > > Taskotron future, I'm sure. :)
> > > I'm not sure where in the phases this should fit, it seems like a
> > > giant forklift of work to me given that the base Fedora distro that
> > > Atomic is going to be pulling it's bits from doesn't even have this
> > > functionality. Am I mistaken or possibly missing something? (Please
> > > correct me if I am)
> >
> > I think initially, this will be basic smoketests. Currently, I believe
> > those are https://github.com/kushaldas/tunirtests — Kushal, can you
> > correct me if I'm wrong. So, we don't need to forklift the whole
> > thing... just get these running automatically.
> >
> >
> >
> > > >> We need people to test, and a process to move things through stages.
> > > >> build all the images, etc. a location to put things, i,e lots of
> > > >> things
> > > This largely piggy backs off my previous point. Automatic
> "graduation"
> > > will require a decent amount of work and probably commitment from the
> > > QA group. I won't speak to their capacity to cater to this but I think
> > > what is ultimately being asked of them should be brought to their
> > > attention as early as possible to get their input.
> >
> > I don't think we can ask a lot more of the QA group — we may need more,
> > new bodies.
> >
> >
> > > > Let's work at building the full list of lots of things. If we want to
> > > > have people testing, I'd say that'd go something like:
> > > >   - every two weeks, latest image to pass is flagged as needing human
> > > >     testing, a la a package update in bodhi
> > > >   - some sort of karma system — maybe it _is_ bodhi, but don't want to
> > > >     block on that
> > > If it were to be Bodhi, would the Atomic tree (or image) then be just
> > > another build artifact pushed through and sync'd like an rpm? So for
> > > at least the near term, it would require human intervention of
> > > submitting an update?
> >
> > Yeah. Although maybe that update could be summitted by a bot of some
> > sort? Presumably the same bot running Tunir. This would have the logic
> > for "two weeks mark has passed; find the latest image to pass tests
> > with this period, submit as update (or raise alarm if nothing has
> > succeeded)".
> >
> > > > Atomic devs have expressed a strong worry about the lack of
> > > > something like this, unless I'm misunderstanding. Possibly an
> > > > alternate image including updates-testing would suffice (or, maybe,
> > > > is actually separately necessary, since atomic doesn't let
> > > > individual testers just pull in individual RPMs to test).
> > > If I remember correctly, there are also concerns about things ending
> > > up in Atomic trees that actually never see the light of day in Fedora
> > > because they might never make it to stable. What's the policy on that
> > > sort of thing? (Or is there one? I suspect this could be uncharted
> > > territory because Atomic is effectively a new Operating System product
> > > with it's own lifecycle aiming to operate within the Fedora umbrella)
> >
> > It's a little bit uncharted territory if we're using F22 as a base
> > rather than Rawhide. For Rawhide, it's happened before (and
> > theoretically should be SOP for a Change proposal which doesn't work
> > out). But even with F22 base, as long as name-epoch-version-release
> > marches forward, and the packages which are subject to such churn are
> > clearly described as volatile, I don't see a problem.
> >
> >
> > > I suspect that if/when the testing automation is in place the double
> > > gate wouldn't be needed, especially if this is just considered a
> > > development release. But I think that's where part of the problem lies
> > > is that the Atomic dev teams wants to use a stable OS as something
> > > that is re-rev'd with components updated and replaced but still call
> > > it "Fedora $VERSION" when in fact, it is not the same as what others
> > > will get from Fedora $VERSION unless they install from either the
> > > Atomic image or this dev tree. (depending on what the decisions are
> > > there)
> >
> > I think mostly the components that will get updated and replaced are
> > under the general Atomic umbrella, right? (Kubernetes, Atomic,
> > Docker...) Do we have concrete examples of things outside of that? I
> > know there was a theoretical example of systemd, but maybe that can be
> > coordinated with a mainline update. And the kernel is on an aggressive
> > update schedule anyway.
> >
>
>  Kubernetes might be removed from the list soon if we can containerize it,
>  there's extra work to do so but that might simplify the atomic builds a bit.
>
>     -Mike
>
> > My suggestion, at least, is to try this approach with an eye towards
> > building the thing you talk about in this next paragraph for post-F23,
> > if it turns out to be necessary.
> >
> > > Basically this again. If anything I think if Fedora Atomic wants to be
> > > it's own product then it should exist slightly differently such that
> > > it has it's own set of koji tags, inherits from $current, but
> > > subplants the components it needs. This would however be a massive
> > > shift because as it turns out that is something that's never been done
> > > before and I think this would require a whole new Bodhi stream since
> > > it's effectively a new product//release. If it's not Rawhide but
> it's
> > > not Fedora $current, then It's basically it's own Distro that's
> just
> > > based on Fedora and if it were to exist within the Fedora umbrella
> > > then I think there need to be policy changes and that's going to be a
> > > giant pile of other work to sort out how that's all going to fit.
> >
> > --
> > Matthew Miller
> > <mattdm(a)fedoraproject.org&gt;
> > Fedora Project Leader
> > _______________________________________________
> > cloud mailing list
> > cloud(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/cloud
> > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> >
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Adam Miller]

On Wed, Jun 17, 2015 at 9:17 AM, Adam Miller
<maxamillion(a)fedoraproject.org&gt; wrote:
>  On Mon, Jun 15, 2015 at 2:23 PM, Michael P. McGrath
> <mmcgrath(a)redhat.com&gt; wrote:
> > ----- Original Message -----
> >> From: "Matthew Miller" <mattdm(a)fedoraproject.org&gt;
> >> To: "Fedora Cloud SIG" <cloud(a)lists.fedoraproject.org&gt;
> >> Sent: Monday, June 15, 2015 10:17:31 AM
> >> Subject: Re: basic plan for two-week atomic images
> >>
> >> On Wed, May 27, 2015 at 11:37:18AM -0500, Adam Miller wrote:
> >> > >> > 2. Images built from those nightly (I think right now only
> rawhide?)
> >> > >> once f22 is final only rawhide.
> >> > > What would it take to make those happen post-release?
> >> > If we could get a clearly defined list of what these items are I would
> >> > be happy to work towards getting them done. I just don't know how that
> >>
> >> Going back to this old thread. :) What's the list you need here? I
> >> assume it's:
> >>
> >> - downloadable Atomic qcow2 (and raw.xz)
> >> - Same image uploaded to EC2 (and hopefully others)
> >> - installer ISO?
> >>
> >
> > At this point I would settle for anything I could put here:
> >
> > http://dl.fedoraproject.org/pub/alt/fedora-atomic/images/testing/
> >
> > and blog about.
>
>  To satisfy this in the near term as we sort out a more appropriate
>  long term solution, we now have F22 + Upddates nightly ISO builds[0]
>  as well as re-enabled vagrant and qcow builds[1].
>
>  For anyone interested in the "secret sauce" here:
>
>  - the ISOs are built in a mock chroot with this script:
>  https://github.com/maxamillion/fedora-atomic-nightly/blob/master/build-at... 
Wanted to post this for posterity, I renamed the script and added some
info to the README:
https://github.com/maxamillion/fedora-atomic-nightly

-AdamM

>  - vagrant/qcow images were re-enabled in the releng script here:
>  https://pagure.io/releng/blob/master/f/scripts/build-cloud-images
>
>  Once we have pungi4 in shape and/or the run-root koji plugin live in
>  production, we will likely be able to move these sort of tasks into a
>  more well defined process better suited for the long term but my hope
>  is that we'll no longer block the projectatomic team.
>
>  -AdamM
>
>  [0] - http://atomic-nightly.cloud.fedoraproject.org/composes
>  [1] -
> http://koji.fedoraproject.org/koji/tasks?start=0&state=all&view=t...
>
> >
> >>
> >> > >> > 3. Images go through automated tests, automatically (tunir;
> not fully
> >> > >> >    automatic yet, right?)
> >> > >> there is zero automated testing available, but we really need it,
> afaik
> >> > >> it is
> >> > >> all manual
> >> > > See <https://fedoraproject.org/wiki/Changes/tunir>;. Plus
> glorious
> >> > > Taskotron future, I'm sure. :)
> >> > I'm not sure where in the phases this should fit, it seems like a
> >> > giant forklift of work to me given that the base Fedora distro that
> >> > Atomic is going to be pulling it's bits from doesn't even have this
> >> > functionality. Am I mistaken or possibly missing something? (Please
> >> > correct me if I am)
> >>
> >> I think initially, this will be basic smoketests. Currently, I believe
> >> those are https://github.com/kushaldas/tunirtests — Kushal, can you
> >> correct me if I'm wrong. So, we don't need to forklift the whole
> >> thing... just get these running automatically.
> >>
> >>
> >>
> >> > >> We need people to test, and a process to move things through
> stages.
> >> > >> build all the images, etc. a location to put things, i,e lots of
> >> > >> things
> >> > This largely piggy backs off my previous point. Automatic
> "graduation"
> >> > will require a decent amount of work and probably commitment from the
> >> > QA group. I won't speak to their capacity to cater to this but I think
> >> > what is ultimately being asked of them should be brought to their
> >> > attention as early as possible to get their input.
> >>
> >> I don't think we can ask a lot more of the QA group — we may need more,
> >> new bodies.
> >>
> >>
> >> > > Let's work at building the full list of lots of things. If we want
> to
> >> > > have people testing, I'd say that'd go something like:
> >> > >   - every two weeks, latest image to pass is flagged as needing human
> >> > >     testing, a la a package update in bodhi
> >> > >   - some sort of karma system — maybe it _is_ bodhi, but don't want
> to
> >> > >     block on that
> >> > If it were to be Bodhi, would the Atomic tree (or image) then be just
> >> > another build artifact pushed through and sync'd like an rpm? So for
> >> > at least the near term, it would require human intervention of
> >> > submitting an update?
> >>
> >> Yeah. Although maybe that update could be summitted by a bot of some
> >> sort? Presumably the same bot running Tunir. This would have the logic
> >> for "two weeks mark has passed; find the latest image to pass tests
> >> with this period, submit as update (or raise alarm if nothing has
> >> succeeded)".
> >>
> >> > > Atomic devs have expressed a strong worry about the lack of
> >> > > something like this, unless I'm misunderstanding. Possibly an
> >> > > alternate image including updates-testing would suffice (or, maybe,
> >> > > is actually separately necessary, since atomic doesn't let
> >> > > individual testers just pull in individual RPMs to test).
> >> > If I remember correctly, there are also concerns about things ending
> >> > up in Atomic trees that actually never see the light of day in Fedora
> >> > because they might never make it to stable. What's the policy on that
> >> > sort of thing? (Or is there one? I suspect this could be uncharted
> >> > territory because Atomic is effectively a new Operating System product
> >> > with it's own lifecycle aiming to operate within the Fedora umbrella)
> >>
> >> It's a little bit uncharted territory if we're using F22 as a base
> >> rather than Rawhide. For Rawhide, it's happened before (and
> >> theoretically should be SOP for a Change proposal which doesn't work
> >> out). But even with F22 base, as long as name-epoch-version-release
> >> marches forward, and the packages which are subject to such churn are
> >> clearly described as volatile, I don't see a problem.
> >>
> >>
> >> > I suspect that if/when the testing automation is in place the double
> >> > gate wouldn't be needed, especially if this is just considered a
> >> > development release. But I think that's where part of the problem lies
> >> > is that the Atomic dev teams wants to use a stable OS as something
> >> > that is re-rev'd with components updated and replaced but still call
> >> > it "Fedora $VERSION" when in fact, it is not the same as what
> others
> >> > will get from Fedora $VERSION unless they install from either the
> >> > Atomic image or this dev tree. (depending on what the decisions are
> >> > there)
> >>
> >> I think mostly the components that will get updated and replaced are
> >> under the general Atomic umbrella, right? (Kubernetes, Atomic,
> >> Docker...) Do we have concrete examples of things outside of that? I
> >> know there was a theoretical example of systemd, but maybe that can be
> >> coordinated with a mainline update. And the kernel is on an aggressive
> >> update schedule anyway.
> >>
> >
> > Kubernetes might be removed from the list soon if we can containerize it,
> > there's extra work to do so but that might simplify the atomic builds a bit.
> >
> >    -Mike
> >
> >> My suggestion, at least, is to try this approach with an eye towards
> >> building the thing you talk about in this next paragraph for post-F23,
> >> if it turns out to be necessary.
> >>
> >> > Basically this again. If anything I think if Fedora Atomic wants to be
> >> > it's own product then it should exist slightly differently such that
> >> > it has it's own set of koji tags, inherits from $current, but
> >> > subplants the components it needs. This would however be a massive
> >> > shift because as it turns out that is something that's never been done
> >> > before and I think this would require a whole new Bodhi stream since
> >> > it's effectively a new product//release. If it's not Rawhide but
> it's
> >> > not Fedora $current, then It's basically it's own Distro that's
> just
> >> > based on Fedora and if it were to exist within the Fedora umbrella
> >> > then I think there need to be policy changes and that's going to be a
> >> > giant pile of other work to sort out how that's all going to fit.
> >>
> >> --
> >> Matthew Miller
> >> <mattdm(a)fedoraproject.org&gt;
> >> Fedora Project Leader
> >> _______________________________________________
> >> cloud mailing list
> >> cloud(a)lists.fedoraproject.org
> >> https://admin.fedoraproject.org/mailman/listinfo/cloud
> >> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> >>
> > _______________________________________________
> > cloud mailing list
> > cloud(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/cloud
> > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct