basic plan for two-week atomic images

Wed May 27 16:37:18 UTC 2015

On Thu, May 21, 2015 at 4:46 PM, Matthew Miller
<mattdm at fedoraproject.org> wrote:
> On Thu, May 21, 2015 at 04:14:51PM -0500, Dennis Gilmore wrote:
>> > 1. F22 trees built nightly (already happening)
>> there is new atomic trees built as part of the package updates
>> process managed by bodhi.
>
> Cool.
>
>> > 2. Images built from those nightly (I think right now only rawhide?)
>> once f22 is final only rawhide.
>
> What would it take to make those happen post-release?

If we could get a clearly defined list of what these items are I would
be happy to work towards getting them done. I just don't know how that
all needs to fit into current processes right now. This is definitely
something I'd like to at least have outlined before the Rel-Eng FAD[0]
so maybe some progress could be made before then and/or during.

I do have an open pull request to enable two week ISO composes with
the rel-eng scripts but it's still being reviewed and iterated on[1].

I also submitted a patch to the pungi 4 dev branch to enable atomic
ISOs[2] be built as a koji task once we migrate to pungi4 and can take
advantage of that capability.

>
>> > 3. Images go through automated tests, automatically (tunir; not fully
>> >    automatic yet, right?)
>> there is zero automated testing available, but we really need it, afaik it is
>> all manual
>
> See <https://fedoraproject.org/wiki/Changes/tunir>. Plus glorious
> Taskotron future, I'm sure. :)

I'm not sure where in the phases this should fit, it seems like a
giant forklift of work to me given that the base Fedora distro that
Atomic is going to be pulling it's bits from doesn't even have this
functionality. Am I mistaken or possibly missing something? (Please
correct me if I am)

>
>> > 4. Every two weeks, latest image to pass those tests gets linked
>> >    on atomic.fpo, again all automatically (NEEDS DESIGN AND CODE)
>> >     - probably need a manual override
>> >     - page will present these images with appropriate setting of
>> >       expectations ("fedora qa not to blame for anything here").
>> >     - need something to happen if there aren't any that pass for
>> >       two weeks, which should not happen I hope, but, y'know)
>> >         * automatic message about image being out of date
>> >         * next successful image automatically posted? Or do we
>> >           skip two weeks?
>> >     - also needs: commitment to follow and fix if things break
>> We need people to test, and a process to move things through stages.
>> build all the images, etc. a location to put things, i,e lots of
>> things
>

This largely piggy backs off my previous point. Automatic "graduation"
will require a decent amount of work and probably commitment from the
QA group. I won't speak to their capacity to cater to this but I think
what is ultimately being asked of them should be brought to their
attention as early as possible to get their input.

> Let's work at building the full list of lots of things. If we want to
> have people testing, I'd say that'd go something like:
>
>   - every two weeks, latest image to pass is flagged as needing human
>     testing, a la a package update in bodhi
>   - some sort of karma system — maybe it _is_ bodhi, but don't want to
>     block on that

If it were to be Bodhi, would the Atomic tree (or image) then be just
another build artifact pushed through and sync'd like an rpm? So for
at least the near term, it would require human intervention of
submitting an update?

>   - enough +1s, it goes to the atomic.fpo page
>   - maybe expectation that new image is flagged on Monday, passed by...
>     Wednesday? Something like that.
>
> Alternately, we could skip that, or slot it into a later phase, and
> just say "these images are auto-generated".
>
> What process is missing for building the images?
>
> For location, I guess
> <https://dl.fedoraproject.org/pub/alt/fedora-atomic/images/>? Maybe
> with separate subdirs for "all", "passed autotest", "passed human
> tests" — and/or a subdir for the two-week "releases" separate from the
> nightlies.
>
>>
>> > Phase II
>> >
>> > 5. Ability to include packages from a side tag, repo, something.
>> >    All full, legit Fedora packages destined for main repo but maybe at
>> >    a different speed. (Implementation needs work)
>> >    - possibly initially just selected packages from updates-testing?
>> really we just need to push things stable. if we do updates-testing composes.
>> we should be able to get the testing done and pushed through. there there is
>> no need to go off and do special things
>
> Atomic devs have expressed a strong worry about the lack of something
> like this, unless I'm misunderstanding. Possibly an alternate image
> including updates-testing would suffice (or, maybe, is actually
> separately necessary, since atomic doesn't let individual testers just
> pull in individual RPMs to test).

If I remember correctly, there are also concerns about things ending
up in Atomic trees that actually never see the light of day in Fedora
because they might never make it to stable. What's the policy on that
sort of thing? (Or is there one? I suspect this could be uncharted
territory because Atomic is effectively a new Operating System product
with it's own lifecycle aiming to operate within the Fedora umbrella)
>
>> > Phase III
>> > 6. Something to only expose Atomic users to _update sets_ that pass the
>> >    tests (because, hey, we have tests!)
>> it would be most ideal to have things signed off as good. we kinda
>> have it for updates with bodhi, we likely need something new or to
>> extend something existing.
>
> It's kind of different from the Bodhi, in that these will be
> integration tests rather than per-package tests.
>
> I'm also worried about sending updates through a human-intervention
> gate _twice_ will be a big bottleneck.

I suspect that if/when the testing automation is in place the double
gate wouldn't be needed, especially if this is just considered a
development release. But I think that's where part of the problem lies
is that the Atomic dev teams wants to use a stable OS as something
that is re-rev'd with components updated and replaced but still call
it "Fedora $VERSION" when in fact, it is not the same as what others
will get from Fedora $VERSION unless they install from either the
Atomic image or this dev tree. (depending on what the decisions are
there)

>
>> > 7. Presumably, there will be a switch to F23 as base at F23 release;
>> >    will there be overlap (beta images) to make that smooth?
>> we probably should do all supported releases and rawhide
>
> I know Atomic devs have previously suggested wanting to encourage
> people to stay on $current, at least while the thing is under such
> rapid development. Possibly the right thing here is to generate
> nightlies for all but point the phase I two-week-selection process at
> only $current.
>

I basically covered this already, but short version is that $current
is no longer $current as soon as you ship things that aren't actually
a part of $current as the distro is currently defined. If this is
something that lands within the scope of the Rings concept such that
what the Atomic devs want to rip/replace is outside of the central
Ring and the outer rings have flexibility then I suspect that would be
fine, but that's for the Council to decide. However given that
Anaconda and systemd might need a rebase that might not be an option.

The Fedora Rel-Eng process needs a solid process definition documented
for the scope of a full release somewhere so that we can then look at
how Atomic can work within that scope. I fear there are going to be a
lot of places they will collide since they are effectively two
different distros, composed of almost the same content, but with
completely different release cycles and cadences. I know Rawhide has
been listed as something that poses challenges but it honestly feels
more well suited for something that wants to move this fast, I just
think it's a hard sell to say "that thing that is cut every 6 months,
I want to re-rev it every 2 weeks but it still be the 'same'"

I might be off base and if I am please feel free to explain to me why I am.

>
>> > 8. What else?
>> >
>> > Phase IV
>> >
>> > 9. Expand tests beyond tunir — test installs on hardware, etc.
>> test everything :) including other parts of fedora. automated tests for cloud
>> images would be good also. I fear we currently do not do anywhere near the
>> testing that we should.
>
> Absolutely.

+1

>
>>
>> > 10. Maybe create a new, special Fedora repository for packages
>> >     with version skew mainline Fedora — e.g. newer systemd
>> >     or hold older Docker version for some reasons. (Up for
>> >     discussion — that's why this is in phase 3!)
>> We could do so, I would want to see a really good reason to do so. I
>> would hope that what is done for one part of fedora does not break or
>> have negative impacts on other parts of fedora, so just pushing the
>> updates in should be good. making rawhide more of a place to be, for
>> me should be our goal. get the new awesome things in there and used
>> by people, leave the stable releases stable
>
> The concern I'm hearing is that rawhide is hard to develop on, because
> parts other than what you're working on are may change from underneath
> you too quicky. So, there's a desire to develop on top of the stable
> release — but since Atomic isn't really an "on top of" thing entirely,
> it might need new systemd or something that the mainline isn't ready
> for. This area _definitely_ needs clear details, I agree.

Basically this again. If anything I think if Fedora Atomic wants to be
it's own product then it should exist slightly differently such that
it has it's own set of koji tags, inherits from $current, but
subplants the components it needs. This would however be a massive
shift because as it turns out that is something that's never been done
before and I think this would require a whole new Bodhi stream since
it's effectively a new product//release. If it's not Rawhide but it's
not Fedora $current, then It's basically it's own Distro that's just
based on Fedora and if it were to exist within the Fedora umbrella
then I think there need to be policy changes and that's going to be a
giant pile of other work to sort out how that's all going to fit.

Again, I might be off in the weeds but that's just how it plays out in my head.

Looking forward to comments.

-AdamM

[0] - https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
[1] - https://lists.fedoraproject.org/pipermail/rel-eng/2015-April/019822.html

>
>
> Thanks Dennis!
>
> --
> Matthew Miller
> <mattdm at fedoraproject.org>
> Fedora Project Leader
> _______________________________________________
> cloud mailing list
> cloud at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/cloud
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct