Pungi 4 milestone builds: proposals

[Adam Williamson]

Hi folks!

So we're now frozen for Alpha and we have an anaconda build with
blocker fixes in updates-testing...I guess this means it's a good time
to decide what to do about milestone builds with Pungi 4. :)

I'm assuming here that we're going to use more or less the same "TCs
and RCs" approach for F24 and just try to adapt it to Pungi 4. It seems
a bit late to make radical changes to the process itself at this point.

I have a couple of ideas. I'm gonna recap, briefly, what (as I
understand it) Pungi 4 expects us to do for milestone builds:

We're expected to run 'pungi-koji --label foo', where 'foo' is a
compose 'label' in Pungi's expected format. AIUI, this is the format
Pungi expects for a compose 'label':

MILESTONE-N.R

where MILESTONE is the milestone (there is a list of valid milestones;
for our purposes we're likely to use Alpha, Beta and RC), 'N' is the
release number, and 'R' is a respin number. Both N and R have to be
integers (though they can have multiple digits; Alpha-11.23 would be a
valid label).

Fedora does not do numbered milestone releases; we just have Alpha,
Beta and Final. RHEL does, which is why the label format has 'N'. In
the RHEL process, as I understand it, they build "Alpha-1.1", then if
it passes whatever requirements are needed for an "Alpha 1" release,
they ship it as Alpha 1; otherwise they build "Alpha-1.2" and ship that
if it passes, etc etc. The "respin" value is considered "private",
and
respins are used for about the same purpose we use TCs/RCs.

It *is* possible to do a 'production' compose with no label, I believe,
by calling 'pungi-koji --no-label'. It would *also* be technically
possible to add new label formats to Pungi. So we do have three options
at the high level:

a. Work with Pungi's currently-expected label format
b. Patch Pungi to support a more Fedora-y label format
c. Don't apply a label to our TC/RC composes at all

I would suggest that a is the best practical choice, though, because we
know this is how Pungi is kind of expected to be used. b is likely to
take time to work out any kinks, and c seems unnecessarily extreme; we
should be able to come up with a usable label format that gives us
valuable information.

So assuming we go with 1) at the high level, I see a few possibilities
for how to do this.

== 1. Just use one value ==

In this design, we'd set one of the numbers to 1 (or I guess 0) for
every single Fedora compose, and just increment the other. I was
thinking we'd lock down the release number (N) and just increment the
respin (R) - so we'd do Alpha-1.1 then Alpha-1.2 then Alpha-1.3 and so
on.

This seems like it'd work fine for Alpha and Beta but be a little odd
for Final. 'Final' is not a valid milestone for Pungi 4, it expects you
to use 'RC' as a milestone and build 'RC1.1', 'RC1.2',
'RC1.3', 'RC2.1'
etc etc. It'd be a bit odd for us to have 'RC1.1' as 'Final TC1',
'RC1.2' as 'Final TC2', 'RC1.3' as 'Final RC1', etc. So we
could
instead lock down R and only increment N, so we'd do 'RC 1.1', 'RC
2.1', 'RC 3.1' and so on. The build called 'RC 1.1' would likely not
really be a "real" RC (in the sense it'd happen while we were not
frozen and not have all blocker bugs fixed), but that's not horrible. I
suppose we could relatively easily add 'Final' to Pungi's list of
milestones too, if we wanted (or 'TC', come to that).

The other possible problem with this is it makes it hard to keep the
TC/RC distinction. We could build the composes as 'Alpha 1.1', 'Alpha
2.1', 'Alpha 3.1' and put 'Alpha 1.1' in a directory called
'Alpha_TC1', 'Alpha 2.1' in 'Alpha_TC2', and 'Alpha 3.1'
in 'Alpha_RC1'
if we wanted, but you couldn't easily map from one name to the other,
because there's no way to know the cutoff where we switched over - if
you only know you're dealing with 'Alpha 2.1', you can't tell if
that's
'Alpha TC2' or 'Alpha RC1'.

The question then is, do we actually need the TC/RC distinction for
anything? I'm not sure we really do. But if we want to keep it, we
could try...

== 2. N indicates TC/RC, R indicates number ==

In this scheme, we'd build e.g. 'Alpha 1.1' (Alpha TC1), 'Alpha 1.2'
(Alpha TC2), 'Alpha 2.1' (Alpha RC1), 'Alpha 2.2' (Alpha RC2).

This seems like the closest possible way to map to our current system.
Again it's a bit weird at Final because there is no 'Final' milestone,
only 'RC', so 'RC1.1' would be 'TC1' and 'RC2.1' would be
'RC1', which
is kinda strange; again we could add a 'Final' milestone to Pungi, I
guess.

I'm just not sure, as per 1), if we really *need* to maintain the TC/RC
distinction at least in terms of how the composes are labelled and
distributed.

== 3. Some kinda hybrid ==

We could combine things, so we do 1.n for Alpha and Beta, then n.1 for
RC (so we have Alpha 1.1, Alpha 1.2, Alpha 1.3, Alpha 1.4, Beta 1.1,
Beta 1.2, Beta 1.3, RC1.1, RC2.1, RC3.1 etc). This is viable, but I'm
not really sure it's necessary. It seems simpler to pick some variant
of 1 or 2.



There's a final question, which is whether we adopt the Pungi 'label'
naming format for all purposes - so we put the composes in directories
named Alpha-1.1, Alpha-1.2, Alpha-1.3 and so on, and we call the
validation events something like 'Test Results:Fedora 24 Alpha 1.1'
rather than 'Test Results:Fedora 24 Alpha TC1'. It's probably simpler
to just go with the label names, but if people are really used to the
TC/RC names we can stick with them if we choose option 2 or 3. It would
be difficult to keep them with option 1. I'm not sure what Dennis'
thoughts are on this.

As a final note, just so people are aware: all Pungi 4 composes have
compose IDs, so a 'production' compose with a label has *both* a
compose ID *and* a label. The compose ID for a production compose is, I
believe, something like 'Fedora-24-20160309.0' - it looks quite a lot
like a nightly compose ID, but without the '.n.' that denotes nightly.
There is no letter identifier for 'production' composes, they just omit
it entirely. This is mostly a thing for me to deal with in tooling,
though...
-- Adam WilliamsonFedora QA Community MonkeyIRC: adamw | Twitter: AdamW_Fedora | XMPP:
adamw AT happyassassin . nethttp://www.happyassassin.net

[Chris Murphy]

On Wed, Mar 9, 2016 at 11:16 AM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  Hi folks!
>
>  So we're now frozen for Alpha and we have an anaconda build with
>  blocker fixes in updates-testing...I guess this means it's a good time
>  to decide what to do about milestone builds with Pungi 4. :)
>
>  I'm assuming here that we're going to use more or less the same "TCs
>  and RCs" approach for F24 and just try to adapt it to Pungi 4. It seems
>  a bit late to make radical changes to the process itself at this point.
>
>  I have a couple of ideas. I'm gonna recap, briefly, what (as I
>  understand it) Pungi 4 expects us to do for milestone builds:
>
>  We're expected to run 'pungi-koji --label foo', where 'foo' is a
>  compose 'label' in Pungi's expected format. AIUI, this is the format
>  Pungi expects for a compose 'label':
>
>  MILESTONE-N.R
>
>  where MILESTONE is the milestone (there is a list of valid milestones;
>  for our purposes we're likely to use Alpha, Beta and RC), 'N' is the
>  release number, and 'R' is a respin number. Both N and R have to be
>  integers (though they can have multiple digits; Alpha-11.23 would be a
>  valid label).
>
>  Fedora does not do numbered milestone releases; we just have Alpha,
>  Beta and Final. RHEL does, which is why the label format has 'N'. In
>  the RHEL process, as I understand it, they build "Alpha-1.1", then if
>  it passes whatever requirements are needed for an "Alpha 1" release,
>  they ship it as Alpha 1; otherwise they build "Alpha-1.2" and ship that
>  if it passes, etc etc. The "respin" value is considered "private",
> and
>  respins are used for about the same purpose we use TCs/RCs.
>
>  It *is* possible to do a 'production' compose with no label, I believe,
>  by calling 'pungi-koji --no-label'. It would *also* be technically
>  possible to add new label formats to Pungi. So we do have three options
>  at the high level:
>
>  a. Work with Pungi's currently-expected label format 
In this format option, MILESTONE-N.R would be
MILESTONE= alpha, beta, RC
N = arbitrary integer
R = arbitrary integer

Can N and R be 0? Or must it be 1-9?

>  So we could
>  instead lock down R and only increment N, so we'd do 'RC 1.1', 'RC
>  2.1', 'RC 3.1' and so on. 
To reduce granularity, lock down either N or R. But if locked down, I
suggest a value of 0 rather than 1 to indicate nullification.


>  The build called 'RC 1.1' would likely not
>  really be a "real" RC (in the sense it'd happen while we were not
>  frozen and not have all blocker bugs fixed), but that's not horrible. I
>  suppose we could relatively easily add 'Final' to Pungi's list of
>  milestones too, if we wanted (or 'TC', come to that). 
At least from the test side, I'm not thinking there was ever a
meaningful distinction between TC and RC for alpha and beta. For
final, I vaguely think TCs and RCs did change behavior in some ways,
maybe it was disabling u-t by default? Or maybe some installer
labeling or warning goes away?

If you want to keep some granularity to distinguish between TC/RC you
could use N=0 to indicate "TC" and N=1  to indicate "RC". :-P

TC's
alpha-0.1
alpha-0.2
alpha-0.3
alpha-0.4
beta-0.1
beta-0.2
beta-0.3
RC-0.1
RC-0.2

legit RC's
alpha-1.1
alpha-1.2
alpha-1.3
alpha-1.4
beta-1.1
beta-1.2
beta-1.3
RC-1.1
RC-1.2

Further, if there is no meaningful distinction between TC/RC during
alpha and beta, then those would always be N=1. You'd only nullify
"RC" with a trailing 0 for the final TC's.


-- 
Chris Murphy

[Chris Murphy]

On Wed, Mar 9, 2016 at 1:47 PM, Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Wed, Mar 9, 2016 at 11:16 AM, Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
> > Hi folks!
> >
> > So we're now frozen for Alpha and we have an anaconda build with
> > blocker fixes in updates-testing...I guess this means it's a good time
> > to decide what to do about milestone builds with Pungi 4. :)
> >
> > I'm assuming here that we're going to use more or less the same "TCs
> > and RCs" approach for F24 and just try to adapt it to Pungi 4. It seems
> > a bit late to make radical changes to the process itself at this point.
> >
> > I have a couple of ideas. I'm gonna recap, briefly, what (as I
> > understand it) Pungi 4 expects us to do for milestone builds:
> >
> > We're expected to run 'pungi-koji --label foo', where 'foo' is a
> > compose 'label' in Pungi's expected format. AIUI, this is the format
> > Pungi expects for a compose 'label':
> >
> > MILESTONE-N.R
> >
> > where MILESTONE is the milestone (there is a list of valid milestones;
> > for our purposes we're likely to use Alpha, Beta and RC), 'N' is the
> > release number, and 'R' is a respin number. Both N and R have to be
> > integers (though they can have multiple digits; Alpha-11.23 would be a
> > valid label).
> >
> > Fedora does not do numbered milestone releases; we just have Alpha,
> > Beta and Final. RHEL does, which is why the label format has 'N'. In
> > the RHEL process, as I understand it, they build "Alpha-1.1", then if
> > it passes whatever requirements are needed for an "Alpha 1" release,
> > they ship it as Alpha 1; otherwise they build "Alpha-1.2" and ship that
> > if it passes, etc etc. The "respin" value is considered
> "private", and
> > respins are used for about the same purpose we use TCs/RCs.
> >
> > It *is* possible to do a 'production' compose with no label, I believe,
> > by calling 'pungi-koji --no-label'. It would *also* be technically
> > possible to add new label formats to Pungi. So we do have three options
> > at the high level:
> >
> > a. Work with Pungi's currently-expected label format
>
>  In this format option, MILESTONE-N.R would be
>  MILESTONE= alpha, beta, RC
>  N = arbitrary integer
>  R = arbitrary integer
>
>  Can N and R be 0? Or must it be 1-9?
>
> > So we could
> > instead lock down R and only increment N, so we'd do 'RC 1.1', 'RC
> > 2.1', 'RC 3.1' and so on.
>
>  To reduce granularity, lock down either N or R. But if locked down, I
>  suggest a value of 0 rather than 1 to indicate nullification.
>
>
> > The build called 'RC 1.1' would likely not
> > really be a "real" RC (in the sense it'd happen while we were not
> > frozen and not have all blocker bugs fixed), but that's not horrible. I
> > suppose we could relatively easily add 'Final' to Pungi's list of
> > milestones too, if we wanted (or 'TC', come to that).
>
>  At least from the test side, I'm not thinking there was ever a
>  meaningful distinction between TC and RC for alpha and beta. For
>  final, I vaguely think TCs and RCs did change behavior in some ways,
>  maybe it was disabling u-t by default? Or maybe some installer
>  labeling or warning goes away?
>
>  If you want to keep some granularity to distinguish between TC/RC you
>  could use N=0 to indicate "TC" and N=1  to indicate "RC". :-P
>
>  TC's
>  alpha-0.1
>  alpha-0.2
>  alpha-0.3
>  alpha-0.4
>  beta-0.1
>  beta-0.2
>  beta-0.3
>  RC-0.1
>  RC-0.2
>
>  legit RC's
>  alpha-1.1
>  alpha-1.2
>  alpha-1.3
>  alpha-1.4
>  beta-1.1
>  beta-1.2
>  beta-1.3
>  RC-1.1
>  RC-1.2
>
>  Further, if there is no meaningful distinction between TC/RC during
>  alpha and beta, then those would always be N=1. You'd only nullify
>  "RC" with a trailing 0 for the final TC's.
>
>
>  --
>  Chris Murphy 
So if "RC-1.8" passes all tests and is a go, how is this renamed? Or
is it not renamed?

I assume you don't want to do a production compose at this point
because then it's a compose that isn't tested, while ostensibly
identical the process has any number of non-deterministic aspects to
it, right? If so, is there an foreseen consequence of just renaming
the filename of the ISOs? Or does the final distributed ISO need to
retain the "RC-1.8" add-on?

It's sort of a minor point. There are build artifacts now as a single
digit since I think Fedora 20, that don't mean anything to the outside
world (e.g. for workstation it's 21-5, 22-3, 23-10 for the past three
releases). But I don't think they hurt anything really, so if this
RC-1.8 add-on is in the distributed ISO, it's only very slightly
crusty. But compare this to how Windows eval downloads are named:

9600.17050.WINBLUE_REFRESH.140317-1640_X64FRE_ENTERPRISE_EVAL_EN-US-IR3_CENA_X64FREE_EN-US_DV9.ISO
10586.0.151029-1700.TH2_RELEASE_CLIENTENTERPRISEEVAL_OEMRET_X64FRE_EN-US.ISO

So yea, that's pretty terrible. One of those is Windows 8 the other
Windows 10. But the world continues to function.



-- 
Chris Murphy

[Adam Williamson]

On Wed, 2016-03-09 at 13:47 -0700, Chris Murphy wrote:

>  a. Work with Pungi's currently-expected label format
>  In this format option, MILESTONE-N.R would be
>  MILESTONE= alpha, beta, RC
>  N = arbitrary integer
>  R = arbitrary integer
>  
>  Can N and R be 0? Or must it be 1-9? 
At least according to the regex that tests their validity, they can be
0, yes.

>  At least from the test side, I'm not thinking there was ever a
>  meaningful distinction between TC and RC for alpha and beta. 
Right, this is my take on it too. The *definition* is clear enough:
it's only an RC if it happens after the freeze *and* there are no
outstanding blockers *and* releng flipped any necessary switches for
'release' behaviour.

There's also a distinction in terms of the content of the compose: for
an RC compose, the filenames and volume labels are supposed to be in
the form of the 'final' release, whereas for TCs, the filenames and
volume labels are supposed to indicate TC-ness.

But so far as the actual test process goes, I don't think there's any
need to distinguish them. Of course when it comes to Go/No-Go time we
need to be sure the thing we're going-or-not-going-with meets all the
'RC' requirements, but it doesn't have to be explicitly labelled an
'RC' for that purpose, I don't think.

For Pungi 4 I'm assuming the 'label' winds up in the image filenames
and volume labels, though I haven't explicitly checked that. So I
suppose if we went with a scheme that didn't distinguish between a TC
and an RC, you couldn't tell just from looking at some random image's
filename or volume label whether it was a TC or RC. But the value of
this has always seemed quite small to me, since you also can't tell an
earlier RC from the final release in this way; if you just have an
image called 'Fedora 24' you don't know if it's actually the final
release, or an earlier RC of the final release.

The TC/RC naming was pretty much done for human consumption, but I'm
not sure it really does much for us.

>   For
>  final, I vaguely think TCs and RCs did change behavior in some ways,
>  maybe it was disabling u-t by default? Or maybe some installer
>  labeling or warning goes away? 
Aside from the image naming, the primary difference is that the
'betanag' warning - the Timbuktu screen in the installer - is disabled.
The point where we disable updates-testing by default is actually
independent of the compose process, I think.

>  If you want to keep some granularity to distinguish between TC/RC
> you
>  could use N=0 to indicate "TC" and N=1  to indicate "RC". :-P 
Hmm, yeah, I like that more than 1 and 2 if we're gonna go that path,
now you suggest it. 0 makes clear that this is something a bit
different, and it lets us use '1' as the number for the solitary actual
milestone release, which is natural.

>  TC's
>  alpha-0.1
>  alpha-0.2
>  alpha-0.3
>  alpha-0.4
>  beta-0.1
>  beta-0.2
>  beta-0.3
>  RC-0.1
>  RC-0.2
>  
>  legit RC's
>  alpha-1.1
>  alpha-1.2
>  alpha-1.3
>  alpha-1.4
>  beta-1.1
>  beta-1.2
>  beta-1.3
>  RC-1.1
>  RC-1.2 
I think this still works a bit better if we add a 'Final' milestone
rather than using the 'RC' milestone, but yeah, this isn't bad. Dennis,
WDYT?

I know releng has kinda treated Final a bit differently in the past;
they basically left out all milestone-y indicators in Final builds (so
they went into directories like 23_TC1 or 23_RC1, for instance, not
23_Final_TC1 and 23_Final_RC1, and the string 'Final' does not appear
in their filenames or volume IDs). But I don't think Pungi 4 really
allows for that, unless I'm missing something, production builds are
required to have some kind of milestone identifier, so the question is
only do we use the 'RC' milestone or add a new one, I guess.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Adam Williamson]

On Wed, 2016-03-09 at 13:57 -0700, Chris Murphy wrote:
>  
>  So if "RC-1.8" passes all tests and is a go, how is this renamed? Or
>  is it not renamed? 
>  I assume you don't want to do a production compose at this point
>  because then it's a compose that isn't tested, while ostensibly
>  identical the process has any number of non-deterministic aspects to
>  it, right? If so, is there an foreseen consequence of just renaming
>  the filename of the ISOs? Or does the final distributed ISO need to
>  retain the "RC-1.8" add-on? 
I believe Dennis' position is that the RC is *exactly what we ship* -
we don't rebuild or rename the images. We just sync the entire tree
exactly as is into the release location. So it wouldn't be a directory
called '24-RC-1.8' or whatever any more, but the image filenames and
volume labels would still contain that identifier, if they originally
did.

>  It's sort of a minor point. There are build artifacts now as a
> single
>  digit since I think Fedora 20, that don't mean anything to the outside
>  world (e.g. for workstation it's 21-5, 22-3, 23-10 for the past three
>  releases). But I don't think they hurt anything really, so if this
>  RC-1.8 add-on is in the distributed ISO, it's only very slightly
>  crusty. 
Right, exactly.
-- Adam WilliamsonFedora QA Community MonkeyIRC: adamw | Twitter: AdamW_Fedora | XMPP:
adamw AT happyassassin . nethttp://www.happyassassin.net