To semi-rolling or not to semi-rolling, that is the question...

[Kevin Kofler]

Doug Ledford wrote:

> On 03/05/2010 04:49 AM, Kevin Kofler wrote:
>> Yet it is the only solution which really satisfies both groups of people.
> 
> You should always be more clear when writing emails such as this.  The
> "Yet it is" above is unclear.  Are you referring to a stable rawhide, or
> the two update stream model for satisfying both parties?

The 2 update stream model.

>> Stabilizing Rawhide is pretty much a lost cause.
> 
> No, it's not.  If we care about rawhide being stable, then it's up to us
> to make it so.  Since we don't currently care, developers do sloppy,
> lazy things in rawhide and don't worry about it.

I think there's a lot more to it than just that, as I explained further 
down.

>> What your proposal further
>> down does is actually introducing an intermediate stream à la Debian
>> testing, which comes with a lot of extra maintainer work as well.
> 
> It comes with less extra work than doing two update streams.  Face it,
> there is *no* solution to this problem that both solves the issue for
> both parties involved and does not include at least *some* extra work
> for you.

Sure, but will yours be *less* extra work? And do we also really need to 
target both groups in the first place (as opposed to focusing on the fast-
moving solution which distinguishes us from other distros), seeing how many 
existing distributions already fill the need for the conservative variant?

[re users who expect a conservative update policy]
> They are fedora users today, ignoring them is exceedingly rude.

It was their choice to use Fedora. They may have made the wrong choice, but 
that doesn't mean they get to turn Fedora into something different. As an 
analogy, if I walk into a gay bar, I don't get to convert it into a bar for 
heterosexuals just because I happen not to be attracted by men.

> And given the fact that more people have stood up requesting a more stable
> update stream than those that have stood up for the semi-rolling update
> stream,

Really? That's not the impression I got.

>> And I also dislike the usage of the term "stable" for this purpose, as it
>> implies our current updates are "unstable", which to most users means
>> "crashy".
> 
> Did you miss the KDE update thread yesterday?  I stand by the name
> stable, and I raise you an "it *was* crashy".

I actually believe the KDE updates are an example of things done right. They 
aren't perfect, but no software is.

> My proposal *is* trying to do that.  Whether or not it is a lost cause
> is a function of *us*.  We make rawhide consumable, or we make it a
> minefield.  Either way, it's our actions at work.  All we have to do is
> care.

No matter what we do, we can't achieve the impossible.

> So?  Rawhide-unstable is just a temporary holding area.  Unlike the
> GA/updates repo combinations for releases, rawhide is not frozen and
> doesn't contain a static set of packages.  That means that
> rawhide-unstable is not like an updates repo where it holds the updates
> forever.  Instead, packages move from rawhide-unstable to rawhide on a
> regular basis, and when that happens, the rawhide-unstable repo shrinks
> (preferably to 0, but if some of the packages aren't ready for the move
> yet then they stay in rawhide-unstable).

I understood that just fine the first time, there was no need to repeat it.

> You don't test locally before you build into the build system?  That
> would be one way.  The other way is to actually know and understand the
> code you are submitting.  Many disruptive changes you simply *know* are
> disruptive because you know and understand the code.  This would be
> determined on a maintainer by maintainer basis.  Then you would also
> have the AutoQA/rpmdiff checks on the packages post-build that would
> attempt to catch disruptive changes that the maintainer missed.

Of course all this is possible and will be done, but that doesn't obviate 
the need for a testing repository. If it did, then why would we have 
updates-testing in the first place?

> But, really, this boils down to what I said earlier.  If we care about
> rawhide, then we make an effort not to break it, it will be better.  If
> we don't, then it will stay the same.  There's no great technical hurdle
> here, only a social one.

The fact that packages cannot be tested adequately without a testing 
repository is just common sense, and it is a technical issue, not a social 
one.

I for one don't see untested or poorly-tested stuff as a viable alternative 
to our current updates. I wouldn't want to use stuff which had no testing 
(except for trivial fixes or urgent stuff such as security fixes, but we've 
had that part of the discussion already; and stuff which is not trivial 
and/or urgent definitely DOES need testing, and there needs to be 
infrastructure for that).

And an unstable repository also containing disruptive changes which are 
known to be disruptive is a poor substitute for a testing repository. It's 
hard to test something when everything else is broken. There's a reason we 
have updates-testing and don't just use Rawhide (in its current form) for 
all our testing.

> Sure it would.  I'm not entirely sure why you think it wouldn't be easy
> to pull packages from rawhide-unstable and run them on rawhide.  As I
> pointed out above, rawhide would regularly dump packages from
> rawhide-unstable to rawhide when they are ready.  That keeps rawhide and
> rawhide-unstable so close that for the most part anything in
> rawhide-unstable *should* work on rawhide with the obvious exception of
> soname bump type issues.

But that ("soname bump type issues") is exactly what I worry about.

> But, that's also part of the reason for having rawhide-unstable,

Which makes it all the more likely that a soname will be higher in rawhide-
unstable than in the consumable rawhide.

> and in the case of soname bumps you simply have not only the library
> that's bumping versions but also all the dependent packages in rawhide-
> unstable so that the entire group can be updated on top of rawhide and
> tested.  This would work perfectly fine.

But what if the maintainer wants to do an unrelated update to one of the 
affected packages? He'll have to build it twice (with different EVRs 
obviously) then.

> No.  As above, most packages should be close enough to rawhide to move
> safely, and soname bumps and the sort will all come as a group so when
> you move the updated library, you move all the dependent apps at the
> same time, no rebuilds are necessary.

That doesn't work if those packages are to get unrelated updates (even if 
they're just trivial bugfix patches). They either need to be built twice or 
the unrelated update withheld until the whole group can move. The path of 
least resistance will be the latter, and so the consumable rawhide is likely 
not to get as many updates as its users would like to have. Especially as 
these soname issues will affect MANY packages, e.g. an OpenSSL soname bump 
affects almost the entire distro! Sure, that soname bump and the related 
updates will EVENTUALLY hit the consumable rawhide (I got that part! There's 
no need to reexplain it again!), but that "eventually" can be up to 1 month 
away even if the first flag day is met, and even longer otherwise. That's 
way too long! So things WILL need to be built twice for the system to work.

> The devel branch builds into rawhide by default.  The rawhide buildroot
> inherits all packages from itself, so anytime a build goes directly into
> rawhide it is immediately available in rawhide.  The rawhide-unstable
> buildroot inherits from rawhide and from itself, so anything in
> rawhide-unstable overrides whatever is in rawhide.  It also builds into
> itself, so builds into rawhide-unstable are immediately visible in
> rawhide-unstable.

I already figured that. It doesn't solve the above problem.

> If you have a package you know will be disruptive, you can opt to use make
> unstable-build instead of make build and it will direct your build to
> rawhide-unstable.

Those really need to be separate branches, not just separate make targets, 
or maintaining 2 versions where needed (and it WILL be needed, see above) 
will be a PITA.

> Let's assume for this exercise that your change is a major update to
> python where all the dependent packages will need rebuilt/touched up.  You
> built your new python in rawhide-unstable.  You then notify devel@ that
> you have updated python in rawhide-unstable and request all python module
> owners to rebuild their packages in unstable against your new python.  You
> could even file bugs to this effect.  You then wait for them to do their
> stuff.  If they are too slow, you could check things out in cvs and do a
> bump and rebuild into unstable yourself.  At some point in time, when the
> majority (or all if you are that picky) of these packages are rebuilt,
> you can put out a call for testing whereby people can do:
> 
> yum --enablerepo=rawhide-unstable update python
> 
> which will pull in the base python package and all the dependent items.
>  People can then test your code.  You wait for feedback from people,
> resolving items as you go, building new things into unstable as needed,
> etc.  Finally, when you are satisfied that this is really at a stable
> point, you file the bodhi ticket for your package plus all the python
> modules in unstable with you, and at the next regular
> rawhide-unstable==>rawhide update they get moved over.

And until this happens, all updates to Python packages have to be built 
twice, once for rawhide-unstable and once for the consumable rawhide.

> No maintainer will *ever* leave stuff in rawhide-unstable.

Yet the whole rest of your paragraph goes on to explain reasons why stuff 
would, in fact, be left in rawhide-unstable. And there's an additional one: 
laziness.

> No, if the breakage isn't fixed, it isn't included in the move on flag
> day.  The flag day is not simply a blind copying of all packages from
> rawhide-unstable to rawhide, it would more likely be a regular bodhi
> push where the people responsible for a package would have filed a bodhi
> ticket listing all the packages necessary to resolve the disruption in a
> bodhi ticket and that ticket would get pushed from rawhide-unstable to
> rawhide.  That adds a little extra work if you have a disruptive change
> in that you now would have to file a bodhi ticket, but I see this as a
> good thing because it forces the developer to do their homework, to
> check that the issue is resolved, to account for all the packages that
> need changed in resolving the issue, and to proactively notify rel-eng
> that the change is actually ready.  These are all good things even if
> they mean an extra 10 minutes of work for the maintainer.

The more stuff you leave in rawhide-unstable that way, the more it will 
diverge from the consumable rawhide, invalidating all your arguments about 
how this will only be a small add-on repo, how everything in it will hit 
rawhide eventually, how the 2 repos are "close" so stuff "usually" doesn't 
need to built twice (which I doubt anyway) etc.

Let's be blunt: either you blindly move everything and break stuff, or you 
don't and the repos will tend to diverge more, not less, over time.

> Now, as I pointed out, there is the distinct possibility that by doing
> this six times as often, each individual time will be much easier to
> deal with, less prone to system breakage, etc.  You may find, to your
> surprise, that it is actually a better way of doing things.  We don't
> know, no one's ever tried it.
[snip]
> What I wrote above will absolutely be *more* consumable than what we
> have today.  Whether or not it would be consumable *enough* to use,
> that's a different matter.  But I guarantee the above steps would
> improve things.

Yet you want to eliminate the option that's known working at the same time 
(in favor of a conservative one which doesn't satisfy our needs at all).

> Most of these last two paragraphs seem to be directed at your vision of
> my proposal which, as I pointed out many paragraphs above, is not the
> same as my actual vision (aka, the whole part about the moving things
> from rawhide-unstable to rawhide, which means they aren't really two
> repos, just that one is merely a temporary holding ground before a push
> moves stuff over to the other, very much unlike a GA/updates repo pair).

Wrong. I understood your proposal perfectly, your repeated explanations were 
not needed, I already "got" all that! It's you who didn't understand my 
objections to it.

        Kevin Kofler