FESCo wants to ban direct stable pushes in Bodhi (urgent call for feedback)

[Kevin Kofler]

Adam Williamson wrote:
> They can't expect anything currently either. We have no project-wide
> commitment to what kind of updates we will ship. Some maintainers ship
> version updates, some don't. There's already no consistency; this
> wouldn't make things any worse.

I'd argue it would, because having to do either 2 sets of updates or only 
conservative updates will draw more maintainers towards the latter. But 
anyway, I know we don't have a concrete policy there right now (mainly 
because we couldn't agree on one).

> Wasn't it your side of the argument who was arguing against telling
> maintainers what to do, and saying it should be 'fun' to contribute to
> Fedora?

I didn't bring up the "fun" argument. My point is that banning direct stable 
pushes prevents us from fixing things for our users ASAP when needed. This 
is all part of duty, not fun.

Other people argued that excess bureaucracy also makes it "unfun" to 
contribute, and they're right. But my point doesn't depend on that.

> If going through a testing process before shipping an update is
> sufficiently 'unfun' to be a problem, surely being required to ship
> updates you do not believe should be shipped is in the same boat? Not
> that I disagree the situation should be less unpredictable than it is at
> the moment, but I don't see why *one* solution to this (make updates
> slightly more cautious) is a terrible imposition on maintainers, but
> another (make all updates more adventurous) is not.

I don't think that having a policy of "always push new versions unless they 
cause breakage, such as [list of bad things an update should never do], or 
do not change anything for Fedora users (e.g. only fix bugs specific to 
other platforms)" would make maintainership "unfun".

>> Yet, in practice, I still think a lot
>> more stuff gets backported in our updates repository than in those
>> backports repositories of other distros.
> 
> Probably true (though in the case of Mandriva, maybe less than you'd
> expect; it's nothing like the wasteland that is Ubuntu backports).
> What's your point?

That your model would lead to fewer "backport" type updates and thus be 
further away from what I and several others would like to see than what we 
have now! You admitting this point also directly contradicts your claims 
elsewhere that it wouldn't lead to fewer "backport" type updates being 
provided.

(By the way, the "backports" term is unfortunate, because it's misleading, 
as the "backports" are actually version upgrades whereas the conservative 
updates are the ones getting backported fixes. Sadly, it's confusing Debian 
jargon (coming from the fact that Debian backports are backports of packages 
from unstable for stable) which made its way into other distributions.)

>> Also because the maintainers don't have to
>> worry about a conservative updates stream at the same time.
> 
> Er, yes they do.

Sorry, what I meant was: "Also because OUR maintainers don't have to worry 
about a conservative updates stream at the same time."

> That's precisely what Mandriva has - twin streams, one stable update
> stream, one backport stream. All maintainers are required to provide
> stable updates for packages in /main. They can *choose* to provide
> /backports upgrades too, but that doesn't absolve them of doing a safe
> /updates stream.

That was exactly my point. This removes the option to only provide the new 
versions (which also fix bugs, often more than bugfixes conservatively 
backported to the old version would ever fix because maintainers cannot be 
expected to backport each and every bugfix for big upstream projects, and if 
they did, they'd essentially ship the new version disguised as backported 
bugfixes) and will lead many maintainers to choose to only provide the 
conservative option. With our policy, they're encouraged to just drop the 
legacy crap and ship the current version.

In addition, in practice, it's quite likely that bugs will often be answered 
with "it's too hard to backport that particular fix, upgrade to the current 
version from backports (or even Cooker/Rawhide/whatever) if you need it". 
There's no way you can really force maintainers to provide an update stream 
which is "stable" under that definition (no upgrades, only backported 
fixes).

>> Having both,
>> we'd have to fix bugs in the conservative updates AND push backports.
> 
> Again, Mandriva's been doing this for years, with a substantially
> smaller maintainer base, without it being a problem.

The problem is that there are fewer updates included in Mandriva backports 
than in our official updates. You may not see this as a problem, but those 
of us who think the version upgrades are an important characteristic of 
Fedora (and I'm one of those) definitely do!

> Obviously the system isn't perfect, no system is; sometimes version bumps
> just go through as updates because it's way too much work to maintain a
> seventeen month bugfix backport list.

Indeed, that's why I'm saying it would be a significant extra burden for us 
to have to maintain that conservative stream, you were the one denying that.

> But most of the time it works as advertised, and has a record of causing
> fewer problems for users of stable releases than the Fedora updates system
> has.

That may be true, but you're comparing apples with oranges. Users of that 
kind of "stable releases" also don't get any of the improvements our users 
get in the updates and even bugs will often only be fixed in the next 
release.

I think that that kind of problems are actually extremely rare in our stable 
updates (also because testing is used where it makes sense, and also because 
regression fixes are often pushed directly to stable to minimize exposure 
time) and that being paranoid about them would just degrade the overall 
Fedora experience. (The phrase "throwing out the baby with the bathwater" 
comes to my mind.)

>> Of
>> course that'd tempt maintainers to just skip the backports step. Whereas
>> our policy is to prefer resolving issues by pushing new upstream releases
> 
> Er, is it? I thought we were discussing the fact that we don't *have* an
> updates policy. That may be the KDE SIG's policy, but it's not Fedora's,
> as far as I know.

Strange, I thought there was a policy written down somewhere, and it's even 
possible it used to be there on some old web page or wiki we no longer use 
and no longer is now.

The only current document I found which sorta implies this is:
https://fedoraproject.org/wiki/Staying_close_to_upstream_projects
"Staying close to upstream versions is helpful when doing version bumps for 
updates that bring in new features. This prevents tedious backporting of 
only security and bug fixes."
but it's not really a policy, it just says backporting is "tedious" and can 
be "prevented".

Another part of that page which implies such a policy is in the "Exceptions" 
section further on:
"For any major issues, waiting on a new release from upstream for a fix can 
be too much of a delay. In these instances, it may be better to backport 
those fixes from upstream or fix the issue by writing your own patch and do 
an update in Fedora."
To me, this says backporting should only be done if there isn't a new 
upstream release to upgrade to.

And this has been the de-facto policy all this time, see e.g.:
http://www.mail-archive.com/fedora-legacy-list@redhat.com/msg01494.html
and see the actual security updates being pushed. The common rule is, 
backport a security fix only if there's no new version to upgrade to or if 
there's a good reason not to upgrade (e.g. for OpenSSL where you'd have to 
rebuild half the distro if you upgrade).

> Many instances have shown that it does not give us the bugfixes 'for
> free'. It comes with the cost of causing regressions. That may be a cost
> which we decide we want to bear, but portraying things in a Panglossian
> manner doesn't help your cause, as it just makes you look like you're
> denying there could ever possibly be any problems with your method.

But that's not a cost for the maintainer (unless the regression breaks 
his/her own system). :-)

Well, having to fix the regression is. But that's much less work than 
backporting every single bugfix, and not doing that would mean our updates 
would have more bugs than now.

>> That also sucks. With Fedora's KDE updates, users can be sure that
>> they'll be as regression-free as humanely possible and we do all we can
>> to keep their updates stable.
> 
> Which means...when they notice something's broken they let you know, and
> you fix it. What's the difference? Or are you seriously telling us
> you're perfect, and there's never been any problem in any KDE update?
> Your little 'as humanly possible' disclaimer suggests you're not really
> saying that, but you could've made it more obvious.

The difference is that our updates go through testing first and (ideally) 
only get pushed to stable when there are no regressions left. This is quite 
unlike the Kubuntu model where they throw out new KDE versions into some 
backports repo as soon as they're built, with no testing, and any regression 
reports are answered with "it's an unsupported backport" and often only 
fixed with the next KDE version, if at all. (I'm not very familiar with what 
Mandriva is doing, so it may suck a bit less there, but still, a not really 
supported backports repository is not quite as reliable as our updates!)

>> On the other hand, users of distros using the
>> backports model just get told "backports are unsupported".
> 
> No, they don't. please don't misrepresent my words. That's not what I
> said at all. What they get told is 'well, backports are unsupported - as
> you know anyway because we spread that message very well and people know
> what they're getting into - but give me some information and I'll do my
> best to fix it'. Which usually happens.

Ideally yes, in practice I've read many complaints about this not happening, 
at least for Kubuntu's and openSUSE's KDE backports (which are both in 
dedicated KDE backports repos, not in generic backports, by the way), and as 
much I'd like to believe Mandriva is perfect, I doubt it.

>> In fact their
>> builds of new KDE releases tend to carry only the same old distro patches
>> as the old version or even to be entirely vanilla, very little is done to
>> e.g. backport regression fixes from the branch. And KDE is just the
>> example I'm most familiar with, I'm pretty sure it's similar with other
>> stuff that gets updated in our stable updates vs. other distros'
>> backports repositories.
> 
> That doesn't match my impression of how Mandriva at least handles KDE
> backports, and there's nothing intrinsic about the /backports model that
> means it must be this way. Or are you saying that if Fedora adopted a
> split repositories model, you would somehow be compelled to make
> the /backports releases suck? I don't understand why that would be. It'd
> be all the same work you currently do, just released in a different
> place.

There's a completely different feeling of responsibility when I'm about to 
push something to an official updates repository than when I'm about to push 
something for an unsupported backports repository.

It's also quite likely that backports-testing would get even fewer testers 
than the current updates-testing (or maybe backports-testing would get most 
of them, which would suck for updates-testing; either way, we'd be splitting 
our already small tester base into 2!), leading to more regressions slipping 
through.

And, while I'm certainly biased, my perception is also that our updates are 
currently a lot more stable than the backports repositories other 
distributions provide, which, if true, would also provide evidence for my 
point. But it's hard to provide some objective metric to prove or disprove 
that perception. All I know is that a few users switching from openSUSE to 
Fedora told us that they subjectively found our KDE updates to be more 
reliable than the ones from the openSUSE KDE4:* backports repos.

>> Another big issue is that people will be drawn to selectively picking
>> only some stuff from the backports repository while staying with the
>> official updates for other stuff, leading to an untestable combinatorial
>> explosion of possible update combinations.
> 
> Theoretically, yeah, that can happen. In practice such situations tend
> to occur pretty rarely and are addressed when they do happen. There is a
> little caveat here that I should have mentioned; MDV's backport policy
> discourages backports for key libraries - so no-one would push a new GTK
> + version as a backport, for instance. That'd just lead to large amounts
> of pain. This is why new versions of KDE for stable MDV releases tend to
> show up on KDE.org rather than MDV backports, because the new version of
> Qt that is usually included would break this guideline.

That's a big failure point of the backports model which you tried to sweep 
under the carpet. It severely limits the types of updates we can do. Indeed, 
we definitely need to push out Qt updates, and as Qt is backwards-compatible 
(modulo bugs), this should not be an issue. But selective upgrades would be 
painful or completely unsupported. Our solution of just pushing out the new 
Qt as an official update which everything else can thus rely on is much more 
effective.

A separate KDE backports repository (which is indeed what most distros with 
backports repositories are using, at least Kubuntu and openSUSE work that 
way too) is a very poor solution. For several of our packages, e.g. 
knetworkmanager, it'd mean we'd have to build THREE versions instead of one 
to provide full service:
* old knetworkmanager for the official updates (and I have no idea how we'd 
be supposed to maintain that as fixing bugs without upgrading is not quite 
feasible for knetworkmanager, it'd probably just bitrot),
* latest knetworkmanager built against the old KDE for the main backports,
* latest knetworkmanager built against the current KDE for the KDE 
backports.

>> (Now I know people can also selectively update
>> from our updates, but if things break, I can just tell them that
>> selective updates are not supported and that they should run "yum update"
>> and come back if their problem still happens after that.)
> 
> So, 'they just get told it's unsupported', as you suggested was a bad
> thing above? :)

Selective updates are just not supportable. But we "support" complete 
updates whereas the backports model considers both complete and selective 
updates from backports unsupported.

>> > And they know they can easily fall back to what's in /updates if they
>> > find /backports to be broken; it gives them an escape route.
>> 
>> That's the only advantage of that model,
> 
> No, it's a side benefit. The main advantage is that those who want
> stable updates can have stable updates, and those who want adventurous
> upgrades can have those.

That assumes that version upgrades are adventurous in the first place.

>> I'm not sure it's worth the
>> drawbacks.
> 
> As I said, if we are happy to jettison many users who'd prefer more
> stable updates, you're correct. If we'd rather keep those users, you're
> not. That's the underlying question.

Most of those users can actually deal with our model of well-tested version 
upgrades (and the rare regression fixes pushed directly to stable to 
minimize exposure time) just fine. I think that compared with distros which 
offer both conservative updates and "backports" (i.e. version upgrades), 
while providing similar currentness as "backports", we're closer to the 
conservative updates in reliability, due to several factors:
* packagers know those are official updates and will be more careful about 
what to push there and when than for an "unsupported" repository (in fact, 
the definition of "backports" often includes "may break things", which leads 
packagers to push disruptive stuff in there which would be best saved for 
Rawhide, i.e. I'm worried packagers will treat backports like our unofficial 
kde-redhat unstable repository rather than like our official updates),
* all our testing for updates to releases focuses on the one updates-testing 
repository, and normally only well-tested updates get pushed,
* packagers do not have to worry about 2 update streams at the same time and 
therefore have more time to fix bugs,
* the many additional bugfixes coming from the new upstream versions offset 
the occasional regression.

If despite all this, the users are still inconvenienced by our updates, then 
chances are they'd be better served by a distribution which is more 
conservative altogether (e.g. RHEL/CentOS or Debian stable).

        Kevin Kofler