FESCo wants to ban direct stable pushes in Bodhi (urgent call for feedback)
Kevin Kofler
kevin.kofler at chello.at
Tue Mar 2 08:45:15 UTC 2010
Adam Williamson wrote:
> They can't expect anything currently either. We have no project-wide
> commitment to what kind of updates we will ship. Some maintainers ship
> version updates, some don't. There's already no consistency; this
> wouldn't make things any worse.
I'd argue it would, because having to do either 2 sets of updates or only
conservative updates will draw more maintainers towards the latter. But
anyway, I know we don't have a concrete policy there right now (mainly
because we couldn't agree on one).
> Wasn't it your side of the argument who was arguing against telling
> maintainers what to do, and saying it should be 'fun' to contribute to
> Fedora?
I didn't bring up the "fun" argument. My point is that banning direct stable
pushes prevents us from fixing things for our users ASAP when needed. This
is all part of duty, not fun.
Other people argued that excess bureaucracy also makes it "unfun" to
contribute, and they're right. But my point doesn't depend on that.
> If going through a testing process before shipping an update is
> sufficiently 'unfun' to be a problem, surely being required to ship
> updates you do not believe should be shipped is in the same boat? Not
> that I disagree the situation should be less unpredictable than it is at
> the moment, but I don't see why *one* solution to this (make updates
> slightly more cautious) is a terrible imposition on maintainers, but
> another (make all updates more adventurous) is not.
I don't think that having a policy of "always push new versions unless they
cause breakage, such as [list of bad things an update should never do], or
do not change anything for Fedora users (e.g. only fix bugs specific to
other platforms)" would make maintainership "unfun".
>> Yet, in practice, I still think a lot
>> more stuff gets backported in our updates repository than in those
>> backports repositories of other distros.
>
> Probably true (though in the case of Mandriva, maybe less than you'd
> expect; it's nothing like the wasteland that is Ubuntu backports).
> What's your point?
That your model would lead to fewer "backport" type updates and thus be
further away from what I and several others would like to see than what we
have now! You admitting this point also directly contradicts your claims
elsewhere that it wouldn't lead to fewer "backport" type updates being
provided.
(By the way, the "backports" term is unfortunate, because it's misleading,
as the "backports" are actually version upgrades whereas the conservative
updates are the ones getting backported fixes. Sadly, it's confusing Debian
jargon (coming from the fact that Debian backports are backports of packages
from unstable for stable) which made its way into other distributions.)
>> Also because the maintainers don't have to
>> worry about a conservative updates stream at the same time.
>
> Er, yes they do.
Sorry, what I meant was: "Also because OUR maintainers don't have to worry
about a conservative updates stream at the same time."
> That's precisely what Mandriva has - twin streams, one stable update
> stream, one backport stream. All maintainers are required to provide
> stable updates for packages in /main. They can *choose* to provide
> /backports upgrades too, but that doesn't absolve them of doing a safe
> /updates stream.
That was exactly my point. This removes the option to only provide the new
versions (which also fix bugs, often more than bugfixes conservatively
backported to the old version would ever fix because maintainers cannot be
expected to backport each and every bugfix for big upstream projects, and if
they did, they'd essentially ship the new version disguised as backported
bugfixes) and will lead many maintainers to choose to only provide the
conservative option. With our policy, they're encouraged to just drop the
legacy crap and ship the current version.
In addition, in practice, it's quite likely that bugs will often be answered
with "it's too hard to backport that particular fix, upgrade to the current
version from backports (or even Cooker/Rawhide/whatever) if you need it".
There's no way you can really force maintainers to provide an update stream
which is "stable" under that definition (no upgrades, only backported
fixes).
>> Having both,
>> we'd have to fix bugs in the conservative updates AND push backports.
>
> Again, Mandriva's been doing this for years, with a substantially
> smaller maintainer base, without it being a problem.
The problem is that there are fewer updates included in Mandriva backports
than in our official updates. You may not see this as a problem, but those
of us who think the version upgrades are an important characteristic of
Fedora (and I'm one of those) definitely do!
> Obviously the system isn't perfect, no system is; sometimes version bumps
> just go through as updates because it's way too much work to maintain a
> seventeen month bugfix backport list.
Indeed, that's why I'm saying it would be a significant extra burden for us
to have to maintain that conservative stream, you were the one denying that.
> But most of the time it works as advertised, and has a record of causing
> fewer problems for users of stable releases than the Fedora updates system
> has.
That may be true, but you're comparing apples with oranges. Users of that
kind of "stable releases" also don't get any of the improvements our users
get in the updates and even bugs will often only be fixed in the next
release.
I think that that kind of problems are actually extremely rare in our stable
updates (also because testing is used where it makes sense, and also because
regression fixes are often pushed directly to stable to minimize exposure
time) and that being paranoid about them would just degrade the overall
Fedora experience. (The phrase "throwing out the baby with the bathwater"
comes to my mind.)
>> Of
>> course that'd tempt maintainers to just skip the backports step. Whereas
>> our policy is to prefer resolving issues by pushing new upstream releases
>
> Er, is it? I thought we were discussing the fact that we don't *have* an
> updates policy. That may be the KDE SIG's policy, but it's not Fedora's,
> as far as I know.
Strange, I thought there was a policy written down somewhere, and it's even
possible it used to be there on some old web page or wiki we no longer use
and no longer is now.
The only current document I found which sorta implies this is:
https://fedoraproject.org/wiki/Staying_close_to_upstream_projects
"Staying close to upstream versions is helpful when doing version bumps for
updates that bring in new features. This prevents tedious backporting of
only security and bug fixes."
but it's not really a policy, it just says backporting is "tedious" and can
be "prevented".
Another part of that page which implies such a policy is in the "Exceptions"
section further on:
"For any major issues, waiting on a new release from upstream for a fix can
be too much of a delay. In these instances, it may be better to backport
those fixes from upstream or fix the issue by writing your own patch and do
an update in Fedora."
To me, this says backporting should only be done if there isn't a new
upstream release to upgrade to.
And this has been the de-facto policy all this time, see e.g.:
http://www.mail-archive.com/fedora-legacy-list@redhat.com/msg01494.html
and see the actual security updates being pushed. The common rule is,
backport a security fix only if there's no new version to upgrade to or if
there's a good reason not to upgrade (e.g. for OpenSSL where you'd have to
rebuild half the distro if you upgrade).
> Many instances have shown that it does not give us the bugfixes 'for
> free'. It comes with the cost of causing regressions. That may be a cost
> which we decide we want to bear, but portraying things in a Panglossian
> manner doesn't help your cause, as it just makes you look like you're
> denying there could ever possibly be any problems with your method.
But that's not a cost for the maintainer (unless the regression breaks
his/her own system). :-)
Well, having to fix the regression is. But that's much less work than
backporting every single bugfix, and not doing that would mean our updates
would have more bugs than now.
>> That also sucks. With Fedora's KDE updates, users can be sure that
>> they'll be as regression-free as humanely possible and we do all we can
>> to keep their updates stable.
>
> Which means...when they notice something's broken they let you know, and
> you fix it. What's the difference? Or are you seriously telling us
> you're perfect, and there's never been any problem in any KDE update?
> Your little 'as humanly possible' disclaimer suggests you're not really
> saying that, but you could've made it more obvious.
The difference is that our updates go through testing first and (ideally)
only get pushed to stable when there are no regressions left. This is quite
unlike the Kubuntu model where they throw out new KDE versions into some
backports repo as soon as they're built, with no testing, and any regression
reports are answered with "it's an unsupported backport" and often only
fixed with the next KDE version, if at all. (I'm not very familiar with what
Mandriva is doing, so it may suck a bit less there, but still, a not really
supported backports repository is not quite as reliable as our updates!)
>> On the other hand, users of distros using the
>> backports model just get told "backports are unsupported".
>
> No, they don't. please don't misrepresent my words. That's not what I
> said at all. What they get told is 'well, backports are unsupported - as
> you know anyway because we spread that message very well and people know
> what they're getting into - but give me some information and I'll do my
> best to fix it'. Which usually happens.
Ideally yes, in practice I've read many complaints about this not happening,
at least for Kubuntu's and openSUSE's KDE backports (which are both in
dedicated KDE backports repos, not in generic backports, by the way), and as
much I'd like to believe Mandriva is perfect, I doubt it.
>> In fact their
>> builds of new KDE releases tend to carry only the same old distro patches
>> as the old version or even to be entirely vanilla, very little is done to
>> e.g. backport regression fixes from the branch. And KDE is just the
>> example I'm most familiar with, I'm pretty sure it's similar with other
>> stuff that gets updated in our stable updates vs. other distros'
>> backports repositories.
>
> That doesn't match my impression of how Mandriva at least handles KDE
> backports, and there's nothing intrinsic about the /backports model that
> means it must be this way. Or are you saying that if Fedora adopted a
> split repositories model, you would somehow be compelled to make
> the /backports releases suck? I don't understand why that would be. It'd
> be all the same work you currently do, just released in a different
> place.
There's a completely different feeling of responsibility when I'm about to
push something to an official updates repository than when I'm about to push
something for an unsupported backports repository.
It's also quite likely that backports-testing would get even fewer testers
than the current updates-testing (or maybe backports-testing would get most
of them, which would suck for updates-testing; either way, we'd be splitting
our already small tester base into 2!), leading to more regressions slipping
through.
And, while I'm certainly biased, my perception is also that our updates are
currently a lot more stable than the backports repositories other
distributions provide, which, if true, would also provide evidence for my
point. But it's hard to provide some objective metric to prove or disprove
that perception. All I know is that a few users switching from openSUSE to
Fedora told us that they subjectively found our KDE updates to be more
reliable than the ones from the openSUSE KDE4:* backports repos.
>> Another big issue is that people will be drawn to selectively picking
>> only some stuff from the backports repository while staying with the
>> official updates for other stuff, leading to an untestable combinatorial
>> explosion of possible update combinations.
>
> Theoretically, yeah, that can happen. In practice such situations tend
> to occur pretty rarely and are addressed when they do happen. There is a
> little caveat here that I should have mentioned; MDV's backport policy
> discourages backports for key libraries - so no-one would push a new GTK
> + version as a backport, for instance. That'd just lead to large amounts
> of pain. This is why new versions of KDE for stable MDV releases tend to
> show up on KDE.org rather than MDV backports, because the new version of
> Qt that is usually included would break this guideline.
That's a big failure point of the backports model which you tried to sweep
under the carpet. It severely limits the types of updates we can do. Indeed,
we definitely need to push out Qt updates, and as Qt is backwards-compatible
(modulo bugs), this should not be an issue. But selective upgrades would be
painful or completely unsupported. Our solution of just pushing out the new
Qt as an official update which everything else can thus rely on is much more
effective.
A separate KDE backports repository (which is indeed what most distros with
backports repositories are using, at least Kubuntu and openSUSE work that
way too) is a very poor solution. For several of our packages, e.g.
knetworkmanager, it'd mean we'd have to build THREE versions instead of one
to provide full service:
* old knetworkmanager for the official updates (and I have no idea how we'd
be supposed to maintain that as fixing bugs without upgrading is not quite
feasible for knetworkmanager, it'd probably just bitrot),
* latest knetworkmanager built against the old KDE for the main backports,
* latest knetworkmanager built against the current KDE for the KDE
backports.
>> (Now I know people can also selectively update
>> from our updates, but if things break, I can just tell them that
>> selective updates are not supported and that they should run "yum update"
>> and come back if their problem still happens after that.)
>
> So, 'they just get told it's unsupported', as you suggested was a bad
> thing above? :)
Selective updates are just not supportable. But we "support" complete
updates whereas the backports model considers both complete and selective
updates from backports unsupported.
>> > And they know they can easily fall back to what's in /updates if they
>> > find /backports to be broken; it gives them an escape route.
>>
>> That's the only advantage of that model,
>
> No, it's a side benefit. The main advantage is that those who want
> stable updates can have stable updates, and those who want adventurous
> upgrades can have those.
That assumes that version upgrades are adventurous in the first place.
>> I'm not sure it's worth the
>> drawbacks.
>
> As I said, if we are happy to jettison many users who'd prefer more
> stable updates, you're correct. If we'd rather keep those users, you're
> not. That's the underlying question.
Most of those users can actually deal with our model of well-tested version
upgrades (and the rare regression fixes pushed directly to stable to
minimize exposure time) just fine. I think that compared with distros which
offer both conservative updates and "backports" (i.e. version upgrades),
while providing similar currentness as "backports", we're closer to the
conservative updates in reliability, due to several factors:
* packagers know those are official updates and will be more careful about
what to push there and when than for an "unsupported" repository (in fact,
the definition of "backports" often includes "may break things", which leads
packagers to push disruptive stuff in there which would be best saved for
Rawhide, i.e. I'm worried packagers will treat backports like our unofficial
kde-redhat unstable repository rather than like our official updates),
* all our testing for updates to releases focuses on the one updates-testing
repository, and normally only well-tested updates get pushed,
* packagers do not have to worry about 2 update streams at the same time and
therefore have more time to fix bugs,
* the many additional bugfixes coming from the new upstream versions offset
the occasional regression.
If despite all this, the users are still inconvenienced by our updates, then
chances are they'd be better served by a distribution which is more
conservative altogether (e.g. RHEL/CentOS or Debian stable).
Kevin Kofler
More information about the devel
mailing list