4:18 p.m.
I see this package is requested to be pushed to stable updates of F8:
https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
Please don't do that. This package is broken - compiz won't start with
it:
https://bugzilla.redhat.com/show_bug.cgi?id=433728
--
Bojan
5:07 p.m.
2008/3/5, Bojan Smojver <bojan(a)rexursive.com>:
I see this package is requested to be pushed to stable updates of F8:
https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
Please don't do that. This package is broken - compiz won't start with
it:
https://bugzilla.redhat.com/show_bug.cgi?id=433728
And it crashes when NetworkManager loads the rt61pci driver during boot..
5:24 p.m.
Bojan Smojver wrote:
I see this package is requested to be pushed to stable updates of
F8:
https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
Please don't do that. This package is broken - compiz won't start with
it:
https://bugzilla.redhat.com/show_bug.cgi?id=433728
b43 seems unstable also.
--
Jonathan Steffan
daMaestro
Fedora Unity - http://fedoraunity.org/
GPG Fingerprint: 93A2 3E2F DC26 5570 3472 5B16 AD12 6CE7 0D86 AF59
8:41 p.m.
On Wednesday 05 March 2008 06:24:56 pm Jonathan Steffan wrote:
Bojan Smojver wrote:
> I see this package is requested to be pushed to stable updates of F8:
>
> https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
>
> Please don't do that. This package is broken - compiz won't start with
> it:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=433728
b43 seems unstable also.
and iwl4965.
In my case, its actually mac80211 going bonkers under the covers. Keep
forgetting to bz it so Linville fixes it... :)
--
Jarod Wilson
jwilson(a)redhat.com
5 a.m.
2008/3/7, Chuck Ebbert <cebbert(a)redhat.com>:
On 03/06/2008 03:01 PM, Bojan Smojver wrote:
> Bojan Smojver <bojan <at> rexursive.com> writes:
>
>> Please don't do that.
>
> I see our protests have been in vain :-(
>
Sorry, we had to release with known bugs. A new kernel will be in
updates-testing
very shortly.
You make it sound like someone forced your hand into breaking our setups.
9:51 a.m.
On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
Sorry, we had to release with known bugs. A new kernel will be in
updates-testing very shortly.
Why did you have to release with known bugs? Why not just wait until the bugs
are fixed? The last three kernel updates broke suspend for me...
-- Benjamin Kreuter
--
Message sent on: Fri Mar 7 10:48:43 EST 2008
10:15 a.m.
On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> Sorry, we had to release with known bugs. A new kernel will be in
> updates-testing very shortly.
Why did you have to release with known bugs? Why not just wait until the
bugs are fixed? The last three kernel updates broke suspend for me...
Uh... If we waited until all the known bugs were fixed, we'd never release
*any* kernel... :)
Despite this kernel making my own iwl4965 unusable, I was fully in favor of
releasing it. In theory, we fixed more problems than we caused, and you're
always welcome to keep running the prior kernel. (I'm actually running a
slightly modified 2.6.24.2-7.fc8 now).
--
Jarod Wilson
jwilson(a)redhat.com
11:13 a.m.
On Friday 07 March 2008 11:15:40 Jarod Wilson wrote:
Uh... If we waited until all the known bugs were fixed, we'd never release
*any* kernel... :)
Perhaps I chose the wrong words. What I intended to say is, why are we
releasing kernels that break key things like network drivers or suspend,
especially after we put so much effort into getting those things to work?
Despite this kernel making my own iwl4965 unusable, I was fully in
favor of
releasing it. In theory, we fixed more problems than we caused, and you're
always welcome to keep running the prior kernel. (I'm actually running a
slightly modified 2.6.24.2-7.fc8 now).
Perhaps I should pay more attention to the kernel development, before
commenting on it. I'm still running a 2.6.23.9 kernel, because it is the
most recent version that does what I need.
-- Ben
--
Jarod Wilson
jwilson(a)redhat.com
--
Message sent on: Fri Mar 7 12:11:25 EST 2008
11:28 a.m.
2008/3/7, Jarod Wilson <jwilson(a)redhat.com>:
On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> > Sorry, we had to release with known bugs. A new kernel will be in
> > updates-testing very shortly.
>
> Why did you have to release with known bugs? Why not just wait until
the
> bugs are fixed? The last three kernel updates broke suspend for me...
Uh... If we waited until all the known bugs were fixed, we'd never release
*any* kernel... :)
Despite this kernel making my own iwl4965 unusable, I was fully in favor
of
releasing it. In theory, we fixed more problems than we caused, and you're
always welcome to keep running the prior kernel. (I'm actually running a
slightly modified 2.6.24.2-7.fc8 now).
I don't think anyone expects perfection, but when breakage goes so far as to
encourage users to petition against an update being marked stable we might
want to reconsider deploying. Not doing so reflects poorly on Fedora as a
project to users in that our update policy looks dangerous to them and
discourages testers from reporting problems since their experience will be
that they are being ignored.
12:02 p.m.
On Friday 07 March 2008 12:28:17 pm David Nielsen wrote:
2008/3/7, Jarod Wilson <jwilson(a)redhat.com>:
> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
> > On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> > > Sorry, we had to release with known bugs. A new kernel will be in
> > > updates-testing very shortly.
> >
> > Why did you have to release with known bugs? Why not just wait until
>
> the
>
> > bugs are fixed? The last three kernel updates broke suspend for me...
>
> Uh... If we waited until all the known bugs were fixed, we'd never
> release *any* kernel... :)
>
> Despite this kernel making my own iwl4965 unusable, I was fully in favor
> of
> releasing it. In theory, we fixed more problems than we caused, and
> you're always welcome to keep running the prior kernel. (I'm actually
> running a slightly modified 2.6.24.2-7.fc8 now).
I don't think anyone expects perfection
Yeah, I know, that was sorta tongue-in-cheek. :)
but when breakage goes so far as
to encourage users to petition against an update being marked stable we
might want to reconsider deploying. Not doing so reflects poorly on Fedora
as a project to users in that our update policy looks dangerous to them and
discourages testers from reporting problems since their experience will be
that they are being ignored.
I suppose there would be that. But we're also supposed to be pretty
leading-edge, and we've been stuck on 2.6.23 for a while... Fun balancing act
sometimes...
I took those of us with problems to be a vocal minority in this case. For the
vast majority of people, that kernel works great, fixes issues with prior
kernels, etc., etc.
--
Jarod Wilson
jwilson(a)redhat.com
1:27 p.m.
On Fri, 7 Mar 2008 18:28:17 +0100
"David Nielsen" <gnomeuser(a)gmail.com> wrote:
2008/3/7, Jarod Wilson <jwilson(a)redhat.com>:
>
> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
> > On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> > > Sorry, we had to release with known bugs. A new kernel will be in
> > > updates-testing very shortly.
> >
> > Why did you have to release with known bugs? Why not just wait until
> the
> > bugs are fixed? The last three kernel updates broke suspend for me...
>
>
> Uh... If we waited until all the known bugs were fixed, we'd never release
> *any* kernel... :)
>
> Despite this kernel making my own iwl4965 unusable, I was fully in favor
> of
> releasing it. In theory, we fixed more problems than we caused, and you're
> always welcome to keep running the prior kernel. (I'm actually running a
> slightly modified 2.6.24.2-7.fc8 now).
I don't think anyone expects perfection, but when breakage goes so far as to
encourage users to petition against an update being marked stable we might
want to reconsider deploying. Not doing so reflects poorly on Fedora as a
project to users in that our update policy looks dangerous to them and
discourages testers from reporting problems since their experience will be
that they are being ignored.
We have an official way to protest an update. You go in the update
system and give it bad karma. If a package gets -3, it gets
auto-unpushed. At the moment the kernel is not exempt from this.
This particular update had a karma level of -1 when it was pushed. If
only two more of you on this thread had bothered to use the mechanisms
we have in place it would have been unpushed. So while I understand
your frustration, it seems to be a bit out of place to deride the
kernel developers for releasing something that people couldn't be
bothered to mark as bad.
josh
9:14 a.m.
On Fri, 7 Mar 2008 13:27:29 -0600, Josh Boyer wrote:
We have an official way to protest an update. You go in the update
system and give it bad karma. If a package gets -3, it gets
auto-unpushed. At the moment the kernel is not exempt from this.
Important information, Josh. Was this announced before? I still wish we
had a memo-list or similar where to inform the contributor community about
things like that.
9:17 a.m.
On Mon, 2008-03-10 at 15:14 +0100, Michael Schwendt wrote:
Important information, Josh. Was this announced before? I still wish
we
had a memo-list or similar where to inform the contributor community about
things like that.
I thought Luke announced Bodhi's karma system and the heuristics around
it via fedora-devel-announce. It should also be documented somewhere in
the bodhi docs. Luke?
--
Jesse Keating
Fedora -- All my bits are free, are yours?
3:41 p.m.
Jesse Keating wrote:
On Mon, 2008-03-10 at 15:14 +0100, Michael Schwendt wrote:
> Important information, Josh. Was this announced before? I still wish we
> had a memo-list or similar where to inform the contributor community about
> things like that.
I thought Luke announced Bodhi's karma system and the heuristics around
it via fedora-devel-announce. It should also be documented somewhere in
the bodhi docs. Luke?
I was aware of this so I know it was posted at some point, but after looking a
few minutes I couldn't find it. Most likely its buried in -devel I guess.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
5:57 p.m.
On Mon, Mar 10, 2008 at 10:17:18AM -0400, Jesse Keating wrote:
On Mon, 2008-03-10 at 15:14 +0100, Michael Schwendt wrote:
> Important information, Josh. Was this announced before? I still wish we
> had a memo-list or similar where to inform the contributor community about
> things like that.
I thought Luke announced Bodhi's karma system and the heuristics around
it via fedora-devel-announce. It should also be documented somewhere in
the bodhi docs. Luke?
Yep, I announced this feature in "Bodhi & Fedora 8 Updates" on October
22nd. This feature is also documented.
luke
1:15 p.m.
On 03/07/2008 02:27 PM, Josh Boyer wrote:
On Fri, 7 Mar 2008 18:28:17 +0100
"David Nielsen" <gnomeuser(a)gmail.com> wrote:
> 2008/3/7, Jarod Wilson <jwilson(a)redhat.com>:
>> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
>>> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
>>>> Sorry, we had to release with known bugs. A new kernel will be in
>>>> updates-testing very shortly.
>>> Why did you have to release with known bugs? Why not just wait until
>> the
>>> bugs are fixed? The last three kernel updates broke suspend for me...
>>
>> Uh... If we waited until all the known bugs were fixed, we'd never release
>> *any* kernel... :)
>>
>> Despite this kernel making my own iwl4965 unusable, I was fully in favor
>> of
>> releasing it. In theory, we fixed more problems than we caused, and you're
>> always welcome to keep running the prior kernel. (I'm actually running a
>> slightly modified 2.6.24.2-7.fc8 now).
>
> I don't think anyone expects perfection, but when breakage goes so far as to
> encourage users to petition against an update being marked stable we might
> want to reconsider deploying. Not doing so reflects poorly on Fedora as a
> project to users in that our update policy looks dangerous to them and
> discourages testers from reporting problems since their experience will be
> that they are being ignored.
We have an official way to protest an update. You go in the update
system and give it bad karma. If a package gets -3, it gets
auto-unpushed. At the moment the kernel is not exempt from this.
People should file bugs instead of just posting a comment in Bodhi.
Bodhi does not provide the same features as bugzilla for tracking
problems -- for example this weekend we had one person in Bodhi
posting the same -1 karma message three times in a row...
1:36 p.m.
On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
People should file bugs instead of just posting a comment in Bodhi.
Bodhi does not provide the same features as bugzilla for tracking
problems -- for example this weekend we had one person in Bodhi
posting the same -1 karma message three times in a row...
Why not both? Reference the bug in a karma posting.
--
Jesse Keating
Fedora -- All my bits are free, are yours?
1:50 p.m.
On 03/10/2008 02:36 PM, Jesse Keating wrote:
On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
> People should file bugs instead of just posting a comment in Bodhi.
> Bodhi does not provide the same features as bugzilla for tracking
> problems -- for example this weekend we had one person in Bodhi
> posting the same -1 karma message three times in a row...
Why not both? Reference the bug in a karma posting.
Negative karma postings should require a bug number...
1:54 p.m.
On Mon, Mar 10, 2008 at 02:50:15PM -0400, Chuck Ebbert wrote:
On 03/10/2008 02:36 PM, Jesse Keating wrote:
> On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
>> People should file bugs instead of just posting a comment in Bodhi.
>> Bodhi does not provide the same features as bugzilla for tracking
>> problems -- for example this weekend we had one person in Bodhi
>> posting the same -1 karma message three times in a row...
>
> Why not both? Reference the bug in a karma posting.
>
Negative karma postings should require a bug number...
Absolutely. We shouldn't be using Bohdi to track bugs against updates - they
should all be in BZ. And if we're going to allow -ve karma to revert a bug
fix, then it must be accompanied by a bug report indicating what's wrong.
Dan.
--
|: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
2:04 p.m.
On Mon, Mar 10, 2008 at 7:54 PM, Daniel P. Berrange <berrange(a)redhat.com> wrote:
On Mon, Mar 10, 2008 at 02:50:15PM -0400, Chuck Ebbert wrote:
> On 03/10/2008 02:36 PM, Jesse Keating wrote:
> > On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
> >> People should file bugs instead of just posting a comment in Bodhi.
> >> Bodhi does not provide the same features as bugzilla for tracking
> >> problems -- for example this weekend we had one person in Bodhi
> >> posting the same -1 karma message three times in a row...
> >
> > Why not both? Reference the bug in a karma posting.
> >
>
> Negative karma postings should require a bug number...
Absolutely. We shouldn't be using Bohdi to track bugs against updates - they
should all be in BZ. And if we're going to allow -ve karma to revert a bug
fix, then it must be accompanied by a bug report indicating what's wrong.
+1
2:18 p.m.
On Mon, 10 Mar 2008 18:54:54 +0000
"Daniel P. Berrange" <berrange(a)redhat.com> wrote:
On Mon, Mar 10, 2008 at 02:50:15PM -0400, Chuck Ebbert wrote:
> On 03/10/2008 02:36 PM, Jesse Keating wrote:
> > On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
> >> People should file bugs instead of just posting a comment in Bodhi.
> >> Bodhi does not provide the same features as bugzilla for tracking
> >> problems -- for example this weekend we had one person in Bodhi
> >> posting the same -1 karma message three times in a row...
> >
> > Why not both? Reference the bug in a karma posting.
> >
>
> Negative karma postings should require a bug number...
Absolutely. We shouldn't be using Bohdi to track bugs against updates - they
should all be in BZ. And if we're going to allow -ve karma to revert a bug
fix, then it must be accompanied by a bug report indicating what's wrong.
I've filed an RFE for this:
https://fedorahosted.org/bodhi/ticket/181
josh
3:44 p.m.
Josh Boyer wrote:
On Mon, 10 Mar 2008 18:54:54 +0000
"Daniel P. Berrange" <berrange(a)redhat.com> wrote:
> On Mon, Mar 10, 2008 at 02:50:15PM -0400, Chuck Ebbert wrote:
>> On 03/10/2008 02:36 PM, Jesse Keating wrote:
>>> On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
>>>> People should file bugs instead of just posting a comment in Bodhi.
>>>> Bodhi does not provide the same features as bugzilla for tracking
>>>> problems -- for example this weekend we had one person in Bodhi
>>>> posting the same -1 karma message three times in a row...
>>> Why not both? Reference the bug in a karma posting.
>>>
>> Negative karma postings should require a bug number...
> Absolutely. We shouldn't be using Bohdi to track bugs against updates - they
> should all be in BZ. And if we're going to allow -ve karma to revert a bug
> fix, then it must be accompanied by a bug report indicating what's wrong.
I've filed an RFE for this:
https://fedorahosted.org/bodhi/ticket/181
josh
This might be better overall, but I just want to point out it is definitely
going to increase the barrier of getting people to use bodhi, which (due to this
thread) is apparently not being used by testers enough as it is.
I would think it is better to have a field users *may* add a bug number to their
comment, so those with the motivation do, but requiring it is probably going to
reduce the negative karma postings.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
5:05 p.m.
On Mon, 10 Mar 2008 13:44:21 -0700
Andrew Farris <lordmorgul(a)gmail.com> wrote:
Josh Boyer wrote:
> On Mon, 10 Mar 2008 18:54:54 +0000
> "Daniel P. Berrange" <berrange(a)redhat.com> wrote:
>
>> On Mon, Mar 10, 2008 at 02:50:15PM -0400, Chuck Ebbert wrote:
>>> On 03/10/2008 02:36 PM, Jesse Keating wrote:
>>>> On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
>>>>> People should file bugs instead of just posting a comment in Bodhi.
>>>>> Bodhi does not provide the same features as bugzilla for tracking
>>>>> problems -- for example this weekend we had one person in Bodhi
>>>>> posting the same -1 karma message three times in a row...
>>>> Why not both? Reference the bug in a karma posting.
>>>>
>>> Negative karma postings should require a bug number...
>> Absolutely. We shouldn't be using Bohdi to track bugs against updates -
they
>> should all be in BZ. And if we're going to allow -ve karma to revert a bug
>> fix, then it must be accompanied by a bug report indicating what's wrong.
>
> I've filed an RFE for this:
>
> https://fedorahosted.org/bodhi/ticket/181
>
> josh
This might be better overall, but I just want to point out it is definitely
going to increase the barrier of getting people to use bodhi, which (due to this
thread) is apparently not being used by testers enough as it is.
I would think it is better to have a field users *may* add a bug number to their
comment, so those with the motivation do, but requiring it is probably going to
reduce the negative karma postings.
Fair points. The RFE is open for discussion and it's exact
implementation will likely need to reflect the results of bringing the
overall proposal to FESCo.
If we want to be overachievers, then we can make bodhi talk to bugzilla
and dump the user into a new bug report immediately before/after doing
a negative karma comment.
josh
5:01 a.m.
On Mon March 10 2008, Daniel P. Berrange wrote:
On Mon, Mar 10, 2008 at 02:50:15PM -0400, Chuck Ebbert wrote:
> On 03/10/2008 02:36 PM, Jesse Keating wrote:
> > On Mon, 2008-03-10 at 14:15 -0400, Chuck Ebbert wrote:
> >> People should file bugs instead of just posting a comment in Bodhi.
> >> Bodhi does not provide the same features as bugzilla for tracking
> >> problems -- for example this weekend we had one person in Bodhi
> >> posting the same -1 karma message three times in a row...
> >
> > Why not both? Reference the bug in a karma posting.
>
> Negative karma postings should require a bug number...
Absolutely. We shouldn't be using Bohdi to track bugs against updates -
they should all be in BZ. And if we're going to allow -ve karma to revert a
bug fix, then it must be accompanied by a bug report indicating what's
wrong.
Imho it should be possible to portion out bad karma via Bugzilla, i.e.
indicating in a bug report, that the bug is caused by an update for the
regarding package. I find it very annoying to use bodhi after I filed a bug
report and also to have to report it for every release seperately in bodhi,
e.g. when I found a bug in a package for F8 and the same version was also
released as an update for F7.
And also when it is possible to provide a bugzilla for a comment in bodhi,
this comment should also be posted to bugzilla to the regarding bug report
imho.
Regards,
Till
4:25 p.m.
On Fri, 2008-03-07 at 11:15 -0500, Jarod Wilson wrote:
On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> > Sorry, we had to release with known bugs. A new kernel will be in
> > updates-testing very shortly.
>
> Why did you have to release with known bugs? Why not just wait until the
> bugs are fixed? The last three kernel updates broke suspend for me...
Uh... If we waited until all the known bugs were fixed, we'd never release
*any* kernel... :)
Despite this kernel making my own iwl4965 unusable, I was fully in favor of
releasing it. In theory, we fixed more problems than we caused, and you're
always welcome to keep running the prior kernel. (I'm actually running a
slightly modified 2.6.24.2-7.fc8 now).
Yes, the real issue here is not all bugs, but regressions. Regressions
are a major problem for Aunt Tillie. Kernel regressions can result in an
unbootable, unusable system. I can't imagine ever deploying Fedora on
Aunt Tillie's machine for exactly this reason, kernel regressions.
Use case: Aunt Tillie diligently keeps her Fedora machine up to date. A
new kernel results in a regression with her hardware. Maybe it doesn't
even boot. What does she do? Can we really expect her to know how to
boot the previous kernel? How is she to even know it is the kernel that
broke? Does she even know what a kernel is? How does she fix it? Booting
the old kernel in GRUB is a one time deal. How does she make it stick?
How does she blacklist the broken kernel? What does she do when 6 more
broken kernels come through the update pipe?
What do *I* do to prevent this? Tell her to not update, and risk
security issues? Should I have blacklisted updating the kernel before
leaving her alone with the machine? Which still leaves the kernel
potentially vulnerable.
This is not theoretical, I ran into this very kind of problem in F7. F7
ran perfectly, initially. A kernel update (a bump from 2.6.21 to 2.6.22,
mind you...) resulted in a reboot loop on my wife's eMachines m6805
(x86_64) laptop. I even bugzilled it right away, though bugzilla's
wonderous search functionality is refusing to find it right now. Many
months and many kernel updates went by, all of them broken. It finally
got fixed when the bug was discovered in the rawhide kernel and ended up
on the F8 release blocker list.
This is a terrible user experience for *me*, let alone Aunt Tillie. I
can't imagine subjecting Aunt Tillie to this without help.
Now, I'm not saying I have the solution to this, and I'm not saying the
solution is easy. But IMHO this really needs to be addressed, somehow,
if Fedora is to ever truly be "ready for the desktop".
4:41 p.m.
Callum Lerwick wrote:
On Fri, 2008-03-07 at 11:15 -0500, Jarod Wilson wrote:
> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
>> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
>>> Sorry, we had to release with known bugs. A new kernel will be in
>>> updates-testing very shortly.
>> Why did you have to release with known bugs? Why not just wait until the
>> bugs are fixed? The last three kernel updates broke suspend for me...
> Uh... If we waited until all the known bugs were fixed, we'd never release
> *any* kernel... :)
>
> Despite this kernel making my own iwl4965 unusable, I was fully in favor of
> releasing it. In theory, we fixed more problems than we caused, and you're
> always welcome to keep running the prior kernel. (I'm actually running a
> slightly modified 2.6.24.2-7.fc8 now).
Yes, the real issue here is not all bugs, but regressions. Regressions
are a major problem for Aunt Tillie. Kernel regressions can result in an
unbootable, unusable system. I can't imagine ever deploying Fedora on
Aunt Tillie's machine for exactly this reason, kernel regressions.
Use case: Aunt Tillie diligently keeps her Fedora machine up to date. A
new kernel results in a regression with her hardware. Maybe it doesn't
even boot. What does she do? Can we really expect her to know how to
boot the previous kernel? How is she to even know it is the kernel that
broke? Does she even know what a kernel is? How does she fix it? Booting
the old kernel in GRUB is a one time deal. How does she make it stick?
How does she blacklist the broken kernel? What does she do when 6 more
broken kernels come through the update pipe?
What do *I* do to prevent this? Tell her to not update, and risk
security issues? Should I have blacklisted updating the kernel before
leaving her alone with the machine? Which still leaves the kernel
potentially vulnerable.
This is not theoretical, I ran into this very kind of problem in F7. F7
ran perfectly, initially. A kernel update (a bump from 2.6.21 to 2.6.22,
mind you...) resulted in a reboot loop on my wife's eMachines m6805
(x86_64) laptop. I even bugzilled it right away, though bugzilla's
wonderous search functionality is refusing to find it right now. Many
months and many kernel updates went by, all of them broken. It finally
got fixed when the bug was discovered in the rawhide kernel and ended up
on the F8 release blocker list.
This is a terrible user experience for *me*, let alone Aunt Tillie. I
can't imagine subjecting Aunt Tillie to this without help.
Now, I'm not saying I have the solution to this, and I'm not saying the
solution is easy. But IMHO this really needs to be addressed, somehow,
if Fedora is to ever truly be "ready for the desktop".
I don't think that "Aunt Tillie" should be using a bleeding edge Linux
distribution such as Fedora provides. And if "Nephew Johnie" installs it
for her and she has problems with it that she can not deal with herself I
think it is "Nephew Johny's" fault for installing it for her. What do you
think?
No distribution names mentioned here but there are several others that I
can think of that are much better suited for Newbies like your "Aunt Tillie".
--
David
4:58 p.m.
David Boles wrote:
Callum Lerwick wrote:
> On Fri, 2008-03-07 at 11:15 -0500, Jarod Wilson wrote:
>> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
>>> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
>>>> Sorry, we had to release with known bugs. A new kernel will be in
>>>> updates-testing very shortly.
>>> Why did you have to release with known bugs? Why not just wait
>>> until the
>>> bugs are fixed? The last three kernel updates broke suspend for me...
>> Uh... If we waited until all the known bugs were fixed, we'd never
>> release *any* kernel... :)
>>
>> Despite this kernel making my own iwl4965 unusable, I was fully in
>> favor of releasing it. In theory, we fixed more problems than we
>> caused, and you're always welcome to keep running the prior kernel.
>> (I'm actually running a slightly modified 2.6.24.2-7.fc8 now).
>
> Yes, the real issue here is not all bugs, but regressions. Regressions
> are a major problem for Aunt Tillie. Kernel regressions can result in an
> unbootable, unusable system. I can't imagine ever deploying Fedora on
> Aunt Tillie's machine for exactly this reason, kernel regressions.
>
> Use case: Aunt Tillie diligently keeps her Fedora machine up to date. A
> new kernel results in a regression with her hardware. Maybe it doesn't
> even boot. What does she do? Can we really expect her to know how to
> boot the previous kernel? How is she to even know it is the kernel that
> broke? Does she even know what a kernel is? How does she fix it? Booting
> the old kernel in GRUB is a one time deal. How does she make it stick?
> How does she blacklist the broken kernel? What does she do when 6 more
> broken kernels come through the update pipe?
>
> What do *I* do to prevent this? Tell her to not update, and risk
> security issues? Should I have blacklisted updating the kernel before
> leaving her alone with the machine? Which still leaves the kernel
> potentially vulnerable.
>
> This is not theoretical, I ran into this very kind of problem in F7. F7
> ran perfectly, initially. A kernel update (a bump from 2.6.21 to 2.6.22,
> mind you...) resulted in a reboot loop on my wife's eMachines m6805
> (x86_64) laptop. I even bugzilled it right away, though bugzilla's
> wonderous search functionality is refusing to find it right now. Many
> months and many kernel updates went by, all of them broken. It finally
> got fixed when the bug was discovered in the rawhide kernel and ended up
> on the F8 release blocker list.
>
> This is a terrible user experience for *me*, let alone Aunt Tillie. I
> can't imagine subjecting Aunt Tillie to this without help.
>
> Now, I'm not saying I have the solution to this, and I'm not saying the
> solution is easy. But IMHO this really needs to be addressed, somehow,
> if Fedora is to ever truly be "ready for the desktop".
>
I don't think that "Aunt Tillie" should be using a bleeding edge Linux
distribution such as Fedora provides. And if "Nephew Johnie" installs it
for her and she has problems with it that she can not deal with herself I
think it is "Nephew Johny's" fault for installing it for her. What do you
think?
Bzzz, wrong answer, regressions are bad . <period that is> Kernel regressions
are worse.
Just because Fedora is trying to be up2date with all the latest and greatest
stuff happening in FOSS land, does not make regressions all of a sudden OK.
You (we?) really need to stop thinking this way if we ever want Fedora to be a
serious distro, thinking this way inevitably leads to reducing Fedora to
nothing more then a toy distro.
Regards,
Hans
6:04 p.m.
Hans de Goede wrote:
David Boles wrote:
> Callum Lerwick wrote:
>> On Fri, 2008-03-07 at 11:15 -0500, Jarod Wilson wrote:
>>> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
>>>> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
>>>>> Sorry, we had to release with known bugs. A new kernel will be in
>>>>> updates-testing very shortly.
>>>> Why did you have to release with known bugs? Why not just wait
>>>> until the
>>>> bugs are fixed? The last three kernel updates broke suspend for me...
>>> Uh... If we waited until all the known bugs were fixed, we'd never
>>> release *any* kernel... :)
>>>
>>> Despite this kernel making my own iwl4965 unusable, I was fully in
>>> favor of releasing it. In theory, we fixed more problems than we
>>> caused, and you're always welcome to keep running the prior kernel.
>>> (I'm actually running a slightly modified 2.6.24.2-7.fc8 now).
>>
>> Yes, the real issue here is not all bugs, but regressions. Regressions
>> are a major problem for Aunt Tillie. Kernel regressions can result in an
>> unbootable, unusable system. I can't imagine ever deploying Fedora on
>> Aunt Tillie's machine for exactly this reason, kernel regressions.
>>
>> Use case: Aunt Tillie diligently keeps her Fedora machine up to date. A
>> new kernel results in a regression with her hardware. Maybe it doesn't
>> even boot. What does she do? Can we really expect her to know how to
>> boot the previous kernel? How is she to even know it is the kernel that
>> broke? Does she even know what a kernel is? How does she fix it? Booting
>> the old kernel in GRUB is a one time deal. How does she make it stick?
>> How does she blacklist the broken kernel? What does she do when 6 more
>> broken kernels come through the update pipe?
>>
>> What do *I* do to prevent this? Tell her to not update, and risk
>> security issues? Should I have blacklisted updating the kernel before
>> leaving her alone with the machine? Which still leaves the kernel
>> potentially vulnerable.
>>
>> This is not theoretical, I ran into this very kind of problem in F7. F7
>> ran perfectly, initially. A kernel update (a bump from 2.6.21 to 2.6.22,
>> mind you...) resulted in a reboot loop on my wife's eMachines m6805
>> (x86_64) laptop. I even bugzilled it right away, though bugzilla's
>> wonderous search functionality is refusing to find it right now. Many
>> months and many kernel updates went by, all of them broken. It finally
>> got fixed when the bug was discovered in the rawhide kernel and ended up
>> on the F8 release blocker list.
>>
>> This is a terrible user experience for *me*, let alone Aunt Tillie. I
>> can't imagine subjecting Aunt Tillie to this without help.
>>
>> Now, I'm not saying I have the solution to this, and I'm not saying the
>> solution is easy. But IMHO this really needs to be addressed, somehow,
>> if Fedora is to ever truly be "ready for the desktop".
>>
>
>
> I don't think that "Aunt Tillie" should be using a bleeding edge Linux
> distribution such as Fedora provides. And if "Nephew Johnie" installs it
> for her and she has problems with it that she can not deal with herself I
> think it is "Nephew Johny's" fault for installing it for her. What do
you
> think?
>
Bzzz, wrong answer, regressions are bad . <period that is> Kernel
regressions are worse.
Just because Fedora is trying to be up2date with all the latest and
greatest stuff happening in FOSS land, does not make regressions all of
a sudden OK.
You (we?) really need to stop thinking this way if we ever want Fedora
to be a serious distro, thinking this way inevitably leads to reducing
Fedora to nothing more then a toy distro.
Regards,
Hans
;-)
I know who you are Hans and I know what you do. I also know many of the
"Aunt Tillie" types. Some use Windows of some form. Some use Mac of some
form. Some use Linux of some form. None. Repeat none of them is in anyway
qualified to use Fedora until you developers make Fedora as 'idiot proof'
as Ubuntu. Oh yeah. And add all the eye-candy too.
Does anyone here recall a user by the name of Karl? Nice enough person but
he had some problems with Fedora. He switched to Ubuntu and ta da!! his
problems went away. Ubuntu is better than Fedora? I don't think so. But
not as 'cutting edge' or as modern and it it qualifies as almost idiot
proof. Out-of-the-box works if you wish. And they get to 'show' where the
3rd party codecs and stuff are located.
Is this current kernel problem acceptable? Of course not. But this is
Rawhide isn't it?
"Aunt Tillie" should stick to knitting and a simpler to use, more user
friendly, Linux. And I really don't want to post names here. but I can
think of five at this minute that are much, much more users friendly and
that have mouch more eye-candy. I used to use one of those. I switched to
Fedora becasue i got bored with them. When/if Fedora becomes 'idiot proof'
the idiots can have it. I will more on. <sigh>
--
David
6:21 p.m.
David Boles wrote:
Is this current kernel problem acceptable? Of course not. But this
is
Rawhide isn't it?
You are aware this was an F8 kernel in question, and not rawhide, right? I'm
100% in favor of breakage occurring to rawhide systems, and I've been using
rawhide and testing for years, but more caution is required for the releases.
That said, I still think if the kernel fixed significant bugs, and just causes
some users to 'be forced' to boot from a prior kernel.. thats a fairly minor
issue whether it was a regression or not. I would argue that learning what a
grub menu is, and how to boot a prior kernel if something breaks, is a barrier
to entry for a Linux user and should be considered as such.
I would never dream of installing linux on a person's computer and NOT telling
them what that boot menu is doing.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
7:22 p.m.
Andrew Farris wrote:
David Boles wrote:
> Is this current kernel problem acceptable? Of course not. But this is
> Rawhide isn't it?
You are aware this was an F8 kernel in question, and not rawhide,
right? I'm 100% in favor of breakage occurring to rawhide systems, and
I've been using rawhide and testing for years, but more caution is
required for the releases.
Yes I am and that is a bad thing. However just how would you expect a
developer to test this kernel on every combinations of hardware out there?
Is that not what 'testing' repos is for? And I would bet dollars to
doughnuts that those that are having problems did not test anything.
Wanna' bet? ;-) And, without the 'testing' just how would the problems
get fixed? And since the 'good', previous, kernel is still there where,
exactly, is this 'disaster'?
That said, I still think if the kernel fixed significant bugs, and
just
causes some users to 'be forced' to boot from a prior kernel.. thats a
fairly minor issue whether it was a regression or not. I would argue
that learning what a grub menu is, and how to boot a prior kernel if
something breaks, is a barrier to entry for a Linux user and should be
considered as such.
I would never dream of installing linux on a person's computer and NOT
telling them what that boot menu is doing.
Good for you. Many don't do that. It shows up all of the time that *most*,
not *all*, but most Linux Newbies install Fedora, or some other
distribution, without reading anything about it. No docs. No release
notes. No errata. Nothing. So just how do you fix that? And do you want to
know what I think the motive is? *Free* software. Stuff that they would
have to buy if they used other OS's.
let me say that I was *not* attacking anyone here. I think that you do a
fantastic job. But? Well 'things' to happen and then 'things' get fixed.
And there should not be a panic when that happens.
--
David
5:37 p.m.
On 03/09/2008 08:22 PM, David Boles wrote:
Andrew Farris wrote:
> You are aware this was an F8 kernel in question, and not rawhide,
> right? I'm 100% in favor of breakage occurring to rawhide systems,
> and I've been using rawhide and testing for years, but more caution is
> required for the releases.
Yes I am and that is a bad thing. However just how would you expect a
developer to test this kernel on every combinations of hardware out there?
Is that not what 'testing' repos is for? And I would bet dollars to
doughnuts that those that are having problems did not test anything.
Wanna' bet? ;-)
Um, this entire thread started because the person who had the problem
(Bojan) noticed the kernel was broken when it was still in the -testing
repo... and told us that it was broken before it got pushed to the
stable repo... and asked us not to with reasons why it was broken...
I'll take those doughnuts if you provide enough for everyone.
6:49 a.m.
On Sun, 09 Mar 2008 19:04:45 -0400, David Boles wrote:
Does anyone here recall a user by the name of Karl? Nice enough
person but
he had some problems with Fedora. He switched to Ubuntu and ta da!! his
problems went away.
The list archives over there don't confirm that impression. The main
Ubuntu users list has less traffic than fedora-list, but not only did the
subscribers ignore some of his posts (which is what the people on
fedora-list ought to have done, too), some of his posts are about various
similar+different problems.
Ubuntu is better than Fedora? I don't think so.
Certainly it was not a switch from "too many problems" to "no problems at
all".
Is this current kernel problem acceptable? Of course not. But this
is
Rawhide isn't it?
No, it isn't.
https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
It's an update of the type "you didn't play the guinea-pig while the
kernel was in updates-testing, so you deserve the breakage when it
appears in the stable repo".
7:05 a.m.
Michael Schwendt wrote:
On Sun, 09 Mar 2008 19:04:45 -0400, David Boles wrote:
> Does anyone here recall a user by the name of Karl? Nice enough person but
> he had some problems with Fedora. He switched to Ubuntu and ta da!! his
> problems went away.
The list archives over there don't confirm that impression. The main
Ubuntu users list has less traffic than fedora-list, but not only did the
subscribers ignore some of his posts (which is what the people on
fedora-list ought to have done, too), some of his posts are about various
similar+different problems.
> Ubuntu is better than Fedora? I don't think so.
Certainly it was not a switch from "too many problems" to "no problems at
all".
> Is this current kernel problem acceptable? Of course not. But this is
> Rawhide isn't it?
No, it isn't.
https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
It's an update of the type "you didn't play the guinea-pig while the
kernel was in updates-testing, so you deserve the breakage when it
appears in the stable repo".
Somebody has to test; this time not enough people did. I hate to see the blame
being lumped on kernel devs for letting this get pushed when they cannot
possibly test it on all hardware. For lack of testers everyone must test.
There is no viable alternative and the rest of our lives work exactly the same
way, not just with software.
If more people with the ability to handle recovery of their machines would run
with updates-testing enabled all the time this sort of thing wouldn't happen as
much.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
9:10 a.m.
On Mon, 10 Mar 2008 05:05:12 -0700, Andrew Farris wrote:
> https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
>
> It's an update of the type "you didn't play the guinea-pig while the
> kernel was in updates-testing, so you deserve the breakage when it
> appears in the stable repo".
Somebody has to test; this time not enough people did.
It was not even requested. No deadline either. Some of us would give it a
try, but we can't be everywhere. Especially not if such a kernel update
goes unnoticed and is pushed to stable after less than a week already,
which makes the window during which to fetch the update from a nearby
mirror even smaller. Further, I'm still not sure what impact negative
feedback has. Please do read through the comments and notice the
dates. Later, on one list, it was ruled that the update is supposed to fix
more bugs than it causes regression. Those people, who reported problems,
were ignored. Effectively, you punished one part of the community for not
testing, and you punished another part of the community for testing,
because they could not stop this update from hitting the stable repo.
I hate to see the blame
being lumped on kernel devs for letting this get pushed when they cannot
possibly test it on all hardware. For lack of testers everyone must test.
Make the "must" more explicit. Require tester feedback, or else extend the
testing-period and ask for more feedback. Explicitly request confirmation
of bug-fixes.
If more people with the ability to handle recovery of their machines
would run
with updates-testing enabled all the time this sort of thing wouldn't happen as
much.
Uh, not that argument again, please. As I've written multiple times in
related threads, I *would* enable updates-testing again as I've done for
quite a long time. But for that to happen I'd like to see whether my
feedback is valued or not. In several test-updates my -1 in bodhi has been
ignored (last with k3b breaking DVD burning), and incidents like that tell
me that testers are not appreciated.
5:08 p.m.
On Sun, 2008-03-09 at 17:41 -0400, David Boles wrote:
Callum Lerwick wrote:
> On Fri, 2008-03-07 at 11:15 -0500, Jarod Wilson wrote:
>> On Friday 07 March 2008 10:51:25 am Benjamin Kreuter wrote:
>>> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
>>>> Sorry, we had to release with known bugs. A new kernel will be in
>>>> updates-testing very shortly.
>>> Why did you have to release with known bugs? Why not just wait until the
>>> bugs are fixed? The last three kernel updates broke suspend for me...
>> Uh... If we waited until all the known bugs were fixed, we'd never release
>> *any* kernel... :)
>>
>> Despite this kernel making my own iwl4965 unusable, I was fully in favor of
>> releasing it. In theory, we fixed more problems than we caused, and you're
>> always welcome to keep running the prior kernel. (I'm actually running a
>> slightly modified 2.6.24.2-7.fc8 now).
>
> Yes, the real issue here is not all bugs, but regressions. Regressions
> are a major problem for Aunt Tillie. Kernel regressions can result in an
> unbootable, unusable system. I can't imagine ever deploying Fedora on
> Aunt Tillie's machine for exactly this reason, kernel regressions.
>
> Use case: Aunt Tillie diligently keeps her Fedora machine up to date. A
> new kernel results in a regression with her hardware. Maybe it doesn't
> even boot. What does she do? Can we really expect her to know how to
> boot the previous kernel? How is she to even know it is the kernel that
> broke? Does she even know what a kernel is? How does she fix it? Booting
> the old kernel in GRUB is a one time deal. How does she make it stick?
> How does she blacklist the broken kernel? What does she do when 6 more
> broken kernels come through the update pipe?
>
> What do *I* do to prevent this? Tell her to not update, and risk
> security issues? Should I have blacklisted updating the kernel before
> leaving her alone with the machine? Which still leaves the kernel
> potentially vulnerable.
>
> This is not theoretical, I ran into this very kind of problem in F7. F7
> ran perfectly, initially. A kernel update (a bump from 2.6.21 to 2.6.22,
> mind you...) resulted in a reboot loop on my wife's eMachines m6805
> (x86_64) laptop. I even bugzilled it right away, though bugzilla's
> wonderous search functionality is refusing to find it right now. Many
> months and many kernel updates went by, all of them broken. It finally
> got fixed when the bug was discovered in the rawhide kernel and ended up
> on the F8 release blocker list.
>
> This is a terrible user experience for *me*, let alone Aunt Tillie. I
> can't imagine subjecting Aunt Tillie to this without help.
>
> Now, I'm not saying I have the solution to this, and I'm not saying the
> solution is easy. But IMHO this really needs to be addressed, somehow,
> if Fedora is to ever truly be "ready for the desktop".
>
I don't think that "Aunt Tillie" should be using a bleeding edge Linux
distribution such as Fedora provides. And if "Nephew Johnie" installs it
for her and she has problems with it that she can not deal with herself I
think it is "Nephew Johny's" fault for installing it for her. What do you
think?
No distribution names mentioned here but there are several others that I
can think of that are much better suited for Newbies like your "Aunt Tillie".
This is one of the big reasons why I'm now intalling centos on machines
(while I continue to use fedora).
The other is that I don't have time to upgrade multiple fedora installs
when they are no longer supported.
R
--
"It's a fine line between denial and faith.
It's much better on my side"
1:30 a.m.
Dnia 2008-03-10, o godz. 09:08:09 Rodd Clarkson <rodd(a)clarkson.id.au>
napisał(a):
The other is that I don't have time to upgrade multiple fedora
installs
when they are no longer supported.
If not the regressions which come with every
release, you'd need 1/100th of the
time and wouldn't be scared to update, aren't I right? For me, you install
CentOS for only one reason, not two.
Lam
11:17 p.m.
On Mon, 2008-03-10 at 07:30 +0100, Leszek Matok wrote:
Dnia 2008-03-10, o godz. 09:08:09 Rodd Clarkson
<rodd(a)clarkson.id.au>
napisał(a):
> The other is that I don't have time to upgrade multiple fedora installs
> when they are no longer supported.
If not the regressions which come with every release, you'd need 1/100th of the
time and wouldn't be scared to update, aren't I right? For me, you install
CentOS for only one reason, not two.
Actually, having built most of the boxes on which I've installed Fedora
for friends and fanily, I can tell you they are modest, but fit for
purpose, and that the problems being had here aren't likely to affect
them.
In fact, they usually use Fedora because they are sick of Windows
viruses and malware, but they rarely update their installs and more
often than not I upgrade because I need to do something and updates are
no longer available for there installs.
long term support is extremely attractive in some cases.
R
--
"It's a fine line between denial and faith.
It's much better on my side"
5:23 p.m.
On Sun, 2008-03-09 at 17:41 -0400, David Boles wrote:
I don't think that "Aunt Tillie" should be using a
bleeding edge Linux
distribution such as Fedora provides. And if "Nephew Johnie" installs
it
for her and she has problems with it that she can not deal with
herself I
think it is "Nephew Johny's" fault for installing it for her. What do
you
think?
I think you missed the point that this is just as painful for Joe Fedora
Developer and CS Major as it is Aunt Tillie.
User friendly for Aunt Tillie, if done properly, translates to quick and
painless for me. Bip, done, and I can get on with my life. Yes, I am
able to type arcane command lines and edit obscure config files. That
doesn't mean I want to. My first Linux was Slackware 3.2 in 1997. I
spent a week just trying to get PPP to work. Took months to get XFree86
to work. I've had enough of doing it the hard way for one lifetime,
thank you. This is the exact reason I *use* Fedora, instead of Slackware
or Gentoo or even Debian. Because it does the best job of Just Working
out of every distribution I've ever tried. But it could be better.
There's two issues here:
1) Preventing bugs.
2) Recovering from bugs.
The first one gets plenty of coverage. Everyone's trying to fix bugs.
But bugs are inevitable. So when a serious kernel regression shows up,
how do you recover from it? If you ask me, very little thought is being
put in to making system recovery user friendly.
It's easy to say "Just boot the previous kernel!", it's much harder to
actually do it. And make it stick. Each new kernel insists on making
itself default. And yes, I've tried setting UPDATEDEFAULT=no
in /etc/sysconfig/kernel, it doesn't seem to work reliably, and it's
definitely not user friendly, and it's not easy to change the default
kernel back once's its been changed. (Now you have to
edit /boot/grub/grub.conf...) I know what you're going to say, just
remove the broken kernel. But then, the next time you update, it just
gets installed again.
Direct manipulation. I, and Aunt Tillie, should be able to set the
default kernel right there in the GRUB menu, with one keypress. Click,
done, move on to something more productive. Such as filing a bug...
Even better, work out a way to detect a failed kernel boot and
automagically drop back to the last known working kernel. I seem to
remember a thread about this a while back. This would work for total
boot failures, but wouldn't really help driver regressions.
6:07 p.m.
Callum Lerwick wrote:
On Sun, 2008-03-09 at 17:41 -0400, David Boles wrote:
> I don't think that "Aunt Tillie" should be using a bleeding edge Linux
> distribution such as Fedora provides. And if "Nephew Johnie" installs
> it
> for her and she has problems with it that she can not deal with
> herself I
> think it is "Nephew Johny's" fault for installing it for her. What do
> you
> think?
I think you missed the point that this is just as painful for Joe Fedora
Developer and CS Major as it is Aunt Tillie.
"just as painful?" Do you mean the error or that "Joe Fedora
Developer"
does not know enough to press a key and select the other kernel to boot? I
have often wondered why Fedora chooses to hide the other kernel. Maybe
"Aunt Tillie", with a little help, could see that 'kernel A' does not
work. Hmm... so maybe I should try 'kernel B'.
User friendly for Aunt Tillie, if done properly, translates to quick
and
painless for me. Bip, done, and I can get on with my life. Yes, I am
able to type arcane command lines and edit obscure config files. That
doesn't mean I want to. My first Linux was Slackware 3.2 in 1997. I
spent a week just trying to get PPP to work. Took months to get XFree86
to work. I've had enough of doing it the hard way for one lifetime,
thank you. This is the exact reason I *use* Fedora, instead of Slackware
or Gentoo or even Debian. Because it does the best job of Just Working
out of every distribution I've ever tried. But it could be better.
There's two issues here:
1) Preventing bugs.
2) Recovering from bugs.
The first one gets plenty of coverage. Everyone's trying to fix bugs.
But bugs are inevitable. So when a serious kernel regression shows up,
how do you recover from it? If you ask me, very little thought is being
put in to making system recovery user friendly.
It's easy to say "Just boot the previous kernel!", it's much harder to
actually do it. And make it stick. Each new kernel insists on making
itself default. And yes, I've tried setting UPDATEDEFAULT=no
in /etc/sysconfig/kernel, it doesn't seem to work reliably, and it's
definitely not user friendly, and it's not easy to change the default
kernel back once's its been changed. (Now you have to
edit /boot/grub/grub.conf...) I know what you're going to say, just
remove the broken kernel. But then, the next time you update, it just
gets installed again.
Direct manipulation. I, and Aunt Tillie, should be able to set the
default kernel right there in the GRUB menu, with one keypress. Click,
done, move on to something more productive. Such as filing a bug...
Even better, work out a way to detect a failed kernel boot and
automagically drop back to the last known working kernel. I seem to
remember a thread about this a while back. This would work for total
boot failures, but wouldn't really help driver regressions.
As I said in another post. Fedora is more cutting edge. There are already
many, perhaps too many, "Aunt Tillie" distributions. User friendly and
loaded down with eye-candy.
Me? I like Fedora the way that it is. Let the other distributions imitate
Windows.
--
David
1:26 p.m.
On 03/09/2008 07:07 PM, David Boles wrote:
Callum Lerwick wrote:
> On Sun, 2008-03-09 at 17:41 -0400, David Boles wrote:
>> I don't think that "Aunt Tillie" should be using a bleeding edge
Linux
>> distribution such as Fedora provides. And if "Nephew Johnie" installs
>> it
>> for her and she has problems with it that she can not deal with
>> herself I
>> think it is "Nephew Johny's" fault for installing it for her. What
do
>> you
>> think?
>
> I think you missed the point that this is just as painful for Joe Fedora
> Developer and CS Major as it is Aunt Tillie.
"just as painful?" Do you mean the error or that "Joe Fedora
Developer"
does not know enough to press a key and select the other kernel to boot? I
have often wondered why Fedora chooses to hide the other kernel. Maybe
"Aunt Tillie", with a little help, could see that 'kernel A' does not
work. Hmm... so maybe I should try 'kernel B'.
The first thing I do after installing Fedora is remove the "hiddenmenu"
line in /etc/grub.conf... why _is_ that the default, anyway??
3:46 p.m.
Chuck Ebbert wrote:
On 03/09/2008 07:07 PM, David Boles wrote:
> Callum Lerwick wrote:
>> On Sun, 2008-03-09 at 17:41 -0400, David Boles wrote:
>>> I don't think that "Aunt Tillie" should be using a bleeding
edge Linux
>>> distribution such as Fedora provides. And if "Nephew Johnie"
installs
>>> it
>>> for her and she has problems with it that she can not deal with
>>> herself I
>>> think it is "Nephew Johny's" fault for installing it for her.
What do
>>> you
>>> think?
>> I think you missed the point that this is just as painful for Joe Fedora
>> Developer and CS Major as it is Aunt Tillie.
>
> "just as painful?" Do you mean the error or that "Joe Fedora
Developer"
> does not know enough to press a key and select the other kernel to boot? I
> have often wondered why Fedora chooses to hide the other kernel. Maybe
> "Aunt Tillie", with a little help, could see that 'kernel A' does
not
> work. Hmm... so maybe I should try 'kernel B'.
>
>
The first thing I do after installing Fedora is remove the "hiddenmenu"
line in /etc/grub.conf... why _is_ that the default, anyway??
Good question. I think its a terrible idea to hide the menu, and I've posted to
at least 5 forum questions where the user simply didn't realize pushing a button
at the hidden menu would let them see their windows install... "omg Fedora
install does not let me boot windows".
A pretty menu, but at what cost?
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
4:47 p.m.
On 03/10/2008 04:46 PM, Andrew Farris wrote:
Chuck Ebbert wrote:
> On 03/09/2008 07:07 PM, David Boles wrote:
>> Callum Lerwick wrote:
>>> On Sun, 2008-03-09 at 17:41 -0400, David Boles wrote:
>>>> I don't think that "Aunt Tillie" should be using a bleeding
edge Linux
>>>> distribution such as Fedora provides. And if "Nephew Johnie"
installs
>>>> it
>>>> for her and she has problems with it that she can not deal with
>>>> herself I
>>>> think it is "Nephew Johny's" fault for installing it for
her. What do
>>>> you
>>>> think?
>>> I think you missed the point that this is just as painful for Joe
>>> Fedora
>>> Developer and CS Major as it is Aunt Tillie.
>>
>> "just as painful?" Do you mean the error or that "Joe Fedora
Developer"
>> does not know enough to press a key and select the other kernel to
>> boot? I
>> have often wondered why Fedora chooses to hide the other kernel. Maybe
>> "Aunt Tillie", with a little help, could see that 'kernel A'
does not
>> work. Hmm... so maybe I should try 'kernel B'.
>>
>>
>
> The first thing I do after installing Fedora is remove the "hiddenmenu"
> line in /etc/grub.conf... why _is_ that the default, anyway??
>
Good question. I think its a terrible idea to hide the menu, and I've
posted to at least 5 forum questions where the user simply didn't
realize pushing a button at the hidden menu would let them see their
windows install... "omg Fedora install does not let me boot windows".
A pretty menu, but at what cost?
Well the timeout should be changed to something reasonable like 5-10 seconds.
5:44 p.m.
On Mon, 2008-03-10 at 17:47 -0400, Chuck Ebbert wrote:
Well the timeout should be changed to something reasonable like 5-10
seconds.
I agree, I often change that timeout to 13 or so, which gives me about
10 or so seconds to change or do something different than the default.
--
Mike Chambers
Madisonville, KY
"The best lil town on Earth!"
6:13 p.m.
Benjamin Kreuter wrote:
On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> Sorry, we had to release with known bugs. A new kernel will be in
> updates-testing very shortly.
Why did you have to release with known bugs? Why not just wait until the bugs
are fixed? The last three kernel updates broke suspend for me...
Then why are you installing them? If this kernel was known to break things,
then when it hits updates don't install it... not rocket science.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
2:58 a.m.
On Mon, Mar 10, 2008 at 12:13 AM, Andrew Farris <lordmorgul(a)gmail.com> wrote:
Benjamin Kreuter wrote:
> On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
>> Sorry, we had to release with known bugs. A new kernel will be in
>> updates-testing very shortly.
>
> Why did you have to release with known bugs? Why not just wait until the bugs
> are fixed? The last three kernel updates broke suspend for me...
Then why are you installing them? If this kernel was known to break things,
then when it hits updates don't install it... not rocket science.
When the yum update applet reports that new updates are available, I
always choose to accept them without checking in advance whether or
not they will render my computer unusable.
Also if I did decide to check if the update would break my system, it
currently is not possible to tell pup to ignore this particular
update.
A try-catch mechanism could provide a fallback solution to kernels.
Kernels could be marked as bootable, not_checked and non_bootable, and
grub could check this flag/status file and switch to the first
bootable kernel if the default is marked as non_bootable. Perhaps a
feature like this is already available in grub?
--
Trond Danielsen
3:22 a.m.
Trond Danielsen wrote:
On Mon, Mar 10, 2008 at 12:13 AM, Andrew Farris
<lordmorgul(a)gmail.com> wrote:
> Benjamin Kreuter wrote:
> > On Thursday 06 March 2008 19:29:23 Chuck Ebbert wrote:
> >> Sorry, we had to release with known bugs. A new kernel will be in
> >> updates-testing very shortly.
> >
> > Why did you have to release with known bugs? Why not just wait until the bugs
> > are fixed? The last three kernel updates broke suspend for me...
>
> Then why are you installing them? If this kernel was known to break things,
> then when it hits updates don't install it... not rocket science.
When the yum update applet reports that new updates are available, I
always choose to accept them without checking in advance whether or
not they will render my computer unusable.
Also if I did decide to check if the update would break my system, it
currently is not possible to tell pup to ignore this particular
update.
I understand this complaint, a method of marking an update as 'never install
this one' is not even available yet in PackageKit, the eventual replacement of
pup in rawhide, and it would be a very good feature to have.
A try-catch mechanism could provide a fallback solution to kernels.
Kernels could be marked as bootable, not_checked and non_bootable, and
grub could check this flag/status file and switch to the first
bootable kernel if the default is marked as non_bootable. Perhaps a
feature like this is already available in grub?
An automated choice no, but you can change the default kernel manually by
changing the 'default=' setting in /boot/grub/grub.conf. This can also be
changed by using the system-config-boot tool (which may not behave right in F8 I
do not remember, but in rawhide it works correctly).
In any case, the try-catch mechanism is in place by keeping the prior kernel
installed, and if it fails you choose the next one in the list next time. It is
not possible to try a kernel until its been booted, and when it does boot if the
kernel itself misbehaves it is not possible to do anything automatic because the
kernel has failed to do what it should do... the kernel is *everything*. In
some relatively minor kernel boot failures I'm sure a method could be there to
set a kernel as 'nonbootable' so it is not tried again, but I suspect most of
those situations the failure is so minor that the user is barely inconvenienced
by it (network or sound not working). In any major error it is just not going
to be feasible to build a 'recover from this disastrous kernel failure'
mechanism.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
2:23 p.m.
On Mon, 2008-03-10 at 01:22 -0700, Andrew Farris wrote:
In any case, the try-catch mechanism is in place by keeping the prior
kernel
installed, and if it fails you choose the next one in the list next time. It is
not possible to try a kernel until its been booted, and when it does boot if the
kernel itself misbehaves it is not possible to do anything automatic because the
kernel has failed to do what it should do...
It's easy. It's much like the dirty flag on filesystems. GRUB writes a
flag somewhere on disk, that indicates "Attempting to boot kernel
2.6.XXXXX". Then somewhere in the early userspace init scripts, you
change the flag to "Finished booting kernel 2.6.XXXX". The next time
grub starts up, it sees this flag and it now knows that that particular
kernel at least made it to userspace, which is a pretty good indication
it's not completely b0rked. If GRUB starts up and sees a "Attempting
boot" flag still remaining, then that's an indication that that kernel
failed to boot.
And I know this idea has come up before... Is this on anyone's to-do
list?
GRUB needs to be smarter. In the old days, unix machines had a monitor
firmware, the kernel never directly restarted the machine, it just
returned control to the firmware... "halt" used to mean something.
6:18 p.m.
On Mon, 10 Mar 2008 14:23:53 -0500, Callum Lerwick wrote:
On Mon, 2008-03-10 at 01:22 -0700, Andrew Farris wrote:
> In any case, the try-catch mechanism is in place by keeping the prior
> kernel installed, and if it fails you choose the next one in the list
> next time. It is not possible to try a kernel until its been booted,
> and when it does boot if the kernel itself misbehaves it is not
> possible to do anything automatic because the kernel has failed to do
> what it should do...
It's easy. It's much like the dirty flag on filesystems. GRUB writes a
flag somewhere on disk, that indicates "Attempting to boot kernel
2.6.XXXXX". Then somewhere in the early userspace init scripts, you
change the flag to "Finished booting kernel 2.6.XXXX". The next time
grub starts up, it sees this flag and it now knows that that particular
kernel at least made it to userspace, which is a pretty good indication
it's not completely b0rked. If GRUB starts up and sees a "Attempting
boot" flag still remaining, then that's an indication that that kernel
failed to boot.
And I know this idea has come up before... Is this on anyone's to-do
list?
I believe redhat has been carrying around a patch in grub which
essentially would allow the above, called "bootonce" patch. It allows
you to couple it with "reboot on kernel panic" and "failback" to a
known
working kernel... I've never tried it or had the use for it myself.
6:30 p.m.
Zing wrote:
On Mon, 10 Mar 2008 14:23:53 -0500, Callum Lerwick wrote:
> On Mon, 2008-03-10 at 01:22 -0700, Andrew Farris wrote:
>> In any case, the try-catch mechanism is in place by keeping the prior
>> kernel installed, and if it fails you choose the next one in the list
>> next time. It is not possible to try a kernel until its been booted,
>> and when it does boot if the kernel itself misbehaves it is not
>> possible to do anything automatic because the kernel has failed to do
>> what it should do...
> It's easy. It's much like the dirty flag on filesystems. GRUB writes a
> flag somewhere on disk, that indicates "Attempting to boot kernel
> 2.6.XXXXX". Then somewhere in the early userspace init scripts, you
> change the flag to "Finished booting kernel 2.6.XXXX". The next time
> grub starts up, it sees this flag and it now knows that that particular
> kernel at least made it to userspace, which is a pretty good indication
> it's not completely b0rked. If GRUB starts up and sees a "Attempting
> boot" flag still remaining, then that's an indication that that kernel
> failed to boot.
>
> And I know this idea has come up before... Is this on anyone's to-do
> list?
I believe redhat has been carrying around a patch in grub which
essentially would allow the above, called "bootonce" patch. It allows
you to couple it with "reboot on kernel panic" and "failback" to a
known
working kernel... I've never tried it or had the use for it myself.
This idea is still going to work for some failures and not for others. I
suppose it is better to have that than nothing, but you have to choose where to
place the 'ok the kernel is booted enough to count' position in the booting
sequence. For that to be really useful it needs to be after any kernel modules
are loaded and fairly deep into the daemon startup I would think.
IMO, The user still needs to know about the grub menu and the presence of
multiple kernels in the end.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
4:24 a.m.
Then why are you installing them? If this kernel was known to break
things,
then when it hits updates don't install it... not rocket science.
How many end-users knew this? Was this announced publicly where
end-users, not developers or testers, could notice this?
Cheers,
Debarshi
--
"From what we get, we can make a living; what we give, however, makes a life."
-- Arthur Ashe
5:24 a.m.
Debarshi Ray wrote:
> Then why are you installing them? If this kernel was known to
break things,
> then when it hits updates don't install it... not rocket science.
How many end-users knew this? Was this announced publicly where
end-users, not developers or testers, could notice this?
Cheers,
Debarshi
I don't know, thats why I said 'if'. It seems to me that most of the users
who
posted here about the issue knew beforehand, or at least the tone of their
emails suggests that.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
9:02 a.m.
On Monday 10 March 2008 06:24:44 Andrew Farris wrote:
I don't know, thats why I said 'if'. It seems to me that most of the users
who posted here about the issue knew beforehand, or at least the tone of
their emails suggests that.
Actually, it is usually pretty difficult for me to figure out if an update
will break something. For example, suspend seems to be eternally listed
as "working," but several kernel updates have broken it. Suspend to disk is
also listed as "working," but I have yet to actually observe this (I am told
it is my graphics driver that is to blame; this is not documented, or wasn't
the last time I checked).
It is also unreasonable to expect everyone to know everything about every
package on their system. My inbox is crowded with messages from ~3 active
mailing lists; imagine if I tried to keep up with messages from the mailing
lists for all of the software that I need working (~12 packages). It would
be pandemonium. That's why we trust package maintainers to maintain our
packages for us; that's also why we get unhappy when packages start
regressing, and why we have updates-testing for packages that might break
something or regress something, to help weed out bugs.
My point is, putting a kernel with obvious and known problems into the
non-testing updates may not be a wise idea. I understand that we are
supposed to be bleeding edge, but that shouldn't require us to bleed (much).
-- Benjamin Kreuter
--
Message sent on: Mon Mar 10 09:51:02 EDT 2008
5:29 a.m.
* Debarshi Ray <debarshi.ray(a)gmail.com> [20080310 10:24]:
> Then why are you installing them? If this kernel was known to
break things,
> then when it hits updates don't install it... not rocket science.
How many end-users knew this? Was this announced publicly where
end-users, not developers or testers, could notice this?
It is a delicate balance, weighing up stability versus always having
the latest packages. To be honest, I thought Fedora would be a middle
ground between the "extremes" of CentOS (stable, should not break) and
something like Gentoo or Rawhide (half your system never works
correctly because you're always running the development version or the
latest Alpha of everything).
This kernel update seems to be due to a foot planted deeply into the
"new version at all costs" territory, damn stability and damn what
users may think. I've rolled back to previous kernel.
I believe occurances like this dents peoples belief in Fedora as a
viable alternative. A *release* is not quite suitable for development
experimentation, that's what Rawhide is for, and no amount of excuses
and "it's not rocket surgery" comments will ever change that.
If Fedora has the principle of "we don't care what users think, we'll
break their systems whenever we like because we care more about
having a higher version number on something than some other
distribution, than doing some proper QA" - perhaps that should be
clearly stated so that end users have a chance of avoiding Fedora
completely.
If I wanted to run Rawhide, I would select it conciously. Pushing
known broken packages as updates in a *release* is showing real
contempt for your userbase.
/Anders
6:17 a.m.
Anders Karlsson wrote:
* Debarshi Ray <debarshi.ray(a)gmail.com> [20080310 10:24]:
>> Then why are you installing them? If this kernel was known to break things,
>> then when it hits updates don't install it... not rocket science.
> How many end-users knew this? Was this announced publicly where
> end-users, not developers or testers, could notice this?
It is a delicate balance, weighing up stability versus always having
the latest packages. To be honest, I thought Fedora would be a middle
ground between the "extremes" of CentOS (stable, should not break) and
something like Gentoo or Rawhide (half your system never works
correctly because you're always running the development version or the
latest Alpha of everything).
This kernel update seems to be due to a foot planted deeply into the
"new version at all costs" territory, damn stability and damn what
users may think. I've rolled back to previous kernel.
I believe occurances like this dents peoples belief in Fedora as a
viable alternative. A *release* is not quite suitable for development
experimentation, that's what Rawhide is for, and no amount of excuses
and "it's not rocket surgery" comments will ever change that.
If Fedora has the principle of "we don't care what users think, we'll
break their systems whenever we like because we care more about
having a higher version number on something than some other
distribution, than doing some proper QA" - perhaps that should be
clearly stated so that end users have a chance of avoiding Fedora
completely.
If I wanted to run Rawhide, I would select it conciously. Pushing
known broken packages as updates in a *release* is showing real
contempt for your userbase.
Again.. as has been stated previously by people with authority around here
(which I am not)... testers did not provide proper feedback on this kernel, via
the tooling in place for it [1]. Proper QA is not the responsibility of someone
else. It is the responsibility of the community. People's lack of
participation in how the system works are their own failure in this case.
Fedora devs have *countless* times proven they care what the users think about
this type of situation, but users who only share what they think after shit
breaks are useless to everyone.
[1] http://bodhi.fedoraproject.org
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
6:58 a.m.
* Andrew Farris <lordmorgul(a)gmail.com> [20080310 12:18]:
[snip]
Again.. as has been stated previously by people with authority around
here
(which I am not)... testers did not provide proper feedback on this kernel,
via the tooling in place for it [1]. Proper QA is not the responsibility
of someone else. It is the responsibility of the community. People's lack
of participation in how the system works are their own failure in this
case.
So the Fedora process is;
"If *we* break it, it's *your* fault because *you* did not test it" ?
I stand by my earlier comment that this shows little but contempt for
the userbase.
Fedora devs have *countless* times proven they care what the users
think
about this type of situation, but users who only share what they think
after shit breaks are useless to everyone.
Oh, I'm sorry. I thought that the idea was to have a three-tier system
with stable, testing and unstable repos (to borrow terms from Debian),
where the idea is to run stable if you intend on doing something
productive, testing if you participate in QA and unstable if you like
living on the edge.
Pushing known broken packages from testing to stable is not
acceptable, just because it did not receive enough QA in testing. Yes,
you get what you pay for, but work ethics should prevent these things
from happening.
Fedora 8 is released, it is supposed to be the "stable" branch, or are
you trying to tell me that Fedora 8 is still in development, even with
Fedora 9 pending? If so, are users supposed to run Fedora 7 to ensure
they are not being treated as lab-rats?
I must be incredibly dense - so I'd appreciate a thorough explanation
about this situation just so I know what to do to avoid having my
systems sabotagued in future. If I understand you right - you are
saying that Fedora does not want "just users" because they are totally
useless and just a burden to the project?
/Anders
7:17 a.m.
Anders Karlsson wrote:
* Andrew Farris <lordmorgul(a)gmail.com> [20080310 12:18]:
[snip]
> Again.. as has been stated previously by people with authority around here
> (which I am not)... testers did not provide proper feedback on this kernel,
> via the tooling in place for it [1]. Proper QA is not the responsibility
> of someone else. It is the responsibility of the community. People's lack
> of participation in how the system works are their own failure in this
> case.
So the Fedora process is;
"If *we* break it, it's *your* fault because *you* did not test it" ?
I stand by my earlier comment that this shows little but contempt for
the userbase.
> Fedora devs have *countless* times proven they care what the users think
> about this type of situation, but users who only share what they think
> after shit breaks are useless to everyone.
Oh, I'm sorry. I thought that the idea was to have a three-tier system
with stable, testing and unstable repos (to borrow terms from Debian),
where the idea is to run stable if you intend on doing something
productive, testing if you participate in QA and unstable if you like
living on the edge.
That is in inaccurate picture which probably has alot to do with your
frustration over this. Debian's unstable and testing are completely separate
repos not intended to get flowed together at a regular timeframe. Fedoras are
meant to be very temporary, for testing. Updates-testing is a place where
things desperately need testing interaction so they can be pushed because they
are only there because they fix known bugs... not because its new stuff that
happens to be unstable and is not getting dropped into the primary repo.
Pushing known broken packages from testing to stable is not
acceptable, just because it did not receive enough QA in testing. Yes,
you get what you pay for, but work ethics should prevent these things
from happening.
It was known to fix some bugs. Feedback was not provided about having major
breakage. What exactly is the proper procedure in your eyes here? Not push the
kernel which is known to fix bugs? I'm really curious.
Fedora 8 is released, it is supposed to be the "stable"
branch, or are
you trying to tell me that Fedora 8 is still in development, even with
Fedora 9 pending? If so, are users supposed to run Fedora 7 to ensure
they are not being treated as lab-rats?
You can answer this question, see above. Do you want known bugs to get fixed?
How many weeks do you want to wait for them? Would you prefer nothing get
released until its had an 'adequate' testing, and never get released if not? Do
you realize that because there are obviously not enough testers this means that
*most* updates will never get released?
Welcome to CentOS (just go install it now). That is not what Fedora is supposed
to be about, at least from my perspective and why I spend my time with it.
I must be incredibly dense - so I'd appreciate a thorough
explanation
about this situation just so I know what to do to avoid having my
systems sabotagued in future. If I understand you right - you are
saying that Fedora does not want "just users" because they are totally
useless and just a burden to the project?
That is not at all what I said, or meant. What I said is people with nothing
but complaints after the fact are helping nothing. That does not mean they
cannot use the distro, nor does it mean the community is not *intended* to work
for them and prevent this. What it means is they are not part of solution or
prevention. Thats it. The community as a whole should have prevented this; yet
in this case it failed to do so. A situation to be learned from, not one to
pack up camp and run away from.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
7:50 a.m.
Andrew Farris wrote:
Anders Karlsson wrote:
> * Andrew Farris <lordmorgul(a)gmail.com> [20080310 12:18]:
> [snip]
>> Again.. as has been stated previously by people with authority around
>> here (which I am not)... testers did not provide proper feedback on
>> this kernel, via the tooling in place for it [1]. Proper QA is not
>> the responsibility of someone else. It is the responsibility of the
>> community. People's lack of participation in how the system works
>> are their own failure in this case.
>
> So the Fedora process is;
> "If *we* break it, it's *your* fault because *you* did not test it" ?
>
> I stand by my earlier comment that this shows little but contempt for
> the userbase.
>
>> Fedora devs have *countless* times proven they care what the users
>> think about this type of situation, but users who only share what
>> they think after shit breaks are useless to everyone.
>
> Oh, I'm sorry. I thought that the idea was to have a three-tier system
> with stable, testing and unstable repos (to borrow terms from Debian),
> where the idea is to run stable if you intend on doing something
> productive, testing if you participate in QA and unstable if you like
> living on the edge.
That is in inaccurate picture which probably has alot to do with your
frustration over this. Debian's unstable and testing are completely
separate repos not intended to get flowed together at a regular
timeframe. Fedoras are meant to be very temporary, for testing.
Updates-testing is a place where things desperately need testing
interaction so they can be pushed because they are only there because
they fix known bugs... not because its new stuff that happens to be
unstable and is not getting dropped into the primary repo.
If this is the expectation, why not enable updates-testing by default
or at least encourage users to use it more? I for one have never
had it enabled (I do lots of rawhide testing though) as I was not aware
of its nature or need.
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
7:59 a.m.
Gary Thomas wrote:
Andrew Farris wrote:
> Anders Karlsson wrote:
>> * Andrew Farris <lordmorgul(a)gmail.com> [20080310 12:18]:
>> [snip]
>>> Again.. as has been stated previously by people with authority
>>> around here (which I am not)... testers did not provide proper
>>> feedback on this kernel, via the tooling in place for it [1].
>>> Proper QA is not the responsibility of someone else. It is the
>>> responsibility of the community. People's lack of participation in
>>> how the system works are their own failure in this case.
>>
>> So the Fedora process is;
>> "If *we* break it, it's *your* fault because *you* did not test it"
?
>>
>> I stand by my earlier comment that this shows little but contempt for
>> the userbase.
>>
>>> Fedora devs have *countless* times proven they care what the users
>>> think about this type of situation, but users who only share what
>>> they think after shit breaks are useless to everyone.
>>
>> Oh, I'm sorry. I thought that the idea was to have a three-tier system
>> with stable, testing and unstable repos (to borrow terms from Debian),
>> where the idea is to run stable if you intend on doing something
>> productive, testing if you participate in QA and unstable if you like
>> living on the edge.
>
> That is in inaccurate picture which probably has alot to do with your
> frustration over this. Debian's unstable and testing are completely
> separate repos not intended to get flowed together at a regular
> timeframe. Fedoras are meant to be very temporary, for testing.
> Updates-testing is a place where things desperately need testing
> interaction so they can be pushed because they are only there because
> they fix known bugs... not because its new stuff that happens to be
> unstable and is not getting dropped into the primary repo.
If this is the expectation, why not enable updates-testing by default
or at least encourage users to use it more? I for one have never
had it enabled (I do lots of rawhide testing though) as I was not aware
of its nature or need.
I think there probably does need to be alot more effect made at publicizing
updates-testing and the need for feedback through bodhi. It should not be
enabled by default because 'users' who aren't part of the feedback process,
and
who don't know how to recover from issues like this, should not be installing
updates-testing. I've always immediately enabled updates-testing on any stable
Fedora install I do for myself, but at present I only have rawhide installed on
machines. It is not a painful process to help provide updates-testing feedback;
the vast majority of the packages end up stable and are not broken like this one
was. Anyone who keeps reasonable backups of their data, and can handle manually
removing a package (and be able to decide which one was the problem), should be
using updates-testing in my opinion.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
11:55 a.m.
On Mon, Mar 10, 2008 at 4:59 AM, Andrew Farris <lordmorgul(a)gmail.com> wrote:
I think there probably does need to be alot more effect made at
publicizing
updates-testing and the need for feedback through bodhi.
Do you have specific suggestions on what we could do to communicate better?
-jef
8:51 a.m.
* Andrew Farris <lordmorgul(a)gmail.com> [20080310 13:18]:
Anders Karlsson wrote:
> * Andrew Farris <lordmorgul(a)gmail.com> [20080310 12:18]:
> [snip]
>> Again.. as has been stated previously by people with authority around
>> here (which I am not)... testers did not provide proper feedback on this
>> kernel, via the tooling in place for it [1]. Proper QA is not the
>> responsibility of someone else. It is the responsibility of the
>> community. People's lack of participation in how the system works are
>> their own failure in this case.
>
> So the Fedora process is;
> "If *we* break it, it's *your* fault because *you* did not test it" ?
>
> I stand by my earlier comment that this shows little but contempt for
> the userbase.
>
>> Fedora devs have *countless* times proven they care what the users think
>> about this type of situation, but users who only share what they think
>> after shit breaks are useless to everyone.
>
> Oh, I'm sorry. I thought that the idea was to have a three-tier system
> with stable, testing and unstable repos (to borrow terms from Debian),
> where the idea is to run stable if you intend on doing something
> productive, testing if you participate in QA and unstable if you like
> living on the edge.
That is in inaccurate picture which probably has alot to do with your
frustration over this. Debian's unstable and testing are completely
separate repos not intended to get flowed together at a regular timeframe.
Fedoras are meant to be very temporary, for testing. Updates-testing is a
place where things desperately need testing interaction so they can be
pushed because they are only there because they fix known bugs... not
because its new stuff that happens to be unstable and is not getting
dropped into the primary repo.
As a suggestion to improve things then, why not regularly have a
roll-call on the standard fedora-list asking for volunteers to
participate in using the updates-testing repo? Preferably with a short
few lines about "how to enable it, and how to disable it again should
you wish to stop participating" ?
> Pushing known broken packages from testing to stable is not
> acceptable, just because it did not receive enough QA in testing. Yes,
> you get what you pay for, but work ethics should prevent these things
> from happening.
It was known to fix some bugs. Feedback was not provided about having
major breakage. What exactly is the proper procedure in your eyes here?
Not push the kernel which is known to fix bugs? I'm really curious.
Fixing bugs is good, but pushing broken packages is bad. I'm not
saying bugs should not be fixed, but knowingly introducing breaks
elsewhere seems to be putting the cart before the horse a bit. If the
bugs fixed were security related, or critical (known data corruptor or
similar), I can see the need.
Proper process... Well, I'm just an end-user, so my opinion does not
weigh much. I would however expect that the process in place to handle
the situation, and in all fairness - it does seem to work quite well
most of the time.
> Fedora 8 is released, it is supposed to be the "stable"
branch, or are
> you trying to tell me that Fedora 8 is still in development, even with
> Fedora 9 pending? If so, are users supposed to run Fedora 7 to ensure
> they are not being treated as lab-rats?
You can answer this question, see above. Do you want known bugs to get
fixed? How many weeks do you want to wait for them? Would you prefer
nothing get released until its had an 'adequate' testing, and never get
released if not? Do you realize that because there are obviously not
enough testers this means that *most* updates will never get released?
Do I want bugs fixed - yes, but not with the pricetag of breaking
other functionality *knowingly*.
What you are describing is a problem with packages not getting
adequate exposure to testing - and I have made a suggestion earlier in
this mail about what could be done to help that.
Welcome to CentOS (just go install it now). That is not what Fedora
is
supposed to be about, at least from my perspective and why I spend my time
with it.
I've got CentOS equivalent running on some systems, and for my X60s I
went with F8 for driver reasons. I'm just surprised about the shift in
focus in Fedora, that's all.
The kernel update did not eat any data for me, it just broke X for
me. Rolling back kernel, problem solved.
> I must be incredibly dense - so I'd appreciate a thorough
explanation
> about this situation just so I know what to do to avoid having my
> systems sabotagued in future. If I understand you right - you are
> saying that Fedora does not want "just users" because they are totally
> useless and just a burden to the project?
That is not at all what I said, or meant. What I said is people with
nothing but complaints after the fact are helping nothing. That does not
mean they cannot use the distro, nor does it mean the community is not
*intended* to work for them and prevent this. What it means is they are
not part of solution or prevention. Thats it. The community as a whole
should have prevented this; yet in this case it failed to do so. A
situation to be learned from, not one to pack up camp and run away from.
And what else are the "just users" supposed to do, after the event,
when they sit with a system that for some reason no longer will let
them start X, use their wireless network or suspend/resume? They *are*
reporting the problem because someone broke something for them,
something that used to work. That's called 'regression', and that is
usually taken quite seriously in software development.
If the required steps are to get the packages exposed to more testers,
then I'd expect steps to be taken to recruit more testers. Advertise
the updates-testing repo more, encourage its use, prod the mailing
lists regularly for participants. Yes, we should learn from this, as
that is what good organisations do. Leverage the information in Smolt
as well, maybe have a push with that, allowing people to tick a box
saying that "Yeah, I could be interested in doing some QA for you"
when submitting the hardware profile and then use that to get as much
coverage of hardware as possible.
Thanks!
/Anders
4:01 p.m.
Anders Karlsson wrote:
* Andrew Farris <lordmorgul(a)gmail.com> [20080310 13:18]:
> Anders Karlsson wrote:
>> * Andrew Farris <lordmorgul(a)gmail.com> [20080310 12:18]:
>> [snip]
>>> Again.. as has been stated previously by people with authority around
>>> here (which I am not)... testers did not provide proper feedback on this
>>> kernel, via the tooling in place for it [1]. Proper QA is not the
>>> responsibility of someone else. It is the responsibility of the
>>> community. People's lack of participation in how the system works are
>>> their own failure in this case.
>> So the Fedora process is;
>> "If *we* break it, it's *your* fault because *you* did not test it"
?
>>
>> I stand by my earlier comment that this shows little but contempt for
>> the userbase.
>>
>>> Fedora devs have *countless* times proven they care what the users think
>>> about this type of situation, but users who only share what they think
>>> after shit breaks are useless to everyone.
>> Oh, I'm sorry. I thought that the idea was to have a three-tier system
>> with stable, testing and unstable repos (to borrow terms from Debian),
>> where the idea is to run stable if you intend on doing something
>> productive, testing if you participate in QA and unstable if you like
>> living on the edge.
> That is in inaccurate picture which probably has alot to do with your
> frustration over this. Debian's unstable and testing are completely
> separate repos not intended to get flowed together at a regular timeframe.
> Fedoras are meant to be very temporary, for testing. Updates-testing is a
> place where things desperately need testing interaction so they can be
> pushed because they are only there because they fix known bugs... not
> because its new stuff that happens to be unstable and is not getting
> dropped into the primary repo.
As a suggestion to improve things then, why not regularly have a
roll-call on the standard fedora-list asking for volunteers to
participate in using the updates-testing repo? Preferably with a short
few lines about "how to enable it, and how to disable it again should
you wish to stop participating" ?
Good ideas.
>> Pushing known broken packages from testing to stable is not
>> acceptable, just because it did not receive enough QA in testing. Yes,
>> you get what you pay for, but work ethics should prevent these things
>> from happening.
> It was known to fix some bugs. Feedback was not provided about having
> major breakage. What exactly is the proper procedure in your eyes here?
> Not push the kernel which is known to fix bugs? I'm really curious.
Fixing bugs is good, but pushing broken packages is bad. I'm not
saying bugs should not be fixed, but knowingly introducing breaks
elsewhere seems to be putting the cart before the horse a bit. If the
bugs fixed were security related, or critical (known data corruptor or
similar), I can see the need.
I agree, 100%. Broken packages shouldn't be shipped. The question is whether
something is seriously broken, or happens to be broken for 1 or 2 people with
strange hardware, odd configurations, or transient problems. Keeping in mind
this is not a distro intended to fit into the 'nothing will ever break'
category. The front page of fedoraproject.org should make that one pretty
clear. The minimal testing feedback received in that case would have made me (a
tester) think it was perfectly reasonable to push it... up to the time the push
comment was posted to bodhi.
Proper process... Well, I'm just an end-user, so my opinion does
not
weigh much. I would however expect that the process in place to handle
the situation, and in all fairness - it does seem to work quite well
most of the time.
>> Fedora 8 is released, it is supposed to be the "stable" branch, or are
>> you trying to tell me that Fedora 8 is still in development, even with
>> Fedora 9 pending? If so, are users supposed to run Fedora 7 to ensure
>> they are not being treated as lab-rats?
> You can answer this question, see above. Do you want known bugs to get
> fixed? How many weeks do you want to wait for them? Would you prefer
> nothing get released until its had an 'adequate' testing, and never get
> released if not? Do you realize that because there are obviously not
> enough testers this means that *most* updates will never get released?
Do I want bugs fixed - yes, but not with the pricetag of breaking
other functionality *knowingly*.
What you are describing is a problem with packages not getting
adequate exposure to testing - and I have made a suggestion earlier in
this mail about what could be done to help that.
I agree, more testing exposure *AND* participation is needed and is the real
root cause of this problem. Updates cannot be shipped knowing they are valid
unless someone tests them, and thats the point of my comments up to now. The
primary assumption of using Fedora is that it is as good as the community helps
make it... a drastically different scenario to RHEL and its important to understand.
> Welcome to CentOS (just go install it now). That is not what
Fedora is
> supposed to be about, at least from my perspective and why I spend my time
> with it.
I've got CentOS equivalent running on some systems, and for my X60s I
went with F8 for driver reasons. I'm just surprised about the shift in
focus in Fedora, that's all.
The kernel update did not eat any data for me, it just broke X for
me. Rolling back kernel, problem solved.
>> I must be incredibly dense - so I'd appreciate a thorough explanation
>> about this situation just so I know what to do to avoid having my
>> systems sabotagued in future. If I understand you right - you are
>> saying that Fedora does not want "just users" because they are totally
>> useless and just a burden to the project?
> That is not at all what I said, or meant. What I said is people with
> nothing but complaints after the fact are helping nothing. That does not
> mean they cannot use the distro, nor does it mean the community is not
> *intended* to work for them and prevent this. What it means is they are
> not part of solution or prevention. Thats it. The community as a whole
> should have prevented this; yet in this case it failed to do so. A
> situation to be learned from, not one to pack up camp and run away from.
And what else are the "just users" supposed to do, after the event,
when they sit with a system that for some reason no longer will let
them start X, use their wireless network or suspend/resume? They *are*
reporting the problem because someone broke something for them,
something that used to work. That's called 'regression', and that is
usually taken quite seriously in software development.
I think its bad, very bad. But the fact that it happened is not the end of the
world either. The regression can be fixed (unregressed!). It happens in
commercial software at *regular intervals* as well (although, the linux
community should be better than that).
If the required steps are to get the packages exposed to more
testers,
then I'd expect steps to be taken to recruit more testers. Advertise
the updates-testing repo more, encourage its use, prod the mailing
lists regularly for participants. Yes, we should learn from this, as
that is what good organisations do. Leverage the information in Smolt
as well, maybe have a push with that, allowing people to tick a box
saying that "Yeah, I could be interested in doing some QA for you"
when submitting the hardware profile and then use that to get as much
coverage of hardware as possible.
Thanks!
/Anders
That is excellent feedback and I hope there are documentation people taking
notice of this thread.
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
10:22 a.m.
On Mon, 10 Mar 2008 04:17:25 -0700, Andrew Farris wrote:
Again.. as has been stated previously by people with authority around
here
(which I am not)... testers did not provide proper feedback on this kernel, via
the tooling in place for it [1].
https://admin.fedoraproject.org/updates/F8/FEDORA-2008-2103
bodhi - 2008-02-28 21:44:05
This update has been pushed to testing
r1348 - 2008-03-01 01:31:17
b43 performance is awful, this looks like fixed in 2.6.24-13
bojan - 2008-03-03 00:30:52
Borked. See bug #433728.
Anonymous Tester - 2008-03-02 18:49:19
rt61pci will connect but only a for a while after which it loses connection and refuses to
reconnect unless you reboot. Same goes for -13
bodhi - 2008-03-06 16:37:42
This update has been pushed to stable
[...]
That was not proper feedback?
What bugs is the kernel supposed to fix?
-> Fixes RTC on powerpc platform (#433262)
No other bugzilla ticket is mentioned in bodhi!
https://admin.fedoraproject.org/updates/search/kernel
is not sorted in any meaningful way, but something which appears
to be random. On page 1, the kernel update is not even listed.
46 Results Found
1 2 3 all
ID Package Type Submitter Date Released
FEDORA-2007-2258 kernel-xen-2.6-2.6.20-2936.fc7 [bugfix] ehabkost 2007-09-28 21:24:58
FEDORA-2008-1422 kernel-2.6.23.15-80.fc7 [security] cebbert 2008-02-11 20:30:00
FEDORA-2008-2167 kernel-2.6.23.17-82.fc7 [bugfix] cebbert 2008-03-01 07:10:16
FEDORA-2007-1130 kernel-2.6.22.1-27.fc7 [security] cebbert 2007-07-20 12:33:20
FEDORA-2007-0409 kernel-2.6.21-1.3228.fc7 [security] davej 2007-06-13 14:06:03
FEDORA-2007-2702 kernel-2.6.23.1-10.fc7 [bugfix] cebbert 2007-10-29 19:08:08
FEDORA-2007-2986 kernel-2.6.23.1-49.fc8 [bugfix] cebbert 2007-11-09 23:35:17
FEDORA-2007-2298 kernel-2.6.22.7-85.fc7 [security] cebbert 2007-09-25 08:29:28
FEDORA-2007-1829 kernel-2.6.22.5-71.fc7 [bugfix] cebbert 2007-08-27 13:49:14
FEDORA-2008-1423 kernel-2.6.23.15-137.fc8 [security] cebbert 2008-02-11 20:30:09
FEDORA-2007-2987 kernel-xen-2.6-2.6.20-2943.fc7 [bugfix] ehabkost 2007-11-29 01:45:50
FEDORA-2008-1328 kernel-xen-2.6-2.6.21-2954.fc8 [bugfix] berrange
FEDORA-2008-0748 kernel-2.6.23.14-107.fc8 [security] kyle 2008-01-24 00:54:57
FEDORA-2007-2748 kernel-2.6.23.1-21.fc7 [bugfix] cebbert 2007-11-05 15:07:45
FEDORA-2008-1878 kvm-60-1.fc8, kernel-2.6.24.2-7.fc8 [enhancement] cebbert 2008-02-21
02:32:18
kernel-2.6.24.3-22.fc8 [bugfix] cebbert
FEDORA-2008-0984 kernel-2.6.23.14-115.fc8 [security] cebbert 2008-02-05 23:11:13
FEDORA-2008-1433 kernel-xen-2.6-2.6.21-2957.fc8 [security] ehabkost 2008-02-13
04:08:37
FEDORA-2007-2556 kernel-2.6.23.1-4.fc7 [enhancement] cebbert 2007-10-15 21:32:08
FEDORA-2008-1796 kernel-xen-2.6-2.6.21.7-2.fc8 [enhancement] markmc 2008-02-19
02:50:07
8:59 a.m.
On Thu, 6 Mar 2008 20:01:13 +0000 (UTC), Bojan Smojver wrote:
Bojan Smojver <bojan <at> rexursive.com> writes:
> Please don't do that.
I see our protests have been in vain :-(
Whatever was in the March 7th updates set killed my network config.
/etc/sysconfig/network-scripts/ifcfg-eth* got renamed with .bak
appended.
9:32 a.m.
As said many times in this topic, this kernel is problematic.
I do not have updatesd running (because of blocking yum access because
of long repos download time), so I'm checking update manually.
Today I saw this kernel in updates list, and didn't updated to it :P .
--
Jakub 'Livio' Rusinek
http://liviopl.jogger.pl/
1:18 p.m.
On Fri, 07 Mar 2008 16:32:03 +0100
"Jakub 'Livio' Rusinek" <jakub.rusinek(a)gmail.com> wrote:
As said many times in this topic, this kernel is problematic.
I do not have updatesd running (because of blocking yum access because
of long repos download time), so I'm checking update manually.
Today I saw this kernel in updates list, and didn't updated to it :P .
Did you mark it with bad karma in the updates system?
josh
6:51 p.m.
Josh Boyer wrote:
On Fri, 07 Mar 2008 16:32:03 +0100
"Jakub 'Livio' Rusinek" <jakub.rusinek(a)gmail.com> wrote:
> As said many times in this topic, this kernel is problematic.
>
> I do not have updatesd running (because of blocking yum access because
> of long repos download time), so I'm checking update manually.
>
> Today I saw this kernel in updates list, and didn't updated to it :P .
Did you mark it with bad karma in the updates system?
He shouldn't do that if he didn't install it and find out...
--
Andrew Farris <lordmorgul(a)gmail.com> www.lordmorgul.net
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
5897
days inactive
5905
days old
70 comments
28 participants
participants (28)
-
Anders Rayner-Karlsson -
Andrew Farris -
Benjamin Kreuter -
Bojan Smojver -
Callum Lerwick -
Christopher Aillon -
Chuck Ebbert -
Daniel P. Berrange -
David -
David Nielsen -
Debarshi Ray -
drago01 -
Gary Thomas -
Hans de Goede -
Jakub 'Livio' Rusinek -
Jarod Wilson -
Jef Spaleta -
Jesse Keating -
Jonathan Steffan -
Josh Boyer -
Leszek Matok -
Luke Macken -
Michael Schwendt -
Mike Chambers -
Rodd Clarkson -
Till Maas -
Trond Danielsen -
Zing