11 a.m.
Hi,
I recently packaged and pushed an update for fedmsg-meta-fedora-infrastructure
to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am sure that
testing a package surely takes more than 40 secs. This makes me really curious
that are the packages really being tested before giving out the karma.
After going through messages in datagrepper[2][3], I found that few people are
giving out karma in one go (4-5 packages under a minute). If these packages
really are not-tested and the karma are given out randomly then I am sure that
this sure going to affect the release, infrastructure and our users.
Does anybody know what is going on?
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2016-f2331b2c8b
[2] https://apps.fedoraproject.org/datagrepper/raw?user=buvaneshkumar
[3] https://apps.fedoraproject.org/datagrepper/raw?user=prakashmishra1598
--
Sayan Chowdhury <https://sayanchowdhury.dgplug.org/>
Senior Software Engineer, Fedora Engineering - Emerging Platform
GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B
Proud to work at The Open Organization!
11:05 a.m.
Hi,
positive karma on a set of packages within short time can happen when
people use fedora-easy-karma. Thats my way of giving karma too, I use
the stuff for some hours, the I run fedora-easy-karma and give feedback
on all packages i've tested (and of course skip the ones I did not test ;).
But +1 just 40 seconds after push is really bad... Sometimes I think
people just give +1 to get the badges...
Greetings,
Christian
On 07/10/2016 06:00 PM, Sayan Chowdhury wrote:
Hi,
I recently packaged and pushed an update for fedmsg-meta-fedora-infrastructure
to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am sure that
testing a package surely takes more than 40 secs. This makes me really curious
that are the packages really being tested before giving out the karma.
After going through messages in datagrepper[2][3], I found that few people are
giving out karma in one go (4-5 packages under a minute). If these packages
really are not-tested and the karma are given out randomly then I am sure that
this sure going to affect the release, infrastructure and our users.
Does anybody know what is going on?
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2016-f2331b2c8b
[2] https://apps.fedoraproject.org/datagrepper/raw?user=buvaneshkumar
[3] https://apps.fedoraproject.org/datagrepper/raw?user=prakashmishra1598
11:18 a.m.
On Sun, 2016-07-10 at 21:30 +0530, Sayan Chowdhury wrote:
Hi,
I recently packaged and pushed an update for fedmsg-meta-fedora-infrastructure
to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am sure that
testing a package surely takes more than 40 secs. This makes me really curious
that are the packages really being tested before giving out the karma.
After going through messages in datagrepper[2][3], I found that few people are
giving out karma in one go (4-5 packages under a minute). If these packages
really are not-tested and the karma are given out randomly then I am sure that
this sure going to affect the release, infrastructure and our users.
Does anybody know what is going on?
There was a testing sprint run by Sumantro recently, and generally he's
been signing up a lot of new testers. Sumantro, can you please check
and make sure folks are properly testing these updates? This update for
instance is involved in producing human-readable forms of fedmsgs; it
would actually be quite complex to test properly, I believe, as you
would need to have a whole test fedmsg hub set up.
"Install the package and see if the system breaks" is an appropriate
form of testing for some packages, but not all, many are not at all
related to typical desktop system functionality. Folks do need to make
sure they understand what a package is for and have actually tested its
functionality before posting feedback.
I do note that the 'Positive feedback' section of
https://fedoraproject.org/wiki/QA:Update_feedback_guidelines is a
little simplistic here, and I'll try to update that.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:19 p.m.
Adam Williamson <adamwill(a)fedoraproject.org> writes:
"Install the package and see if the system breaks" is an
appropriate
form of testing for some packages, but not all, many are not at all
related to typical desktop system functionality. Folks do need to make
sure they understand what a package is for and have actually tested its
functionality before posting feedback.
I was bitten by this last week but didn't have time to post about it.
Something recently reviewed that I didn't consider suitable for stable
-- partly because I was asking legal about it, though legal seems to be
a black hole -- was automatically pushed through to f24 stable. (I got
in and edited the other releases to stop the same thing happening for
other releases as soon as I realized what had happened but too late for
f24.)
I didn't realize that this automation actually got things pushed rather
than simply marked as able to be pushed. That's likely documented,
though I haven't had time to check the documentation, and presumably
it's my fault. However, I guess it's not sufficiently obvious to new
maintainers and I think it's wrong for it to be the default in bodhi.
I think all the +1 comments simply said "LGTM" whose meaning I actually
had to guess; it's not something I'm used to as an experienced
maintainer and English speaker, and it's not in VERA via dict. Also if
people are going to comment on release quality, perhaps they could look
at upstream issues for the relevant version, particularly ones
specifically relevant to Fedora. In this case I've raised a number,
mostly not addressed in the upstream release, so I'm less convinced as a
user that it looked good.
12:31 p.m.
On Mon, Jul 11, 2016 at 10:19 AM, Dave Love <d.love(a)liverpool.ac.uk> wrote:
I didn't realize that this automation actually got things pushed
rather
than simply marked as able to be pushed. That's likely documented,
though I haven't had time to check the documentation, and presumably
it's my fault. However, I guess it's not sufficiently obvious to new
maintainers and I think it's wrong for it to be the default in bodhi.
It's very simple to control whether something is pushed automatically, and
it's also easy
to control the amount of karma required for the automatic push. Not only
is it documented, it is in the file you
modify when submitting the update. You have to READ. If you are
submitting updates it is assumed you
understand what you are doing. LGTM is Looks Good To Me.
9:34 a.m.
"Gerald B. Cox" <gbcox(a)bzb.us> writes:
It's very simple to control whether something is pushed
automatically, and
it's also easy
to control the amount of karma required for the automatic push.
Yes, I edited it.
Not only
is it documented, it is in the file you
modify when submitting the update. You have to READ.
What file? There is tick box at the bottom of the web page that you
might assume had sensible defaults, and this isn't the same interface as
the one I originally used to do this, which may have read something
different.
If you are
submitting updates it is assumed you
understand what you are doing.
Maybe I'm uniquely incompetent, unable to hold all the complexity in my
head between reviews/updates, and unable to deal with the breakage and
nasty interfaces of the tools, but experience suggests that's unlikely.
I may simply be willing to admit to errors and confusion in the face of
hostile responses to attempts to improve things; I see that too often,
and assume it drives off potential contributors. Are we expected to
stop packaging if we fail to understand something about the pretty
involved process -- which isn't exactly stable anyhow?
This is about actual problems due to what I consider a bad default and
understanding what it does. You can address that either by driving off
volunteers or trying to improve the process.
LGTM is Looks Good To Me.
Yes, I know, but it's not necessarily obvious, and also is useless
without the value of "LG", or possibly "M". If someone runs one of
my
packages successfully on their HPC system, or fails to, it's useful to
know. "Looks good" from someone without any context (like knowing what
"M" does) isn't useful.
It's obviously easy to make unjustified assumptions about what's
obvious, even if we can READ English. It's not as if "karma" even seems
to be used according to the dictionary, and some of this is exposed to
users, if not as badly as it used to be.
--
DOMINUS ILLUMINATIO MEA
1:37 a.m.
I got karma on packages that were in stable already for a long time. The karma system is
too tolerant in my eyes.
Why is it possible to give +1 to stable packages? Bodhi may deny or at least impede such
actions, -1 or neutral comments should be still possible though for users having issues.
In case of testing packages, maybe it's a good idea to delay write access to karma
system for some hours after new package got it into bodhi. We already have a blocking
phase for edits from maintainer there, could it be extended to the karma system for both
pending and testing status?
1:56 a.m.
On Mon, Jul 11, 2016 at 9:37 AM, Raphael Groner
<raphgro(a)fedoraproject.org> wrote:
Why is it possible to give +1 to stable packages? Bodhi may deny or
at least impede such actions, -1 or neutral comments should be still possible though for
users having issues.
I disagree with allowing -1. At best, it's meaningless after a package
is in stable, and may actually cause problems through confusion.
I think disallowing *any* kind of comments after a package is in
stable would be ok -- the comment field could be replaced with a link
that points to a Bugzilla form prepopulated with the correct product,
component, version, and a link to the update in the description.
3:08 a.m.
Hi,
On Mon, Jul 11, 2016 at 12:26 PM, Ville Skyttä <ville.skytta(a)iki.fi> wrote:
On Mon, Jul 11, 2016 at 9:37 AM, Raphael Groner
<raphgro(a)fedoraproject.org> wrote:
> Why is it possible to give +1 to stable packages? Bodhi may deny or at least impede
such actions, -1 or neutral comments should be still possible though for users having
issues.
I disagree with allowing -1. At best, it's meaningless after a package
is in stable, and may actually cause problems through confusion.
I think disallowing *any* kind of comments after a package is in
stable would be ok -- the comment field could be replaced with a link
that points to a Bugzilla form prepopulated with the correct product,
component, version, and a link to the update in the description.
Any kind of feedback after package gets into updates repository should
be denied, I can't think of a case where package moves to updates
repository and bodhi still allows to add comments/karma. Any feedback
after that can be either reported in bugzilla or to the package
maintainers directly. Its not like providing positive or negative
karma will make any changes to already pushed package to updates
repository.
Regards,
Parag.
3:49 a.m.
Any kind of feedback after package gets into updates repository
should
be denied, I can't think of a case where package moves to updates
repository and bodhi still allows to add comments/karma. Any feedback
after that can be either reported in bugzilla or to the package
maintainers directly. Its not like providing positive or negative
karma will make any changes to already pushed package to updates
repository.
I do have valid use cases. We've had examples of important packages having been pushed
stable too quickly and breaking some computers (think kernel, mesa). The bodhi comments
can then be used to provide a description of the problem, suggest any workarounds or
fixes, and also link to more information e.g. in the bugzilla or common bugs wiki, etc.
The Bodhi pages are easy to find (if I perform a system update and the computer breaks, I
look at what was included in the last update, and look at any bodhi feedback for those
packages), so it makes perfect sense to inform people there. It's much harder to
search bugzilla to find issues with a particular package.
So, the ability to comment post-push should remain. However, it would make sense to tweak
email notifications somewhat (so that everyone is not spammed with each new comment long
after the package is pushed stable). That's what I assume annoys people.
4 a.m.
On 07/11/2016 10:49 AM, Kamil Paral wrote:
> Any kind of feedback after package gets into updates repository
should
> be denied, I can't think of a case where package moves to updates
> repository and bodhi still allows to add comments/karma. Any feedback
> after that can be either reported in bugzilla or to the package
> maintainers directly. Its not like providing positive or negative
> karma will make any changes to already pushed package to updates
> repository.
I do have valid use cases. We've had examples of important packages having been
pushed stable too quickly and breaking some computers (think kernel, mesa). The bodhi
comments can then be used to provide a description of the problem, suggest any workarounds
or fixes, and also link to more information e.g. in the bugzilla or common bugs wiki, etc.
The Bodhi pages are easy to find (if I perform a system update and the computer breaks, I
look at what was included in the last update, and look at any bodhi feedback for those
packages), so it makes perfect sense to inform people there. It's much harder to
search bugzilla to find issues with a particular package.
Well, IMHO for description of the problem and fixes Bugzilla is the
right place. It's a bug in a stable package then, no testing feedback
anymore.
Greetings,
Christian
4:44 a.m.
On 07/11/2016 10:49 AM, Kamil Paral wrote:
>> Any kind of feedback after package gets into updates repository should
>> be denied, I can't think of a case where package moves to updates
>> repository and bodhi still allows to add comments/karma. Any feedback
>> after that can be either reported in bugzilla or to the package
>> maintainers directly. Its not like providing positive or negative
>> karma will make any changes to already pushed package to updates
>> repository.
>
> I do have valid use cases. We've had examples of important packages having
> been pushed stable too quickly and breaking some computers (think kernel,
> mesa). The bodhi comments can then be used to provide a description of the
> problem, suggest any workarounds or fixes, and also link to more
> information e.g. in the bugzilla or common bugs wiki, etc. The Bodhi pages
> are easy to find (if I perform a system update and the computer breaks, I
> look at what was included in the last update, and look at any bodhi
> feedback for those packages), so it makes perfect sense to inform people
> there. It's much harder to search bugzilla to find issues with a
> particular package.
>
Well, IMHO for description of the problem and fixes Bugzilla is the
right place. It's a bug in a stable package then, no testing feedback
anymore.
Yes, of course, Bugzilla is the right place to deal with the bug. However, Bodhi is a much
better place to inform affected people about it (e.g. link the bug). That's what I was
trying to convey.
6:56 a.m.
Dne 11.7.2016 v 11:00 Christian Dersch napsal(a):
On 07/11/2016 10:49 AM, Kamil Paral wrote:
>> Any kind of feedback after package gets into updates repository should
>> be denied, I can't think of a case where package moves to updates
>> repository and bodhi still allows to add comments/karma. Any feedback
>> after that can be either reported in bugzilla or to the package
>> maintainers directly. Its not like providing positive or negative
>> karma will make any changes to already pushed package to updates
>> repository.
> I do have valid use cases. We've had examples of important packages having been
pushed stable too quickly and breaking some computers (think kernel, mesa). The bodhi
comments can then be used to provide a description of the problem, suggest any workarounds
or fixes, and also link to more information e.g. in the bugzilla or common bugs wiki, etc.
The Bodhi pages are easy to find (if I perform a system update and the computer breaks, I
look at what was included in the last update, and look at any bodhi feedback for those
packages), so it makes perfect sense to inform people there. It's much harder to
search bugzilla to find issues with a particular package.
>
Well, IMHO for description of the problem and fixes Bugzilla is the
right place. It's a bug in a stable package then, no testing feedback
anymore.
But I think it is good feedback loop for testers. I would like to know
about package, which I approved, but later it was discovered, that
something was wrong with it. I might do better job testing next time
(since I probably won't notice new BZ).
Vít
8:17 a.m.
Hi,
On Mon, Jul 11, 2016 at 2:19 PM, Kamil Paral <kparal(a)redhat.com> wrote:
> Any kind of feedback after package gets into updates repository
should
> be denied, I can't think of a case where package moves to updates
> repository and bodhi still allows to add comments/karma. Any feedback
> after that can be either reported in bugzilla or to the package
> maintainers directly. Its not like providing positive or negative
> karma will make any changes to already pushed package to updates
> repository.
I do have valid use cases. We've had examples of important packages having been
pushed stable too quickly and breaking some computers (think kernel, mesa). The bodhi
comments can then be used to provide a description of the problem, suggest any workarounds
or fixes, and also link to more information e.g. in the bugzilla or common bugs wiki, etc.
The Bodhi pages are easy to find (if I perform a system update and the computer breaks, I
look at what was included in the last update, and look at any bodhi feedback for those
packages), so it makes perfect sense to inform people there. It's much harder to
search bugzilla to find issues with a particular package.
In this case its better to disable karma and just let users provide
feedback. Its really annoying to get just +1 karma on your package
when that package is already stable.
>
> So, the ability to comment post-push should remain. However, it would make sense to
tweak email notifications somewhat (so that everyone is not spammed with each new comment
long after the package is pushed stable). That's what I assume annoys people.
> --
> devel mailing list
> devel(a)lists.fedoraproject.org
> https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
11:24 a.m.
On Mon, 2016-07-11 at 06:37 +0000, Raphael Groner wrote:
I got karma on packages that were in stable already for a long time.
The karma system is too tolerant in my eyes.
Why is it possible to give +1 to stable packages? Bodhi may deny or at least impede such
actions, -1 or neutral comments should be still possible though for users having issues.
In case of testing packages, maybe it's a good idea to delay write
access to karma system for some hours after new package got it into
bodhi. We already have a blocking phase for edits from maintainer
there, could it be extended to the karma system for both pending and
testing status?
The 'blocking phase' isn't some kind of by-policy cool-down period,
it's simply a technical thing: updates are locked while any kind of
'push' involving that update is in progress, for strictly technical
reasons (if you edit the update in such a way that it shouldn't be
pushed, while it's being pushed, things can get out of sync).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:49 a.m.
W dniu 10.07.2016 o 18:00, Sayan Chowdhury pisze:
I recently packaged and pushed an update for
fedmsg-meta-fedora-infrastructure to bodhi and exactly 40 secs[1]
later I got a +1 to the update. I am sure that testing a package
surely takes more than 40 secs. This makes me really curious that are
the packages really being tested before giving out the karma.
What about situation when maintainer X scratch built package Y, got it
tested by few people (let name them A, C, E, F) before submitting it to
stable-updates?
Once package enters stable-updates A, C, E, F give +1 to package as it
got built in exactly same environment as one they were testing.
3:59 a.m.
W dniu 10.07.2016 o 18:00, Sayan Chowdhury pisze:
What about situation when maintainer X scratch built package Y, got it
tested by few people (let name them A, C, E, F) before submitting it to
stable-updates?
Once package enters stable-updates A, C, E, F give +1 to package as it
got built in exactly same environment as one they were testing.
Are you talking about secondary architectures? I am not sure, do people build there
packages when they are in still in testing for primary architectures? When packages reach
stable (with a good test result) and people start to build for secondary architectures,
they could still want to give karma in bodhi.
4:07 a.m.
On Mon, 11 Jul 2016 08:59:54 -0000
"Raphael Groner" <raphgro(a)fedoraproject.org> wrote:
> W dniu 10.07.2016 o 18:00, Sayan Chowdhury pisze:
>
>
> What about situation when maintainer X scratch built package Y, got
> it tested by few people (let name them A, C, E, F) before
> submitting it to stable-updates?
>
> Once package enters stable-updates A, C, E, F give +1 to package as
> it got built in exactly same environment as one they were testing.
Are you talking about secondary architectures? I am not sure, do
people build there packages when they are in still in testing for
primary architectures? When packages reach stable (with a good test
result) and people start to build for secondary architectures, they
could still want to give karma in bodhi.
we are following fX-updates-testing tags in primary koji to build
updates as soon as possible on secondary arches
Dan
4:26 a.m.
W dniu 11.07.2016 o 10:59, Raphael Groner pisze:
> W dniu 10.07.2016 o 18:00, Sayan Chowdhury pisze:
>
>
> What about situation when maintainer X scratch built package Y, got
> it tested by few people (let name them A, C, E, F) before
> submitting it to stable-updates?
>
> Once package enters stable-updates A, C, E, F give +1 to package as
> it got built in exactly same environment as one they were testing.
Are you talking about secondary architectures?
No.
10:11 a.m.
On Sun, Jul 10, 2016 at 9:00 AM, Sayan Chowdhury <
sayan.chowdhury2012(a)gmail.com> wrote:
After going through messages in datagrepper[2][3], I found that few people
are
giving out karma in one go (4-5 packages under a minute). If these packages
really are not-tested and the karma are given out randomly then I am sure
that
this sure going to affect the release, infrastructure and our users.
As other people have mentioned, the reason behind 4-5 packages under a
minute is more than
likely fedora-easy-karma - and there is absolutely nothing wrong with that;
it simplifies the process
and allows people to quickly and easily give package feedback.
Regarding the comment regarding karma left 40 seconds after the package was
pushed... it could
be that the person earlier had downloaded the package directly from koji,
because they did not
want to wait DAYS for the package to be pushed to the repository.
I don't believe the karma system was or is intended to be the end all / be
all; but rather an early
warning metric to packagers to tell them if something has gone south.
10:31 a.m.
On 07/11/2016 05:11 PM, Gerald B. Cox wrote:
Regarding the comment regarding karma left 40 seconds after the
package
was pushed... it could
be that the person earlier had downloaded the package directly from
koji, because they did not
want to wait DAYS for the package to be pushed to the repository.
IMHO that's a strong reason against a freeze on giving karma until the
package has been in testing for a certain amount of time. In fact, I
sometimes give +1 on updates that haven't been pushed at all yet. One
example [1]: I reported a bug, I received a notification that a bugfix
is pending, I immediately downloaded the package, tested it and gave my
feedback. Now, if there was a freeze, I would have to come back later
and see if I'm already allowed to give feedback. Most likely, I would
have never given my +1.
Regards,
Till
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2016-d0febaacf2
11:25 a.m.
On Mon, Jul 11, 2016 at 8:31 AM, Till Hofmann <thofmann(a)fedoraproject.org>
wrote:
IMHO that's a strong reason against a freeze on giving karma
until the
package has been in testing for a certain amount of time. In fact, I
sometimes give +1 on updates that haven't been pushed at all yet. One
example [1]: I reported a bug, I received a notification that a bugfix
is pending, I immediately downloaded the package, tested it and gave my
feedback. Now, if there was a freeze, I would have to come back later
and see if I'm already allowed to give feedback. Most likely, I would
have never given my +1.
Yep... the lag many times is insufferable. Another instance... I have been
testing
a package since February which was awaiting package review. It was
approved then went
to the koji/bodhi process 5 days ago... it finally showed up in the testing
repository this morning.
You SHOULD NOT assume that people are waiting for your package to be pushed
before using it.
Again, karma is intended to be the canary in the coal mine to give
packagers some
feedback - it isn't the gold seal that your package has gone through some
rigorous testing
process.
10:12 a.m.
First, have you contacted them? This looks like a misunderstanding between the way you
(and probably most fedora packagers + bodhi developers) think "karma" works and
the way they understand it.
My perspective is being someone who often tests packages but doesn't package them.
From my point of view, the "karma" ("Is the update generally
functional?") field is not quite clear to users of bodhi. I only know because
I've been using it for quite a long time and understand that some packages are
auto-pushed to stable after reaching a specific karma limit.
Testers providing karma in <2 minutes probably just installed the update and if it
1. doesn't break dnf
2. succeeds to run the application
3. (maybe) doesn't crash on 5 seconds of testing
they give it +1.
I suggest to take these actions:
1. Change or extend the phrase "Is the update generally functional?" to have
more details.
2. Link the "Is the update generally functional?" phrase to some more
documentation, e.g. on Fedora Wiki
3. Talk to those testers
4. If this happens often (I can't say, someone needs to do research), have somebody
look at the data (analyze it) and contact testers with "suspicious" testing
behavior.
11:22 a.m.
On Mon, 2016-07-11 at 15:12 +0000, Christian Stadelmann wrote:
First, have you contacted them? This looks like a misunderstanding
between the way you (and probably most fedora packagers + bodhi
developers) think "karma" works and the way they understand it.
My perspective is being someone who often tests packages but doesn't
package them.
From my point of view, the "karma" ("Is the update generally
functional?") field is not quite clear to users of bodhi. I only know
because I've been using it for quite a long time and understand that
some packages are auto-pushed to stable after reaching a specific
karma limit.
Yeah, I think this is a good point. Honestly it's a bit QA's fault,
because there's lots of stuff in the new Bodhi (2) which we asked for
but haven't entirely got around to doing much with yet, like the
various feedback types. For now we've got the feedback types, but in
essence only the bits that were there in Bodhi 1 really 'matter' much,
except to manual review by the update submitter.
It might be good if the Bodhi UI linked to
https://fedoraproject.org/wiki/QA:Update_feedback_guidelines , for one
thing.
Testers providing karma in <2 minutes probably just installed the
update and if it
1. doesn't break dnf
2. succeeds to run the application
3. (maybe) doesn't crash on 5 seconds of testing
they give it +1.
Sometimes this is a perfectly appropriate level of testing, though.
It's quite difficult to write 'perfect' guidelines for this, because
there's a big human experience element to it, knowing how important
different packages are, how significant the changes in a given update
are...
I suggest to take these actions:
1. Change or extend the phrase "Is the update generally functional?" to have
more details.
2. Link the "Is the update generally functional?" phrase to some more
documentation, e.g. on Fedora Wiki
Like the feedback guidelines :P good idea.
3. Talk to those testers
4. If this happens often (I can't say, someone needs to do research),
have somebody look at the data (analyze it) and contact testers with
"suspicious" testing behavior.
We have actually already talked to the relevant people in this case,
and as others have pointed out, there are valid reasons for early
feedback, like downloading updates from Koji.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
11:49 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Sun, 2016-07-10 at 21:30 +0530, Sayan Chowdhury wrote:
Hi,
I recently packaged and pushed an update for fedmsg-meta-fedora-infrastructure
to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am sure that
testing a package surely takes more than 40 secs. This makes me really curious
that are the packages really being tested before giving out the karma.
After going through messages in datagrepper[2][3], I found that few people are
giving out karma in one go (4-5 packages under a minute). If these packages
really are not-tested and the karma are given out randomly then I am sure that
this sure going to affect the release, infrastructure and our users.
Does anybody know what is going on?
So I've been discussing this with various people in the last few days,
and one specific idea has come out of that which I'd like to float.
We've been hesitant to suggest this before as we thought packagers
might not like the idea, but we figured it can't hurt to at least
suggest it.
The idea is this: there could be a requirement for all packages to
provide at least *some* kind of 'how to test' information.
Looked at from the perspective of a new tester, the current system is
quite difficult to handle when it comes to upgrades of packages which
aren't obviously part of the critical path (e.g. kernel) or a well-
known GUI application package (e.g. firefox).
How do we *expect* a new tester to respond when they come to an update
for fedmsg-meta-fedora-infrastructure , in Bodhi or fedora-easy-karma?
It's very difficult - probably impossible - for them to know or work
out what they should actually do to test this package.
Of course, writing instructions for every single package is a lot of
work, but right now we have test cases for very few packages, and it
would definitely help if we could significantly increase that number.
What do people think about this idea?
To be clear, the idea would be to have general-purpose instructions for
basic functionality testing of each package, not requiring new 'how to
test' text to be written for every individual package update,
specifically tailored to the changes in that update.
The way the system works at present is that Bodhi will show all wiki
pages in a specific category based on the package name. For package
'foo', Bodhi's web interface and fedora-easy-karma will list all wiki
pages in the category "Category:Package_foo_test_cases" . These are
expected to be typical 'test case' pages, though there isn't actually
any technical *requirement* for them to be, or enforcement of that.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
noon
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
2016-07-12 10:49 GMT-06:00 Adam Williamson <adamwill(a)fedoraproject.org>:
On Sun, 2016-07-10 at 21:30 +0530, Sayan Chowdhury wrote:
> Hi,
>
> I recently packaged and pushed an update for
fedmsg-meta-fedora-infrastructure
> to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am
sure that
> testing a package surely takes more than 40 secs. This makes me really
curious
> that are the packages really being tested before giving out the karma.
>
> After going through messages in datagrepper[2][3], I found that few
people are
> giving out karma in one go (4-5 packages under a minute). If these
packages
> really are not-tested and the karma are given out randomly then I am
sure that
> this sure going to affect the release, infrastructure and our users.
>
> Does anybody know what is going on?
So I've been discussing this with various people in the last few days,
and one specific idea has come out of that which I'd like to float.
We've been hesitant to suggest this before as we thought packagers
might not like the idea, but we figured it can't hurt to at least
suggest it.
The idea is this: there could be a requirement for all packages to
provide at least *some* kind of 'how to test' information.
Looked at from the perspective of a new tester, the current system is
quite difficult to handle when it comes to upgrades of packages which
aren't obviously part of the critical path (e.g. kernel) or a well-
known GUI application package (e.g. firefox).
How do we *expect* a new tester to respond when they come to an update
for fedmsg-meta-fedora-infrastructure , in Bodhi or fedora-easy-karma?
It's very difficult - probably impossible - for them to know or work
out what they should actually do to test this package.
Of course, writing instructions for every single package is a lot of
work, but right now we have test cases for very few packages, and it
would definitely help if we could significantly increase that number.
Sound like a good idea for a virtual FAD, this way more people can help
What do people think about this idea?
To be clear, the idea would be to have general-purpose instructions for
basic functionality testing of each package, not requiring new 'how to
test' text to be written for every individual package update,
specifically tailored to the changes in that update.
The way the system works at present is that Bodhi will show all wiki
pages in a specific category based on the package name. For package
'foo', Bodhi's web interface and fedora-easy-karma will list all wiki
pages in the category "Category:Package_foo_test_cases" . These are
expected to be typical 'test case' pages, though there isn't actually
any technical *requirement* for them to be, or enforcement of that.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
--
devel mailing list
devel(a)lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
1:27 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 09:49:14AM -0700, Adam Williamson wrote:
To be clear, the idea would be to have general-purpose instructions
for
basic functionality testing of each package, not requiring new 'how to
test' text to be written for every individual package update,
specifically tailored to the changes in that update.
It is a good idea in general, but lack of such instructions is not an
excuse for not performing the one implicit set of test cases in most
updates: verifying fixes to bug reports attached to the update. If a
tester does not know how to do that, then they should not be giving
karma at all, regardless of whether or not there is documentation on
sanity tests for the package.
Siddhesh
1:38 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, 2016-07-12 at 23:57 +0530, Siddhesh Poyarekar wrote:
On Tue, Jul 12, 2016 at 09:49:14AM -0700, Adam Williamson wrote:
>
> To be clear, the idea would be to have general-purpose instructions for
> basic functionality testing of each package, not requiring new 'how to
> test' text to be written for every individual package update,
> specifically tailored to the changes in that update.
It is a good idea in general, but lack of such instructions is not an
excuse for not performing the one implicit set of test cases in most
updates: verifying fixes to bug reports attached to the update. If a
tester does not know how to do that, then they should not be giving
karma at all, regardless of whether or not there is documentation on
sanity tests for the package.
This isn't really correct, because there is no simple relationship
between 'bugs claimed to be fixed actually are fixed' and 'update
should be released'. Both of these are possible:
1) an update which fixes the bugs it claims to fix, but should *NOT* be
released
2) an update which does not fix all the bugs it claims to fix, but
*SHOULD* be released
An example of 1) is an update which claims to fix a minor bug, and
does, but creates a *major* bug. e.g., fixes a typo in the package
description, but causes the app not to run at all. This update should
be given -1 karma (negative response to 'Is the update generally
functional?'), not +1.
An example of 2) is an update which claims to provides a critical
security fix and a trivial bug fix, and *does* fix the security issue,
but the trivial bug fix doesn't work. This update should be given +1
karma (positive response to 'Is the update generally functional?'), not
-1.
This is in fact *why*, in Bodhi 2.0, there are separate feedback
entries for each individual bug listed by the update - so testers can
separate 'does or does not fix bug X' feedback from 'update generally
works' feedback.
It is possible (in my opinion) for a tester to reasonably provide 'Is
the update generally functional?' feedback without actually checking
all or even any of the claimed bug fixes, and I've done this myself
quite often. It can quite often be difficult (if the bug is in a very
complex use case) or impossible (if the bug is specific to e.g. a piece
of hardware you don't own) to check bug fixes.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1:48 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 11:38:01AM -0700, Adam Williamson wrote:
This isn't really correct, because there is no simple
relationship
between 'bugs claimed to be fixed actually are fixed' and 'update
should be released'. Both of these are possible:
1) an update which fixes the bugs it claims to fix, but should *NOT* be
released
2) an update which does not fix all the bugs it claims to fix, but
*SHOULD* be released
An example of 1) is an update which claims to fix a minor bug, and
does, but creates a *major* bug. e.g., fixes a typo in the package
description, but causes the app not to run at all. This update should
be given -1 karma (negative response to 'Is the update generally
functional?'), not +1.
An example of 2) is an update which claims to provides a critical
security fix and a trivial bug fix, and *does* fix the security issue,
but the trivial bug fix doesn't work. This update should be given +1
karma (positive response to 'Is the update generally functional?'), not
-1.
Sure, and I said nothing to contradict that. A good test involves
verification of fixes *and* regression tests, not one or the other.
However using lack of regression tests as an excuse for not verifying
fixes (and more importantly, still leaving positive feedback) is not
acceptable. The points you mention go into intricacies of testing
feedback whereas I am talking about the very basics.
This is in fact *why*, in Bodhi 2.0, there are separate feedback
entries for each individual bug listed by the update - so testers can
separate 'does or does not fix bug X' feedback from 'update generally
works' feedback.
It is possible (in my opinion) for a tester to reasonably provide 'Is
the update generally functional?' feedback without actually checking
all or even any of the claimed bug fixes, and I've done this myself
quite often. It can quite often be difficult (if the bug is in a very
complex use case) or impossible (if the bug is specific to e.g. a piece
of hardware you don't own) to check bug fixes.
Sure, such feedback is very useful, but I don't think it should be
accompanied by a karma +1. The 'update generally functional' should
only have options of karma 0 and -1 IMO.
Siddhesh
2:16 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Wed, 2016-07-13 at 00:18 +0530, Siddhesh Poyarekar wrote:
On Tue, Jul 12, 2016 at 11:38:01AM -0700, Adam Williamson wrote:
>
> This isn't really correct, because there is no simple relationship
> between 'bugs claimed to be fixed actually are fixed' and 'update
> should be released'. Both of these are possible:
>
> 1) an update which fixes the bugs it claims to fix, but should *NOT* be
> released
> 2) an update which does not fix all the bugs it claims to fix, but
> *SHOULD* be released
>
> An example of 1) is an update which claims to fix a minor bug, and
> does, but creates a *major* bug. e.g., fixes a typo in the package
> description, but causes the app not to run at all. This update should
> be given -1 karma (negative response to 'Is the update generally
> functional?'), not +1.
>
> An example of 2) is an update which claims to provides a critical
> security fix and a trivial bug fix, and *does* fix the security issue,
> but the trivial bug fix doesn't work. This update should be given +1
> karma (positive response to 'Is the update generally functional?'), not
> -1.
Sure, and I said nothing to contradict that. A good test involves
verification of fixes *and* regression tests, not one or the other.
However using lack of regression tests as an excuse for not verifying
fixes (and more importantly, still leaving positive feedback) is not
acceptable. The points you mention go into intricacies of testing
feedback whereas I am talking about the very basics.
This is setting far too high a bar for a project like Fedora. We take
the feedback we can get, we are not in a position to demand all update
testers perform comprehensive testing of all possible facets of an
update. It is always important to bear in mind that Bodhi is a system
designed to do the best we can to provide some moderate level of update
quality checking, it has never been intended or expected that Bodhi
feedback is of professional-level quality and comprehensiveness.
>
> This is in fact *why*, in Bodhi 2.0, there are separate feedback
> entries for each individual bug listed by the update - so testers can
> separate 'does or does not fix bug X' feedback from 'update generally
> works' feedback.
>
> It is possible (in my opinion) for a tester to reasonably provide 'Is
> the update generally functional?' feedback without actually checking
> all or even any of the claimed bug fixes, and I've done this myself
> quite often. It can quite often be difficult (if the bug is in a very
> complex use case) or impossible (if the bug is specific to e.g. a piece
> of hardware you don't own) to check bug fixes.
Sure, such feedback is very useful, but I don't think it should be
accompanied by a karma +1. The 'update generally functional' should
only have options of karma 0 and -1 IMO.
I'm sorry, but I still don't agree, and this would clearly be a major
departure from how the current system is actually designed and intended
to be used.
Note that *only* responses to 'generally functional' are actually used
to calculate the karma score. All the other feedback items have no
relevance to scoring or any kind of actual technical gating mechanism,
they are purely informational to the update submitter at this point in
time. If 'generally functional' had no +1, it would be impossible for
any update ever to be auto-pushed or pushed ahead of the 'waiting
period', because no update could ever get a positive karma score.
As a quick refresher, Bodhi 1.x did not have multiple feedback 'types'
(items in the Feedback section in the web UI), it simply let each
tester give a single +1 or -1 'vote', along with comments, for each
update. In Bodhi 2.x, the 'is the update generally functional' feedback
item exactly matches the entirety of what Bodhi 1.x allowed testers to
do; we *added on* the additional feedback items to provide a clear way
for testers to provide feedback on whether bugs were fixed and test
cases satisfied, without the confusion of whether a non-fixed bug
should result in a '-1'.
Bodhi 2 is generally designed to give us the ability to be smarter
about gating - in theory, we could have more complex policies than
using the simple numeric 'score' based on the 'generally functional'
feedback, like never autopushing an update which got a -1 for 'Does the
system's basic functionality continue to work after this update?'.
We can also allow customization at the package level ('all updates for
package foo must get a +1 for test case bar') or update level ('don't
push this update stable unless there's +2 for RHBZ #123456'). There's
actually an initial implementation of this in current Bodhi, but I
haven't seen many packagers using it. In most cases, none of the
feedback items besides 'Does the system's basic functionality continue
to work after this update?' actually does anything beyond communicate
the response to the update submitter.
I have revised https://fedoraproject.org/wiki/QA:Update_feedback_guidelines
again to better cover the Bodhi 2 design and provide specific guidance
on how to provide responses to the different feedback items; the
previous text hadn't really been updated for Bodhi 2 at all.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
9:47 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 12:16:59PM -0700, Adam Williamson wrote:
This is setting far too high a bar for a project like Fedora. We
take
the feedback we can get, we are not in a position to demand all update
testers perform comprehensive testing of all possible facets of an
update. It is always important to bear in mind that Bodhi is a system
designed to do the best we can to provide some moderate level of update
quality checking, it has never been intended or expected that Bodhi
feedback is of professional-level quality and comprehensiveness.
But then we're setting the bar too low by allowing *anyone* to set
karma for the sake of it. You might as well just let developers push
packages to stable if they're 'confident' about it. If volume of
contributions is a concern then I think we need to look at other ways
to promote that without compromising on quality.
If setting the initial bar is too much then we need a way to put in
control mechanisms (and maybe have a mentorship program) when we see
a pattern of incorrect behaviour.
Note that *only* responses to 'generally functional' are
actually used
to calculate the karma score. All the other feedback items have no
relevance to scoring or any kind of actual technical gating mechanism,
they are purely informational to the update submitter at this point in
time. If 'generally functional' had no +1, it would be impossible for
any update ever to be auto-pushed or pushed ahead of the 'waiting
period', because no update could ever get a positive karma score.
... which is not enough. The definition of 'generally functional' is
vague, as all of us agree and we have seen examples of that being
misused in the past. Requiring devel to document their packages is
one step forward and requiring testers to provide more detailed
feedback on what they tested (so that devel can decide if that is
sufficient) should be the step after that.
We can also allow customization at the package level ('all
updates for
package foo must get a +1 for test case bar') or update level ('don't
push this update stable unless there's +2 for RHBZ #123456'). There's
actually an initial implementation of this in current Bodhi, but I
haven't seen many packagers using it. In most cases, none of the
This is what I am looking for.
Siddhesh
11:12 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 7:47 PM, Siddhesh Poyarekar <sid(a)reserved-bit.com>
wrote:
... which is not enough. The definition of 'generally
functional' is
vague, as all of us agree and we have seen examples of that being
misused in the past. Requiring devel to document their packages is
one step forward and requiring testers to provide more detailed
feedback on what they tested (so that devel can decide if that is
sufficient) should be the step after that.
Instead of concentrating on testers, what about the packagers who don't
even test their
applications before throwing them over the wall to bodhi. I've seen
packages that didn't even
get past a simple dnf requisite test because the requires statement in the
spec file was
screwed up. Now we have packagers blaming bodhi because they can't read
the instructions,
or assuming that a tester hasn't adequately tested a package because it was
just released
by bodhi (ignoring the fact that a tester could have gotten that same
package days ago from
koji).
As others have already pointed out bodhi was never intended to be a robust
testing system.
It was intended to give the packager a high level indication of whether or
not there are issues. There is
nothing wrong with "generally functional". At least you're getting some
feedback.
If you're concerned about the quality of the testing, then simply raise the
karma limit, or turn off the auto push.
If you start putting up barriers to people providing feedback you'll soon
find you have none. Which would
you rather have, an idea that many people have installed your package and
haven't noticed any issues or
a nice complicated test plan that no one bothers to run and you are left
with no feedback. If anything, if
there is a test script, that script should be run by the packager BEFORE
they send their application to
bodhi. That would be more helpful.
12:25 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 09:12:25PM -0700, Gerald B. Cox wrote:
Instead of concentrating on testers, what about the packagers who
don't
even test their
applications before throwing them over the wall to bodhi. I've seen
packages that didn't even
get past a simple dnf requisite test because the requires statement in the
spec file was
screwed up. Now we have packagers blaming bodhi because they can't read
the instructions,
or assuming that a tester hasn't adequately tested a package because it was
just released
by bodhi (ignoring the fact that a tester could have gotten that same
package days ago from
koji).
As others have already pointed out bodhi was never intended to be a robust
testing system.
It was intended to give the packager a high level indication of whether or
not there are issues. There is
nothing wrong with "generally functional". At least you're getting some
feedback.
If you're concerned about the quality of the testing, then simply raise the
karma limit, or turn off the auto push.
I came back to clarify after thinking about it a bit more, but you
beat me to it :)
Devel and testing are not two sides and I realize that my initial tone
seems to indicate that, so I take that back. I fully accept the idea
that developers need to document ways to test their application.
However at the same time, there is currently a reasonably high barrier
to becoming a packager for Fedora and it might not be a bad idea to
have something similar for QE.
An individual wanting to get started in Fedora packaging has to prove
their competence and understanding of the packaging guidelines by
commenting on package review requests in addition to submitting their
own package for review. There is a side door to this in the form of
co-maintainership but that too is gated by a Fedora contributor.
What I am looking for is a mechanism similar to that for testing.
Maybe one way to do this would be to allow everyone to comment but
restrict karma rights only to individuals who are in a specific group
and then have that group gated by a testing mentor. This in itself
will give plenty of opportunity for mentoring.
If you start putting up barriers to people providing feedback
you'll soon
find you have none. Which would
you rather have, an idea that many people have installed your package and
haven't noticed any issues or
a nice complicated test plan that no one bothers to run and you are left
with no feedback. If anything, if
there is a test script, that script should be run by the packager BEFORE
they send their application to
bodhi. That would be more helpful.
Barriers to entry for any engineering task should be sufficiently high
that it does not grossly compromise on quality. Developers not using
the automatic testing framework is obviously their fault, but that is
a conversation for a different thread.
Siddhesh
12:34 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Wed, 2016-07-13 at 10:55 +0530, Siddhesh Poyarekar wrote:
An individual wanting to get started in Fedora packaging has to
prove
their competence and understanding of the packaging guidelines by
commenting on package review requests in addition to submitting their
own package for review. There is a side door to this in the form of
co-maintainership but that too is gated by a Fedora contributor.
What I am looking for is a mechanism similar to that for testing.
Maybe one way to do this would be to allow everyone to comment but
restrict karma rights only to individuals who are in a specific group
and then have that group gated by a testing mentor. This in itself
will give plenty of opportunity for mentoring.
We have a high bar for packagers because packagers have an awful lot of
power to cause problems both intentionally and unintentionally. People
filing update feedback don't: all they can do is influence, to a
degree, whether updates submitted by packagers go out or not. Bodhi is,
as Gerald said, a rough sanity check mechanism, it is not intended and
never *has* been intended to be a perfect gating mechanism at which you
can throw anything and be sure you'll get the right result. Packagers
still ultimately are held responsible for the quality of their updates.
We (QA) are certainly happy to look into any possible karma errors, as
we have done in this case, quite extensively, and made changes to the
feedback guidelines as a result. But I don't think it's really viable
or appropriate to construct a wall as high as the one for packagers
just to be allowed to leave karma.
Note we had a 'proventesters' mechanism for a while which attempted to
give feedback from 'trusted' testers somewhat more significance than
feedback from any registered user, but it didn't work out entirely
great and got mothballed...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:26 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, 12 Jul 2016 09:49:14 -0700
Adam Williamson <adamwill(a)fedoraproject.org> wrote:
So I've been discussing this with various people in the last few
days,
and one specific idea has come out of that which I'd like to float.
We've been hesitant to suggest this before as we thought packagers
might not like the idea, but we figured it can't hurt to at least
suggest it.
The idea is this: there could be a requirement for all packages to
provide at least *some* kind of 'how to test' information.
Looked at from the perspective of a new tester, the current system is
quite difficult to handle when it comes to upgrades of packages which
aren't obviously part of the critical path (e.g. kernel) or a well-
known GUI application package (e.g. firefox).
How do we *expect* a new tester to respond when they come to an update
for fedmsg-meta-fedora-infrastructure , in Bodhi or fedora-easy-karma?
It's very difficult - probably impossible - for them to know or work
out what they should actually do to test this package.
Of course, writing instructions for every single package is a lot of
work, but right now we have test cases for very few packages, and it
would definitely help if we could significantly increase that number.
What do people think about this idea?
Well, I'm not sure. How would the requirement be enforced?
Packages without such test cases couldn't get any +1s?
I think thats probably a bit harsh. We could try and encourage people
to add them however, perhaps series of badges for writing test case
pages? Or a note/warning on the bodhi page or fedora-easy-karma noting
that there's no tests?
Some packages are definitely going to be harder than others...
kevin
5:45 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, 2016-07-12 at 23:32 +0100, Jonathan Wakely wrote:
On 12/07/16 16:26 -0600, Kevin Fenzi wrote:
>
> Some packages are definitely going to be harder than others...
For example, a libfoo, and its libfoo-devel subpackage.
Bodhi works at the source package level, not binary package level.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
9:51 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 03:45:54PM -0700, Adam Williamson wrote:
Bodhi works at the source package level, not binary package level.
I think Jon's point was with respect to the scope of testing. With
glibc (or libstdc++ that Jon would be concerned with), an ideal set of
sanity tests would cover the library as well as its development files*.
It will be a lot of work for each tester though. We had talked about
using tunir or taskatron to automate some of it last year, but I don't
think there has been any progress on that front.
Siddhesh
* Well with glibc it would be library, devel, locales, networking and
boot, to begin to name a few things.
12:26 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Wed, 2016-07-13 at 08:21 +0530, Siddhesh Poyarekar wrote:
On Tue, Jul 12, 2016 at 03:45:54PM -0700, Adam Williamson wrote:
>
> Bodhi works at the source package level, not binary package level.
I think Jon's point was with respect to the scope of testing. With
glibc (or libstdc++ that Jon would be concerned with), an ideal set of
sanity tests would cover the library as well as its development files*.
It will be a lot of work for each tester though. We had talked
about
using tunir or taskatron to automate some of it last year, but I don't
think there has been any progress on that front.
Siddhesh
* Well with glibc it would be library, devel, locales, networking and
boot, to begin to name a few things.
It would not be 'a lot of work', it would be a gigantic, totally
unsustainable burden. I honestly think you're shooting *way* too high
here. Even with all the recent volunteers, we have like a couple dozen
people who volunteer to spend *some* time testing updates. There is no
way we are going to be running rigorous full functionality tests on
every update, and honestly, it's not even reasonable to *expect* that,
because it should be done far earlier in the process than 'distribution
package updates'. What we can realistically do for distribution package
updates is basic functionality tests and (if feasible) bug
verification.
We have, what, ~10k packages in the distro. We have at least a few
hundred just on the *critical path*. Some of those are things like, oh
say, *entire desktop environments*, which you'd be looking at a week of
solid full time work to test *fully*. We have database stacks, whole
things like fedmsg that you need a solid couple of days' reading to set
up a dev environment of, stuff like FreeIPA which RH has entire teams
dedicated to testing...
we are not going to be doing comprehensive testing of all of this
stuff, either manually or automated, at the distribution level. It's
not feasible and it's not even *sensible*, because it'd essentially be
a huge project of doing work in the wrong place. Comprehensive
functionality testing of glibc does not belong in a distribution, it
belongs in the glibc project. What makes sense to test at the
distribution level is 'does this glibc, which we trust upstream to
assert is at least vaguely functional in and of itself, have any
unfortunate interactions with the rest of Fedora?', which boils down
to, get a few people to install it and run with it for a day or two and
see if anything looks wonky. That's the level of testing we
realistically have the resources for - either doing manually, or
writing automated tests for - at the distribution level.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:38 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 10:26:20PM -0700, Adam Williamson wrote:
It would not be 'a lot of work', it would be a gigantic,
totally
unsustainable burden. I honestly think you're shooting *way* too high
here. Even with all the recent volunteers, we have like a couple dozen
I agree it is a massive task, which is why it hasn't gotten off the
ground for glibc over the last year. However I remain optimistic that
someone someday will do at least a fraction of the automation :)
Siddhesh
1:52 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Wed, 2016-07-13 at 11:08 +0530, Siddhesh Poyarekar wrote:
On Tue, Jul 12, 2016 at 10:26:20PM -0700, Adam Williamson wrote:
>
> It would not be 'a lot of work', it would be a gigantic, totally
> unsustainable burden. I honestly think you're shooting *way* too high
> here. Even with all the recent volunteers, we have like a couple dozen
I agree it is a massive task, which is why it hasn't gotten off the
ground for glibc over the last year. However I remain optimistic that
someone someday will do at least a fraction of the automation :)
FWIW, as someone who is working on this, I don't think we can
realistically aim to do distribution-level automated testing with per-
package granularity. We actually have all the bits in place to do
something like that if we wanted to - I could have some kind of PoC
using existing openQA tests in a week or so - but I just kinda don't
think it's the way to go.
I think a more profitable angle at the distribution level is to define
what it is we actually think a distribution should do, and test whether
updates change *those* things. That's an appropriate and manageable
level of automated testing that we can actually achieve.
It doesn't necessarily plug into the current Bodhi design precisely,
but that's not a particularly good reason to pick one approach over
another.
Of course, we don't *have* to pick one thing or the other necessarily;
we can certainly provide all the appropriate hooks for packages to do
automated update testing, this is something folks are already looking
at, and there's no reason to stand in the way of maintainers / teams
who want to implement package-level automated tests for distribution
updates.
But from the perspective of the Fedora QA team, I don't think the best
thing we could do with our time is, you know, draw up a big list of
packages and start working down it, writing automated tests for one
package at a time. If we started doing that we might make some sort of
vaguely noticeable dent by, oh, say, 2026 or so. ;)
Personally I think it's more useful to do, well, the kinds of things
we're doing: *generic* package tests like the ones taskotron runs at
present (there are more we could do there), and automating the
requirements that are already encoded in the release criteria, which is
what we've been doing with openQA. If we can complete that task - or
get further along with it - so we can run a subset of openQA tests for
every update and say 'OK, well installing this update doesn't break the
high-level functions we define as "what the distribution should do"' -
I think that's interesting.
Just my thoughts, though.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:01 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, 2016-07-12 at 23:52 -0700, Adam Williamson wrote:
Of course, we don't *have* to pick one thing or the other necessarily;
we can certainly provide all the appropriate hooks for packages to do
automated update testing, this is something folks are already looking
at, and there's no reason to stand in the way of maintainers / teams
who want to implement package-level automated tests for distribution
updates.
Just to follow on from that a bit, I guess it's worth noting that the
current Bodhi system just isn't a great fit for some packages; it's a
bit of a one-size-fits-all model, but it fits certain packages much
better than others. Bodhi's *existence* is a result of several cases
where updates got pushed out that badly broke basic system
functionality for a large number of users, and we got terrible press;
thus it's kinda designed around avoiding *that*. If you look at the
design of Bodhi 1 especially you can see how that's the problem it was
trying to solve; Bodhi 2 is kinda an attempt to solve some limitations
we rubbed up against in Bodhi 1 use, but it's still fundamentally an
elaboration on the same basic design.
So Bodhi works best for updates to familiar tools and applications, and
things that are involved in basic system functionality for lots of
testers. You can really see the value of Bodhi best when someone, say,
pushes an update that breaks boot on encrypted systems (or something
like that) and the update gets a flood of negative karma and doesn't go
out. That's Bodhi doing exactly what it was designed to do, and that's
what it's best at. It's almost equally good when there's, say, a
trivial update to gedit, and it gets a prompt flood of positive karma,
so the update goes out with almost no delay.
But Bodhi kinda struggles with, well, the kind of update that started
this whole discussion. fedmsg-meta-fedora-infrastructure is an
extremely specialized package with an extremely specialized function
that, realistically, Joe Q. Random Tester is not going to be able to
test effectively. In an ideal world, Bodhi just isn't a very good fit
for it...but Bodhi is all we have, so we make the best of it. I don't
actually know that there's any sensible instruction we can give rookie
testers for giving feedback on a fedmsg-meta-fedora-infrastructure
update besides 'just leave it alone and move onto something easier'.
So the system's definitely sub-optimal in this case, and it would be
great to have something better. If we could provide the necessary hooks
for automated testing this might actually be a good case for it, if the
fedmsg maintainers were willing to develop some kind of automated test
that deployed a fedmsg instance and ran some basic checks on it, which
we could hook up to appropriate package updates...the biggest objection
to more heavy automation of package update testing is that automation
can miss problems a human would catch, but I think a very specialized
package like this is actually a good example of a case where automation
is more likely to do a good job than Random Human Testing.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:05 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, Jul 12, 2016 at 11:52:45PM -0700, Adam Williamson wrote:
FWIW, as someone who is working on this, I don't think we can
realistically aim to do distribution-level automated testing with per-
package granularity. We actually have all the bits in place to do
something like that if we wanted to - I could have some kind of PoC
using existing openQA tests in a week or so - but I just kinda don't
think it's the way to go.
I think a more profitable angle at the distribution level is to define
what it is we actually think a distribution should do, and test whether
updates change *those* things. That's an appropriate and manageable
level of automated testing that we can actually achieve.
Agreed, the automated tests are an ideal that we have set for
ourselves as the glibc team and it doesn't actually plug into the
distribution's goals for QE. The only real relationship with the
distribution is that we want to leverage the infrastructure to do our
automated tests so that we don't have to set up our own on sourceware
or wherever else.
But from the perspective of the Fedora QA team, I don't think the
best
thing we could do with our time is, you know, draw up a big list of
packages and start working down it, writing automated tests for one
package at a time. If we started doing that we might make some sort of
vaguely noticeable dent by, oh, say, 2026 or so. ;)
Heh, no, the 'someone' I talked about would be one of us glibc
maintainers or maybe some poor soul who thinks it would be a great
project to dive into, not Fedora QE.
We have been a bit lazy about it so far because (in addition to the
fact that we don't have volunteers yet) a lot of the distribution
level tests are reasonable sanity checks for glibc.
Siddhesh
12:23 p.m.
New subject: Requiring package test instructions
On 07/13/2016 01:38 AM, Siddhesh Poyarekar wrote:
On Tue, Jul 12, 2016 at 10:26:20PM -0700, Adam Williamson wrote:
> It would not be 'a lot of work', it would be a gigantic, totally
> unsustainable burden. I honestly think you're shooting *way* too high
> here. Even with all the recent volunteers, we have like a couple dozen
I agree it is a massive task, which is why it hasn't gotten off the
ground for glibc over the last year. However I remain optimistic that
someone someday will do at least a fraction of the automation :)
There are many
levels of functionality before you get to the coverage
you're talking about.
I have bugzilla cases against distributed packages that
* failed to correctly update during routine system updates (last one
yesterday, 1355916)
* failed to start (crashed before getting to first prompt/window)
* started the application but crashed on a simplest function
I think the functionality you're talking about (checking correctness of
bug fixes, etc) should be left to the original bug reporters. After all,
they raised the issue so they are invested in the result.
Automation of those checks is tricky: bugzilla does ask for 'steps to
reproduce' and actual vs. expected results but we'd have to really come
up with a better technology to translate that into automated test cases.
It's something that looks desirable and even doable, but not quite easy.
12:14 a.m.
New subject: Requiring package test instructions
On Wed, Jul 13, 2016 at 01:23:04PM -0400, Przemek Klosowski wrote:
I think the functionality you're talking about (checking
correctness of bug
fixes, etc) should be left to the original bug reporters. After all, they
raised the issue so they are invested in the result.
Agreed.
Automation of those checks is tricky: bugzilla does ask for
'steps to
reproduce' and actual vs. expected results but we'd have to really come up
with a better technology to translate that into automated test cases. It's
something that looks desirable and even doable, but not quite easy.
Oh no, that is not what we're aiming for. We are aiming for a set of
automated *sanity tests* that run for every update. Over time maybe
we could add regression tests to it that can't be added to the
upstream testsuite for some reason (requiring root privs for example,
much easier to do in a vm), but for now sanity tests will have to do.
Siddhesh
3 a.m.
New subject: Requiring package test instructions
Oh no, that is not what we're aiming for. We are aiming for a
set of
automated *sanity tests* that run for every update. Over time maybe
we could add regression tests to it that can't be added to the
upstream testsuite for some reason (requiring root privs for example,
much easier to do in a vm), but for now sanity tests will have to do.
Siddhesh
We're not yet at the point where we can offer dist-git style checks for every package
maintainer (it's our goal, though), but if you have any useful generic sanity check
which can be run on any package after it is built in Koji, we can most probably start
executing it right away. We already run rpmlint, depcheck, upgradepath, abicheck and
soon-to-be-deployed rpmgrill.
Feel free to ping us on #fedora-qa or discuss at qa-devel mailing list to talk about
details.
Thanks,
Kamil
4:21 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On 13/07/16 08:21 +0530, Siddhesh Poyarekar wrote:
On Tue, Jul 12, 2016 at 03:45:54PM -0700, Adam Williamson wrote:
> Bodhi works at the source package level, not binary package level.
That's irrelevant. If a source package only provides a library for
other packages to link against then testing it is non-trivial.
Something like libjpeg-turbo is easily testabel because it includes
some command-line utilities that can be tested (e.g. run djpeg to
decompress a JPEG file). Not all packages that install libraries
include such utilities.
I think Jon's point was with respect to the scope of testing.
With
glibc (or libstdc++ that Jon would be concerned with), an ideal set of
sanity tests would cover the library as well as its development files*.
Right, or Boost. Boost has no applications you can test, it only
installs libraries and headers. 31 shared libraries, from 25 almost
entirely unrelated sub-libraries. And several times that number of
header-only sub-libraries.
To test that properly you'd need to compile several dozen applications
that use different pieces of Boost and check they link and run OK.
That was my point. Testing a package that only provides a library for
other packages to build against is non-trivial.
10:23 a.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Wed, 2016-07-13 at 10:21 +0100, Jonathan Wakely wrote:
On 13/07/16 08:21 +0530, Siddhesh Poyarekar wrote:
>
> On Tue, Jul 12, 2016 at 03:45:54PM -0700, Adam Williamson wrote:
> >
> > Bodhi works at the source package level, not binary package level.
That's irrelevant. If a source package only provides a library for
other packages to link against then testing it is non-trivial.
Something like libjpeg-turbo is easily testabel because it includes
some command-line utilities that can be tested (e.g. run djpeg to
decompress a JPEG file). Not all packages that install libraries
include such utilities.
>
> I think Jon's point was with respect to the scope of testing. With
> glibc (or libstdc++ that Jon would be concerned with), an ideal set of
> sanity tests would cover the library as well as its development files*.
Right, or Boost. Boost has no applications you can test, it only
installs libraries and headers. 31 shared libraries, from 25 almost
entirely unrelated sub-libraries. And several times that number of
header-only sub-libraries.
To test that properly you'd need to compile several dozen applications
that use different pieces of Boost and check they link and run OK.
That was my point. Testing a package that only provides a library for
other packages to build against is non-trivial.
Ah, I see what you meant. What I usually suggest for library testing is
to run whatever applications you have installed that use that library,
and make sure they work for what you usually do with them.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
5:45 p.m.
New subject: Requiring package test instructions (was: Re: Too fast karma on
Bodhi updates)
On Tue, 2016-07-12 at 16:26 -0600, Kevin Fenzi wrote:
On Tue, 12 Jul 2016 09:49:14 -0700
Adam Williamson <adamwill(a)fedoraproject.org> wrote:
>
> So I've been discussing this with various people in the last few days,
> and one specific idea has come out of that which I'd like to float.
>
> We've been hesitant to suggest this before as we thought packagers
> might not like the idea, but we figured it can't hurt to at least
> suggest it.
>
> The idea is this: there could be a requirement for all packages to
> provide at least *some* kind of 'how to test' information.
>
> Looked at from the perspective of a new tester, the current system is
> quite difficult to handle when it comes to upgrades of packages which
> aren't obviously part of the critical path (e.g. kernel) or a well-
> known GUI application package (e.g. firefox).
>
> How do we *expect* a new tester to respond when they come to an update
> for fedmsg-meta-fedora-infrastructure , in Bodhi or fedora-easy-karma?
> It's very difficult - probably impossible - for them to know or work
> out what they should actually do to test this package.
>
> Of course, writing instructions for every single package is a lot of
> work, but right now we have test cases for very few packages, and it
> would definitely help if we could significantly increase that number.
>
> What do people think about this idea?
Well, I'm not sure. How would the requirement be enforced?
For new packages, it could be a requirement at the package review stage.
Packages without such test cases couldn't get any +1s?
For maximum lulz, for existing packages, how about you can't submit an
update till you provide some test instructions? ;)
(/me buys shares in pitchforks and flaming torches)
There's various levels of 'enforcement' we could have, for now I didn't
want to get too far into the details and just discuss the idea.
I think thats probably a bit harsh. We could try and encourage
people
to add them however, perhaps series of badges for writing test case
pages? Or a note/warning on the bodhi page or fedora-easy-karma noting
that there's no tests?
Some packages are definitely going to be harder than others...
kevin
--
devel mailing list
devel(a)lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
4:41 a.m.
New subject: Requiring package test instructions
Dne 12.7.2016 v 18:49 Adam Williamson napsal(a):
The idea is this: there could be a requirement for all packages to
provide at least *some* kind of 'how to test' information.
If the package should be tested by human, I'd expect that there will be
some additional value, i.e. the human can search for the "how to test"
information by him self on the internet, the human has some interest and
knowledge about the package, try something I was not thinking about when
writing the "how to test" script etc. Following some "how to test"
guidelines makes not much sense to me, since I do these tests by myself.
If I should provide "how to test" information, it should be probably
script which should be run by AutoQA or something. But TBH, I have no
idea how to provide such script. There is precisely zero information
about this stuff on places I would expect (this one [1] as an example).
There is also zero support in tools such as fedpkg and dist-git.
I am pretty sure this would be more valuable then some "how to test"
document.
Vít
[1] https://fedoraproject.org/wiki/Join_the_package_collection_maintainers
11:44 a.m.
New subject: Requiring package test instructions
On Wed, 2016-07-13 at 11:41 +0200, Vít Ondruch wrote:
Dne 12.7.2016 v 18:49 Adam Williamson napsal(a):
>
>
> The idea is this: there could be a requirement for all packages to
> provide at least *some* kind of 'how to test' information.
If the package should be tested by human, I'd expect that there will be
some additional value, i.e. the human can search for the "how to test"
information by him self on the internet, the human has some interest and
knowledge about the package, try something I was not thinking about when
writing the "how to test" script etc. Following some "how to test"
guidelines makes not much sense to me, since I do these tests by myself.
People filing karma don't necessarily have 'interest and knowledge
about the package'; fedora-easy-karma lists *every* package you
currently have installed from updates-testing, which may include ones
that were installed by default which the tester doesn't normally use,
dependencies and so on.
Not all packagers actually test their packages before submitting them.
Most packagers probably at least run the code somehow, but often they
use their development checkout rather than the actual package they
produced, for instance, or they only verify the fix they're sending out
rather than re-testing the entire thing. This is trivially demonstrable
by the fact that we *do* still sometimes get completely broken updates
showing up.
Even if you *do* test your package properly, one thing Bodhi testing
provides is broader testing, in several ways. Humans almost always find
ways to do things differently; even if you provide very specific
testing instructions you'll probably find there's *some* variance in
exactly how people carry them out, and a tester may well find an issue
you didn't just because they happen to perform the test a slightly
different way. There's also testing environments. You probably test
your package on your system, in the desktop environment you usually
run. But what if it breaks on a different arch? Or you run GNOME, and
your update doesn't work in KDE for some reason? Or it breaks when some
config file setting is set the opposite way to the way you have it set?
And so on and so on - the possibilities are endless. But Bodhi testing
can at least help to get things checked in more configurations than
just yours.
If I should provide "how to test" information, it should be
probably
script which should be run by AutoQA or something. But TBH, I have no
idea how to provide such script. There is precisely zero information
about this stuff on places I would expect (this one [1] as an example).
There is also zero support in tools such as fedpkg and dist-git.
If you're willing to do this, that's great, and we would definitely
like to enable it. I know the Taskotron (it hasn't been AutoQA for a
while) team is working on enabling generic dist-git integration (which
is basically what you're asking for), but I don't know the current
status of that...CCing tflink and kparal who may have more information.
I am pretty sure this would be more valuable then some "how to
test"
document.
I think it varies substantially by package, but I don't think we're
realistically going to get sufficient automated testing for all
packages, including ones that are important to users.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
9:35 a.m.
New subject: Requiring package test instructions
On Wed, Jul 13, 2016 at 09:44:36AM -0700, Adam Williamson wrote:
On Wed, 2016-07-13 at 11:41 +0200, Vít Ondruch wrote:
> If I should provide "how to test" information, it should be probably
> script which should be run by AutoQA or something. But TBH, I have no
> idea how to provide such script. There is precisely zero information
> about this stuff on places I would expect (this one [1] as an example).
> There is also zero support in tools such as fedpkg and dist-git.
If you're willing to do this, that's great, and we would definitely
like to enable it. I know the Taskotron (it hasn't been AutoQA for a
while) team is working on enabling generic dist-git integration (which
is basically what you're asking for), but I don't know the current
status of that...CCing tflink and kparal who may have more information.
> I am pretty sure this would be more valuable then some "how to test"
> document.
I think it varies substantially by package, but I don't think we're
realistically going to get sufficient automated testing for all
packages, including ones that are important to users.
I don't think koschei has been mentioned in this thread yet.
For library type packages that's the testing we want to do, and if
I'm not mistaken, koschei compilations already are started for dependent
packages when an rpm hits updates-testing, if enabled for those packages.
This kind of testing complements the usability testing that is
suitable for leaf packages.
So adding testing instructions might help for some types of packages,
for other types we should be working on integrating koschei more
closely, in particular flagging the update that breaks builds.
Zbyszek
10:44 a.m.
New subject: Requiring package test instructions
On Sat, 2016-07-16 at 14:35 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 13, 2016 at 09:44:36AM -0700, Adam Williamson wrote:
>
> On Wed, 2016-07-13 at 11:41 +0200, Vít Ondruch wrote:
> >
> > If I should provide "how to test" information, it should be probably
> > script which should be run by AutoQA or something. But TBH, I have no
> > idea how to provide such script. There is precisely zero information
> > about this stuff on places I would expect (this one [1] as an example).
> > There is also zero support in tools such as fedpkg and dist-git.
>
> If you're willing to do this, that's great, and we would definitely
> like to enable it. I know the Taskotron (it hasn't been AutoQA for a
> while) team is working on enabling generic dist-git integration (which
> is basically what you're asking for), but I don't know the current
> status of that...CCing tflink and kparal who may have more information.
>
> >
> > I am pretty sure this would be more valuable then some "how to test"
> > document.
>
> I think it varies substantially by package, but I don't think we're
> realistically going to get sufficient automated testing for all
> packages, including ones that are important to users.
I don't think koschei has been mentioned in this thread yet.
For library type packages that's the testing we want to do, and if
I'm not mistaken, koschei compilations already are started for dependent
packages when an rpm hits updates-testing, if enabled for those packages.
This kind of testing complements the usability testing that is
suitable for leaf packages.
So adding testing instructions might help for some types of packages,
for other types we should be working on integrating koschei more
closely, in particular flagging the update that breaks builds.
Well, there can be functional aspects to library testing too; take GTK+
as an interesting case, you can have a GTK+ update which passes all its
internal tests and all or most packages rebuild fine against it, but it
changes some bit of CSS or something and makes text go white in
Firefox, or something like that. It's happened before. ;)
But yeah, there are definitely cases - as I said - where the style of
human testing that Bodhi is kinda designed around doesn't really fit,
and we could definitely improve things.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3:41 p.m.
Sayan Chowdhury wrote:
I recently packaged and pushed an update for
fedmsg-meta-fedora-infrastructure to bodhi and exactly 40 secs[1] later I
got a +1 to the update. I am sure that testing a package surely takes more
than 40 secs. This makes me really curious that are the packages really
being tested before giving out the karma.
After going through messages in datagrepper[2][3], I found that few people
are giving out karma in one go (4-5 packages under a minute). If these
packages really are not-tested and the karma are given out randomly then I
am sure that this sure going to affect the release, infrastructure and our
users.
Does anybody know what is going on?
The broken system where giving karma:
* is the only way to allow the maintainer to push the package before some
arbitrary deadline,
* in many cases (lazy maintainers enabling that "autokarma" nonsense) even
AUTOMATICALLY gets things out to stable,
* in addition, giving karma automatically (without any validation) gives
some of that "badges" nonsense (which counterproductively rewards a lot of
bad behavior including broken builds!) as a reward,
gives the perverse incentive to game the system with fake karma.
Fedora really needs to move back to a system in which the decision to push
an update is purely made by a human known to be competent, i.e., the
package's maintainer. Any feedback on Bodhi should be purely informative (to
the maintainer, who then does the actual call). That is the only way to stop
encouraging counterproductive behavior.
Kevin Kofler
2864
days inactive
2870
days old
53 comments
21 participants
participants (21)
-
Adam Williamson -
Christian Dersch -
Christian Stadelmann -
Dan Horák -
Dave Love -
Gerald B. Cox -
Jonathan Wakely -
Kamil Paral -
Kevin Fenzi -
Kevin Kofler -
Marcin Juszkiewicz -
Parag Nemade -
Przemek Klosowski -
Raphael Groner -
Sayan Chowdhury -
Siddhesh Poyarekar -
Till Hofmann -
Ville Skyttä -
Vít Ondruch -
William Moreno -
Zbigniew Jędrzejewski-Szmek