8:22 a.m.
Hi guys,
(note: devel, packaging and test lists) previously I've done a little experiment
and counted how many packages are likely to have upstream test suites and how
many don't:
http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
In general around 35% do have test suites, the rest don't.
My goal is to bring down the number of packages which ship without any sort of
test suite inside their code base.
The first step is to identify them and track them in Bugzilla.
My question is:
**Is everyone, especially package maintainers OK with me filing 1000+ bugs ?**
Last time I did so (around 100 bugs) it got a few people unhappy so better ask
this time!
If you are unhappy seeing such many bugs and having your mailbox full with
notifications from Bugzilla please reply with a better proposal and why do you
consider it better.
Thanks,
Alex
8:27 a.m.
On Fri, Feb 21, 2014 at 04:22:42PM +0200, Alexander Todorov wrote:
Hi guys,
(note: devel, packaging and test lists) previously I've done a
little experiment and counted how many packages are likely to have
upstream test suites and how many don't:
http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
In general around 35% do have test suites, the rest don't.
[...]
My question is:
**Is everyone, especially package maintainers OK with me filing 1000+ bugs ?**
Last time I did so (around 100 bugs) it got a few people unhappy so
better ask this time!
Is it correct that you're only going to be filing bugs when upstream
tarballs already contain test suites, but they are just not enabled in
the Fedora package? If so, it seems reasonable to file a bug
encouraging the packager to enable the existing test suite.
My concern is that the packager is not the right person to create a
new test suite for a package.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
8:51 a.m.
На 21.02.2014 16:27, Richard W.M. Jones написа:
Is it correct that you're only going to be filing bugs when
upstream
tarballs already contain test suites, but they are just not enabled in
the Fedora package?
Hi Richard,
I meant just the opposite. However I will also do what you suggest but this will
result in far less number of bugs (probably around 100).
I want to track which packages *DO NOT* have any tests and later be able to
focus on creating them (be it working with volunteers, GSoC participants or
whoever is willing to step up to this task).
I don't intend to force package maintainers to write tests if they don't want to
(or can't).
--
Alex
8:53 a.m.
On 21/02/14 14:51, Alexander Todorov wrote:
I want to track which packages *DO NOT* have any tests and later be
able
to focus on creating them (be it working with volunteers, GSoC
participants or whoever is willing to step up to this task).
Why would you file a bug in the Fedora bug tracker when the package has
no test suite upstream? That makes no sense - if the upstream package
has no tests then the bug belongs upstream not in Fedora.
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
8:57 a.m.
On Fri, Feb 21, 2014 at 02:53:55PM +0000, Tom Hughes wrote:
On 21/02/14 14:51, Alexander Todorov wrote:
>I want to track which packages *DO NOT* have any tests and later be able
>to focus on creating them (be it working with volunteers, GSoC
>participants or whoever is willing to step up to this task).
Why would you file a bug in the Fedora bug tracker when the package
has no test suite upstream? That makes no sense - if the upstream
package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning
filing bugs where upstream *does* have a test suite, but the RPM
specfile has not enabled it in %check.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
8:58 a.m.
On 21/02/14 14:57, Daniel P. Berrange wrote:
On Fri, Feb 21, 2014 at 02:53:55PM +0000, Tom Hughes wrote:
> On 21/02/14 14:51, Alexander Todorov wrote:
>
>> I want to track which packages *DO NOT* have any tests and later be able
>> to focus on creating them (be it working with volunteers, GSoC
>> participants or whoever is willing to step up to this task).
>
> Why would you file a bug in the Fedora bug tracker when the package
> has no test suite upstream? That makes no sense - if the upstream
> package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning
filing bugs where upstream *does* have a test suite, but the RPM
specfile has not enabled it in %check.
That was what I thought he really meant originally, and that was what
the wiki page suggested, but it doesn't seem to be what he is saying in
the message I quoted.
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
9:08 a.m.
На 21.02.2014 16:58, Tom Hughes написа:
On 21/02/14 14:57, Daniel P. Berrange wrote:
> On Fri, Feb 21, 2014 at 02:53:55PM +0000, Tom Hughes wrote:
>> On 21/02/14 14:51, Alexander Todorov wrote:
>>
>>> I want to track which packages *DO NOT* have any tests and later be able
>>> to focus on creating them (be it working with volunteers, GSoC
>>> participants or whoever is willing to step up to this task).
>>
>> Why would you file a bug in the Fedora bug tracker when the package
>> has no test suite upstream? That makes no sense - if the upstream
>> package has no tests then the bug belongs upstream not in Fedora.
>
> I don't believe that is what he is proposing. I read it as meaning
> filing bugs where upstream *does* have a test suite, but the RPM
> specfile has not enabled it in %check.
That was what I thought he really meant originally, and that was what the wiki
page suggested, but it doesn't seem to be what he is saying in the message I
quoted.
Guys I can do both.
1) Report packages which *have* test suites but they are *not* executed in %check
2) Report packages which *don't* have any test suites at all.
1) is easy but I'm more interested in 2)
--
Alex
9:16 a.m.
On Friday, 21 February 2014 at 16:08, Alexander Todorov wrote:
[...]
Guys I can do both.
1) Report packages which *have* test suites but they are *not* executed in %check
2) Report packages which *don't* have any test suites at all.
1) is easy but I'm more interested in 2)
Could you provide the rationale for 2? Why do you want to do it and
what does Fedora gain by doing it?
IMHO, 1 is the only one worth having filed as a bugs in our bugzilla.
You can track 2 any way you like, but mass bug filing is the wrong way.
It's not a bug in the Fedora package that upstream doesn't have a test
suite and I'll be sure to close such bugs as WONTFIX.
Personally, I do try to run the test suite in %check for my packages,
provided that it doesn't require (much) patching and is easy to do.
Regards,
Dominik
--
Fedora http://fedoraproject.org/wiki/User:Rathann
RPMFusion http://rpmfusion.org | MPlayer http://mplayerhq.hu
"Faith manages."
-- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"
9:20 a.m.
На 21.02.2014 17:16, Dominik 'Rathann' Mierzejewski написа:
On Friday, 21 February 2014 at 16:08, Alexander Todorov wrote:
[...]
> Guys I can do both.
>
> 1) Report packages which *have* test suites but they are *not* executed in %check
>
> 2) Report packages which *don't* have any test suites at all.
>
> 1) is easy but I'm more interested in 2)
Could you provide the rationale for 2? Why do you want to do it and
what does Fedora gain by doing it?
I want to work on improving 2). Though Bugzilla is a nice way to track it. Seems
like others think otherwise so I'll stick to the wiki.
--
Alex
7:54 a.m.
On Fri, Feb 21, 2014 at 04:16:59PM +0100, Dominik 'Rathann' Mierzejewski wrote:
> 2) Report packages which *don't* have any test suites at
all.
Could you provide the rationale for 2? Why do you want to do it and
what does Fedora gain by doing it?
Better code quality, less risk of regressions, easier automatic QA.
I think that this is a good initiative, and absolutely fits with something
Fedora can and should help with -- but I also agree that bugzilla probably
isn't the best place to track it. (It's kind of a shame, though, because a
bug tracker would be nicer than a wiki page. We just don't have the right
kind of flexiblity in our bugzilla setup to do it.)
--
Matthew Miller -- Fedora Project -- <mattdm(a)fedoraproject.org>
9:06 a.m.
На 21.02.2014 16:57, Daniel P. Berrange написа:
> Why would you file a bug in the Fedora bug tracker when the
package
> has no test suite upstream? That makes no sense - if the upstream
> package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning
filing bugs where upstream *does* have a test suite, but the RPM
specfile has not enabled it in %check.
Daniel,
I'm proposing quite the opposite. Tom got it right.
--
Alex
9:09 a.m.
On Fri, Feb 21, 2014 at 05:06:03PM +0200, Alexander Todorov wrote:
На 21.02.2014 16:57, Daniel P. Berrange написа:
>>Why would you file a bug in the Fedora bug tracker when the package
>>has no test suite upstream? That makes no sense - if the upstream
>>package has no tests then the bug belongs upstream not in Fedora.
>
>I don't believe that is what he is proposing. I read it as meaning
>filing bugs where upstream *does* have a test suite, but the RPM
>specfile has not enabled it in %check.
>
Daniel,
I'm proposing quite the opposite. Tom got it right.
In that case you should not file this against Fedora bugzilla. Any
such requests should go upstream exclusively.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:14 a.m.
Looks like reporting missing test suites in Bugzilla is not accepted. I guess
it's just me who prefers Bugzilla compared to other media.
I *will use the Wiki* for this.
On the topic of tests not executed in %check I *will use Bugzilla* but Alexander
Kurtakov brings up another angle - tests executed in %build. This needs more
investigation. I will start a separate thread for it.
Thanks for your answers,
Alex
8:59 a.m.
На 21.02.2014 16:53, Tom Hughes написа:
Why would you file a bug in the Fedora bug tracker when the package
has no test
suite upstream? That makes no sense - if the upstream package has no tests then
the bug belongs upstream not in Fedora.
Same reason you file kernel bugs in Bugzilla.redhat.com and not
bugzilla.kernel.org.
It's easier to use and to automate than me going around every possible bug
tracker in the upstream world and filing bugs.
Also easier for volunteers to find and work on them.
--
Alex
8:57 a.m.
New subject: May I file 1000 bugs aka upstream test suite tracking
----- Original Message -----
From: "Alexander Todorov" <atodorov(a)redhat.com>
To: "Discussion of RPM packaging standards and practices for Fedora"
<packaging(a)lists.fedoraproject.org>,
"Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Cc: "For testing and quality assurance of Fedora releases"
<test(a)lists.fedoraproject.org>
Sent: Friday, February 21, 2014 4:51:52 PM
Subject: Re: [Fedora-packaging] May I file 1000 bugs aka upstream test suite tracking
На 21.02.2014 16:27, Richard W.M. Jones написа:
> Is it correct that you're only going to be filing bugs when upstream
> tarballs already contain test suites, but they are just not enabled in
> the Fedora package?
Hi Richard,
I meant just the opposite. However I will also do what you suggest but this
will
result in far less number of bugs (probably around 100).
I want to track which packages *DO NOT* have any tests and later be able to
focus on creating them (be it working with volunteers, GSoC participants or
whoever is willing to step up to this task).
These are upstream bugs and having them open in Fedora bz would do no good.
While this value is valuable it's not Fedora bug and list of packages that are missing
test suites in general would better be kept on e.g. wiki page where volunteers can pick
one and work with upstream to add test suite.
For packages that has test suites but don't compile/run it's another story.
Alexander Kurtakov
Red Hat Eclipse team
I don't intend to force package maintainers to write tests if they don't want
to
(or can't).
--
Alex
--
test mailing list
test(a)lists.fedoraproject.org
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
8:55 a.m.
On Fri, Feb 21, 2014 at 04:22:42PM +0200, Alexander Todorov wrote:
Hi guys,
(note: devel, packaging and test lists) previously I've done a
little experiment and counted how many packages are likely to have
upstream test suites and how many don't:
http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
In general around 35% do have test suites, the rest don't.
My goal is to bring down the number of packages which ship without
any sort of test suite inside their code base.
The first step is to identify them and track them in Bugzilla.
My question is:
**Is everyone, especially package maintainers OK with me filing 1000+ bugs ?**
Last time I did so (around 100 bugs) it got a few people unhappy so
better ask this time!
If you have code that can fairly reliably detect whether a test suite
exists in the source tar.gz, then I think you would be justified
in filing bugs for spec files which have not enabled the test suite.
What I wouldn't do is blindly mass file bugs against every package
and ask the maintainer to investigate whether the test suite exists
or not. Basically you want to minimize the risk of false bug reports
against tests not being enabled, to avoid wasting maintainers time.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:01 a.m.
На 21.02.2014 16:55, Daniel P. Berrange написа:
If you have code that can fairly reliably detect whether a test
suite
exists in the source tar.gz, then I think you would be justified
in filing bugs for spec files which have not enabled the test suite.
At present I'm aware of 11 different location (directory or file names) which
represent tests. Not sure how accurate is this though.
What I wouldn't do is blindly mass file bugs against every
package
and ask the maintainer to investigate whether the test suite exists
or not.
I try to avoid that to the extent possible.
--
Alex
3713
days inactive
3716
days old
packaging@lists.fedoraproject.org
16 comments
7 participants
participants (7)
-
Aleksandar Kurtakov -
Alexander Todorov -
Daniel P. Berrange -
Dominik 'Rathann' Mierzejewski -
Matthew Miller -
Richard W.M. Jones -
Tom Hughes