Packages with missing %check

[Alexander Todorov]

Hi guys,
I have identified 551 packages on the Fedora 20 source DVD which are missing a 
%check section in their spec files but are very likely to have a test suite. See
https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...

For a few of them I already had bug open previously (will filter the list later 
when I run my scripts against latest Rawhide).

I have the following questions:

1) Do we consider this a bug and if yes what priority do you give it? From last 
week discussions it looks like most people prefer to have tests executed in %check.

2) Last week Alexander Kurtakov brought up the issue of packages executing their 
test suite during build. How to detect such packages?  OTOH we can have some 
false negatives (also see 3).


3) Another proposal (sorry don't remember who proposed it) was to have %check 
with a comment why the test suite is not executed (e.g. requires network) or why 
it is executed in %build.

I support this as it makes it clear to the QA person what is happening.
This comment can be later extended to explain why a particular package is 
missing test suite (e.g. man pages or packages containing data files).


How about making %check a packaging requirement in all cases - run tests or add 
a comment explaining why not, how to run them (e.g. make test) or why there are 
no tests for this package.


--
Alex

[Mikolaj Izdebski]

On 02/25/2014 11:45 AM, Alexander Todorov wrote:
>  Hi guys,
>  I have identified 551 packages on the Fedora 20 source DVD which are
>  missing a %check section in their spec files but are very likely to have
>  a test suite. See
>
> https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...
I can see a lot of Java packages on that list.  Most Java packages run
their test suites in %build and don't have %check section at all, so
they would be false-positives.

>  1) Do we consider this a bug and if yes what priority do you give
> it?
>  From last week discussions it looks like most people prefer to have
>  tests executed in %check. 
If upstream includes test suite but it is not ran without any comment
then I consider this as a bug.  If some tests cannot be ran because of
missing dependencies or because they require network access etc. then I
try to disable these test cases and run all others.

>  2) Last week Alexander Kurtakov brought up the issue of packages
>  executing their test suite during build. How to detect such packages? 
>  OTOH we can have some false negatives (also see 3). 
For packages using Maven (most common Java build system) you can use
heuristic implemented in the following script:
  http://mizdebsk.fedorapeople.org/scripts/wr-testsuitecheck
I used it to review my 250+ packages I maintain and it has low
false-positive ratio.

>  3) Another proposal (sorry don't remember who proposed it) was to
> have
>  %check with a comment why the test suite is not executed (e.g. requires
>  network) or why it is executed in %build. 
Commenting why tests are skipped is a very good thing, but I don't like
the idea of adding empty %check sections to my 250+ packages just for
the sake of documenting that tests are ran in %build "because that's
what we do in Java world".

-- 
Mikolaj Izdebski
IRC: mizdebsk

[Mat Booth]

On 25 February 2014 11:19, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:

>  On 02/25/2014 11:45 AM, Alexander Todorov wrote:
>  > 3) Another proposal (sorry don't remember who proposed it) was to have
>  > %check with a comment why the test suite is not executed (e.g. requires
>  > network) or why it is executed in %build.
>
>  Commenting why tests are skipped is a very good thing, but I don't like
>  the idea of adding empty %check sections to my 250+ packages just for
>  the sake of documenting that tests are ran in %build "because that's
>  what we do in Java world".
>
>  
Agreed, it seems like busy work to me that adds very little value to anyone
familiar with Java packages. Wouldn't it be easier to change the whatever
tool is generating this report to accommodate for this? "If package invokes
%mvn_build then don't expect there to be a %check section" seems like a
reasonable heuristic to me.


-- 
Mat Booth
http://fedoraproject.org/get-fedora

[Alexander Todorov]

На  4.03.2014 20:36, Mat Booth написа:
>  On 25 February 2014 11:19, Mikolaj Izdebski
> <mizdebsk(a)redhat.com&gt; wrote:
>
> > On 02/25/2014 11:45 AM, Alexander Todorov wrote:
> >> 3) Another proposal (sorry don't remember who proposed it) was to have
> >> %check with a comment why the test suite is not executed (e.g. requires
> >> network) or why it is executed in %build.
> >
> > Commenting why tests are skipped is a very good thing, but I don't like
> > the idea of adding empty %check sections to my 250+ packages just for
> > the sake of documenting that tests are ran in %build "because that's
> > what we do in Java world".
> >
> >
>  Agreed, it seems like busy work to me that adds very little value to anyone
>  familiar with Java packages. 
You are forgetting everyone that is not so familiar with Java.

Also I didn't ask you (as a package owner) to do it explicitly, I've asked you 
to accept a patch which should be much more easier.


>  Wouldn't it be easier to change the whatever
>  tool is generating this report to accommodate for this? "If package invokes
>  %mvn_build then don't expect there to be a %check section" seems like a
>  reasonable heuristic to me.
>
>  
See https://bugzilla.redhat.com/show_bug.cgi?id=1072417#c4 to avoid repeating 
myself.


Even if the tool uses heuristics to exclude some groups of packages it will not 
be obvious why there's no %check section. It could be because tests are executed 
in %build, because they need root or network access and are disabled, because 
the test framework used is not available (see DHCP) or anything else.

A small comment makes it much more clear and straight forward.

--
Alex

[Stanislav Ochotnicky]

On Wed 05 Mar 2014 11:23:23 AM CET Alexander Todorov wrote:

>  На  4.03.2014 20:36, Mat Booth написа:
> > On 25 February 2014 11:19, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
> >
> >> On 02/25/2014 11:45 AM, Alexander Todorov wrote:
> >>> 3) Another proposal (sorry don't remember who proposed it) was to have
> >>> %check with a comment why the test suite is not executed (e.g. requires
> >>> network) or why it is executed in %build.
> >>
> >> Commenting why tests are skipped is a very good thing, but I don't like
> >> the idea of adding empty %check sections to my 250+ packages just for
> >> the sake of documenting that tests are ran in %build "because that's
> >> what we do in Java world".
> >>
> >>
> > Agreed, it seems like busy work to me that adds very little value to anyone
> > familiar with Java packages.
>
>  You are forgetting everyone that is not so familiar with Java. 
Why are you filing bugs (with patches) you don't understand then? Have
you asked *anybody* to help you out with exclusions? Have you gotten in
touch with Java|Perl|PythonX|Y|Z SIG and ask for their assistance in better
identification of testsuites and if they are being run?

You are putting cart before the horse rushing the whole mass bug filing
process without understanding consequences. You are going to be
ignored. Your bugs are going to get closed/wontfix. You are going to get
annoyed. You are not going to achieve much (if anything).

>  Also I didn't ask you (as a package owner) to do it explicitly,
> I've asked you
>  to accept a patch which should be much more easier. 
Patch which contains text which you haven't verified is
correct. Quoting:

+%check
+# tests are executed during %build
+

How do *you* *know* they are executed during build? Do you even know how
to recognize a difference between following:

ant dist test javadoc
ant dist javadoc
ant jar
%mvn_build
%mvn_build -f
%mvn_build -Dtestnotmatchpattern=\*
%mvn_build -- -Dmaven.test.skip=true
mvn-rpmbuild -Dmaven.test.failure.ignore=true
...

Hint: Except "%mvn_build" and "ant dist test javadoc" these
executions
most likely don't run tests or if they do they ignore the failures

> > Wouldn't it be easier to change the whatever
> > tool is generating this report to accommodate for this? "If package invokes
> > %mvn_build then don't expect there to be a %check section" seems like a
> > reasonable heuristic to me.
> >
> >
>
>  See https://bugzilla.redhat.com/show_bug.cgi?id=1072417#c4 to avoid repeating
>  myself.
>
>
>  Even if the tool uses heuristics to exclude some groups of packages it will not
>  be obvious why there's no %check section. It could be because tests are executed
>  in %build, because they need root or network access and are disabled, because
>  the test framework used is not available (see DHCP) or anything else. 
Right, so your patches are basically worthless and you are planning to file
possibly hundreds of bugs where maintainers will most likely manually
close as WONTFIX/NOTABUG because it's clear to anyone working in their
respective SIG what's happening there.

>  A small comment makes it much more clear and straight forward.
I want my packages to run tests. If there are tests upstream but I am
not running them, sure there should be a comment or even better a
FutureFeature bug to fix the testsuite. We are mostly doing all of this
in Java SIG packages. Adding useless comment to spec files is not the
way to improve things.

If you want to improve stuff then focus on identifying packages which
should run tests but don't.

--
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Software Engineer - Developer Experience

PGP: 7B087241
Red Hat Inc.                               http://cz.redhat.com

[Alexander Todorov]

На  5.03.2014 14:12, Stanislav Ochotnicky написа:
>  Why are you filing bugs (with patches) you don't understand then? 
This is a foolish statement to make without knowing what I do and don't know or 
understand.


>  Patch which contains text which you haven't verified is
>  correct. Quoting:
>
>  +%check
>  +# tests are executed during %build
>  +
>
>  How do *you* *know* they are executed during build? 
FYI, first I got a list of possible packages which have their tests run in 
%build from mizdebsk, then I inspected them and built them *by hand* to verify 
that was indeed correct (e.g. apache-commons-codec, apache-comons-logging, 
python-blivet, python-urlgrabber, etc.)

If you find a patch which is incorrect (that is not running the test suite 
properly or stating an invalid comment) point me to it and I will work to update 
it.

--
Alex

[Stanislav Ochotnicky]

On Wed 05 Mar 2014 03:57:17 PM CET Alexander Todorov wrote:
>  На  5.03.2014 14:12, Stanislav Ochotnicky написа:
> >
> > Why are you filing bugs (with patches) you don't understand then?
>
>  This is a foolish statement to make without knowing what I do and don't know or
>  understand. 
That's the whole point though. Several people from Java SIG feel the
same way about those patches...

> > Patch which contains text which you haven't verified is
> > correct. Quoting:
> >
> > +%check
> > +# tests are executed during %build
> > +
> >
> > How do *you* *know* they are executed during build?
>
>  FYI, first I got a list of possible packages which have their tests run in
>  %build from mizdebsk, then I inspected them and built them *by hand* to verify
>  that was indeed correct (e.g. apache-commons-codec, apache-comons-logging,
>  python-blivet, python-urlgrabber, etc.) 
So you are filing bugs for components which *are* running tests. Is it
so weird that we consider that a non-issue while we have possibly
hundreds of packages which are not running tests at all?

>  If you find a patch which is incorrect (that is not running the test
> suite
>  properly or stating an invalid comment) point me to it and I will work to update
>  it. 
Every single patch is incorrect due to one problem:
$ rpmlint apache-commons-codec.spec
apache-commons-codec.spec:58: W: macro-in-comment %build
0 packages and 1 specfiles checked; 0 errors, 1 warnings.

Sure it's a minor thing, but I'd hope as a QA guy you might appreciate
the irony.

Putting that aside, I've always worked with one though when working on
Java packaging (quoting Exupéry)
 [snip] perfection is attained not when there is nothing more to add, but
 when there is nothing more to remove.

I am yet to see a convincing argument for empty %check sections
improving Fedora either for users or developers.

--
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Software Engineer - Developer Experience

PGP: 7B087241
Red Hat Inc.                               http://cz.redhat.com

[Mat Booth]

On 5 March 2014 10:23, Alexander Todorov <atodorov(a)redhat.com&gt; wrote:

>  На  4.03.2014 20:36, Mat Booth написа:
>
>   On 25 February 2014 11:19, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
> >
> >  On 02/25/2014 11:45 AM, Alexander Todorov wrote:
> >>
> >>> 3) Another proposal (sorry don't remember who proposed it) was to have
> >>> %check with a comment why the test suite is not executed (e.g. requires
> >>> network) or why it is executed in %build.
> >>>
> >>
> >> Commenting why tests are skipped is a very good thing, but I don't like
> >> the idea of adding empty %check sections to my 250+ packages just for
> >> the sake of documenting that tests are ran in %build "because that's
> >> what we do in Java world".
> >>
> >>
> >>  Agreed, it seems like busy work to me that adds very little value to
> > anyone
> > familiar with Java packages.
> >
>
>  You are forgetting everyone that is not so familiar with Java.
>
>  Also I didn't ask you (as a package owner) to do it explicitly, I've asked
>  you to accept a patch which should be much more easier.
>
>
>
>   Wouldn't it be easier to change the whatever
> > tool is generating this report to accommodate for this? "If package
> > invokes
> > %mvn_build then don't expect there to be a %check section" seems like a
> > reasonable heuristic to me.
> >
> >
> >
>  See https://bugzilla.redhat.com/show_bug.cgi?id=1072417#c4 to avoid
>  repeating myself.
>
>
>  Even if the tool uses heuristics to exclude some groups of packages it
>  will not be obvious why there's no %check section. It could be because
>  tests are executed in %build, because they need root or network access and
>  are disabled, because the test framework used is not available (see DHCP)
>  or anything else.
>
>  
If the tool excludes a package based on a heuristic, it can also tell you
*why* it was excluded (the heuristic was added for a reason!) A comment in
the SPEC is unnecessary duplication of information at that point.

[Adam Williamson]

On Tue, 2014-03-04 at 18:36 +0000, Mat Booth wrote:
>  On 25 February 2014 11:19, Mikolaj Izdebski
> <mizdebsk(a)redhat.com&gt; wrote:
>  
>  > On 02/25/2014 11:45 AM, Alexander Todorov wrote:
>  > > 3) Another proposal (sorry don't remember who proposed it) was to have
>  > > %check with a comment why the test suite is not executed (e.g. requires
>  > > network) or why it is executed in %build.
>  >
>  > Commenting why tests are skipped is a very good thing, but I don't like
>  > the idea of adding empty %check sections to my 250+ packages just for
>  > the sake of documenting that tests are ran in %build "because that's
>  > what we do in Java world".
>  >
>  >
>  Agreed, it seems like busy work to me that adds very little value to anyone
>  familiar with Java packages. Wouldn't it be easier to change the whatever
>  tool is generating this report to accommodate for this? "If package invokes
>  %mvn_build then don't expect there to be a %check section" seems like a
>  reasonable heuristic to me. 
It was my suggestion initially, and I think it's a decent idea just from
a general 'legibility of spec to outside readers' perspective. From that
perspective, though, it may reasonable to assume an outside reader knows
about how tests are run in Javaland, especially if this is documented as
a general thing somewhere in the Java packaging guidelines. I wasn't
suggesting it as an idiotic, inflexible Policy Requirement, just as a
thing to consider.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Michael Schwendt]

On Tue, 25 Feb 2014 12:45:11 +0200, Alexander Todorov wrote:

>  Hi guys,
>  I have identified 551 packages on the Fedora 20 source DVD which are missing a 
>  %check section in their spec files but are very likely to have a test suite. See
>
> https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...
>  
Could you add a short classifier to each src.rpm name, which sums up why
your checker believes there is a test-suite available that's not run in
%check. For example, "make-check-target" found or "tests-subdir" found
or
something like that?

[Alexander Todorov]

На 25.02.2014 13:40, Michael Schwendt написа:
>  On Tue, 25 Feb 2014 12:45:11 +0200, Alexander Todorov wrote:
>
> > Hi guys,
> > I have identified 551 packages on the Fedora 20 source DVD which are missing a
> > %check section in their spec files but are very likely to have a test suite. See
> >
> https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...
> >
>
>  Could you add a short classifier to each src.rpm name, which sums up why
>  your checker believes there is a test-suite available that's not run in
>  %check. For example, "make-check-target" found or "tests-subdir"
> found or
>  something like that?
>  
Not at the moment but I will think about it. Currently the WITHOUT files are 
just the diff between WITH files with diff formatting cleared away.

e.g. with-tests-WITHOUT-check == diff with-check with-tests


--
Alex

[Michael Schwendt]

On Tue, 25 Feb 2014 13:47:01 +0200, Alexander Todorov wrote:

>  >>
> https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...
>  >>
>  >
>  > Could you add a short classifier to each src.rpm name, which sums up why
>  > your checker believes there is a test-suite available that's not run in
>  > %check. For example, "make-check-target" found or "tests-subdir"
> found or
>  > something like that?
>  >
>  
>  Not at the moment but I will think about it. Currently the WITHOUT files are 
>  just the diff between WITH files with diff formatting cleared away.
>  
>  e.g. with-tests-WITHOUT-check == diff with-check with-tests 
Well, your check may be too simplified. I've had a look at 

  /mnt/fedora/l/libetpan-1.1-7.fc20.src.rpm

and it contains a "tests" subdir with a few test programs, but no test-suite
to run automatically.

[Alexander Todorov]

На 25.02.2014 13:57, Michael Schwendt написа:
>  On Tue, 25 Feb 2014 13:47:01 +0200, Alexander Todorov wrote:
>
> >>>
> https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...
> >>>
> >>
> >> Could you add a short classifier to each src.rpm name, which sums up why
> >> your checker believes there is a test-suite available that's not run in
> >> %check. For example, "make-check-target" found or
> "tests-subdir" found or
> >> something like that?
> >>
> >
> > Not at the moment but I will think about it. Currently the WITHOUT files are
> > just the diff between WITH files with diff formatting cleared away.
> >
> > e.g. with-tests-WITHOUT-check == diff with-check with-tests
>
>  Well, your check may be too simplified. I've had a look at
>
>     /mnt/fedora/l/libetpan-1.1-7.fc20.src.rpm
>
>  and it contains a "tests" subdir with a few test programs, but no test-suite
>  to run automatically.
>  
I said it didn't have a %check section in the spec file, not that it was missing 
tests. Hope its more clear now.

Whether or not the contents of the tests/ directory is a bunch of simple scripts 
to run or a proper test suite I can't tell without a manual inspection.

--
Alex

[Michael Schwendt]

On Tue, 25 Feb 2014 16:32:37 +0200, Alexander Todorov wrote:

>  > Well, your check may be too simplified. I've had a look at
>  >
>  >    /mnt/fedora/l/libetpan-1.1-7.fc20.src.rpm
>  >
>  > and it contains a "tests" subdir with a few test programs, but no
> test-suite
>  > to run automatically.
>  >
>  
>  I said it didn't have a %check section in the spec file, not that it was missing 
>  tests. Hope its more clear now. 
Hmmm, if mere existance of a "tests" subdir is considered as "very likely
to have a test suite", that puts a lot on the TODO list to avoid such
false positives.

 | missing a %check section in their spec files but are very likely
 | to have a test suite
 
>  Whether or not the contents of the tests/ directory is a bunch of
> simple scripts 
>  to run or a proper test suite I can't tell without a manual inspection. 
Hence my suggestion to classify your findings a bit based on what your
checker script has examined.

[Adam Jackson]

On Tue, 2014-02-25 at 12:45 +0200, Alexander Todorov wrote:

>  1) Do we consider this a bug and if yes what priority do you give it?
> From last 
>  week discussions it looks like most people prefer to have tests executed in %check.
I don't consider %check to be an appropriate way to run tests, so no, I
don't consider it a bug.  Stating it broadly: we don't have a separate
phase for it in koji, which means we can't adequately set up tests in
general; and adding it to koji would be a mistake, because build
construction and build validation are fundamentally different phases.

This is an argument against %check, not against testing in general.  We
should be relying on rpmbuild less, not more.  rpm doesn't even have
anything like Requires(check), does it?

- ajax

[Matthias Runge]

On Tue, Feb 25, 2014 at 08:50:18AM -0500, Adam Jackson wrote:
>  On Tue, 2014-02-25 at 12:45 +0200, Alexander Todorov wrote:
>  
>  > 1) Do we consider this a bug and if yes what priority do you give it? From last 
>  > week discussions it looks like most people prefer to have tests executed in %check.
>  
>  I don't consider %check to be an appropriate way to run tests, so no, I
>  don't consider it a bug.  Stating it broadly: we don't have a separate
>  phase for it in koji, which means we can't adequately set up tests in
>  general; and adding it to koji would be a mistake, because build
>  construction and build validation are fundamentally different phases.
>  
>  This is an argument against %check, not against testing in general.  We
>  should be relying on rpmbuild less, not more.  rpm doesn't even have
>  anything like Requires(check), does it? 
Hmm, then we should probably about enhancing this? Like adding a 
separate phase of for %check to rpmnbuild?

IMHO, that still belongs to building a package, as I don't want
a package pushed, when checks are not executed successfully. 

Checks could be executed in a different phase, e.g before acutally
pushing to a repo. Separating from build would have another benefit: 
being able to execute tests, e.g when required packages were updated.
That'd require significant changes in our infrastructure and in many
packages, as test suites and test data currently don't belong to 
most packages.
-- 
Matthias Runge <mrunge(a)matthias-runge.de&gt;

[Adam Jackson]

On Tue, 2014-02-25 at 15:45 +0100, Matthias Runge wrote:
>  On Tue, Feb 25, 2014 at 08:50:18AM -0500, Adam Jackson wrote:
>  > This is an argument against %check, not against testing in general.  We
>  > should be relying on rpmbuild less, not more.  rpm doesn't even have
>  > anything like Requires(check), does it?
>  
>  Hmm, then we should probably about enhancing this? Like adding a 
>  separate phase of for %check to rpmnbuild? 
This is exactly not "relying on rpmbuild less", so no, I don't think we
should do that.  I was attempting to use rpmbuild's existing inadequacy
as an argument for not bothering with it, not suggesting we enhance a
bad tool.

>  IMHO, that still belongs to building a package, as I don't want
>  a package pushed, when checks are not executed successfully.  
You are failing to distinguish between "pushed to package git" and
"pushed to an installable repository", which is a mistake.

- ajax

[Matthias Runge]

On Tue, Feb 25, 2014 at 10:01:06AM -0500, Adam Jackson wrote:
>  You are failing to distinguish between "pushed to package
> git" and
>  "pushed to an installable repository", which is a mistake. 
I'm distinguishing:
1. package compiles successfully
2. rpmbuild manages to include all files into an rpm package
3. including that rpm in a package repository

In this last step (3) could be executed a test/check if that
compiled package actually works; that could be e.g unit tests or
whatever.

I was not talking about a git repo (or so), which may be used as 
backend for any package descriptions, patches etc.

-- 
Matthias Runge <mrunge(a)matthias-runge.de&gt;

[Richard W.M. Jones]

On Tue, Feb 25, 2014 at 03:45:51PM +0100, Matthias Runge wrote:
>  On Tue, Feb 25, 2014 at 08:50:18AM -0500, Adam Jackson wrote:
>  > On Tue, 2014-02-25 at 12:45 +0200, Alexander Todorov wrote:
>  > 
>  > > 1) Do we consider this a bug and if yes what priority do you give it? From last
>
>  > > week discussions it looks like most people prefer to have tests executed in
> %check.
>  > 
>  > I don't consider %check to be an appropriate way to run tests, so no, I
>  > don't consider it a bug.  Stating it broadly: we don't have a separate
>  > phase for it in koji, which means we can't adequately set up tests in
>  > general; and adding it to koji would be a mistake, because build
>  > construction and build validation are fundamentally different phases.
>  > 
>  > This is an argument against %check, not against testing in general.  We
>  > should be relying on rpmbuild less, not more.  rpm doesn't even have
>  > anything like Requires(check), does it?
>  
>  Hmm, then we should probably about enhancing this? Like adding a 
>  separate phase of for %check to rpmnbuild? 
There is a proposal to allow better testing of packages, and groups of
packages.  Actually at least two proposals but the one I'm thinking of
now is:

  https://fedoraproject.org/wiki/Taskotron

Previous discussion:

  https://lists.fedoraproject.org/pipermail/devel/2013-December/193267.html
& https://lists.fedoraproject.org/pipermail/devel/2014-January/thread.html#...

>  IMHO, that still belongs to building a package, as I don't want
>  a package pushed, when checks are not executed successfully.  
Some testing can be done in %check, and some testing needs to be done
outside %check because either it involves multiple packages, or
because it's too heavyweight for %check, or because it involves some
extra permissions (eg. root, a VM, etc).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Adam Williamson]

On Tue, 2014-02-25 at 22:38 +0000, Richard W.M. Jones wrote:
>  On Tue, Feb 25, 2014 at 03:45:51PM +0100, Matthias Runge wrote:
>  > On Tue, Feb 25, 2014 at 08:50:18AM -0500, Adam Jackson wrote:
>  > > On Tue, 2014-02-25 at 12:45 +0200, Alexander Todorov wrote:
>  > > 
>  > > > 1) Do we consider this a bug and if yes what priority do you give it? From
> last 
>  > > > week discussions it looks like most people prefer to have tests executed
> in %check.
>  > > 
>  > > I don't consider %check to be an appropriate way to run tests, so no, I
>  > > don't consider it a bug.  Stating it broadly: we don't have a separate
>  > > phase for it in koji, which means we can't adequately set up tests in
>  > > general; and adding it to koji would be a mistake, because build
>  > > construction and build validation are fundamentally different phases.
>  > > 
>  > > This is an argument against %check, not against testing in general.  We
>  > > should be relying on rpmbuild less, not more.  rpm doesn't even have
>  > > anything like Requires(check), does it?
>  > 
>  > Hmm, then we should probably about enhancing this? Like adding a 
>  > separate phase of for %check to rpmnbuild?
>  
>  There is a proposal to allow better testing of packages, and groups of
>  packages.  Actually at least two proposals but the one I'm thinking of
>  now is:
>  
>    https://fedoraproject.org/wiki/Taskotron 
Just for the sake of accuracy, taskotron isn't so much a proposal as a
thing undergoing full-steam-ahead development, which fesco is kind of
expecting to be available for fedora.next:

https://bitbucket.org/fedoraqa

it is the replacement for AutoQA, which is a thing that currently exists
and which we currently use for Real Stuff.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Andrew Lutomirski]

On Tue, Feb 25, 2014 at 2:38 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Tue, Feb 25, 2014 at 03:45:51PM +0100, Matthias Runge wrote:
> > On Tue, Feb 25, 2014 at 08:50:18AM -0500, Adam Jackson wrote:
> > > On Tue, 2014-02-25 at 12:45 +0200, Alexander Todorov wrote:
> > >
> > > > 1) Do we consider this a bug and if yes what priority do you give it? From
> last
> > > > week discussions it looks like most people prefer to have tests executed in
> %check.
> > >
> > > I don't consider %check to be an appropriate way to run tests, so no, I
> > > don't consider it a bug.  Stating it broadly: we don't have a separate
> > > phase for it in koji, which means we can't adequately set up tests in
> > > general; and adding it to koji would be a mistake, because build
> > > construction and build validation are fundamentally different phases.
> > >
> > > This is an argument against %check, not against testing in general.  We
> > > should be relying on rpmbuild less, not more.  rpm doesn't even have
> > > anything like Requires(check), does it?
> >
> > Hmm, then we should probably about enhancing this? Like adding a
> > separate phase of for %check to rpmnbuild?
>
>  There is a proposal to allow better testing of packages, and groups of
>  packages.  Actually at least two proposals but the one I'm thinking of
>  now is:
>
>    https://fedoraproject.org/wiki/Taskotron
>
>  Previous discussion:
>
>    https://lists.fedoraproject.org/pipermail/devel/2013-December/193267.html
>  & https://lists.fedoraproject.org/pipermail/devel/2014-January/thread.html#...
>
> > IMHO, that still belongs to building a package, as I don't want
> > a package pushed, when checks are not executed successfully.
>
>  Some testing can be done in %check, and some testing needs to be done
>  outside %check because either it involves multiple packages, or
>  because it's too heavyweight for %check, or because it involves some
>  extra permissions (eg. root, a VM, etc). 
Just to mention: there are probably many packages where the equivalent
of %check can't be run without access to a source tree, so Taskotron
can't usefully replace %check.  I maintain a package like that.

--Andy

[Adam Williamson]

On Tue, 2014-02-25 at 18:35 -0800, Andrew Lutomirski wrote:

>  Just to mention: there are probably many packages where the
> equivalent
>  of %check can't be run without access to a source tree, so Taskotron
>  can't usefully replace %check.  I maintain a package like that. 
How do you get from that premise to that conclusion? We know where the
source tree is...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Dridi Boukelmoune]

On Wed, Feb 26, 2014 at 4:41 AM, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  On Tue, 2014-02-25 at 18:35 -0800, Andrew Lutomirski wrote:
>
> > Just to mention: there are probably many packages where the equivalent
> > of %check can't be run without access to a source tree, so Taskotron
> > can't usefully replace %check.  I maintain a package like that.
>
>  How do you get from that premise to that conclusion? We know where the
>  source tree is... 
I don't think this would be a good idea to avoid such tests in %check.
If you do that you have to later fetch the source code again, build it
again and finally you can run the tests.

Dridi

>  --
>  Adam Williamson
>  Fedora QA Community Monkey
>  IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
>  http://www.happyassassin.net
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Adam Jackson]

On Wed, 2014-02-26 at 11:00 +0100, Dridi Boukelmoune wrote:

>  I don't think this would be a good idea to avoid such tests in
> %check.
>  If you do that you have to later fetch the source code again, build it
>  again and finally you can run the tests. 
No you don't.  There's no reason the final rpms need to be the only
output of a koji build.  Just save the built tree as another build-time
artifact.

- ajax

[Colin Walters]

On Wed, Feb 26, 2014 at 10:24 AM, Adam Jackson <ajax(a)redhat.com&gt; wrote:
>   Just save the built tree as another build-time
>  artifact.
>   
We do this already with glib2:
http://pkgs.fedoraproject.org/cgit/glib2.git/commit/?id=25351c50

And that's the general idea - assemble a tree containing -test 
subpackages, and run the tests in that tree.  For example, I really 
want the glib2 tests to run after the kernel changes.

There's no runner for InstalledTests yet though for Fedora - but I 
already have working code, just need to port from gnome-continuous.

[Stanislav Ochotnicky]

Adam Williamson <awilliam(a)redhat.com&gt; writes:

>  On Tue, 2014-02-25 at 18:35 -0800, Andrew Lutomirski wrote:
>
> > Just to mention: there are probably many packages where the equivalent
> > of %check can't be run without access to a source tree, so Taskotron
> > can't usefully replace %check.  I maintain a package like that.
>
>  How do you get from that premise to that conclusion? We know where the
>  source tree is... 
Because unit tests are designed to be run as part of the build
process. It's not impossible to run them *after* the build, but good
luck making it work reliably across all packages without manual work.

As I see it:
 * %check (or equivalent) is for unit testing
 * taskotron (or equivalent) is integration testing

Each has its own value

--
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Software Engineer - Developer Experience

PGP: 7B087241
Red Hat Inc.                               http://cz.redhat.com

[Colin Walters]

On Wed, Feb 26, 2014 at 5:01 AM, Stanislav Ochotnicky 
<sochotnicky(a)redhat.com&gt; wrote:
>  
>  Because unit tests are designed to be run as part of the build
>  process. It's not impossible to run them *after* the build, but good
>  luck making it work reliably across all packages without manual work.
>   
The https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
initative, as implemented by gnome-continuous, takes these "unit tests" 
as you call them and runs them as what you call integration tests.  
(Personally, I think distinguishing them is a broken idea.  No one runs 
just one bit of software, they run a tree - a complete system)

For example, after glib changes, I rerun the *gtk* tests.  After gtk 
changes, I rerun *application* tests.

This simple change of taking existing valuable tests that were run at 
once most per build and turning them into something run 50 or more 
times a day made them much more valuable.  It also revealed many of 
them were full of race conditions...

[Stanislav Ochotnicky]

On Wed 26 Feb 2014 01:41:36 PM CET Colin Walters wrote:

>  On Wed, Feb 26, 2014 at 5:01 AM, Stanislav Ochotnicky
>  <sochotnicky(a)redhat.com&gt; wrote:
> >
> > Because unit tests are designed to be run as part of the build
> > process. It's not impossible to run them *after* the build, but good
> > luck making it work reliably across all packages without manual work.
> >
>  The https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
>  initative, as implemented by gnome-continuous, takes these "unit tests"
>  as you call them and runs them as what you call integration tests. 
"I" didn't name them. I used standard names for different testing levels
as defined by software engineering bodies. Quoting from SWEBOK:

Software testing is usually performed at different levels along the
development and maintenance processes. That is to say, the target of the
test can vary: a single module, a group of such modules (related by
purpose, use, behavior, or structure), or a whole system. Three big test
stages can be conceptually distinguished, namely Unit, Integration, and
System. No process model is implied, nor are any of those three stages
assumed to have greater importance than the other two.

>  (Personally, I think distinguishing them is a broken idea.  No one
> runs
>  just one bit of software, they run a tree - a complete system) 
Your personal opinion goes against what best practices of software
engineering have been for at least past decade. Perhaps you'll not hold
it against me if I am going to dismissive about your personal opinion in
this case.

>  For example, after glib changes, I rerun the *gtk* tests.  After gtk
>  changes, I rerun *application* tests. 
During making glib changes you should run glib unit tests to have some
basic level of assurance you didn't introduce regressions or unwanted
changes. After that you can run integration/system tests to ensure that
gtk or application tests still pass. Simply put unit, integration and
system tests are run during different phases of development or
maintenance.

>  This simple change of taking existing valuable tests that were run
> at
>  once most per build and turning them into something run 50 or more
>  times a day made them much more valuable.  It also revealed many of
>  them were full of race conditions... 
It's great that your change helped with discovering issues but perhaps
your original testsuite was mixing different levels of testing in the
same code. Unit tests are supposed to be quick, dirty solutions using
mock objects and other "hacks" to allow of testing with small
granularity. I could probably go on but this would quickly become off
topic for fedora-devel and there's tons of literature on introduction to
software testing.

--
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Software Engineer - Developer Experience

PGP: 7B087241
Red Hat Inc.                               http://cz.redhat.com

[Colin Walters]

On Wed, Feb 26, 2014 at 8:09 AM, Stanislav Ochotnicky 
<sochotnicky(a)redhat.com&gt; wrote:
>  
>  "I" didn't name them. I used standard names for different testing 
>  levels
>  as defined by software engineering bodies. Quoting from SWEBOK:
>   
Yes, I think they're wrong.   Well, "suboptimal" is a better word.

>  During making glib changes you should run glib unit tests to have
> some
>  basic level of assurance you didn't introduce regressions or unwanted
>  changes.
>   
The *very first* test I run is "does the OS still boot"?  That's called 
"smoketest" for me, and it only takes a few minutes.

Yes, after every change, even just an updated translation, I boot the 
OS, run through systemd, gdm, gnome-shelll, and everything and ensure 
it still logs in.

This is *before* I run the GTK+ tests or the glib tests or any other 
test.  Why?  Because if that fails, there's *no point* to running the 
other tests.  The system is broken.  The originating change is 
investigated and is up for reverting.   Whether or not the GKeyFile 
test pass or not is irrelevant.

(Also, there is the fact that InstalledTests are guaranteed to be run 
in a logged in desktop, not a mock chroot, so the above needs to work 
anyways)

>  It's great that your change helped with discovering issues but
> perhaps
>  your original testsuite was mixing different levels of testing in the
>  same code. Unit tests are supposed to be quick, dirty solutions using
>  mock objects and other "hacks" to allow of testing with small
>  granularity.
>   
Ah, but if one makes "integration tests" very fast and easy to run as I

have, then there's less need for "quick and dirty".

[Adam Jackson]

On Wed, 2014-02-26 at 13:11 +0000, Colin Walters wrote:

>  Ah, but if one makes "integration tests" very fast and easy
> to run as
>  I have, then there's less need for "quick and dirty". 
Which is sort of the crux of my argument against %check.  "Hey, we found
this hammer, it smells kind of funny and it's got the claw on both ends
and it's made of some really crumbly plastic, but maybe we can still hit
some nails with it?"  You're going to want a better hammer anyway, put
down the toy.

- ajax

[Miloslav Trmač]

2014-02-26 14:11 GMT+01:00 Colin Walters <walters(a)verbum.org&gt;:

>  During making glib changes you should run glib unit tests to have
> some
>  basic level of assurance you didn't introduce regressions or unwanted
>  changes.
>
>  The *very first* test I run is "does the OS still boot"?  That's called
>  "smoketest" for me, and it only takes a few minutes.
>  
That seems to be optimizing for bugs that break the boot, when bugs that
occur in less-frequently used parts of the system are far more common; a
lot of software is not used, or not critical, in the boot path.

Yes, after every change, even just an updated translation, I boot the OS,
>  run through systemd, gdm, gnome-shelll, and everything and ensure it
> still
>  logs in.
>  
That would make sense if running the whole battery of tests after every
commit were *both* free and very fast. What I see around me, however, is
that there is never quite as much hardware as everyone would want to use,
and full test runs of the system take days.  So, testing after every commit
is impractical, and when one starts with "the system fails to boot",
backtracking down to the offending commit in *the whole system* is rather a
lot of work.

Starting with the small-scale unit tests is a more efficient use of
resources: the tests are smaller and run faster, so it is actually
practical to run them after every commit (or at the very least before any
git push or other release of the component visible to other people).
    Mirek

[Matthias Clasen]

On Wed, 2014-02-26 at 16:58 +0100, Miloslav Trmač wrote:
>  2014-02-26 14:11 GMT+01:00 Colin Walters <walters(a)verbum.org&gt;:
>          The *very first* test I run is "does the OS still
> boot"?
>          That's called "smoketest" for me, and it only takes a few
>          minutes.
>  
>  
>  That seems to be optimizing for bugs that break the boot, when bugs
>  that occur in less-frequently used parts of the system are far more
>  common; a lot of software is not used, or not critical, in the boot
>  path. 
It is optimizing for answering the most important question first, which
is: Can I tag this build / push this to the thousands of developers and
testers which be blocked from testing the less-frequently used parts of
the system if their system doesn't boot anymore after the upgrade.

[Miloslav Trmač]

2014-02-26 17:46 GMT+01:00 Matthias Clasen <mclasen(a)redhat.com&gt;:

>  On Wed, 2014-02-26 at 16:58 +0100, Miloslav Trmač wrote:> That
> seems to be
>  optimizing for bugs that break the boot, when bugs
>  > that occur in less-frequently used parts of the system are far more
>  > common; a lot of software is not used, or not critical, in the boot
>  > path.
>
>  It is optimizing for answering the most important question first, which
>  is: Can I tag this build / push this to the thousands of developers and
>  testers which be blocked from testing the less-frequently used parts of
>  the system if their system doesn't boot anymore after the upgrade.
>  
Are you saying that the boot path should have tests, and the
less-frequently used parts of the system should be verified by seeing
whether any human users notice breakage?  I don't think "if it boots, ship
it" is an acceptable quality target.
    Mirek

[Adam Jackson]

On Wed, 2014-02-26 at 17:50 +0100, Miloslav Trmač wrote:

>  Are you saying that the boot path should have tests, 
Yes, that is what was being said.

>   and the less-frequently used parts of the system should be verified
>  by seeing whether any human users notice breakage?  
No, that was neither said nor implied.

- ajax

[Colin Walters]

On Wed, Feb 26, 2014 at 11:50 AM, Miloslav Trmač <mitr(a)volny.cz&gt; wrote:
>  
>  Are you saying that the boot path should have tests, and the 
>  less-frequently used parts of the system should be verified by seeing 
>  whether any human users notice breakage? 
>   
No.  First, it's more that in order to run any other tests, the system 
must boot.

Now of course there's non-test "reporting" stuff like static code 
analysis, code style checkers...those things make sense in a mock 
chroot to me. They're not significantly more demanding than a simple 
build.

>  I don't think "if it boots, ship it" is an acceptable
> quality target.
>   
The neat thing is that I can easily define multiple, progressively 
enhanced quality targets, and I do in fact.

For each ref, I have "buildmaster" in the string.  All that happened 
with this branch is that we stuck the RPMs together.  We don't know if 
it boots.

But quickly after that, we do boot.  That tree gets tagged as 
"smoketested" if it succeeds, and you can track just that tree.  It 
introduces a few minutes of latency.

After smoketested, there's a vast array of potential tests to run.  I 
think it makes sense to have a "baseline" per tree.  For example, for 
server/docker-io, it might mean that docker can start and pull images 
and run some basic things.

(One could argue for a per-tree smoketested that is this type of thing, 
but I think it makes sense to have a distinct phase which is the basic 
fundamental requirements for anything at all, such as having a 
functional systemd)

[Richard W.M. Jones]

On Wed, Feb 26, 2014 at 04:58:43PM +0100, Miloslav Trmač wrote:
>  2014-02-26 14:11 GMT+01:00 Colin Walters <walters(a)verbum.org&gt;:
>  
>  > During making glib changes you should run glib unit tests to have some
>  > basic level of assurance you didn't introduce regressions or unwanted
>  > changes.
>  >
>  > The *very first* test I run is "does the OS still boot"?  That's
> called
>  > "smoketest" for me, and it only takes a few minutes.
>  >
>  
>  That seems to be optimizing for bugs that break the boot, when bugs that
>  occur in less-frequently used parts of the system are far more common; a
>  lot of software is not used, or not critical, in the boot path. 
But bugs which break the boot prevent you from testing everything else.

Libguestfs currently is the de-facto test of bugs that break the boot,
and TBH it's not a job I enjoy having.  It happens too often in
Rawhide, and a simple test (in %check or elsewhere) could fix it.  I
even wrote a simple tool to perform the test:

http://people.redhat.com/~rjones/qemu-sanity-check/

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Miloslav Trmač]

2014-02-26 22:53 GMT+01:00 Richard W.M. Jones <rjones(a)redhat.com&gt;:

>  On Wed, Feb 26, 2014 at 04:58:43PM +0100, Miloslav Trmač wrote:
>  > 2014-02-26 14:11 GMT+01:00 Colin Walters <walters(a)verbum.org&gt;:
>  >
>  > > During making glib changes you should run glib unit tests to have some
>  > > basic level of assurance you didn't introduce regressions or unwanted
>  > > changes.
>  > >
>  > > The *very first* test I run is "does the OS still boot"?  That's
> called
>  > > "smoketest" for me, and it only takes a few minutes.
>  > >
>  >
>  > That seems to be optimizing for bugs that break the boot, when bugs that
>  > occur in less-frequently used parts of the system are far more common; a
>  > lot of software is not used, or not critical, in the boot path.
>
>  But bugs which break the boot prevent you from testing everything else.
>  
Only if I would reboot boot my primary workstation into the new untested
software, which I don't do.

Sure, there's the kernel and the graphics drivers and the like which need
physical hardware and booting, but these are the rare exceptions within the
package universe.

As for the case that "I need to reboot so that the gtk2 test suite catches
glib2 bugs", most of these cases are missing tests in glib2.  Yes, the
final reboot and re-running all individual test suites in the new
environment might be useful, if there is time and capacity to do such a
test, but it shouldn't be the primary execution mode of the tests; it's too
costly and unfocused, and too far removed from the programmer in the
think/edit/build/test cycle.
    Mirek

[Richard W.M. Jones]

On Wed, Feb 26, 2014 at 11:12:50PM +0100, Miloslav Trmač wrote:
>  2014-02-26 22:53 GMT+01:00 Richard W.M. Jones
> <rjones(a)redhat.com&gt;:
>  > But bugs which break the boot prevent you from testing everything else.
>  >
>  
>  Only if I would reboot boot my primary workstation into the new untested
>  software, which I don't do.
>  
>  Sure, there's the kernel and the graphics drivers and the like which need
>  physical hardware and booting, but these are the rare exceptions within the
>  package universe.
>  
>  As for the case that "I need to reboot so that the gtk2 test suite catches
>  glib2 bugs", most of these cases are missing tests in glib2.  Yes, the
>  final reboot and re-running all individual test suites in the new
>  environment might be useful, if there is time and capacity to do such a
>  test, but it shouldn't be the primary execution mode of the tests; it's too
>  costly and unfocused, and too far removed from the programmer in the
>  think/edit/build/test cycle. 
I think you may be missing a point here.  We're talking about
automated regression testing.  The entire stack is tested "all-up" (to
use the Apollo phrase), automatically, in a virtual machine.  It
happens without human intervention and (in the os-tree case) in many
thousands of different configurations.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Adam Williamson]

On Wed, 2014-02-26 at 21:53 +0000, Richard W.M. Jones wrote:
>  On Wed, Feb 26, 2014 at 04:58:43PM +0100, Miloslav Trmač wrote:
>  > 2014-02-26 14:11 GMT+01:00 Colin Walters <walters(a)verbum.org&gt;:
>  > 
>  > > During making glib changes you should run glib unit tests to have some
>  > > basic level of assurance you didn't introduce regressions or unwanted
>  > > changes.
>  > >
>  > > The *very first* test I run is "does the OS still boot"?  That's
> called
>  > > "smoketest" for me, and it only takes a few minutes.
>  > >
>  > 
>  > That seems to be optimizing for bugs that break the boot, when bugs that
>  > occur in less-frequently used parts of the system are far more common; a
>  > lot of software is not used, or not critical, in the boot path.
>  
>  But bugs which break the boot prevent you from testing everything else.
>  
>  Libguestfs currently is the de-facto test of bugs that break the boot,
>  and TBH it's not a job I enjoy having. 
Er. There are quite a lot of us running Rawhide, y'know. We tend to
notice when our systems stop booting as well.

>    It happens too often in
>  Rawhide, and a simple test (in %check or elsewhere) could fix it. 
You can't really test a system boot in a package's %check. That's very
definitely not what it's for.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Richard W.M. Jones]

On Wed, Feb 26, 2014 at 06:50:56PM -0800, Adam Williamson wrote:
>  On Wed, 2014-02-26 at 21:53 +0000, Richard W.M. Jones wrote:
>  >   It happens too often in
>  > Rawhide, and a simple test (in %check or elsewhere) could fix it.
>  
>  You can't really test a system boot in a package's %check. That's very
>  definitely not what it's for. 
It turns out you can.  Read the link I posted:

http://people.redhat.com/~rjones/qemu-sanity-check/

Yes, it doesn't catch every possible problem, but the kernel is
sometimes pushed out with major problems that could be caught with
simple testing.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

[Adam Williamson]

On Thu, 2014-02-27 at 08:09 +0000, Richard W.M. Jones wrote:
>  On Wed, Feb 26, 2014 at 06:50:56PM -0800, Adam Williamson wrote:
>  > On Wed, 2014-02-26 at 21:53 +0000, Richard W.M. Jones wrote:
>  > >   It happens too often in
>  > > Rawhide, and a simple test (in %check or elsewhere) could fix it.
>  > 
>  > You can't really test a system boot in a package's %check. That's very
>  > definitely not what it's for.
>  
>  It turns out you can.  Read the link I posted:
>  
>  http://people.redhat.com/~rjones/qemu-sanity-check/ 
I've read it. I'm not exactly convinced.

>  Yes, it doesn't catch every possible problem, but the kernel is
>  sometimes pushed out with major problems that could be caught with
>  simple testing. 
IIRC you had one example for this, and the bug turned out to be racy and
hardware-dependent, which is why the kernel devs didn't see it in the
tests they *do* actually do.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Richard W.M. Jones]

On Thu, Feb 27, 2014 at 12:12:59AM -0800, Adam Williamson wrote:
>  On Thu, 2014-02-27 at 08:09 +0000, Richard W.M. Jones wrote:
>  > On Wed, Feb 26, 2014 at 06:50:56PM -0800, Adam Williamson wrote:
>  > > On Wed, 2014-02-26 at 21:53 +0000, Richard W.M. Jones wrote:
>  > > >   It happens too often in
>  > > > Rawhide, and a simple test (in %check or elsewhere) could fix it.
>  > > 
>  > > You can't really test a system boot in a package's %check. That's
> very
>  > > definitely not what it's for.
>  > 
>  > It turns out you can.  Read the link I posted:
>  > 
>  > http://people.redhat.com/~rjones/qemu-sanity-check/
>  
>  I've read it. I'm not exactly convinced. 
If there is a specific problem, please mention what it is.  Otherwise
there is no action I can take.

>  > Yes, it doesn't catch every possible problem, but the kernel
> is
>  > sometimes pushed out with major problems that could be caught with
>  > simple testing.
>  
>  IIRC you had one example for this, and the bug turned out to be racy and
>  hardware-dependent, which is why the kernel devs didn't see it in the
>  tests they *do* actually do. 
*One*?  There's loads of them.  Regularly the kernel doesn't boot on
qemu, most recently just two days ago (although that "fixed itself"
before I managed to get around to filing a bug).

JUST the bugs in the kernel & qemu filed or fixed in 2013/2014:

https://bugzilla.redhat.com/show_bug.cgi?id=872831
https://bugzilla.redhat.com/show_bug.cgi?id=1033971
https://bugzilla.redhat.com/show_bug.cgi?id=1034433
https://bugzilla.redhat.com/show_bug.cgi?id=998722
https://bugzilla.redhat.com/show_bug.cgi?id=991808
https://bugzilla.redhat.com/show_bug.cgi?id=857026
https://bugzilla.redhat.com/show_bug.cgi?id=962079
https://bugzilla.redhat.com/show_bug.cgi?id=863978

non-x86:
https://bugzilla.redhat.com/show_bug.cgi?id=1066581
https://bugzilla.redhat.com/show_bug.cgi?id=1036742

non-Fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=998065

In one sense none of this is surprising at all.  The kernel
(especially) and qemu are pushed out with next to no testing.  They
are huge complex pieces of software.  The fact they work some of the
time is a miracle really.

And none of this includes the frequent unreproducible heisenbugs that
I see.  I only file bugs about things that are reproducible for
obvious reasons, but if you run the libguestfs test suite you'll see
that it often fails because of problems that are obviously in the
kernel but can't be reproduced when run a second time.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Josh Boyer]

On Thu, Feb 27, 2014 at 4:49 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Thu, Feb 27, 2014 at 12:12:59AM -0800, Adam Williamson wrote:
> > On Thu, 2014-02-27 at 08:09 +0000, Richard W.M. Jones wrote:
> > > On Wed, Feb 26, 2014 at 06:50:56PM -0800, Adam Williamson wrote:
> > > > On Wed, 2014-02-26 at 21:53 +0000, Richard W.M. Jones wrote:
> > > > >   It happens too often in
> > > > > Rawhide, and a simple test (in %check or elsewhere) could fix it.
> > > >
> > > > You can't really test a system boot in a package's %check.
> That's very
> > > > definitely not what it's for.
> > >
> > > It turns out you can.  Read the link I posted:
> > >
> > > http://people.redhat.com/~rjones/qemu-sanity-check/
> >
> > I've read it. I'm not exactly convinced.
>
>  If there is a specific problem, please mention what it is.  Otherwise
>  there is no action I can take.
>
> > > Yes, it doesn't catch every possible problem, but the kernel is
> > > sometimes pushed out with major problems that could be caught with
> > > simple testing.
> >
> > IIRC you had one example for this, and the bug turned out to be racy and
> > hardware-dependent, which is why the kernel devs didn't see it in the
> > tests they *do* actually do.
>
>  *One*?  There's loads of them.  Regularly the kernel doesn't boot on
>  qemu, most recently just two days ago (although that "fixed itself"
>  before I managed to get around to filing a bug).
>
>  JUST the bugs in the kernel & qemu filed or fixed in 2013/2014:
>
>  https://bugzilla.redhat.com/show_bug.cgi?id=872831
>  https://bugzilla.redhat.com/show_bug.cgi?id=1033971
>  https://bugzilla.redhat.com/show_bug.cgi?id=1034433
>  https://bugzilla.redhat.com/show_bug.cgi?id=998722
>  https://bugzilla.redhat.com/show_bug.cgi?id=991808
>  https://bugzilla.redhat.com/show_bug.cgi?id=857026
>  https://bugzilla.redhat.com/show_bug.cgi?id=962079
>  https://bugzilla.redhat.com/show_bug.cgi?id=863978
>
>  non-x86:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1066581
>  https://bugzilla.redhat.com/show_bug.cgi?id=1036742
>
>  non-Fedora:
>  https://bugzilla.redhat.com/show_bug.cgi?id=998065
>
>  In one sense none of this is surprising at all.  The kernel
>  (especially) and qemu are pushed out with next to no testing.  They
>  are huge complex pieces of software.  The fact they work some of the
>  time is a miracle really. 
The kernels that are built are tested at a minimum on 3 machines
before I even put them to git.  After they are built in koji we have
an autotest framework that pulls down the build and runs a set of
tests on them in multiple KVM guests.

I appreciate that you do find kernel bugs and you're very responsible
in reporting and pushing upstream where appropriate.  Thank you for
that.  However, please don't make assumptions and disparage the kernel
maintainers because you found a few bugs using QEMU.

Also, if you'd like to point out the testing aspect and some of the
deficiencies we have around how rawhide works, please make sure you're
speaking about rawhide.  The release update kernels are much more
thoroughly tested because we have a framework that prevents builds
from landing in a repo there.  If we had a manageable way to do that
in rawhide, I'd be happy to use it.

As for running the sanity check in %check, it would bloat the
BuildRequires for the kernel significantly and would increase already
long build times.

josh

[Richard W.M. Jones]

On Thu, Feb 27, 2014 at 08:04:10AM -0500, Josh Boyer wrote:
>  The kernels that are built are tested at a minimum on 3 machines
>  before I even put them to git. 
I didn't realize this happened, so my apologies for making claims
about testing which aren't true.  Is it possible you can boot them in
qemu as part of this process?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Josh Boyer]

On Thu, Feb 27, 2014 at 8:13 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Thu, Feb 27, 2014 at 08:04:10AM -0500, Josh Boyer wrote:
> > The kernels that are built are tested at a minimum on 3 machines
> > before I even put them to git.
>
>  I didn't realize this happened, so my apologies for making claims
>  about testing which aren't true.  Is it possible you can boot them in
>  qemu as part of this process? 
Yes, actually.  I don't want to duplicate everything the KVM autotests
will do, but a quick boot test in a VM on a box I have is not
unreasonable at all.  I'll try and make sure it's done going forward.
If you could also keep doing the tests in libguestfs, that would be
appreciated as well.  I know it's after it's pushed to rawhide, but
I'd rather catch bugs where we can than have that dropped.

josh

[Toshio Kuratomi]

On Feb 26, 2014 5:16 AM, "Colin Walters" <walters(a)verbum.org&gt; wrote:
>  On Wed, Feb 26, 2014 at 8:09 AM, Stanislav Ochotnicky <
sochotnicky(a)redhat.com&gt; wrote:
> >  
> > "I" didn't name
> them. I used standard names for different testing levels 
as defined by software
engineering bodies. Quoting from SWEBOK:
>  Yes, I think they're wrong.   Well, "suboptimal" is a better word.
>
> > During making glib changes you should run glib unit tests to have some 
basic
level of assurance you didn't introduce regressions or unwanted
changes.
>  The *very first* test I run is "does the OS still boot"?  That's called
"smoketest" for me, and it only takes a few minutes.
>  Yes, after every change, even just an updated translation, I boot the OS, 
run
through systemd, gdm, gnome-shelll, and everything and ensure it still
logs in.
>  This is *before* I run the GTK+ tests or the glib tests or any other 
test.  Why? 
Because if that fails, there's *no point* to running the other
tests.  The system is broken.  The originating change is investigated and
is up for reverting.   Whether or not the GKeyFile test pass or not is
irrelevant.
>  (Also, there is the fact that InstalledTests are guaranteed to be run in 
a logged
in desktop, not a mock chroot, so the above needs to work anyways)
> > It's great that your change helped with discovering issues but perhaps 
your
original testsuite was mixing different levels of testing in the same
code. Unit tests are supposed to be quick, dirty solutions using mock
objects and other "hacks" to allow of testing with small granularity.
>  Ah, but if one makes "integration tests" very fast and easy to run as I
have, then there's less need for "quick and dirty".
>  
Integration tests and unittests really have different
purposes.
Integration tests test that things that you want to do aren't broken by a
change.  Unittests test that individual functional units aren't broken by a
change.

For a developer who can run their tests right after each commit there's
likely not a lot of difference.  For someone who's getting a code drop with
hundreds of commits in it, however, it's often easier to debug what's
causing a failing unittest than what's causing a failing integration test.
The reason is that ideally unittests isolate a small piece of code for
testing.  when it fails, the Unittest itself provides the clues to the
person debugging as to where the failure lies.

-Toshio

[Dridi Boukelmoune]

On Tue, Feb 25, 2014 at 2:50 PM, Adam Jackson <ajax(a)redhat.com&gt; wrote:
>  On Tue, 2014-02-25 at 12:45 +0200, Alexander Todorov wrote:
>
> > 1) Do we consider this a bug and if yes what priority do you give it? From last
> > week discussions it looks like most people prefer to have tests executed in %check.
>
>  I don't consider %check to be an appropriate way to run tests, so no, I
>  don't consider it a bug.  Stating it broadly: we don't have a separate
>  phase for it in koji, which means we can't adequately set up tests in
>  general; and adding it to koji would be a mistake, because build
>  construction and build validation are fundamentally different phases. 
I'd rather not discard %check when (blind guess) most of the test
suites fit well there. Unless you're testing the package from the
outside (let's say integration tests), you don't need another hook for
the tests. Even for integration tests %check might still be relevant
if you manage to keep them self-contained.

I'm not saying I'm against a hook for non %check-able test suites :)

Dridi

>  This is an argument against %check, not against testing in general. 
> We
>  should be relying on rpmbuild less, not more.  rpm doesn't even have
>  anything like Requires(check), does it?
>
>  - ajax
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

[Toshio Kuratomi]

On Tue, Feb 25, 2014 at 12:45:11PM +0200, Alexander Todorov wrote:
>  Hi guys,
>  I have identified 551 packages on the Fedora 20 source DVD which are
>  missing a %check section in their spec files but are very likely to
>  have a test suite. See
>
> https://github.com/atodorov/fedora-scripts/blob/master/sample-data/fedora...
>  
>  For a few of them I already had bug open previously (will filter the
>  list later when I run my scripts against latest Rawhide).
>  
>  I have the following questions:
>  
>  1) Do we consider this a bug and if yes what priority do you give it?
>  From last week discussions it looks like most people prefer to have
>  tests executed in %check.
>   
https://fedoraproject.org/wiki/Packaging:Guidelines#Test_Suites

So in some set of cases this would be a bug.  I don't have an estimate of
how many test suites would be "practical to" execute the test suite though.

-Toshio

[David Howells]

Alexander Todorov <atodorov(a)redhat.com&gt; wrote:

>  How about making %check a packaging requirement in all cases - run
> tests or
>  add a comment explaining why not, how to run them (e.g. make test) or why
>  there are no tests for this package. 
Does %check install the package and run the tests as root?  For the keyutils
package that I maintain, this would be a requirement to be able to run the
testsuite because:

 (1) There are certain operations that cannot be tested if you're not root.

 (2) The kernel must be able to upcall to executables installed by the rpm.

I imagine this applies to some other packages also.

David

[Richard W.M. Jones]

On Wed, Feb 26, 2014 at 01:56:22PM +0000, David Howells wrote:
>  Alexander Todorov <atodorov(a)redhat.com&gt; wrote:
>  
>  > How about making %check a packaging requirement in all cases - run tests or
>  > add a comment explaining why not, how to run them (e.g. make test) or why
>  > there are no tests for this package.
>  
>  Does %check install the package and run the tests as root? 
No, of course not.

>  For the keyutils
>  package that I maintain, this would be a requirement to be able to run the
>  testsuite because:
>  
>   (1) There are certain operations that cannot be tested if you're not root.
>  
>   (2) The kernel must be able to upcall to executables installed by the rpm.
>  
>  I imagine this applies to some other packages also. 
You can throw up a VM to test this, if you really wanted to do it in
%check.  libguestfs does this.  However it's not especially easy, and
it's fairly ugly.

For the sorts of tests you are talking about it's much better to test
the final RPM installed in a full OS environment.  That is what (I
hope) Taskotron is trying to do.  Also what Colin is doing with his
ostree testing effort.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Adam Williamson]

On Wed, 2014-02-26 at 21:56 +0000, Richard W.M. Jones wrote:

>  For the sorts of tests you are talking about it's much better to
> test
>  the final RPM installed in a full OS environment.  That is what (I
>  hope) Taskotron is trying to do. 
Well, that's *one* of the things it does, yes (as AutoQA did before it).
Both taskotron and AutoQA were conceived as fairly generic testing
frameworks, though. They aren't intended to be inextricably tied to
distribution packages or koji or bodhi or anything else like that. 

In practice AutoQA turned out to be somewhat coupled at least to some
Fedora-specific infrastructure, and that's one of the reasons behind the
taskotron re-design:

http://tirfa.com/down-with-test-automation-long-live-task-automation.html

So, yeah: running tests on packages in full OS environments is certainly
one of the major things we want to do with Taskotron, but that's not all
it's for. We also want, for instance, to do testing on Fedora
deliverables - one of the things we had working very briefly in AutoQA,
for instance, was automated testing of the daily Rawhide boot.iso
composes. The "RATS" test could run through a minimal install, see if it
worked, and provide somewhat useful information when it didn't. Not
package-level testing at all.

Other references on taskotron's design:
https://fedoraproject.org/wiki/Taskotron ,
https://fedoraproject.org/wiki/User:Tflink/taskotron_development_plan ,
http://tirfa.com/how-is-taskbot-different-from-autoqa.html (when it was
called Taskbot).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Alexander Todorov]

На 26.02.2014 15:56, David Howells написа:
>  Alexander Todorov <atodorov(a)redhat.com&gt; wrote:
>
> > How about making %check a packaging requirement in all cases - run tests or
> > add a comment explaining why not, how to run them (e.g. make test) or why
> > there are no tests for this package.
>
>  Does %check install the package and run the tests as root?  For the keyutils
>  package that I maintain, this would be a requirement to be able to run the
>  testsuite because:
>
>    (1) There are certain operations that cannot be tested if you're not root.
>
>    (2) The kernel must be able to upcall to executables installed by the rpm.
>
>  I imagine this applies to some other packages also.
>  
I does, but not sure to how many.  %check is executed during rpmbuild time right 
after %build so limitations apply.

--
Alex

[Dridi Boukelmoune]

On Thu, Feb 27, 2014 at 11:18 AM, Alexander Todorov <atodorov(a)redhat.com&gt; wrote:
>  На 26.02.2014 15:56, David Howells написа:
>
> > Alexander Todorov <atodorov(a)redhat.com&gt; wrote:
> >
> >> How about making %check a packaging requirement in all cases - run tests
> >> or
> >> add a comment explaining why not, how to run them (e.g. make test) or why
> >> there are no tests for this package.
> >
> >
> > Does %check install the package and run the tests as root?  For the
> > keyutils
> > package that I maintain, this would be a requirement to be able to run the
> > testsuite because:
> >
> >   (1) There are certain operations that cannot be tested if you're not
> > root.
> >
> >   (2) The kernel must be able to upcall to executables installed by the
> > rpm.
> >
> > I imagine this applies to some other packages also.
> >
>
>  I does, but not sure to how many.  %check is executed during rpmbuild time
>  right after %build so limitations apply. 
s/%build/%install/
https://fedoraproject.org/wiki/How_to_create_an_RPM_package#.25check_section

>  --
>  Alex
>
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct