Running a test from dist-git

[Richard W.M. Jones]

As directed by:

  https://fedoraproject.org/wiki/CI/Tests

I tried to add a test to libguestfs dist-git:

  https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/tests/tests.yml

It's very simple initially.  It just needs to run
‘libguestfs-test-tool’ (a program installed by the libguestfs package
which does some self-tests).  Ideally as non-root, but I think I've
set it up to run as root to start with.

I did a scratch build initially, but it seems like the tests only run
from what's been pushed to dist-git.  In any case the test didn't look
as if it had run.  So I pushed the test to dist-git -- see above --
and did another scratch build.  However I've still no real idea if the
test really ran.

I got an email containing this link:

 
https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...

which looks like there are lots of failures, but it's also not clear
if the test ran / tried to run / didn't run / if this link has nothing
to do tests.

So, that was the end of my experiment.  Not a very good result :-(

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Miro Hrončok]

On 27.9.2018 23:30, Richard W.M. Jones wrote:
>  As directed by:
>  
>     https://fedoraproject.org/wiki/CI/Tests
>  
>  I tried to add a test to libguestfs dist-git:
>  
>     https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/tests/tests.yml
>  
>  It's very simple initially.  It just needs to run
>  ‘libguestfs-test-tool’ (a program installed by the libguestfs package
>  which does some self-tests).  Ideally as non-root, but I think I've
>  set it up to run as root to start with.
>  
>  I did a scratch build initially, but it seems like the tests only run
>  from what's been pushed to dist-git.  In any case the test didn't look
>  as if it had run.  So I pushed the test to dist-git -- see above --
>  and did another scratch build.  However I've still no real idea if the
>  test really ran.
>  
>  I got an email containing this link:
>  
>    
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
>  
>  which looks like there are lots of failures, but it's also not clear
>  if the test ran / tried to run / didn't run / if this link has nothing
>  to do tests.
>  
>  So, that was the end of my experiment.  Not a very good result :-( 
I had very similar experience.

One thing that's useful is that the test run if you create a src.fp.o 
Pull Request.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miroslav Vadkerti]

Hi,

On Thu, Sep 27, 2018 at 11:32 PM Richard W.M. Jones <rjones(a)redhat.com&gt;
wrote:

>  As directed by:
>
>    https://fedoraproject.org/wiki/CI/Tests
>
>  I tried to add a test to libguestfs dist-git:
>
>
>  https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/tests/tests.yml
>
>  It's very simple initially.  It just needs to run
>  ‘libguestfs-test-tool’ (a program installed by the libguestfs package
>  which does some self-tests).  Ideally as non-root, but I think I've
>  set it up to run as root to start with.
>  
Cool :)



>  I did a scratch build initially, but it seems like the tests only run
>  from what's been pushed to dist-git. 

That is correct. Tests are not bundled into the src.rpm.


>    In any case the test didn't look
>  as if it had run.  So I pushed the test to dist-git -- see above --
>  and did another scratch build.  However I've still no real idea if the
>  test really ran.
>
>  I got an email containing this link:
>
>
>
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...

Nice, seems fedora notifications works again :)


>  which looks like there are lots of failures, but it's also not clear
>  if the test ran / tried to run / didn't run / if this link has nothing
>  to do tests.
>  
Yeah, sorry for that, must have been som outage. I agree seeing that is not
very helpful, to somebody
not experienced in the CI pipeline. Also rescheduling anything else as a
pull-request testing is
not easily possible currently :( In the future this should be possible via
fedpkg.


>  So, that was the end of my experiment.  Not a very good result :-(
>  
For this time I rescheduled the run:


https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...

Seems it will end up with something more meaningful this time ...

HTH,
/M



>  Rich.
>
>  --
>  Richard Jones, Virtualization Group, Red Hat
>  http://people.redhat.com/~rjones
>  Read my programming and virtualization blog: http://rwmj.wordpress.com
>  libguestfs lets you edit virtual machines.  Supports shell scripting,
>  bindings from many languages.  http://libguestfs.org
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Miroslav Vadkerti :: Principal QE :: BaseOS QE Security
IRC mvadkert #osci #baseosci #qe :: GPG 0x7B5B2E95
Desk Phone [phone] :: Mobile [phone]
Red Hat Czech s.r.o, Purkyňova 99/71, 612 00, Brno, CR

[Richard W.M. Jones]

On Mon, Oct 01, 2018 at 11:26:59AM +0200, Miroslav Vadkerti wrote:
>  For this time I rescheduled the run:
>  
>  
>
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
It appears to have failed, but I've got no clue why.  How do I see
the output of the command?

I think it would be easier if we could just drop a shell script into
tests/ and have it run that and display the output.  The current
system seems elaborately overengineered and I don't understand why.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Al Stone]

On 10/2/18 1:40 AM, Richard W.M. Jones wrote:
>  On Mon, Oct 01, 2018 at 11:26:59AM +0200, Miroslav Vadkerti wrote:
> > For this time I rescheduled the run:
> >
> >
> >
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
>  
>  It appears to have failed, but I've got no clue why.  How do I see
>  the output of the command?
>  
>  I think it would be easier if we could just drop a shell script into
>  tests/ and have it run that and display the output.  The current
>  system seems elaborately overengineered and I don't understand why.
>  
>  Rich.
>   
Hear, hear. + a dozen or so.

I've been trying to add testing shell scripts I already have that work
properly from the command line, and are even part of the RPM build, so
thought it would be nice to convert them to this new automated system.
So far, it just makes my brain hurt.

-- 
ciao,
al
-----------------------------------
Al Stone
Software Engineer
Red Hat, Inc.
ahs3(a)redhat.com
-----------------------------------

[Miroslav Vadkerti]

Hi,

sorry for the late response :(

On Tue, Oct 2, 2018 at 9:42 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:

>  On Mon, Oct 01, 2018 at 11:26:59AM +0200, Miroslav Vadkerti wrote:
>  > For this time I rescheduled the run:
>  >
>  >
>  >
>
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
>
>  It appears to have failed, but I've got no clue why.  How do I see
>  the output of the command?
>  
Ugh, logs are now gone. Which package were you testing? We can take a look
what was the problem.


>  I think it would be easier if we could just drop a shell script into
>  tests/ and have it run that and display the output.  The current
>  system seems elaborately overengineered and I don't understand why.
>  
Well, it was decided that we will use Ansible to wrap around our tests. Not
my idea. I agree the results could be presented better though, even if the
runner is Ansbile. We are preparing a dashboard to view the result of
testing, to give a better user experience in the future.

If the pipeline goes through (i.e. tests were run), the pipeline ends up
with Pass (test passed) or Unstable state (tests failed). If the test
failed, the pipeline did not execute the testing. In the former case you
should find logs in the Artifacts tab in Jenkins (test results are prefixed
with PASS_ or FAIL_). If you encounter an error, we are here to help an
investigate.

HTH,
/M



>  Rich.
>
>  --
>  Richard Jones, Virtualization Group, Red Hat
>  http://people.redhat.com/~rjones
>  Read my programming and virtualization blog: http://rwmj.wordpress.com
>  virt-top is 'top' for virtual machines.  Tiny program with many
>  powerful monitoring features, net stats, disk stats, logging, etc.
>  http://people.redhat.com/~rjones/virt-top
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 
Miroslav Vadkerti :: Principal QE :: BaseOS QE Security
IRC mvadkert #osci #baseosci #qe :: GPG 0x7B5B2E95
Desk Phone [phone] :: Mobile [phone]
Red Hat Czech s.r.o, Purkyňova 99/71, 612 00, Brno, CR

[Richard W.M. Jones]

On Wed, Oct 10, 2018 at 11:26:01PM +0200, Miroslav Vadkerti wrote:
>  Hi,
>  
>  sorry for the late response :(
>  
>  On Tue, Oct 2, 2018 at 9:42 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  
>  > On Mon, Oct 01, 2018 at 11:26:59AM +0200, Miroslav Vadkerti wrote:
>  > > For this time I rescheduled the run:
>  > >
>  > >
>  > >
>  >
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
>  >
>  > It appears to have failed, but I've got no clue why.  How do I see
>  > the output of the command?
>  >
>  
>  Ugh, logs are now gone. Which package were you testing? We can take a look
>  what was the problem. 
https://src.fedoraproject.org/rpms/libguestfs

The test is to run a single command (libguestfs-test-tool).  This
command is already installed by the package, so there's no need for
any test harness or script.  If the command fails (ie. exit code != 0)
then it's vital we see the complete output in order to diagnose the
problem.

https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/tests/tests.yml

Ideally we'd want to run the test as root and non-root, but for now I
just had it run the test as root.

>  > I think it would be easier if we could just drop a shell script
> into
>  > tests/ and have it run that and display the output.  The current
>  > system seems elaborately overengineered and I don't understand why.
>  >
>  
>  Well, it was decided that we will use Ansible to wrap around our tests. Not
>  my idea. I agree the results could be presented better though, even if the
>  runner is Ansbile. We are preparing a dashboard to view the result of
>  testing, to give a better user experience in the future. 
It wasn't very obvious to start off with, but it seems as if the
"standard test roles" are designed for the simple case of "just run a
script".  However the use of YAML combined with the fact that you
can't easily run a test locally makes the whole thing very awkward.
(Also for some reason I didn't investigate yet, my copy of emacs
appears to hate YAML.)

>  If the pipeline goes through (i.e. tests were run), the pipeline ends
> up
>  with Pass (test passed) or Unstable state (tests failed). If the test
>  failed, the pipeline did not execute the testing. In the former case you
>  should find logs in the Artifacts tab in Jenkins (test results are prefixed
>  with PASS_ or FAIL_). If you encounter an error, we are here to help an
>  investigate. 
Do the tests run if we submit scratch builds?

Anyway feel free to submit a new build of libguestfs if you want to
see if you can make the test run.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Petr Šplíchal]

On Thu, 11 Oct 2018 at 09:49, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Wed, Oct 10, 2018 at 11:26:01PM +0200, Miroslav Vadkerti wrote:
>  > Hi,
>  >
>  > sorry for the late response :(
>  >
>  > On Tue, Oct 2, 2018 at 9:42 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  >
>  > > On Mon, Oct 01, 2018 at 11:26:59AM +0200, Miroslav Vadkerti wrote:
>  > > > For this time I rescheduled the run:
>  > > >
>  > > >
>  > > >
>  > >
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
>  > >
>  > > It appears to have failed, but I've got no clue why.  How do I see
>  > > the output of the command?
>  > >
>  >
>  > Ugh, logs are now gone. Which package were you testing? We can take a look
>  > what was the problem.
>
>  https://src.fedoraproject.org/rpms/libguestfs
>
>  The test is to run a single command (libguestfs-test-tool).  This
>  command is already installed by the package, so there's no need for
>  any test harness or script.  If the command fails (ie. exit code != 0)
>  then it's vital we see the complete output in order to diagnose the
>  problem.
>
>  https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/tests/tests.yml 
I've created pull request with fixed tests.yml file:

    https://src.fedoraproject.org/rpms/libguestfs/pull-request/4

In the pull request you would directly see link to the test
results including the complete output of the test. Unfortunately
there is currently issue with propagating results to Pagure:

    https://pagure.io/fedora-ci/general/issue/10

We are currently looking into this, it should be fixed soon.
Meanwhile it's still possible to see results of the pipeline here:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
https://fedoraproject.org/wiki/CI/Pipeline#Instances

The latest libguestfs job is this:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...

Test output can be found under that "Artifact" tab:

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr...

Hope this helps.

psss...

>  Ideally we'd want to run the test as root and non-root, but for
> now I
>  just had it run the test as root.
>
>  > > I think it would be easier if we could just drop a shell script into
>  > > tests/ and have it run that and display the output.  The current
>  > > system seems elaborately overengineered and I don't understand why.
>  > >
>  >
>  > Well, it was decided that we will use Ansible to wrap around our tests. Not
>  > my idea. I agree the results could be presented better though, even if the
>  > runner is Ansbile. We are preparing a dashboard to view the result of
>  > testing, to give a better user experience in the future.
>
>  It wasn't very obvious to start off with, but it seems as if the
>  "standard test roles" are designed for the simple case of "just run a
>  script".  However the use of YAML combined with the fact that you
>  can't easily run a test locally makes the whole thing very awkward.
>  (Also for some reason I didn't investigate yet, my copy of emacs
>  appears to hate YAML.)
>
>  > If the pipeline goes through (i.e. tests were run), the pipeline ends up
>  > with Pass (test passed) or Unstable state (tests failed). If the test
>  > failed, the pipeline did not execute the testing. In the former case you
>  > should find logs in the Artifacts tab in Jenkins (test results are prefixed
>  > with PASS_ or FAIL_). If you encounter an error, we are here to help an
>  > investigate.
>
>  Do the tests run if we submit scratch builds? 
AFAIK, testing on scratch builds is not currently enabled.

>
> Anyway feel free to submit a new build of libguestfs if you want to
> see if you can make the test run.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]