Playing around with Sikuli

[Tim Flink]

I've been wanting to try out Sikuli (http://www.sikuli.org) for a while
now and finally got around to making it build and work on Fedora.

The basic concept is to create GUI test cases based on images instead of
absolute (x,y) coordinates (like selenium), scripted key sequences
(openqa) or using an accessability layer (like dogtail).

I've put together a tarball with source and build scripts if anyone
else wants to try it out:

http://tflink.fedorapeople.org/tools/sikuli/README_build-sikuli.txt
http://tflink.fedorapeople.org/tools/sikuli/fedora-sikuli-20120220.tar.gz

The distribution method is a really dirty hack but I've been able to
build on both F15 and F16 after installing a huge number of -devel
dependencies and building all of Sikuli's direct dependencies from
source. Xpresser (https://wiki.ubuntu.com/Xpresser) might be another
option - it's a re-implementation of the concepts from Sikuli. It seems
to be missing some of Sikuli's features but would likely be quite a bit
easier to build and integrate.

One of the things I have in mind is to attempt automating at least some
of the installation test cases because a lot of them pass most of the
time and they can be a bit mind-numbing to do over and over again. I'm
of the opinion that our human resources could be put to better use than
monkey-button-pushing their way through simple test cases.

A non-trivial amount of work would be needed before we could fully
utilize this (fixing the code so that it is package-able, better
integration with VNC etc.) but I'm wondering if it could be a good
candidate for automating some of the installation test cases.

I've written two test cases for Sikuli thus far: a simple DVD graphical
install and graphical firstboot. I had to workaround some timing issues
due to occasional lag in the VM's VNC interface but they seem to be
working well with F17 Alpha RC2 for now.

F17 DVD Basic Graphical Install:
http://tflink.fedorapeople.org/tools/sikuli/f17-basicGraphicalDVD_sikuli....

F17 Graphical Firstboot:
http://tflink.fedorapeople.org/tools/sikuli/f17-graphicalFirstboot_sikuli...
 - NOTE: This only works some of the time due to rhbz#750527 which was
   closed with no bug comment by the developers. I'm going to bug them
   about this but will likely update the test case to work with
   unpredictable order of the firstboot screens.

To run those test cases, extract them somewhere and point the Sikuli
IDE at the extracted directories.

I'm going to be running my basic graphical install testcase for
different F17 releases to get an initial feel for how fragile the test
cases are and how much effort would be required to maintain these kinds
of test cases over a release.

Anyhow, long email over. Any thoughts on whether this might be worth
investigating farther?

Tim

[Vitezslav Humpa]

Hi Tim,

I know that folks at Spice QE have been playing around with Sikuli.
I can imagine you'd like to use this for automating anaconda tests
especially in virtual environments in which spice folks have been
operating too. If you'd like some tips etc., contacting Lubos Kocman
might be helpful.

--
Vita Humpa
Desktop QE

----- Original Message -----
>  I've been wanting to try out Sikuli (http://www.sikuli.org) for
> a
>  while
>  now and finally got around to making it build and work on Fedora.
>  
>  The basic concept is to create GUI test cases based on images instead
>  of
>  absolute (x,y) coordinates (like selenium), scripted key sequences
>  (openqa) or using an accessability layer (like dogtail).
>  
>  I've put together a tarball with source and build scripts if anyone
>  else wants to try it out:
>  
>  http://tflink.fedorapeople.org/tools/sikuli/README_build-sikuli.txt
>  http://tflink.fedorapeople.org/tools/sikuli/fedora-sikuli-20120220.tar.gz
>  
>  The distribution method is a really dirty hack but I've been able to
>  build on both F15 and F16 after installing a huge number of -devel
>  dependencies and building all of Sikuli's direct dependencies from
>  source. Xpresser (https://wiki.ubuntu.com/Xpresser) might be another
>  option - it's a re-implementation of the concepts from Sikuli. It
>  seems
>  to be missing some of Sikuli's features but would likely be quite a
>  bit
>  easier to build and integrate.
>  
>  One of the things I have in mind is to attempt automating at least
>  some
>  of the installation test cases because a lot of them pass most of the
>  time and they can be a bit mind-numbing to do over and over again.
>  I'm
>  of the opinion that our human resources could be put to better use
>  than
>  monkey-button-pushing their way through simple test cases.
>  
>  A non-trivial amount of work would be needed before we could fully
>  utilize this (fixing the code so that it is package-able, better
>  integration with VNC etc.) but I'm wondering if it could be a good
>  candidate for automating some of the installation test cases.
>  
>  I've written two test cases for Sikuli thus far: a simple DVD
>  graphical
>  install and graphical firstboot. I had to workaround some timing
>  issues
>  due to occasional lag in the VM's VNC interface but they seem to be
>  working well with F17 Alpha RC2 for now.
>  
>  F17 DVD Basic Graphical Install:
>  http://tflink.fedorapeople.org/tools/sikuli/f17-basicGraphicalDVD_sikuli....
>  
>  F17 Graphical Firstboot:
>  http://tflink.fedorapeople.org/tools/sikuli/f17-graphicalFirstboot_sikuli...
>   - NOTE: This only works some of the time due to rhbz#750527 which
>   was
>     closed with no bug comment by the developers. I'm going to bug
>     them
>     about this but will likely update the test case to work with
>     unpredictable order of the firstboot screens.
>  
>  To run those test cases, extract them somewhere and point the Sikuli
>  IDE at the extracted directories.
>  
>  I'm going to be running my basic graphical install testcase for
>  different F17 releases to get an initial feel for how fragile the
>  test
>  cases are and how much effort would be required to maintain these
>  kinds
>  of test cases over a release.
>  
>  Anyhow, long email over. Any thoughts on whether this might be worth
>  investigating farther?
>  
>  Tim
>  
>  _______________________________________________
>  autoqa-devel mailing list
>  autoqa-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/autoqa-devel
>

[Tim Flink]

On Tue, 21 Feb 2012 05:00:53 -0500 (EST)
Vitezslav Humpa <vhumpa(a)redhat.com&gt; wrote:

>  Hi Tim,
>  
>  I know that folks at Spice QE have been playing around with Sikuli.
>  I can imagine you'd like to use this for automating anaconda tests
>  especially in virtual environments in which spice folks have been
>  operating too. If you'd like some tips etc., contacting Lubos Kocman
>  might be helpful. 
Good to know - adding the ability to connect to a VM over VNC or Spice
was one of the things that I wanted to look into.

Thanks for the tip, I'll make sure to contact Lubos.

Tim

[Ryan]

Thank you for the package and instructions. 

After everything builds I am left with two files in 
the folder where you need to run the shell script from. (forms-1.2.1.jar & 
sikuli-ide.sh)
When I run the "sh sikuli-ide.sh", I get this error: "Unable to access
jarfile 
./sikuli-ide.jar"

Any ideas why?
(I am running Fedora 15 (64bit))

Thanks!

[Tim Flink]

On Tue, 29 May 2012 21:44:45 +0000 (UTC)
Ryan <rhnrksn(a)gmail.com&gt; wrote:

>  Thank you for the package and instructions. 
>  
>  After everything builds I am left with two files in 
>  the folder where you need to run the shell script from.
>  (forms-1.2.1.jar & sikuli-ide.sh)
>  When I run the "sh sikuli-ide.sh", I get this error: "Unable to
>  access jarfile ./sikuli-ide.jar"
>  
>  Any ideas why?
>  (I am running Fedora 15 (64bit)) 
Off the top of my head, no but I don't think that I tried building it
on F15.

Do the jar files exist? Were there any errors when building?

Tim