Eclipse UI Tests

[Andrew Overholt]

Hi,

I was talking with Billy last night about Eclipse unit tests and I ran the
UI ones this morning.  There are 806 tests and we get 152 Errors and 40
Failures.  144 of the 152 Errors are from
org.eclipse.ui.tests.dnd.DragTestSuite in which Billy said to expect errors
(since this is 3.1M5a I'm testing).

I'd like to set things up to run these automatically (Aaron is looking into
it) and get some hard data that we can work with.

The one thing that bothered me is that the tests took about 43 minutes to
run on my P4 3 GHz with 1 GB RAM.  Billy says the tests take only about 10
minutes normally.  I'll try to spend some time figuring out where the time
goes.  This could have been my machine/setup but they took about the same
amount of time to run on my laptop which is a much less beefier machine so
I sort of doubt it.

This is how Billy taught me to run the tests (for future reference):

. start Eclipse with a new workspace
  eclipse -data testworkspace
. switch to CVS perspective and add the Eclipse repo
  (dev.eclipse.org - /home/eclipse - anonymous - pserver)
. expand the Eclipse repo and expand versions
. expand org.eclipse.releng
. expand I20050219-1500 (the tag for 3.1M5a)
. expand maps
. double-click on ui.map
. note that plugin(a)org.eclipse.ui.tests=v20050218-1200
. close ui.map and expand org.eclipse.ui.tests
. check out org.eclipse.ui.tests v20050218-1200
. switch to the Java perspective
. once your workspace finishes building, there will be errors in a few
  things ... we only care about the errors in plugin.xml so edit it to
  comment out or remove the two offending lines
. expand Eclipse UI Tests -> org.eclipse.ui.tests
. right-click on UiTestSuite.java and click Run As -> JUnit Plug-in Test
. beside Package Explorer and Hierarchy, you should get a JUnit tab;
  quickly select it and select the Hierarchy tab within before the tests
  start up
. let the tests run without playing with your computer because some of the
  tests rely on focus

Andrew

[Aaron Luchko]

On Tue, 2005-04-05 at 10:55 -0400, Andrew Overholt wrote:
>  Hi,
>  
>  I was talking with Billy last night about Eclipse unit tests and I ran the
>  UI ones this morning.  There are 806 tests and we get 152 Errors and 40
>  Failures.  144 of the 152 Errors are from
>  org.eclipse.ui.tests.dnd.DragTestSuite in which Billy said to expect errors
>  (since this is 3.1M5a I'm testing).
>  
>  I'd like to set things up to run these automatically (Aaron is looking into
>  it) and get some hard data that we can work with. 
I've been working on running them with a headless eclipse as outlined
here,
http://download.eclipse.org/eclipse/downloads/drops/S-3.1M5a-200502191500...
and using 3.0 have been able to run them however using 3.1 M5 I've been
unable to successfully run them due to a number of issues.

First the instructions on the page appear to be incomplete, first the
runtests script requires a directory
test-eclipse/eclipse
which is not created by the script, as well the line 
unzip -qq -o -C eclipse-junit-tests-*.zip */plugins/org.eclipse.test*
does not fully expand the tests.

To run them successfully on 3.0.1 I run 
unzip eclipse-SDK-*.zip
unzip eclipse-junit-*.zip
mkdir test-eclipse
cp -r eclipse test-eclipse/   
./runtests -os linux -ws gtk -arch x86 -vm /usr/lib/jvm/java-1.4.2-
ibm-1.4.2.1//jre/bin/java -noclean

which works, however of M5 using the identical steps (with zips changed
to tar.gz) I get

Buildfile: test.xml

all:

jdttext:

runtests:

setup:

junit:

init:

BUILD FAILED
/notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
testing/test.xml:299: The following error occurred while executing this
line:
/notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
testing/test.xml:155: The following error occurred while executing this
line:
/notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
testing/test.xml:84: The following error occurred while executing this
line:
/notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
testing/test.xml:65: The following error occurred while executing this
line:
/notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
testing/test-
eclipse/eclipse/plugins/org.eclipse.jdt.text.tests_3.1.0/test.xml:17:
org.eclipse.ant.core.AntSecurityException

Total time: 1 second


Aaron

[Benjamin Konrath]

On Tue, 2005-04-05 at 13:54 -0400, Aaron Luchko wrote:
>  /notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
>  testing/test-
>  eclipse/eclipse/plugins/org.eclipse.jdt.text.tests_3.1.0/test.xml:17:
>  org.eclipse.ant.core.AntSecurityException 
I think this is bug in 3.1M5a. I encountered a similar problem when I
was generating the tarballs for the CDT. Maybe you should try it again
with 3.1M6 and see how it goes.

Cheers, Ben

[Aaron Luchko]

On Tue, 2005-04-05 at 14:02 -0400, Ben Konrath wrote:
>  On Tue, 2005-04-05 at 13:54 -0400, Aaron Luchko wrote:
>  > /notnfs/eclipse-tests/nightlies/eclipse-tests-devel/eclipse-
>  > testing/test-
>  > eclipse/eclipse/plugins/org.eclipse.jdt.text.tests_3.1.0/test.xml:17:
>  > org.eclipse.ant.core.AntSecurityException
>  
>  I think this is bug in 3.1M5a. I encountered a similar problem when I
>  was generating the tarballs for the CDT. Maybe you should try it again
>  with 3.1M6 and see how it goes.
>   
Yup that fixes it, thanks for the tip, I was hoping something would be
fixed in M6 that would allow them to work but hadn't yet got around to
testing it.

Well fixes it as far as here,

BUILD FAILED
/notnfs/eclipse-tests/temp/eclipse-testing/test.xml:300: The following
error occurred while executing this line:
/notnfs/eclipse-tests/temp/eclipse-testing/test.xml:161: The following
error occurred while executing this line:
/notnfs/eclipse-tests/temp/eclipse-testing/test.xml:84: The following
error occurred while executing this line:
/notnfs/eclipse-tests/temp/eclipse-testing/test.xml:65: The following
error occurred while executing this line:
/notnfs/eclipse-tests/temp/eclipse-testing/test-
eclipse/eclipse/plugins/org.eclipse.releng.tests_3.1.0/test.xml:31:
Directory /builds/t creation was not successful for an unknown reason

these tests didn't work in M5 either, but removing the relEng test
plugin and target in the test.xml fixes it, unfortunately this means
that we won't be able to do automated tests until we change the rpms to
M6.

Aaron

[Phil Muldoon]

>  these tests didn't work in M5 either, but removing the relEng
> test
>  plugin and target in the test.xml fixes it, unfortunately this means
>  that we won't be able to do automated tests until we change the rpms to
>  M6. 
The Eclipse CDT is the blocker on M6 at the moment as Eclipse 3.0 CDT M5
will flat not work with Eclipse 3.1 M6. So with that in mind, I thought
it best to hold off an additional week to to let the Eclipse CDT 3.0 M6
build be published, as the schedules are so close. This will ensure that
our CDT users are not left stranded. One week, one way or the other, is
not too much time. So soon, very soon... ;)

Regards

Phil Muldoon

[Phil Muldoon]

>   So with that in mind, I thought
>  it best to hold off an additional week to to let the Eclipse CDT 3.0 M6
>  build be published, as the schedules are so close. This will ensure that
>  our CDT users are not left stranded. One week, one way or the other, is
>  not too much time. So soon, very soon... ;) 
There is a candidate integration build that the Eclipse CDT team has
released for M6. As yet it is untagged in the eclipse repository, so I
built a test CDT M6 build and put the rpm/srpm on:

http://people.redhat.com/pmuldoon/cdt

I did a quick smoke test, and it appears to work with Eclipse M6 ok.
Feedback appreciated.

Regards

Phil