Re: Functional tests as User Stories

[wes hayutin]

On Wed, 2010-03-03 at 11:24 -0400, Devan Goodwin wrote:
> The Java functional tests evolved first, they're kind of a hybrid, no
> deployed Candlepin required, you have access to the database for setup
> purposes, but then (I think) they fire up a servlet container
> internally (Grizzy?) and test against that. Some benefits to this but
> this doesn't catch problems that sometimes surface in Tomcat/JBoss.
>
> Personally I'd prefer we port them to Python suite. (or something else
> should we replace that) I like testing this outside against a full
> environment as we can technically point it anywhere. It will clutter
> up the database (and thus should never be run against an important db)
> and is a little harder to inject setup data (but this basically just
> boils down to uploading a cert right now), IMO we need that kind of
> external testing to have some degree of confidence that this thing is
> working.
>
> I'd vote we boil testing down to just Java unit tests, and Python nosetests.
>
> As for Cucumber it's blowing my mind and I don't yet see how it even
> works. :) Have to dig into it.
>
> Can QA fit either solution (Cucumber/Python tests) into their automated systems?

The quick answer is no, not currently.

The other answer is that I *think* dev and qa will have different goals
and view points when it comes to functional tests. I dont want QA to get
too involved in how dev sets up their tests.  QA works with several
teams and products, trying to get each product or feature to conform to
something QA wants is probably not ideal. However, I do want QA to
contribute to what ever tests development sets up.

The QA team and automation team will be writing api and command line
function tests for candlepin and entitlements starting next sprint.
This effort will probably be separate from development, but does not
have to be.  We've had some initial conversations and we just have one
requirement for our tests, testng. http://testng.org/doc/index.html

That does also limit the tests to be written in jvm languages like java,
jython, groovy etc.  testng is currently a requirements because it
offers us:
1. robust test ordering, skipping etc.
2. the ability to automatically write/update test cases. TCMS api
3 the ability to automatically read bugzilla data.  bugzilla api
4. some very nice logging.

The major disadvantage here is that the tests we write don't make it
back to the community. I understand the community benefits more if we
write tests in the candlepin, cobbler repo's, and hopefully we can
figure that out at some point.  I saw that cukes can use jruby, so maybe
thats an option but I'm not sure.

We'd like to have our entire framework and tests available to the
community, but we've been told not to do that.

I can keep the candlepin team updated regarding automated functional
tests, maybe we can find more ways to integrate work.

Thanks
> Cheers,
>
> Devan
>
> On Wed, Mar 3, 2010 at 11:02 AM, Justin Harris  wrote:
> > I agree with this %100.  In my view I would think that we have two kinds of tests:
> >
> > Unit Tests - Junit
> > Functional Tests - Cucumber?  or python tests or whatever
> >
> > Functional tests test the API via REST.
> > Unit tests test small (class level) functionality
> >
> > What am I missing?  What is the purpose of the Java Functional Tests?
> > Justin
> >
> > ----- "Bryan Kearney"  wrote:
> > > I vote for a minimal number of testy things. RIght now I see:
> > >
> > > Junit
> > > Java Functional Tests
> > > python scripts
> > > nosetests
> > >
> > > if cucumber were to replace one of these.. great!
> > >
> > > -- bk
> > >
> > > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > > I don't know if anyone has heard of Cucumber (http://cukes.info/) -
> > >
> > > its a BDD framework in ruby, but it seems like a nice way to define
> > > specific user stories that are directly executable.  Has anyone played
> > > around with this?  It could be a nice fit for candlepin...
> > > > > Justin
> > > > >
> > > > > candlepin mailing list
> > > > > [e-mail]
> > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > >
> > > > cucumber is HOT (admittedly i haven't played much with it).
> > > > -d
> > > > _______________________________________________
> > > > candlepin mailing list
> > > > [e-mail]
> > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > >
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin