Functional tests as User Stories

[Justin Harris]

I don't know if anyone has heard of Cucumber (http://cukes.info/) - its a BDD framework in ruby, but it seems like a nice way to define specific user stories that are directly executable.  Has anyone played around with this?  It could be a nice fit for candlepin...

- Justin

[Dmitri Dolguikh]

On 03/03/2010 10:50 AM, Justin Harris wrote:
> I don't know if anyone has heard of Cucumber (http://cukes.info/) - its a BDD framework in ruby, but it seems like a nice way to define specific user stories that are directly executable.  Has anyone played around with this?  It could be a nice fit for candlepin...
> Justin
>
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

cucumber is HOT (admittedly i haven't played much with it).
-d

[Bryan Kearney]

I vote for a minimal number of testy things. RIght now I see:

Junit
Java Functional Tests
python scripts
nosetests

if cucumber were to replace one of these.. great!

-- bk

On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> On 03/03/2010 10:50 AM, Justin Harris wrote:
> > I don't know if anyone has heard of Cucumber (http://cukes.info/) - its a BDD framework in ruby, but it seems like a nice way to define specific user stories that are directly executable.  Has anyone played around with this?  It could be a nice fit for candlepin...
> > Justin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin
>
> cucumber is HOT (admittedly i haven't played much with it).
> -d
> _______________________________________________
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

[Justin Harris]

I agree with this %100.  In my view I would think that we have two kinds of tests:

Unit Tests - Junit
Functional Tests - Cucumber?  or python tests or whatever

Functional tests test the API via REST.
Unit tests test small (class level) functionality

What am I missing?  What is the purpose of the Java Functional Tests?

- Justin

----- "Bryan Kearney"  wrote:
> I vote for a minimal number of testy things. RIght now I see:
>
> Junit
> Java Functional Tests
> python scripts
> nosetests
>
> if cucumber were to replace one of these.. great!
>
> -- bk
>
> On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > I don't know if anyone has heard of Cucumber (http://cukes.info/) -
>
> its a BDD framework in ruby, but it seems like a nice way to define
> specific user stories that are directly executable.  Has anyone played
> around with this?  It could be a nice fit for candlepin...
> > > Justin
> > >
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > cucumber is HOT (admittedly i haven't played much with it).
> > -d
> > _______________________________________________
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin
>
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

[Devan Goodwin]

The Java functional tests evolved first, they're kind of a hybrid, no
deployed Candlepin required, you have access to the database for setup
purposes, but then (I think) they fire up a servlet container
internally (Grizzy?) and test against that. Some benefits to this but
this doesn't catch problems that sometimes surface in Tomcat/JBoss.

Personally I'd prefer we port them to Python suite. (or something else
should we replace that) I like testing this outside against a full
environment as we can technically point it anywhere. It will clutter
up the database (and thus should never be run against an important db)
and is a little harder to inject setup data (but this basically just
boils down to uploading a cert right now), IMO we need that kind of
external testing to have some degree of confidence that this thing is
working.

I'd vote we boil testing down to just Java unit tests, and Python nosetests.

As for Cucumber it's blowing my mind and I don't yet see how it even
works. :) Have to dig into it.

Can QA fit either solution (Cucumber/Python tests) into their automated systems?

Cheers,

Devan

On Wed, Mar 3, 2010 at 11:02 AM, Justin Harris  wrote:
> I agree with this %100.  In my view I would think that we have two kinds of tests:
>
> Unit Tests - Junit
> Functional Tests - Cucumber?  or python tests or whatever
>
> Functional tests test the API via REST.
> Unit tests test small (class level) functionality
>
> What am I missing?  What is the purpose of the Java Functional Tests?
>
> - Justin
>
> ----- "Bryan Kearney"  wrote:
> > I vote for a minimal number of testy things. RIght now I see:
> >
> > Junit
> > Java Functional Tests
> > python scripts
> > nosetests
> >
> > if cucumber were to replace one of these.. great!
> >
> > -- bk
> >
> > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > I don't know if anyone has heard of Cucumber (http://cukes.info/) -
> >
> > its a BDD framework in ruby, but it seems like a nice way to define
> > specific user stories that are directly executable.  Has anyone played
> > around with this?  It could be a nice fit for candlepin...
> > > > - Justin
> > > > _______________________________________________
> > > > candlepin mailing list
> > > > [e-mail]
> > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > >
> > > cucumber is HOT (admittedly i haven't played much with it).
> > > -d
> > > _______________________________________________
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin
>
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

[Jesus M. Rodriguez]

We're not checking functionaltests for checkstyle, and I have no idea who's
running them nowadays, can I just remove them all? Or do we need them
around until we write equivalent nosetests?

jesus

[wes hayutin]

On Wed, 2010-03-03 at 11:24 -0400, Devan Goodwin wrote:
> The Java functional tests evolved first, they're kind of a hybrid, no
> deployed Candlepin required, you have access to the database for setup
> purposes, but then (I think) they fire up a servlet container
> internally (Grizzy?) and test against that. Some benefits to this but
> this doesn't catch problems that sometimes surface in Tomcat/JBoss.
>
> Personally I'd prefer we port them to Python suite. (or something else
> should we replace that) I like testing this outside against a full
> environment as we can technically point it anywhere. It will clutter
> up the database (and thus should never be run against an important db)
> and is a little harder to inject setup data (but this basically just
> boils down to uploading a cert right now), IMO we need that kind of
> external testing to have some degree of confidence that this thing is
> working.
>
> I'd vote we boil testing down to just Java unit tests, and Python nosetests.
>
> As for Cucumber it's blowing my mind and I don't yet see how it even
> works. :) Have to dig into it.
>
> Can QA fit either solution (Cucumber/Python tests) into their automated systems?

The quick answer is no, not currently.

The other answer is that I *think* dev and qa will have different goals
and view points when it comes to functional tests. I dont want QA to get
too involved in how dev sets up their tests.  QA works with several
teams and products, trying to get each product or feature to conform to
something QA wants is probably not ideal. However, I do want QA to
contribute to what ever tests development sets up.

The QA team and automation team will be writing api and command line
function tests for candlepin and entitlements starting next sprint.
This effort will probably be separate from development, but does not
have to be.  We've had some initial conversations and we just have one
requirement for our tests, testng. http://testng.org/doc/index.html

That does also limit the tests to be written in jvm languages like java,
jython, groovy etc.  testng is currently a requirements because it
offers us:
1. robust test ordering, skipping etc.
2. the ability to automatically write/update test cases. TCMS api
3 the ability to automatically read bugzilla data.  bugzilla api
4. some very nice logging.

The major disadvantage here is that the tests we write don't make it
back to the community. I understand the community benefits more if we
write tests in the candlepin, cobbler repo's, and hopefully we can
figure that out at some point.  I saw that cukes can use jruby, so maybe
thats an option but I'm not sure.

We'd like to have our entire framework and tests available to the
community, but we've been told not to do that.

I can keep the candlepin team updated regarding automated functional
tests, maybe we can find more ways to integrate work.

Thanks
> Cheers,
>
> Devan
>
> On Wed, Mar 3, 2010 at 11:02 AM, Justin Harris  wrote:
> > I agree with this %100.  In my view I would think that we have two kinds of tests:
> >
> > Unit Tests - Junit
> > Functional Tests - Cucumber?  or python tests or whatever
> >
> > Functional tests test the API via REST.
> > Unit tests test small (class level) functionality
> >
> > What am I missing?  What is the purpose of the Java Functional Tests?
> > Justin
> >
> > ----- "Bryan Kearney"  wrote:
> > > I vote for a minimal number of testy things. RIght now I see:
> > >
> > > Junit
> > > Java Functional Tests
> > > python scripts
> > > nosetests
> > >
> > > if cucumber were to replace one of these.. great!
> > >
> > > -- bk
> > >
> > > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > > I don't know if anyone has heard of Cucumber (http://cukes.info/) -
> > >
> > > its a BDD framework in ruby, but it seems like a nice way to define
> > > specific user stories that are directly executable.  Has anyone played
> > > around with this?  It could be a nice fit for candlepin...
> > > > > Justin
> > > > >
> > > > > candlepin mailing list
> > > > > [e-mail]
> > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > >
> > > > cucumber is HOT (admittedly i haven't played much with it).
> > > > -d
> > > > _______________________________________________
> > > > candlepin mailing list
> > > > [e-mail]
> > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > >
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin

[Bryan Kearney]

What was the outcome of this? Do we have a plan to solidify / modify / 
or any other "fy" the various test bits?

-- bk

On 03/03/2010 10:24 AM, Devan Goodwin wrote:
> The Java functional tests evolved first, they're kind of a hybrid, no
> deployed Candlepin required, you have access to the database for setup
> purposes, but then (I think) they fire up a servlet container
> internally (Grizzy?) and test against that. Some benefits to this but
> this doesn't catch problems that sometimes surface in Tomcat/JBoss.
>
> Personally I'd prefer we port them to Python suite. (or something else
> should we replace that) I like testing this outside against a full
> environment as we can technically point it anywhere. It will clutter
> up the database (and thus should never be run against an important db)
> and is a little harder to inject setup data (but this basically just
> boils down to uploading a cert right now), IMO we need that kind of
> external testing to have some degree of confidence that this thing is
> working.
>
> I'd vote we boil testing down to just Java unit tests, and Python nosetests.
>
> As for Cucumber it's blowing my mind and I don't yet see how it even
> works. :) Have to dig into it.
>
> Can QA fit either solution (Cucumber/Python tests) into their automated systems?
>
> Cheers,
>
> Devan
>
> On Wed, Mar 3, 2010 at 11:02 AM, Justin Harris  wrote:
> > I agree with this %100.  In my view I would think that we have two kinds of tests:
> >
> > Unit Tests - Junit
> > Functional Tests - Cucumber?  or python tests or whatever
> >
> > Functional tests test the API via REST.
> > Unit tests test small (class level) functionality
> >
> > What am I missing?  What is the purpose of the Java Functional Tests?
> > Justin
> >
> > ----- "Bryan Kearney"  wrote:
> > > I vote for a minimal number of testy things. RIght now I see:
> > >
> > > Junit
> > > Java Functional Tests
> > > python scripts
> > > nosetests
> > >
> > > if cucumber were to replace one of these.. great!
> > >
> > > -- bk
> > >
> > > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > > I don't know if anyone has heard of Cucumber (http://cukes.info/) -
> > >
> > > its a BDD framework in ruby, but it seems like a nice way to define
> > > specific user stories that are directly executable.  Has anyone played
> > > around with this?  It could be a nice fit for candlepin...
> > > > > - Justin
> > > > >
> > > > > candlepin mailing list
> > > > > [e-mail]
> > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > >
> > > > cucumber is HOT (admittedly i haven't played much with it).
> > > > -d
> > > > _______________________________________________
> > > > candlepin mailing list
> > > > [e-mail]
> > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > >
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin

[Justin Harris]

Personally, I would like to give Cucumber an evaluation to see if it could fit in well with buildr plus our extreme agile development. :)  I'm happy to let it fall by the wayside if folks are opposed to it, however.

- Justin

----- "Bryan Kearney"  wrote:
> What was the outcome of this? Do we have a plan to solidify / modify /
>
> or any other "fy" the various test bits?
>
> -- bk
>
> On 03/03/2010 10:24 AM, Devan Goodwin wrote:
> > The Java functional tests evolved first, they're kind of a hybrid,
>
> no
> > deployed Candlepin required, you have access to the database for
>
> setup
> > purposes, but then (I think) they fire up a servlet container
> > internally (Grizzy?) and test against that. Some benefits to this
>
> but
> > this doesn't catch problems that sometimes surface in Tomcat/JBoss.
> >
> > Personally I'd prefer we port them to Python suite. (or something
>
> else
> > should we replace that) I like testing this outside against a full
> > environment as we can technically point it anywhere. It will
>
> clutter
> > up the database (and thus should never be run against an important
>
> db)
> > and is a little harder to inject setup data (but this basically
>
> just
> > boils down to uploading a cert right now), IMO we need that kind of
> > external testing to have some degree of confidence that this thing
>
> is
> > working.
> >
> > I'd vote we boil testing down to just Java unit tests, and Python
>
> nosetests.
> > As for Cucumber it's blowing my mind and I don't yet see how it
>
> even
> > works. :) Have to dig into it.
> >
> > Can QA fit either solution (Cucumber/Python tests) into their
>
> automated systems?
> > Cheers,
> >
> > Devan
> >
> > On Wed, Mar 3, 2010 at 11:02 AM, Justin Harris
>
> wrote:
> > > I agree with this %100.  In my view I would think that we have two
>
> kinds of tests:
> > > Unit Tests - Junit
> > > Functional Tests - Cucumber?  or python tests or whatever
> > >
> > > Functional tests test the API via REST.
> > > Unit tests test small (class level) functionality
> > >
> > > What am I missing?  What is the purpose of the Java Functional
>
> Tests?
> > > Justin
> > >
> > > ----- "Bryan Kearney"  wrote:
> > > > I vote for a minimal number of testy things. RIght now I see:
> > > >
> > > > Junit
> > > > Java Functional Tests
> > > > python scripts
> > > > nosetests
> > > >
> > > > if cucumber were to replace one of these.. great!
> > > >
> > > > -- bk
> > > >
> > > > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > > > I don't know if anyone has heard of Cucumber
>
> (http://cukes.info/) -
> > > > its a BDD framework in ruby, but it seems like a nice way to
>
> define
> > > > specific user stories that are directly executable.  Has anyone
>
> played
> > > > around with this?  It could be a nice fit for candlepin...
> > > > > > - Justin
> > > > > >
> > > > > > candlepin mailing list
> > > > > > [e-mail]
> > > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > > >
> > > > > cucumber is HOT (admittedly i haven't played much with it).
> > > > > -d
> > > > > _______________________________________________
> > > > > candlepin mailing list
> > > > > [e-mail]
> > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > >
> > > > candlepin mailing list
> > > > [e-mail]
> > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > >
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
>
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

[Jesus M. Rodriguez]

If you want Cucumber to replace the other things, I suggest getting it
setup to run at least one of the test scenarios and show how it will
fit in our process.

If it seems like a good fit we can decide then. I don't want
to say let's do it without knowing how it will work, same
for not wanting to just give up on it.

jesus

On Wed, Mar 10, 2010 at 9:09 AM, Justin Harris  wrote:
> Personally, I would like to give Cucumber an evaluation to see if it could fit in well with buildr plus our extreme agile development. :)  I'm happy to let it fall by the wayside if folks are opposed to it, however.
>
> - Justin
>
> ----- "Bryan Kearney"  wrote:
> > What was the outcome of this? Do we have a plan to solidify / modify /
> >
> > or any other "fy" the various test bits?
> >
> > -- bk
> >
> > On 03/03/2010 10:24 AM, Devan Goodwin wrote:
> > > The Java functional tests evolved first, they're kind of a hybrid,
> >
> > no
> > > deployed Candlepin required, you have access to the database for
> >
> > setup
> > > purposes, but then (I think) they fire up a servlet container
> > > internally (Grizzy?) and test against that. Some benefits to this
> >
> > but
> > > this doesn't catch problems that sometimes surface in Tomcat/JBoss.
> > >
> > > Personally I'd prefer we port them to Python suite. (or something
> >
> > else
> > > should we replace that) I like testing this outside against a full
> > > environment as we can technically point it anywhere. It will
> >
> > clutter
> > > up the database (and thus should never be run against an important
> >
> > db)
> > > and is a little harder to inject setup data (but this basically
> >
> > just
> > > boils down to uploading a cert right now), IMO we need that kind of
> > > external testing to have some degree of confidence that this thing
> >
> > is
> > > working.
> > >
> > > I'd vote we boil testing down to just Java unit tests, and Python
> >
> > nosetests.
> > > As for Cucumber it's blowing my mind and I don't yet see how it
> >
> > even
> > > works. :) Have to dig into it.
> > >
> > > Can QA fit either solution (Cucumber/Python tests) into their
> >
> > automated systems?
> > > Cheers,
> > >
> > > Devan
> > >
> > > On Wed, Mar 3, 2010 at 11:02 AM, Justin Harris
> >
> > wrote:
> > > > I agree with this %100.  In my view I would think that we have two
> >
> > kinds of tests:
> > > > Unit Tests - Junit
> > > > Functional Tests - Cucumber?  or python tests or whatever
> > > >
> > > > Functional tests test the API via REST.
> > > > Unit tests test small (class level) functionality
> > > >
> > > > What am I missing?  What is the purpose of the Java Functional
> >
> > Tests?
> > > > - Justin
> > > >
> > > > ----- "Bryan Kearney"  wrote:
> > > > > I vote for a minimal number of testy things. RIght now I see:
> > > > >
> > > > > Junit
> > > > > Java Functional Tests
> > > > > python scripts
> > > > > nosetests
> > > > >
> > > > > if cucumber were to replace one of these.. great!
> > > > >
> > > > > -- bk
> > > > >
> > > > > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > > > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > > > > I don't know if anyone has heard of Cucumber
> >
> > (http://cukes.info/) -
> > > > > its a BDD framework in ruby, but it seems like a nice way to
> >
> > define
> > > > > specific user stories that are directly executable.  Has anyone
> >
> > played
> > > > > around with this?  It could be a nice fit for candlepin...
> > > > > > >
> > > > > > >     - Justin
> > > > > > > _______________________________________________
> > > > > > > candlepin mailing list
> > > > > > > [e-mail]
> > > > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > > > > >
> > > > > > cucumber is HOT (admittedly i haven't played much with it).
> > > > > > -d
> > > > > > _______________________________________________
> > > > > > candlepin mailing list
> > > > > > [e-mail]
> > > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > > >
> > > > > candlepin mailing list
> > > > > [e-mail]
> > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > >
> > > > candlepin mailing list
> > > > [e-mail]
> > > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin
>
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

[Jesus M. Rodriguez]

Sorry for top posting

jesus

[Justin Harris]

Oh I agree completely - just an evaluation was all I was suggesting.  No sense in make drastic changes without some solid data.

----- "Jesus M. Rodriguez"  wrote:
> If you want Cucumber to replace the other things, I suggest getting it
> setup to run at least one of the test scenarios and show how it will
> fit in our process.
>
> If it seems like a good fit we can decide then. I don't want
> to say let's do it without knowing how it will work, same
> for not wanting to just give up on it.
>
> jesus
>
> On Wed, Mar 10, 2010 at 9:09 AM, Justin Harris 
> wrote:
> > Personally, I would like to give Cucumber an evaluation to see if it
>
> could fit in well with buildr plus our extreme agile development. :)
>  I'm happy to let it fall by the wayside if folks are opposed to it,
> however.
> > - Justin
> >
> > ----- "Bryan Kearney"  wrote:
> > > What was the outcome of this? Do we have a plan to solidify /
>
> modify /
> > > or any other "fy" the various test bits?
> > >
> > > -- bk
> > >
> > > On 03/03/2010 10:24 AM, Devan Goodwin wrote:
> > > > The Java functional tests evolved first, they're kind of a
>
> hybrid,
> > > no
> > > > deployed Candlepin required, you have access to the database for
> > >
> > > setup
> > > > purposes, but then (I think) they fire up a servlet container
> > > > internally (Grizzy?) and test against that. Some benefits to this
> > >
> > > but
> > > > this doesn't catch problems that sometimes surface in
>
> Tomcat/JBoss.
> > > > Personally I'd prefer we port them to Python suite. (or something
> > >
> > > else
> > > > should we replace that) I like testing this outside against a
>
> full
> > > > environment as we can technically point it anywhere. It will
> > >
> > > clutter
> > > > up the database (and thus should never be run against an
>
> important
> > > db)
> > > > and is a little harder to inject setup data (but this basically
> > >
> > > just
> > > > boils down to uploading a cert right now), IMO we need that kind
>
> of
> > > > external testing to have some degree of confidence that this
>
> thing
> > > is
> > > > working.
> > > >
> > > > I'd vote we boil testing down to just Java unit tests, and Python
> > >
> > > nosetests.
> > > > As for Cucumber it's blowing my mind and I don't yet see how it
> > >
> > > even
> > > > works. :) Have to dig into it.
> > > >
> > > > Can QA fit either solution (Cucumber/Python tests) into their
> > >
> > > automated systems?
> > > > Cheers,
> > > >
> > > > Devan
> > > >
> > > > On Wed, Mar 3, 2010 at 11:02 AM, Justin
>
> Harris
> > > wrote:
> > > > > I agree with this %100.  In my view I would think that we have
>
> two
> > > kinds of tests:
> > > > > Unit Tests - Junit
> > > > > Functional Tests - Cucumber?  or python tests or whatever
> > > > >
> > > > > Functional tests test the API via REST.
> > > > > Unit tests test small (class level) functionality
> > > > >
> > > > > What am I missing?  What is the purpose of the Java Functional
> > >
> > > Tests?
> > > > > - Justin
> > > > >
> > > > > ----- "Bryan Kearney"  wrote:
> > > > > > I vote for a minimal number of testy things. RIght now I see:
> > > > > >
> > > > > > Junit
> > > > > > Java Functional Tests
> > > > > > python scripts
> > > > > > nosetests
> > > > > >
> > > > > > if cucumber were to replace one of these.. great!
> > > > > >
> > > > > > -- bk
> > > > > >
> > > > > > On 03/03/2010 09:52 AM, Dmitri Dolguikh wrote:
> > > > > > > On 03/03/2010 10:50 AM, Justin Harris wrote:
> > > > > > >> I don't know if anyone has heard of Cucumber
> > >
> > > (http://cukes.info/) -
> > > > > > its a BDD framework in ruby, but it seems like a nice way to
> > >
> > > define
> > > > > > specific user stories that are directly executable.  Has anyone
> > >
> > > played
> > > > > > around with this?  It could be a nice fit for candlepin...
> > > > > > >>
> > > > > > >>     - Justin
> > > > > > >> _______________________________________________
> > > > > > >> candlepin mailing list
> > > > > > >> [e-mail]
> > > > > > >> https://fedorahosted.org/mailman/listinfo/candlepin
> > > > > > >>
> > > > > > > cucumber is HOT (admittedly i haven't played much with it).
> > > > > > > -d
> > > > > > > _______________________________________________
> > > > > > > candlepin mailing list
> > > > > > > [e-mail]
> > > > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > > > >
> > > > > > candlepin mailing list
> > > > > > [e-mail]
> > > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > > > >
> > > > > candlepin mailing list
> > > > > [e-mail]
> > > > > https://fedorahosted.org/mailman/listinfo/candlepin
> > >
> > > candlepin mailing list
> > > [e-mail]
> > > https://fedorahosted.org/mailman/listinfo/candlepin
> >
> > candlepin mailing list
> > [e-mail]
> > https://fedorahosted.org/mailman/listinfo/candlepin
>
> candlepin mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/candlepin

[Bryan Kearney]

I like this but one question:

On 03/10/2010 09:49 AM, Justin Harris wrote:
> Oh I agree completely - just an evaluation was all I was suggesting.  No sense in make drastic changes without some solid data.

Which tests will it replace? Java Functional, Java Unit, nosetests?

-- bk

Sorry for bi-posting.

[Justin Harris]

----- "Bryan Kearney"  wrote:
> I like this but one question:
>
> On 03/10/2010 09:49 AM, Justin Harris wrote:
> > Oh I agree completely - just an evaluation was all I was suggesting.
>
> No sense in make drastic changes without some solid data.
> >
>
> Which tests will it replace? Java Functional, Java Unit, nosetests?

I would think this would replace all of our functional tests (the deprecated java ones and the nosetests) - we definitely want to keep the finer grained Java unit tests, as the cucumber tests would test candlepin only at the API level.
> -- bk
>
> Sorry for bi-posting.

[Dmitri Dolguikh]

My idea writing those java functional tests was to have tests setup the 
*complete* test environment. But if they don't catch all issues (what 
were the issues in jboss environment?) then they are certainly less 
useful than python-based ones.

I don't have a strong preference on which framework to use, I only want 
tests to be easy and quick to setup (otherwise *less* tests will be 
written).

cheers,
-d

[Devan Goodwin]

On Wed, Mar 10, 2010 at 11:55 AM, Dmitri Dolguikh  wrote:
> My idea writing those java functional tests was to have tests setup the
> *complete* test environment. But if they don't catch all issues (what
> were the issues in jboss environment?) then they are certainly less
> useful than python-based ones.

Lots of deployment issues in both tomcat and jboss won't be caught by
these, it also doesn't keep us properly in tune with exactly what's
getting serialized in and out as the Java objects kind of hide it all.
I've got much more confidence if we can point the suite at a
particular deployment and see the results, so I'd argue the nosetests
(or replacement) *must* exist, and thus it probably makes more sense
to just do one functional test suite instead of two.
> I don't have a strong preference on which framework to use, I only want
> tests to be easy and quick to setup (otherwise *less* tests will be
> written).

Me either, right now I understand the nosetests and I don't understand
Cucumber, but if we do a POC and it looks easy to write and maintain
then it's fine with me.

Devan