Cloud Test Overview

[Mike Ruckman]

Hey fellow fedorans!

I've made some slight edits to the existing "Test Overview" [0] on the wiki
and would like some feedback. It's meant as a high-level overview of all the 
things we think we need to test for the Fedora 21 cloud product. 

It's not intended to be an exhaustive list, just a basic statement of intent.
So if you have a couple minutes to look it over (it's short) and look for
anything I might have missed, it would be greatly appreciated.

Thanks!

[0] https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Test_Overview

-- 
// Mike 
--
Fedora QA
freenode: roshi
http://roshi.fedorapeople.org

[milanisko k]

Guys,

are the Cloud Image news being published on the Fedmsg channels?
I'd like to investigate integration of RedHatQE's  test runner and suite
<https://github.com/RedHatQE/dva>;.

Thanks,
milan



2014-10-03 3:01 GMT+02:00 Mike Ruckman <roshi(a)fedoraproject.org&gt;:

>  Hey fellow fedorans!
>
>  I've made some slight edits to the existing "Test Overview" [0] on the
> wiki
>  and would like some feedback. It's meant as a high-level overview of all
>  the
>  things we think we need to test for the Fedora 21 cloud product.
>
>  It's not intended to be an exhaustive list, just a basic statement of
>  intent.
>  So if you have a couple minutes to look it over (it's short) and look for
>  anything I might have missed, it would be greatly appreciated.
>
>  Thanks!
>
>  [0] https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Test_Overview
>
>  --
>  // Mike
>  --
>  Fedora QA
>  freenode: roshi
>  http://roshi.fedorapeople.org
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>

[Matthew Miller]

On Fri, Oct 03, 2014 at 01:57:43PM +0200, milanisko k wrote:
>  are the Cloud Image news being published on the Fedmsg channels?
>  I'd like to investigate integration of RedHatQE's  test runner and suite
>  <https://github.com/RedHatQE/dva>;. 
Amazingly, yes -- that plan is slowly, slowly coming to fruition.

Take a look at http://fedimg.readthedocs.org/en/latest/ for the Fedimg
service, and particularly

  http://fedimg.readthedocs.org/en/latest/consumer/
  http://fedimg.readthedocs.org/en/latest/messaging/



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[milanisko k]

Thanks,
seems fedimg is going to be both an image upload and test service, right?

Cheers,
milan

2014-10-03 14:22 GMT+02:00 Matthew Miller <mattdm(a)fedoraproject.org&gt;:

>  On Fri, Oct 03, 2014 at 01:57:43PM +0200, milanisko k wrote:
>  > are the Cloud Image news being published on the Fedmsg channels?
>  > I'd like to investigate integration of RedHatQE's  test runner and suite
>  > <https://github.com/RedHatQE/dva>;.
>
>  Amazingly, yes -- that plan is slowly, slowly coming to fruition.
>
>  Take a look at http://fedimg.readthedocs.org/en/latest/ for the Fedimg
>  service, and particularly
>
>    http://fedimg.readthedocs.org/en/latest/consumer/
>    http://fedimg.readthedocs.org/en/latest/messaging/
>
>
>
>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>

[Matthew Miller]

On Tue, Oct 07, 2014 at 05:28:27PM +0200, milanisko k wrote:
>  seems fedimg is going to be both an image upload and test service,
> right? 
The idea is for it to do some really simple initial smoketests, to save
further testing systems from wasting time. I don't think we want it to grow
to become yet another test harness.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[milanisko k]

Cool, thanks.
Smoke-testing at this step is a reasonable thing to do.
I'm asking because our RCM is currently in the process of  adopting a
strategy similar to fedimg so I thought we could donate couple of extra
cycles should  the community benefit from our test harness.
Anyways, we'd be delighted to share.

Cheers,
milan

2014-10-07 17:47 GMT+02:00 Matthew Miller <mattdm(a)fedoraproject.org&gt;:

>  On Tue, Oct 07, 2014 at 05:28:27PM +0200, milanisko k wrote:
>  > seems fedimg is going to be both an image upload and test service, right?
>
>  The idea is for it to do some really simple initial smoketests, to save
>  further testing systems from wasting time. I don't think we want it to grow
>  to become yet another test harness.
>
>
>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>

[Mike Ruckman]

On Fri, Oct 03, 2014 at 01:57:43PM +0200, milanisko k wrote:
>  Guys,
>  
>  are the Cloud Image news being published on the Fedmsg channels?
>  I'd like to investigate integration of RedHatQE's  test runner and suite
>  <https://github.com/RedHatQE/dva>;.
>  
>  Thanks,
>  milan
>  
>   
Milan,

Currently fedimg has some smoketests in it, but the plan for the future is to
decouple the tests from fedimg and move the testing to Taskotron. We're still
a ways out before that can happen, but it's the end goal we're looking to 
accomplish.

-- 
// Mike 
--
Fedora QA
freenode: roshi
http://roshi.fedorapeople.org

[milanisko k]

Fair enough. Not sure about the Taskotron features (plan) but our harness
already supports couple of things which are a must for the load we've been
facing wr. images validation:
- parallel instances handling
- parallel test cases execution
- test case dependencies
- remote (bash) code execution (rpyc)
- expect-like test support (paramiko + bunch of handy methods)
- dry-run mode
- event-driven (gevent)
- capable of handling ~200 simultaneous connections from a single process
- custom testing modules support
- test case tags, stages, filters and applicability handling
- instance reboot handling
- plain YAML result files, easily to translate to e.g. xunit
- GPLd, GitHub-hosted, open-source
These give us a workhorse for which we're missing a fedimg listener atm.
Actually, we have to implement one anyway because of our release
engineering adopting a process similar to fedmsg.

Sorry for the advertisement ;) but we're missing a community and should the
community miss a harness, there's one worth of considering I think...

Cheers,
milan


2014-10-08 1:21 GMT+02:00 Mike Ruckman <roshi(a)fedoraproject.org&gt;:

>  On Fri, Oct 03, 2014 at 01:57:43PM +0200, milanisko k wrote:
>  > Guys,
>  >
>  > are the Cloud Image news being published on the Fedmsg channels?
>  > I'd like to investigate integration of RedHatQE's  test runner and suite
>  > <https://github.com/RedHatQE/dva>;.
>  >
>  > Thanks,
>  > milan
>  >
>  >
>
>  Milan,
>
>  Currently fedimg has some smoketests in it, but the plan for the future is
>  to
>  decouple the tests from fedimg and move the testing to Taskotron. We're
>  still
>  a ways out before that can happen, but it's the end goal we're looking to
>  accomplish.
>
>  --
>  // Mike
>  --
>  Fedora QA
>  freenode: roshi
>  http://roshi.fedorapeople.org
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>

[Joe Brockmeier]

On 10/02/2014 08:01 PM, Mike Ruckman wrote:
>  Hey fellow fedorans!
>  
>  I've made some slight edits to the existing "Test Overview" [0] on the
> wiki
>  and would like some feedback. It's meant as a high-level overview of all the 
>  things we think we need to test for the Fedora 21 cloud product. 
>  
>  It's not intended to be an exhaustive list, just a basic statement of intent.
>  So if you have a couple minutes to look it over (it's short) and look for
>  anything I might have missed, it would be greatly appreciated. 
This: "The product WG aims to increase support for Docker with a custom
container image."

This is in conjunction with the Base WG, as they're producing the Docker
images, yeah?

A few small edits on my part. Changed Atomic from "such as Docker
containers" to "Docker containers" -- we don't have any plans ATM to
support other container formats.

I'm still skeptical about the "adopting cattle" thing, really.

Best,

jzb
-- 
Joe Brockmeier | Principal Cloud & Storage Analyst
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/

[Mike Ruckman]

On Fri, Oct 03, 2014 at 10:03:00AM -0500, Joe Brockmeier wrote:
>  On 10/02/2014 08:01 PM, Mike Ruckman wrote:
>  > Hey fellow fedorans!
>  > 
>  > I've made some slight edits to the existing "Test Overview" [0] on the
> wiki
>  > and would like some feedback. It's meant as a high-level overview of all the 
>  > things we think we need to test for the Fedora 21 cloud product. 
>  > 
>  > It's not intended to be an exhaustive list, just a basic statement of intent.
>  > So if you have a couple minutes to look it over (it's short) and look for
>  > anything I might have missed, it would be greatly appreciated.
>  
>  This: "The product WG aims to increase support for Docker with a custom
>  container image."
>  
>  This is in conjunction with the Base WG, as they're producing the Docker
>  images, yeah? 
Yeah, that's right :/ I'll update that bit. Good catch.

>  
>  A few small edits on my part. Changed Atomic from "such as Docker
>  containers" to "Docker containers" -- we don't have any plans ATM to
>  support other container formats.
>   
I wasn't 100% sure we were only doing docker, so I went with vague :)

>  I'm still skeptical about the "adopting cattle" thing,
> really.
>   
I don't know how it would work, but we have it listed as a feature in the PRD.
https://fedoraproject.org/wiki/Cloud/Cloud_PRD#Feature:_Cloud_-.3E_Server
So until we vote on it, I left it in there.

>  Best,
>  
>  jzb
>  -- 
>  Joe Brockmeier | Principal Cloud & Storage Analyst
>  jzb(a)redhat.com | http://community.redhat.com/
>  Twitter: @jzb  | http://dissociatedpress.net/
>   

-- 
// Mike 
--
Fedora QA
freenode: roshi
http://roshi.fedorapeople.org

[Matthew Miller]

On Fri, Oct 03, 2014 at 09:54:23AM -0600, Mike Ruckman wrote:
>  > I'm still skeptical about the "adopting cattle"
> thing, really.
>  I don't know how it would work, but we have it listed as a feature in the PRD.
>  https://fedoraproject.org/wiki/Cloud/Cloud_PRD#Feature:_Cloud_-.3E_Server
>  So until we vote on it, I left it in there. 
Hope to have something on this soon. Also relates to converation about
upgrades on the main -devel list....

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Joe Brockmeier]

On 10/03/2014 11:38 AM, Matthew Miller wrote:
>  On Fri, Oct 03, 2014 at 09:54:23AM -0600, Mike Ruckman wrote:
> >> I'm still skeptical about the "adopting cattle" thing, really.
> > I don't know how it would work, but we have it listed as a feature in the PRD.
> > https://fedoraproject.org/wiki/Cloud/Cloud_PRD#Feature:_Cloud_-.3E_Server
> > So until we vote on it, I left it in there.
>  
>  Hope to have something on this soon. Also relates to converation about
>  upgrades on the main -devel list.... 
Ah, is there one? I am subscribed but filter -devel... something we need
to look in on?

Best,

jzb
-- 
Joe Brockmeier | Principal Cloud & Storage Analyst
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb  | http://dissociatedpress.net/

[Dusty Mabe]

On Thu, Oct 02, 2014 at 07:01:14PM -0600, Mike Ruckman wrote:
>  Hey fellow fedorans!
>  
>  I've made some slight edits to the existing "Test Overview" [0] on the
> wiki
>  and would like some feedback. It's meant as a high-level overview of all the 
>  things we think we need to test for the Fedora 21 cloud product. 
>  
>  It's not intended to be an exhaustive list, just a basic statement of intent.
>  So if you have a couple minutes to look it over (it's short) and look for
>  anything I might have missed, it would be greatly appreciated.
>  
>  Thanks!
>  
>  [0] https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Test_Overview 

Looks good to me. The longer I am around the more I will be able to
make informed suggestions on descriptions like this. 

Dusty