2010-04-14 - Bodhi and AutoQA recap
by James Laska
# Bodhi Updates and AutoQA Discussion
# Date: 2010-04-14
# Time: 14:00 UTC (11:00 EDT, 15:00 CET)
# Participants: lmacken, kparal, jlaska, jkeating, wwoods
= Discussion Topics =
== Goals ==
* Do *not* add more steps for package maintainers (unless the autoqa
test fails)
== Background: depcheck ==
* triggered on post-bodhi-update hook
* inputs: update NVRs, target repo, [other pending packages]
** Why "other pending packages"?
*** Can't handle interdependent packages without it
== Current Bodhi ==
* What states/stages do updates go through?
** See workflow below
* How do we recognize them through the API?
** See workflow below
== Review bodhi update workflow ==
Package built by maintainer
| koji tag dist-fxx-updates-candidate
V
Bodhi update created by maintainer
| koji tag dist-fxx-updates-candidate
| bodhi (status: pending, request: None)
V
Bodhi update requested by maintainer (for stable or testing)
| koji tag dist-fxx-updates-candidate
| bodhi (status: pending, request: ['testing' or 'stable'])
| bodhi tags the package with the *-pending tag, sends message to bus
V
pending
| could be koji tag dist-fxx-updates(-testing)-pending
| koji uses newRepo to make repos
V
Wait for newRepo <---+
| |
| FAIL (remains in pending)
V |
Acceptance testing ---+
|
PASS - autoqa gives +1 to update (future work: autoqa sets 'test-ok'
tag/field)
|
V
Available for pushing (still in dist-fxx-updates(-testing)-pending)
|
V
Pushed by releng (package moved from -candidate/-testing to
-updates(-testing) and drops -pending tag)
(-testing updates can now go to "update requested" with request:
'stable')
== What does depcheck/autoqa need? ==
* Needs a repo of pending packages
* Package sanity requirement: https://fedorahosted.org/autoqa/ticket/139
** Multiple tests are needed for complete package acceptance, the autoqa
+1 should happen after all have completed
* un-depsolved packages = the *-pending tags
* depsolved packages (eligible for push) = *-pending tag & +1 from
autoqa
= Open Questions =
Q. Are we going to need a lot more koji tags?
A. The tag could live on the bodhi side, just need a way to determine
which builds are eligible for pushing to requested repo (stable,
testing)
Q. How to note that AutoQA has passed acceptance tests?
A. Agreement that there should be AutoQA specific karma in bodhi. This
lets maintainers and other testers see that the response is from an
automated tool
Q. Re-test everything once a maintainer retracts an update request?
A. Yes - any time the -pending set changes, we need to re-run the
depcheck/acceptance test
Q. What happens when things change and an update which once passed
acceptance no longer does?
A.
Q. What about the tests beyond depcheck, individual package acceptance
tests
A. This process would only relate to package acceptance testing in
accordance with the criteria
(https://fedoraproject.org/wiki/Package_update_acceptance_criteria).
Functional testing will occur once the packages/updates are moved to
updates-testing.
Q. How to handle packages that failed testing, and then a new NVR comes
in a new update request?
A.
Q. If a maintainer revokes a push request what now?
A. Need to remove the -pending tag
Q. What do we do when a firefox security update is headed out, but
breaks other packages?
A.
Q. What to do when acceptance test fails
A. Don't add negative karma, add karma 0 with a comment in the bodhi
request
Q. We have 4 planned automated acceptance tests, how to make sure all
run before approving the update
A.
= Action Items =
[ ] - Update the bodhi-1.0 watcher to detect builds (quick'n'dirty
method to watch for koji *-pending tags) (wwoods)
[ ] - Integrate use of mash into acceptance tests (jkeating && wwoods)
[ ] - Create new koji tags for *-pending (jkeating)
[ ] - Bodhi-1.0 changes? (lmacken)
[ ] - Add -pending tag when update is requested
[ ] - Magic handling for autoqa's +1 karma (only show autoqa-approved
stuff)
[ ] - Bodhi-2.0 roadmap changes?
- Tickets in bodhi trac, against 2.0 milestone
[ ] - Move package acceptance test plan out of Draft (kparal && jlaska)
= Next Meeting =
2010-04-28 - Another 'touch-base' meeting
14 years
2010-04-16 - AutoQA resultsdb meeting recap
by James Laska
# AutoQA ResultsDB Discussion
# Date: 2010-04-16
# Time: 14:00 UTC (10:00 EDT, 15:00 CET)
# Attendees: kparal, jskladan, wwoods, jlaska
= Previous action items =
* [jskladan+kparal] Create updated DB schema according to the discussion
** https://fedoraproject.org/wiki/AutoQA_resultsdb_schema#Result
-> rename 'Test' table to 'TestCase'
-> type of the test is not stored in the DB, add it somewhere, or use
tags? We must have the mandatory keyvals stored in the database (cached?
from wiki) to have fast access to the keyval pairs
->-> we don't have to keep this data inside DB if we are just happy with
that result not displaying in the frontend; DB itself doesn't care about
invalid data
* [wwoods?] Study MediaWiki RPC mechanism
* [kparal] Define default (common) key-values for basic test classes
(installation tests, repo tests, package tests)
**
https://fedoraproject.org/wiki/AutoQA_resultsdb_schema#Default_key-values...
-> install key/val pairs exist in .treeinfo
(http://download.fedoraproject.org/pub/fedora/linux/development/13/i386/os...)
* [jskladan] Communicate with Autotest developers how to get Job ID in
autotest client
**
https://fedorahosted.org/pipermail/autoqa-devel/2010-March/000326.html
* [jlaska] Try to set up a qpid instance and send some test data through
it (from provider to listener)
(jkeating (Oxf13) is a good person to talk to about this!)UI
** https://fedorahosted.org/autoqa/ticket/132#comment:1
= ResultsDB API =
* 'Input' interface
<https://fedoraproject.org/wiki/AutoQA_resultsdb_API>
* Is this satisfactory? Do we any other methods for pushing data
into
the ResultsDB
* Tagging - kparal and I discussed, if there could be any reasonable
tag, which one would like to use _from inside_ the testrun. (All the
tags I could think of were either about Tests, not Testruns, or they'd
make sense to be applied after a real person went through the results.)
* 'Output' interface
* Fedora Message Bus (FMB)
* What's the state?
-> In progress, Bodhi will may soon talk over the bus,
* Will we set up some testing instance?
-> There should be running Qpid instance somewhere
-> talk to infrastructure guys about current state and if/how can
we use it
* My understanding of using the FMB is, that we'll just post new
'events' to the FMB, and whoever wants to will consume that
information,
is that right? If so, will the consumers be responsible for storing the
data, for their further use, or will they be able to query ResultsDB?
-> Mostly we'll send events, that will inform users that 'new data
is in the resultsDB'
* Querying ResultsDB
* What is the best approach
1) Set of predefined functions
2) Possibility to execute SQL queries
-> not secure, if needed we should add API
3) Combination of both
* Predefined functions:
* What functions will we provide at the beginning?
-> we'll create a set of functions _we_ need at the beginning
(we'll find out, while creating the front-end)
* Will we add more if somebody asks?
* How to tell, if the demand is valid?
= Open Questions =
Q. Whether or not front-ends will be message bus consumers, or query the
resultsdb directly?
A.
Q. Are we at the point where we can add a built-in unit test suite for
resultsdb?
A. Jskladan noted there is a ticket for this already (see
https://fedorahosted.org/autoqa/ticket/144)
Q. How to proceed with prototyping?
A. Jskladan plans to experiment with a python xmlrpc to get started.
Wwoods recommended moving to TurboGears when the time is right.
Q. Next steps?
A. Prototype the API ... use this to get data in and out of resultsdb.
Wwoods noted that it can be helpful to think of a specific use case
first (installation, package acceptance).
Q. What front-end should we focus on first.
A. Wwoods+jlaska felt the current draft package acceptance test plan
(https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_accept...) was best to start with.
Q. What to do with 'Is rawhide broken' test plan?
A. One suggestion was to rename it to 'Is branched broken'.
= Action Items =
* [all] *Keep It Simple*
* [ ] - talk to Fedora Infrastructure to figure out current msgbus state
and how to use it
* [ ] - ask lmacken || toshio whether TurboGears2 is still prefered
Fedora solution
14 years
Re: [PATCH] Ticket 65, 130: getting autotest test-ID to the client
by Kamil Paral
----- "Josef Skladanka" <jskladan(a)redhat.com> wrote:
> Added new option to autoqa --autotest-server which can be used
> to set hostname of the autotest server, which will hold the results
> of the test. Since our infrastructure has Autotest and AutoQA
> on the same machine, default value for this option is the hostname
> of current machine (socket.gethostname()).
Maybe it would be nice also to support this option in /etc/autoqa.conf,
so in case we need to use it we don't have to specify the command
line option all the time.
14 years
[PATCH] Ticket 65, 130: getting autotest test-ID to the client
by Josef Skladanka
Added new option to autoqa --autotest-server which can be used
to set hostname of the autotest server, which will hold the results
of the test. Since our infrastructure has Autotest and AutoQA
on the same machine, default value for this option is the hostname
of current machine (socket.gethostname()).
Also the templates of control were updated, so they propagate
the basic URL to the Autotest results directory.
Test_class_templates were updated, so they create the whole
URL to the results.
---
autoqa | 6 ++++++
hooks/post-koji-build/control.template | 3 ++-
hooks/post-koji-build/test_class_template.py | 5 ++++-
hooks/post-repo-update/control.template | 3 ++-
hooks/post-repo-update/test_class_template.py | 5 ++++-
hooks/post-tree-compose/control.template | 3 ++-
hooks/post-tree-compose/test_class_template.py | 5 ++++-
7 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/autoqa b/autoqa
index efb94e3..9f41c00 100755
--- a/autoqa
+++ b/autoqa
@@ -27,6 +27,7 @@ import optparse
import tempfile
import StringIO
import urlgrabber
+import socket
from ConfigParser import *
from subprocess import call
@@ -139,6 +140,9 @@ parser.add_option('--local', action='store_true',
dest='local',
help='Do not schedule jobs - run test(s) directly on the local
machine')
parser.add_option('-l', '--list-tests', action='store_true',
dest='listtests',
help='list the tests for the given hookname - do not run any tests')
+parser.add_option('--autotest-server', action='store',
default=socket.gethostname(),
+ help='Sets the autotest-server hostname. Used for creating URLs to
results.\
+Hostname of the local machine is used by default.')
# Read and validate the hookname
# Check for no args, or just -h/--help
if len(sys.argv) == 1 or sys.argv[1] in ('-h', '--help'):
@@ -197,6 +201,8 @@ for arch in opts.arch:
# N.B. process_testdata may grow new keyword arguments if we add
new autoqa
# args that add another loop here..
testdata = hook.process_testdata(opts, args, arch=arch)
+ if not 'autotest_server' in testdata.keys():
+ testdata['autotest_server'] = opts.autotest_server
# XXX FIXME: tests need to be able to indicate that they do not
require
# any specific arch (e.g. rpmlint can run on any arch)
for test in testlist:
diff --git a/hooks/post-koji-build/control.template
b/hooks/post-koji-build/control.template
index b90260f..b7c5261 100644
--- a/hooks/post-koji-build/control.template
+++ b/hooks/post-koji-build/control.template
@@ -15,4 +15,5 @@ TEST_CATEGORY = 'Functional'
# kojitag: koji tag applied to this package
job.run_test('testclassname', name=name, envr=envr, kojitag=kojitag,
- config=autoqa_conf)
+ config=autoqa_conf,
+ autotest_url="http://%s/results/%s/" % (autotest_server,
job.tag))
diff --git a/hooks/post-koji-build/test_class_template.py
b/hooks/post-koji-build/test_class_template.py
index cf91865..462dc91 100644
--- a/hooks/post-koji-build/test_class_template.py
+++ b/hooks/post-koji-build/test_class_template.py
@@ -35,7 +35,10 @@ class post_repo_update_test_class_name(test.test):
#utils.system('yum -y install yum-utils')
pass
- def run_once(self, envr, name, kojitag):
+ def run_once(self, envr, name, kojitag, autotest_url):
+ # results_url points to the directory on autotest-server with
+ #the sotred results
+ results_url = "%s%s/" % (autotest_url, self.__class__.__name__)
parentlist = ' '.split(parents)
cmd = 'test_binary --url %s' % baseurl
# You can get stuff from the [test] section of autoqa.conf
like this
diff --git a/hooks/post-repo-update/control.template
b/hooks/post-repo-update/control.template
index 50ba58d..204dc69 100644
--- a/hooks/post-repo-update/control.template
+++ b/hooks/post-repo-update/control.template
@@ -19,4 +19,5 @@ TEST_CATEGORY = 'Functional'
job.run_test('testclassname', baseurl=url,
parents=parents,
reponame=reponame,
- config=autoqa_conf)
+ config=autoqa_conf,
+ autotest_url="http://%s/results/%s/" % (autotest_server,
job.tag))
diff --git a/hooks/post-repo-update/test_class_template.py
b/hooks/post-repo-update/test_class_template.py
index e274f98..c0c0137 100644
--- a/hooks/post-repo-update/test_class_template.py
+++ b/hooks/post-repo-update/test_class_template.py
@@ -33,7 +33,10 @@ class post_repo_update_test_class_name(test.test):
#utils.system('yum -y install yum-utils')
pass
- def run_once(self, baseurl, parents, reponame):
+ def run_once(self, baseurl, parents, reponame, autotest_url):
+ # results_url points to the directory on autotest-server with
+ #the sotred results
+ results_url = "%s%s/" % (autotest_url, self.__class__.__name__)
parentlist = ' '.split(parents)
cmd = 'test_binary --url %s' % baseurl
# You can get stuff from the [test] section of autoqa.conf
like this
diff --git a/hooks/post-tree-compose/control.template
b/hooks/post-tree-compose/control.template
index e327799..ef8ea35 100644
--- a/hooks/post-tree-compose/control.template
+++ b/hooks/post-tree-compose/control.template
@@ -17,4 +17,5 @@ TEST_CATEGORY = 'Functional'
job.run_test('testclassname', baseurl=url,
treename=treename,
- config=autoqa_conf)
+ config=autoqa_conf,
+ autotest_url="http://%s/results/%s/" % (autotest_server,
job.tag))
diff --git a/hooks/post-tree-compose/test_class_template.py
b/hooks/post-tree-compose/test_class_template.py
index 0ffab9d..6084fb1 100644
--- a/hooks/post-tree-compose/test_class_template.py
+++ b/hooks/post-tree-compose/test_class_template.py
@@ -33,7 +33,10 @@ class post_tree_compose_test_class_name(test.test):
#utils.system('yum -y install yum-utils')
pass
- def run_once(self, baseurl, treename):
+ def run_once(self, baseurl, treename, autotest_url):
+ # results_url points to the directory on autotest-server with
+ #the sotred results
+ results_url = "%s%s/" % (autotest_url, self.__class__.__name__)
cmd = 'test_binary --url %s' % baseurl
# You can get stuff from the [test] section of autoqa.conf
like this
email = self.config.get('test','result_email')
--
1.6.6.1
14 years
[AutoQA] #139: pst: Find a way how to test packages coming to updates-testing
by fedora-badges
#139: pst: Find a way how to test packages coming to updates-testing
--------------------+-------------------------------------------------------
Reporter: kparal | Owner:
Type: task | Status: new
Priority: major | Milestone: Package Sanity Tests
Component: tests | Version: 1.0
Keywords: pst |
--------------------+-------------------------------------------------------
We must find a way how to test package sanity of packages coming to
updates-testing, that means those packages there are not yet in any repo.
The problem here are the dependencies, which may already be present in
updates-testing (ideal case), but often may not be. In this case the
update author should have probably specified all depending packages in one
"set" when creating the update in Bodhi. We must communicate with Bodhi,
find out this set, detect all packages this set contains, download them,
and install them together with our main package. Does Bodhi have API
capable of this?
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/139>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
14 years
[AutoQA] #146: ResultsDB: Propose ResultsDB API - API for the frontends
by fedora-badges
#146: ResultsDB: Propose ResultsDB API - API for the frontends
-----------------------+----------------------------------------------------
Reporter: jskladan | Owner:
Type: task | Status: new
Priority: major | Milestone: Resultdb
Component: docs/wiki | Version: 1.0
Keywords: |
-----------------------+----------------------------------------------------
In #131, we have a API for the tests side of the resultsdb, i.e. API for
storing the information.
We should discuss the API we want to provide for digging information from
the resultsdb - do we want the frontends to have limited set of functions,
of do we want to give them a chance to run any select/join query they'd
like?
Binded with #145
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/146>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
14 years
