[AutoQA] #352: Determine AutoQA Functional Self Test Cases
by fedora-badges
#352: Determine AutoQA Functional Self Test Cases
--------------------+-------------------------------------------------------
Reporter: tflink | Owner:
Type: task | Status: new
Priority: major | Milestone: Self Test
Component: core | Keywords:
--------------------+-------------------------------------------------------
In order to start forming a better self test suite for AutoQA, we need to
start cataloging situations that will make for good functional regression
tests.
Instead of focusing on how to get them all run, the idea is to describe
the situations that would make for good tests. Formatting and categorizing
the tests will be left for another ticket.
Examples of jobs that have caused problems in the past or continue to
cause strange problems would be helpful.
An extremely simplistic example for depcheck might look like:
=== Package Descriptions ===
|| '''Package''' || '''Depends On''' ||
|| foo-1.0-1 || bar-1.0-1 ||
|| foo-2.0-1 || bar-2.0-1 ||
=== Repo Contents ===
|| '''Name''' || '''Packages''' ||
|| stable || foo-1.0-1, bar-1.0-1 ||
|| pending || foo-2.0-1 ||
=== Expected Results ===
Failure because foo-2.0-1 depends on bar-2.0-1 which is not available.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/352>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #330: upgradepath: improve behavior when pushing update to multiple releases
by fedora-badges
#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
Reporter: kparal | Owner:
Type: enhancement | Status: new
Priority: major | Milestone: 0.5.0
Component: tests | Keywords:
-------------------------+--------------------------------------------------
This ticket stems from my talk to mcepl and our following discussion:
https://fedorahosted.org/pipermail/autoqa-devel/2011-May/002215.html
Package maintainers usually push fixes to many Fedora releases at once.
Upgradepath usually fails for all but the highest one. We should improve
the output.
1. If upgradepath passes, no changes.
2. If upgradepath fails, check update-pending repos for the failed
sections and determine whether the same proposed update is available
there. If it is, check upgradepath again and determine whether test would
pass if all those proposed updates for all releases were pushed at the
same time. Inform the user about it.
3. If the condition in 2) is valid, change FAIL result to
NEEDS_INSPECTION.
I'll gladly take this ticket when I finish my current work.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #348: anaconda upgrade test
by fedora-badges
#348: anaconda upgrade test
----------------------+-----------------------------------------------------
Reporter: hongqing | Owner: hongqing
Type: task | Status: new
Priority: major | Milestone: Automate installation test plan
Component: core | Keywords:
----------------------+-----------------------------------------------------
To implement test case:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_New_Bootloader
. we have already implemented Fedora installation automation from DVD,
next step we will upgrade the previous release. "virsh' will be used to
operate on the guest. [[BR]]
The tasks include:[[BR]]
* define upgrade steps [[BR]]
* install previous release [[BR]]
* modify guest xml to boot from new release[[BR]]
* upgrade
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/348>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #354: Custom 404 page for autoqa.fp.o
by fedora-badges
#354: Custom 404 page for autoqa.fp.o
----------------------+-----------------------------------------------------
Reporter: kparal | Owner:
Type: task | Status: new
Priority: minor | Milestone: 0.6.0
Component: autotest | Keywords:
----------------------+-----------------------------------------------------
Quoting tflink:
For some reason, this came to mind when I was thinking about the bodhi
link issue but how do we feel about putting a custom 404 on autoqa.fp.o?
Since we don't keep logs forever, anything that links to old results
goes to a generic 404 without more explanation than "could not find the
file you were looking for". Instead, we could do a custom 404 that
contains:
- Explanation of log retention policy
- Relevant AutoQA or FedoraQA links
It would involve a small change to the apache config and some static
HTML on the autotest server. Pretty easy finger food :)
https://fedorahosted.org/pipermail/autoqa-devel/2011-July/002546.html
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/354>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #102: post-iso-build hook
by fedora-badges
#102: post-iso-build hook
----------------------+-----------------------------------------------------
Reporter: wwoods | Owner:
Type: task | Status: new
Priority: major | Milestone: Automate installation test plan
Component: watchers | Version: 1.0
Keywords: |
----------------------+-----------------------------------------------------
We need one (or more) standard locations where new iso images get posted
for testing, and a watcher that can monitor for new iso images and launch
appropriate installation/sanity tests.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/102>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #209: complete auto CD install test driver
by fedora-badges
#209: complete auto CD install test driver
-------------------+--------------------------------------------------------
Reporter: liam | Owner:
Type: task | Status: new
Priority: major | Milestone: Automate installation test plan
Component: tests | Version: 1.0
Keywords: |
-------------------+--------------------------------------------------------
complete the cd_install.py as dvd_install.py to support CD install, with
some functions like automatically swap CD when CD1 install was completed.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/209>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #365: update mediakit_sanity.py to accept different images, DVD, Live, netinst.iso
by fedora-badges
#365: update mediakit_sanity.py to accept different images, DVD, Live,
netinst.iso
-------------------------+--------------------------------------------------
Reporter: hongqing | Owner: hongqing
Type: enhancement | Status: new
Priority: minor | Milestone: Automate installation test plan
Component: tests | Keywords:
-------------------------+--------------------------------------------------
mediakit_sanity test covers the DVD.iso sanity test cases. But the
Live.iso and netinst.iso are different with DVD.iso, they cannot be done
repoclosure and fileconficts tests. mediakit_sanity should be updated to
accept Live.iso and netinst.iso sanity tests.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/365>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #364: test.py: 'Unknown result keyword' handling in cmp_result()
by fedora-badges
#364: test.py: 'Unknown result keyword' handling in cmp_result()
----------------------+-----------------------------------------------------
Reporter: pschindl | Owner:
Type: defect | Status: new
Priority: minor | Milestone: Finger Food
Component: core | Keywords:
----------------------+-----------------------------------------------------
Test.py raise exception when unknown result keyword is used in
PrettyLog.summary_detail dictionary:
{{{
Traceback (most recent call last):
File "/usr/share/autotest/client/bin/parallel.py", line 18, in
fork_start
l()
File "/usr/share/autotest/client/bin/job.py", line 522, in <lambda>
l = lambda : test.runtest(self, url, tag, args, dargs)
File "/usr/share/autotest/client/bin/test.py", line 115, in runtest
job.sysinfo.log_after_each_iteration)
File "/usr/share/autotest/client/common_lib/test.py", line 710, in
runtest
mytest._exec(args, dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 419, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 630, in
_call_test_function
raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled ValueError: tuple.index(x): x not in tuple
Traceback (most recent call last):
File "/usr/share/autotest/client/common_lib/test.py", line 623, in
_call_test_function
return func(*args, **dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 291, in
execute
postprocess_profiled_run, args, dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 215, in
_call_run_once
self.postprocess_iteration()
File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 135, in
postprocess_iteration
self.post_results()
File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 183, in
post_results
logname = self._write_log(test_detail)
File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 313, in
_write_log
log.write(test_detail.create_output(full_log=full_log))
File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 561, in
create_output
return self.log.create()
File "/usr/lib/python2.7/site-packages/autoqa/pretty_log.py", line 245,
in create
summary_detail = self._make_summary_detail()
File "/usr/lib/python2.7/site-packages/autoqa/pretty_log.py", line 87,
in _make_summary_detail
cmp=lambda x,y: td.cmp_result(x,y))
File "/usr/lib/python2.7/site-packages/autoqa/pretty_log.py", line 87,
in <lambda>
cmp=lambda x,y: td.cmp_result(x,y))
File "/usr/lib/python2.7/site-packages/autoqa/test.py", line 499, in
cmp_result
index1 = self.result_order.index(result1) if result1 is not None else
-1
ValueError: tuple.index(x): x not in tuple
}}}
Problem is in cmp_result() from test.py. It compares only results which
are in result_order tuple. We can write some reasonable error message or
we can rewrite cmp_result to handle this problem (then it is needed to
check if it won't break something).
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/364>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
[AutoQA] #349: Depcheck - prefilter the list of packages passed to yum instance's rpmdb
by fedora-badges
#349: Depcheck - prefilter the list of packages passed to yum instance's rpmdb
----------------------+-----------------------------------------------------
Reporter: jskladan | Owner:
Type: task | Status: new
Priority: major | Milestone: 0.6.0
Component: core | Keywords:
----------------------+-----------------------------------------------------
Recent problem with libreport
<http://autoqa.fedoraproject.org/results/140482-autotest/172.16.0.20/depch...>
is most probably caused by depcheck selecting too old packages (i.e.
ignoring the newest packages in repos) due to the fact that YUM selects
packages alphabeticaly on fist-comes-first served basis.
Because YUM is hacked into believing that all packages are installed by
passing the content of _whole_ pkgsack into the rpmdb, it contains
multiple versions of the same package.
I belive, that this can be easily solved by pre-filtering the list, so it
contains only the newest from each package.
<code>
# This YumBase object will act like all repo packages are already
installed
y = YumDepcheck()
y.tsInfo = yum.transactioninfo.TransactionData()
y.tsInfo.debug = 1
# this line is causing the problems
y.rpmdb = MetaSackRPMDB(yum_repos.pkgSack)
y.xsack = yum.packageSack.PackageSack()
# Hacky way to set up the databases (this is copied from yum's
testbase.py)
y.tsInfo.setDatabases(y.rpmdb, y.xsack)
</code>
the yum_repos.pkgSack is a list of tuples NAEVR <code>[('libreport',
'x86_64', '0', '2.0.2', '5.fc15'), ('libreport', 'i686', '0', '2.0.2',
'5.fc15'), ('libreport', 'x86_64', '0', '2.0.4', '1.fc15'), ('libreport',
'i686', '0', '2.0.4', '1.fc15')]</code>
After the filtering, only the newest package for each architecture should
be left. Newest = the first while sorting using E,V,R as keys with
decreasing importance in this order. I.e. <code> [('libreport', 'x86_64',
'0', '2.0.4', '1.fc15'), ('libreport', 'i686', '0', '2.0.4',
'1.fc15')]</code>. koji_utils.compareEVR() can be used to compare EVRs.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/349>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 8 months
