autoqa-devel September 2010

autoqa-devel@lists.fedorahosted.org

8 participants
88 discussions

[AutoQA] #170: Initscript review: net-tools
by fedora-badges 11 Oct '10

11 Oct '10

#170: Initscript review: net-tools ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: minor | Milestone: Initscripts review Component: tests | Version: 1.0 Keywords: | ----------------------+----------------------------------------------------- Check the test according to the <https://fedorahosted.org/autoqa/wiki/initscripts> -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/170> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

[AutoQA] #166: Initscript review: iputils
by fedora-badges 11 Oct '10

11 Oct '10

#166: Initscript review: iputils ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: minor | Milestone: Initscripts review Component: tests | Version: 1.0 Keywords: | ----------------------+----------------------------------------------------- Check the test according to the <https://fedorahosted.org/autoqa/wiki/initscripts> -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/166> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

upgradepath and updates-testing - summary of problems
by Kamil Paral 07 Oct '10

07 Oct '10

This is a summary of all problems related to upgradepath test and the updates-testing repo. First of all, upgrade path requirement is not properly defined anywhere. The best what I could find is this: https://fedoraproject.org/wiki/Package_maintainer_responsibilities#Miscella… That means we work mainly on assumptions. For base and -updates repo, this is not a big problem, the task is pretty straightforward: Algorithm example ----------------- Repo set RS = base + updates For every Fedora release F: For every package P in RS of F: NVR of P <= NVR of P in RS of F+1 We must check this constraint for every pending package update, right before it is pushed. We have this already implemented, and when it is enforced, we will guard all the common users (common user == using just base + updates repo set) against upgrade path problems. For -updates-testing repo however, things are not as easy. Problem #1: What is the desired repo set? ----------------------------------------- In the algorithm example, repo set of F and repo set of F+1 were equal (base + updates repos). How does repo set look like when -updates-testing is in place? Do we want to check it this way? a) F base + updates + updates-testing => F+1 base + updates + updates-testing or this way? b) F base + updates + updates-testing => F+1 base + updates Option a) is simple, we use the same approach as in the algorithm example. But it may not really guard the user, see following problems. Option b) guards the user well, but it has some serious drawbacks. It basically says that you must first push into F+1-updates before you can push to F-updates-testing. That can have a large impact on package maintainers workflow. If you consider that it may take one or two weeks before your update is pushed into -updates, then it may a whole month or more before your update propagates from Rawhide to Rawhide-3. Example: 1. In all repos there is foo-2.0. 2. I want to push foo-3.0 into Rawhide, F14, F13 and F12. 3. First I must push to Rawhide. 4. Once it is in Rawhide, I can push to F14-updates-testing. 5. Now I must wait until testers give me enough karma, and then push it into F14-updates. 6. Only now (not before) I can push F13-updates-testing. 7. Now I must wait until testers give me enough karma, and then push it into F13-updates. 8. Only now (not before) I can push F12-updates-testing. ... Only FESCo may say whether this is the desired workflow for package maintainers. It is certainly different from current practice. Problem #2: Removing repositories --------------------------------- Option a) in Problem #1 sounded nice, didn't it? Well, not so much. A problem arises when you disable -updates-testing repo. Either by hand (a power-user downloaded a few packages from this repo because he wanted to test new versions of his favorite software or test some bugfix, and then he disabled this repo again) or automatically (anaconda disabled -updates-testing on distribution upgrade - I don't know whether this is actually done, but anaconda developers don't know either:)). After you disable this repo, you still might have some packages installed that are newer than those in F+1 base+updates. That is a problem, we haven't saved this user from upgrade path problems in this case. Problem #3: Unpushing packages ------------------------------ Even more juicy problem exists - unpushing packages from -updates-testing. If we use Option a) in Problem #1, we still can't have any confidence in our results. Because our results will apply only in that precise moment when we check it. An hour later the corresponding package may be unpushed from F+1 updates-testing, but left intact in F updates-testing (or you already downloaded it, that's the same). And voilà, we have an upgrade path problem. Example: 1. foo-3.0 is pushed into F12-updates-testing and F13-updates-testing. 2. User installs foo-3.0 from F12-updates-testing. 3. foo-3.0 is unpushed from F13-updates-testing. 4. Upgrade path problem. Conclusion ---------- 1. We don't know exactly how upgradepath constraint is defined. 2. We have no idea what -updates-testing should be tackled. 3. We should ask FESCo what's the preferred approach. 4. Current proposed solution is: "Updates-testing is for power-users, they will handle possible problems. After we receive some info how to proceed in this matter, we can extend our code." 5. Basic base+updates repo checking seems simple and is already implemented. All common users should be safe. If you see some logical mistakes or gaps in this email, please correct it. There might be some further issues that didn't occur to me. Thanks, Kamil

2 1

[AutoQA] #226: upgradepath: crash for -updates-testing koji tag
by fedora-badges 07 Oct '10

07 Oct '10

#226: upgradepath: crash for -updates-testing koji tag --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: defect | Status: new Priority: major | Milestone: Package Update Acceptance Test Plan Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- It seems that upgradepath crashes when we try to give it *-updates-testing koji tag. {{{ [root@aqd autoqa]# autoqa post-bodhi-update --title freeciv --target-tag dist-f13-updates-testing freeciv-3.0-1.fc13 --local -t upgradepath ... 11:35:12 INFO | ERROR: Entered bad kojitag 11:35:12 ERROR| Exception escaping from test: Traceback (most recent call last): File "/usr/share/autotest/client/common_lib/test.py", line 384, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/share/autotest/client/common_lib/test.py", line 577, in _call_test_function raise error.UnhandledTestFail(e) UnhandledTestFail: Unhandled UnboundLocalError: local variable 'matching_build' referenced before assignment Traceback (most recent call last): File "/usr/share/autotest/client/common_lib/test.py", line 570, in _call_test_function return func(*args, **dargs) File "/usr/share/autotest/client/common_lib/test.py", line 279, in execute postprocess_profiled_run, args, dargs) File "/usr/share/autotest/client/common_lib/test.py", line 201, in _call_run_once self.run_once(*args, **dargs) File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71, in newf f_result = f(*args, **kwargs) #call the decorated function File "/usr/share/autotest/client/site_tests/upgradepath/upgradepath.py", line 92, in run_once self.envr_list.add(matching_build['envr']) UnboundLocalError: local variable 'matching_build' referenced before assignment }}} -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/226> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #134: ResultsDB: create resultsdb prototype
by fedora-badges 06 Oct '10

06 Oct '10

#134: ResultsDB: create resultsdb prototype ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: major | Milestone: Resultdb Component: tests | Version: 1.0 Keywords: | ----------------------+----------------------------------------------------- Create prototype of database and software which will implement dbSchema from #133 and API from #131 This prototype should also include some test suite, which will be used to determine, if all the functionality is OK. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/134> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

Re: rpmlint improvements
by Kamil Paral 04 Oct '10

04 Oct '10

----- "Alexander Todorov" <atodorov(a)redhat.com> wrote: > Hello folks, > I want to discuss topics in tickets: > https://fedorahosted.org/autoqa/ticket/149 > https://fedorahosted.org/autoqa/ticket/150 > > I did a test run of rpmlint against the latest RHEL6 tree and filed > some bugs. > Some were legitimate but others turned out to be false negatives and > developers > got angry. > > Then it turned out that rpmlint needs more features in order to be > used reliably > without making developers unhappy. So I talked to twoerner who is > rpmlint > maintainer for RHEL6 and exchanged some ideas. > > What we think is a good idea and needs to be present in upstream is > having a > rpmlint-data package which will provide rules.$pkg-name files for > packages that > need special handling. Then rpmlint will take into account those files > and act > accordingly. The rules files will be maintained by a separate > maintainer and not > by the packagers. Also any change into those rules/exceptions needs to > be > approved with a review process to avoid pkg maintainers manipulating > the rules > simply to silence out rpmlint output. This will guarantee accurate > results. > > This data package will be optional to rpmlint and not affect core > functionality. > It will most likely be different for Fedora and RHEL as well. This is > more or > less the same as proposed in ticket 149 with lintian tool for Debian. > > What do you think? Any drawbacks/gotchas you may think of? Hello Alexander, thanks for looking into this and sorry for late response. The big difference between lintian and your proposed solution is that lintian allows to specify "overrides" inside the affected packages [1] (files located at /usr/share/lintian/overrides/<package> or at debian/source/lintian-overrides), but your solution centralizes all those files in a single package rpmlint-data. I wonder what the better approach is. Per-package solution obviously gives more freedom (and trust) to individual package maintainers and is easier to manage (for them). Centralized approach is stricter (you have to ask for inclusion, review process can be easily enforced), therefore quality may be higher, on the other hand it may annoy and discourage a lot of people. ("Just another set of policies to follow.") Originally I was thinking about the per-package (lintian-like) approach. My idea is that we can still apply some policies on top of it, but it is much more convenient for package maintainers in the rest of the cases. The policy might look like: Every line whitelisting an Error message must be sponsored by some different package maintainer. On the other hand, Warning and Info messages might be whitelisted without restrictions. That's only one example of a possible policy, certainly there is plenty of solutions. As Michal Scherer mentioned [2], rpmlint already supports "config files" in which we can filter out some output (based on regexp I think). So we are already able to do this inside autoqa - just read the per-package definition file, create custom rpmlint config and run rpmlint with this config. Of course, having this feature upstream would be much better for us. The package maintainers should see the same output when they run rpmlint locally as they see when rpmlint is run inside autoqa. That is possible only when rpmlint upstream supports these whitelists definitions directly. [1] http://lintian.debian.org/manual/ch2.html#s2.4 [2] https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000861.html > > > The rules files format need to support the following (as identified so > far): > > * List file paths/directories with special permissions and uid/gid > settings. For > example CUPS has such. audit also has such (in RHEL for certification > purposes) An important note here - our proposed solution talked just about whitelisting some warning/error messages. Also lintian overrides files provide just whitelisting of output messages. What you talk about here (and what you sent in your patch [3]) is a little different. You just don't whitelist some output, you "fix" (override) the input. For example you don't just define "don't show me the line that warns me about incorrect permissions on /etc/passwd", but you also define *what the correct permissions are*. Per package. That is a little different (and somewhat harder task). I don't know whether rpmlint developers will be inclined to so big change (redefining what the correct input is). But allowing output whitelisting per-package could be easier to convince them to adopt. >From my point of view, I would rather start with simple whitelisting too. Your approach seems to bring more challenging tasks and I think it's better to start small. What do you think? [3] https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000860.html > > * Ignore some test cases for particular package(or file). For > example: > gcc.x86_64: E: devel-dependency glibc-devel > gcc.x86_64: W: devel-file-in-non-devel-package > > This looks normal since gcc is a compiler but will be an error for > another > package (e.g. an application). > > jlaska told me that Fedora QA already uses rpmlint. Have you guys > identified > other areas/tests that need special handling or exclusion ? What are > those? > > Have you thought about the file format? It needs to be easy to > maintain and easy > to read - probably just plain text. It also needs to be flexible so > that we can > add to it without breaking compatibility. We can probably have a > separate file > (format) for every test/package that rpmlint performs. > > Just to let you know that I'm happy to work on the rpmlint code and > push it > upstream if noone else has spare cycles. Your effort is greatly appreciated, thanks. > > Regards, > Alexander. > > > _______________________________________________ > autoqa-devel mailing list > autoqa-devel(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/autoqa-devel

3 3

Several small fixes
by James Laska 01 Oct '10

01 Oct '10

Greetings, Nothing urgent, just a few small patches I'd like to apply to master. Nothing noteworthy with the first two, but the third might be more interesting given recent upgradepath discussion. Kamil might already have this in his pending upgradepath changes. Thoughts/concerns welcome. > 0001-Tell-post-tree-compose-that-composes-are-enabled-for.patch The repoinfo.conf configuration value 'composes' was enabled so that post-tree-compose would know which entries to monitor for installation images. This patch turns on composes for the Branched release (F14). I've also updated the wiki documentation as well (https://fedoraproject.org/wiki/How_to_update_AutoQA_repoinfo.conf) > 0002-Correct-missing-config_loader-import.patch Not sure why this never surfaced until now, but this patch corrects a missing import from the rats_install.py script. > 0003-Enable-owner-fedoraproject.org-cc-for-upgradepath-re.patch I think this might make more sense to move out of the tests, and into lib/python/test.py, but until we do that, this patch turns on cc of test results for upgradepath. Josef also pointed out that we may want a way for tests to enable/disable sending output to maintainers. I don't have any great ideas on that, so for now, this patch just mimics our rpmlint and rpmguard by including $package-owner(a)fedoraproject.org. Thanks, James

2 5

Re: [PATCH] tidy up upgradepath test code
by Kamil Paral 01 Oct '10

01 Oct '10

----- "Kamil Páral" <kparal(a)redhat.com> wrote: > This patch introduces no new features and (hopefully) no new bugs. It > just reformats the code to be more readable and maintainable - now > the > semantics and variables are very similar to those used in rpmlint and > rpmguard tests, so it's easy to read through the code. > --- > tests/upgradepath/upgradepath.py | 99 > ++++++++++++++++++++++++++------------ > 1 files changed, 68 insertions(+), 31 deletions(-) Although this patch looks large, it is really just a reformatting of a few variables. If there are no objections against it, I'll push it tomorrow into master. Thanks, Kamil

1 1

timing-critical tests - depcheck, upgradepath
by Kamil Paral 30 Sep '10

30 Sep '10

This ticket: https://fedorahosted.org/autoqa/ticket/228 and this patch: https://fedorahosted.org/pipermail/autoqa-devel/2010-September/001195.html forced me to do some thinking. We have some tests that are really timing-critical - depcheck and upgradepath. We can't run them in an arbitrary time as we can do with rpmlint and similar. We must run them just before the packages are pushed - not earlier, not later. In other words, we are given set of packages to examine and a repository in some well-defined state. We examine the packages and the repo and say "OK" or "not OK". Someone (Bodhi) must guarantee that neither the repo nor the set of packages change between our examination and actual push of those packages. There must be no steps in between. Otherwise our result would be misleading. For upgradepath we have even stricter requirements - *no* repo must change between our examination and actual push. Now that we know the requirements, we can ask some questions: As for the ticket: Can we really rely on checking koji -pending tags, instead of communicating with Bodhi? I don't know how the -pending tags are implemented. But can we be sure that Bodhi takes exactly same set of packages in the exactly same order as we do, when it tries to push them? As for the patch: I have a feeling that for these aforementioned tests there's not really any sense in sending email reports. These tests either pass or fail, just before push. If our test fails, then the push fails and maintainers will be probably notified by Bodhi anyway, won't they? And more importantly, we currently run upgradepath when packages are *requested* for push, but the actual push might be some hours away, is that right? So the result we produce is not really valid, in the meantime a lot of things may change. That also speaks against sending some notifications. But maybe I just got some stuff wrong. What do you think?

2 1

Current state of ResultsDB
by Josef Skladanka 30 Sep '10

30 Sep '10

Hello gang! Currently, there is a instance of ResultsDB core, running at <http://test1185.test.redhat.com/wsgi/resultsdb/xmlrpc> The tested & working functionality is "storing the data into the DB" using beforementioned xmlrpc interface. At the moment, "using resultsdb as storage" is implemented in the jskladan branch in git <http://test1185.test.redhat.com/wsgi/resultsdb/xmlrpc> Precisely these patches: <http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=46b4019b7608a7947b…> <http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=e1404be74b646ec5e0…> As you can see, the only changes are in the AutoQATest base class (lib/python/test.py), and of course the API library for ResultsDB is addded (lib/python/resuldsdb.py or lib/python/logging.py [in the first patch]). The rest is simply taken care of automagically. This patchset does not make use of the phases, as these would require changes into the sources of individual tests and can be postponed until the basic functionality is in the master. But the functionality should be pretty much at the same level, as the mailing list provides (although resultsdb is not _at the moment_ storing the "outputs" variable). I have also created a wiki metadataa drafts: Testplan: https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Accepta… Testcases: https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmguard_Testcase_Meta… https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmlint_Testcase_Metad… ... As you can see, the main focus is on the Testplan part. The idea behind this wiki page is to provide a simple, yet whole description of the testplan, so we can make a frontend, which will take it, and visualize the 'progress' of the testplan for given envr (which seems to be the common denominator of all the arguments given to a test). To explain what does it all mean: * "testcases" : _stripped_ - this has two means 1) sum up all the tests covered in the testplan 2) provide 'aliases' for further use in the metadata - so we can write "Initscripts" instead of the whole URL * "testcase_classes" : ["mandatory", "introspection", "advisory"] - this value specifies testcase 'classes' or 'types' - this has utterly semantic meaning, and these can be _anything_ what you find meaningfull. The idea behind this is to be able to somehow group the test in the frontend AND to provide some means to bind the results of all the standalone tests into the "testplan result". See below. * "mandatory" : {"testcases": ["Package sanity", "Repo sanity", "Conflicts", "Upgrade path"], "pass": ["PASSED", "WAIVED"], on_fail":"FAILED"} - this says, that the ["Package sanity", "Repo sanity", "Conflicts", "Upgrade path"] tests will be considered "passed", if the result stored in ResultsDB is in ["PASSED", "WAIVED"]. - If any of theses tests have a different result, the whole testplan is considered to be "failed" (on_fail":"FAILED") * "introspection" : {"testcases": ["Rpmlint", "Initscripts"], "pass": ["PASSED", "WAIVED", "INFO"], "on_fail":"FAILED"} - the same as the mandatory tests, but also satisfies with "INFO" as a result of the test. * "advisory" : {"testcases": ["Rpmlint", "Rpmguard"], "pass": ["PASSED", "WAIVED", "INFO", "NEEDS_INSPECTION"], "on_fail":"NEEDS_INSPECTION"} - if any of these testcases' result is out of the "pass" set, mark the testplan as "NEEDS_INSPECTION". You noticed, that Rpmlint is both in the "introspecition" and "advisory" groups, this is based on the current state of the testplan <https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptan…> - I'm not really sure how to base the "no errors" and "no warnings" into that, based solely on the result, but that's not the real issue for now. This brought me to one more possible thing to think about: are the current "result states" (RUNNING, PASSED, INFO, FAILED, ABORTED, CRASHED, WAIVED, NEEDS_INSPECTION) enough? AND - are we able to sort these according to the 'urgency' or 'importance' or 'relevance' or how to call it? Let's say we have two tests in a testplan, one is ABORTED and the second is CRASHED - what would you like to see as 'overall result' Aborted or Crashed? I'm not sure, if I was able to describe it clearly (I'm really hitting the limits of my descriptive skills here :-D), so feel free to ask as many questions, as needed. Joza

2 3

← Newer
1
2
3
4
5
6
7
8
9
Older →

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel September 2010