autoqa-devel June 2011

autoqa-devel@lists.fedorahosted.org

12 participants
97 discussions

[AutoQA] #298: test.py - split postprocess_iteration reporting into standalone methods
by fedora-badges 15 Jun '11

15 Jun '11

#298: test.py - split postprocess_iteration reporting into standalone methods ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: minor | Milestone: 0.5.0 Component: core | Keywords: ----------------------+----------------------------------------------------- At the moment, postprocess_iteration() handles the whole reporting/sending results. When the run_once ends, this method takes the content of self.{result, summary, highlights, outputs} and automagically sends an email to the mailing list, creates output.log file, etc. This approach was very reasonable for tests which actually test just one thing (update/build/etc), because just a single result is to be sent. With the new tests like Depcheck and Upgradepath, we'd love to be able to force-send several results as the test proceeds - e.g. to be able to send bodhi comment to every update, but send just one overall email... So what I'd like to have is: 1) Take postprocess_iteration, and split it into standalone methods according to the 'destination' of the report (e.g. send_email(), create_output_log(), ...). These will be able to take {result, summary, highlights, outputs} parameters, which will override the 'automagicall' self.{result, summary, highlights, outputs}. I.e. if I set result parameter, summary, highlights and outputs will be filled using the 'self' variables, and so on. 2) Take the bodhi-reporting method (as used in depcheck and upgradepath), and move it to the AutoQATest class. 3) Add a method report_results(), again with the same parameters ({result, summary, highlights, outputs}) and one more (e.g. a dictionary) to control which reporting methods to call (i.e. to be able to say "send email, store in resultsdb, but do not send bodhi comment"). By default, this will call all the reporting routines. 4) In postprocess_iteration(), call the report_results(), so the 'automagical reporting' behaviour is not changed. 5) Add a variable, which will turn on/off the call of report_results() (True as default) in postprocess_iteration() (needs to be some attribute of AutoQATest class, since postprocess_iteration() does not take arguments). This is for the wrapper-writer, to be able to control if results are to be sent in the postprocess_iteration() or not (imagine Upgradepath - we will report all the results in the run_once(), and do not need postprocess iteration to actually send anything). ------------------------------- This is also preparation for resultsdb, because it solves the problem with reporting multiple results from one test. Also turning the resultsdb reporting on is a matter of adding one method, and calling it from report_results() (and adding more types of reporting [I'm looking at you fedora message bus!] in the future will be also this simple. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/298> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

Re: [AutoQA] #319: Create HTML log output if possible
by fedora-badges 15 Jun '11

15 Jun '11

#319: Create HTML log output if possible --------------------+------------------------------------------------------- Reporter: kparal | Owner: vhumpa Type: task | Status: closed Priority: major | Milestone: 0.5.0 Component: core | Resolution: fixed Keywords: | --------------------+------------------------------------------------------- Changes (by kparal): * status: new => closed * resolution: => fixed * component: documentation => core Comment: I have changed the wiki docs here: https://fedoraproject.org/w/index.php?title=Writing_AutoQA_Tests&action=his… -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/319#comment:9> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

AutoQA pkg tests (Half baked)
by Athmane Madjoudj 14 Jun '11

14 Jun '11

Hello, This is my first attempt to integrate my small test suite (currently server-oriented) into AutoQA framework, I've not yet tested running testes through autoqa, I need to install some deps, next week I'll try to check [1]. git repo: git://fedorapeople.org/~athmane/autoqa.git tested are in: tests/pkg_tests/tests [1] http://repos.fedorapeople.org/repos/fedora-qa/autoqa/ -- Athmane Madjoudj

2 2

[AutoQA] #339: update mediakit_sanity to skip boot.iso existence test or warn boot.iso's absence
by fedora-badges 14 Jun '11

14 Jun '11

#339: update mediakit_sanity to skip boot.iso existence test or warn boot.iso's absence ----------------------+----------------------------------------------------- Reporter: hongqing | Owner: Type: task | Status: new Priority: major | Milestone: Automate installation test plan Component: core | Keywords: ----------------------+----------------------------------------------------- now the existences of all images, listed in .treeinfo [images-i386] section, are tested. In DVD.iso, boot.iso is listed in .treeinfo, but actually it does not exist. so the mediakit_sanity test always fails. I think the solution is to skip the boot.iso existence test, since it is not required in DVD.iso or we can just print warning information rather than fail it. or any other ideas? -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/339> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

Re: [AutoQA] #316: depcheck: extract possible cause of failure
by fedora-badges 10 Jun '11

10 Jun '11

#316: depcheck: extract possible cause of failure -------------------------+-------------------------------------------------- Reporter: kparal | Owner: jskladan Type: enhancement | Status: closed Priority: major | Milestone: 0.5.0 Component: tests | Resolution: fixed Keywords: | -------------------------+-------------------------------------------------- Changes (by kparal): * status: assigned => closed * resolution: => fixed Comment: Pushed into master as 8a2b9ce2410009da8a4715ea0e33342ef46d4b2f . This doesn't need any documentation changes I believe, because it will be covered in ticket #326. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/316#comment:3> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Re: [AutoQA] #318: Provide access to test documentation
by fedora-badges 10 Jun '11

10 Jun '11

#318: Provide access to test documentation ---------------------------+------------------------------------------------ Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.5.0 Component: documentation | Resolution: Keywords: | ---------------------------+------------------------------------------------ Changes (by kparal): * component: core => documentation Comment: Pushed into master as 8a2b9ce2410009da8a4715ea0e33342ef46d4b2f . We could probably mention this in our wiki documentation ("if you create your own test, you can provide some documentation for it at <url>..."). -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/318#comment:3> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Re: [AutoQA] #319: Create HTML log output if possible
by fedora-badges 10 Jun '11

10 Jun '11

#319: Create HTML log output if possible ---------------------------+------------------------------------------------ Reporter: kparal | Owner: vhumpa Type: task | Status: new Priority: major | Milestone: 0.5.0 Component: documentation | Resolution: Keywords: | ---------------------------+------------------------------------------------ Changes (by kparal): * component: core => documentation Comment: Pushed into master as 8a2b9ce2410009da8a4715ea0e33342ef46d4b2f . This will definitely need some documentation adjustments. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/319#comment:8> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

[PATCH] pretty logs
by Kamil Paral 10 Jun '11

10 Jun '11

This patch involves tickets: https://fedorahosted.org/autoqa/ticket/315 - Create per-item logs for multi-item tests https://fedorahosted.org/autoqa/ticket/316 - depcheck: extract possible cause of failure https://fedorahosted.org/autoqa/ticket/318 - Provide access to test documentation https://fedorahosted.org/autoqa/ticket/298 - test.py - split postprocess_iteration reporting into standalone methods https://fedorahosted.org/autoqa/ticket/319 - Create HTML log output if possible and it is the result of my, jskladan's and vhumpa's work. I'll the describe the patch in detail here, but I decided not to attach it nor to post it into reviewboard, because it is simply too large. Also I expect that it will receive a little more polish in the following days, so it's not absolutely finalized yet. But if you see anything you would like to change, please, send your comments. You can display the patch by command: $ git fetch $ git diff origin/master..origin/pretty -M Most important changes: 1. TestDetail class All the test's outcomes are now stored in a single class TestDetail. It works as a simple container for test result, summary, output, highlights and a few additional information. It is easier now to call methods that operate on these information, you just pass them your TestDetail instance. By default every test has a default "main" TestDetail instance created in self.detail. You can access self.detail to change the values of result, summary, etc. For a simple single-item tests (like rpmlint), this main TestDetail instance is all they need. For complex multi-item tests (like upgradepath) you need to create a TestDetail instance for every "report" you want to create. In the case of upgradepath and depcheck this means creating separate TestDetail for every Bodhi update tested. 2. TestDetail.update_result() method TestDetail class remembers list of result keywords sorted by importance. If you use TestDetail.update_result(result) method, it will set the test result only if your provided test result is more important (i.e. "worse") then the current test result. This way you can easily handle all error states inside your script and only as the last line include: self.detail.update_result('PASSED') 3. self.log() method We used to do a pretty complicated tasks when printing and saving some output. E.g. you would print in out on the stderr, and then you would append it to self.outputs and maybe even self.highlights. Now you just use: self.log(message, stderr=True, highlight=True) For multi-item tests (like upgradepath) you can also use self.log() to print to several TestDetail instances at once. 4. automatic log creation, different terminology We now have several types of logs (debug, output.log and pretty log). We want to have them transparent and simple for test authors. Debug log is just for us, the developers, so we don't mention it. The former output.log was renamed to full.log and we mention it as little as possible, preferably no at all. And the pretty log is referenced simply as "log" or "test log" and that's the only one we hope test authors will work with. Full log (full.log) is created automatically at the end of every test run. It contains everything that has been logged via self.log() method (provided that @param printout==True). It does not contain output of "print foo" messages (that's just in the debug log). Test log (aka pretty log) is also created automatically at the end of every test run and it's populated from main TestDetail instance (self.detail). Except for the case when the test author has created some test log already manually, then we don't create another one automatically. Test log is created in HTML and this file is used for reporting results everywhere (it is either sent or linked). 5. self.post_results() method There is now a single method for reporting results. It takes care of everything - creating log file, sending email, sending opt-in email and sending Bodhi comment. The first two are mandatory and the last two are optional - you have to provide correct parameters. By default the results are posted automatically at the end of the test run. It means for simple tests where you don't need anything fancy (like Bodhi comments or opt-in emails), you don't have to care about reporting results, everything is handled for you. For more advanced uses you have to call the self.post_results() method. For multi-items tests like upgradepath or depcheck you'll need to call self.post_results(test_detail) for every "item" (Bodhi update in our case) you want to report results for. 6. Test logs are in HTML Test logs are now in HTML. It will allow us to do pretty formatting, nice highlighting, etc. When sending as an email, only a plaintext overview header and a link to proper HTML log is included in the email body. Example: http://kparal.fedorapeople.org/autoqa/upgradepath2.html 7. All tests modified All tests were re-written to the new architecture. Most of them don't take any advantage of the new log features yet. rpmlint example: http://kparal.fedorapeople.org/autoqa/rpmlint.html Upgradepath and depcheck received more love and they should creater nice per-update logs with just the relevant info included. Depcheck now tries to filter out irrelevant messages and keep only the interesting ones in the log. upgradepath example: http://kparal.fedorapeople.org/autoqa/upgradepath.html depcheck example: http://kparal.fedorapeople.org/autoqa/depcheck.html We were working on it hard even today. Some code may be still rough. All comments welcome. Thanks, Kamil

2 4

Unit Testing Dependencies for AutoQA
by Tim Flink 10 Jun '11

10 Jun '11

Kamil asked a good question in his review of the code for comment spam reduction that made me realize that I haven't done enough explaining of how the unit test code is currently working. The question was whether python-virtualenv and python-pip (required for running 'make test') should be listed as build dependencies in autoqa.spec. My feeling on this is 'probably not' because of the way that those dependencies are being used. I'm open to other ideas, though. The current way that I have implemented 'make test' does the following: - create virtualenv if it doesn't already exist * install required packages (py.test etc.) if the env is being created - activate virtualenv - run tests - deactivate virtualenv The main reason that I wrote it this way is that an appropriate version of py.test still hasn't been reviewed for inclusion in Fedora and the version that does exist is not new enough to run these tests. I dropped the ball on that one and once we get 0.5.0 released, I will resume work on getting the needed packages reviewed so that they get into rawhide before F16 branch. I'm not sure that we want to be using virtualenv or pip during rpm packaging. I think that I would rather see direct execution of the tests during rpm build and I'm not even sure that the use of virtualenv for 'make test' is acceptable for Fedora packaging standards. Then again, AutoQA isn't a Fedora package yet so maybe we don't want to worry about this now. Thoughts? Tim

2 6

[PATCH] AutoQA Comment Spam Reduction
by Tim Flink 10 Jun '11

10 Jun '11

This patch involves fixes for tickets in preparation for 0.5.0 release: * https://fedorahosted.org/autoqa/ticket/314 - Decrease the volume of 'PASSED' email sent to maintainers from bodhi * https://fedorahosted.org/autoqa/ticket/258 - implement 'make test' This is a smaller change than the pretty log stuff, so I created a review request in reviewboard based off of a diff from master: - https://fedorahosted.org/reviewboard/r/149/ Otherwise, you can check the diff from master using the commands: $ git fetch $ git diff origin/master..origin/tflink -M Important changes: 1. Addition of bodhi_update_state: 2. bodhi_utils._is_bodhi_comment_email_needed() 3. Adding a parameter to bodhi_post_testresult() 4. [bodhi_email] section added to autoqa.conf 5. Tests for the code associated with #314 ------------------------------------------------------------ | Addition of bodhi_update_state: ------------------------------------------------------------ This object encapsulates information on the state of tests for a given bodhi update. The idea was to encapsulate the logic and make it relatively easy to understand. ------------------------------------------------------------ | bodhi_utils._is_bodhi_comment_email_needed() ------------------------------------------------------------ This was added to bodhi_utils to encapsulate the logic behind whether or not to send an email. _parse_result_from_comment() and _get_bodhi_email_config() were added to support this method ------------------------------------------------------------ | Adding a parameter to bodhi_post_testresult() ------------------------------------------------------------ Since the state of an update's tests depends on the koji tag currently being tested (upgradepath doesn't run on *-updates-testing), the current koji tag became an important part of determining whether or not to send an email. ------------------------------------------------------------ | [bodhi_email] section added to autoqa.conf ------------------------------------------------------------ To support the requirement of changing email behavior without code modifications, more configuration settings were needed. Adding a new section to autoqa.conf was deemed to be the easiest and most self-contained method of getting this done. ------------------------------------------------------------ | Tests for the code associated with #314 ------------------------------------------------------------ This kind of code is a poster child for unit tests, in my mind. Relatively simple code that has subtle changes in logic based on many changing inputs. The tests are written for py.test > 2.0.1 and are in lib/autoqa/tests/

2 7

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel June 2011