Improving the testability of AutoQA
by Tim Flink
As I've been working on the email reduction feature, I've been rather
frustrated with how difficult it is to test AutoQA as a framework. I've
written scripts to make testing depcheck (minus commenting and build
gathering) easier but I have yet to find a reasonable way to exercise
the whole AutoQA stack other than running jobs on my dev system with
comments disabled and hope that they finish before production does.
Before I go farther, I realize that what I'm discussing would be a
non-trivial amount of work and I'm kind of thinking that it would be a
little too much for 0.5.0. Then again, more testing could help improve
the release. Thoughts on that part would be welcome.
I can think of two methods that we could use to make testing easier and
start down the road of retiring our mascot:
- Refactor bodhi_utils and koji_utils so that they can be stubbed out
in testing
- Find an alternative to the production instances of koji and bodhi
Either way, setting up the conditions for self-tests (making mock rpms,
setting up the metadata, creating repos) is going to be a decent amount
of work and will likely end up with fragile, high maintenance tests if
we go beyond basic smoke tests.
There were concerns over the refactoring I proposed for koji_utils a
while back so I assume that isn't the way we want to proceed for now.
I looked into setting up test instances of bodhi and koji but I'm rather
intimidated by the amount of maintenance and hacking that would be
required to induce the conditions that we would need for more complete
testing on a regular basis.
With this in mind, I started hacking at a mockup for replacing bodhi and
koji for testing purposes that does nothing more than implement the
parts of those interfaces that we're using in AutoQA. It took a little
while to reverse engineer the interfaces to bodhi and koji but I do have
some code that is able to fool the koji client with hard-coded results
(haven't tried it with AutoQA yet nor have I gotten very far on mocking
up bodhi). I can send out code if anyone wants to see it (kind of ugly
ATM, though).
While finishing the mockup would also be non-trivial, I think that it
will be more manageable and flexible in the long run in addition to
being better suited to our needs since we can add in testability
features as needed without having to bug other teams about features that
other people may never use.
Anyhow, thoughts on the concepts? On the timing (try to have some for
0.5.0 or not)? Anything that I didn't consider?
Tim
12 years, 4 months
A puzzle about the ways of adding repos when installing
by Tao Wu
Hi folks,
I'm trying to build a hard drive auto installation script but have a
puzzle on the several methods of locating repo.
As we know, there are at least 3 ways to locate the iso image:
1. add the boot argument 'askmethod', then choose the location when
anaconda prompts, like:
https://fedoraproject.org/wiki/QA/TestCases/InstallSourceHardDrive
2. add the boot argument 'repo=hd:<device>:/<path>', like:
http://fedoraproject.org/wiki/Anaconda/Options#repo
3. write the repo location into kickstart file, like:
https://fedoraproject.org/wiki/Anaconda/Kickstart#install
4. add the repo graphically (this method does not support using iso
file), like:
https://fedoraproject.org/wiki/QA:Testcase_Ftp_Repository
Can someone tell me that are these methods the same to anaconda, and if
they are different, which one can be treat as the third one(kickstart
method)?
Thanks!
--
Best Regards,
Tao Wu
12 years, 4 months
A puzzle about the ways of adding repos when installing
by Tao Wu
Hi folks,
I'm trying to build a hard drive auto installation script but have a
puzzle on the several methods of locating repo.
As we know, there are at least 3 ways to locate the iso image:
1. add the boot argument 'askmethod', then choose the location when
anaconda prompts, like:
https://fedoraproject.org/wiki/QA/TestCases/InstallSourceHardDrive
2. add the boot argument 'repo=hd:<device>:/<path>', like:
http://fedoraproject.org/wiki/Anaconda/Options#repo
3. write the repo location into kickstart file, like:
https://fedoraproject.org/wiki/Anaconda/Kickstart#install
4. add the repo graphically (this method does not support using iso
file), like:
https://fedoraproject.org/wiki/QA:Testcase_Ftp_Repository
Can someone tell me that are these methods the same to anaconda, and if
they are different, which one can be treat as the third one(kickstart
method)?
Thanks!
--
Best Regards,
Tao Wu
12 years, 4 months
dealing with i686 builds
by Tim Flink
One of the things that I haven't dealt with yet for email reduction is
i686 packages.
At the moment, I assume that there are 2 or 3 tests that will be run
depending on the koji tag:
- depcheck i386
- depcheck x86_64
- upgradepath noarch (not run for updates-testing)
These 2 or 3 tests are used to determine the state of an update's tests
and that state is used as part of the decision process to send an email
or not.
I'm trying to figure out the best way to deal with i686 packages and the
things that I have come up with so far are:
- Ignore them and accept that emails will not be sent for those updates
- Interface with bodhi/koji to determine whether an update has i686
builds and adjust the state determination as needed
I remember Kamil found a list of i686 packages (but can't seem to find
the email at the moment) but I'll find the list of updates that would be
affected by not handling i686 builds.
Any other thoughts on how to go about doing this or preferences on one
over the other?
Tim
12 years, 4 months
Release Readyness
by Tim Flink
Well, it's been 2 weeks since the last time this question was asked and
it's time to do it again:
Are we ready for release?
My vote is going to be for no again but I'm also wondering if we should
re-evaluate next week instead of 2 weeks from now. I think that the
email reduction feature is getting close to being done and it sounds
like the logging feature is also getting close.
Thoughts?
Tim
12 years, 4 months
[AutoQA] #271: Create autotestd systemd.service file for F15 (and beyond)
by fedora-badges
#271: Create autotestd systemd.service file for F15 (and beyond)
-----------------------+----------------------------------------------------
Reporter: jlaska | Owner:
Type: task | Status: new
Priority: major | Milestone: Package Update Acceptance Test Plan
Component: packaging | Keywords:
-----------------------+----------------------------------------------------
07:41:40 Viking-Ice: jlaska: nr.1 upstream man pages nr.2 comparing
converting
https://fedoraproject.org/wiki/User:Johannbg/QA/Systemd/compatability (
The Fedora autotest-server package includes a sysVinit script. A
systemd.service file will be needed for f15 and beyond. Guidance from
Viking-Ice included below ...
{{{
07:41:40 Viking-Ice: jlaska: nr.1 upstream man pages nr.2 comparing
converting
https://fedoraproject.org/wiki/User:Johannbg/QA/Systemd/compatability (
you can side by side view them ) nr.3 scim through
Lennart's blog
07:42:36 jlaska: Viking-Ice: perfect, just the bread-crumb trail I
needed. Thank you
07:43:04 * jlaska goes to create a ticket with these instructions (need
to eventually create an autotestd.service)
07:44:02 Viking-Ice: I deliberately designed the wiki page so those
wanting to convert could side by side compare what already had been
converted for you
know ah.. they solve this problem that way moment..
07:45:14 <-- juhp [~petersen(a)66.187.239.10] has quit (Ping timeout: 240
seconds)
07:46:51 Viking-Ice: jlaska: throw this link into your pool
http://0pointer.de/blog/projects/systemd-for-admins-3.html
07:48:21 Viking-Ice: jlaska: what he mentioned there is the skeleton to
be built upon as inn the first example of the service file. Always start
like that
then gradually build on top of that if needed
}}}
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/271>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
12 years, 4 months
