Re: rpmlint improvements
by Kamil Paral
----- "Alexander Todorov" <atodorov(a)redhat.com> wrote:
> Kamil Paral wrote:
> > A. From my opinion it's pretty obvious I would like to see one
> config file
> > per package, distributed in that package -- the "lintian way". Those
> config
> > files could be installed into /usr/share/rpmlint/whitelist/<package>
> or
> > similar location. We can already use this approach for autoqa -- we
> can
> > extract this file from the binary package (or similarly named file
> for
> > the source package) and provide it with --file option to rpmlint.
> But, it
> > would be very beneficial if upstream rpmlint also gained these
> capabilities,
> > the maintainers would then see the same output on their localhost as
> the
> > output provided by autoqa. So, convincing rpmlint upstream to
> support these
> > lintian-style per-package config files would be absolutely great.
>
> Can you explain? Do you mean that rpmlint will read its default config
> +
> /usr/share/rpmlint/whitelist/<package> ?
Yes, I would love to if rpmlint has internal support for per-package
config file (the actual path doesn't matter). There are some advantages:
1. rpmlint can use it even if you don't have that package installed, it
simply extracts it from the rpm file.
2. You can be sure that only relevant config file gets loaded for your
package. That means you don't waste performance on loaded several
thousand other config files and also you can be sure that no other
config file by accident influences your results.
3. You surely don't want to have several thousand *.config files directly
in /etc/rpmlint, it would be a mess. At least one more dir is needed
to keep it separated.
4. These files should be stored in /etc at all I believe. They are not
config files, which a system administrator would like to change.
They are more like a definition file. Those file will be changed only
by the package maintainer and will be distributed with next package
release. They should not be in /etc.
5. Package maintainers get the same output on their machines as we do
on ours.
> Currently it does read the
> default
> config + /etc/rpmlint/*config which is a bit different but can serve
> the same
> functionality if whitelisting filters begin with the package name.
I'm afraid it is not so. Take this example:
addFilter('audit.* /etc/audisp/plugins.d 0750')
This will filter not only results for audit package, but also for audit-libs
(same source package) and audit-viewer (different source package).
I think regular expressions are not enough in this case, you will get
similar collisions all the time. Also, it would be hell to manage.
13 years, 6 months
Re: distinguishing ABORTED and CRASHED
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> On Fri, 2010-10-08 at 09:56 -0400, Kamil Paral wrote:
> > I would like to reach a consensus how ABORTED and CRASHED test
> status
> > should be used. My idea is this:
> >
> > CRASHED should be used for programming errors. Any fault that is
> caused
> > by our code should be reported as CRASHED.
> >
> > ABORTED, otoh, should be used for irrevocable failures of
> third-parties.
> > Every time I catch an exception that is not caused by our code, but
> > for example it is a network failure, external script crash, etc,
> and
> > there's no point in continuing the test, it should be reported as
> ABORTED.
> >
> > Both statuses mean a failure for us, but we will be easily able to
> > distinguish our failures and other services' failures.
> >
> > Together with this proposal I would change our exception catcher
> this way:
> >
> > Status is set to CRASHED on any uncaught Exception, with one
> > exception -- when error.JobComplete is raised and uncaught, we
> would
> > not override the status.
> >
> > That will allow easily end the test immediately (deep in the call
> stack)
> > when some external party fails. We just set self.result to ABORTED
> and
> > call raise error.JobComplete.
> >
> > What do you think?
>
> Certainly makes sense. I'm having a hard time thinking about how
> you'd
> instrument the code to distinguish between the two, but I don't think
> that's needed for me to give a +1 :)
>
> Can you give some more examples to help distinguish CRASHED and
> ABORTED
> results?
The intention is, that if we see an ABORTED test we can just waive our
hand and say "ah well, something (koji,bodhi,etc) failed, let's re-run
the test" and we don't have to examine the failure closer. Because we will
know that that failure was not our fault. Only if the failure happens again,
then we will investigate more.
On the other hand, when we see a CRASHED test, we will have to inspect it,
because it might be a bug in our code.
So it's just meant to simplify the maintenance.
Code example is simple. By default all uncaught exceptions (e.g. network
error) mark the test as CRASHED. If you want different behavior, you must
wrap the corresponding code like:
try:
//download from koji
except IOError: //or other error
self.status = "ABORTED"
raise
You don't have to do it, then it will be reported as CRASHED.
I'm not sure I've answered your question, have I?
(We may also want to add some detailed description to that exception. I'm
not Python guru, but it should be possible. That's just an implementation
matter.)
13 years, 6 months
Re: rpmlint improvements
by Kamil Paral
----- "seth vidal" <skvidal(a)fedoraproject.org> wrote:
> On Thu, 2010-10-07 at 09:53 -0400, Kamil Paral wrote:
>
> > Some technical details now:
> >
> > A. From my opinion it's pretty obvious I would like to see one
> config file
> > per package, distributed in that package -- the "lintian way". Those
> config
> > files could be installed into /usr/share/rpmlint/whitelist/<package>
> or
> > similar location. We can already use this approach for autoqa -- we
> can
> > extract this file from the binary package (or similarly named file
> for
> > the source package) and provide it with --file option to rpmlint.
> But, it
> > would be very beneficial if upstream rpmlint also gained these
> capabilities,
> > the maintainers would then see the same output on their localhost as
> the
> > output provided by autoqa. So, convincing rpmlint upstream to
> support these
> > lintian-style per-package config files would be absolutely great.
> >
> >
>
> one config file per pkg? or one per branch per package?
>
> Would it make since to have:
>
> fedpkg rpmlint
>
> which checks the pkg in the branch you're currently in
>
> using .rpmlint as the config for that pkg, if it exists.
>
> and then have autoqa draw the .rpmlint file from the git branch it is
> building in the same way?
>
> then the pkger has control over it and if there is a different
> maintainer for el5 vs f13 they don't have to quibble over rules.
>
> thoughts?
> -sv
Oh, this starts to be really complicated :) My idea was one config
file per binary package.
I have no experience with package maintenance, but different Fedora
releases of the same package are represented just by different git
branches, right? So if that need occurs, there should be no problem
in keeping different config file versions for f13 and el5 branch,
am I right?
The main difference is just that I propose to include that file directly
in that binary package. That means AutoQA doesn't have to download
it from anywhere, and (more importantly) arbitrary rpmlint run
produces the same result as our AutoQA run -- of course just if this
feature is supported in upstream rpmlint.
Did I get your comment right?
13 years, 6 months
distinguishing ABORTED and CRASHED
by Kamil Paral
I would like to reach a consensus how ABORTED and CRASHED test status
should be used. My idea is this:
CRASHED should be used for programming errors. Any fault that is caused
by our code should be reported as CRASHED.
ABORTED, otoh, should be used for irrevocable failures of third-parties.
Every time I catch an exception that is not caused by our code, but
for example it is a network failure, external script crash, etc, and
there's no point in continuing the test, it should be reported as ABORTED.
Both statuses mean a failure for us, but we will be easily able to
distinguish our failures and other services' failures.
Together with this proposal I would change our exception catcher this way:
Status is set to CRASHED on any uncaught Exception, with one
exception -- when error.JobComplete is raised and uncaught, we would
not override the status.
That will allow easily end the test immediately (deep in the call stack)
when some external party fails. We just set self.result to ABORTED and
call raise error.JobComplete.
What do you think?
13 years, 6 months
Re: rpmlint improvements
by Kamil Paral
----- "Alexander Todorov" <atodorov(a)redhat.com> wrote:
> Hi all,
> finally I've had the opportunity to play with additional rpmlint
> config files.
> Current version will read all files that match /etc/rpmlint/*config.
> See
> https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000861.html
> for more info.
>
> If we put our configuration there then the changes are in effect on
> the next
> rpmlint run. Attached is a config file for the audit package which
> will filter
> out all errors/warnings about non-standard permissions. I've tested it
> and it
> works well.
>
>
> It looks like what's left to do on our side is to scan all packages
> and decide
> what we want to filter and what is a real error. Also decide between
> centralized
> vs. per package configuration and start creating those config files
> with help
> from devel.
>
> My vote is for the centralized approach and the exclusion rules be
> administered
> by QE folks. What are your thoughts?
Hello Alexander,
thanks for your work. My opinion is similar to jlaska's.
1. If some is going to be a central authority, I don't believe it should
be QA. We don't have the manpower for doing this, nor the required
knowledge (at least I don't have it -- I'm not a package maintainer and
I won't be able to decide for many requests whether they are valid or not).
This whole process seems to me very similar to package review process,
so packaging team could be better suited for this. Anyone, I'm sure FESCo
would have a word in this.
2. I would vote for the permissive way rather than restrictive way.
Permissive way is the way of Fedora. Compare it to the package review.
There is a package review in the beginning, but after that noone really
approves your changes, so you can change practically anything, even
contrary to the guidelines. Someone can report you a bug against your
spec file, but that's it. We just rely on good behavior and dutifulness
of the maintainer. It seems weird to me to have one process with all
that freedom and another process (rpmlint output whitelisting) really
restricted.
3. To ensure people don't hide problems with simply whitelisting every
error message, a document with proper guidelines should be created.
Maintainers then can refer to these guidelines or ask in devel-list.
Still, there may be some maintainers who simply ignore the guidelines
and whitelist everything, sure. In this case, we can do regular/irregular
random/all-packages review and check whether all the whitelist definitions
are valid. (Actually I would love to see this process even for package
review, every package's spec file should be re-evaluated every few years.)
That surely needs some manpower, yes. Probably a similar manpower to the
centralized approach. The advantage is that a lack of manpower does not
block the process, and also we want to keep the maintainers happy with
Fedora. Centralized approaches usually don't achieve that. Surely we won't
have everything 100% correct this way. But things never are 100%.
Some technical details now:
A. From my opinion it's pretty obvious I would like to see one config file
per package, distributed in that package -- the "lintian way". Those config
files could be installed into /usr/share/rpmlint/whitelist/<package> or
similar location. We can already use this approach for autoqa -- we can
extract this file from the binary package (or similarly named file for
the source package) and provide it with --file option to rpmlint. But, it
would be very beneficial if upstream rpmlint also gained these capabilities,
the maintainers would then see the same output on their localhost as the
output provided by autoqa. So, convincing rpmlint upstream to support these
lintian-style per-package config files would be absolutely great.
To close it up -- by following the KISS principle, we should start small.
We may have different ideas on centralized/distributed maintenance, but we
still need the plumbing, either way. We should discuss our needs with rpmlint
upstream in the first place. Then we can start a test round by asking
interested maintainers to create their whitelist files. Based on the result
we can then decide what the future approach should be.
13 years, 6 months
upgradepath and updates-testing - summary of problems
by Kamil Paral
This is a summary of all problems related to upgradepath test and the
updates-testing repo.
First of all, upgrade path requirement is not properly defined anywhere.
The best what I could find is this:
https://fedoraproject.org/wiki/Package_maintainer_responsibilities#Miscel...
That means we work mainly on assumptions.
For base and -updates repo, this is not a big problem, the task is pretty
straightforward:
Algorithm example
-----------------
Repo set RS = base + updates
For every Fedora release F:
For every package P in RS of F:
NVR of P <= NVR of P in RS of F+1
We must check this constraint for every pending package update, right before
it is pushed.
We have this already implemented, and when it is enforced, we will guard
all the common users (common user == using just base + updates repo set)
against upgrade path problems.
For -updates-testing repo however, things are not as easy.
Problem #1: What is the desired repo set?
-----------------------------------------
In the algorithm example, repo set of F and repo set of F+1 were equal (base +
updates repos). How does repo set look like when -updates-testing is in place?
Do we want to check it this way?
a) F base + updates + updates-testing => F+1 base + updates + updates-testing
or this way?
b) F base + updates + updates-testing => F+1 base + updates
Option a) is simple, we use the same approach as in the algorithm example. But
it may not really guard the user, see following problems.
Option b) guards the user well, but it has some serious drawbacks. It basically
says that you must first push into F+1-updates before you can push to
F-updates-testing. That can have a large impact on package maintainers workflow.
If you consider that it may take one or two weeks before your update is pushed
into -updates, then it may a whole month or more before your update propagates
from Rawhide to Rawhide-3.
Example:
1. In all repos there is foo-2.0.
2. I want to push foo-3.0 into Rawhide, F14, F13 and F12.
3. First I must push to Rawhide.
4. Once it is in Rawhide, I can push to F14-updates-testing.
5. Now I must wait until testers give me enough karma, and then push it into
F14-updates.
6. Only now (not before) I can push F13-updates-testing.
7. Now I must wait until testers give me enough karma, and then push it into
F13-updates.
8. Only now (not before) I can push F12-updates-testing.
...
Only FESCo may say whether this is the desired workflow for package
maintainers. It is certainly different from current practice.
Problem #2: Removing repositories
---------------------------------
Option a) in Problem #1 sounded nice, didn't it? Well, not so much.
A problem arises when you disable -updates-testing repo. Either by hand
(a power-user downloaded a few packages from this repo because he wanted
to test new versions of his favorite software or test some bugfix, and then
he disabled this repo again) or automatically (anaconda disabled
-updates-testing on distribution upgrade - I don't know whether this is
actually done, but anaconda developers don't know either:)).
After you disable this repo, you still might have some packages installed
that are newer than those in F+1 base+updates. That is a problem, we haven't
saved this user from upgrade path problems in this case.
Problem #3: Unpushing packages
------------------------------
Even more juicy problem exists - unpushing packages from -updates-testing.
If we use Option a) in Problem #1, we still can't have any confidence in
our results. Because our results will apply only in that precise moment when
we check it. An hour later the corresponding package may be unpushed from
F+1 updates-testing, but left intact in F updates-testing (or you already
downloaded it, that's the same). And voilà, we have an upgrade path problem.
Example:
1. foo-3.0 is pushed into F12-updates-testing and F13-updates-testing.
2. User installs foo-3.0 from F12-updates-testing.
3. foo-3.0 is unpushed from F13-updates-testing.
4. Upgrade path problem.
Conclusion
----------
1. We don't know exactly how upgradepath constraint is defined.
2. We have no idea what -updates-testing should be tackled.
3. We should ask FESCo what's the preferred approach.
4. Current proposed solution is: "Updates-testing is for power-users, they
will handle possible problems. After we receive some info how to proceed
in this matter, we can extend our code."
5. Basic base+updates repo checking seems simple and is already implemented.
All common users should be safe.
If you see some logical mistakes or gaps in this email, please correct it.
There might be some further issues that didn't occur to me.
Thanks,
Kamil
13 years, 6 months
[AutoQA] #226: upgradepath: crash for -updates-testing koji tag
by fedora-badges
#226: upgradepath: crash for -updates-testing koji tag
--------------------+-------------------------------------------------------
Reporter: kparal | Owner: kparal
Type: defect | Status: new
Priority: major | Milestone: Package Update Acceptance Test Plan
Component: tests | Version: 1.0
Keywords: |
--------------------+-------------------------------------------------------
It seems that upgradepath crashes when we try to give it *-updates-testing
koji tag.
{{{
[root@aqd autoqa]# autoqa post-bodhi-update --title freeciv --target-tag
dist-f13-updates-testing freeciv-3.0-1.fc13 --local -t upgradepath
...
11:35:12 INFO | ERROR: Entered bad kojitag
11:35:12 ERROR| Exception escaping from test:
Traceback (most recent call last):
File "/usr/share/autotest/client/common_lib/test.py", line 384, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 577, in
_call_test_function
raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled UnboundLocalError: local variable
'matching_build' referenced before assignment
Traceback (most recent call last):
File "/usr/share/autotest/client/common_lib/test.py", line 570, in
_call_test_function
return func(*args, **dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 279, in
execute
postprocess_profiled_run, args, dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 201, in
_call_run_once
self.run_once(*args, **dargs)
File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71,
in newf
f_result = f(*args, **kwargs) #call the decorated function
File "/usr/share/autotest/client/site_tests/upgradepath/upgradepath.py",
line 92, in run_once
self.envr_list.add(matching_build['envr'])
UnboundLocalError: local variable 'matching_build' referenced before
assignment
}}}
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/226>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 6 months
[AutoQA] #235: Use of exceptions
by fedora-badges
#235: Use of exceptions
-----------------------+----------------------------------------------------
Reporter: kparal | Owner:
Type: task | Status: new
Priority: major | Milestone: Outdated documentation
Component: docs/wiki | Version: 1.0
Keywords: |
-----------------------+----------------------------------------------------
Use of exceptions - AutoQA tests may no longer raise exceptions
arbitrarily (like TestFail etc). Any exception raised means that the test
will end immediately and will be reported as CRASHED. If the tests
finishes fine (it doesn't matter if it passed or failed, it just matters
it has been executed fully), no exception must be raised.
We should fix the docs.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/235>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 6 months