New subject: distinguishing ABORTED and CRASHED

11 Oct 2010


      ----- "James Laska" jlaska@redhat.com wrote:
...
On Fri, 2010-10-08 at 09:56 -0400, Kamil Paral wrote:
...
I would like to reach a consensus how ABORTED and CRASHED test
status
...
should be used. My idea is this:
CRASHED should be used for programming errors. Any fault that is
caused
...
by our code should be reported as CRASHED.
ABORTED, otoh, should be used for irrevocable failures of
third-parties.
...
Every time I catch an exception that is not caused by our code, but
for example it is a network failure, external script crash, etc,
and
...
there's no point in continuing the test, it should be reported as
ABORTED.
...
Both statuses mean a failure for us, but we will be easily able to
distinguish our failures and other services' failures.
Together with this proposal I would change our exception catcher
this way:
...
Status is set to CRASHED on any uncaught Exception, with one 
  exception -- when error.JobComplete is raised and uncaught, we
would
...
not override the status.
That will allow easily end the test immediately (deep in the call
stack)
...
when some external party fails. We just set self.result to ABORTED
and
...
call raise error.JobComplete.
What do you think?
Certainly makes sense.  I'm having a hard time thinking about how
you'd
instrument the code to distinguish between the two, but I don't think
that's needed for me to give a +1 :)
Can you give some more examples to help distinguish CRASHED and
ABORTED
results?
The intention is, that if we see an ABORTED test we can just waive our
hand and say "ah well, something (koji,bodhi,etc) failed, let's re-run
the test" and we don't have to examine the failure closer. Because we will
know that that failure was not our fault. Only if the failure happens again,
then we will investigate more.
On the other hand, when we see a CRASHED test, we will have to inspect it,
because it might be a bug in our code.
So it's just meant to simplify the maintenance.
Code example is simple. By default all uncaught exceptions (e.g. network
error) mark the test as CRASHED. If you want different behavior, you must
wrap the corresponding code like:
try:
  //download from koji
except IOError: //or other error
  self.status = "ABORTED"
  raise
You don't have to do it, then it will be reported as CRASHED.
I'm not sure I've answered your question, have I?
(We may also want to add some detailed description to that exception. I'm
not Python guru, but it should be possible. That's just an implementation
matter.)

Re: distinguishing ABORTED and CRASHED