stage: hanging jobs

[Kamil Paral]

I noticed today that we have about 50 queued x86_64/noarch depchecks on our staging
server. The problem was in this job:

http://autoqa-stg.fedoraproject.org/results/2335-autotest/virt27.qa/debug/

It was hanging for two days in the "running" state. This seems like an autotest
problem, it should detect when connection times out and it should abort the job
automatically. (But now I'm not sure if we didn't make some adjustments the
autotest's watchdog timer, we need to investigate).

I aborted the hanging job and all but the last two depchecks.

If this happens again we should investigate and report to autotest devs.

[Tim Flink]

On Mon, 28 Nov 2011 09:20:05 -0500 (EST)
Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  I noticed today that we have about 50 queued x86_64/noarch depchecks
>  on our staging server. The problem was in this job:
>  
>  http://autoqa-stg.fedoraproject.org/results/2335-autotest/virt27.qa/debug/
>  
>  It was hanging for two days in the "running" state. This seems like
>  an autotest problem, it should detect when connection times out and
>  it should abort the job automatically. (But now I'm not sure if we
>  didn't make some adjustments the autotest's watchdog timer, we need
>  to investigate).
>  
>  I aborted the hanging job and all but the last two depchecks.
>  
>  If this happens again we should investigate and report to autotest
>  devs. 
Have you seen the same thing in production? We upgraded to the newer
version of autotest last week.

Tim

[Kamil Paral]

>  On Mon, 28 Nov 2011 09:20:05 -0500 (EST)
>  Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  
>  > I noticed today that we have about 50 queued x86_64/noarch
>  > depchecks
>  > on our staging server. The problem was in this job:
>  > 
>  > http://autoqa-stg.fedoraproject.org/results/2335-autotest/virt27.qa/debug/
>  > 
>  > It was hanging for two days in the "running" state. This seems like
>  > an autotest problem, it should detect when connection times out and
>  > it should abort the job automatically. (But now I'm not sure if we
>  > didn't make some adjustments the autotest's watchdog timer, we need
>  > to investigate).
>  > 
>  > I aborted the hanging job and all but the last two depchecks.
>  > 
>  > If this happens again we should investigate and report to autotest
>  > devs.
>  
>  Have you seen the same thing in production? We upgraded to the newer
>  version of autotest last week.
>  
>  Tim 
I didn't. But I have to say I never cared much about production, James always handled
that. But I don't remember such problem.

It can be caused by the new autotest, true.

[Kamil Paral]

>  > On Mon, 28 Nov 2011 09:20:05 -0500 (EST)
>  > Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  > 
>  > > I noticed today that we have about 50 queued x86_64/noarch
>  > > depchecks
>  > > on our staging server. The problem was in this job:
>  > > 
>  > > http://autoqa-stg.fedoraproject.org/results/2335-autotest/virt27.qa/debug/
>  > > 
>  > > It was hanging for two days in the "running" state. This seems
>  > > like
>  > > an autotest problem, it should detect when connection times out
>  > > and
>  > > it should abort the job automatically. (But now I'm not sure if
>  > > we
>  > > didn't make some adjustments the autotest's watchdog timer, we
>  > > need
>  > > to investigate).
>  > > 
>  > > I aborted the hanging job and all but the last two depchecks.
>  > > 
>  > > If this happens again we should investigate and report to
>  > > autotest
>  > > devs.
>  > 
>  > Have you seen the same thing in production? We upgraded to the
>  > newer
>  > version of autotest last week. 
I checked production and ... surprise surprise ... there are 4 jobs hanging from Nov 26!
It seems that Nov 26 was a bad day.

http://autoqa.fedoraproject.org/results/238579-autotest/qa02.qa.fedorapro...
http://autoqa.fedoraproject.org/results/238665-autotest/qa07.qa.fedorapro...
http://autoqa.fedoraproject.org/results/238685-autotest/qa03.qa.fedorapro...
http://autoqa.fedoraproject.org/results/238799-autotest/qa01.qa.fedorapro...

I'll abort them all. We will need to report bug in autotest. The reproducer will be
tricky however.

[Tim Flink]

On Mon, 28 Nov 2011 10:34:52 -0500 (EST)
Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  I checked production and ... surprise surprise ... there are 4 jobs
>  hanging from Nov 26! It seems that Nov 26 was a bad day.
>  
>  http://autoqa.fedoraproject.org/results/238579-autotest/qa02.qa.fedorapro...
>  http://autoqa.fedoraproject.org/results/238665-autotest/qa07.qa.fedorapro...
>  http://autoqa.fedoraproject.org/results/238685-autotest/qa03.qa.fedorapro...
>  http://autoqa.fedoraproject.org/results/238799-autotest/qa01.qa.fedorapro...
>  
>  I'll abort them all. We will need to report bug in autotest. The
>  reproducer will be tricky however. 
Are we sure that this is an autotest bug? Something in the back of my
head says that we have the timeout before killing a job set to something
crazy high.

Tim

[Kamil Paral]

>  On Mon, 28 Nov 2011 10:34:52 -0500 (EST)
>  Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  
>  > I checked production and ... surprise surprise ... there are 4 jobs
>  > hanging from Nov 26! It seems that Nov 26 was a bad day.
>  > 
>  >
> http://autoqa.fedoraproject.org/results/238579-autotest/qa02.qa.fedorapro...
>  >
> http://autoqa.fedoraproject.org/results/238665-autotest/qa07.qa.fedorapro...
>  >
> http://autoqa.fedoraproject.org/results/238685-autotest/qa03.qa.fedorapro...
>  >
> http://autoqa.fedoraproject.org/results/238799-autotest/qa01.qa.fedorapro...
>  > 
>  > I'll abort them all. We will need to report bug in autotest. The
>  > reproducer will be tricky however.
>  
>  Are we sure that this is an autotest bug? Something in the back of my
>  head says that we have the timeout before killing a job set to
>  something
>  crazy high. 
Yes, I believe we did set it to something crazy high because the timer was buggy and some
jobs were being cancelled in a few minutes. I'll try to check.

[Hongqing Yang]

----- Original Message -----
>  From: "Kamil Paral" <kparal(a)redhat.com&gt;
>  To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org&gt;
>  Sent: Monday, November 28, 2011 11:34:52 PM
>  Subject: Re: stage: hanging jobs
>  
>  > > On Mon, 28 Nov 2011 09:20:05 -0500 (EST)
>  > > Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  > > 
>  > > > I noticed today that we have about 50 queued x86_64/noarch
>  > > > depchecks
>  > > > on our staging server. The problem was in this job:
>  > > > 
>  > > >
> http://autoqa-stg.fedoraproject.org/results/2335-autotest/virt27.qa/debug/
>  > > > 
>  > > > It was hanging for two days in the "running" state. This seems
>  > > > like
>  > > > an autotest problem, it should detect when connection times out
>  > > > and
>  > > > it should abort the job automatically. (But now I'm not sure if
>  > > > we
>  > > > didn't make some adjustments the autotest's watchdog timer, we
>  > > > need
>  > > > to investigate).
>  > > > 
>  > > > I aborted the hanging job and all but the last two depchecks.
>  > > > 
>  > > > If this happens again we should investigate and report to
>  > > > autotest
>  > > > devs.
>  > > 
>  > > Have you seen the same thing in production? We upgraded to the
>  > > newer
>  > > version of autotest last week.
>  
>  I checked production and ... surprise surprise ... there are 4 jobs
>  hanging from Nov 26! It seems that Nov 26 was a bad day.
>  
>  http://autoqa.fedoraproject.org/results/238579-autotest/qa02.qa.fedorapro...
>  http://autoqa.fedoraproject.org/results/238665-autotest/qa07.qa.fedorapro...
>  http://autoqa.fedoraproject.org/results/238685-autotest/qa03.qa.fedorapro...
>  http://autoqa.fedoraproject.org/results/238799-autotest/qa01.qa.fedorapro...
>  
>  I'll abort them all. We will need to report bug in autotest. The
>  reproducer will be tricky however. 
I want to ask one question here by the way, all the jobs are queued but running 
on my local machine, what configuration needed to make them run? Thanks.

Hongqing

[Tim Flink]

On Mon, 28 Nov 2011 21:44:16 -0500 (EST)
Hongqing Yang <hoyang(a)redhat.com&gt; wrote:

>  I want to ask one question here by the way, all the jobs are queued
>  but running on my local machine, what configuration needed to make
>  them run? Thanks. 
Have you set up labels for the clients yet? If there are no labels, the
jobs won't go out to the clients.

It'll be some combination of:
fcXX (insert version for XX), virt, i386 or x86_64

Tim

[Hongqing Yang]

----- Original Message -----
>  From: "Tim Flink" <tflink(a)redhat.com&gt;
>  To: autoqa-devel(a)lists.fedorahosted.org
>  Sent: Tuesday, November 29, 2011 10:52:24 AM
>  Subject: Re: stage: hanging jobs
>  
>  On Mon, 28 Nov 2011 21:44:16 -0500 (EST)
>  Hongqing Yang <hoyang(a)redhat.com&gt; wrote:
>  
>  > I want to ask one question here by the way, all the jobs are queued
>  > but running on my local machine, what configuration needed to make
>  > them run? Thanks.
>  
>  Have you set up labels for the clients yet? If there are no labels,
>  the
>  jobs won't go out to the clients.
>  
>  It'll be some combination of:
>  fcXX (insert version for XX), virt, i386 or x86_64
>  
>  Tim
>   
Hi, Tim,

I have labels 'virt_capable' and 'x86_64', I will add fc15 and try. 

Thanks,
Hongqing

[Kamil Paral]

>  > > I want to ask one question here by the way, all the jobs
> are
>  > > queued
>  > > but running on my local machine, what configuration needed to
>  > > make
>  > > them run? Thanks.
>  > 
>  > Have you set up labels for the clients yet? If there are no labels,
>  > the
>  > jobs won't go out to the clients.
>  > 
>  > It'll be some combination of:
>  > fcXX (insert version for XX), virt, i386 or x86_64
>  > 
>  > Tim
>  > 
>  
>  Hi, Tim,
>  
>  I have labels 'virt_capable' and 'x86_64', I will add fc15 and try.
>  
>  Thanks,
>  Hongqing 
Here's the full list:
https://fedoraproject.org/wiki/Managing_autotest_labels

[Hongqing Yang]

----- Original Message -----
>  From: "Kamil Paral" <kparal(a)redhat.com&gt;
>  To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org&gt;
>  Sent: Tuesday, November 29, 2011 4:08:28 PM
>  Subject: Re: stage: hanging jobs
>  
>  > > > I want to ask one question here by the way, all the jobs are
>  > > > queued
>  > > > but running on my local machine, what configuration needed to
>  > > > make
>  > > > them run? Thanks.
>  > > 
>  > > Have you set up labels for the clients yet? If there are no
>  > > labels,
>  > > the
>  > > jobs won't go out to the clients.
>  > > 
>  > > It'll be some combination of:
>  > > fcXX (insert version for XX), virt, i386 or x86_64
>  > > 
>  > > Tim
>  > > 
>  > 
>  > Hi, Tim,
>  > 
>  > I have labels 'virt_capable' and 'x86_64', I will add fc15 and try.
>  > 
>  > Thanks,
>  > Hongqing 
Hi, All,

I have added all the labels, but the jobs are still queued. 

Hongqing

[Hongqing Yang]

----- Original Message -----
>  From: "Hongqing Yang" <hoyang(a)redhat.com&gt;
>  To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org&gt;
>  Sent: Tuesday, December 6, 2011 4:50:45 PM
>  Subject: Re: stage: hanging jobs
>  
>  
>  
>  ----- Original Message -----
>  > From: "Kamil Paral" <kparal(a)redhat.com&gt;
>  > To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org&gt;
>  > Sent: Tuesday, November 29, 2011 4:08:28 PM
>  > Subject: Re: stage: hanging jobs
>  > 
>  > > > > I want to ask one question here by the way, all the jobs are
>  > > > > queued
>  > > > > but running on my local machine, what configuration needed to
>  > > > > make
>  > > > > them run? Thanks.
>  > > > 
>  > > > Have you set up labels for the clients yet? If there are no
>  > > > labels,
>  > > > the
>  > > > jobs won't go out to the clients.
>  > > > 
>  > > > It'll be some combination of:
>  > > > fcXX (insert version for XX), virt, i386 or x86_64
>  > > > 
>  > > > Tim
>  > > > 
>  > > 
>  > > Hi, Tim,
>  > > 
>  > > I have labels 'virt_capable' and 'x86_64', I will add fc15 and
>  > > try.
>  > > 
>  > > Thanks,
>  > > Hongqing
>  
>  Hi, All,
>  
>  I have added all the labels, but the jobs are still queued.
>  
>  Hongqing
>   
I think that is because my host is locked.

Hongqing

[Kamil Paral]

Again I found two hanging tests:

http://autoqa-stg.fedoraproject.org/results/4507-autotest/virt27.qa/debug/
http://autoqa-stg.fedoraproject.org/results/4508-autotest/virt24.qa/debug/

I aborted them and also all the tests that were queued since then.

[Kamil Paral]

>  Again I found two hanging tests:
>  
>  http://autoqa-stg.fedoraproject.org/results/4507-autotest/virt27.qa/debug/
>  http://autoqa-stg.fedoraproject.org/results/4508-autotest/virt24.qa/debug/
>  
>  I aborted them and also all the tests that were queued since then. 
There is this setting in global_config.ini:

job_timeout_default: 72
job_max_runtime_hrs_default: 72

The value is in hours. I tried to change it locally to one hour and it aborts a hanged job
correctly. I intend to set these values on our staging (and production, if no one objects)
server:

job_timeout_default: 1
job_max_runtime_hrs_default: 2

I suppose the first value sets the timeout from the last message received from the client,
the second value sets the timeout for the whole runtime (I might be wrong, I found no
documentation apart from [1]).

I'll also mention that in our "Install autotest" wiki page.

That should help a bit.


[1] http://autotest.kernel.org/wiki/GlobalConfig#AUTOTEST_WEB

[Kamil Paral]

>  > Again I found two hanging tests:
>  > 
>  > http://autoqa-stg.fedoraproject.org/results/4507-autotest/virt27.qa/debug/
>  > http://autoqa-stg.fedoraproject.org/results/4508-autotest/virt24.qa/debug/
>  > 
>  > I aborted them and also all the tests that were queued since then.
>  
>  There is this setting in global_config.ini:
>  
>  job_timeout_default: 72
>  job_max_runtime_hrs_default: 72
>  
>  The value is in hours. I tried to change it locally to one hour and
>  it aborts a hanged job correctly. I intend to set these values on
>  our staging (and production, if no one objects) server:
>  
>  job_timeout_default: 1
>  job_max_runtime_hrs_default: 2
>  
>  I suppose the first value sets the timeout from the last message
>  received from the client, the second value sets the timeout for the
>  whole runtime (I might be wrong, I found no documentation apart from
>  [1]).
>  
>  I'll also mention that in our "Install autotest" wiki page.
>  
>  That should help a bit.
>  
>  
>  [1] http://autotest.kernel.org/wiki/GlobalConfig#AUTOTEST_WEB 
Hmm, after I set this up on the staging server, some tests were aborted while queued.
I'll talk to Lucas about the exact meaning of those options.

[Kamil Paral]

>  > > Again I found two hanging tests:
>  > > 
>  > > http://autoqa-stg.fedoraproject.org/results/4507-autotest/virt27.qa/debug/
>  > > http://autoqa-stg.fedoraproject.org/results/4508-autotest/virt24.qa/debug/
>  > > 
>  > > I aborted them and also all the tests that were queued since
>  > > then.
>  > 
>  > There is this setting in global_config.ini:
>  > 
>  > job_timeout_default: 72
>  > job_max_runtime_hrs_default: 72
>  > 
>  > The value is in hours. I tried to change it locally to one hour and
>  > it aborts a hanged job correctly. I intend to set these values on
>  > our staging (and production, if no one objects) server:
>  > 
>  > job_timeout_default: 1
>  > job_max_runtime_hrs_default: 2
>  > 
>  > I suppose the first value sets the timeout from the last message
>  > received from the client, the second value sets the timeout for the
>  > whole runtime (I might be wrong, I found no documentation apart
>  > from
>  > [1]).
>  > 
>  > I'll also mention that in our "Install autotest" wiki page.
>  > 
>  > That should help a bit.
>  > 
>  > 
>  > [1] http://autotest.kernel.org/wiki/GlobalConfig#AUTOTEST_WEB
>  
>  Hmm, after I set this up on the staging server, some tests were
>  aborted while queued. I'll talk to Lucas about the exact meaning of
>  those options.
>   
Lucas explained the options better at
https://github.com/autotest/autotest/wiki/GlobalConfig . The options will also be
described inside the config file in then next autotest release. I decided to leave
job_timeout_default to the default 72 hours, and set job_max_runtime_hrs_default to 2
hours. I'll apply this change to staging and production server. That should protect us
from hanging jobs.

I also documented the recommended value here:
https://fedoraproject.org/w/index.php?title=Install_and_configure_autotes...