Builds on buildsys.fedoraproject.org fails, but the build self will work.

[Frank Büttner]

Hello,
something with the build system is wrong.
The build self will work, but the system says failed.
Have anyone see this also?

Build ID: 36210 and 36211.

Thanks.

[Michael Schwendt]

On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:

>  Hello,
>  something with the build system is wrong.
>  The build self will work, but the system says failed.
>  Have anyone see this also?
>  
>  Build ID: 36210 and 36211. 
It has been a topic on fedora-maintainer multiple times. Here's the
corresponding ticket:

https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125

[Frank Büttner]

Michael Schwendt schrieb:
>  On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
>  
> > Hello,
> > something with the build system is wrong.
> > The build self will work, but the system says failed.
> > Have anyone see this also?
> >
> > Build ID: 36210 and 36211.
>  
>  It has been a topic on fedora-maintainer multiple times. Here's the
>  corresponding ticket:
>  
>  https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
>   
Yes, it's my problem too. I hope it will be fixed soon.

[Michael Schwendt]

On Sat, 08 Sep 2007 11:01:24 +0200, Frank Büttner wrote:

>  Michael Schwendt schrieb:
>  > On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
>  > 
>  >> Hello,
>  >> something with the build system is wrong.
>  >> The build self will work, but the system says failed.
>  >> Have anyone see this also?
>  >>
>  >> Build ID: 36210 and 36211.
>  > 
>  > It has been a topic on fedora-maintainer multiple times. Here's the
>  > corresponding ticket:
>  > 
>  > https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
>  > 
>  
>  Yes, it's my problem too. I hope it will be fixed soon. 
Consider this fixed [1]. Resubmit your failed build-jobs for Extras
and EPEL and see.

After I'd noticed that approx. 5 out of 6 noarch build-jobs on the
ppc2 server failed with the same symptoms, I've picked yum-cron as a
test pkg for an online-debugging session on the ppc2 host. As another
pkg that had failed very often, I've tested the final patch with
ctapi-common.

The reason for failure was a race condition between plague 0.4.4.1 and
mock. The plague builder did not monitor the mock state accurately
enough. Mock could change its state (e.g. from prepping/setup to
building) while the plague builder was sleeping. In the cases I've
observed, mock had exited already when plague woke up and made a wrong
assumption about the state mock was in when it exited.

[1] http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch

[Jesse Keating]

On Wed, 19 Sep 2007 03:59:38 +0200
Michael Schwendt <mschwendt.tmp0701.nospam(a)arcor.de&gt; wrote:

>  The reason for failure was a race condition between plague 0.4.4.1
> and
>  mock. The plague builder did not monitor the mock state accurately
>  enough. Mock could change its state (e.g. from prepping/setup to
>  building) while the plague builder was sleeping. In the cases I've
>  observed, mock had exited already when plague woke up and made a wrong
>  assumption about the state mock was in when it exited.
>  
>  [1]
>  http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch 
This is really awesome that you were able to track this down.  Thank
you very much for this work.

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?

[Dan Williams]

On Wed, 2007-09-19 at 03:59 +0200, Michael Schwendt wrote:
>  On Sat, 08 Sep 2007 11:01:24 +0200, Frank Büttner wrote:
>  
>  > Michael Schwendt schrieb:
>  > > On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
>  > > 
>  > >> Hello,
>  > >> something with the build system is wrong.
>  > >> The build self will work, but the system says failed.
>  > >> Have anyone see this also?
>  > >>
>  > >> Build ID: 36210 and 36211.
>  > >
>  > > It has been a topic on fedora-maintainer multiple times. Here's the
>  > > corresponding ticket:
>  > > 
>  > > https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
>  > > 
>  > 
>  > Yes, it's my problem too. I hope it will be fixed soon.
>  
>  Consider this fixed [1]. Resubmit your failed build-jobs for Extras
>  and EPEL and see.
>  
>  After I'd noticed that approx. 5 out of 6 noarch build-jobs on the
>  ppc2 server failed with the same symptoms, I've picked yum-cron as a
>  test pkg for an online-debugging session on the ppc2 host. As another
>  pkg that had failed very often, I've tested the final patch with
>  ctapi-common.
>  
>  The reason for failure was a race condition between plague 0.4.4.1 and
>  mock. The plague builder did not monitor the mock state accurately
>  enough. Mock could change its state (e.g. from prepping/setup to
>  building) while the plague builder was sleeping. In the cases I've
>  observed, mock had exited already when plague woke up and made a wrong
>  assumption about the state mock was in when it exited.
>  
>  [1] http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch 
Workaround looks sane, diagnosis is correct, thanks for tracking this
down and fixing it.  Out of curiosity, did a mock update or a hardware
change exacerbate the issue?  We only just started seeing this after ~2
years of using plague.

Dan

[Dennis Gilmore]

Once upon a time Tuesday 18 September 2007, Dan Williams wrote:

>  Workaround looks sane, diagnosis is correct, thanks for tracking
> this
>  down and fixing it.  Out of curiosity, did a mock update or a hardware
>  change exacerbate the issue?  We only just started seeing this after ~2
>  years of using plague.
>
>  Dan 
not really sure,  mock has not been updated on the builders for awhile.  there 
has been no hardware changes.  the only real difference that i see is that 
plague is being used less now.  we are building EPEL-4 EPEL-5 and Extras 6 in 
plague.  currently we have been keeping 4 cores available for x86_64 and 4 
for ppc.  

Dennis

[Michael Schwendt]

On Tue, 18 Sep 2007 22:36:15 -0400, Dan Williams wrote:

>  > [1]
> http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch
>  
>  Workaround looks sane, diagnosis is correct, thanks for tracking this
>  down and fixing it.  Out of curiosity, did a mock update or a hardware
>  change exacerbate the issue?  We only just started seeing this after ~2
>  years of using plague. 
I have doubts that the .../mock-state/status file is in sync with
mock. It either shows symptoms of buffering or changes too
quickly. It stays in "prep" for most of the time, then very
quickly steps through "setu" -> "buil" -> "done". So
quickly that
plague-builder cannot sleep for 3 secs without missing the state
changes before mock is done.

Perhaps this is different for longer builds which don't run for just
6-8 minutes.

Look at this debug log excerpt made when plague-builder was monitoring
the "prepping" step (the "string" value is the four chars from the
mock-state file after a refresh):

Running _status_prepping: prepping
string = prep
_watch_mock(): waitpid = 0,0
continuing _status_prepping: prepping
string = prep
Running _status_prepping: prepping
string = done

      ^ here mock-state changed while plague-builder was sleeping for 3s
      => from "prep" to mock-exit in plague-builder's view (!)
      => when not reloading mock's state file to verify

string = done
_watch_mock(): waitpid = 4139,0
_watch_mock(): PID 4139 exit status 0
Setting Status Good! - failed

      ^ this is based on the assumption that a mock-exit during
      "prepping" would always be an error condition despite the
      zero exit code

init
clean
prep
This may take a while
setup
build
ending
done

    ^ this is what mock has processed actually while plague-builder
    assumed that mock still was doing only "prep"+"setu"

Results and/or logs in:
/mnt/build/builder_work/46bb3113e41a4745e4b1217cb9d5b6b8ef5cc73b/result
Cleaning up the buildroot...
   /usr/bin/setarch ppc32 /usr/bin/mock clean
--uniqueext=46bb3113e41a4745e4b1217cb9d5b6b8ef5cc73b -r fedora-6-ppc-core