11:18 a.m.
postgresql is currently failing to rebuild in rawhide:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
This seems quite repeatable, in koji, but the package builds fine on my
workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
environments. Shouldn't this have exactly the same results? Can anyone
suggest how I might reproduce the failure someplace where I can debug it?
regards, tom lane
12:47 p.m.
On Wed, Nov 9, 2011 at 11:18 AM, Tom Lane <tgl(a)redhat.com> wrote:
postgresql is currently failing to rebuild in rawhide:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
This seems quite repeatable, in koji, but the package builds fine on my
workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
environments. Shouldn't this have exactly the same results? Can anyone
suggest how I might reproduce the failure someplace where I can debug it?
I'm having a similar problem with OpenImageIO. I can build against TBB
in a mock environment but it fails in koji.
Richard
12:49 p.m.
Richard Shaw wrote:
On Wed, Nov 9, 2011 at 11:18 AM, Tom Lane <tgl(a)redhat.com>
wrote:
> postgresql is currently failing to rebuild in rawhide:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
> This seems quite repeatable, in koji, but the package builds fine on my
> workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
> environments. Shouldn't this have exactly the same results? Can anyone
> suggest how I might reproduce the failure someplace where I can debug it?
I'm having a similar problem with OpenImageIO. I can build against TBB
in a mock environment but it fails in koji.
I had a similar problem with iwhd.
All tests passed locally(F16), but "make check" in koji would always fail.
I ended up leaving %check disabled in the spec file.
Need to recheck...
1:05 p.m.
On 11/09/2011 08:49 PM, Jim Meyering wrote:
Richard Shaw wrote:
> On Wed, Nov 9, 2011 at 11:18 AM, Tom Lane<tgl(a)redhat.com> wrote:
>> postgresql is currently failing to rebuild in rawhide:
>> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
>> This seems quite repeatable, in koji, but the package builds fine on my
>> workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
>> environments. Shouldn't this have exactly the same results? Can anyone
>> suggest how I might reproduce the failure someplace where I can debug it?
>
> I'm having a similar problem with OpenImageIO. I can build against TBB
> in a mock environment but it fails in koji.
I had a similar problem with iwhd.
All tests passed locally(F16), but "make check" in koji would always fail.
I ended up leaving %check disabled in the spec file.
Need to recheck...
I've seen some similar incidents too:
At some point rpm test-suite was failing because of DNS issues on (some
of) the builders, causing unexpected "Could not canonicalize hostname"
spew in test output.
Sqlite has had a couple of rather entertaining cases: one had to do with
the filesystem and how many syncs sqlite tests were expecting to count
on it, this was different the builder vs fedora-of-the-time era. Another
more recent one was that sqlite test-suite tries to test failure when
opening file descriptors, and in order to do this it opens ~2000 fds.
This starts failing on "normal" systems but on the builders the max
number of fds is much higher, causing the test to fail because there was
no failure.
The only way to figure these out (that I know of) is to have some logic
in %check to detect test-suite failure and dump out what failure logs
you might have into koji's build.log so you have *some* clue where to
look. Eg in rpm I have this (relying on "make check" return code wont
work thanks to something in autofoo masking it)
%if %{with check}
%check
make check
[ "$(ls -A tests/rpmtests.dir)" ] && cat tests/rpmtests.log
%endif
- Panu -
1:10 p.m.
On Wed, Nov 9, 2011 at 2:05 PM, Panu Matilainen
<pmatilai(a)laiskiainen.org> wrote:
On 11/09/2011 08:49 PM, Jim Meyering wrote:
> Richard Shaw wrote:
>> On Wed, Nov 9, 2011 at 11:18 AM, Tom Lane<tgl(a)redhat.com> wrote:
>>> postgresql is currently failing to rebuild in rawhide:
>>> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
>>> This seems quite repeatable, in koji, but the package builds fine on my
>>> workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
>>> environments. Shouldn't this have exactly the same results? Can anyone
>>> suggest how I might reproduce the failure someplace where I can debug it?
>>
>> I'm having a similar problem with OpenImageIO. I can build against TBB
>> in a mock environment but it fails in koji.
>
> I had a similar problem with iwhd.
> All tests passed locally(F16), but "make check" in koji would always fail.
> I ended up leaving %check disabled in the spec file.
>
> Need to recheck...
I've seen some similar incidents too:
At some point rpm test-suite was failing because of DNS issues on (some
of) the builders, causing unexpected "Could not canonicalize hostname"
spew in test output.
Sqlite has had a couple of rather entertaining cases: one had to do with
the filesystem and how many syncs sqlite tests were expecting to count
on it, this was different the builder vs fedora-of-the-time era. Another
more recent one was that sqlite test-suite tries to test failure when
opening file descriptors, and in order to do this it opens ~2000 fds.
This starts failing on "normal" systems but on the builders the max
number of fds is much higher, causing the test to fail because there was
no failure.
The only way to figure these out (that I know of) is to have some logic
in %check to detect test-suite failure and dump out what failure logs
you might have into koji's build.log so you have *some* clue where to
look. Eg in rpm I have this (relying on "make check" return code wont
work thanks to something in autofoo masking it)
%if %{with check}
%check
make check
[ "$(ls -A tests/rpmtests.dir)" ] && cat tests/rpmtests.log
%endif
While I hesitate to point it out because of how rare it actually does
impact things, it's also a good idea to keep in mind that mock
duplicates the userspace package set, but it is still running on the
host kernel. The koji builders are all RHEL 6.x, which is obviously a
different kernel version than a f15/f16 desktop.
As I said, it's fairly rare for that to be a reason for failure, but
it does happen.
josh
3:36 p.m.
On Nov 9, 2011, at 9:18 AM, Tom Lane wrote:
postgresql is currently failing to rebuild in rawhide:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
This seems quite repeatable, in koji, but the package builds fine on my
workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
environments. Shouldn't this have exactly the same results? Can anyone
suggest how I might reproduce the failure someplace where I can debug it?
Mock rawhide targets make use of the nightly published rawhide tree. Koji makes use of an
up to the minute updated chroot of rawhide built packages, which can include many new
things not in the nightly tree.
This can lead to a difference of environments. Make sure all the same packages are being
used.
You can ask koji to spit out a mock config stub that you could use to gain access to the
packages used in koji at that moment in time:
$ koji mock-config --tag f17-build --arch x86_64 --topurl
http://kojipkgs.fedoraproject.org/ f17
It's a bit convoluted so I'm working on a fedpkg command that will use details
from the host machine and the current package branch to fill in those details.
--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
4:54 p.m.
Jesse Keating <jkeating(a)redhat.com> writes:
On Nov 9, 2011, at 9:18 AM, Tom Lane wrote:
> postgresql is currently failing to rebuild in rawhide:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
> This seems quite repeatable, in koji, but the package builds fine on my
> workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
> environments. Shouldn't this have exactly the same results? Can anyone
> suggest how I might reproduce the failure someplace where I can debug it?
Mock rawhide targets make use of the nightly published rawhide tree.
Koji makes use of an up to the minute updated chroot of rawhide built packages, which can
include many new things not in the nightly tree.
That was the first thing I thought of, but the failure originally
occurred yesterday in koji, and I still can't reproduce it today in
mock. I also looked at the root.log and didn't notice any differences
in installed package versions. (Although it seemed that the root.log
was being generated by a slightly different python script, because the
line numbers in the left margin were different. Should I be worried
about that?)
regards, tom lane
8 a.m.
On 11/09/2011 11:54 PM, Tom Lane wrote:
Jesse Keating<jkeating(a)redhat.com> writes:
> On Nov 9, 2011, at 9:18 AM, Tom Lane wrote:
>> postgresql is currently failing to rebuild in rawhide:
>> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
>> This seems quite repeatable, in koji, but the package builds fine on my
>> workstation under mock's fedora-rawhide-x86_64 and fedora-rawhide-i386
>> environments. Shouldn't this have exactly the same results? Can anyone
>> suggest how I might reproduce the failure someplace where I can debug it?
> Mock rawhide targets make use of the nightly published rawhide tree. Koji makes use
of an up to the minute updated chroot of rawhide built packages, which can include many
new things not in the nightly tree.
That was the first thing I thought of, but the failure originally
occurred yesterday in koji, and I still can't reproduce it today in
mock.
I don't know what has changed from yesterday (instead that I've upgraded
to F16), but now I have the same error in local mock as it is in koji
when building postgresql (yesterday the build succeeded every-time in
local mock - same as Tom wrote).
Honza
10:35 a.m.
Honza Horak <hhorak(a)redhat.com> writes:
On 11/09/2011 11:54 PM, Tom Lane wrote:
> That was the first thing I thought of, but the failure originally
> occurred yesterday in koji, and I still can't reproduce it today in
> mock.
I don't know what has changed from yesterday (instead that
I've upgraded
to F16), but now I have the same error in local mock as it is in koji
when building postgresql (yesterday the build succeeded every-time in
local mock - same as Tom wrote).
Huh, today I can reproduce it too --- and I did not change anything at
all on that machine. Is it possible that mock lags two days behind
rawhide, not just one?
The actual failure seems to be due to some change in Python error
traceback behavior. Which is curious because it doesn't look like the
python package changed versions since yesterday. I'm not much of a
Python person ... is there any package besides the core python package
that would have an influence on error handling behavior seen by
applications using libpython2.7.so?
Also, looking closer, why is it that rawhide still contains
python-2.7.2-16.fc17, when according to
http://pkgs.fedoraproject.org/gitweb/?p=python.git
that version was replaced two weeks ago due to the recent glibc issues?
And perhaps even more to the point, why does rawhide contain
glibc-2.14.90-15.1.x86_64
which according to the glibc git repository is a two-day-old f16 build,
not rawhide at all? (And why is glibc ignoring the convention to use
%{?dist} in Release:?)
regards, tom lane
11:18 a.m.
On 2011-11-10 8:35, Tom Lane wrote:
(And why is glibc ignoring the convention to use %{?dist} in
Release:?)
There is a bug open for this. Note that dist tags are still optional.
https://bugzilla.redhat.com/show_bug.cgi?id=676755
1:42 p.m.
I wrote:
[ concerning failure to rebuild postgresql in rawhide ]
The actual failure seems to be due to some change in Python error
traceback behavior.
After more thorough investigation, it seems to be a gcc optimization bug:
https://bugzilla.redhat.com/show_bug.cgi?id=752905
which makes the timing even weirder, because AFAICS gcc was last built
for rawhide on 2011-10-31. If the bug was injected then, how did it not
propagate to mock's view of rawhide until today? Something is rotten in
the state of Denmark.
regards, tom lane
5:58 p.m.
On Thu, 2011-11-10 at 11:35 -0500, Tom Lane wrote:
Also, looking closer, why is it that rawhide still contains
python-2.7.2-16.fc17, when according to
http://pkgs.fedoraproject.org/gitweb/?p=python.git
that version was replaced two weeks ago due to the recent glibc issues?
We haven't actually done a full rebuild for the glibc issues in Rawhide.
The packages which have a 'rebuilt for glibc' note in their Rawhide
branch git history are a result of dgilmore making a mistake: we planned
to rebuild a specific set of affected packages in F16 for the Final
release, but by mistake, this change was initially applied to master
branch rather than f16 branch. When we noticed the mistake, the mass
rebuild was cancelled, the changes applied to the f16 branch instead,
and the mass rebuild done for the f16 branch. So the upshot is that
there were a few dozen packages which have the 'rebuilt for glibc' note
in their Rawhide changelogs but where that build was never actually
submitted.
We probably should do a more thoroughgoing look at what packages should
still be rebuilt in Rawhide and F16 repos for the glibc issue - we only
ever actually did the most important F16 ones at the time - but that's
where things stand ATM.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
11:12 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
El Wed, 09 Nov 2011 12:18:45 -0500
Tom Lane <tgl(a)redhat.com> escribió:
postgresql is currently failing to rebuild in rawhide:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
This seems quite repeatable, in koji, but the package builds fine on
my workstation under mock's fedora-rawhide-x86_64 and
fedora-rawhide-i386 environments. Shouldn't this have exactly the
same results? Can anyone suggest how I might reproduce the failure
someplace where I can debug it?
regards, tom lane
so the builders are all rhel6 and the buildroots all have dns
resolution disabled. we are all running an older mock due to the number
of bugs were were hitting with newer mocks and pty handling. i need to
sit down and do some extensive testing of newer mocks and make sure im
happy that things are ok.
so if your using a different mock and a different kernel you could see
some differences.
Dennis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
iEYEARECAAYFAk69V3sACgkQkSxm47BaWfe4cgCgsNZ0td0N9k93q0FZf9qJL6iW
3rYAoKPqZ+uIUgaXs8qs5tgvNLfTWnep
=j8xs
-----END PGP SIGNATURE-----
11:43 a.m.
Dennis Gilmore <dennis(a)ausil.us> writes:
Tom Lane <tgl(a)redhat.com> escribió:
> postgresql is currently failing to rebuild in rawhide:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
so the builders are all rhel6 and the buildroots all have dns
resolution disabled. we are all running an older mock due to the number
of bugs were were hitting with newer mocks and pty handling.
The answer turned out to be that glibc-2.14.90-15.1 broke it, but for
some reason it took *two* days for that to show up in mock's image of
rawhide. Is that expected? Could we do something to cut down the lag?
I find it hard to visualize a situation where building in a stale image
of rawhide is especially useful.
regards, tom lane
1:23 p.m.
On Nov 11, 2011, at 9:43 AM, Tom Lane wrote:
Dennis Gilmore <dennis(a)ausil.us> writes:
> Tom Lane <tgl(a)redhat.com> escribió:
>> postgresql is currently failing to rebuild in rawhide:
>> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
> so the builders are all rhel6 and the buildroots all have dns
> resolution disabled. we are all running an older mock due to the number
> of bugs were were hitting with newer mocks and pty handling.
The answer turned out to be that glibc-2.14.90-15.1 broke it, but for
some reason it took *two* days for that to show up in mock's image of
rawhide. Is that expected? Could we do something to cut down the lag?
I find it hard to visualize a situation where building in a stale image
of rawhide is especially useful.
regards, tom lane
You can use the koji command I gave earlier to create mock configs that will use koji
content directly. Otherwise we don't have much we can do about the amount of time it
takes to get to a nightly tree and get that nightly tree synced to all the mirrors one
might use.
- jlk
6:55 p.m.
On Fri, 11 Nov 2011 12:43:28 -0500
Tom Lane <tgl(a)redhat.com> wrote:
Dennis Gilmore <dennis(a)ausil.us> writes:
> Tom Lane <tgl(a)redhat.com> escribió:
>> postgresql is currently failing to rebuild in rawhide:
>> http://koji.fedoraproject.org/koji/taskinfo?taskID=3499379
> so the builders are all rhel6 and the buildroots all have dns
> resolution disabled. we are all running an older mock due to the
> number of bugs were were hitting with newer mocks and pty handling.
The answer turned out to be that glibc-2.14.90-15.1 broke it, but for
some reason it took *two* days for that to show up in mock's image of
rawhide. Is that expected? Could we do something to cut down the
lag? I find it hard to visualize a situation where building in a
stale image of rawhide is especially useful.
Another part of this delay is that glibc maintainers are building only
for f16. So, they build, submit a update for f16, then tag it into f17.
So, there's a delay there before the glibc maintainer tags it.
finished building: Tue, 08 Nov 2011 14:36:26 UTC
tagged into f17: Tue Nov 8 15:18:56 UTC 2011
Granted only about 40min in this case...
kevin
4545
days inactive
4550
days old
15 comments
12 participants
participants (12)
-
Adam Williamson -
Dennis Gilmore -
Garrett Holmstrom -
Honza Horak -
Jesse Keating -
Jesse Keating -
Jim Meyering -
Josh Boyer -
Kevin Fenzi -
Panu Matilainen -
Richard Shaw -
Tom Lane