6:01 a.m.
https://koji.fedoraproject.org/koji/taskinfo?taskID=23008582
This has been stuck in the final tagging stage for must be at least 1
hour now, possibly 2 hours.
Is there anything that can be done?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
7:52 a.m.
On Thu, Nov 09, 2017 at 12:01:44PM +0000, Richard W.M. Jones wrote:
https://koji.fedoraproject.org/koji/taskinfo?taskID=23008582
This has been stuck in the final tagging stage for must be at least 1
hour now, possibly 2 hours.
Still tagging! I'm starting to wonder if it's broken.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
4:58 a.m.
On Thu, Nov 09, 2017 at 01:52:19PM +0000, Richard W.M. Jones wrote:
On Thu, Nov 09, 2017 at 12:01:44PM +0000, Richard W.M. Jones wrote:
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=23008582
>
> This has been stuck in the final tagging stage for must be at least 1
> hour now, possibly 2 hours.
Still tagging! I'm starting to wonder if it's broken.
24 hours and counting ...
I talked to someone on #fedora-admin about this and it is caused by
Koji having to load the metadata (files, dependencies etc) of every
RPM into its database. Combined with the fact this happens to be
running on s390x and it's texlive, makes it slow.
Still, this is blocking the builds of anything that depends on tex,
which is quite a lot of things because documentation tends to be built
using tools that use tex.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
8:33 a.m.
On 11/10/2017 1058Z, Richard W.M. Jones wrote:
24 hours and counting ...
I talked to someone on #fedora-admin about this and it is caused by
Koji having to load the metadata (files, dependencies etc) of every
RPM into its database. Combined with the fact this happens to be
running on s390x and it's texlive, makes it slow.
Can you be more specific? Is it slow because:
1. the s390x has a slow or inappropriate CPU for the task?
2. the VM on the s390x has not enough real RAM, and is page thrashing on a spinning
harddrive?
3. the database uses a quadratic algorithm that hurts in this case?
4. the database has very high CPU overhead in general?
5. "every RPM" means all 30,000 .rpm in Fedora, not just the 300 for
texlive?
(And if so, then why isn't tagging slow for _every_ package?)
6. some other specific reason(s)?
Indexing 1GB of data should take no more than a few minutes.
--
9:01 a.m.
On Fri, Nov 10, 2017 at 06:33:09AM -0800, John Reiser wrote:
On 11/10/2017 1058Z, Richard W.M. Jones wrote:
>24 hours and counting ...
>
>I talked to someone on #fedora-admin about this and it is caused by
>Koji having to load the metadata (files, dependencies etc) of every
>RPM into its database. Combined with the fact this happens to be
>running on s390x and it's texlive, makes it slow.
Can you be more specific? Is it slow because:
1. the s390x has a slow or inappropriate CPU for the task?
2. the VM on the s390x has not enough real RAM, and is page thrashing on a spinning
harddrive?
3. the database uses a quadratic algorithm that hurts in this case?
4. the database has very high CPU overhead in general?
5. "every RPM" means all 30,000 .rpm in Fedora, not just the 300 for
texlive?
(And if so, then why isn't tagging slow for _every_ package?)
6. some other specific reason(s)?
Indexing 1GB of data should take no more than a few minutes.
I don't have any access to the machine so I'm not in a position to say.
I will note however that the previous successful build of texlive took
4 hours in total (that's building, adding to the database and
tagging). This one has been running for 30 hours and still has not
completed.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
11:47 a.m.
On 11/10/2017 07:01 AM, Richard W.M. Jones wrote:
On Fri, Nov 10, 2017 at 06:33:09AM -0800, John Reiser wrote:
> On 11/10/2017 1058Z, Richard W.M. Jones wrote:
>
>> 24 hours and counting ...
>>
>> I talked to someone on #fedora-admin about this and it is caused by
>> Koji having to load the metadata (files, dependencies etc) of every
>> RPM into its database. Combined with the fact this happens to be
>> running on s390x and it's texlive, makes it slow.
>
> Can you be more specific? Is it slow because:
> 1. the s390x has a slow or inappropriate CPU for the task?
> 2. the VM on the s390x has not enough real RAM, and is page thrashing on a spinning
harddrive?
> 3. the database uses a quadratic algorithm that hurts in this case?
> 4. the database has very high CPU overhead in general?
> 5. "every RPM" means all 30,000 .rpm in Fedora, not just the 300 for
texlive?
> (And if so, then why isn't tagging slow for _every_ package?)
> 6. some other specific reason(s)?
>
> Indexing 1GB of data should take no more than a few minutes.
I don't have any access to the machine so I'm not in a position to say.
I will note however that the previous successful build of texlive took
4 hours in total (that's building, adding to the database and
tagging). This one has been running for 30 hours and still has not
completed.
The S390x builders are not in the same datacenter as all the rest of the
builders, the koji hub and it's database. They are not particularly
slow, but sending all the data back and forth over the net is likely the
bottleneck here.
It's particularly bad (if I understand correctly how it works) because
one builder manages the parent task and farms out the builds to other
builders, then collects them back and uploads them to the hub. So, if a
s390x builder is controlling the parent task it will have to pull all
the resuts back accross the net to itself, then push them back accross
the net to the hub.
All that said, I don't see where s390x is involved in the current build.
The parent task is being handled by buildvm-23, but indeed it seems like
something has gone wrong. I can try and free that task to another
builder, but that might cause it to fail. I suppose I can resubmit after
that... so will give it a try
kevin
12:43 p.m.
On Fri, Nov 10, 2017 at 09:47:02AM -0800, Kevin Fenzi wrote:
All that said, I don't see where s390x is involved in the current
build.
The parent task is being handled by buildvm-23, but indeed it seems like
something has gone wrong. I can try and free that task to another
builder, but that might cause it to fail. I suppose I can resubmit after
that... so will give it a try
It seems as if the build has been resubmitted. It was on an s390x
builder (according to the hostname), but when I reloaded the page just
a few moments ago it's now on buildvm-02.phx2, and in the final stage
already.
Thanks for the explanation & fix.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
12:59 p.m.
On 11/10/2017 10:43 AM, Richard W.M. Jones wrote:
On Fri, Nov 10, 2017 at 09:47:02AM -0800, Kevin Fenzi wrote:
> All that said, I don't see where s390x is involved in the current build.
> The parent task is being handled by buildvm-23, but indeed it seems like
> something has gone wrong. I can try and free that task to another
> builder, but that might cause it to fail. I suppose I can resubmit after
> that... so will give it a try
It seems as if the build has been resubmitted. It was on an s390x
builder (according to the hostname), but when I reloaded the page just
a few moments ago it's now on buildvm-02.phx2, and in the final stage
already.
Thanks for the explanation & fix
yeah, I freed it from the buildvm-23 it was on, and buildvm-02 got it.
I guess we will wait a few hours and see if it finishes.
Texlive is pretty much the worst case example for the build system, and
has broken things in various tools a number of times. From it's 5000
subpackages, it's > 2GB src.rpm, etc.
Hopefully it will complete now, if it doesn't we need to look more,
likely for a koji hub bug. ;(
kevin
3:02 a.m.
It looks as if the texlive saga continues!
https://koji.fedoraproject.org/koji/taskinfo?taskID=23085154
Started yesterday evening and still going 15 hours later.
It has picked an armv7 host this time.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
9:53 a.m.
On 11/13/2017 01:02 AM, Richard W.M. Jones wrote:
It looks as if the texlive saga continues!
https://koji.fedoraproject.org/koji/taskinfo?taskID=23085154
Started yesterday evening and still going 15 hours later.
It has picked an armv7 host this time.
I canceled and started a new one.
It wasn't done this morning, and we looked and it turns out we are
hitting a bug in koji where it's seeing duplicate NVR's for some
subpackages from the partially imported build, but not failing correctly.
Kalev has started a new one with subpackages bumped properly.
We have hopes for this build. ;)
kevin
6:48 a.m.
2017-11-13 10:53 GMT-05:00 Kevin Fenzi <kevin(a)scrye.com>:
On 11/13/2017 01:02 AM, Richard W.M. Jones wrote:
> It looks as if the texlive saga continues!
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=23085154
>
> Started yesterday evening and still going 15 hours later.
> It has picked an armv7 host this time.
I canceled and started a new one.
It wasn't done this morning, and we looked and it turns out we are
hitting a bug in koji where it's seeing duplicate NVR's for some
subpackages from the partially imported build, but not failing correctly.
Kalev has started a new one with subpackages bumped properly.
We have hopes for this build. ;)
Can you share the link for the build job? So people can watch it :)
I have been trying since last Friday to build sagemath, and always
fails due to armv7hl texlive:
DEBUG util.py:439: Error:
DEBUG util.py:439: Problem 1: package
texlive-scheme-basic-6:svn25923.0-36.20160520.fc28.7.noarch requires
texlive-collection-basic, but none of the providers can be installed
DEBUG util.py:439: - package
texlive-collection-basic-6:svn41149-36.20160520.fc28.7.noarch requires
dvipdfmx, but none of the providers can be installed
DEBUG util.py:439: - package
texlive-6:2016-36.20160520.fc28.7.armv7hl requires
texlive-scheme-basic, but none of the providers can be installed
DEBUG util.py:439: - package
texlive-dvipdfmx-bin-6:svn40273-36.20160520.fc28.7.armv7hl requires
texlive-xetex-bin, but none of the providers can be installed
DEBUG util.py:439: - conflicting requests
DEBUG util.py:439: - nothing provides libpoppler.so.71 needed by
texlive-xetex-bin-6:svn41091-36.20160520.fc28.7.armv7hl
DEBUG util.py:439: Problem 2: package
R-core-devel-3.4.2-2.fc28.armv7hl requires texinfo-tex, but none of
the providers can be installed
DEBUG util.py:439: - package texinfo-tex-6.5-1.fc28.armv7hl
requires tex(tex), but none of the providers can be installed
DEBUG util.py:439: - package R-devel-3.4.2-2.fc28.armv7hl requires
R-core-devel = 3.4.2-2.fc28, but none of the providers can be
installed
DEBUG util.py:439: - package
texlive-collection-basic-6:svn41149-36.20160520.fc28.7.noarch requires
dvipdfmx, but none of the providers can be installed
DEBUG util.py:439: - package R-3.4.2-2.fc28.armv7hl requires
R-devel = 3.4.2-2.fc28, but none of the providers can be installed
DEBUG util.py:439: - package
texlive-dvipdfmx-bin-6:svn40273-36.20160520.fc28.7.armv7hl requires
texlive-xetex-bin, but none of the providers can be installed
DEBUG util.py:439: - conflicting requests
DEBUG util.py:439: - nothing provides libpoppler.so.71 needed by
texlive-xetex-bin-6:svn41091-36.20160520.fc28.7.armv7hl
DEBUG util.py:439: Problem 3: package
texlive-collection-latexrecommended-6:svn35765.0-36.20160520.fc28.7.noarch
requires texlive-collection-latex, but none of the providers can be
installed
DEBUG util.py:439: - package R-core-3.4.2-2.fc28.armv7hl requires
tex(latex), but none of the providers can be installed
DEBUG util.py:439: - package
texlive-collection-latex-6:svn41011-36.20160520.fc28.7.noarch requires
texlive-collection-basic, but none of the providers can be installed
DEBUG util.py:439: - package python3-rpy-2.9.0-1.fc28.armv7hl
requires libR.so, but none of the providers can be installed
DEBUG util.py:439: - package
texlive-collection-basic-6:svn41149-36.20160520.fc28.7.noarch requires
dvipdfmx, but none of the providers can be installed
DEBUG util.py:439: - package rpy-2.9.0-1.fc28.armv7hl requires
python3-rpy = 2.9.0-1.fc28, but none of the providers can be installed
DEBUG util.py:439: - package
texlive-dvipdfmx-bin-6:svn40273-36.20160520.fc28.7.armv7hl requires
texlive-xetex-bin, but none of the providers can be installed
DEBUG util.py:439: - conflicting requests
DEBUG util.py:439: - nothing provides libpoppler.so.71 needed by
texlive-xetex-bin-6:svn41091-36.20160520.fc28.7.armv7hl
kevin
Thanks,
Paulo
7:36 a.m.
On Tue, Nov 14, 2017 at 07:48:07AM -0500, Paulo César Pereira de Andrade wrote:
2017-11-13 10:53 GMT-05:00 Kevin Fenzi <kevin(a)scrye.com>:
> On 11/13/2017 01:02 AM, Richard W.M. Jones wrote:
>> It looks as if the texlive saga continues!
>>
>> https://koji.fedoraproject.org/koji/taskinfo?taskID=23085154
>>
>> Started yesterday evening and still going 15 hours later.
>> It has picked an armv7 host this time.
>
> I canceled and started a new one.
>
> It wasn't done this morning, and we looked and it turns out we are
> hitting a bug in koji where it's seeing duplicate NVR's for some
> subpackages from the partially imported build, but not failing correctly.
>
> Kalev has started a new one with subpackages bumped properly.
>
> We have hopes for this build. ;)
Can you share the link for the build job? So people can watch it :)
https://koji.fedoraproject.org/koji/packageinfo?packageID=5402
Unfortunately the latest one is still building after ~ 22 hours ...
It seems like it might have got stuck again.
I have been trying since last Friday to build sagemath, and always
fails due to armv7hl texlive:
[...]
Yup, that's the same problem I'm having.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
10:27 a.m.
We're 24 hours into a new build and it still isn't working ...
At this rate it'll soon be 7 days since we've been able to do
any build in Rawhide that depends on TeX, emacs, ... (which is
quite a lot of builds since most documentation tools depend
on TeX directly or indirectly).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
5:58 a.m.
We're now 48 hours into the build and it hasn't completed, probably
indicating that something is going on.
And -- although this is *not* in any way the fault of any of the
people involved -- it has now been 7 days since we've been able to
build anything in Rawhide that depends on texlive. Is there something
else we could try, like untagging the new poppler?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
10:03 a.m.
On 11/16/2017 03:58 AM, Richard W.M. Jones wrote:
We're now 48 hours into the build and it hasn't completed, probably
indicating that something is going on.
And -- although this is *not* in any way the fault of any of the
people involved -- it has now been 7 days since we've been able to
build anything in Rawhide that depends on texlive.
Yes, we know. :)
Patrick looked at it in great depth the other day and we thought we had
a solution, but apparently not. He's planning on digging more on it
today. It seemed that when uploading the results it had a db transaction
open longer than the firewall "ESTABLISHED" rules, so when the client
tried to use it next it was dropped by the firewall. But there may be
something further after this issue we are hitting now.
I'm not sure what more we can do...
Is there something
else we could try, like untagging the new poppler?
I don't think thats an option at this point.
kevin
5:20 a.m.
2017-11-16 16:49 GMT-05:00 Tom Callaway <tcallawa(a)redhat.com>:
On 11/16/2017 11:03 AM, Kevin Fenzi wrote:
> I'm not sure what more we can do...
While not solving this immediate problem, I am working on redoing the
texlive package to make it less evil, in my spare cycles.
Maybe now there is a possibility of having a texlive/repository-module ?
Somewhat similar to what is descrived at
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
The problem is maintaining 1.5k+ packages, and creating such number
o individual packages, and every time an update is done (assuming 1
month+), need to remove like 5+ and add new 5+.
Anyway in the Fedora way :), should split binaries in one package, and
have multiple texlive-collection-* packages (-scheme probably would not
work, so, need to be virtual packages).
~tom
Thanks,
Paulo
5:41 a.m.
On Thu, Nov 16, 2017 at 08:03:58AM -0800, Kevin Fenzi wrote:
On 11/16/2017 03:58 AM, Richard W.M. Jones wrote:
>
> We're now 48 hours into the build and it hasn't completed, probably
> indicating that something is going on.
>
> And -- although this is *not* in any way the fault of any of the
> people involved -- it has now been 7 days since we've been able to
> build anything in Rawhide that depends on texlive.
Yes, we know. :)
Patrick looked at it in great depth the other day and we thought we had
a solution, but apparently not. He's planning on digging more on it
today. It seemed that when uploading the results it had a db transaction
open longer than the firewall "ESTABLISHED" rules, so when the client
tried to use it next it was dropped by the firewall. But there may be
something further after this issue we are hitting now.
I'm not sure what more we can do...
The latest build has now succeeded. I'm going to try a dependent
build now.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
10:28 a.m.
On 11/17/2017 03:41 AM, Richard W.M. Jones wrote:
On Thu, Nov 16, 2017 at 08:03:58AM -0800, Kevin Fenzi wrote:
> On 11/16/2017 03:58 AM, Richard W.M. Jones wrote:
>>
>> We're now 48 hours into the build and it hasn't completed, probably
>> indicating that something is going on.
>>
>> And -- although this is *not* in any way the fault of any of the
>> people involved -- it has now been 7 days since we've been able to
>> build anything in Rawhide that depends on texlive.
>
> Yes, we know. :)
>
> Patrick looked at it in great depth the other day and we thought we had
> a solution, but apparently not. He's planning on digging more on it
> today. It seemed that when uploading the results it had a db transaction
> open longer than the firewall "ESTABLISHED" rules, so when the client
> tried to use it next it was dropped by the firewall. But there may be
> something further after this issue we are hitting now.
>
> I'm not sure what more we can do...
The latest build has now succeeded. I'm going to try a dependent
build now.
Yeah, thanks to a ton of work from Patrick. :)
The bottleneck is the part where it's finished all the builds and needs
to check the noarch packages built on each arch to make sure they are
the same.
Patrick now has some ideas for a real fix in koji, so hopefully we will
be past this issue soon.
kevin
10:50 a.m.
On 11/17/2017 1628Z, Kevin Fenzi wrote:
The bottleneck is the part where it's finished all the builds and
needs
to check the noarch packages built on each arch to make sure they are
the same.
Patrick now has some ideas for a real fix in koji, so hopefully we will
be past this issue soon.
Please share the bugzilla or other tracking link for this problem, work,
or fix? One idea to consider is for each builder on each $ARCH to produce
a sha256 checksum as each individual .rpm is built. Then the files of
collected checksums for each $ARCH can be sorted and used as input to
diff or comm. This gives information even if some builder(s) are
not yet finished.
--
11:08 a.m.
On 11/17/2017 08:50 AM, John Reiser wrote:
On 11/17/2017 1628Z, Kevin Fenzi wrote:
> The bottleneck is the part where it's finished all the builds and needs
> to check the noarch packages built on each arch to make sure they are
> the same.
>
> Patrick now has some ideas for a real fix in koji, so hopefully we will
> be past this issue soon.
Please share the bugzilla or other tracking link for this problem, work,
or fix? One idea to consider is for each builder on each $ARCH to produce
a sha256 checksum as each individual .rpm is built. Then the files of
collected checksums for each $ARCH can be sorted and used as input to
diff or comm. This gives information even if some builder(s) are
not yet finished.
I don't know how the test is currently implemented... thats something to
discuss with koji developers.
https://pagure.io/koji/issue/715
kevin
2348
days inactive
2359
days old
21 comments
6 participants
participants (6)
-
John Reiser -
Kevin Fenzi -
Matthew Miller -
Paulo César Pereira de Andrade -
Richard W.M. Jones -
Tom Callaway