1:01 a.m.
Hi,
I have tried to enable debuginfo for Chromium but it cannot fit the disk.
During local build it has 160GB (on x86_64) and Koji shows (moreover AFAIK
shared for multiple builders):
https://kojipkgs.fedoraproject.org//work/tasks/7099/46377099/hw_info.log
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 138G 71G 67G 52% /
Is there some way forward?
In 2013:
https://lists.fedoraproject.org/pipermail/devel/2013-January/176673.html
Looking at space we have, we are just going to bump them all to 150 or
so. Hopefully that will keep us for a while.
Jan
diff --git a/chromium-debug.patch b/chromium-debug.patch
new file mode 100644
index 0000000..181521a
--- /dev/null
+++ b/chromium-debug.patch
@@ -0,0 +1,12 @@
+--- chromium-83.0.4103.116/build/config/compiler/BUILD.gn.debug 2020-06-29
19:07:24.928680424 +0200
++++ chromium-83.0.4103.116/build/config/compiler/BUILD.gn 2020-06-29 19:40:23.063688281
+0200
+@@ -2277,7 +2277,8 @@ config("symbols") {
+ # flag, so we can use use -g1 for pnacl and nacl-clang compiles.
+ # gcc nacl is is_nacl && !is_clang, pnacl and nacl-clang are &&
is_clang.
+ if (!is_nacl || is_clang) {
+- cflags += [ "-g2" ]
++ cflags += [ "-g3" ]
++ cflags += [ "-fdebug-types-section" ]
+ }
+
+ if (is_clang && !is_nacl && !use_xcode_clang) {
diff --git a/chromium.spec b/chromium.spec
index 69dcead..3d9bfff 100644
--- a/chromium.spec
+++ b/chromium.spec
@@ -45,9 +45,8 @@
%global headlessbuilddir out/Headless
%global remotingbuilddir out/Remoting
-# Debuginfo packages aren't very useful here. If you need to debug
-# you should do a proper debug build (not implemented in this spec yet)
-%global debug_package %{nil}
+# Debuginfo would run out of 32-bit 4GB DWARF so use -fdebug-types-section.
+%global _find_debuginfo_dwz_opts %{nil}
# %%{nil} for Stable; -beta for Beta; -dev for Devel
# dash in -beta and -dev is intentional !
@@ -163,7 +162,7 @@ Name: chromium%{chromium_channel}%{nsuffix}
Name: chromium%{chromium_channel}
%endif
Version: %{majorversion}.0.4103.116
-Release: 3%{?dist}
+Release: 3debugtypes%{?dist}
%if %{?freeworld}
%if %{?shared}
# chromium-libs-media-freeworld
@@ -286,6 +285,8 @@ Patch205: chromium-81.0.4044.92-fix-vaapi-on-intel.patch
# Apply these patches to work around EPEL8 issues
Patch300: chromium-76.0.3809.132-rhel8-force-disable-use_gnome_keyring.patch
+# Enable chromium-debuginfo.rpm
+Patch900: chromium-debug.patch
# Use chromium-latest.py to generate clean tarball from released build tarballs, found
here:
# http://build.chromium.org/buildbot/official/
@@ -877,6 +878,7 @@ udev.
%if 0%{?rhel} == 8
%patch300 -p1 -b .disblegnomekeyring
%endif
+%patch900 -p1 -b .debug
# Change shebang in all relevant files in this directory and all subdirectories
# See `man find` for how the `-exec command {} +` syntax works
@@ -959,12 +961,12 @@ popd
# Core defines are flags that are true for both the browser and headless.
CHROMIUM_CORE_GN_DEFINES=""
-CHROMIUM_CORE_GN_DEFINES+=' is_debug=false'
+CHROMIUM_CORE_GN_DEFINES+=' is_debug=false symbol_level=2'
%ifarch x86_64 aarch64
CHROMIUM_CORE_GN_DEFINES+=' system_libdir="lib64"'
%endif
CHROMIUM_CORE_GN_DEFINES+=' google_api_key="%{api_key}"
google_default_client_id="%{default_client_id}"
google_default_client_secret="%{default_client_secret}"'
-CHROMIUM_CORE_GN_DEFINES+=' is_clang=false use_sysroot=false use_gold=false
fieldtrial_testing_like_official_build=true use_lld=false
rtc_enable_symbol_export=true'
+CHROMIUM_CORE_GN_DEFINES+=' is_clang=false use_sysroot=false use_gold=true
fieldtrial_testing_like_official_build=true use_lld=false
rtc_enable_symbol_export=true'
%if %{freeworld}
CHROMIUM_CORE_GN_DEFINES+=' ffmpeg_branding="ChromeOS"
proprietary_codecs=true'
%else
@@ -992,7 +994,7 @@ CHROMIUM_BROWSER_GN_DEFINES+=' is_component_ffmpeg=true
is_component_build=true'
%else
CHROMIUM_BROWSER_GN_DEFINES+=' is_component_ffmpeg=false
is_component_build=false'
%endif
-CHROMIUM_BROWSER_GN_DEFINES+=' blink_symbol_level=0
enable_hangout_services_extension=true'
+CHROMIUM_BROWSER_GN_DEFINES+=' blink_symbol_level=2
enable_hangout_services_extension=true'
CHROMIUM_BROWSER_GN_DEFINES+=' use_aura=true'
CHROMIUM_BROWSER_GN_DEFINES+=' enable_widevine=true'
%if %{use_vaapi}
@@ -1379,11 +1381,6 @@ ulimit -n 4096
. /opt/rh/devtoolset-%{dts_version}/enable
%endif
-# Decrease the debuginfo verbosity, so it compiles in koji
-%ifarch %{ix86}
-%global optflags %(echo %{optflags} | sed 's/-g /-g1 /')
-%endif
-
export
PYTHONPATH="../../third_party/pyjson5/src:../../third_party/catapult/third_party/google-endpoints"
echo
6:43 a.m.
On Wed, 1 Jul 2020 at 02:09, Jan Kratochvil <jan.kratochvil(a)redhat.com> wrote:
Hi,
I have tried to enable debuginfo for Chromium but it cannot fit the disk.
During local build it has 160GB (on x86_64) and Koji shows (moreover AFAIK
shared for multiple builders):
https://kojipkgs.fedoraproject.org//work/tasks/7099/46377099/hw_info.log
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 138G 71G 67G 52% /
Is there some way forward?
In 2013:
https://lists.fedoraproject.org/pipermail/devel/2013-January/176673.html
Looking at space we have, we are just going to bump them all to 150 or
so. Hopefully that will keep us for a while.
In the end we have limited resources and you are running into those
limits. We can either have fewer builders with more disk space and a
long queue of builds or we can have more builders with smaller disk
space. We are especially limited during and after the move of hardware
because it takes a considerable amount of time to bring things back
up. People are going to have to be patient or find other ways to do
their work if that is not working. If this is not acceptable, please
bring it up with our management chain.
Jan
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Stephen J Smoogen.
7 a.m.
On Wed, 01 Jul 2020 13:43:57 +0200, Stephen John Smoogen wrote:
In the end we have limited resources and you are running into those
limits. We can either have fewer builders with more disk space and a
long queue of builds or we can have more builders with smaller disk
space.
Good to know disk space really is a limiting factor there (as it is in general
cheap compared to CPUs+RAM).
Sure a best step would be to have multiple (two) kinds of builders with
different disk sizes and blacklist/whitelist packages to them. I am aware that
idea is obvious, is there a Koji patch needed or even administration of two
kinds of builders is too much?
Jan
7:47 a.m.
On Wed, Jul 01, 2020 at 02:00:09PM +0200, Jan Kratochvil wrote:
On Wed, 01 Jul 2020 13:43:57 +0200, Stephen John Smoogen wrote:
> In the end we have limited resources and you are running into those
> limits. We can either have fewer builders with more disk space and a
> long queue of builds or we can have more builders with smaller disk
> space.
Good to know disk space really is a limiting factor there (as it is in general
cheap compared to CPUs+RAM).
Sure a best step would be to have multiple (two) kinds of builders
with different disk sizes and blacklist/whitelist packages to
them. I am aware that idea is obvious, is there a Koji patch needed
or even administration of two kinds of builders is too much?
I know from the RISC-V Koji instance that given enough permissions it
is possible to direct jobs to particular build hosts. We do it
manually for jobs we know will be very large/long (hello there gcc!)
Maybe a way to direct jobs based on regexp of their name would be a
useful feature to propose/implement upstream?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
8:12 a.m.
On Wednesday, July 1, 2020 2:00:09 PM CEST Jan Kratochvil wrote:
On Wed, 01 Jul 2020 13:43:57 +0200, Stephen John Smoogen wrote:
> In the end we have limited resources and you are running into those
> limits. We can either have fewer builders with more disk space and a
> long queue of builds or we can have more builders with smaller disk
> space.
Good to know disk space really is a limiting factor there (as it is in
general cheap compared to CPUs+RAM).
Sure a best step would be to have multiple (two) kinds of builders with
different disk sizes and blacklist/whitelist packages to them.
I like the idea but would suggest to rather implement denylist/allowlist
so that someone does not have to rename those lists later on:
https://www.redhat.com/en/blog/making-open-source-more-inclusive-eradicat...
Kamil
I am aware that
idea is obvious, is there a Koji patch needed or even administration
of two kinds of builders is too much?
Jan
8:14 a.m.
On Wed, 1 Jul 2020 at 08:09, Jan Kratochvil <jan.kratochvil(a)redhat.com> wrote:
On Wed, 01 Jul 2020 13:43:57 +0200, Stephen John Smoogen wrote:
> In the end we have limited resources and you are running into those
> limits. We can either have fewer builders with more disk space and a
> long queue of builds or we can have more builders with smaller disk
> space.
Good to know disk space really is a limiting factor there (as it is in general
cheap compared to CPUs+RAM).
Disk space is only cheap if you don't care about the speed or the
amount of usable disk space you have. [You can have a system with 200
TB of disk space from a bunch of 4TB sata disks but find that only 20
TB is speedy and the rest has a fall over in performance because none
of the drives or controllers have enough cache] Every time we have
tried to get larger disks, the amount of complaints for slow builders
goes up exponentially. Trying for other disk layouts to increase IO
ends up with other downtime complaints (oh look RAID-10 failed on 2
drives in the same pair.. there goes everything).
Disk space which meets the IO needs of builders ends up being as
expensive as CPU cores and RAM amounts. The faster the IO needs, the
smaller the disk space per disk. You can fix this by having more
disks, but the more disks per system, the higher the power and cooling
costs per system.
In the end, there is no such thing as a free lunch.
Sure a best step would be to have multiple (two) kinds of builders
with
different disk sizes and blacklist/whitelist packages to them. I am aware that
idea is obvious, is there a Koji patch needed or even administration of two
kinds of builders is too much?
I believe we have done this in the past for the x86_64 arch. The s390,
ppc64le, and aarch64 architectures have exceedingly limited resources
to try this with. At the moment, we are still trying to bring up what
we had in PHX2 to IAD2.
At the moment, the sysadmin team for Fedora are running on fumes. We
have worked several months of 10+ hour days+weekends to make the move
a success and we will be working at least a month more to bring up the
rest of the servers. At the moment we have about 10% of the hardware
we shipped back in operation and there are at least 2 to 3 weeks to
get the rest up. Getting up larger builders is going to have to wait.
--
Stephen J Smoogen.
8:38 a.m.
On Wed, 01 Jul 2020 15:14:09 +0200, Stephen John Smoogen wrote:
At the moment we have about 10% of the hardware
we shipped back in operation and there are at least 2 to 3 weeks to
get the rest up. Getting up larger builders is going to have to wait.
Sure there is no hurry with larger builders. I watch this problem of missing
debuginfo for years.
Thanks,
Jan
9:41 a.m.
On Wed, 1 Jul 2020 at 09:38, Jan Kratochvil <jan.kratochvil(a)redhat.com> wrote:
On Wed, 01 Jul 2020 15:14:09 +0200, Stephen John Smoogen wrote:
> At the moment we have about 10% of the hardware
> we shipped back in operation and there are at least 2 to 3 weeks to
> get the rest up. Getting up larger builders is going to have to wait.
Sure there is no hurry with larger builders. I watch this problem of missing
debuginfo for years.
I would like to apologize for my tone in earlier emails. I should have
said that we are going to look at doing this once we had gotten the
move done and instead I went into lecturing and condescending mode.
Thanks,
Jan
--
Stephen J Smoogen.
10:46 a.m.
On Wed, Jul 01, 2020 at 03:38:26PM +0200, Jan Kratochvil wrote:
On Wed, 01 Jul 2020 15:14:09 +0200, Stephen John Smoogen wrote:
> At the moment we have about 10% of the hardware
> we shipped back in operation and there are at least 2 to 3 weeks to
> get the rest up. Getting up larger builders is going to have to wait.
Sure there is no hurry with larger builders. I watch this problem of missing
debuginfo for years.
So, there's a lot of things we could do here... but I'd like to know
more details:
* Is this only on x86_64 that you need this? Or all arches?
* you're guessing you need 160GB? Is that pretty accurate?
The space on the builders currently has been somewhat constrained by the
space we have available along with a desire to keep things pretty
similar so we don't have 'special' machines.
Of course thats already out the window as different arches and all the
buildhw's are different machines.
Short term we can probibly spin up a x86_64 builder with more disk space
and direct chromium scratch builds there? Or... the buildhw-x86*
machines actually have more disk than the buildvm-x86 vm's... but it's
not fully allocated. I can fix that and they would have ~300GB or so
each, would that be enough? in that case you could submit and cancel
until you got a buildhw-x86 ?
Finally as people mentioned upthread, koji has a concept called
'channels'. We already have a number of them. The hub has a policy
config that can direct things to builders subscribed to specific
channels. For example, all the livemedia tasks run on hosts in the
livemedia channel.
Before the move I created a 'heavybuilder' channel for chromium
(official builds). This is because our normal aarch64 buildvm's take
about 24-30 hours to build it, but we have 2 buildhw-a64's that have a
ton of cpu and memory (but not much disk space so we can't put a lot of
vm's on them). I just yesterday got this resetup. These builders take an
hour or two to build aarch64 chromium.
Anyhow, I'd suggest filing an infrastructure ticket with your needs and
we can see what we can come up with either short or long term for you.
kevin
4:22 p.m.
On Wed, 01 Jul 2020 17:46:23 +0200, Kevin Fenzi wrote:
* Is this only on x86_64 that you need this? Or all arches?
Sure we should have this on all arches but I haven't tested non-x86_64 yet.
There possibly may be also some memory problems even on x86_64 (it builds on
my machine w/64GB but Koji has only 16GB + big swap).
* you're guessing you need 160GB? Is that pretty accurate?
$ du -shc `git ls-files` `cat sources |sed 's/^.*(\(.*\)).*$/\1/'`
chromium-83.0.4103.116-2/ BUILDROOT/
... ...
161G chromium-83.0.4103.116/
14G BUILDROOT/
176G total
I can fix that and they would have ~300GB or so each, would that be
enough?
It depends what other packages are being built on the same host, doesn't it?
in that case you could submit and cancel until you got a buildhw-x86
?
I am not Chromium maintainer (Tom 'spot' Callaway is). I am not even sure Spot
will accept this change (although then why not). I do not want to talk for
Spot if he likes to do the builds this way.
Anyhow, I'd suggest filing an infrastructure ticket with your
needs and
we can see what we can come up with either short or long term for you.
I will file a ticket after Spot says his opinion (or not).
Thanks,
Jan
7:04 p.m.
On Wed, Jul 01, 2020 at 11:22:26PM +0200, Jan Kratochvil wrote:
On Wed, 01 Jul 2020 17:46:23 +0200, Kevin Fenzi wrote:
> * Is this only on x86_64 that you need this? Or all arches?
Sure we should have this on all arches but I haven't tested non-x86_64 yet.
There possibly may be also some memory problems even on x86_64 (it builds on
my machine w/64GB but Koji has only 16GB + big swap).
> * you're guessing you need 160GB? Is that pretty accurate?
$ du -shc `git ls-files` `cat sources |sed 's/^.*(\(.*\)).*$/\1/'`
chromium-83.0.4103.116-2/ BUILDROOT/
... ...
161G chromium-83.0.4103.116/
14G BUILDROOT/
176G total
> I can fix that and they would have ~300GB or so each, would that be enough?
It depends what other packages are being built on the same host, doesn't it?
Yes, somewhat. The vast majority of the time that is the empty set.
It's only when there's a mass build of something do all the builders
really fill up.
> in that case you could submit and cancel until you got a
buildhw-x86 ?
I am not Chromium maintainer (Tom 'spot' Callaway is). I am not even sure Spot
will accept this change (although then why not). I do not want to talk for
Spot if he likes to do the builds this way.
I have fixed the buildhw-x86 boxes, they all now have ~365GB now.
Sorry, I misunderstood you... I thought you were wanting to test a
scratch build that needed more space. You are proposing a change that
will make official chromium builds take that extra space always?
Our very fast a64 builders do not always have that amount of space, so
it would have to fall back to the aarch64 buildvm's in that arch.
> Anyhow, I'd suggest filing an infrastructure ticket with
your needs and
> we can see what we can come up with either short or long term for you.
I will file a ticket after Spot says his opinion (or not).
Sure.
kevin
6:36 a.m.
On Thu, 02 Jul 2020 02:04:48 +0200, Kevin Fenzi wrote:
I have fixed the buildhw-x86 boxes, they all now have ~365GB now.
Yes, it has built fine:
https://koji.fedoraproject.org/koji/taskinfo?taskID=46455864
Sorry, I misunderstood you... I thought you were wanting to test a
scratch build that needed more space. You are proposing a change that
will make official chromium builds take that extra space always?
Yes (I can do scratch builds on my machine faster). But I have succeeded
getting buildhw-* only on the 4th time.
Our very fast a64 builders do not always have that amount of space,
so
it would have to fall back to the aarch64 buildvm's in that arch.
aarch64 needs 187G + BUILDROOT (which got deleted), it may be 205GB total.
i686 cannot build chromium with debuginfo as linker runs out of 32-bit address
space, one would have to cross-build it which is sure not worth it.
> I will file a ticket after Spot says his opinion (or not).
Sure.
Spot has not said anyway yet.
Jan
7:52 a.m.
On Thu, 02 Jul 2020 02:04:48 +0200, Kevin Fenzi wrote:
On Wed, Jul 01, 2020 at 11:22:26PM +0200, Jan Kratochvil wrote:
> I will file a ticket after Spot says his opinion (or not).
Sure.
https://src.fedoraproject.org/rpms/chromium/pull-request/27
12:55 a.m.
On Fri, 03 Jul 2020 14:52:00 +0200, Jan Kratochvil wrote:
On Thu, 02 Jul 2020 02:04:48 +0200, Kevin Fenzi wrote:
> On Wed, Jul 01, 2020 at 11:22:26PM +0200, Jan Kratochvil wrote:
> > I will file a ticket after Spot says his opinion (or not).
>
> Sure.
https://src.fedoraproject.org/rpms/chromium/pull-request/27
So Spot says:
I'm not sure that the additional build time and storage demands of
this are worthwhile in practice.
Not sure what to do when after all the compiler tools effort the debuginfo is
not worth even the time building it.
Jan
2:11 p.m.
On Tue, Jul 07, 2020 at 07:55:09AM +0200, Jan Kratochvil wrote:
On Fri, 03 Jul 2020 14:52:00 +0200, Jan Kratochvil wrote:
> On Thu, 02 Jul 2020 02:04:48 +0200, Kevin Fenzi wrote:
> > On Wed, Jul 01, 2020 at 11:22:26PM +0200, Jan Kratochvil wrote:
> > > I will file a ticket after Spot says his opinion (or not).
> >
> > Sure.
>
> https://src.fedoraproject.org/rpms/chromium/pull-request/27
So Spot says:
I'm not sure that the additional build time and storage demands of
this are worthwhile in practice.
Not sure what to do when after all the compiler tools effort the debuginfo is
not worth even the time building it.
I am not sure what to tell you here. Perhaps you could describe the
reason you are working on the chromium debuginfo? Is it broken? Missing?
Less useful that normal?
All I see in this thread is that you were working on it and it increased
disk space usage, I must have missed the background/goal here?
kevin
11:15 a.m.
On Tue, 07 Jul 2020 21:11:31 +0200, Kevin Fenzi wrote:
I am not sure what to tell you here. Perhaps you could describe the
reason you are working on the chromium debuginfo? Is it broken? Missing?
Less useful that normal?
Missing. Completely.
And it is not just about enabling it (=remove its disable) because with
default Fedora building options (no -fdebug-types-section and using DWZ) the
build will fail. That is because debuginfo will exceed 4GB which is the limit
of DWARF32 format and GNU tools do not much support DWARF64.
Enable debuginfo for x86_64 and aarch64.
https://src.fedoraproject.org/rpms/chromium/pull-request/27
All I see in this thread is that you were working on it and it
increased
disk space usage, I must have missed the background/goal here?
The goal is to have debuginfo for Chromium. Or why other Fedora packages have
debuginfo but Chromium does not?
Thanks,
Jan
1:05 p.m.
On Thu, Jul 09, 2020 at 06:15:54PM +0200, Jan Kratochvil wrote:
On Tue, 07 Jul 2020 21:11:31 +0200, Kevin Fenzi wrote:
> I am not sure what to tell you here. Perhaps you could describe the
> reason you are working on the chromium debuginfo? Is it broken? Missing?
> Less useful that normal?
Missing. Completely.
And it is not just about enabling it (=remove its disable) because with
default Fedora building options (no -fdebug-types-section and using DWZ) the
build will fail. That is because debuginfo will exceed 4GB which is the limit
of DWARF32 format and GNU tools do not much support DWARF64.
Enable debuginfo for x86_64 and aarch64.
https://src.fedoraproject.org/rpms/chromium/pull-request/27
> All I see in this thread is that you were working on it and it increased
> disk space usage, I must have missed the background/goal here?
The goal is to have debuginfo for Chromium. Or why other Fedora packages have
debuginfo but Chromium does not?
Well, if you can't convince the maintainer, you're welcome to escalate
it to FESCo I suppose. I don't think there's a guideline that requires
them to exist, but they are definitely handy sometimes.
Perhaps a compromise could be to generate them for rawhide builds, but
not stable releases and if someone needs to debug things ask them to use
the rawhide one/debuginfo?
kevin
PS: I'm subscribed to the list, there's no need to specifically cc me on
posts.
6:53 a.m.
On Fri, 10 Jul 2020 20:05:07 +0200, Kevin Fenzi wrote:
Perhaps a compromise could be to generate them for rawhide builds,
but
not stable releases and if someone needs to debug things ask them to use
the rawhide one/debuginfo?
That is not useful, system debuginfos are there to catch bugs easily and also
to catch difficult to reproduce bugs.
If you need to change the build to troubleshoot it then you can already build
a real debug version - particularly to use -O0 -g (instead of -O2 -g as in
Fedora) and also to enable package-specific compile-time debugging options.
I agree, there is slightly easier to download a Rawhide build than to build it
yourself but in both cases you are out of luck for unreproducible bugs.
Jan
1399
days inactive
1409
days old
17 comments
5 participants
participants (5)
-
Jan Kratochvil -
Kamil Dudka -
Kevin Fenzi -
Richard W.M. Jones -
Stephen John Smoogen