Remove gcc, gcc-c++ and make from minimal build root
Vít Ondruch
vondruch at redhat.com
Tue Jan 13 15:03:20 UTC 2015
Forgot to add this was measured on my Lenovo T440s, with SSD and 12 GB
RAM, running F21 with mock 1.2.3, configured to use LVM plugin and DNF
(see attached config files). And network? Who knows, probably fast.
Should be 1Gb/s if I am not mistaken.
Not sure how similar/different the tests could look on Fedoras Koji
builders.
Vít
Dne 13.1.2015 v 15:56 Vít Ondruch napsal(a):
> Dne 13.1.2015 v 14:02 Richard W.M. Jones napsal(a):
>> On Tue, Jan 13, 2015 at 01:31:06PM +0100, Sven Lankes wrote:
>>> On Tue, Jan 13, 2015 at 12:16:42PM +0000, Richard W.M. Jones wrote:
>>>
>>>> That's simply an idiotic thing to say. What is the progress involved
>>>> in adding a new BR to thousands of packages?
>>> Am I wrong assuming that it would be rather easy to add the BR to all
>>> packages semi-automatically previous to a mass rebuild so that people
>>> who want to get rid of gcc for their packages can easily remove it?
>>>
>>> What am I missing?
>> What is missing is there is no quantifcation yet of the benefit.
>> How much faster will builds be?
>
> Since you are asking I did 5 times:
>
> $ mock -r minimal --scrub all && time mock -r minimal --init
>
> Where the "Total" is DNF's output of package download speed and amount
> of downloaded data obtained from mock output. Other DNF information
> are shown on top of each section as well.
>
>
> == with current minimal build root, i.e. "install @buildsys-build"
>
> Install 164 Packages
>
> Total download size: 119 M
> Installed size: 445 M
>
> 1)
> Total 26 MB/s | 119 MB
> 00:04
> real 0m51.364s
> user 0m33.050s
> sys 0m6.454s
>
> 2)
> Total 13 MB/s | 119 MB
> 00:09
> real 0m51.479s
> user 0m33.024s
> sys 0m3.400s
>
> 3)
> Total 19 MB/s | 119 MB
> 00:06
> real 0m52.373s
> user 0m33.171s
> sys 0m7.121s
>
> 4)
> Total 21 MB/s | 119 MB
> 00:05
> real 0m54.213s
> user 0m32.999s
> sys 0m7.122s
>
> 5)
> Total 13 MB/s | 119 MB
> 00:09
> real 0m56.093s
> user 0m32.708s
> sys 0m5.745s
>
>
> == without gcc, gcc-c++ and make, i.e. "install rpm-build shadow-utils
> util-linux which"
>
> Install 153 Packages
>
> Total download size: 81 M
> Installed size: 350 M
>
> 1)
> Total 14 MB/s | 81 MB
> 00:05
> real 0m49.789s
> user 0m28.926s
> sys 0m6.864s
>
> 2)
> Total 17 MB/s | 81 MB
> 00:04
> real 0m44.010s
> user 0m28.965s
> sys 0m3.434s
>
> 3)
> Total 18 MB/s | 81 MB
> 00:04
> real 0m45.860s
> user 0m29.032s
> sys 0m5.730s
>
> 4)
> Total 19 MB/s | 81 MB
> 00:04
> real 0m42.047s
> user 0m28.944s
> sys 0m3.099s
>
> 5)
> Total 14 MB/s | 81 MB
> 00:05
> real 0m47.079s
> user 0m29.839s
> sys 0m3.978s
>
>
> == without gcc, gcc-c++, make and without perl (rhbz#1158860, note
> that the rpm packages are local in this case), i.e. "install rpm-build
> shadow-utils util-linux which"
>
> Install 124 Packages
>
> Total size: 69 M
> Total download size: 68 M
> Installed size: 310 M
>
> 1)
> Total 11 MB/s | 68 MB
> 00:06
> real 0m44.725s
> user 0m27.644s
> sys 0m6.126s
>
> 2)
> Total 12 MB/s | 68 MB
> 00:05
> real 0m46.253s
> user 0m27.624s
> sys 0m7.056s
>
> 3)
> Total 6.5 MB/s | 68 MB
> 00:10
> real 0m50.296s
> user 0m27.567s
> sys 0m6.064s
>
> 4)
> Total 12 MB/s | 68 MB
> 00:05
> real 0m46.078s
> user 0m27.581s
> sys 0m7.698s
>
> 5)
> Total 16 MB/s | 68 MB
> 00:04
> real 0m44.307s
> user 0m27.830s
> sys 0m5.293s
>
> == Only rpmbuild install "install rpm-build shadow-utils"
>
> Install 111 Packages
>
> Total size: 61 M
> Total download size: 60 M
> Installed size: 286 M
>
> 1)
> Total 20 MB/s | 60 MB
> 00:03
> real 0m37.848s
> user 0m26.158s
> sys 0m2.676s
>
> 2)
> Total 13 MB/s | 60 MB
> 00:04
> real 0m44.756s
> user 0m26.442s
> sys 0m5.185s
>
> 3)
> Total 16 MB/s | 60 MB
> 00:03
> real 0m42.703s
> user 0m26.118s
> sys 0m3.174s
>
> 4)
> Total 13 MB/s | 60 MB
> 00:04
> real 0m42.365s
> user 0m26.439s
> sys 0m4.193s
>
> 5)
> Total 5.9 MB/s | 60 MB
> 00:10
> real 0m48.940s
> user 0m26.608s
> sys 0m4.576s
>
> == Minimal root I can shell in, i.e. "install bash shadow-utils"
>
> Install 48 Packages
>
> Total download size: 31 M
> Installed size: 176 M
>
> 1)
> Total 8.3 MB/s | 31 MB
> 00:03
> real 0m36.745s
> user 0m22.019s
> sys 0m4.936s
>
> 2)
> Total 9.5 MB/s | 31 MB
> 00:03
> real 0m32.608s
> user 0m21.805s
> sys 0m2.719s
>
> 3)
> Total 11 MB/s | 31 MB
> 00:02
> real 0m34.111s
> user 0m22.170s
> sys 0m3.651s
>
> 4)
> Total 12 MB/s | 31 MB
> 00:02
> real 0m41.141s
> user 0m22.595s
> sys 0m5.535s
>
> 5)
> Total 8.4 MB/s | 31 MB
> 00:03
> real 0m34.303s
> user 0m21.851s
> sys 0m3.839s
>
>
> And several times during the test I encountered bug similar to:
>
> Downloading Packages:
> Error: Error downloading packages:
> Curl error: Failure when receiving data from the peer for
> ftp://mirror.slu.cz/fedora/linux/development/rawhide/x86_64/os/Packages/e/elfutils-libelf-0.161-1.fc22.x86_64.rpm
> [response reading failed]
>
> Less packages to download, less possible issues.
>
>
> You can interpret these data yourself, but with less packages in build
> root, I can see:
>
> 1) Saved build time
> 2) Saved network bandwidth
> 3) Saved storage
> 4) Less things to break
>
>
>> Versus how long will it take to
>> either modify all these packages / write and run the script to do
>> that? Is A > B?
>
> There are things which are hard to quantify. If you want to prohibit
> any change, this is probably right question to ask. Of course
> answering such questions also takes some time, I hope you will count
> it into the final equation at the end, although hard to say on which
> side it should belong.
>
>
> Vít
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20150113/bf5abfdb/attachment-0001.html>
-------------- next part --------------
config_opts['root'] = 'minimal'
config_opts['target_arch'] = 'x86_64'
config_opts['legal_host_arches'] = ('x86_64',)
#config_opts['chroot_setup_cmd'] = 'install @buildsys-build'
#config_opts['chroot_setup_cmd'] = 'install rpm-build shadow-utils util-linux which'
#config_opts['chroot_setup_cmd'] = 'install rpm-build shadow-utils'
config_opts['chroot_setup_cmd'] = 'install bash shadow-utils'
#rpm-build
#shadow-utils
#bash
#bzip2
#coreutils
#cpio
#diffutils
#fedora-release
#findutils
#gawk
#gcc
#gcc-c++
#grep
#gzip
#info
#make
#patch
#redhat-rpm-config
#sed
#tar
#unzip
#util-linux
#which
#xz
config_opts['dist'] = 'rawhide' # only useful for --resultdir variable subst
config_opts['extra_chroot_dirs'] = [ '/run/lock', ]
config_opts['releasever'] = '22'
config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1
syslog_ident=mock
syslog_device=
# repos
[fedora]
name=fedora
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch
failovermethod=priority
[local]
name=local
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
cost=2000
enabled=0
[debug]
name=Fedora Rawhide - x86_64 - Debug
failovermethod=priority
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=$basearch
enabled=0
[rpm]
name=rpm
baseurl=file:///var/lib/mock/fedora-rawhide-x86_64/result/
enabled=0
"""
-------------- next part --------------
# mock defaults
# vim:tw=0:ts=4:sw=4:et:
#
# This config file is for site-specific default values that apply across all
# configurations. Options specified in this config file can be overridden in
# the individual mock config files.
#
# The site-defaults.cfg delivered by default has NO options set. Only set
# options here if you want to override the defaults.
#
# Entries in this file follow the same format as other mock config files.
# config_opts['foo'] = bar
#############################################################################
#
# Things that we recommend you set in site-defaults.cfg:
#
# config_opts['basedir'] = '/var/lib/mock/'
# config_opts['cache_topdir'] = '/var/cache/mock'
# Note: the path pointed to by basedir and cache_topdir must be owned
# by group 'mock' and must have mode: g+rws
# config_opts['rpmbuild_timeout'] = 0
# config_opts['use_host_resolv'] = True
# You can configure log format to pull from logging.ini formats of these names:
# config_opts['build_log_fmt_name'] = "unadorned"
# config_opts['root_log_fmt_name'] = "detailed"
# config_opts['state_log_fmt_name'] = "state"
#
# mock will normally set up a minimal chroot /dev.
# If you want to use a pre-configured /dev, disable this and use the bind-mount
# plugin to mount your special /dev
# config_opts['internal_dev_setup'] = True
#
# internal_setarch defaults to 'True' if the python 'ctypes' package is
# available. It is in the python std lib on >= python 2.5. On older versions,
# it is available as an addon. On systems w/o ctypes, it will default to 'False'
# config_opts['internal_setarch'] = False
#
# the cleanup_on_* options allow you to automatically clean and remove the
# mock build directory, but only take effect if --resultdir is used.
# config_opts provides fine-grained control. cmdline only has big hammer
#
# config_opts['cleanup_on_success'] = 1
# config_opts['cleanup_on_failure'] = 1
# The default package manager is Yum
config_opts['package_manager'] = 'dnf'
# If you want to use DNF, set it to 'dnf'. To use DNf you need to have dnf and
# dnf-plugins-core installed
# You can configure Yum, DNF, rpm and rpmbuild executable paths if you need to
# use different versions that the system-wide ones
# config_opts['yum_command'] = '/usr/bin/yum'
# config_opts['yum_builddep_command'] = '/usr/bin/yum-builddep'
# config_opts['dnf_command'] = '/usr/bin/dnf'
# config_opts['rpm_command'] = '/bin/rpm'
# config_opts['rpmbuild_command'] = '/usr/bin/rpmbuild'
#
# By default a Yum/DNF update is performed before each rebuild
# config_opts['update_before_build'] = True
# if you want mock to automatically run createrepo on the rpms in your
# resultdir.
# config_opts['createrepo_on_rpms'] = False
# config_opts['createrepo_command'] = '/usr/bin/createrepo_c -d -q -x *.src.rpm'
# if you want mock to backup the contents of a result dir before clean
# config_opts['backup_on_clean'] = False
# config_opts('backup_base_dir'] = config_opts['basedir'] + "backup"
# if you want to speed up the package installation and the build process, mock
# can use nosync library to skip fsync and related calls from programs called
# from within mock. It needs nosync library to be installed and for multilib
# target, it requires both architectures of nosync to be present. If these
# conditions aren't met, it won't be enabled
config_opts['nosync'] = True
# if you cannot or don't want to install both architectures of nosync and still
# want mock to use it, you can force it, but then expect a lot of (generally
# harmless) error messages from ld.so when a 32bit program is executed
# config_opts['nosync_force'] = False
#############################################################################
#
# plugin related. Below are the defaults. Change to suit your site
# policy. site-defaults.cfg is a good place to do this.
#
# NOTE: Some of the caching options can theoretically affect build
# reproducability. Change with care.
#
# config_opts['plugin_conf']['package_state_enable'] = False
# config_opts['plugin_conf']['ccache_enable'] = True
# config_opts['plugin_conf']['ccache_opts'] = {}
# config_opts['plugin_conf']['ccache_opts']['max_cache_size'] = '4G'
# config_opts['plugin_conf']['ccache_opts']['compress'] = None
# config_opts['plugin_conf']['ccache_opts']['dir'] = "%(cache_topdir)s/%(root)s/ccache/"
# config_opts['plugin_conf']['yum_cache_enable'] = True
# config_opts['plugin_conf']['yum_cache_opts'] = {}
# config_opts['plugin_conf']['yum_cache_opts']['max_age_days'] = 30
# config_opts['plugin_conf']['yum_cache_opts']['dir'] = "%(cache_topdir)s/%(root)s/yum_cache/"
config_opts['plugin_conf']['root_cache_enable'] = False
# config_opts['plugin_conf']['root_cache_opts'] = {}
# config_opts['plugin_conf']['root_cache_opts']['max_age_days'] = 15
# config_opts['plugin_conf']['root_cache_opts']['dir'] = "%(cache_topdir)s/%(root)s/root_cache/"
# config_opts['plugin_conf']['root_cache_opts']['compress_program'] = "pigz"
# config_opts['plugin_conf']['root_cache_opts']['extension'] = ".gz"
# config_opts['plugin_conf']['root_cache_opts']['exclude_dirs'] = ["./proc", "./sys", "./dev",
# "./tmp/ccache", "./var/cache/yum" ]
#
# bind mount plugin is enabled by default but has no configured directories to
# mount
# config_opts['plugin_conf']['bind_mount_enable'] = True
# config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/host/path', '/bind/mount/path/in/chroot/' ))
#
# config_opts['plugin_conf']['tmpfs_enable'] = False
# config_opts['plugin_conf']['tmpfs_opts'] = {}
# config_opts['plugin_conf']['tmpfs_opts']['required_ram_mb'] = 1024
# config_opts['plugin_conf']['tmpfs_opts']['max_fs_size'] = '768m'
# config_opts['plugin_conf']['tmpfs_opts']['mode'] = '0755'
# config_opts['plugin_conf']['tmpfs_opts']['keep_mounted'] = False
#
# config_opts['plugin_conf']['chroot_scan_enable'] = False
# config_opts['plugin_conf']['chroot_scan_opts'] = [ "core(\.\d+)?", "\.log$",]
#
# lvm_root plugin is not enabled by default and is distributed in separate
# subpackage mock-lvm. If you want to use it, it's recommended to disable the
# root_cache plugin, otherwise you'd be caching twice.
config_opts['plugin_conf']['lvm_root_enable'] = True
# config_opts['plugin_conf']['lvm_root_opts'] = {}
# You need to give it a volume group with sufficient space. It won't touch any
# of the existing logical volumes, so you can use the same volume group you use
# for other purposes. It requires a name of the VG (not device path).
# config_opts['plugin_conf']['lvm_root_opts']['volume_group'] = 'my_vg'
# You need to specify the size that will mock's thinpool occupy. For regular
# packages with small set of dependencies, 2G should suffice. For large packages
# such as libreoffice, you should set it to at least 5 GB otherwise you may run
# out of space and the build process will be blocked
# config_opts['plugin_conf']['lvm_root_opts']['size'] = '2G'
# You can specify alternative pool metadata size, format is the same as size.
# Default value is determined by lvcreate based on size
# config_opts['plugin_conf']['lvm_root_opts']['poolmetadatasize'] = None
config_opts['plugin_conf']['lvm_root_opts'] = {
'volume_group': 'vg_fedora',
'size': '15G',
'pool_name': 'mock',
}
#
### If you want to compress mock log, enable this plugin
# config_opts['plugin_conf']['compress_logs_enable'] = False
# config_opts['plugin_conf']['compress_logs_opts'] = {}
### Command used to compress logs - e.g. "/usr/bin/xz -9"
# config_opts['plugin_conf']['compress_logs_opts']['command'] = ""
#############################################################################
#
# environment for chroot
#
# config_opts['environment']['TERM'] = 'vt100'
# config_opts['environment']['SHELL'] = '/bin/bash'
# config_opts['environment']['HOME'] = '/builddir'
# config_opts['environment']['HOSTNAME'] = 'mock'
# config_opts['environment']['PATH'] = '/usr/bin:/bin:/usr/sbin:/sbin'
# config_opts['environment']['PROMPT_COMMAND'] = 'printf "\033]0;<mock-chroot>\007<mock-chroot>"'
# config_opts['environment']['LANG'] = os.environ.setdefault('LANG', 'en_US.UTF-8')
# config_opts['environment']['TZ'] = os.environ.setdefault('TZ', 'EST5EDT')
#############################################################################
#
# Things that you can change, but we dont recommend it:
# config_opts['chroothome'] = '/builddir'
# config_opts['clean'] = True
#############################################################################
#
# Things that must be adjusted if SCM integration is used:
#
# config_opts['scm'] = True
# config_opts['scm_opts']['method'] = 'git'
# config_opts['scm_opts']['cvs_get'] = 'cvs -d /srv/cvs co SCM_BRN SCM_PKG'
# config_opts['scm_opts']['git_get'] = 'git clone SCM_BRN git://localhost/SCM_PKG.git SCM_PKG'
# config_opts['scm_opts']['svn_get'] = 'svn co file:///srv/svn/SCM_PKG/SCM_BRN SCM_PKG'
# config_opts['scm_opts']['spec'] = 'SCM_PKG.spec'
# config_opts['scm_opts']['ext_src_dir'] = '/dev/null'
# config_opts['scm_opts']['write_tar'] = True
# config_opts['scm_opts']['git_timestamps'] = True
# config_opts['scm_opts']['exclude_vcs'] = True
# These options are also recognized but usually defined in cmd line
# with --scm-option package=<pkg> --scm-option branch=<branch>
# config_opts['scm_opts']['package'] = 'mypkg'
# config_opts['scm_opts']['branch'] = 'master'
#############################################################################
#
# Things that are best suited for individual chroot config files:
#
# MUST SET (in individual chroot cfg file):
# config_opts['root'] = 'name-of-yum-build-dir'
# config_opts['target_arch'] = 'i386'
# config_opts['yum.conf'] = ''
# config_opts['yum_common_opts'] = []
#
# CAN SET, defaults usually work ok:
# config_opts['chroot_setup_cmd'] = 'install buildsys-build'
# config_opts['log_config_file'] = 'logging.ini'
# config_opts['more_buildreqs']['srpm_name-version-release'] = 'dependencies'
# config_opts['macros']['%Add_your_macro_name_here'] = "add macro value here"
# config_opts['files']['path/name/no/leading/slash'] = "put file contents here."
# config_opts['chrootuid'] = os.getuid()
# config_opts['releasever'] = '20'
# If you change chrootgid, you must also change "mock" to the correct group
# name in this line of the mock PAM config:
# auth sufficient pam_succeed_if.so user ingroup mock use_uid quiet
# config_opts['chrootgid'] = grp.getgrnam("mock")[2]
# config_opts['useradd'] = '/usr/sbin/useradd -m -u %(uid)s -g %(gid)s -d %(home)s -n %(user)s' # Fedora/RedHat
#
# Security related
# config_opts['no_root_shells'] = False
#
# Proxy settings (https_proxy, ftp_proxy, and no_proxy can also be set)
# config_opts['http_proxy'] = 'http://localhost:3128'
#
# Extra dirs to be created when the chroot is initialized
# This is just a list of strings representing chroot paths such as:
# [ '/run/lock', ]
# config_opts['extra_chroot_dirs'] = []
More information about the devel
mailing list