mergerepos dies with IOError: [Errno 28] No space left on device

[John Florian]

I'm trying to get a new Koji deployment up and running atop Fedora 21.  I'm mostly
there (I think).  I was able to "koji regen-repo ..." but my first build
attempts were failing due to not having the useradd tool in the build root, despite this:

koji list-groups dist-fedora21-build
build  [dist-fedora21-build]
  bash: None, default  [dist-fedora21-build]
  bzip2: None, default  [dist-fedora21-build]
  coreutils: None, default  [dist-fedora21-build]
  cpio: None, default  [dist-fedora21-build]
  diffutils: None, default  [dist-fedora21-build]
  fedora-release: None, default  [dist-fedora21-build]
  findutils: None, default  [dist-fedora21-build]
  gawk: None, default  [dist-fedora21-build]
  gcc: None, default  [dist-fedora21-build]
  gcc-c++: None, default  [dist-fedora21-build]
  grep: None, default  [dist-fedora21-build]
  gzip: None, default  [dist-fedora21-build]
  info: None, default  [dist-fedora21-build]
  make: None, default  [dist-fedora21-build]
  patch: None, default  [dist-fedora21-build]
  redhat-rpm-config: None, default  [dist-fedora21-build]
  rpm-build: None, default  [dist-fedora21-build]
  sed: None, default  [dist-fedora21-build]
  shadow-utils: None, default  [dist-fedora21-build]
  tar: None, default  [dist-fedora21-build]
  unzip: None, default  [dist-fedora21-build]
  util-linux: None, default  [dist-fedora21-build]
  which: None, default  [dist-fedora21-build]
  xz: None, default  [dist-fedora21-build]
srpm-build  [dist-fedora21-build]
  bash: None, default  [dist-fedora21-build]
  curl: None, default  [dist-fedora21-build]
  cvs: None, default  [dist-fedora21-build]
  fedora-release: None, default  [dist-fedora21-build]
  fedpkg-minimal: None, default  [dist-fedora21-build]
  gnupg2: None, default  [dist-fedora21-build]
  make: None, default  [dist-fedora21-build]
  redhat-rpm-config: None, default  [dist-fedora21-build]
  rpm-build: None, default  [dist-fedora21-build]
  shadow-utils: None, default  [dist-fedora21-build

I thus reasoned this is because yum has no idea where to get shadow-utils from which I
could rectify by setting up an external-repo pointing to our local mirror of the Fedora 21
repo.  Keeping it simple, I ignored updates and stuck to the base release repo only for
the moment.  With the ext. repo added, I tried the regen-repo command again and now I see
from the mergerepos.log:

$ /usr/libexec/kojid/mergerepos -a x86_64 -b
/mnt/koji/repos/dist-fedora21-build/20/x86_64/blocklist -o /var/tmp/koji/tasks/66/66/repo
-g /mnt/koji/repos/dist-fedora21-build/20/groups/comps.xml -r
file:///var/tmp/koji/tasks/66/66/repo_20_premerge/ -r
http://mdct-00fs.dartcontainer.com/ftp/pub/fedora/21/Everything/x86_64/os/
JF: Set self.yumbase.conf.cachedir: /tmp/tmpyXkTJb
Adding repo: file:///var/tmp/koji/tasks/66/66/repo_20_premerge/
Adding repo: http://mdct-00fs.dartcontainer.com/ftp/pub/fedora/21/Everything/x86_64/os/
JF: Writing origins: /tmp/tmpyXkTJb/pkgorigins
Traceback (most recent call last):
  File "/usr/libexec/kojid/mergerepos", line 278, in <module>
    main(sys.argv[1:])
  File "/usr/libexec/kojid/mergerepos", line 272, in main
    merge.merge_repos()
  File "/usr/libexec/kojid/mergerepos", line 162, in merge_repos
    self.sort_and_filter()
  File "/usr/libexec/kojid/mergerepos", line 232, in sort_and_filter
    origins.write('%s\t%s\n' % (pkg_nvra, repo.urls[0]))
IOError: [Errno 28] No space left on device

The lines prefixed "JF: " are my own added messages since the original error
didn't provide any details about what device was full.  Looking at
RepoMerge.__init__(),  I don't see any configurable way to change where the cachedir
is located and since /tmp is tmpfs this currently seems doomed.  Have I missed something
or done something stupid or is this a bug?

--
John Florian

[John Florian]

I was able to resolve this issue, but only by hacking on /usr/libexec/kojid/mergerepos
(from koji-builder-1.9.0-10.fc21.gitcd45e886.noarch), specifically with:

--- /usr/libexec/kojid/mergerepos.orig  2015-01-27 23:41:00.000000000 -0500
+++ /usr/libexec/kojid/mergerepos       2015-04-01 17:06:36.164997686 -0400
@@ -125,7 +125,7 @@
             self.yumbase.preconf.debuglevel = 2
         else:
             self.yumbase._getConfig('/dev/null', init_plugins=False,
debuglevel=2)
-        self.yumbase.conf.cachedir = tempfile.mkdtemp()
+        self.yumbase.conf.cachedir = tempfile.mkdtemp(dir='/var/tmp')
         self.yumbase.conf.cache = 0
         self.archlist = arches
         self.mdconf.groupfile = groupfile

Obviously this is a dirty hack.  I suspect though the more correct approach would be to
use the "workdir" setting from kojid.conf.  I was rather surprised when it
didn't do this already, but then workdir was relocated not all that long ago[1].

[1] https://lists.fedoraproject.org/pipermail/buildsys/2014-May/004293.html

FWIW, this was failing on a VM with 2 GiB of RAM.  With the above patch, my repo-gen ran
longer, but now was getting SIGKILL, from the OOMK as it turns out.  Raising the VM to 4
GiB avoided the visit from the OOMK and I have no idea if that would've made my tmpfs
/tmp large enough to avoid this whole mess or not.

--
John Florian

From: buildsys-bounces(a)lists.fedoraproject.org
[mailto:[e-mail]] On Behalf Of John Florian
Sent: Wednesday, April 01, 2015 14:46
To: buildsys
Subject: mergerepos dies with IOError: [Errno 28] No space left on device

I'm trying to get a new Koji deployment up and running atop Fedora 21.  I'm mostly
there (I think).  I was able to "koji regen-repo ..." but my first build
attempts were failing due to not having the useradd tool in the build root, despite this:

koji list-groups dist-fedora21-build
build  [dist-fedora21-build]
  bash: None, default  [dist-fedora21-build]
  bzip2: None, default  [dist-fedora21-build]
  coreutils: None, default  [dist-fedora21-build]
  cpio: None, default  [dist-fedora21-build]
  diffutils: None, default  [dist-fedora21-build]
  fedora-release: None, default  [dist-fedora21-build]
  findutils: None, default  [dist-fedora21-build]
  gawk: None, default  [dist-fedora21-build]
  gcc: None, default  [dist-fedora21-build]
  gcc-c++: None, default  [dist-fedora21-build]
  grep: None, default  [dist-fedora21-build]
  gzip: None, default  [dist-fedora21-build]
  info: None, default  [dist-fedora21-build]
  make: None, default  [dist-fedora21-build]
  patch: None, default  [dist-fedora21-build]
  redhat-rpm-config: None, default  [dist-fedora21-build]
  rpm-build: None, default  [dist-fedora21-build]
  sed: None, default  [dist-fedora21-build]
  shadow-utils: None, default  [dist-fedora21-build]
  tar: None, default  [dist-fedora21-build]
  unzip: None, default  [dist-fedora21-build]
  util-linux: None, default  [dist-fedora21-build]
  which: None, default  [dist-fedora21-build]
  xz: None, default  [dist-fedora21-build]
srpm-build  [dist-fedora21-build]
  bash: None, default  [dist-fedora21-build]
  curl: None, default  [dist-fedora21-build]
  cvs: None, default  [dist-fedora21-build]
  fedora-release: None, default  [dist-fedora21-build]
  fedpkg-minimal: None, default  [dist-fedora21-build]
  gnupg2: None, default  [dist-fedora21-build]
  make: None, default  [dist-fedora21-build]
  redhat-rpm-config: None, default  [dist-fedora21-build]
  rpm-build: None, default  [dist-fedora21-build]
  shadow-utils: None, default  [dist-fedora21-build

I thus reasoned this is because yum has no idea where to get shadow-utils from which I
could rectify by setting up an external-repo pointing to our local mirror of the Fedora 21
repo.  Keeping it simple, I ignored updates and stuck to the base release repo only for
the moment.  With the ext. repo added, I tried the regen-repo command again and now I see
from the mergerepos.log:

$ /usr/libexec/kojid/mergerepos -a x86_64 -b
/mnt/koji/repos/dist-fedora21-build/20/x86_64/blocklist -o /var/tmp/koji/tasks/66/66/repo
-g /mnt/koji/repos/dist-fedora21-build/20/groups/comps.xml -r
file:///var/tmp/koji/tasks/66/66/repo_20_premerge/<file:///\\var\tmp\koji\tasks\66\66\repo_20_premerge\>
-r http://mdct-00fs.dartcontainer.com/ftp/pub/fedora/21/Everything/x86_64/os/
JF: Set self.yumbase.conf.cachedir: /tmp/tmpyXkTJb
Adding repo:
file:///var/tmp/koji/tasks/66/66/repo_20_premerge/<file:///\\var\tmp\koji\tasks\66\66\repo_20_premerge\>
Adding repo: http://mdct-00fs.dartcontainer.com/ftp/pub/fedora/21/Everything/x86_64/os/
JF: Writing origins: /tmp/tmpyXkTJb/pkgorigins
Traceback (most recent call last):
  File "/usr/libexec/kojid/mergerepos", line 278, in <module>
    main(sys.argv[1:])
  File "/usr/libexec/kojid/mergerepos", line 272, in main
    merge.merge_repos()
  File "/usr/libexec/kojid/mergerepos", line 162, in merge_repos
    self.sort_and_filter()
  File "/usr/libexec/kojid/mergerepos", line 232, in sort_and_filter
    origins.write('%s\t%s\n' % (pkg_nvra, repo.urls[0]))
IOError: [Errno 28] No space left on device

The lines prefixed "JF: " are my own added messages since the original error
didn't provide any details about what device was full.  Looking at
RepoMerge.__init__(),  I don't see any configurable way to change where the cachedir
is located and since /tmp is tmpfs this currently seems doomed.  Have I missed something
or done something stupid or is this a bug?

--
John Florian