Bootstrapping New Koji Server

Tue Oct 22 22:17:12 UTC 2013

On 10/22/13 1:56 PM, John.Florian at dart.biz wrote:
> First off, I apologize for the lengthy message, but I hope to eventually
> contribute to this project via code and/or docs.  (I've already updated
> the Server Setup wiki page where I felt knowledgeable enough.)
>
> I'm loosely following the wiki page[1] for this process but may have
> myself confused about what some steps do and what overall is required.
>   I want a Koji instance for building internally developed packages.  I
> would rarely, if ever, want to rebuild a package already provided in
> Fedora.  I've got lots of experience using rpmbuild and recently a
> little with mock, but am trying for a full fledged setup with Koji now
> to improve my workflow.  For now, I am focusing on builds for F19 only,
> but will eventually add others to basically follow Fedora's EoL schedule.
>
> Thus far, I have:
>
>  1. A basic setup working and can now login thanks to Mike's excellent
>     help.
>  2. A "f19" tag and a child "f19-build" tag for arches i386 and x86_64.
>  3. Successfully imported the SRPMs for my locally developed packages
>     only.  I didn't import any binary RPMs as I wanted to see that Koji
>     could do that as a test.  I also didn't import anything of the
>     Fedora-provided packages since I'm not interested in building those.

You should never import srpms without importing the corresponding binary 
rpms.  Importing is only used to bootstrap an environment, and 
bootstrapping requires that binary rpms are available to populate the 
minimal buildroot or resolve BuildRequires.  Import only a srpm creates 
a new "build" in Koji with no binary rpms, which can prevent inherited 
binary rpms or binary rpms from external repos from getting used.

>  4. A "f19" build target.
>  5. A build group named "build" associated with my "f19-build" tag.
> 6.
>
> I've tried doing a build of one of my SRPMs that I had imported and keep
> meeting failure.  Examination of the root.log revealed only a few
> things, but none that sounded to worrisome:
>
> DEBUG util.py:266:  There is no installed groups file.
> DEBUG util.py:266:  Warning: Group build does not have any packages to
> install.
>
> When I got to the mock_output.log however, I saw this which I'm guessing
> is the reason for failure:
>
> ERROR: Could not find useradd in chroot, maybe the install failed?
>
> Seeing this I figured it was due to skipping the step for populating the
> "build" group with packages to be installed in the minimal buildroot.  I
> skipped this because the instructions fail for me since I my FAS account
> isn't set up sufficiently to let me to auth against the real Fedora Koji
> instance, i.e., I don't have a signed user cert.

You shouldn't need a cert to get the information required, Koji can be 
called anonymously for read-only operations.  Try using "koji --noauth".

> Knowing that my
> current mock setup can manage this, I dug into its config and stumbled
> upon the "buildsys-build" package name and thus came up with:
>
> $ yum group info buildsys-build
> Loaded plugins: langpacks
> There is no installed groups file.
> Maybe run: yum groups mark convert
>
> Group: Buildsystem building group
>   Group-Id: buildsys-build
>   Mandatory Packages:
>      bash
>      bzip2
>      coreutils
>      cpio
>      diffutils
>      fedora-release
>      findutils
>      gawk
>      gcc
>      gcc-c++
>      grep
>      gzip
>      info
>      make
>      patch
>      redhat-rpm-config
>      rpm-build
>      sed
>      shadow-utils
>      tar
>      unzip
>      util-linux
>      which
>      xz
>
> This looks like a minimal buildroot to me, so I commenced with that list:
>
> $ koji add-group-pkg f19-build build bash bzip2 coreutils cpio diffutils
> fedora-release findutils gawk gcc gcc-c++ grep gzip info make patch
> redhat-rpm-config rpm-build sed shadow-utils tar unzip util-linux which xz
>
> I believe these are now "in place" correctly since:
>
> $ koji list-groups f19-build
> build  [f19-build]
>    bash: None, default  [f19-build]
>    bzip2: None, default  [f19-build]
>    coreutils: None, default  [f19-build]
>    cpio: None, default  [f19-build]
>    diffutils: None, default  [f19-build]
>    fedora-release: None, default  [f19-build]
>    findutils: None, default  [f19-build]
>    gawk: None, default  [f19-build]
>    gcc: None, default  [f19-build]
>    gcc-c++: None, default  [f19-build]
>    grep: None, default  [f19-build]
>    gzip: None, default  [f19-build]
>    info: None, default  [f19-build]
>    make: None, default  [f19-build]
>    patch: None, default  [f19-build]
>    redhat-rpm-config: None, default  [f19-build]
>    rpm-build: None, default  [f19-build]
>    sed: None, default  [f19-build]
>    shadow-utils: None, default  [f19-build]
>    tar: None, default  [f19-build]
>    unzip: None, default  [f19-build]
>    util-linux: None, default  [f19-build]
>    which: None, default  [f19-build]
>    xz: None, default  [f19-build]
>
> 1.
>
> However, when I resubmitted my build request, it failed and I see the
> same warnings/errors in the logs.  I also see this in /var/log/kojid.log:
>
> 2013-10-22 16:48:57,582 [DEBUG] koji.TaskManager: bins: {'1:noarch': 1,
> '2:noarch': 1, '1:x86_64': 1, '1:i386': 1, '2:x86_64': 1, '2:i386': 1}
> 2013-10-22 16:49:12,598 [DEBUG] koji: Traceback (most recent call last):
>    File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1895,
> in _callMethod
>      return self._sendCall(handler, headers, request)
>    File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1806,
> in _sendCall
>      return self._sendOneCall(handler, headers, request)
>    File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1825,
> in _sendOneCall
>      cnx.send(request)
>    File "/usr/lib64/python2.7/httplib.py", line 805, in send
>      self.sock.sendall(data)
>    File "/usr/lib/python2.7/site-packages/koji/ssl/SSLConnection.py",
> line 111, in sendall
>      self.close()
>    File "/usr/lib/python2.7/site-packages/koji/ssl/SSLConnection.py",
> line 82, in close
>      self.shutdown()
>    File "/usr/lib/python2.7/site-packages/koji/ssl/SSLConnection.py",
> line 53, in shutdown
>      self.__dict__["conn"].shutdown()
> Error: []
>
> 2013-10-22 16:49:12,599 [INFO] koji: Try #1 for call 1328
> (listBuildroots) failed: []

Not sure what's happening there, but it seems like either the network or 
SSL libraries have changed significantly in python 2.7.  We'll have to 
look into it further.

>
> I suspect I'm really close now, but have done something silly that's not
> obvious to me.  Any pointers will be greatly appreciated.
>
> [1] https://fedoraproject.org/wiki/Koji/ServerBootstrap
>
> --
> John Florian