New Koji Setup - regen-repo task fails

[Kirk Harr]

Hello,

When setting up a new Koji setup, I submit a regen-repo task which is picked up by the
single builder host that I have setup, which fails with a message in the kojid.log file:

<snip>
2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call
last):
  File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in runTask
    response = (handler.run(),)
  File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in
call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 3536, in handler
    raise koji.GenericError, "Repo directory missing: %s" % path
GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085
</snip>

This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside
/mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the
builder host share with each other. Is there any cause for this error to occur when the
directory referenced as missing does actually exist?

Thanks,

Kirk Harr
Systems and Network Engineer
End Point Corporation

[Mike McLean]

On 02/12/2015 03:00 PM, Kirk Harr wrote:
>  When setting up a new Koji setup, I submit a regen-repo task which is
> picked up by the single builder host that I have setup, which fails with a message in the
> kojid.log file:
>  
>  <snip>
>  2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent
> call last):
>    File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in
> runTask
>      response = (handler.run(),)
>    File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run
>      return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
>    File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in
> call_with_argcheck
>      return func(*args, **kwargs)
>    File "/usr/sbin/kojid", line 3536, in handler
>      raise koji.GenericError, "Repo directory missing: %s" % path
>  GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085
>  </snip>
>  
>  This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside
> /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the
> builder host share with each other. Is there any cause for this error to occur when the
> directory referenced as missing does actually exist? 

The test that appears to be failing is:
>  if not os.path.isdir(path):
>      raise koji.GenericError, "Repo directory missing: %s" % path 
This test happens just after the hub call to host.repoInit (which
creates the stub repo dir with basic data).

A few possibilities:
1) some sort of race issue where the new directory is still unknown to
the builder
2) some sort of permission error where the kojid process cannot see the
directory (does gluster do anything like "root squash"?)
3) the mount isn't set up on the builder after all
4) some sort of path mismatch (e.g. mounted in the wrong place)

[Kirk Harr]

Hi Mike,

Thanks for your response, for the possibilities:

1) This is possible, if so is there a way to retry the regen-repo task
using the same repo ID so that it can try with the directory in place?
2) GlusterFS does have root-squash functionality, but it is disabled by
default and is off in this case.
3) I confirmed the mount is in place and I can edit files on the remote
disk from the builder.
4) Confirmed that the builder has the volume mounted at /mnt/koji

Any ideas are appreciated as I have been working with this setup for a
bit now and I cannot get any further into the setup or building RPMs
until I can figure out why it cannot build its repository information.

Thanks,

Kirk Harr
Systems and Network Engineer
End Point Corporation


On 2/13/2015 10:15 AM, Mike McLean wrote:
>  On 02/12/2015 03:00 PM, Kirk Harr wrote:
> > When setting up a new Koji setup, I submit a regen-repo task which is picked up by
> the single builder host that I have setup, which fails with a message in the kojid.log
> file:
> >
> > <snip>
> > 2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent
> call last):
> >   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161, in
> runTask
> >     response = (handler.run(),)
> >   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in run
> >     return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
> >   File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in
> call_with_argcheck
> >     return func(*args, **kwargs)
> >   File "/usr/sbin/kojid", line 3536, in handler
> >     raise koji.GenericError, "Repo directory missing: %s" % path
> > GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085
> > </snip>
> >
> > This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host inside
> /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and the
> builder host share with each other. Is there any cause for this error to occur when the
> directory referenced as missing does actually exist?
>  
>  
>  The test that appears to be failing is:
> > if not os.path.isdir(path):
> >     raise koji.GenericError, "Repo directory missing: %s" % path
>  
>  This test happens just after the hub call to host.repoInit (which
>  creates the stub repo dir with basic data).
>  
>  A few possibilities:
>  1) some sort of race issue where the new directory is still unknown to
>  the builder
>  2) some sort of permission error where the kojid process cannot see the
>  directory (does gluster do anything like "root squash"?)
>  3) the mount isn't set up on the builder after all
>  4) some sort of path mismatch (e.g. mounted in the wrong place)
>  
>  
>  --
>  buildsys mailing list
>  buildsys(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/buildsys
>

[Mike McLean]

On 02/13/2015 01:53 PM, Kirk Harr wrote:
>  Hi Mike,
>
>  Thanks for your response, for the possibilities:
>
>  1) This is possible, if so is there a way to retry the regen-repo task
>  using the same repo ID so that it can try with the directory in place? 
No, but you could stick a nice sleep (and probably some debugging 
output) just after the call to self.session.host.repoInit in 
NewRepoTask.handler() (in kojid).

Use self.logger.warn() to log anything you want to check on, probably 
before and after a time.sleep(60) or similar.

Things you might want to log:
self.logger.warn("Got repo: %r", path)
self.logger.warn("Stat says: %r", os.lstat(path))
...etc

>  2) GlusterFS does have root-squash functionality, but it is disabled
> by
>  default and is off in this case. 
>  3) I confirmed the mount is in place and I can edit files on the
> remote
>  disk from the builder. 
Aside: RW should not be required for normal use, unless you are using 
the runroot plugin. Unless that is the case, I would recommend not 
granting the builders write permission to /mnt/koji

>  4) Confirmed that the builder has the volume mounted at /mnt/koji
>
>  Any ideas are appreciated as I have been working with this setup for a
>  bit now and I cannot get any further into the setup or building RPMs
>  until I can figure out why it cannot build its repository information.
>
>  Thanks,
>
>  Kirk Harr
>  Systems and Network Engineer
>  End Point Corporation
>
>
>  On 2/13/2015 10:15 AM, Mike McLean wrote:
> > On 02/12/2015 03:00 PM, Kirk Harr wrote:
> >> When setting up a new Koji setup, I submit a regen-repo task which is picked up
> by the single builder host that I have setup, which fails with a message in the kojid.log
> file:
> >>
> >> <snip>
> >> 2015-02-12 14:48:35,496 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most
> recent call last):
> >>    File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1161,
> in runTask
> >>      response = (handler.run(),)
> >>    File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 157, in
> run
> >>      return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
> >>    File "/usr/lib/python2.7/site-packages/koji/util.py", line 139, in
> call_with_argcheck
> >>      return func(*args, **kwargs)
> >>    File "/usr/sbin/kojid", line 3536, in handler
> >>      raise koji.GenericError, "Repo directory missing: %s" % path
> >> GenericError: Repo directory missing: /mnt/koji/repos/dist-centos6-build/1085
> >> </snip>
> >>
> >> This directory /mnt/koji/repos/dist-centos6-build/1085 does exist on the host
> inside /mnt/koji which is a GlusterFS share which the Kojira/Koji-hub/Koji-web host and
> the builder host share with each other. Is there any cause for this error to occur when
> the directory referenced as missing does actually exist?
> >
> >
> > The test that appears to be failing is:
> >> if not os.path.isdir(path):
> >>      raise koji.GenericError, "Repo directory missing: %s" % path
> >
> > This test happens just after the hub call to host.repoInit (which
> > creates the stub repo dir with basic data).
> >
> > A few possibilities:
> > 1) some sort of race issue where the new directory is still unknown to
> > the builder
> > 2) some sort of permission error where the kojid process cannot see the
> > directory (does gluster do anything like "root squash"?)
> > 3) the mount isn't set up on the builder after all
> > 4) some sort of path mismatch (e.g. mounted in the wrong place)
> >
> >
> > --
> > buildsys mailing list
> > buildsys(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/buildsys
> >
>  --
>  buildsys mailing list
>  buildsys(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/buildsys
>

[Kirk Harr]

Hi Mike,

I added those logging lines to the kojid python and then when attempting a repository
refresh the output that is caught in the log is:
<snip>
2015-02-26 17:44:14,886 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent call
last):
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1161, in runTask
    response = (handler.run(),)
  File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 157, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 3408, in handler
    self.logger.warn("Stat says: %r", os.lstat(path))
OSError: [Errno 61] No data available: '/mnt/koji/repos/dist-centos6-build/1127’
</snip>
Its appears the lstat test on the path is returning this error, but I am not familiar with
OSError 61. I was able to run ls on the directory referenced, so it definitely exists but
I am starting to believe there is something else preventing the file read from that
location. Any idea what could cause an error of this type?

Thanks again for your help in tracing down the cause of this problem.

Kirk Harr
Systems and Network Engineer
End Point Corporation


>  On Feb 19, 2015, at 4:19 PM, Mike McLean <mikem(a)redhat.com&gt;
> wrote:
>  
>  self.logger.warn("Got repo: %r", path)
>  self.logger.warn("Stat says: %r", os.lstat(path))

[Mike McLean]

On 02/26/2015 05:49 PM, Kirk Harr wrote:
>  Hi Mike,
>
>  I added those logging lines to the kojid python and then when attempting a repository
> refresh the output that is caught in the log is:
>  <snip>
>  2015-02-26 17:44:14,886 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent
> call last):
>     File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1161, in
> runTask
>       response = (handler.run(),)
>     File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 157, in run
>       return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
>     File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
> call_with_argcheck
>       return func(*args, **kwargs)
>     File "/usr/sbin/kojid", line 3408, in handler
>       self.logger.warn("Stat says: %r", os.lstat(path))
>  OSError: [Errno 61] No data available: '/mnt/koji/repos/dist-centos6-build/1127’
>  </snip>
>  Its appears the lstat test on the path is returning this error, but I am not familiar
> with OSError 61. I was able to run ls on the directory referenced, so it definitely exists
> but I am starting to believe there is something else preventing the file read from that
> location. Any idea what could cause an error of this type? 
selinux maybe?

Did you trying inserting a long sleep after the repoInit call?
I think you said that the directory was there (presumably on the 
builder's mount) when you looked for it after the fact. So either
1) there is some race issue where the builder does not see the fs update 
in time
2) somehow the kojid process does not have access to read it (even if 
your root shell does)

Maybe try running 'watch ls -l /mnt/koji/repos/dist-centos6-build/' in a 
root shell on the builder while you replicate the problem (to see if you 
can observe a lag.

>  Thanks again for your help in tracing down the cause of this
> problem.
>
>  Kirk Harr
>  Systems and Network Engineer
>  End Point Corporation
>
>
> > On Feb 19, 2015, at 4:19 PM, Mike McLean <mikem(a)redhat.com&gt; wrote:
> >
> > self.logger.warn("Got repo: %r", path)
> > self.logger.warn("Stat says: %r", os.lstat(path))
>
>  --
>  buildsys mailing list
>  buildsys(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/buildsys
>

[Kirk Harr]

Hi Mike,

After troubleshooting the issue for a bit, we were able to get the Koji setup working, and
can now generate the build root repositories and build packages. The main issue was that
the builders needed to use /tmp/koji as the ‘workdir’ which then removed any of the
strange permissions errors that would occur when the builder would attempt to process the
task. I had originally had this set to the shared volume for the Kojira/Koji-hub server on
/mnt/koji, which was not working as expected.

Thanks for your assistance in tracing it down,

Kirk Harr
Systems and Network Engineer
End Point Corporation



>  On Mar 27, 2015, at 2:33 PM, Mike McLean <mikem(a)redhat.com&gt;
> wrote:
>  
>  On 02/26/2015 05:49 PM, Kirk Harr wrote:
> > Hi Mike,
> > 
> > I added those logging lines to the kojid python and then when attempting a repository
> refresh the output that is caught in the log is:
> > <snip>
> > 2015-02-26 17:44:14,886 [WARNING] koji.TaskManager: TRACEBACK: Traceback (most recent
> call last):
> >   File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1161, in
> runTask
> >     response = (handler.run(),)
> >   File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 157, in run
> >     return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
> >   File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
> call_with_argcheck
> >     return func(*args, **kwargs)
> >   File "/usr/sbin/kojid", line 3408, in handler
> >     self.logger.warn("Stat says: %r", os.lstat(path))
> > OSError: [Errno 61] No data available: '/mnt/koji/repos/dist-centos6-build/1127’
> > </snip>
> > Its appears the lstat test on the path is returning this error, but I am not familiar
> with OSError 61. I was able to run ls on the directory referenced, so it definitely exists
> but I am starting to believe there is something else preventing the file read from that
> location. Any idea what could cause an error of this type?
>  
>  selinux maybe?
>  
>  Did you trying inserting a long sleep after the repoInit call?
>  I think you said that the directory was there (presumably on the builder's mount)
> when you looked for it after the fact. So either
>  1) there is some race issue where the builder does not see the fs update in time
>  2) somehow the kojid process does not have access to read it (even if your root shell
> does)
>  
>  Maybe try running 'watch ls -l /mnt/koji/repos/dist-centos6-build/' in a root
> shell on the builder while you replicate the problem (to see if you can observe a lag.
>  
> > Thanks again for your help in tracing down the cause of this problem.
> > 
> > Kirk Harr
> > Systems and Network Engineer
> > End Point Corporation
> > 
> > 
> >> On Feb 19, 2015, at 4:19 PM, Mike McLean <mikem(a)redhat.com&gt; wrote:
> >> 
> >> self.logger.warn("Got repo: %r", path)
> >> self.logger.warn("Stat says: %r", os.lstat(path))
> > 
> > --
> > buildsys mailing list
> > buildsys(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/buildsys
> > 
>  
>  --
>  buildsys mailing list
>  buildsys(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/buildsys