Koji regen-repo error

[Didier FABERT]

Hi all,

I have strange error since last week on my koji. All regen-repo tasks are 
failed with error LockError: [Errno 37] No locks available

Koji works perfectly since January month. Directory /mnt/koji is a nfs mount 
but i can touch a new file on it. Database is ok too.

I have two builders, and error appear on both.

Any idea ?

== Client log ==
koji regen-repo centos-6-build
Regenerating repo for tag centos-6-build
Watching tasks (this may be safely interrupted)...
4888 newRepo (centos-6-build): free
4888 newRepo (centos-6-build): free -> open (builder-vbox2.b2pweb.com)
  4889 createrepo (x86_64): free
  4890 createrepo (i386): free
  4889 createrepo (x86_64): free -> open (builder-vbox2.b2pweb.com)
  4890 createrepo (i386): free -> open (builder-vbox2.b2pweb.com)
  4889 createrepo (x86_64): open (builder-vbox2.b2pweb.com) -> FAILED: 
LockError: [Errno 37] No locks available
  0 free  2 open  0 done  1 failed
4888 newRepo (centos-6-build): open (builder-vbox2.b2pweb.com) -> FAILED: 
LockError: [Errno 37] No locks available
  0 free  1 open  0 done  2 failed
  4890 createrepo (i386): open (builder-vbox2.b2pweb.com) -> FAILED: 
LockError: [Errno 37] No locks available
  0 free  0 open  0 done  3 failed

4888 newRepo (centos-6-build) failed

== Builder log ==
/usr/sbin/kojid --verbose --fg
2013-11-15 14:46:47,576 [INFO] {12485} koji.build:73 Starting up
2013-11-15 14:47:03,363 [INFO] {12485} koji.TaskManager:1078 Attempting to 
take task 4891
2013-11-15 14:47:03,711 [INFO] {12485} koji.TaskManager:679 pids: {4891: 
12568}
2013-11-15 14:47:03,775 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': None, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:19,053 [INFO] {12485} koji.TaskManager:679 pids: {4891: 
12568}
2013-11-15 14:47:19,117 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:19,312 [INFO] {12485} koji.TaskManager:1078 Attempting to 
take task 4892
2013-11-15 14:47:19,662 [INFO] {12485} koji.TaskManager:679 pids: {4891: 
12568, 4892: 12636}
2013-11-15 14:47:19,727 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': None, 'id': 4892, 'weight': 1.5}
2013-11-15 14:47:19,728 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:19,925 [INFO] {12485} koji.TaskManager:1078 Attempting to 
take task 4893
2013-11-15 14:47:20,277 [INFO] {12485} koji.TaskManager:679 pids: {4891: 
12568, 4892: 12636, 4893: 12638}
2013-11-15 14:47:20,342 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': None, 'id': 4893, 'weight': 1.5}
2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': None, 'id': 4892, 'weight': 1.5}
2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
2013-11-15 14:47:27,662 [WARNING] {12636} koji.TaskManager:1181 TRACEBACK: 
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in 
runTask
    response = (handler.run(),)
  File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in 
call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 2845, in handler
    self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
  File "/usr/sbin/kojid", line 2895, in create_local_repo
    status = log_output(self.session, cmd[0], cmd, logfile, 
self.getUploadDir(), logerror=True)
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in 
log_output
    incremental_upload(session, remotename, outfd, uploadpath)
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in 
incremental_upload
    fast_incremental_upload(session, fname, fd, path, retries, logger)
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in 
fast_incremental_upload
    result = session.rawUpload(contents, offset, path, fname, overwrite=True)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in 
__call__
    return self.__func(self.__name,args,opts)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in 
_callMethod
    raise err
LockError: [Errno 37] No locks available

2013-11-15 14:47:33,658 [WARNING] {12638} koji.TaskManager:1181 TRACEBACK: 
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in 
runTask
    response = (handler.run(),)
  File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in 
call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 2845, in handler
    self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
  File "/usr/sbin/kojid", line 2895, in create_local_repo
    status = log_output(self.session, cmd[0], cmd, logfile, 
self.getUploadDir(), logerror=True)
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in 
log_output
    incremental_upload(session, remotename, outfd, uploadpath)
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in 
incremental_upload
    fast_incremental_upload(session, fname, fd, path, retries, logger)
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in 
fast_incremental_upload
    result = session.rawUpload(contents, offset, path, fname, overwrite=True)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in 
__call__
    return self.__func(self.__name,args,opts)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in 
_callMethod
    raise err
LockError: [Errno 37] No locks available

2013-11-15 14:47:35,625 [INFO] {12485} koji.TaskManager:679 pids: {4891: 
12568, 4892: 12636, 4893: 12638}
2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:681 open task: 
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True}
2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:693 Waking up task: 
{'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True}
2013-11-15 14:47:35,691 [INFO] {12485} koji.TaskManager:846 Task 4892 (pid 
12636) exited with status 0
2013-11-15 14:47:35,715 [INFO] {12485} koji.TaskManager:991 Expiring 
subsession 7036 (task 4892)
2013-11-15 14:47:35,780 [INFO] {12485} koji.TaskManager:846 Task 4893 (pid 
12638) exited with status 0
2013-11-15 14:47:35,803 [INFO] {12485} koji.TaskManager:991 Expiring 
subsession 7037 (task 4893)
2013-11-15 14:47:36,640 [WARNING] {12568} koji.TaskManager:1181 TRACEBACK: 
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in 
runTask
    response = (handler.run(),)
  File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in 
call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 2811, in handler
    results = self.wait(subtasks.values(), all=True, failany=True)
  File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 234, in wait
    return dict(self.session.host.taskWaitResults(self.id,subtasks))
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in 
__call__
    return self.__func(self.__name,args,opts)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in 
_callMethod
    raise err
LockError: [Errno 37] No locks available

2013-11-15 14:47:51,152 [INFO] {12485} koji.TaskManager:679 pids: {4891: 
12568}
2013-11-15 14:47:51,215 [INFO] {12485} koji.TaskManager:846 Task 4891 (pid 
12568) exited with status 0
2013-11-15 14:47:51,239 [INFO] {12485} koji.TaskManager:991 Expiring 
subsession 7035 (task 4891)

[Mike Bonnet]

On 11/15/13 5:48 AM, Didier Fabert wrote:
> Hi all,
>
> I have strange error since last week on my koji. All regen-repo tasks are
> failed with error LockError: [Errno 37] No locks available
>
> Koji works perfectly since January month. Directory /mnt/koji is a nfs mount
> but i can touch a new file on it. Database is ok too.
>
> I have two builders, and error appear on both.
>
> Any idea ?
>
> == Client log ==
> koji regen-repo centos-6-build
> Regenerating repo for tag centos-6-build
> Watching tasks (this may be safely interrupted)...
> 4888 newRepo (centos-6-build): free
> 4888 newRepo (centos-6-build): free -> open (builder-vbox2.b2pweb.com)
>    4889 createrepo (x86_64): free
>    4890 createrepo (i386): free
>    4889 createrepo (x86_64): free -> open (builder-vbox2.b2pweb.com)
>    4890 createrepo (i386): free -> open (builder-vbox2.b2pweb.com)
>    4889 createrepo (x86_64): open (builder-vbox2.b2pweb.com) -> FAILED:
> LockError: [Errno 37] No locks available
>    0 free  2 open  0 done  1 failed
> 4888 newRepo (centos-6-build): open (builder-vbox2.b2pweb.com) -> FAILED:
> LockError: [Errno 37] No locks available
>    0 free  1 open  0 done  2 failed
>    4890 createrepo (i386): open (builder-vbox2.b2pweb.com) -> FAILED:
> LockError: [Errno 37] No locks available
>    0 free  0 open  0 done  3 failed
>
> 4888 newRepo (centos-6-build) failed
>
> == Builder log ==
> /usr/sbin/kojid --verbose --fg
> 2013-11-15 14:46:47,576 [INFO] {12485} koji.build:73 Starting up
> 2013-11-15 14:47:03,363 [INFO] {12485} koji.TaskManager:1078 Attempting to
> take task 4891
> 2013-11-15 14:47:03,711 [INFO] {12485} koji.TaskManager:679 pids: {4891:
> 12568}
> 2013-11-15 14:47:03,775 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': None, 'id': 4891, 'weight': 0.10000000000000001}
> 2013-11-15 14:47:19,053 [INFO] {12485} koji.TaskManager:679 pids: {4891:
> 12568}
> 2013-11-15 14:47:19,117 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
> 2013-11-15 14:47:19,312 [INFO] {12485} koji.TaskManager:1078 Attempting to
> take task 4892
> 2013-11-15 14:47:19,662 [INFO] {12485} koji.TaskManager:679 pids: {4891:
> 12568, 4892: 12636}
> 2013-11-15 14:47:19,727 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': None, 'id': 4892, 'weight': 1.5}
> 2013-11-15 14:47:19,728 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
> 2013-11-15 14:47:19,925 [INFO] {12485} koji.TaskManager:1078 Attempting to
> take task 4893
> 2013-11-15 14:47:20,277 [INFO] {12485} koji.TaskManager:679 pids: {4891:
> 12568, 4892: 12636, 4893: 12638}
> 2013-11-15 14:47:20,342 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': None, 'id': 4893, 'weight': 1.5}
> 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': None, 'id': 4892, 'weight': 1.5}
> 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001}
> 2013-11-15 14:47:27,662 [WARNING] {12636} koji.TaskManager:1181 TRACEBACK:
> Traceback (most recent call last):
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in
> runTask
>      response = (handler.run(),)
>    File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
>      return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
>    File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
> call_with_argcheck
>      return func(*args, **kwargs)
>    File "/usr/sbin/kojid", line 2845, in handler
>      self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
>    File "/usr/sbin/kojid", line 2895, in create_local_repo
>      status = log_output(self.session, cmd[0], cmd, logfile,
> self.getUploadDir(), logerror=True)
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in
> log_output
>      incremental_upload(session, remotename, outfd, uploadpath)
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in
> incremental_upload
>      fast_incremental_upload(session, fname, fd, path, retries, logger)
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in
> fast_incremental_upload
>      result = session.rawUpload(contents, offset, path, fname, overwrite=True)
>    File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in
> __call__
>      return self.__func(self.__name,args,opts)
>    File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in
> _callMethod
>      raise err
> LockError: [Errno 37] No locks available
>
> 2013-11-15 14:47:33,658 [WARNING] {12638} koji.TaskManager:1181 TRACEBACK:
> Traceback (most recent call last):
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in
> runTask
>      response = (handler.run(),)
>    File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
>      return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
>    File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
> call_with_argcheck
>      return func(*args, **kwargs)
>    File "/usr/sbin/kojid", line 2845, in handler
>      self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
>    File "/usr/sbin/kojid", line 2895, in create_local_repo
>      status = log_output(self.session, cmd[0], cmd, logfile,
> self.getUploadDir(), logerror=True)
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in
> log_output
>      incremental_upload(session, remotename, outfd, uploadpath)
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in
> incremental_upload
>      fast_incremental_upload(session, fname, fd, path, retries, logger)
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in
> fast_incremental_upload
>      result = session.rawUpload(contents, offset, path, fname, overwrite=True)
>    File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in
> __call__
>      return self.__func(self.__name,args,opts)
>    File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in
> _callMethod
>      raise err
> LockError: [Errno 37] No locks available
>
> 2013-11-15 14:47:35,625 [INFO] {12485} koji.TaskManager:679 pids: {4891:
> 12568, 4892: 12636, 4893: 12638}
> 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:681 open task:
> {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True}
> 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:693 Waking up task:
> {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True}
> 2013-11-15 14:47:35,691 [INFO] {12485} koji.TaskManager:846 Task 4892 (pid
> 12636) exited with status 0
> 2013-11-15 14:47:35,715 [INFO] {12485} koji.TaskManager:991 Expiring
> subsession 7036 (task 4892)
> 2013-11-15 14:47:35,780 [INFO] {12485} koji.TaskManager:846 Task 4893 (pid
> 12638) exited with status 0
> 2013-11-15 14:47:35,803 [INFO] {12485} koji.TaskManager:991 Expiring
> subsession 7037 (task 4893)
> 2013-11-15 14:47:36,640 [WARNING] {12568} koji.TaskManager:1181 TRACEBACK:
> Traceback (most recent call last):
>    File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in
> runTask
>      response = (handler.run(),)
>    File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run
>      return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
>    File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in
> call_with_argcheck
>      return func(*args, **kwargs)
>    File "/usr/sbin/kojid", line 2811, in handler
>      results = self.wait(subtasks.values(), all=True, failany=True)
>    File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 234, in wait
>      return dict(self.session.host.taskWaitResults(self.id,subtasks))
>    File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in
> __call__
>      return self.__func(self.__name,args,opts)
>    File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in
> _callMethod
>      raise err
> LockError: [Errno 37] No locks available
>
> 2013-11-15 14:47:51,152 [INFO] {12485} koji.TaskManager:679 pids: {4891:
> 12568}
> 2013-11-15 14:47:51,215 [INFO] {12485} koji.TaskManager:846 Task 4891 (pid
> 12568) exited with status 0
> 2013-11-15 14:47:51,239 [INFO] {12485} koji.TaskManager:991 Expiring
> subsession 7035 (task 4891)

koji.LockError is only raised in two places, both around fnctl.lockf():

http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf

The fcntl manpage says:

ENOLCK Too many segment locks open, lock table is full, or a remote 
locking protocol failed (e.g., locking over NFS).

Are you mounting /mnt/koji over nfs?  Maybe the nfslock service died? 
Is something else holding a lot of locks?

[Didier FABERT]

On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
> On 11/15/13 5:48 AM, Didier Fabert wrote:
> > Hi all,
> >
> > I have strange error since last week on my koji. All regen-repo tasks are
> > failed with error LockError: [Errno 37] No locks available
> >
> > Koji works perfectly since January month. Directory /mnt/koji is a nfs
> > mount but i can touch a new file on it. Database is ok too.
> >
> > I have two builders, and error appear on both.
> >
> > Any idea ?
>
> koji.LockError is only raised in two places, both around fnctl.lockf():
>
> http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
>
> The fcntl manpage says:
>
> ENOLCK Too many segment locks open, lock table is full, or a remote
> locking protocol failed (e.g., locking over NFS).
>
> Are you mounting /mnt/koji over nfs?  Maybe the nfslock service died?
> Is something else holding a lot of locks?

Yes, you're right, nfslock was dead on NFS server (el5)
After restarting nfslock service on my NFS server, all is OK.

Thanks a lot Mike to pointing me to the right direction.

[Peter Bojtos]

The problem appeared today again, although I set up in kojira to keep repos for 1 week. The root log's end is this:

DEBUG util.py:314:  Executing command: /usr/bin/repoquery -c /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
DEBUG util.py:264:  Traceback (most recent call last):
DEBUG util.py:264:    File "/usr/bin/repoquery", line 1510, in <module>
DEBUG util.py:264:      main(sys.argv)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 1504, in main
DEBUG util.py:264:      repoq.runQuery(regexs)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 982, in runQuery
DEBUG util.py:264:      pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 903, in matchPkgs
DEBUG util.py:264:      pkgs = self.returnPkgList(patterns=items)
DEBUG util.py:264:    File "/usr/bin/repoquery", line 856, in returnPkgList
DEBUG util.py:264:      pkgs = self.pkgSack.returnNewestByNameArch(**kwargs)
DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda>
DEBUG util.py:264:      pkgSack = property(fget=lambda self: self._getSacks(),
DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks
DEBUG util.py:264:      self.repos.populateSack(which=repos)
DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py", line 309, in populateSack
DEBUG util.py:264:      self.doSetup()
DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py", line 134, in doSetup
DEBUG util.py:264:      self.retrieveAllMD()
DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py", line 92, in retrieveAllMD
DEBUG util.py:264:      downloading = repo._commonRetrieveDataMD_list(mdtypes)
DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in _commonRetrieveDataMD_list
DEBUG util.py:264:      os.rename(local, local + '.old.tmp')
DEBUG util.py:264:  OSError: [Errno 2] No such file or directory
DEBUG util.py:354:  Child return code was: 1
DEBUG util.py:314:  Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}

The server is:
koji-hub-1.8.0-1.el6.noarch

The builder version is:
koji-builder-1.8.0-1.fc17.noarch

Any suggestion, where to start to debug?

Thank you,
Peter

----- Eredeti üzenet -----
> Feladó: "Didier Fabert" 
> Címzett: "Discussion of Fedora build system" 
> Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19
> Tárgy: Re: Koji regen-repo error
>
> On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
> > On 11/15/13 5:48 AM, Didier Fabert wrote:
> > > Hi all,
> > >
> > > I have strange error since last week on my koji. All regen-repo tasks are
> > > failed with error LockError: [Errno 37] No locks available
> > >
> > > Koji works perfectly since January month. Directory /mnt/koji is a nfs
> > > mount but i can touch a new file on it. Database is ok too.
> > >
> > > I have two builders, and error appear on both.
> > >
> > > Any idea ?
> >
> > koji.LockError is only raised in two places, both around fnctl.lockf():
> >
> > http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
> >
> > The fcntl manpage says:
> >
> > ENOLCK Too many segment locks open, lock table is full, or a remote
> > locking protocol failed (e.g., locking over NFS).
> >
> > Are you mounting /mnt/koji over nfs?  Maybe the nfslock service died?
> > Is something else holding a lot of locks?
>
> Yes, you're right, nfslock was dead on NFS server (el5)
> After restarting nfslock service on my NFS server, all is OK.
>
> Thanks a lot Mike to pointing me to the right direction.
>
> --
> Didier
> --
> buildsys mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

[Mike Bonnet]

On 12/5/13 9:36 AM, Bojtos Péter wrote:
> The problem appeared today again, although I set up in kojira to keep repos for 1 week. The root log's end is this:
>
> DEBUG util.py:314:  Executing command: /usr/bin/repoquery -c /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
> DEBUG util.py:264:  Traceback (most recent call last):
> DEBUG util.py:264:    File "/usr/bin/repoquery", line 1510, in <module>
> DEBUG util.py:264:      main(sys.argv)
> DEBUG util.py:264:    File "/usr/bin/repoquery", line 1504, in main
> DEBUG util.py:264:      repoq.runQuery(regexs)
> DEBUG util.py:264:    File "/usr/bin/repoquery", line 982, in runQuery
> DEBUG util.py:264:      pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs)
> DEBUG util.py:264:    File "/usr/bin/repoquery", line 903, in matchPkgs
> DEBUG util.py:264:      pkgs = self.returnPkgList(patterns=items)
> DEBUG util.py:264:    File "/usr/bin/repoquery", line 856, in returnPkgList
> DEBUG util.py:264:      pkgs = self.pkgSack.returnNewestByNameArch(**kwargs)
> DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda>
> DEBUG util.py:264:      pkgSack = property(fget=lambda self: self._getSacks(),
> DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks
> DEBUG util.py:264:      self.repos.populateSack(which=repos)
> DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py", line 309, in populateSack
> DEBUG util.py:264:      self.doSetup()
> DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py", line 134, in doSetup
> DEBUG util.py:264:      self.retrieveAllMD()
> DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py", line 92, in retrieveAllMD
> DEBUG util.py:264:      downloading = repo._commonRetrieveDataMD_list(mdtypes)
> DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in _commonRetrieveDataMD_list
> DEBUG util.py:264:      os.rename(local, local + '.old.tmp')
> DEBUG util.py:264:  OSError: [Errno 2] No such file or directory
> DEBUG util.py:354:  Child return code was: 1
> DEBUG util.py:314:  Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
>
> The server is:
> koji-hub-1.8.0-1.el6.noarch
>
> The builder version is:
> koji-builder-1.8.0-1.fc17.noarch

This looks like a problem with mock or repoquery.  What version of those 
do you have installed on your builder?
> Any suggestion, where to start to debug?
>
> Thank you,
> Peter
>
> ----- Eredeti üzenet -----
> > Feladó: "Didier Fabert" 
> > Címzett: "Discussion of Fedora build system" 
> > Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19
> > Tárgy: Re: Koji regen-repo error
> >
> > On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
> > > On 11/15/13 5:48 AM, Didier Fabert wrote:
> > > > Hi all,
> > > >
> > > > I have strange error since last week on my koji. All regen-repo tasks are
> > > > failed with error LockError: [Errno 37] No locks available
> > > >
> > > > Koji works perfectly since January month. Directory /mnt/koji is a nfs
> > > > mount but i can touch a new file on it. Database is ok too.
> > > >
> > > > I have two builders, and error appear on both.
> > > >
> > > > Any idea ?
> > >
> > > koji.LockError is only raised in two places, both around fnctl.lockf():
> > >
> > > http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
> > >
> > > The fcntl manpage says:
> > >
> > > ENOLCK Too many segment locks open, lock table is full, or a remote
> > > locking protocol failed (e.g., locking over NFS).
> > >
> > > Are you mounting /mnt/koji over nfs?  Maybe the nfslock service died?
> > > Is something else holding a lot of locks?
> >
> > Yes, you're right, nfslock was dead on NFS server (el5)
> > After restarting nfslock service on my NFS server, all is OK.
> >
> > Thanks a lot Mike to pointing me to the right direction.
> >
> > --
> > Didier
> > --
> > buildsys mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
> --
> buildsys mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

[Peter Bojtos]

Mock version is:
mock-1.1.32-1.fc17.noarch

Repoquery is in the following package:
yum-utils-1.1.31-4.fc17.noarch

Peter

----- Eredeti üzenet -----
> Feladó: "Mike Bonnet" 
> Címzett: [e-mail]
> Elküldött üzenetek: Csütörtök, 2013. December 5. 18:43:48
> Tárgy: Re: Koji regen-repo error
>
> On 12/5/13 9:36 AM, Bojtos Péter wrote:
> > The problem appeared today again, although I set up in kojira to keep repos
> > for 1 week. The root log's end is this:
> >
> > DEBUG util.py:314:  Executing command: /usr/bin/repoquery -c
> > /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf
> > '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' >
> > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env
> > {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME':
> > 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
> > DEBUG util.py:264:  Traceback (most recent call last):
> > DEBUG util.py:264:    File "/usr/bin/repoquery", line 1510, in <module>
> > DEBUG util.py:264:      main(sys.argv)
> > DEBUG util.py:264:    File "/usr/bin/repoquery", line 1504, in main
> > DEBUG util.py:264:      repoq.runQuery(regexs)
> > DEBUG util.py:264:    File "/usr/bin/repoquery", line 982, in runQuery
> > DEBUG util.py:264:      pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs)
> > DEBUG util.py:264:    File "/usr/bin/repoquery", line 903, in matchPkgs
> > DEBUG util.py:264:      pkgs = self.returnPkgList(patterns=items)
> > DEBUG util.py:264:    File "/usr/bin/repoquery", line 856, in returnPkgList
> > DEBUG util.py:264:      pkgs =
> > self.pkgSack.returnNewestByNameArch(**kwargs)
> > DEBUG util.py:264:    File
> > "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda>
> > DEBUG util.py:264:      pkgSack = property(fget=lambda self:
> > self._getSacks(),
> > DEBUG util.py:264:    File
> > "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in _getSacks
> > DEBUG util.py:264:      self.repos.populateSack(which=repos)
> > DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py",
> > line 309, in populateSack
> > DEBUG util.py:264:      self.doSetup()
> > DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py",
> > line 134, in doSetup
> > DEBUG util.py:264:      self.retrieveAllMD()
> > DEBUG util.py:264:    File "/usr/lib/python2.7/site-packages/yum/repos.py",
> > line 92, in retrieveAllMD
> > DEBUG util.py:264:      downloading =
> > repo._commonRetrieveDataMD_list(mdtypes)
> > DEBUG util.py:264:    File
> > "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in
> > _commonRetrieveDataMD_list
> > DEBUG util.py:264:      os.rename(local, local + '.old.tmp')
> > DEBUG util.py:264:  OSError: [Errno 2] No such file or directory
> > DEBUG util.py:354:  Child return code was: 1
> > DEBUG util.py:314:  Executing command: ['/bin/umount', '-n', '-l',
> > '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env
> > {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME':
> > 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
> >
> > The server is:
> > koji-hub-1.8.0-1.el6.noarch
> >
> > The builder version is:
> > koji-builder-1.8.0-1.fc17.noarch
>
> This looks like a problem with mock or repoquery.  What version of those
> do you have installed on your builder?
> > Any suggestion, where to start to debug?
> >
> > Thank you,
> > Peter
> >
> > ----- Eredeti üzenet -----
> > > Feladó: "Didier Fabert" 
> > > Címzett: "Discussion of Fedora build system"
> > >
> > > Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19
> > > Tárgy: Re: Koji regen-repo error
> > >
> > > On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
> > > > On 11/15/13 5:48 AM, Didier Fabert wrote:
> > > > > Hi all,
> > > > >
> > > > > I have strange error since last week on my koji. All regen-repo tasks
> > > > > are
> > > > > failed with error LockError: [Errno 37] No locks available
> > > > >
> > > > > Koji works perfectly since January month. Directory /mnt/koji is a nfs
> > > > > mount but i can touch a new file on it. Database is ok too.
> > > > >
> > > > > I have two builders, and error appear on both.
> > > > >
> > > > > Any idea ?
> > > >
> > > > koji.LockError is only raised in two places, both around fnctl.lockf():
> > > >
> > > > http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
> > > >
> > > > The fcntl manpage says:
> > > >
> > > > ENOLCK Too many segment locks open, lock table is full, or a remote
> > > > locking protocol failed (e.g., locking over NFS).
> > > >
> > > > Are you mounting /mnt/koji over nfs?  Maybe the nfslock service died?
> > > > Is something else holding a lot of locks?
> > >
> > > Yes, you're right, nfslock was dead on NFS server (el5)
> > > After restarting nfslock service on my NFS server, all is OK.
> > >
> > > Thanks a lot Mike to pointing me to the right direction.
> > >
> > > --
> > > Didier
> > > --
> > > buildsys mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/buildsys
> >
> > --
> > buildsys mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
> --
> buildsys mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

[Peter Bojtos]

Dear alll,

I applied Remi's patch from the bugzilla ticket: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=969776
It solved the problem. It should be in a newer mock package in fedora/EPEL.

Thank you for your help,
Peter

----- Eredeti üzenet -----
> Feladó: "Bojtos Péter" 
> Címzett: "Discussion of Fedora build system" 
> Elküldött üzenetek: Csütörtök, 2013. December 5. 19:16:23
> Tárgy: Re: Koji regen-repo error
>
> Mock version is:
> mock-1.1.32-1.fc17.noarch
>
> Repoquery is in the following package:
> yum-utils-1.1.31-4.fc17.noarch
>
> Peter
>
> ----- Eredeti üzenet -----
> > Feladó: "Mike Bonnet" 
> > Címzett: [e-mail]
> > Elküldött üzenetek: Csütörtök, 2013. December 5. 18:43:48
> > Tárgy: Re: Koji regen-repo error
> >
> > On 12/5/13 9:36 AM, Bojtos Péter wrote:
> > > The problem appeared today again, although I set up in kojira to keep
> > > repos
> > > for 1 week. The root log's end is this:
> > >
> > > DEBUG util.py:314:  Executing command: /usr/bin/repoquery -c
> > > /var/lib/mock/dist-slxrv3-build-7179-1529/root//etc/yum.conf -a --qf
> > > '%{nevra} %{buildtime} %{size} %{pkgid} %{repoid}' >
> > > /var/lib/mock/dist-slxrv3-build-7179-1529/result/available_pkgs with env
> > > {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash',
> > > 'HOSTNAME':
> > > 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
> > > DEBUG util.py:264:  Traceback (most recent call last):
> > > DEBUG util.py:264:    File "/usr/bin/repoquery", line 1510, in <module>
> > > DEBUG util.py:264:      main(sys.argv)
> > > DEBUG util.py:264:    File "/usr/bin/repoquery", line 1504, in main
> > > DEBUG util.py:264:      repoq.runQuery(regexs)
> > > DEBUG util.py:264:    File "/usr/bin/repoquery", line 982, in runQuery
> > > DEBUG util.py:264:      pkgs = self.matchPkgs(items,
> > > plain_pkgs=plain_pkgs)
> > > DEBUG util.py:264:    File "/usr/bin/repoquery", line 903, in matchPkgs
> > > DEBUG util.py:264:      pkgs = self.returnPkgList(patterns=items)
> > > DEBUG util.py:264:    File "/usr/bin/repoquery", line 856, in
> > > returnPkgList
> > > DEBUG util.py:264:      pkgs =
> > > self.pkgSack.returnNewestByNameArch(**kwargs)
> > > DEBUG util.py:264:    File
> > > "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in
> > >
> > > <lambda>
> > > DEBUG util.py:264:      pkgSack = property(fget=lambda self:
> > > self._getSacks(),
> > > DEBUG util.py:264:    File
> > > "/usr/lib/python2.7/site-packages/yum/__init__.py", line 779, in
> > > _getSacks
> > > DEBUG util.py:264:      self.repos.populateSack(which=repos)
> > > DEBUG util.py:264:    File
> > > "/usr/lib/python2.7/site-packages/yum/repos.py",
> > > line 309, in populateSack
> > > DEBUG util.py:264:      self.doSetup()
> > > DEBUG util.py:264:    File
> > > "/usr/lib/python2.7/site-packages/yum/repos.py",
> > > line 134, in doSetup
> > > DEBUG util.py:264:      self.retrieveAllMD()
> > > DEBUG util.py:264:    File
> > > "/usr/lib/python2.7/site-packages/yum/repos.py",
> > > line 92, in retrieveAllMD
> > > DEBUG util.py:264:      downloading =
> > > repo._commonRetrieveDataMD_list(mdtypes)
> > > DEBUG util.py:264:    File
> > > "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1540, in
> > > _commonRetrieveDataMD_list
> > > DEBUG util.py:264:      os.rename(local, local + '.old.tmp')
> > > DEBUG util.py:264:  OSError: [Errno 2] No such file or directory
> > > DEBUG util.py:354:  Child return code was: 1
> > > DEBUG util.py:314:  Executing command: ['/bin/umount', '-n', '-l',
> > > '/var/lib/mock/dist-slxrv3-build-7179-1529/root/dev/pts'] with env
> > > {'LANG': 'en_US.UTF-8', 'TERM': 'vt100', 'SHELL': '/bin/bash',
> > > 'HOSTNAME':
> > > 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
> > >
> > > The server is:
> > > koji-hub-1.8.0-1.el6.noarch
> > >
> > > The builder version is:
> > > koji-builder-1.8.0-1.fc17.noarch
> >
> > This looks like a problem with mock or repoquery.  What version of those
> > do you have installed on your builder?
> > > Any suggestion, where to start to debug?
> > >
> > > Thank you,
> > > Peter
> > >
> > > ----- Eredeti üzenet -----
> > > > Feladó: "Didier Fabert" 
> > > > Címzett: "Discussion of Fedora build system"
> > > >
> > > > Elküldött üzenetek: Kedd, 2013. November 19. 10:28:19
> > > > Tárgy: Re: Koji regen-repo error
> > > >
> > > > On Friday 15 November 2013 09:04:11 Mike Bonnet wrote:
> > > > > On 11/15/13 5:48 AM, Didier Fabert wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I have strange error since last week on my koji. All regen-repo tasks
> > > > > > are
> > > > > > failed with error LockError: [Errno 37] No locks available
> > > > > >
> > > > > > Koji works perfectly since January month. Directory /mnt/koji is a nfs
> > > > > > mount but i can touch a new file on it. Database is ok too.
> > > > > >
> > > > > > I have two builders, and error appear on both.
> > > > > >
> > > > > > Any idea ?
> > > > >
> > > > > koji.LockError is only raised in two places, both around fnctl.lockf():
> > > > >
> > > > > http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
> > > > >
> > > > > The fcntl manpage says:
> > > > >
> > > > > ENOLCK Too many segment locks open, lock table is full, or a remote
> > > > > locking protocol failed (e.g., locking over NFS).
> > > > >
> > > > > Are you mounting /mnt/koji over nfs?  Maybe the nfslock service died?
> > > > > Is something else holding a lot of locks?
> > > >
> > > > Yes, you're right, nfslock was dead on NFS server (el5)
> > > > After restarting nfslock service on my NFS server, all is OK.
> > > >
> > > > Thanks a lot Mike to pointing me to the right direction.
> > > >
> > > > --
> > > > Didier
> > > > --
> > > > buildsys mailing list
> > > > [e-mail]
> > > > https://admin.fedoraproject.org/mailman/listinfo/buildsys
> > >
> > > --
> > > buildsys mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/buildsys
> >
> > --
> > buildsys mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
> --
> buildsys mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/buildsys