On 11/15/13 5:48 AM, Didier Fabert wrote:
Hi all,
I have strange error since last week on my koji. All regen-repo tasks are failed with error LockError: [Errno 37] No locks available
Koji works perfectly since January month. Directory /mnt/koji is a nfs mount but i can touch a new file on it. Database is ok too.
I have two builders, and error appear on both.
Any idea ?
== Client log == koji regen-repo centos-6-build Regenerating repo for tag centos-6-build Watching tasks (this may be safely interrupted)... 4888 newRepo (centos-6-build): free 4888 newRepo (centos-6-build): free -> open (builder-vbox2.b2pweb.com) 4889 createrepo (x86_64): free 4890 createrepo (i386): free 4889 createrepo (x86_64): free -> open (builder-vbox2.b2pweb.com) 4890 createrepo (i386): free -> open (builder-vbox2.b2pweb.com) 4889 createrepo (x86_64): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 2 open 0 done 1 failed 4888 newRepo (centos-6-build): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 1 open 0 done 2 failed 4890 createrepo (i386): open (builder-vbox2.b2pweb.com) -> FAILED: LockError: [Errno 37] No locks available 0 free 0 open 0 done 3 failed
4888 newRepo (centos-6-build) failed
== Builder log == /usr/sbin/kojid --verbose --fg 2013-11-15 14:46:47,576 [INFO] {12485} koji.build:73 Starting up 2013-11-15 14:47:03,363 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4891 2013-11-15 14:47:03,711 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:03,775 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,053 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:19,117 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,312 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4892 2013-11-15 14:47:19,662 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636} 2013-11-15 14:47:19,727 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4892, 'weight': 1.5} 2013-11-15 14:47:19,728 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:19,925 [INFO] {12485} koji.TaskManager:1078 Attempting to take task 4893 2013-11-15 14:47:20,277 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636, 4893: 12638} 2013-11-15 14:47:20,342 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4893, 'weight': 1.5} 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': None, 'id': 4892, 'weight': 1.5} 2013-11-15 14:47:20,343 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001} 2013-11-15 14:47:27,662 [WARNING] {12636} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2845, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2895, in create_local_repo status = log_output(self.session, cmd[0], cmd, logfile, self.getUploadDir(), logerror=True) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in log_output incremental_upload(session, remotename, outfd, uploadpath) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in incremental_upload fast_incremental_upload(session, fname, fd, path, retries, logger) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in fast_incremental_upload result = session.rawUpload(contents, offset, path, fname, overwrite=True) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:33,658 [WARNING] {12638} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2845, in handler self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo) File "/usr/sbin/kojid", line 2895, in create_local_repo status = log_output(self.session, cmd[0], cmd, logfile, self.getUploadDir(), logerror=True) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 153, in log_output incremental_upload(session, remotename, outfd, uploadpath) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 48, in incremental_upload fast_incremental_upload(session, fname, fd, path, retries, logger) File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 87, in fast_incremental_upload result = session.rawUpload(contents, offset, path, fname, overwrite=True) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:35,625 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568, 4892: 12636, 4893: 12638} 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:681 open task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True} 2013-11-15 14:47:35,690 [INFO] {12485} koji.TaskManager:693 Waking up task: {'waiting': True, 'id': 4891, 'weight': 0.10000000000000001, 'alert': True} 2013-11-15 14:47:35,691 [INFO] {12485} koji.TaskManager:846 Task 4892 (pid 12636) exited with status 0 2013-11-15 14:47:35,715 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7036 (task 4892) 2013-11-15 14:47:35,780 [INFO] {12485} koji.TaskManager:846 Task 4893 (pid 12638) exited with status 0 2013-11-15 14:47:35,803 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7037 (task 4893) 2013-11-15 14:47:36,640 [WARNING] {12568} koji.TaskManager:1181 TRACEBACK: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/koji/daemon.py", line 1162, in runTask response = (handler.run(),) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 147, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck return func(*args, **kwargs) File "/usr/sbin/kojid", line 2811, in handler results = self.wait(subtasks.values(), all=True, failany=True) File "/usr/lib/python2.6/site-packages/koji/tasks.py", line 234, in wait return dict(self.session.host.taskWaitResults(self.id,subtasks)) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1552, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1914, in _callMethod raise err LockError: [Errno 37] No locks available
2013-11-15 14:47:51,152 [INFO] {12485} koji.TaskManager:679 pids: {4891: 12568} 2013-11-15 14:47:51,215 [INFO] {12485} koji.TaskManager:846 Task 4891 (pid 12568) exited with status 0 2013-11-15 14:47:51,239 [INFO] {12485} koji.TaskManager:991 Expiring subsession 7035 (task 4891)
koji.LockError is only raised in two places, both around fnctl.lockf():
http://docs.python.org/release/2.6.8/library/fcntl.html#fcntl.lockf
The fcntl manpage says:
ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).
Are you mounting /mnt/koji over nfs? Maybe the nfslock service died? Is something else holding a lot of locks?