Currently, when we sign and push built packages, we move them out of a "repodir" while the build server may be copying new packages to that directory.
We have not cleaned up repodir for a long time either, because it could not be done automatically without risk. The corresponding section in the push script contains a "todo" comment only. [Recently I've used a separate script, however, which cleans up the directory after a push, provided that the build system is idle and no package is about to be copied into the directory at the same time.]
I've looked at Repo.py _update_repo() where I see nothing which prevents concurrent access to this repodir. Is there anything I'm missing when thinking that it is possible to add a simple lock file at the root of repodir in order to control access to this directory? The build master would lock the file when copying new packages to it. And the push script would lock the file when moving packages and when cleaning up. Both would sleep when they cannot acquire the lock.
On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote:
Currently, when we sign and push built packages, we move them out of a "repodir" while the build server may be copying new packages to that directory.
We have not cleaned up repodir for a long time either, because it could not be done automatically without risk. The corresponding section in the push script contains a "todo" comment only. [Recently I've used a separate script, however, which cleans up the directory after a push, provided that the build system is idle and no package is about to be copied into the directory at the same time.]
I've looked at Repo.py _update_repo() where I see nothing which prevents concurrent access to this repodir. Is there anything I'm missing when thinking that it is possible to add a simple lock file at the root of repodir in order to control access to this directory? The build master would lock the file when copying new packages to it. And the push script would lock the file when moving packages and when cleaning up. Both would sleep when they cannot acquire the lock.
Attached patch explains in code what I have in mind.
On Sat, 2006-05-20 at 16:50 +0200, Michael Schwendt wrote:
On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote:
Currently, when we sign and push built packages, we move them out of a "repodir" while the build server may be copying new packages to that directory.
We have not cleaned up repodir for a long time either, because it could not be done automatically without risk. The corresponding section in the push script contains a "todo" comment only. [Recently I've used a separate script, however, which cleans up the directory after a push, provided that the build system is idle and no package is about to be copied into the directory at the same time.]
I've looked at Repo.py _update_repo() where I see nothing which prevents concurrent access to this repodir. Is there anything I'm missing when thinking that it is possible to add a simple lock file at the root of repodir in order to control access to this directory? The build master would lock the file when copying new packages to it. And the push script would lock the file when moving packages and when cleaning up. Both would sleep when they cannot acquire the lock.
Attached patch explains in code what I have in mind.
Thanks, a modified version of this patch has been committed to HEAD, and I plan to commit one to STABLE_4 as well.
Dan
On Sat, 2006-05-20 at 16:50 +0200, Michael Schwendt wrote:
On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote:
Currently, when we sign and push built packages, we move them out of a "repodir" while the build server may be copying new packages to that directory.
We have not cleaned up repodir for a long time either, because it could not be done automatically without risk. The corresponding section in the push script contains a "todo" comment only. [Recently I've used a separate script, however, which cleans up the directory after a push, provided that the build system is idle and no package is about to be copied into the directory at the same time.]
I've looked at Repo.py _update_repo() where I see nothing which prevents concurrent access to this repodir. Is there anything I'm missing when thinking that it is possible to add a simple lock file at the root of repodir in order to control access to this directory? The build master would lock the file when copying new packages to it. And the push script would lock the file when moving packages and when cleaning up. Both would sleep when they cannot acquire the lock.
Attached patch explains in code what I have in mind.
Patch is applied to the Extras buildsys build server now too, so feel free to use the lockfile stuff in scripts and whatever on extras64.
Dan
On Fri, 02 Jun 2006 12:38:40 -0400, Dan Williams wrote:
Patch is applied to the Extras buildsys build server now too, so feel free to use the lockfile stuff in scripts and whatever on extras64.
Great. Found a bug, though:
--- Repo.py.~1.26.~ 2006-06-02 17:45:30.000000000 +0200 +++ Repo.py 2006-06-02 22:40:36.000000000 +0200 @@ -207,7 +207,7 @@ # Try to open the lockfile, creating it if it doesn't exist try: lockfile = open(self._lockfile_path, 'w') - except OSError, (errno, strerr): + except IOError, (errno, strerr): print "Repo Error (%s): opening lockfile %s failed. Output: (errno %d) '%s'" % (target_string, self._lockfile_path, errno, strerr)
On Fri, 2006-06-02 at 22:42 +0200, Michael Schwendt wrote:
On Fri, 02 Jun 2006 12:38:40 -0400, Dan Williams wrote:
Patch is applied to the Extras buildsys build server now too, so feel free to use the lockfile stuff in scripts and whatever on extras64.
Great. Found a bug, though:
Thanks, fixed in CVS HEAD and STABLE_4, and on extras64 as well.
Dan
--- Repo.py.~1.26.~ 2006-06-02 17:45:30.000000000 +0200 +++ Repo.py 2006-06-02 22:40:36.000000000 +0200 @@ -207,7 +207,7 @@ # Try to open the lockfile, creating it if it doesn't exist try: lockfile = open(self._lockfile_path, 'w')
except OSError, (errno, strerr):
except IOError, (errno, strerr): print "Repo Error (%s): opening lockfile %s failed. Output: (errno %d) '%s'" % (target_string, self._lockfile_path, errno, strerr)-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
buildsys@lists.fedoraproject.org
-
Dan Williams -
Michael Schwendt