Adding a lock file to plague server Repo.py _update_repo()

[Michael Schwendt]

Currently, when we sign and push built packages, we move them out of a
"repodir" while the build server may be copying new packages to that
directory.

We have not cleaned up repodir for a long time either, because it could
not be done automatically without risk. The corresponding section in the
push script contains a "todo" comment only. [Recently I've used a separate
script, however, which cleans up the directory after a push, provided that
the build system is idle and no package is about to be copied into the
directory at the same time.]

I've looked at Repo.py _update_repo() where I see nothing which prevents
concurrent access to this repodir. Is there anything I'm missing when
thinking that it is possible to add a simple lock file at the root of
repodir in order to control access to this directory?  The build master
would lock the file when copying new packages to it. And the push script
would lock the file when moving packages and when cleaning up. Both would
sleep when they cannot acquire the lock.

[PATCH] Re: Adding a lock file to plague server Repo.py _update_repo()

[Michael Schwendt]

On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote:

>  Currently, when we sign and push built packages, we move them out of
> a
>  "repodir" while the build server may be copying new packages to that
>  directory.
>  
>  We have not cleaned up repodir for a long time either, because it could
>  not be done automatically without risk. The corresponding section in the
>  push script contains a "todo" comment only. [Recently I've used a separate
>  script, however, which cleans up the directory after a push, provided that
>  the build system is idle and no package is about to be copied into the
>  directory at the same time.]
>  
>  I've looked at Repo.py _update_repo() where I see nothing which prevents
>  concurrent access to this repodir. Is there anything I'm missing when
>  thinking that it is possible to add a simple lock file at the root of
>  repodir in order to control access to this directory?  The build master
>  would lock the file when copying new packages to it. And the push script
>  would lock the file when moving packages and when cleaning up. Both would
>  sleep when they cannot acquire the lock. 
Attached patch explains in code what I have in mind.

[PATCH] Re: Adding a lock file to plague server Repo.py _update_repo()

[Dan Williams]

On Sat, 2006-05-20 at 16:50 +0200, Michael Schwendt wrote:
>  On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote:
>  
>  > Currently, when we sign and push built packages, we move them out of a
>  > "repodir" while the build server may be copying new packages to that
>  > directory.
>  > 
>  > We have not cleaned up repodir for a long time either, because it could
>  > not be done automatically without risk. The corresponding section in the
>  > push script contains a "todo" comment only. [Recently I've used a
> separate
>  > script, however, which cleans up the directory after a push, provided that
>  > the build system is idle and no package is about to be copied into the
>  > directory at the same time.]
>  > 
>  > I've looked at Repo.py _update_repo() where I see nothing which prevents
>  > concurrent access to this repodir. Is there anything I'm missing when
>  > thinking that it is possible to add a simple lock file at the root of
>  > repodir in order to control access to this directory?  The build master
>  > would lock the file when copying new packages to it. And the push script
>  > would lock the file when moving packages and when cleaning up. Both would
>  > sleep when they cannot acquire the lock.
>  
>  Attached patch explains in code what I have in mind. 
Thanks, a modified version of this patch has been committed to HEAD, and
I plan to commit one to STABLE_4 as well.

Dan

[PATCH] Re: Adding a lock file to plague server Repo.py _update_repo()

[Dan Williams]

On Sat, 2006-05-20 at 16:50 +0200, Michael Schwendt wrote:
>  On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote:
>  
>  > Currently, when we sign and push built packages, we move them out of a
>  > "repodir" while the build server may be copying new packages to that
>  > directory.
>  > 
>  > We have not cleaned up repodir for a long time either, because it could
>  > not be done automatically without risk. The corresponding section in the
>  > push script contains a "todo" comment only. [Recently I've used a
> separate
>  > script, however, which cleans up the directory after a push, provided that
>  > the build system is idle and no package is about to be copied into the
>  > directory at the same time.]
>  > 
>  > I've looked at Repo.py _update_repo() where I see nothing which prevents
>  > concurrent access to this repodir. Is there anything I'm missing when
>  > thinking that it is possible to add a simple lock file at the root of
>  > repodir in order to control access to this directory?  The build master
>  > would lock the file when copying new packages to it. And the push script
>  > would lock the file when moving packages and when cleaning up. Both would
>  > sleep when they cannot acquire the lock.
>  
>  Attached patch explains in code what I have in mind. 
Patch is applied to the Extras buildsys build server now too, so feel
free to use the lockfile stuff in scripts and whatever on extras64.

Dan

[PATCH] Re: Adding a lock file to plague server Repo.py _update_repo()

[Michael Schwendt]

On Fri, 02 Jun 2006 12:38:40 -0400, Dan Williams wrote:

>  Patch is applied to the Extras buildsys build server now too, so
> feel
>  free to use the lockfile stuff in scripts and whatever on extras64. 
Great. Found a bug, though:

--- Repo.py.~1.26.~	2006-06-02 17:45:30.000000000 +0200
+++ Repo.py	2006-06-02 22:40:36.000000000 +0200
@@ -207,7 +207,7 @@
         # Try to open the lockfile, creating it if it doesn't exist
         try:
             lockfile = open(self._lockfile_path, 'w')
-        except OSError, (errno, strerr):
+        except IOError, (errno, strerr):
             print "Repo Error (%s): opening lockfile %s failed.  Output: (errno %d)
'%s'" % (target_string,
                     self._lockfile_path, errno, strerr)

[PATCH] Re: Adding a lock file to plague server Repo.py _update_repo()

[Dan Williams]

On Fri, 2006-06-02 at 22:42 +0200, Michael Schwendt wrote:
>  On Fri, 02 Jun 2006 12:38:40 -0400, Dan Williams wrote:
>  
>  > Patch is applied to the Extras buildsys build server now too, so feel
>  > free to use the lockfile stuff in scripts and whatever on extras64.
>  
>  Great. Found a bug, though: 
Thanks, fixed in CVS HEAD and STABLE_4, and on extras64 as well.

Dan

>  --- Repo.py.~1.26.~	2006-06-02 17:45:30.000000000 +0200
>  +++ Repo.py	2006-06-02 22:40:36.000000000 +0200
>  @@ -207,7 +207,7 @@
>           # Try to open the lockfile, creating it if it doesn't exist
>           try:
>               lockfile = open(self._lockfile_path, 'w')
>  -        except OSError, (errno, strerr):
>  +        except IOError, (errno, strerr):
>               print "Repo Error (%s): opening lockfile %s failed.  Output: (errno %d)
> '%s'" % (target_string,
>                       self._lockfile_path, errno, strerr)
>   
>  
>  --
>  Fedora-buildsys-list mailing list
>  Fedora-buildsys-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-buildsys-list