Pungis work directory.

List overview All Threads
Download

newer

older

Trying to exclude kernel-PAE from...

F7 pungi-0.3.5-1.fc7 version ==...

Joel Andres Granados

13 Apr 2007 13 Apr '07

1:34 p.m.

I ran pungi once and created a fedora iso. When I ran it again with the same config file, and forgot to erase the previous work directory files, pungi shows a traceback telling me that some file already exists. I have two proposals for this situation. 1. create a timestamp directory inside the working directory for each pungi run. The problem with this approach is that the use is left with a bunch of directories that are named after whatever time.time() spit out (not very pretty). But alas it does away with the ugly traceback message. The diff for the first solution:

--- pungi-0.3.0 2007-04-02 23:17:25.000000000 +0200 +++ pungi-0.3.1-JG 2007-04-13 14:27:22.000000000 +0200 @@ -18,6 +18,7 @@ import yum

from ConfigParser import SafeConfigParser +from time import time

def main(): # Set some default variables, can be overrided in config file @@ -76,7 +77,8 @@

if not opts.destdir == "*CONFFILE*": config.set('default', 'destdir', opts.destdir) - + config.set('default', 'destdir', os.path.join(config.get('default','destdir'), + str(int(time())))) destdir = config.get('default', 'destdir')

if not os.path.exists(destdir):

2. Just tell the user to erase the things he/she has in the directory. The diff for the second solution:

--- pungi-0.3.0 2007-04-13 15:03:26.000000000 +0200 +++ pungi-0.3.1-JG 2007-04-13 15:02:22.000000000 +0200 @@ -18,6 +18,7 @@ import yum

from ConfigParser import SafeConfigParser +from os.path import isdir

def main(): # Set some default variables, can be overrided in config file @@ -77,7 +78,7 @@ if not opts.destdir == "*CONFFILE*": config.set('default', 'destdir', opts.destdir)

- destdir = config.get('default', 'destdir') + destdir = check_destdir(config.get('default', 'destdir'))

if not os.path.exists(destdir): try: @@ -180,4 +181,10 @@ manifestfile.close() return pkglist

+ def check_destdir(workpath): + if isdir(workpath) and len(os.listdir(workpath)) > 0: + print >> sys.stderr, "Please erase all items from the directory %s before continuing." % workpath + exit(1) + return workpath + main()

Show replies by date

Jesse Keating

15 May 15 May

6:18 p.m.

On Friday 13 April 2007 09:34:44 Joel Andres Granados wrote:

...

I ran pungi once and created a fedora iso. When I ran it again with the same config file, and forgot to erase the previous work directory files, pungi shows a traceback telling me that some file already exists. I have two proposals for this situation.

create a timestamp directory inside the working directory for each

pungi run. The problem with this approach is that the use is left with a bunch of directories that are named after whatever time.time() spit out (not very pretty). But alas it does away with the ugly traceback

This needs more thought as I took in a patch that separates out the different pungi tasks to be called separately, so the work dir _could_ exist in some cases. Will think more on this.

-- Jesse Keating Release Engineer: Fedora

Joel Andres Granados

16 May 16 May

8:34 a.m.

Jesse Keating wrote:

...

On Friday 13 April 2007 09:34:44 Joel Andres Granados wrote:

...
I ran pungi once and created a fedora iso. When I ran it again with the same config file, and forgot to erase the previous work directory files, pungi shows a traceback telling me that some file already exists. I have two proposals for this situation.

create a timestamp directory inside the working directory for each

pungi run. The problem with this approach is that the use is left with a bunch of directories that are named after whatever time.time() spit out (not very pretty). But alas it does away with the ugly traceback

This needs more thought as I took in a patch that separates out the different pungi tasks to be called separately, so the work dir _could_ exist in some cases. Will think more on this.

-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

Yes, depending on the command line options, the working directory is either used or not. IMHO I think the second solution (Tell the user what is happening) is the best way to go. In this way if the directory is needed and is already there the user will receive a message, but if the directory is not needed pungi will be silent about the situation. I'll revisit this patch today and see if it behaves like expected. I'll post any results on the list.

Regards.

Joel Andres Granados

5:22 p.m.

Joel Andres Granados wrote:

...

Jesse Keating wrote:

...
On Friday 13 April 2007 09:34:44 Joel Andres Granados wrote:

...
I ran pungi once and created a fedora iso. When I ran it again with the same config file, and forgot to erase the previous work directory files, pungi shows a traceback telling me that some file already exists. I have two proposals for this situation.

create a timestamp directory inside the working directory for each

pungi run. The problem with this approach is that the use is left with a bunch of directories that are named after whatever time.time() spit out (not very pretty). But alas it does away with the ugly traceback

This needs more thought as I took in a patch that separates out the different pungi tasks to be called separately, so the work dir _could_ exist in some cases. Will think more on this.

-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

Yes, depending on the command line options, the working directory is either used or not. IMHO I think the second solution (Tell the user what is happening) is the best way to go. In this way if the directory is needed and is already there the user will receive a message, but if the directory is not needed pungi will be silent about the situation. I'll revisit this patch today and see if it behaves like expected. I'll post any results on the list.

Regards.

-- Fedora-buildsys-list mailing list Fedora-buildsys-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

Looked into this a little more and came up with something completely different than my previous post. :)

I tested to see what would happen with each if the options (-G -B -P -S -I) and I found out that on G,B and I tracedback, in the other options it didn't. When I tested again (I don't know why I tested two times), I found that the tracebacks where different in G, B and I. So basically the trace backs depend on the state of the directory. If the directory is in one state the traceback happens in one line, if it is in another state it has the possibility of occurring in another. So I thought that the best way to fix it would be to check for the existence of the directory in every directory creation, so I looked for every occurrence of "os.makedirs":

/usr/lib/python2.5/site-packages/pypungi/gather.py:64: os.makedirs(logdir) /usr/lib/python2.5/site-packages/pypungi/gather.py:275: os.makedirs(pkgdir) /usr/lib/python2.5/site-packages/pypungi/gather.py:342: os.makedirs(pkgdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:45: os.makedirs(self.workdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:165: os.makedirs(docsdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:263: os.makedirs("%s-disc%d/SRPMS" % (timber.dist_dir, i)) /usr/lib/python2.5/site-packages/pypungi/pungi.py:293: os.makedirs('%s-disc1' % self.topdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:320: os.makedirs(self.isodir) /usr/bin/pungi:90: os.makedirs(destdir) /usr/bin/pungi:99: os.makedirs(cachedir)

I just don't like it... IMO its just not right to check for the existence of the directory in every line!!!

So I take what I said in my last post back, and am sticking with the "create a new root directory for each execution" solution. I changed it a little. Instead of using time.time(), I used time.localtime()'s elements. So now the directory would be something like "/srv/pungi/Fedora/2007-5-16-1430/..." IMO it looks much better than what time.time() spits out.

The diff is attached.

Regards

--- pungi-Original/pungi 2007-05-15 16:43:54.000000000 +0200 +++ /usr/bin/pungi 2007-05-15 18:38:15.000000000 +0200 @@ -18,6 +18,7 @@ import yum

from ConfigParser import SafeConfigParser +from time import localtime

def main(): # Set some default variables, can be overrided in config file @@ -77,6 +78,11 @@ if not opts.destdir == "*CONFFILE*": config.set('default', 'destdir', opts.destdir)

+ t = localtime() + tstamp = "%s-%s-%s-%s%s" % (t[0],t[1],t[2],t[3],t[4]) + config.set('default', 'destdir', + os.path.join(config.get('default','destdir'),tstamp)) + destdir = config.get('default', 'destdir')

if not os.path.exists(destdir):

Joel Andres Granados

5 Jun 5 Jun

5:27 p.m.

...

I tested to see what would happen with each if the options (-G -B -P -S -I) and I found out that on G,B and I tracedback, in the other options it didn't. When I tested again (I don't know why I tested two times), I found that the tracebacks where different in G, B and I. So basically the trace backs depend on the state of the directory. If the directory is in one state the traceback happens in one line, if it is in another state it has the possibility of occurring in another. So I thought that the best way to fix it would be to check for the existence of the directory in every directory creation, so I looked for every occurrence of "os.makedirs":

/usr/lib/python2.5/site-packages/pypungi/gather.py:64: os.makedirs(logdir) /usr/lib/python2.5/site-packages/pypungi/gather.py:275: os.makedirs(pkgdir) /usr/lib/python2.5/site-packages/pypungi/gather.py:342: os.makedirs(pkgdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:45: os.makedirs(self.workdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:165: os.makedirs(docsdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:263: os.makedirs("%s-disc%d/SRPMS" % (timber.dist_dir, i)) /usr/lib/python2.5/site-packages/pypungi/pungi.py:293: os.makedirs('%s-disc1' % self.topdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:320: os.makedirs(self.isodir) /usr/bin/pungi:90: os.makedirs(destdir) /usr/bin/pungi:99: os.makedirs(cachedir)

Actually more lines must be changed. All the shutils lines that create a directory in one way or another have the potential to traceback with the "directory already exists" message.

...

I just don't like it... IMO its just not right to check for the existence of the directory in every line!!!

So I take what I said in my last post back, and am sticking with the "create a new root directory for each execution" solution. I changed it a little. Instead of using time.time(), I used time.localtime()'s elements. So now the directory would be something like "/srv/pungi/Fedora/2007-5-16-1430/..." IMO it looks much better than what time.time() spits out.

Considering the state of pungi in which it is separated by stages this ^^^^^ solution is not really very useful :(. An option in which the user selects the name of the tree he wants to use is needed (not pretty). Came up with another solution that until now has pretty much solved the problem and has not broken pungi (in my tests). The solutions simply clears any directory that pungi is going to create. So for each line in which a directory is created I added a line to erase it. The diff is attached. Regards.

-- Joel Andres Granados diff --git a/pypungi/gather.py b/pypungi/gather.py index 52a5f03..67048ae 100755 --- a/pypungi/gather.py +++ b/pypungi/gather.py @@ -305,6 +305,7 @@ class Gather(yum.YumBase): target = os.path.join(pkgdir, os.path.basename(remote)) if os.path.exists(target): os.remove(target) # avoid traceback after interrupted download + if os.path.isfile(target):os.remove(target)#make sure to refresh os.link(local, target) continue @@ -319,7 +320,9 @@ class Gather(yum.YumBase): if not os.path.exists(local) or not os.path.samefile(path, local): shutil.copy2(path, local) - os.link(local, os.path.join(pkgdir, os.path.basename(remote))) + target = os.path.join(pkgdir, os.path.basename(remote)) + if os.path.isfile(target): os.remove(target)#make sure to refresh + os.link(local, target) def downloadSRPMs(self): @@ -368,11 +371,14 @@ class Gather(yum.YumBase): if not self.config.has_option('default', 'quiet'): self.logger.info("%s already exists and appears to be complete" % local) - if os.path.exists(os.path.join(pkgdir, os.path.basename(remote))) and self.verifyCachePkg(pkg, os.path.join(pkgdir, os.path.basename(remote))) == pkg.packagesize: + if os.path.exists(os.path.join(pkgdir, os.path.basename(remote))) and \ + self.verifyCachePkg(pkg, os.path.join(pkgdir, os.path.basename(remote))) == pkg.packagesize: if not self.config.has_option('default', 'quiet'): self.logger.info("%s already exists in tree and appears to be complete" % local) else: - os.link(local, os.path.join(pkgdir, os.path.basename(remote))) + target = os.path.join(pkgdir, os.path.basename(remote)) + if os.path.isfile(target): os.remove(target) + os.link(local, target) continue # Disable cache otherwise things won't download @@ -386,4 +392,6 @@ class Gather(yum.YumBase): if not os.path.exists(local) or not os.path.samefile(path, local): shutil.copy2(path, local) - os.link(local, os.path.join(pkgdir, os.path.basename(remote))) + target = os.path.join(pkgdir, os.path.basename(remote)) + if os.path.isfile(target): os.remove(target) + os.link(local, target) diff --git a/pypungi/pungi.py b/pypungi/pungi.py index 3f74669..2389131 100755 --- a/pypungi/pungi.py +++ b/pypungi/pungi.py @@ -162,7 +162,8 @@ class Pungi: for pattern in self.config.get('default', 'relnotedirre').split(): dirres.append(re.compile(pattern)) - os.makedirs(docsdir) + if not os.path.exists(docsdir): + os.makedirs(docsdir) # Expload the packages we list as relnote packages pkgs = os.listdir(os.path.join(self.topdir, self.config.get('default', 'product_path'))) @@ -201,7 +202,9 @@ class Pungi: for regex in dirres: if regex.match(directory) and not os.path.exists(os.path.join(self.topdir, directory)): log.info("Copying release note dir %s" % directory) - shutil.copytree(os.path.join(dirpath, directory), os.path.join(self.topdir, directory)) + target = os.path.join(self.topdir, directory) + shutil.rmtree(target, ignore_errors=True) + shutil.copytree(os.path.join(dirpath, directory), target) def doSplittree(self): @@ -260,10 +263,11 @@ class Pungi: # this is stolen from splittree.py in anaconda-runtime. Blame them if its ugly (: for i in range(timber.src_list[0], timber.src_list[-1] + 1): + if not os.path.exists("%s-disc%d/SRPMS" % (timber.dist_dir, i)): os.makedirs("%s-disc%d/SRPMS" % (timber.dist_dir, i)) - timber.linkFiles(timber.dist_dir, - "%s-disc%d" %(timber.dist_dir, i), - timber.common_files) + timber.linkFiles(timber.dist_dir, + "%s-disc%d" %(timber.dist_dir, i), + timber.common_files) timber.splitSRPMS() log.info("splitSRPMS complete") @@ -289,7 +293,8 @@ class Pungi: createrepo.append('media://%s' % mediaid) createrepo.append('--outputdir') - if self.config.getint('default', 'discs') == 1: + if self.config.getint('default', 'discs') == 1 and \ + not os.path.exists('%s-disc1' % self.topdir): os.makedirs('%s-disc1' % self.topdir) createrepo.append('%s-disc1' % self.topdir) @@ -317,7 +322,8 @@ class Pungi: anaruntime = '/usr/lib/anaconda-runtime/boot' discinfofile = os.path.join(self.topdir, '.discinfo') # we use this a fair amount - os.makedirs(self.isodir) + if not os.path.exists(self.isodir): + os.makedirs(self.isodir) # setup the base command mkisofs = ['/usr/bin/mkisofs'] @@ -401,9 +407,10 @@ class Pungi: content[content.index('ALL\n')] = ','.join([str(x) for x in range(1, self.config.getint('default', 'discs') + 1)]) + '\n' open(discinfofile, 'w').writelines(content) - # move the main repodata out of the way to use the split repodata - shutil.move(os.path.join(self.topdir, 'repodata'), os.path.join(self.config.get('default', 'destdir'), - 'repodata-%s' % self.config.get('default', 'arch'))) + # move the main repodata directory out of the way to use the split repodata + target = os.path.join(self.config.get('default', 'destdir'), 'repodata-%s' % self.config.get('default', 'arch')) + shutil.rmtree(target, ignore_errors=True) + shutil.move(os.path.join(self.topdir, 'repodata'), target) shutil.copytree('%s-disc1/repodata' % self.topdir, os.path.join(self.topdir, 'repodata')) # setup the extra mkisofs args @@ -448,9 +455,10 @@ class Pungi: # return the .discinfo file if not self.config.get('default', 'arch') == 'source': - shutil.move(os.path.join(self.config.get('default', 'destdir'), '.discinfo-%s' % self.config.get('default', 'arch')), discinfofile) + shutil.move(os.path.join(self.config.get('default', 'destdir'), + '.discinfo-%s' % self.config.get('default', 'arch')), discinfofile) - shutil.rmtree(os.path.join(self.topdir, 'repodata')) # remove our copied repodata + shutil.rmtree(os.path.join(self.topdir, 'repodata'), ignore_errors=True) # remove our copied repodata shutil.move(os.path.join(self.config.get('default', 'destdir'), 'repodata-%s' % self.config.get('default', 'arch')), os.path.join(self.topdir, 'repodata')) @@ -512,6 +520,7 @@ class Pungi: for directory in dirs: if directory.startswith('os-disc') or directory.startswith('SRPM-disc'): + shutil.rmtree(os.path.join(self.workdir,directory), ignore_errors=True) shutil.move(os.path.join(self.archdir, directory), os.path.join(self.workdir, directory)) log.info("CreateIsos is done.")

6392

Age (days ago)

6445

Last active (days ago)

buildsys@lists.fedoraproject.org

4 comments

2 participants

tags (0)

participants (2)

Jesse Keating
Joel Andres Granados