Kojid unable to regen-repo in Fedora 15

[Anthony Joseph Messina]

I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15.  My
koji hub is still running Fedora 14.

When running a regen-repo, I now get this error:

2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
(most recent call last):
  File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
runTask
    response = (handler.run(),)
  File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
    return self.handler(*self.params,**self.opts)
  File "/usr/sbin/kojid", line 2552, in handler
    self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
  File "/usr/sbin/kojid", line 2606, in create_local_repo
    % parseStatus(status, ' '.join(cmd))
GenericError: failed to create repo: /usr/bin/createrepo -vd -o
/tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages
-i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g
/mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
/mnt/koji/packages/ exited with status 1

Any ideas on where I should go?  Thanks in advance. -A

[Mike McLean]

On 05/25/2011 09:51 PM, Anthony Messina wrote:
> I'm running a Kojid instance, freshly upgraded from Fedora 14 ->  15.  My
> koji hub is still running Fedora 14.
>
> When running a regen-repo, I now get this error:
>
> 2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback

...
> /mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
> /mnt/koji/packages/ exited with status 1
>
> Any ideas on where I should go?  Thanks in advance. -A

Is there a createrepo.log for that task?

[Anthony Joseph Messina]

On 05/25/2011 10:24 PM, Mike McLean wrote:
> On 05/25/2011 09:51 PM, Anthony Messina wrote:
> > I'm running a Kojid instance, freshly upgraded from Fedora 14 ->  15.  My
> > koji hub is still running Fedora 14.
> >
> > When running a regen-repo, I now get this error:
> >
> > 2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
>
> ...
> > /mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
> > /mnt/koji/packages/ exited with status 1
> >
> > Any ideas on where I should go?  Thanks in advance. -A
> Is there a createrepo.log for that task?
>
> buildsys mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

Well the errors I get now are a little different, but I cannot regen a
repo just the same on an F15 kojid (F14 koji-hub).

Logs attached...

[Dennis Gilmore]

On Wednesday, May 25, 2011 08:51:18 PM Anthony Messina wrote:
> I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15.  My
> koji hub is still running Fedora 14.
>
> When running a regen-repo, I now get this error:
>
> 2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
> (most recent call last):
>   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
> runTask
>     response = (handler.run(),)
>   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
>     return self.handler(*self.params,**self.opts)
>   File "/usr/sbin/kojid", line 2552, in handler
>     self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
>   File "/usr/sbin/kojid", line 2606, in create_local_repo
>     % parseStatus(status, ' '.join(cmd))
> GenericError: failed to create repo: /usr/bin/createrepo -vd -o
> /tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages
> -i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g
> /mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
> /mnt/koji/packages/ exited with status 1
>
> Any ideas on where I should go?  Thanks in advance. -A

I need to look into this,  Its due to changes in createrepo, if you grab the 
f14 createrepo it will work as a workaround

Dennis

[Anthony Joseph Messina]

On 05/26/2011 03:55 PM, Dennis Gilmore wrote:
> I need to look into this,  Its due to changes in createrepo, if you grab the 
> f14 createrepo it will work as a workaround
>
> Dennis

Thanks, I can confirm that createrepo-0.9.8-5.fc14.noarch works without
issue.  -A

[Anthony Joseph Messina]

On 05/26/2011 03:55 PM, Dennis Gilmore wrote:
> On Wednesday, May 25, 2011 08:51:18 PM Anthony Messina wrote:
> > > I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15.  My
> > > koji hub is still running Fedora 14.
> > >
> > > When running a regen-repo, I now get this error:
> > >
> > > 2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
> > > (most recent call last):
> > >   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
> > > runTask
> > >     response = (handler.run(),)
> > >   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
> > >     return self.handler(*self.params,**self.opts)
> > >   File "/usr/sbin/kojid", line 2552, in handler
> > >     self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
> > >   File "/usr/sbin/kojid", line 2606, in create_local_repo
> > >     % parseStatus(status, ' '.join(cmd))
> > > GenericError: failed to create repo: /usr/bin/createrepo -vd -o
> > > /tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages
> > > -i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g
> > > /mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
> > > /mnt/koji/packages/ exited with status 1
> > >
> > > Any ideas on where I should go?  Thanks in advance. -A
>
> I need to look into this,  Its due to changes in createrepo, if you grab the 
> f14 createrepo it will work as a workaround
>
> Dennis

Dennis, were you able to find a resolution in createrepo for this issue
on F15?  Just checking.  Thanks.  -A

[Anthony Joseph Messina]

On 05/26/2011 03:55 PM, Dennis Gilmore wrote:
> On Wednesday, May 25, 2011 08:51:18 PM Anthony Messina wrote:
> > > I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15.  My
> > > koji hub is still running Fedora 14.
> > >
> > > When running a regen-repo, I now get this error:
> > >
> > > 2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
> > > (most recent call last):
> > >   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
> > > runTask
> > >     response = (handler.run(),)
> > >   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
> > >     return self.handler(*self.params,**self.opts)
> > >   File "/usr/sbin/kojid", line 2552, in handler
> > >     self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
> > >   File "/usr/sbin/kojid", line 2606, in create_local_repo
> > >     % parseStatus(status, ' '.join(cmd))
> > > GenericError: failed to create repo: /usr/bin/createrepo -vd -o
> > > /tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages
> > > -i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g
> > > /mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
> > > /mnt/koji/packages/ exited with status 1
> > >
> > > Any ideas on where I should go?  Thanks in advance. -A
>
> I need to look into this,  Its due to changes in createrepo, if you grab the 
> f14 createrepo it will work as a workaround
>
> Dennis

Based on Seth's recommendation in
https://bugzilla.redhat.com/show_bug.cgi?id=718635

upgrading to createrepo-0.9.9-5.fc16.noarch from rawhide seems to
resolve this issue.

[Anthony Joseph Messina]

On 07/29/2011 04:41 PM, Anthony Messina wrote:
> Based on Seth's recommendation in
> https://bugzilla.redhat.com/show_bug.cgi?id=718635
>
> upgrading to createrepo-0.9.9-5.fc16.noarch from rawhide seems to
> resolve this issue.

Well, while using either createrepo-0.9.9-4.fc15.noarch or
createrepo-0.9.9-5.fc16.noarch, I am able to successfully regenerate a
repo in Koji, but building a package in the resulting repo fails.

With the new createrepo versions, the repodata files contain sums:
tasks/2898/2898,
['9b19d6d5388866de3256ccf433020b2675678c74-filelists.xml.gz',
'da149cf3eedf43067a2409be98f09f6ac59b1d80-primary.xml.gz',
'14f7b241420a4283ebb06d9b5ded53789a52a716-comps.xml',
'cb4259761f5364bf6203071af4166b237fcdb54b-comps.xml.gz',
'5ddafbf985de1e52f3a4d86d210788dcdf9d12c9-other.sqlite.bz2',
'repomd.xml', '0c3c735cacccc61ee33be459ebb534c22428df38-pkgorigins.gz',
'42e1da88ccdfbdf44350fcc0fcdc554495f3f5c8-filelists.sqlite.bz2',
'a1c6e2a342d4b6d809a50ea2837f6ff8ae2b2c06-primary.sqlite.bz2',
'701919d49a2a98ff685164900e49a1989dfeff1a-other.xml.gz']

With the older stable f14 version, the repodata files don't:
tasks/2878/2878, ['primary.xml.gz', 'comps.xml', 'other.sqlite.bz2',
'filelists.sqlite.bz2', 'filelists.xml.gz', 'repomd.xml',
'primary.sqlite.bz2', 'pkgorigins.gz', 'other.xml.gz', 'comps.xml.gz']

The builds fail with:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
runTask
    response = (handler.run(),)
  File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
    return self.handler(*self.params,**self.opts)
  File "/usr/sbin/kojid", line 2129, in handler
    broot.init()
  File "/usr/sbin/kojid", line 373, in init
    self.session.host.setBuildRootList(self.id,self.getPackageList())
  File "/usr/sbin/kojid", line 456, in getPackageList
    self.markExternalRPMs(ret)
  File "/usr/sbin/kojid", line 559, in markExternalRPMs
    fo = koji.openRemoteFile(relpath, **opts)
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1376,
in openRemoteFile
    fo = open(fn)
IOError: [Errno 2] No such file or directory:
'/mnt/koji//repos/dist-f15-build/386/x86_64/repodata/pkgorigins.gz'

Referring to https://bugzilla.redhat.com/show_bug.cgi?id=718635 (a
createrepo ticket), is this now a Koji issue?

[Paul Howarth]

On Wed, 03 Aug 2011 20:54:34 -0500
Anthony Messina  wrote:
> On 07/29/2011 04:41 PM, Anthony Messina wrote:
> > Based on Seth's recommendation in
> > https://bugzilla.redhat.com/show_bug.cgi?id=718635
> >
> > upgrading to createrepo-0.9.9-5.fc16.noarch from rawhide seems to
> > resolve this issue.
>
> Well, while using either createrepo-0.9.9-4.fc15.noarch or
> createrepo-0.9.9-5.fc16.noarch, I am able to successfully regenerate a
> repo in Koji, but building a package in the resulting repo fails.
>
> With the new createrepo versions, the repodata files contain sums:
> tasks/2898/2898,
> ['9b19d6d5388866de3256ccf433020b2675678c74-filelists.xml.gz',
> 'da149cf3eedf43067a2409be98f09f6ac59b1d80-primary.xml.gz',
> '14f7b241420a4283ebb06d9b5ded53789a52a716-comps.xml',
> 'cb4259761f5364bf6203071af4166b237fcdb54b-comps.xml.gz',
> '5ddafbf985de1e52f3a4d86d210788dcdf9d12c9-other.sqlite.bz2',
> 'repomd.xml',
> '0c3c735cacccc61ee33be459ebb534c22428df38-pkgorigins.gz',
> '42e1da88ccdfbdf44350fcc0fcdc554495f3f5c8-filelists.sqlite.bz2',
> 'a1c6e2a342d4b6d809a50ea2837f6ff8ae2b2c06-primary.sqlite.bz2',
> '701919d49a2a98ff685164900e49a1989dfeff1a-other.xml.gz']
>
> With the older stable f14 version, the repodata files don't:
> tasks/2878/2878, ['primary.xml.gz', 'comps.xml', 'other.sqlite.bz2',
> 'filelists.sqlite.bz2', 'filelists.xml.gz', 'repomd.xml',
> 'primary.sqlite.bz2', 'pkgorigins.gz', 'other.xml.gz', 'comps.xml.gz']
>
> The builds fail with:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114,
> in runTask
>     response = (handler.run(),)
>   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in
> run return self.handler(*self.params,**self.opts)
>   File "/usr/sbin/kojid", line 2129, in handler
>     broot.init()
>   File "/usr/sbin/kojid", line 373, in init
>     self.session.host.setBuildRootList(self.id,self.getPackageList())
>   File "/usr/sbin/kojid", line 456, in getPackageList
>     self.markExternalRPMs(ret)
>   File "/usr/sbin/kojid", line 559, in markExternalRPMs
>     fo = koji.openRemoteFile(relpath, **opts)
>   File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1376,
> in openRemoteFile
>     fo = open(fn)
> IOError: [Errno 2] No such file or directory:
> '/mnt/koji//repos/dist-f15-build/386/x86_64/repodata/pkgorigins.gz'
>
> Referring to https://bugzilla.redhat.com/show_bug.cgi?id=718635 (a
> createrepo ticket), is this now a Koji issue?

Can koji be persuaded to use "createrepo --simple-md-filenames"?

Paul.

[Anthony Joseph Messina]

On 08/04/2011 01:35 AM, Paul Howarth wrote:
> On Wed, 03 Aug 2011 20:54:34 -0500
> Anthony Messina  wrote:
> > > On 07/29/2011 04:41 PM, Anthony Messina wrote:
> > > > > Based on Seth's recommendation in
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=718635
> > > > >
> > > > > upgrading to createrepo-0.9.9-5.fc16.noarch from rawhide seems to
> > > > > resolve this issue.
> > >
> > > Well, while using either createrepo-0.9.9-4.fc15.noarch or
> > > createrepo-0.9.9-5.fc16.noarch, I am able to successfully regenerate a
> > > repo in Koji, but building a package in the resulting repo fails.
> > >
> > > With the new createrepo versions, the repodata files contain sums:
> > > tasks/2898/2898,
> > > ['9b19d6d5388866de3256ccf433020b2675678c74-filelists.xml.gz',
> > > 'da149cf3eedf43067a2409be98f09f6ac59b1d80-primary.xml.gz',
> > > '14f7b241420a4283ebb06d9b5ded53789a52a716-comps.xml',
> > > 'cb4259761f5364bf6203071af4166b237fcdb54b-comps.xml.gz',
> > > '5ddafbf985de1e52f3a4d86d210788dcdf9d12c9-other.sqlite.bz2',
> > > 'repomd.xml',
> > > '0c3c735cacccc61ee33be459ebb534c22428df38-pkgorigins.gz',
> > > '42e1da88ccdfbdf44350fcc0fcdc554495f3f5c8-filelists.sqlite.bz2',
> > > 'a1c6e2a342d4b6d809a50ea2837f6ff8ae2b2c06-primary.sqlite.bz2',
> > > '701919d49a2a98ff685164900e49a1989dfeff1a-other.xml.gz']
> > >
> > > With the older stable f14 version, the repodata files don't:
> > > tasks/2878/2878, ['primary.xml.gz', 'comps.xml', 'other.sqlite.bz2',
> > > 'filelists.sqlite.bz2', 'filelists.xml.gz', 'repomd.xml',
> > > 'primary.sqlite.bz2', 'pkgorigins.gz', 'other.xml.gz', 'comps.xml.gz']
> > >
> > > The builds fail with:
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114,
> > > in runTask
> > >     response = (handler.run(),)
> > >   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in
> > > run return self.handler(*self.params,**self.opts)
> > >   File "/usr/sbin/kojid", line 2129, in handler
> > >     broot.init()
> > >   File "/usr/sbin/kojid", line 373, in init
> > >     self.session.host.setBuildRootList(self.id,self.getPackageList())
> > >   File "/usr/sbin/kojid", line 456, in getPackageList
> > >     self.markExternalRPMs(ret)
> > >   File "/usr/sbin/kojid", line 559, in markExternalRPMs
> > >     fo = koji.openRemoteFile(relpath, **opts)
> > >   File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1376,
> > > in openRemoteFile
> > >     fo = open(fn)
> > > IOError: [Errno 2] No such file or directory:
> > > '/mnt/koji//repos/dist-f15-build/386/x86_64/repodata/pkgorigins.gz'
> > >
> > > Referring to https://bugzilla.redhat.com/show_bug.cgi?id=718635 (a
> > > createrepo ticket), is this now a Koji issue?
>
> Can koji be persuaded to use "createrepo --simple-md-filenames"?
>
> Paul.

If there exists such a way, I am not sure how to go about invoking it.  -A

[Anthony Joseph Messina]

On 08/04/2011 01:35 AM, Paul Howarth wrote:
> On Wed, 03 Aug 2011 20:54:34 -0500
> Anthony Messina  wrote:
> > > On 07/29/2011 04:41 PM, Anthony Messina wrote:
> > > > > Based on Seth's recommendation in
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=718635
> > > > >
> > > > > upgrading to createrepo-0.9.9-5.fc16.noarch from rawhide seems to
> > > > > resolve this issue.
> > >
> > > Well, while using either createrepo-0.9.9-4.fc15.noarch or
> > > createrepo-0.9.9-5.fc16.noarch, I am able to successfully regenerate a
> > > repo in Koji, but building a package in the resulting repo fails.
> > >
> > > With the new createrepo versions, the repodata files contain sums:
> > > tasks/2898/2898,
> > > ['9b19d6d5388866de3256ccf433020b2675678c74-filelists.xml.gz',
> > > 'da149cf3eedf43067a2409be98f09f6ac59b1d80-primary.xml.gz',
> > > '14f7b241420a4283ebb06d9b5ded53789a52a716-comps.xml',
> > > 'cb4259761f5364bf6203071af4166b237fcdb54b-comps.xml.gz',
> > > '5ddafbf985de1e52f3a4d86d210788dcdf9d12c9-other.sqlite.bz2',
> > > 'repomd.xml',
> > > '0c3c735cacccc61ee33be459ebb534c22428df38-pkgorigins.gz',
> > > '42e1da88ccdfbdf44350fcc0fcdc554495f3f5c8-filelists.sqlite.bz2',
> > > 'a1c6e2a342d4b6d809a50ea2837f6ff8ae2b2c06-primary.sqlite.bz2',
> > > '701919d49a2a98ff685164900e49a1989dfeff1a-other.xml.gz']
> > >
> > > With the older stable f14 version, the repodata files don't:
> > > tasks/2878/2878, ['primary.xml.gz', 'comps.xml', 'other.sqlite.bz2',
> > > 'filelists.sqlite.bz2', 'filelists.xml.gz', 'repomd.xml',
> > > 'primary.sqlite.bz2', 'pkgorigins.gz', 'other.xml.gz', 'comps.xml.gz']
> > >
> > > The builds fail with:
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114,
> > > in runTask
> > >     response = (handler.run(),)
> > >   File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in
> > > run return self.handler(*self.params,**self.opts)
> > >   File "/usr/sbin/kojid", line 2129, in handler
> > >     broot.init()
> > >   File "/usr/sbin/kojid", line 373, in init
> > >     self.session.host.setBuildRootList(self.id,self.getPackageList())
> > >   File "/usr/sbin/kojid", line 456, in getPackageList
> > >     self.markExternalRPMs(ret)
> > >   File "/usr/sbin/kojid", line 559, in markExternalRPMs
> > >     fo = koji.openRemoteFile(relpath, **opts)
> > >   File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1376,
> > > in openRemoteFile
> > >     fo = open(fn)
> > > IOError: [Errno 2] No such file or directory:
> > > '/mnt/koji//repos/dist-f15-build/386/x86_64/repodata/pkgorigins.gz'
> > >
> > > Referring to https://bugzilla.redhat.com/show_bug.cgi?id=718635 (a
> > > createrepo ticket), is this now a Koji issue?
>
> Can koji be persuaded to use "createrepo --simple-md-filenames"?
>
> Paul.

Ok, I've looked into this a bit and may be going about this in
completely the wrong way, but...

Modifying line 2577
(https://fedorahosted.org/koji/browser/builder/kojid#L2577) does not work.

cmd = ['/usr/bin/createrepo', '-vd', '--simple-md-filenames', '-o',
self.outdir, '-u', self.options.topurl]

However, modifying mergerepos to add the following after line 96
(https://fedorahosted.org/koji/browser/builder/mergerepos#L96) allows
the koji repo to be regenerated AND allows packages to be built from
that repo:

self.mdconf.unique_md_filenames = False

I am not certain that this is THE way to fix it, but it is a workaround
for now.

Thanks for pointing me in the right direction.  -A