unicode error with mergerepos against CentOS 5.

[Steve Traylen]

Hi,

$ rpm -qf /usr/libexec/kojid/mergerepos
koji-builder-1.3.1-1.fc10.noarch

Running agaist Fedora repositories all looks well but against CentOS.

/usr/libexec/kojid/mergerepos -a x86_64 \
   -o repo  \
   -r  http://swissmirror.silyus.net/centos/5.2/os/i386/

Adding repo: http://swissmirror.silyus.net/centos/5.2/os/i386/
1/400 - Deployment_Guide-or-IN-5.2-9.el5.centos.noarch
2/400 - 1:kde-i18n-Icelandic-3.5.4-1.noarch
....
....
328/400 - system-config-users-1.2.51-4.el5.noarch
329/400 - yum-priorities-1.1.10-9.el5.centos.noarch
Traceback (most recent call last):
 File "/usr/libexec/kojid/mergerepos", line 241, in <module>
   main(sys.argv[1:])
 File "/usr/libexec/kojid/mergerepos", line 236, in main
   merge.write_metadata()
 File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
   mdgen.doPkgMetadata()
 File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
332, in doPkgMetadata
   self.writeMetadataDocs(packages)
 File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
472, in writeMetadataDocs
   self.primaryfile.write(po.xml_dump_primary_metadata())
 File "/usr/lib/python2.5/site-packages/yum/packages.py", line 943,
in xml_dump_primary_metadata
   msg += misc.to_unicode(self._dump_format_items())
 File "/usr/lib/python2.5/site-packages/yum/packages.py", line 809,
in _dump_format_items
   msg += self._dump_pco('provides')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
38: ordinal not in range(128)

Tried with a few other repos as well, e.g the i386 one , the two
merged , ... it fails the same
though apparently  at different packages.

[Seth Vidal]

On Mon, 16 Mar 2009, Steve Traylen wrote:
> Hi,
>
> $ rpm -qf /usr/libexec/kojid/mergerepos
> koji-builder-1.3.1-1.fc10.noarch
>
> Running agaist Fedora repositories all looks well but against CentOS.
>
> /usr/libexec/kojid/mergerepos -a x86_64 \
>   -o repo  \
>   -r  http://swissmirror.silyus.net/centos/5.2/os/i386/
>
> Adding repo: http://swissmirror.silyus.net/centos/5.2/os/i386/
> 1/400 - Deployment_Guide-or-IN-5.2-9.el5.centos.noarch
> 2/400 - 1:kde-i18n-Icelandic-3.5.4-1.noarch
> ....
> ....
> 328/400 - system-config-users-1.2.51-4.el5.noarch
> 329/400 - yum-priorities-1.1.10-9.el5.centos.noarch
> Traceback (most recent call last):
> File "/usr/libexec/kojid/mergerepos", line 241, in <module>
>   main(sys.argv[1:])
> File "/usr/libexec/kojid/mergerepos", line 236, in main
>   merge.write_metadata()
> File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
>   mdgen.doPkgMetadata()
> File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
> 332, in doPkgMetadata
>   self.writeMetadataDocs(packages)
> File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
> 472, in writeMetadataDocs
>   self.primaryfile.write(po.xml_dump_primary_metadata())
> File "/usr/lib/python2.5/site-packages/yum/packages.py", line 943,
> in xml_dump_primary_metadata
>   msg += misc.to_unicode(self._dump_format_items())
> File "/usr/lib/python2.5/site-packages/yum/packages.py", line 809,
> in _dump_format_items
>   msg += self._dump_pco('provides')
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
> 38: ordinal not in range(128)
>
> Tried with a few other repos as well, e.g the i386 one , the two
> merged , ... it fails the same
> though apparently  at different packages.

the metadata you're pulling in was written with a much older createrepo 
which was less picky about some of the data it wrote back out. So you end 
up with some pkgs with dodgy data in the changelogs.

-sv

[Steve Traylen]

On Mon, Mar 16, 2009 at 1:29 PM, Seth Vidal  wrote:
> On Mon, 16 Mar 2009, Steve Traylen wrote:
> > Hi,
> >
> > $ rpm -qf /usr/libexec/kojid/mergerepos
> > koji-builder-1.3.1-1.fc10.noarch
> >
> > Running agaist Fedora repositories all looks well but against CentOS.
> >
> > /usr/libexec/kojid/mergerepos -a x86_64 \
> >  -o repo  \
> >  -r  http://swissmirror.silyus.net/centos/5.2/os/i386/
> >
> > Adding repo: http://swissmirror.silyus.net/centos/5.2/os/i386/
> > 1/400 - Deployment_Guide-or-IN-5.2-9.el5.centos.noarch
> > 2/400 - 1:kde-i18n-Icelandic-3.5.4-1.noarch
> > ....
> > ....
> > 328/400 - system-config-users-1.2.51-4.el5.noarch
> > 329/400 - yum-priorities-1.1.10-9.el5.centos.noarch
> > Traceback (most recent call last):
> > File "/usr/libexec/kojid/mergerepos", line 241, in <module>
> >  main(sys.argv[1:])
> > File "/usr/libexec/kojid/mergerepos", line 236, in main
> >  merge.write_metadata()
> > File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
> >  mdgen.doPkgMetadata()
> > File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
> > 332, in doPkgMetadata
> >  self.writeMetadataDocs(packages)
> > File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
> > 472, in writeMetadataDocs
> >  self.primaryfile.write(po.xml_dump_primary_metadata())
> > File "/usr/lib/python2.5/site-packages/yum/packages.py", line 943,
> > in xml_dump_primary_metadata
> >  msg += misc.to_unicode(self._dump_format_items())
> > File "/usr/lib/python2.5/site-packages/yum/packages.py", line 809,
> > in _dump_format_items
> >  msg += self._dump_pco('provides')
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
> > 38: ordinal not in range(128)
> >
> > Tried with a few other repos as well, e.g the i386 one , the two
> > merged , ... it fails the same
> > though apparently  at different packages.
>
> the metadata you're pulling in was written with a much older createrepo
> which was less picky about some of the data it wrote back out. So you end up
> with some pkgs with dodgy data in the changelogs.

Mirroring in CentOS5 and then recreating from scratch the repodirs with a Fedora
10 createrepo the the repository is created spitting out some warnings.

1556/2382 - CentOS/ghostscript-devel-8.15.2-9.1.el5_1.1.i386.rpm
iso-8859-1 encoding on /usr/lib/aspell-0.60/�slenska.alias
1598/2382 - CentOS/avalon-logkit-javadoc-1.2-4jpp.3.i386.rpm
iso-8859-1 encoding on Ville Skytt� <ville.skytta at iki.fi> - 1:0.2-1jpp

and then the merge fails with.

/usr/libexec/kojid/mergerepos -a i386  \
    -o repo  \
    -r http://skojihub.cern.ch/mirror/centos/5/updates/i386 \
    -r http://skojihub.cern.ch/mirror/centos/5/os/i386

Adding repo: http://skojihub.cern.ch/mirror/centos/5/updates/i386
 Adding repo: http://skojihub.cern.ch/mirror/centos/5/os/i386
 1/2607 - openssl-perl-0.9.8b-10.el5_2.1.i386
 2/2607 - 1:cups-1.2.4-11.18.el5_2.3.i386
 ....
1353/2607 - ImageMagick-c++-6.2.8.0-4.el5_1.1.i386
1354/2607 - mesa-libGL-6.5.1-7.5.el5.i386
Traceback (most recent call last):
  File "/usr/libexec/kojid/mergerepos", line 241, in <module>
    main(sys.argv[1:])
  File "/usr/libexec/kojid/mergerepos", line 236, in main
    merge.write_metadata()
  File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
    mdgen.doPkgMetadata()
  File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
332, in doPkgMetadata
    self.writeMetadataDocs(packages)
  File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line
472, in writeMetadataDocs
    self.primaryfile.write(po.xml_dump_primary_metadata())
  File "/usr/lib/python2.5/site-packages/yum/packages.py", line 943,
in xml_dump_primary_metadata
    msg += misc.to_unicode(self._dump_format_items())
  File "/usr/lib/python2.5/site-packages/yum/packages.py", line 809,
in _dump_format_items
    msg += self._dump_pco('provides')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
38: ordinal not in range(128)

Have read around a bit on .spec file character and there looks to be some
recommendations but nothing concrete? e.g
http://wiki.mandriva.com/en/Policies/Charset#In_spec_file_itself

Steve
> -sv
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Seth Vidal]

On Wed, 18 Mar 2009, Steve Traylen wrote:
> > the metadata you're pulling in was written with a much older createrepo
> > which was less picky about some of the data it wrote back out. So you end up
> > with some pkgs with dodgy data in the changelogs.
>
> Mirroring in CentOS5 and then recreating from scratch the repodirs with a Fedora
> 10 createrepo the the repository is created spitting out some warnings.

What ver of yum and createrepo do you have when you're running both 
createrepo and mergerepo?

-sv

[Steve Traylen]

On Wed, Mar 18, 2009 at 2:18 PM, Seth Vidal  wrote:
> On Wed, 18 Mar 2009, Steve Traylen wrote:
> > > the metadata you're pulling in was written with a much older createrepo
> > > which was less picky about some of the data it wrote back out. So you end
> > > up
> > > with some pkgs with dodgy data in the changelogs.
> >
> > Mirroring in CentOS5 and then recreating from scratch the repodirs with a
> > Fedora
> > 10 createrepo the the repository is created spitting out some warnings.
>
> What ver of yum and createrepo do you have when you're running both
> createrepo and mergerepo?

$ rpm -qf /usr/bin/yum \
              /usr/bin/mergerepo \
              /usr/bin/createrepo \
              /usr/libexec/kojid/mergerepos
yum-3.2.21-2.fc10.noarch
createrepo-0.9.6-3.fc10.noarch
createrepo-0.9.6-3.fc10.noarch
koji-builder-1.3.1-1.fc10.noarch
> -sv
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Steve Traylen]

On Wed, Mar 18, 2009 at 2:23 PM, Steve Traylen  wrote:
> On Wed, Mar 18, 2009 at 2:18 PM, Seth Vidal  wrote:
> > On Wed, 18 Mar 2009, Steve Traylen wrote:
> > > > the metadata you're pulling in was written with a much older createrepo
> > > > which was less picky about some of the data it wrote back out. So you end
> > > > up
> > > > with some pkgs with dodgy data in the changelogs.
> > >
> > > Mirroring in CentOS5 and then recreating from scratch the repodirs with a
> > > Fedora
> > > 10 createrepo the the repository is created spitting out some warnings.
> >
> > What ver of yum and createrepo do you have when you're running both
> > createrepo and mergerepo?
>
> $ rpm -qf /usr/bin/yum \
>              /usr/bin/mergerepo \
>              /usr/bin/createrepo \
>              /usr/libexec/kojid/mergerepos
> yum-3.2.21-2.fc10.noarch
> createrepo-0.9.6-3.fc10.noarch
> createrepo-0.9.6-3.fc10.noarch
> koji-builder-1.3.1-1.fc10.noarch

Hi Seth,

Is there any workaround or anything for this?

Steve
> > -sv
> >
> > --
> > Fedora-buildsys-list mailing list
> > [e-mail]
> > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
>
> --
> Steve Traylen

[Seth Vidal]

On Fri, 27 Mar 2009, Steve Traylen wrote:
> On Wed, Mar 18, 2009 at 2:23 PM, Steve Traylen  wrote:
> > On Wed, Mar 18, 2009 at 2:18 PM, Seth Vidal  wrote:
> > > On Wed, 18 Mar 2009, Steve Traylen wrote:
> > > > > the metadata you're pulling in was written with a much older createrepo
> > > > > which was less picky about some of the data it wrote back out. So you end
> > > > > up
> > > > > with some pkgs with dodgy data in the changelogs.
> > > >
> > > > Mirroring in CentOS5 and then recreating from scratch the repodirs with a
> > > > Fedora
> > > > 10 createrepo the the repository is created spitting out some warnings.
> > >
> > > What ver of yum and createrepo do you have when you're running both
> > > createrepo and mergerepo?
> >
> > $ rpm -qf /usr/bin/yum \
> >              /usr/bin/mergerepo \
> >              /usr/bin/createrepo \
> >              /usr/libexec/kojid/mergerepos
> > yum-3.2.21-2.fc10.noarch
> > createrepo-0.9.6-3.fc10.noarch
> > createrepo-0.9.6-3.fc10.noarch
> > koji-builder-1.3.1-1.fc10.noarch
>
> Hi Seth,
>
> Is there any workaround or anything for this?
>
> Steve

I doubt it'll help much - but try 0.9.7 and 3.2.22 that  are in rawhide. 
We fixed a few more unicode explosions but I'm sure we're still missing 
some

-sv

[Steve Traylen]

2009/3/27 Seth Vidal :
> On Fri, 27 Mar 2009, Steve Traylen wrote:
> > On Wed, Mar 18, 2009 at 2:23 PM, Steve Traylen  wrote:
> > > On Wed, Mar 18, 2009 at 2:18 PM, Seth Vidal 
> > > wrote:
> > > > On Wed, 18 Mar 2009, Steve Traylen wrote:
> > > > > > the metadata you're pulling in was written with a much older
> > > > > > createrepo
> > > > > > which was less picky about some of the data it wrote back out. So you
> > > > > > end
> > > > > > up
> > > > > > with some pkgs with dodgy data in the changelogs.
> > > > >
> > > > > Mirroring in CentOS5 and then recreating from scratch the repodirs with
> > > > > a
> > > > > Fedora
> > > > > 10 createrepo the the repository is created spitting out some warnings.
> > > >
> > > > What ver of yum and createrepo do you have when you're running both
> > > > createrepo and mergerepo?
> > >
> > > $ rpm -qf /usr/bin/yum \
> > >              /usr/bin/mergerepo \
> > >              /usr/bin/createrepo \
> > >              /usr/libexec/kojid/mergerepos
> > > yum-3.2.21-2.fc10.noarch
> > > createrepo-0.9.6-3.fc10.noarch
> > > createrepo-0.9.6-3.fc10.noarch
> > > koji-builder-1.3.1-1.fc10.noarch
> >
> > Hi Seth,
> >
> > Is there any workaround or anything for this?
> >
> > Steve
>
> I doubt it'll help much - but try 0.9.7 and 3.2.22 that  are in rawhide. We
> fixed a few more unicode explosions but I'm sure we're still missing some
>
> -sv

Same with rawhide.

Workaround , removing by hand the copyright symbol present in
primary.xml.
recorded here:
http://createrepo.baseurl.org/ticket/3
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list