Troubles running F9 mock chroot under F11

[Daniel Drake]

Hi,

A number of difficulties/unfortunate circumstances are combining and
causing me a headache. I'm looking for help/ideas on getting around
these...

I am trying to build a customized version of the OLPC XS school server
for the OLPC deployment here in Nepal. The latest XS release is based
on F9. An F11/F12 update is on the cards, but the XS development team
is small and has more pressing priorities right now.

The internet connection here at the office is too slow to make local
builds, and also the power get turned off every night.

We do have a F11 box at the ISP which has a satisfactory internet
connection and reliable power, and we do have a speedy connection from
the office to that box. This box is also used to build other software
components for the deployment so there are multiple reasons why it
makes sense for us to run the school server build there.

The XS build system uses revisor and the upstream version is built on
a F9 box. My problems originate from having to build our customized
version from F11. We do not have the hardware or space to install a F9
box there, and we cannot downgrade our F11 system.

The first thing I tried is to use the F11 revisor to build the F9 XS
release. No luck - it fails on anaconda buildinstall due to big
differences in the F11 anaconda on the host system vs the F9 anaconda
in the target media. Not too surprising.

I looked into using pungi instead, but the documentation states:
"Pungi needs to run on the arch it is composing, as root, and with an
install of what it is composing, eg if you are composing Fedora 8, you
need to be running Fedora 8."

I then tried to create a F9 chroot using mock, with the intention of
running revisor or pungi inside. This doesn't work, because mock
creates a v9 berkeley DB inside the chroot, but the libraries/apps
inside the chroot only support bdb v8. So running "rpm -qa" inside a
fresh F9 chroot on F11 gives you these errors:
    mock-chroot> rpm -qa
    rpmdb: /var/lib/rpm/Packages: unsupported hash version: 9
    error: cannot open Packages index using db3 - Invalid argument (22)
    error: cannot open Packages database in /var/lib/rpm
And revisor and pungi fail in the same way, even  though the Pungi
docs suggest this kind of thing should be possible:
https://fedorahosted.org/pungi/wiki/PungiDocs/RunningPungiInMock

Finally I tried to use db_dump on the F11 host to dump the database
using the v9 tools, to go into the chroot and use db_load to import it
using the v8 tools, but this also results in a v9 database being
loaded :(

Any further ideas or suggestions?

Thanks,
Daniel

[Martin Langhoff]

On Tue, Sep 15, 2009 at 12:02 PM, Daniel Drake  wrote:
> I then tried to create a F9 chroot using mock, with the intention of
> running revisor or pungi inside. This doesn't work, because mock
> creates a v9 berkeley DB inside the chroot, but the libraries/apps
> inside the chroot only support bdb v8. So running "rpm -qa" inside a
> fresh F9 chroot on F11 gives you these errors:
>    mock-chroot> rpm -qa
>    rpmdb: /var/lib/rpm/Packages: unsupported hash version: 9
>    error: cannot open Packages index using db3 - Invalid argument (22)
>    error: cannot open Packages database in /var/lib/rpm

I keep my build machine of F9 due to similar issues I saw building F7
from F9 -- however, ISTR there's been some discussion of this
recently. Hmmm, a bit of googling leads to a nice thread

http://www.mail-archive.com/[e-mail]/msg02210.html

which if you read in depth seems to indicate that either of:

rm -f /var/lib/rpm/__db*
   /bin/rpm --rebuilddb

fixes the problem. Probably either triggers the other.

For obvious reasons I am interested in the results -- let me know if it works.

m

[Daniel Drake]

2009/9/15 Martin Langhoff :
> I keep my build machine of F9 due to similar issues I saw building F7
> from F9 -- however, ISTR there's been some discussion of this
> recently. Hmmm, a bit of googling leads to a nice thread
>
> http://www.mail-archive.com/[e-mail]/msg02210.html
>
> which if you read in depth seems to indicate that either of:
>
> rm -f /var/lib/rpm/__db*
>   /bin/rpm --rebuilddb
>
> fixes the problem. Probably either triggers the other.

Yeah, I found that thread too. It doesn't help. There are no files
that match __db* and "rpm --rebuilddb" complains with the same error.
Still stumped.

Daniel

[Server-devel] Troubles running F9 mock chroot under F11

[Jerry Vonau]

On Tue, 2009-09-15 at 15:47 +0545, Daniel Drake wrote:
> Hi,
>
> A number of difficulties/unfortunate circumstances are combining and
> causing me a headache. I'm looking for help/ideas on getting around
> these...
>
> I am trying to build a customized version of the OLPC XS school server
> for the OLPC deployment here in Nepal. The latest XS release is based
> on F9. An F11/F12 update is on the cards, but the XS development team
> is small and has more pressing priorities right now.

I have an F11 iso you could test ;)
> The internet connection here at the office is too slow to make local
> builds, and also the power get turned off every night.
>
> We do have a F11 box at the ISP which has a satisfactory internet
> connection and reliable power, and we do have a speedy connection from
> the office to that box. This box is also used to build other software
> components for the deployment so there are multiple reasons why it
> makes sense for us to run the school server build there.
>
> The XS build system uses revisor and the upstream version is built on
> a F9 box. My problems originate from having to build our customized
> version from F11. We do not have the hardware or space to install a F9
> box there, and we cannot downgrade our F11 system.

Are you just adding rpms to the install media? Or are you trying
something more difficult? I have a process in mind if you're just adding
rpms to the mix...
> The first thing I tried is to use the F11 revisor to build the F9 XS
> release. No luck - it fails on anaconda buildinstall due to big
> differences in the F11 anaconda on the host system vs the F9 anaconda
> in the target media. Not too surprising.

Revisor used to lie to buildinstall about the host environment, using
the version_from parameter, which just calls buildinstall based on your
target's version of Fedora (in /usr/lib/revisor/scripts) 
from: /usr/lib/python2.6/site-packages/revisor/pungi.py
        # setup the buildinstall call
        if os.access("scripts/%s-buildinstall" % self.cfg.version_from,
os.R_OK):
            buildinstall.extend([os.path.abspath("scripts/%
s-buildinstall" % self.cfg.version_from)])
        elif os.access("/usr/lib/revisor/scripts/%s-buildinstall" %
self.cfg.version_from, os.R_OK):
            buildinstall.extend(["/usr/lib/revisor/scripts/%
s-buildinstall" % self.cfg.version_from])
        else:

buildinstall.extend(['/usr/lib/anaconda-runtime/buildinstall'])
        #buildinstall.append('TMPDIR=%s' % self.workdir) # TMPDIR broken
in buildinstall

# FIXME: Determine options from the anaconda-runtime version
        if self.cfg.version_from in [ "F9", "F10", "F11", "DEVEL" ]:
            buildinstall.append('--debug')

However, I see that the older buildinstall(s) are not present any
more(?)! (File a bug I guess)  If you were to add the buildinstall from
F9's anaconda in revisor's script directory as F9-buildinstall, then the
buildinstall from F9 should be used instead of the one on the host
system.
> I looked into using pungi instead, but the documentation states:
> "Pungi needs to run on the arch it is composing, as root, and with an
> install of what it is composing, eg if you are composing Fedora 8, you
> need to be running Fedora 8."

Funny, how would you build a rawhide beta/preview release? ;) just
kidding...  I've been able to build a lesser version of Fedora than what
is running on the build host in the past, but it has been awhile since I
tried.
> I then tried to create a F9 chroot using mock, with the intention of
> running revisor or pungi inside. This doesn't work, because mock
> creates a v9 berkeley DB inside the chroot, but the libraries/apps
> inside the chroot only support bdb v8. So running "rpm -qa" inside a
> fresh F9 chroot on F11 gives you these errors:
>     mock-chroot> rpm -qa
>     rpmdb: /var/lib/rpm/Packages: unsupported hash version: 9
>     error: cannot open Packages index using db3 - Invalid argument (22)
>     error: cannot open Packages database in /var/lib/rpm
> And revisor and pungi fail in the same way, even  though the Pungi
> docs suggest this kind of thing should be possible:
> https://fedorahosted.org/pungi/wiki/PungiDocs/RunningPungiInMock

Just spinning up a test release using F11 as the host to build
XS-server-f9, using revisor as used in the Makefile at:
http://dev.laptop.org/git/projects/xs-livecd/tree/
> Finally I tried to use db_dump on the F11 host to dump the database
> using the v9 tools, to go into the chroot and use db_load to import it
> using the v8 tools, but this also results in a v9 database being
> loaded :(
>
> Any further ideas or suggestions?

Maybe, just need to know what your trying to do.

Jerry

[Server-devel] Troubles running F9 mock chroot under F11

[Daniel Drake]

2009/9/15 Jerry Vonau :
> Are you just adding rpms to the install media? Or are you trying
> something more difficult? I have a process in mind if you're just adding
> rpms to the mix...

Just adding RPMs would be enough, but also we're customizing the
kickstart file a little.
> However, I see that the older buildinstall(s) are not present any
> more(?)! (File a bug I guess)  If you were to add the buildinstall from
> F9's anaconda in revisor's script directory as F9-buildinstall, then the
> buildinstall from F9 should be used instead of the one on the host
> system.

I did that and it now fails at a later point. I first had to modify pungi.py
+        buildinstall.append('--output')
         buildinstall.append(self.topdir)

and the end result is:

Linking in release notes:
######################################## 100.0%
Size of the installation tree is 518 MB
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/revisor/__init__.py", line 528, in run
    self.base.run()
  File "/usr/lib/python2.6/site-packages/revisor/base.py", line 106, in run
    self.cli.run()
  File "/usr/lib/python2.6/site-packages/revisor/cli.py", line 44, in run
    self.base.lift_off()
  File "/usr/lib/python2.6/site-packages/revisor/base.py", line 867, in lift_off
    self.buildInstallationMedia()
  File "/usr/lib/python2.6/site-packages/revisor/base.py", line 1478,
in buildInstallationMedia
    f = open(os.path.join(mypungi.topdir,"isolinux","isolinux.cfg"),"rw+")
IOError: [Errno 2] No such file or directory:
'/var/tmp/revisor-pungi/0.5.2/xs-f9-i386/i386/os/isolinux/isolinux.cfg'
Traceback occurred, please report a bug at http://fedorahosted.org/revisor

The size should be more like 850mb.

Did you have any luck in your own experiment?

Daniel

[Server-devel] Troubles running F9 mock chroot under F11

[Jerry Vonau]

On Wed, 2009-09-16 at 10:32 +0545, Daniel Drake wrote:
> 2009/9/15 Jerry Vonau :
> > Are you just adding rpms to the install media? Or are you trying
> > something more difficult? I have a process in mind if you're just adding
> > rpms to the mix...
>
> Just adding RPMs would be enough, but also we're customizing the
> kickstart file a little.

That should be do-able using mkslim (read it first) from xs-livecd's git
repo, along with my idea to use a pre-configured "updates repo" on the
iso.

http://lists.laptop.org/pipermail/server-devel/2009-February/002937.html

You would create an "overlay tree" in lets say /tmp/xsupdates/, this
will hold what files you want to add/change on the iso. Now just make a
tree for the files in /tmp/xsupdates/. Create a directory updates,
populate it with rpms and run createrepo against it. If you wish to
replace/add a file on the iso, just have them be in the same place in
the xsupdates directory, as it would be on the iso. eg: xsupdates/ks.cfg
xsupdates/isolinux/isolinux.cfg. Then call

mkslim <path to iso> <output dir> /tmp/xsupdates

Remember to add the repo line to the kickstart file or add them as a
boot argument, for usb:
repo --name=updates --baseurl=file:///mnt/isodir/updates 
for cdrom:
repo --name=updates --baseurl=file:///mnt/stage2/updates

The usb method is tested, while I have not tested the cdrom iso
> > However, I see that the older buildinstall(s) are not present any
> > more(?)! (File a bug I guess)  If you were to add the buildinstall from
> > F9's anaconda in revisor's script directory as F9-buildinstall, then the
> > buildinstall from F9 should be used instead of the one on the host
> > system.
>
> I did that and it now fails at a later point. I first had to modify pungi.py
>    buildinstall.append('--output')
>    buildinstall.append(self.topdir)
>
> and the end result is:
>
> Linking in release notes:
> ######################################## 100.0%
> Size of the installation tree is 518 MB
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/revisor/__init__.py", line 528, in run
>     self.base.run()
>   File "/usr/lib/python2.6/site-packages/revisor/base.py", line 106, in run
>     self.cli.run()
>   File "/usr/lib/python2.6/site-packages/revisor/cli.py", line 44, in run
>     self.base.lift_off()
>   File "/usr/lib/python2.6/site-packages/revisor/base.py", line 867, in lift_off
>     self.buildInstallationMedia()
>   File "/usr/lib/python2.6/site-packages/revisor/base.py", line 1478,
> in buildInstallationMedia
>     f = open(os.path.join(mypungi.topdir,"isolinux","isolinux.cfg"),"rw+")
> IOError: [Errno 2] No such file or directory:
> '/var/tmp/revisor-pungi/0.5.2/xs-f9-i386/i386/os/isolinux/isolinux.cfg'
> Traceback occurred, please report a bug at http://fedorahosted.org/revisor
>
> The size should be more like 850mb.
>
> Did you have any luck in your own experiment?

No, I stopped when it bombed out, had to do my real work, must of been
at the point you got past with the patched pungi.py.

Jerry

[Server-devel] Troubles running F9 mock chroot under F11

[Daniel Drake]

2009/9/16 Jerry Vonau :
> That should be do-able using mkslim (read it first) from xs-livecd's git
> repo, along with my idea to use a pre-configured "updates repo" on the
> iso.
>
> http://lists.laptop.org/pipermail/server-devel/2009-February/002937.html

Thanks! Got it working as follows:
 1. extract ISO
 2. copy in new ks file
 3. add more RPMs to Packages/ (using creative use of yumdownloader to
make sure that deps come with the new RPMs)
 4. createrepo --database --groupfile repodata/comps.xml .
 5. remove stuff that mkslim removes
 6. mkisofs

no need to mess with bdb stuff any more :)

Daniel

[Server-devel] Troubles running F9 mock chroot under F11

[Martin Langhoff]

On Thu, Sep 17, 2009 at 1:47 PM, Daniel Drake  wrote:
> Thanks! Got it working as follows:
>  1. extract ISO
>  2. copy in new ks file
>  3. add more RPMs to Packages/ (using creative use of yumdownloader to
> make sure that deps come with the new RPMs)
>  4. createrepo --database --groupfile repodata/comps.xml .
>  5. remove stuff that mkslim removes
>  6. mkisofs
>
> no need to mess with bdb stuff any more :)

Good to hear it's worked!

Thinking about future support for that deployment and also for the XS
(as "upstream") -- is it in your plans to document what rpms and ks
changes you are using?

cheers,

m

[Server-devel] Troubles running F9 mock chroot under F11

[Jerry Vonau]

On Wed, 2009-09-16 at 12:13 -0500, Jerry Vonau wrote:
> On Wed, 2009-09-16 at 10:32 +0545, Daniel Drake wrote:
> > 2009/9/15 Jerry Vonau :
> > > However, I see that the older buildinstall(s) are not present any
> > > more(?)! (File a bug I guess)  If you were to add the buildinstall from
> > > F9's anaconda in revisor's script directory as F9-buildinstall, then the
> > > buildinstall from F9 should be used instead of the one on the host
> > > system.

Grab the source rpm for revisor, in the /scripts you'll find the
F9-buildinstall, use that one.
> > I did that and it now fails at a later point. I first had to modify pungi.py
> >    buildinstall.append('--output')
> >    buildinstall.append(self.topdir)
> >
> > and the end result is:

At what line did you add this? I got this to go with this:

@@ -322,6 +322,8 @@
         buildinstall.append(self.topdir)

if self.cfg.version_from in [ "F9" ]:
+            buildinstall.append('--output')
+            buildinstall.append(self.topdir)
             for burl in repository_baseurls:
                 buildinstall.append('%s' % burl)
> > Linking in release notes:
> > ######################################## 100.0%
> > Size of the installation tree is 518 MB
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.6/site-packages/revisor/__init__.py", line 528, in run
> >     self.base.run()
> >   File "/usr/lib/python2.6/site-packages/revisor/base.py", line 106, in run
> >     self.cli.run()
> >   File "/usr/lib/python2.6/site-packages/revisor/cli.py", line 44, in run
> >     self.base.lift_off()
> >   File "/usr/lib/python2.6/site-packages/revisor/base.py", line 867, in lift_off
> >     self.buildInstallationMedia()
> >   File "/usr/lib/python2.6/site-packages/revisor/base.py", line 1478,
> > in buildInstallationMedia
> >     f = open(os.path.join(mypungi.topdir,"isolinux","isolinux.cfg"),"rw+")
> > IOError: [Errno 2] No such file or directory:
> > '/var/tmp/revisor-pungi/0.5.2/xs-f9-i386/i386/os/isolinux/isolinux.cfg'
> > Traceback occurred, please report a bug at http://fedorahosted.org/revisor
> >
> > The size should be more like 850mb.
> >
> > Did you have any luck in your own experiment?
>
> No, I stopped when it bombed out, had to do my real work, must of been
> at the point you got past with the patched pungi.py.

Think that has to do with not having the F9-buildinstall from revisor,
the stock one didn't play nice for me either. I ended up 571 rpms on the
test iso. I haven't tried to test it yet..

Jerry

[Jan Kratochvil]

On Tue, 15 Sep 2009 12:02:38 +0200, Daniel Drake wrote:
> We do have a F11 box

[...]
> I then tried to create a F9 chroot using mock, with the intention of
> running revisor or pungi inside. This doesn't work, because mock
> creates a v9 berkeley DB inside the chroot, but the libraries/apps
> inside the chroot only support bdb v8. So running "rpm -qa" inside a
> fresh F9 chroot on F11 gives you these errors:
>     mock-chroot> rpm -qa
>     rpmdb: /var/lib/rpm/Packages: unsupported hash version: 9
>     error: cannot open Packages index using db3 - Invalid argument (22)
>     error: cannot open Packages database in /var/lib/rpm

There is a bug in mock it uses the host rpm to build the database instead of
the guest rpm.  In some minor version changes either
    `rm /var/lib/mock/XXX/root/var/lib/rpm/__db.*'
or
    `rpm -r /var/lib/mock/XXX/root --rebuilddb'
    (which also removes __db.*)
helps but in this F9<->F11 case already /var/lib/rpm/Packages is incompatible:
    /var/lib/mock/fedora-9-x86_64/root/var/lib/rpm/Packages: Berkeley DB (Hash, version 8, native byte-order)
vs.
    /var/lib/mock/fedora-9-x86_64/root/var/lib/rpm/Packages: Berkeley DB (Hash, version 9, native byte-order)

I understand it may be difficult for mock to bootstrap the chroot with the
guest rpm version - not existing at that time yet.

[...]
> Finally I tried to use db_dump on the F11 host to dump the database
> using the v9 tools, to go into the chroot and use db_load to import it
> using the v8 tools, but this also results in a v9 database being
> loaded :(

`db_load' on F9 is already too new for its rpm in use but compat-db works:

mock -r fedora-9-x86_64 --init
mock -r fedora-9-x86_64 --install compat-db
db_dump -f /var/lib/mock/fedora-9-x86_64/root/tmp/pkgdb /var/lib/mock/fedora-9-x86_64/root/var/lib/rpm/Packages 
rm /var/lib/mock/fedora-9-x86_64/root/var/lib/rpm/*
mock -r fedora-9-x86_64 --shell
    db45_load -f /tmp/pkgdb /var/lib/rpm/Packages
    rpm --rebuilddb
    rpm -q bash
    	bash-3.2-23.fc9.x86_64

While F9 is EOLed by Fedora I find this as a real problem running epel (both
epel-4 and epel-5) in mock on any recent Fedoras.

Another workaround would be to run the guest in kvm (or xen on older CPUs).

Regards,
Jan

[Jan Kratochvil]

Hi,

filed as:
    https://bugzilla.redhat.com/show_bug.cgi?id=523698

how to possibly fix the problem by a backport from rpm5.org as suggested by
Jeff Johnson.

Thanks,
Jan

[devzero2000]

On Wed, Sep 16, 2009 at 3:31 PM, Jan Kratochvil 
wrote:
> Hi,
>
> filed as:
>        https://bugzilla.redhat.com/show_bug.cgi?id=523698
>
> how to possibly fix the problem by a backport from rpm5.org as suggested

by
> Jeff Johnson.

For rpm 4.4 the backport was already filled but reject.

https://bugzilla.redhat.com/show_bug.cgi?id=464752
> Thanks,
> Jan
>
> --
> fedora-devel-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-devel-list

[Panu Matilainen]

On Wed, 16 Sep 2009, devzero2000 wrote:
> On Wed, Sep 16, 2009 at 3:31 PM, Jan Kratochvil
>  wrote:
> > Hi,
> >
> > filed as:
> >        https://bugzilla.redhat.com/show_bug.cgi?id=523698
> >
> > how to possibly fix the problem by a backport from rpm5.org as
>
> suggested by
> > Jeff Johnson.
>
> For rpm 4.4 the backport was already filled but reject.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=464752

Since you apparently missed the point: the WONTFIX is for that particular 
patch, not the issue itself. Also the bug is against RHEL 5, we're talking 
Fedora here.

The suggested patch is patently unsafe as it just blasts away any 
incompatible environment without caring whether the environment is 
actually in use by a different rpm version (which can happen in chroot 
environments and during upgrades) or not. It also blasts away the 
transaction lock while at it.

Citing BDB documentation: "The result of attempting to forcibly destroy 
the environment when it is in use is unspecified. Processes using an 
environment often maintain open file descriptors for shared regions within 
it. On UNIX systems, the environment removal will usually succeed, and 
processes that have already joined the region will continue to run in that 
region without change. However, processes attempting to join the 
environment will either fail or create new regions."

And yes, "funny stuff" happens when you got two different rpm/db versions 
accessing the same db. Feel free to try it out, you dont need to take my 
word for it.

That rpm leaves the environment around even when not in use, causing these 
silly issues is largely because the rpmdb open+close is racy. It's racy 
even with the environment present, just to a lesser degree so it usually 
gets away with it. Fixing the raciness will allow removing the environment 
when not in use, mostly curing the disease instead of treating the 
symptom. In addition to that, detecting and removing an incompatible (or 
corrupted) enviroment is a perfectly reasonable thing to do IF appropriate 
care is taken not to remove an active but incompatible environment.

- Panu -