Debugging booting problems with F9 on SD card...

[Martin Langhoff]

On the mission of getting a XS image running on the XO, I have abused
imagecreator (part of livecdtools) to create an ext3 image based on
the XS ISO we distribute.

Now I am trying to make the image boot on the XO, and it is not
working. It fails in trying to mount (and then switchroot to)
/dev/root. I suspect that it's the initrd mount process that is
failing there, but I am a bit at a loss on where to focus my debugging
next. OFW appears to read the initrd alright, and yet...

 - The whole image is i686 (while trying to convince imagecreator to
set the arch to i586) so that may be a source of problems.

 - The resulting ext3 image is using vanilla F9 kernel
2.6.27.19-78.2.30.fc9.686 -- no olpc-specific kernels here. From what
I understand, all the fixes applied to F10 initrds have also been
applied to recent F9 kernels.

 - I've created vmlinuz / initrd symlinks to make things easier -- at
least during debugging. Does anything in the kernel rpm %post set
those up or update them if they're found? Or update olpc.fth? Either
way, everytime a new kernel is installed I need to update
*something*...

 - I have also set /etc/fstab to mount /dev/mmcblk0p1 as /

 - My olpc.fth is built thus:
cat > /boot/olpc.fth << EOF
\ Boot script
" ro root=/dev/mmcblk0p1 rootfstype=ext3 rootdelay=5
console=ttyS0,115200 console=tty0 fbcon=font:SUN12x22" to boot-file
" sd:\boot\initrd" to ramdisk
" sd:\boot\vmlinuz" to boot-device
unfreeze
boot

EOF

 - Funny enough, unfreeze doesn't seem to work, and the boot fails
while consistently showing a pretty screen.

Hints welcome...

cheers,



m
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Server-devel] Debugging booting problems with F9 on SD card...

[Jerry Vonau]

I can download the image, or at least the initrd.img, from where? 

Jerry

On Mon, 2009-04-13 at 12:29 +0200, Martin Langhoff wrote:
>  On the mission of getting a XS image running on the XO, I have
> abused
>  imagecreator (part of livecdtools) to create an ext3 image based on
>  the XS ISO we distribute.
>  
>  Now I am trying to make the image boot on the XO, and it is not
>  working. It fails in trying to mount (and then switchroot to)
>  /dev/root. I suspect that it's the initrd mount process that is
>  failing there, but I am a bit at a loss on where to focus my debugging
>  next. OFW appears to read the initrd alright, and yet...
>  
>   - The whole image is i686 (while trying to convince imagecreator to
>  set the arch to i586) so that may be a source of problems.
>  
>   - The resulting ext3 image is using vanilla F9 kernel
>  2.6.27.19-78.2.30.fc9.686 -- no olpc-specific kernels here. From what
>  I understand, all the fixes applied to F10 initrds have also been
>  applied to recent F9 kernels.
>  
>   - I've created vmlinuz / initrd symlinks to make things easier -- at
>  least during debugging. Does anything in the kernel rpm %post set
>  those up or update them if they're found? Or update olpc.fth? Either
>  way, everytime a new kernel is installed I need to update
>  *something*...
>  
>   - I have also set /etc/fstab to mount /dev/mmcblk0p1 as /
>  
>   - My olpc.fth is built thus:
>  cat > /boot/olpc.fth << EOF
>  \ Boot script
>  " ro root=/dev/mmcblk0p1 rootfstype=ext3 rootdelay=5
>  console=ttyS0,115200 console=tty0 fbcon=font:SUN12x22" to boot-file
>  " sd:\boot\initrd" to ramdisk
>  " sd:\boot\vmlinuz" to boot-device
>  unfreeze
>  boot
>  
>  EOF
>  
>   - Funny enough, unfreeze doesn't seem to work, and the boot fails
>  while consistently showing a pretty screen.
>  
>  Hints welcome...
>  
>  cheers,
>  
>  
>  
>  m

[Server-devel] Debugging booting problems with F9 on SD card...

[Martin Langhoff]

On Mon, Apr 13, 2009 at 1:44 PM, Jerry Vonau <jvonau(a)shaw.ca&gt; wrote:
>  I can download the image, or at least the initrd.img, from where?
scp'ing to http://dev.laptop.org/~martin these 2 images (around 409MB size):

2438d2a520dd8b245f8f5e25e730e158853df34d  ks-200904110217.img.gz
6165f36d5885eabbec65accb3aa49458a67a8e63  ks-200904132252.img.gz

but the connection is slow, you might get better mileage by grabbing
the latest updates to the xs-livecd repository, and issuing:

 sudo ./util/mkext3img OLPC-School-Server-0.5.2-i386.iso
imgcreator/anaconda-f9.ks

Which will give you an ext3 img file.

The second image has i586 kernel (and 386 glibc and openssl). Both
need some manual steps...

 - edit /etc/fstab
 - symlink vmlinuz, initrd

cheers,



martin
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Alexander Boström]

Maybe something is missing from the initrd. The process of creating it 
is a bit of a mystery to me, but making sure /dev/root is set up right 
before generating the initrd seems to work. Try this:

Boot the XO into the regular Sugar system.
Insert SD card, open a root shell.
mount -o remount,dev /media/something
cd /media/something
Create a proper /dev/root:
stat /dev/mmcblk0p1 - stat outputs major/minor in hex
mknod b dev/root X Y - mknod wants decimal
stat /dev/mmcblk0p1 dev/root - make sure they're the same
mount --bind /sys sys
mount --bind /proc proc
mount --bind /dev/pts dev/pts
chroot .
new-kernel-pkg --package kernel --mkinitrd --depmod \
--install 2.6.27.19-78.2.30.fc9.686
exit
sync

And yes, it would be great if new-kernel-pkg could be made to update 
olpc.fth.

/abo

[Server-devel] Debugging booting problems with F9 on SD card...

[Jerry Vonau]

On Mon, 2009-04-13 at 14:44 +0200, Alexander Boström wrote:
>  Maybe something is missing from the initrd. The process of creating
> it 
>  is a bit of a mystery to me, but making sure /dev/root is set up right 
>  before generating the initrd seems to work. Try this:
>   
The initrd is missing the modules mmc_block and sdhci...
I'm going to try to create /etc/sysconfig/mkinitrd with the missing
modules listed and re-creating the initrd in the %post of the kickstart
file.

Jerry

[Server-devel] Debugging booting problems with F9 on SD card...

[Jerry Vonau]

On Mon, 2009-04-13 at 08:26 -0500, Jerry Vonau wrote:
>  On Mon, 2009-04-13 at 14:44 +0200, Alexander Boström wrote:
>  > Maybe something is missing from the initrd. The process of creating it 
>  > is a bit of a mystery to me, but making sure /dev/root is set up right 
>  > before generating the initrd seems to work. Try this:
>  > 
>  The initrd is missing the modules mmc_block and sdhci...
>  I'm going to try to create /etc/sysconfig/mkinitrd with the missing
>  modules listed and re-creating the initrd in the %post of the kickstart
>  file.
>  
>  Jerry 
So far I have added to %post:

cat > /etc/sysconfig/mkinitrd << EOF
MODULES="sdhci mmc_block"
EOF

cat > /etc/fstab.xs << EOF
/dev/mmcblk0p1  /         ext3    defaults,noatime 0 0
devpts     /dev/pts  devpts  gid=5,mode=620   0 0
tmpfs      /dev/shm  tmpfs   defaults         0 0
proc       /proc     proc    defaults         0 0
sysfs      /sys      sysfs   defaults         0 0
EOF

/sbin/mkinitrd --fstab=/etc/fstab.xs /boot/initrd.img `ls /lib/modules`

dd'ing the image to CF.. now for testing..

Jerry

[Server-devel] Debugging booting problems with F9 on SD card...

[Martin Langhoff]

On Mon, Apr 13, 2009 at 5:01 PM, Jerry Vonau <jvonau(a)shaw.ca&gt; wrote:
>  So far I have added to %post:
>
>  cat > /etc/sysconfig/mkinitrd << EOF
>  MODULES="sdhci mmc_block"
>  EOF 
I just tried a variation on that, and it didn't work. It does load
sdhci and mmc_block now, but it still fails. There's something else at
play...

cheers,



martin
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Server-devel] Debugging booting problems with F9 on SD card...

[Martin Langhoff]

On Tue, Apr 14, 2009 at 9:26 AM, Martin Langhoff
<martin.langhoff(a)gmail.com&gt; wrote:
>  I just tried a variation on that, and it didn't work. It does
> load
>  sdhci and mmc_block now, but it still fails. There's something else at
>  play... 
If I replace the kernel with an OLPC kernel, at least it tries to open
mmcblk0p1 -- and fails, because the drivers we need are built as
modules.

Have you had any success?

cheers,


martin
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Server-devel] Debugging booting problems with F9 on SD card...

[Martin Langhoff]

On Tue, Apr 14, 2009 at 2:24 PM, Martin Langhoff
<martin.langhoff(a)gmail.com&gt; wrote:
>  If I replace the kernel with an OLPC kernel, at least it tries to
> open
>  mmcblk0p1 -- and fails, because the drivers we need are built as
>  modules.
>
>  Have you had any success? 
Using dsd's XO-alt-distro scripts we can get a kernel build that has
all the relevant bits for the XO, and builds the MMC  modules we want
statically.

As they are, dsd's scripts setup an initrd that fails exactly like the
other ones. But as the modules we want are linked statically, we don't
need the initrd. And without referring to the initrd, the image boots
like a champ.

So now I have an XS booting on the XO :-) -- with a hand-compiled
kernel, so not quite the ideal thing yet.

The fact that merely adding an initrd breaks it means there must be
something wrong (read: non-XO-compatible) with how initrds are built
under F9.

I've reviewed the changelog of F-10s mkinitrd, but nothing jumps at
me, and it's not trivial to use it on F9 as it pulls a lots of deps.

Any ideas? Help?

cheers,



m
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Server-devel] Debugging booting problems with F9 on SD card...

[Jerry Vonau]

On Tue, 2009-04-14 at 20:45 +0200, Martin Langhoff wrote:
>  On Tue, Apr 14, 2009 at 2:24 PM, Martin Langhoff
>  <martin.langhoff(a)gmail.com&gt; wrote:
>  > If I replace the kernel with an OLPC kernel, at least it tries to open
>  > mmcblk0p1 -- and fails, because the drivers we need are built as
>  > modules.
>  >
>  > Have you had any success?
>   
not much really with F9..

>  Using dsd's XO-alt-distro scripts we can get a kernel build that
> has
>  all the relevant bits for the XO, and builds the MMC  modules we want
>  statically.
>   
What version of the kernel does/can it build? I've looked at scripts,
it's looking at a local source tree, can we rig it work a remote git? or
maybe use a source tree based on the source rpm layout? I wouldn't have
time till the weekend to look at any of that.  


>  As they are, dsd's scripts setup an initrd that fails exactly
> like the
>  other ones. But as the modules we want are linked statically, we don't
>  need the initrd. And without referring to the initrd, the image boots
>  like a champ.
>   
Try to move around some large files and see if the kernel crashes, see
below.

>  So now I have an XS booting on the XO :-) -- with a hand-compiled
>  kernel, so not quite the ideal thing yet.
>   
That would work if somebody maintains a kernel and source rpms (repo
maybe?) for an XO kernel. No, that is not ideal. 

>  The fact that merely adding an initrd breaks it means there must be
>  something wrong (read: non-XO-compatible) with how initrds are built
>  under F9.
>   
The livecd based stuff doesn't seem to have this issue, more like the
initrd needs a bit of a sleep while the module loads and finds the MMC
card, or as a module, it sort of broken(see below). 

I have F11's anaconda booting, formatting and installing to the MMC
card, only to have it kernel opps when installing the rpms.  

Thoughts anybody?

Jerry

[Peter Robinson]

Hi Martin,

What would it take to move the next XS Server release to either F10 or
F11 as F9 will be EOL as soon as F11 comes out in a month or so (one
month after F11 is out F9 will be EOL).

Peter

On Mon, Apr 13, 2009 at 11:29 AM, Martin Langhoff
<martin.langhoff(a)gmail.com&gt; wrote:
>  On the mission of getting a XS image running on the XO, I have
> abused
>  imagecreator (part of livecdtools) to create an ext3 image based on
>  the XS ISO we distribute.
>
>  Now I am trying to make the image boot on the XO, and it is not
>  working. It fails in trying to mount (and then switchroot to)
>  /dev/root. I suspect that it's the initrd mount process that is
>  failing there, but I am a bit at a loss on where to focus my debugging
>  next. OFW appears to read the initrd alright, and yet...
>
>   - The whole image is i686 (while trying to convince imagecreator to
>  set the arch to i586) so that may be a source of problems.
>
>   - The resulting ext3 image is using vanilla F9 kernel
>  2.6.27.19-78.2.30.fc9.686 -- no olpc-specific kernels here. From what
>  I understand, all the fixes applied to F10 initrds have also been
>  applied to recent F9 kernels.
>
>   - I've created vmlinuz / initrd symlinks to make things easier -- at
>  least during debugging. Does anything in the kernel rpm %post set
>  those up or update them if they're found? Or update olpc.fth? Either
>  way, everytime a new kernel is installed I need to update
>  *something*...
>
>   - I have also set /etc/fstab to mount /dev/mmcblk0p1 as /
>
>   - My olpc.fth is built thus:
>  cat > /boot/olpc.fth << EOF
>  \ Boot script
>  " ro root=/dev/mmcblk0p1 rootfstype=ext3 rootdelay=5
>  console=ttyS0,115200 console=tty0 fbcon=font:SUN12x22" to boot-file
>  " sd:\boot\initrd" to ramdisk
>  " sd:\boot\vmlinuz" to boot-device
>  unfreeze
>  boot
>
>  EOF
>
>   - Funny enough, unfreeze doesn't seem to work, and the boot fails
>  while consistently showing a pretty screen.
>
>  Hints welcome...
>
>  cheers,
>
>
>
>  m
>  --
>   martin.langhoff(a)gmail.com
>   martin(a)laptop.org -- School Server Architect
>   - ask interesting questions
>   - don't get distracted with shiny stuff  - working code first
>   - http://wiki.laptop.org/go/User:Martinlanghoff
>
>  _______________________________________________
>  Fedora-olpc-list mailing list
>  Fedora-olpc-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-olpc-list
>

[Martin Langhoff]

On Wed, Apr 15, 2009 at 11:08 AM, Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>  What would it take to move the next XS Server release to either F10
> or
>  F11 as F9 will be EOL as soon as F11 comes out in a month or so (one
>  month after F11 is out F9 will be EOL). 
It's a bit of a recurring question. I am swamped with feature
development for a while. My hope is to port to F11 or even F12 when I
can. Help is welcome and Jerry is doing a lot of work on F10/11.

But qualifiying something as an XS release is a sizable workload t is
to do all the testing to ensure it is stable, installs/upgrades and
works as desired. It takes quite a bit of effort and gear. And I want
to limit how many Fedora releases I have to support. (My plan is to
transition to a stable RHEL/Centos asap, hence my F11/F12 targets.)

But that for the future.

At this stage, people are needing to boot our F9 based XS-0.5.x and
upcoming 0.6, and that's what I am asking for help with.

Can you help me with that? :-) There's an odd issue with booting when
an initrd is involved.

cheers,


martin
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Peter Robinson]

> > What would it take to move the next XS Server release to either
> F10 or
> > F11 as F9 will be EOL as soon as F11 comes out in a month or so (one
> > month after F11 is out F9 will be EOL).
>
>  It's a bit of a recurring question. I am swamped with feature
>  development for a while. My hope is to port to F11 or even F12 when I
>  can. Help is welcome and Jerry is doing a lot of work on F10/11. 
Is there a current status of list and a todo list around somewhere, I
can help from a distro/packaging perspective but aren't much help from
development side of things

>  But qualifiying something as an XS release is a sizable workload t
> is
>  to do all the testing to ensure it is stable, installs/upgrades and
>  works as desired. It takes quite a bit of effort and gear. And I want
>  to limit how many Fedora releases I have to support. (My plan is to
>  transition to a stable RHEL/Centos asap, hence my F11/F12 targets.) 
Is the process documented somewhere?

>  But that for the future.
>
>  At this stage, people are needing to boot our F9 based XS-0.5.x and
>  upcoming 0.6, and that's what I am asking for help with.
>
>  Can you help me with that? :-) There's an odd issue with booting when
>  an initrd is involved. 
Not massively knowledgeable from a initrd perspective.

Peter

[Sameer Verma]

On Wed, Apr 15, 2009 at 2:41 AM, Martin Langhoff
<martin.langhoff(a)gmail.com&gt; wrote:
>  On Wed, Apr 15, 2009 at 11:08 AM, Peter Robinson
> <pbrobinson(a)gmail.com&gt; wrote:
> > What would it take to move the next XS Server release to either F10 or
> > F11 as F9 will be EOL as soon as F11 comes out in a month or so (one
> > month after F11 is out F9 will be EOL).
>
>  It's a bit of a recurring question. I am swamped with feature
>  development for a while. My hope is to port to F11 or even F12 when I
>  can. Help is welcome and Jerry is doing a lot of work on F10/11.
>
>  But qualifiying something as an XS release is a sizable workload t is
>  to do all the testing to ensure it is stable, installs/upgrades and
>  works as desired. It takes quite a bit of effort and gear. And I want
>  to limit how many Fedora releases I have to support. (My plan is to
>  transition to a stable RHEL/Centos asap, hence my F11/F12 targets.)
>  
+1

Fedora releases are too fast a moving target. I personally use Ubuntu
LTS or Debian Stable for my servers for this very reason.

Sameer

>  But that for the future.
>
>  At this stage, people are needing to boot our F9 based XS-0.5.x and
>  upcoming 0.6, and that's what I am asking for help with.
>
>  Can you help me with that? :-) There's an odd issue with booting when
>  an initrd is involved.
>
>  cheers,
>
>
>  martin
>  --
>   martin.langhoff(a)gmail.com
>   martin(a)laptop.org -- School Server Architect
>   - ask interesting questions
>   - don't get distracted with shiny stuff  - working code first
>   - http://wiki.laptop.org/go/User:Martinlanghoff
>  _______________________________________________
>  Devel mailing list
>  Devel(a)lists.laptop.org
>  http://lists.laptop.org/listinfo/devel

[Greg DeKoenigsberg]

On Wed, 15 Apr 2009, Sameer Verma wrote:

>  On Wed, Apr 15, 2009 at 2:41 AM, Martin Langhoff
>  <martin.langhoff(a)gmail.com&gt; wrote:
> > On Wed, Apr 15, 2009 at 11:08 AM, Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
> >> What would it take to move the next XS Server release to either F10 or
> >> F11 as F9 will be EOL as soon as F11 comes out in a month or so (one
> >> month after F11 is out F9 will be EOL).
> >
> > It's a bit of a recurring question. I am swamped with feature
> > development for a while. My hope is to port to F11 or even F12 when I
> > can. Help is welcome and Jerry is doing a lot of work on F10/11.
> >
> > But qualifiying something as an XS release is a sizable workload t is
> > to do all the testing to ensure it is stable, installs/upgrades and
> > works as desired. It takes quite a bit of effort and gear. And I want
> > to limit how many Fedora releases I have to support. (My plan is to
> > transition to a stable RHEL/Centos asap, hence my F11/F12 targets.)
> >
>
>  +1
>
>  Fedora releases are too fast a moving target. I personally use Ubuntu
>  LTS or Debian Stable for my servers for this very reason. 
Choose CentOS.  The workflow for managing development atop Centos and 
Fedora are essentially the same.

--g

--
Computer Science professors should be teaching open source.
Help make it happen.   Visit http://teachingopensource.org.

[Martin Langhoff]

On Wed, Apr 15, 2009 at 6:01 PM, Greg Dekoenigsberg <gdk(a)redhat.com&gt; wrote:
>  Choose CentOS.  The workflow for managing development atop Centos and
> Fedora
>  are essentially the same. 
That's the plan for the XS (I'm sure gregdek knows it, but it's good
to broadcast the msg) :-)

In the meantime, about that initrd brokenness... any takers?

cheers,


m
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Greg DeKoenigsberg]

On Wed, 15 Apr 2009, Martin Langhoff wrote:

>  On Wed, Apr 15, 2009 at 6:01 PM, Greg Dekoenigsberg
> <gdk(a)redhat.com&gt; wrote:
> > Choose CentOS.  The workflow for managing development atop Centos and Fedora
> > are essentially the same.
>
>  That's the plan for the XS (I'm sure gregdek knows it, but it's good
>  to broadcast the msg) :-)
>
>  In the meantime, about that initrd brokenness... any takers? 
Is there a bug I can herd people towards?

--g

--
Computer Science professors should be teaching open source.
Help make it happen.   Visit http://teachingopensource.org.

[Martin Langhoff]

On Wed, Apr 15, 2009 at 6:47 PM, Greg Dekoenigsberg <gdk(a)redhat.com&gt; wrote:
>  Is there a bug I can herd people towards? 
Not a bug as such, as I haven't narrowed down steps to repro, but
here's the start of the relevant threads:

http://lists.laptop.org/pipermail/server-devel/2009-April/003209.html
http://lists.laptop.org/pipermail/server-devel/2009-April/003225.html

If anyone has an XO and reasonable kernel + initrd debugging chops,
it's an interesting problem to tackle.

It is -- unapologetically -- a F9 thing. The current XS is F9, and
there's a push to make it usable on the XO hw for small pilots that
are starting soon (as "yesterday").

cheers,



m
-- 
 martin.langhoff(a)gmail.com
 martin(a)laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

[Peter Robinson]

>  Choose CentOS.  The workflow for managing development atop Centos and Fedora
>  are essentially the same. 
I have no doubt about it, and I've read that going forward that is the
plan for XS. The reason I originally asked the question about what we
need to move it forward to F11 was so that we can get any of the core
stuff that XS needs (if there is any that isn't there already) is
there in F11 so that the move to what will ultimately be centos 6 is
in place and it then makes it easier moving forward.

Peter