[PATCH] mkinitrd rescue mode

[Jeff Layton]

I've opened a BZ ticket with a patch to add a rescue mode to the current
devel version of mkinitrd, but wanted to post a message about it here
since I know this is where most of the discussion occurs. The BZ ticket
with the patch is here:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=159287

Essentially this patch adds some functionality to mkinitrd and nash.
This adds some tools to the initrd images (busybox and fsck, in
particular), and adds a function to nash to allow running a shell before
the switchroot occurs.

This gives you some ability to troubleshoot booting problems, and gives
the ability to do some rescue-type work without having to boot to the
CD. This is a big plus for people that run boxes remotely and don't have
easy physical access to them.

The patch needs a little work,but a backported version basically worked
for me on RHEL4. I have not yet tested it on Fedora but it should
basically work there.

Essentially, I'm sending this to solicit some feedback on it. Anyone
have comments or concerns with this idea?

-- 
Jeffrey Layton <jlayton(a)redhat.com&gt;

[Jeremy Katz]

On Wed, 2005-06-01 at 15:16 -0400, Jeffrey Layton wrote:
>  Essentially this patch adds some functionality to mkinitrd and nash.
>  This adds some tools to the initrd images (busybox and fsck, in
>  particular), and adds a function to nash to allow running a shell before
>  the switchroot occurs. 
This will have a relatively significant impact on the size of the
initramfs and thus on the memory profile.  Although I see it's not
enabled by default, which is at least a plus  It also won't handle
filesystems other than ext3.

>  This gives you some ability to troubleshoot booting problems, and
> gives
>  the ability to do some rescue-type work without having to boot to the
>  CD. This is a big plus for people that run boxes remotely and don't have
>  easy physical access to them. 
PXE, copying the pxeboot vmlinuz/initrd into your grub.conf and a few
other things can give you a way to boot rescue mode without having to
put in a CD.  If you're having to run mkinitrd with different command
line arguments, then you've already lost if you've really hosed your
system badly.

Honestly, I'd really rather see improvements to rescue mode than
something like this.

Jeremy

[Jeff Layton]

On Thu, 2005-06-02 at 11:23 -0400, Jeremy Katz wrote:
>  This will have a relatively significant impact on the size of the
>  initramfs and thus on the memory profile.  Although I see it's not
>  enabled by default, which is at least a plus  It also won't handle
>  filesystems other than ext3.
>   
This is just a proof-of-concept sort of thing at this stage. A better
patch would detect the filesystem type of / and copy the fsck for that
partition here. Then if you had other filesystems you could mount / and
use those.

This does increase the memory profile of the initramfs by about 1.5M.
Isn't that memory freed after the switchroot occurs, though?

It's not enabled by default in this incarnation, but my preference would
be that it is in a later one.

>  > This gives you some ability to troubleshoot booting problems,
> and gives
>  > the ability to do some rescue-type work without having to boot to the
>  > CD. This is a big plus for people that run boxes remotely and don't have
>  > easy physical access to them.
>  
>  PXE, copying the pxeboot vmlinuz/initrd into your grub.conf and a few
>  other things can give you a way to boot rescue mode without having to
>  put in a CD.  If you're having to run mkinitrd with different command
>  line arguments, then you've already lost if you've really hosed your
>  system badly.
>  
>  Honestly, I'd really rather see improvements to rescue mode than
>  something like this. 
Both of those methods mean you'll be booting to a different kernel than
the one that is giving you problems. The main reason I rolled this patch
is to allow you to inspect (and possibly repair) the state of the system
when you're having problems booting. If you update to a new kernel, and
that kernel isn't detecting your drives correctly, then that is very
difficult to troubleshoot once you boot to a rescue kernel.

-- 
Jeffrey Layton <jlayton(a)redhat.com&gt;

[Jeremy Katz]

On Thu, 2005-06-02 at 13:06 -0400, Jeffrey Layton wrote:
>  On Thu, 2005-06-02 at 11:23 -0400, Jeremy Katz wrote:
>  > This will have a relatively significant impact on the size of the
>  > initramfs and thus on the memory profile.  Although I see it's not
>  > enabled by default, which is at least a plus  It also won't handle
>  > filesystems other than ext3.
>  > 
>  
>  This is just a proof-of-concept sort of thing at this stage. A better
>  patch would detect the filesystem type of / and copy the fsck for that
>  partition here. Then if you had other filesystems you could mount / and
>  use those. 
Note that only fsck.ext[23] are statically linked right now, which adds
some complications.

>  This does increase the memory profile of the initramfs by about
> 1.5M.
>  Isn't that memory freed after the switchroot occurs, though? 
The impression I've gotten is that the memory isn't freed with initramfs
since you can't actually unmount the original rootfs.

>  It's not enabled by default in this incarnation, but my
> preference would
>  be that it is in a later one. 
1.5 M additional overhead on all installs is less than ideal.

>  > > This gives you some ability to troubleshoot booting
> problems, and gives
>  > > the ability to do some rescue-type work without having to boot to the
>  > > CD. This is a big plus for people that run boxes remotely and don't have
>  > > easy physical access to them.
>  > 
>  > PXE, copying the pxeboot vmlinuz/initrd into your grub.conf and a few
>  > other things can give you a way to boot rescue mode without having to
>  > put in a CD.  If you're having to run mkinitrd with different command
>  > line arguments, then you've already lost if you've really hosed your
>  > system badly.
>  > 
>  > Honestly, I'd really rather see improvements to rescue mode than
>  > something like this.
>  
>  Both of those methods mean you'll be booting to a different kernel than
>  the one that is giving you problems. The main reason I rolled this patch
>  is to allow you to inspect (and possibly repair) the state of the system
>  when you're having problems booting. If you update to a new kernel, and
>  that kernel isn't detecting your drives correctly, then that is very
>  difficult to troubleshoot once you boot to a rescue kernel. 
If the kernel doesn't detect your drives correctly, then how exactly
does this let you "repair" things?  Most problems I've had are long
before the driver is going to screw things up.

Jeremy

[Bill Rugolsky Jr.]

Jeffrey,

This is a nice start at fulfilling a request I made more than a year ago
(that Jeremy dismissed):

   http://www.redhat.com/archives/fedora-devel-list/2004-January/msg00901.html

On Thu, Jun 02, 2005 at 04:17:22PM -0400, Jeremy Katz wrote:
>  > This does increase the memory profile of the initramfs by about
> 1.5M.
>  > Isn't that memory freed after the switchroot occurs, though?
>  
>  The impression I've gotten is that the memory isn't freed with initramfs
>  since you can't actually unmount the original rootfs. 
 
One is expected to delete files before doing the switchroot.  Unlike a ramdisk,
the ramfs pages are immediately reclaimed.

>  > Both of those methods mean you'll be booting to a different
> kernel than
>  > the one that is giving you problems. The main reason I rolled this patch
>  > is to allow you to inspect (and possibly repair) the state of the system
>  > when you're having problems booting. If you update to a new kernel, and
>  > that kernel isn't detecting your drives correctly, then that is very
>  > difficult to troubleshoot once you boot to a rescue kernel.
>  
>  If the kernel doesn't detect your drives correctly, then how exactly
>  does this let you "repair" things?  Most problems I've had are long
>  before the driver is going to screw things up. 
Potential uses:

o  Repairing a busted root logical volume (which can happen when snapshotting
   the root filesystem for example -- been there, done that).

o  Shrinking filesystems, including the root filesystem.
   The various resizers only allow for online growth.

Having a simple method to create an initramfs image that includes tools plus
their load dependencies (pulling in shared libraries, etc.), whatever the size,
would be a win.  One could decide what is useful locally and put it in a
special initramfs image, e.g.,

	/boot/initrd-$KVER-rescue.img

With a remote filesystem mount (nfs, cifs, etc.) one could have access to
a full set of tools with a simple "mount."

Booting with grub is a simple matter of including a stanza for rescue, or
editing the initrd line manually at the grub menu.

	Bill Rugolsky

[Jeff Layton]

>  On Thu, Jun 02, 2005 at 04:17:22PM -0400, Jeremy Katz wrote:
>  > > This does increase the memory profile of the initramfs by about 1.5M.
>  > > Isn't that memory freed after the switchroot occurs, though?
>  > 
>  > The impression I've gotten is that the memory isn't freed with initramfs
>  > since you can't actually unmount the original rootfs.
>   
>  One is expected to delete files before doing the switchroot.  Unlike a ramdisk,
>  the ramfs pages are immediately reclaimed.
>   
That shouldn't be too hard to handle then -- just delete busybox and
fsck* before the switchroot. Even better would be if there were some way
to get at the initramfs (is it mountable?) after the switchroot and
clean the thing out entirely.

>  > If the kernel doesn't detect your drives correctly, then how
> exactly
>  > does this let you "repair" things?  Most problems I've had are long
>  > before the driver is going to screw things up.
>   
It won't let you repair anything in that particular case, but it at
least would let you determine what the problem is. That's often very
difficult. Being able to go into rescue mode and then from there being
able to do:

dmesg | more

would be be very helpful.

>  Potential uses:
>  
>  o  Repairing a busted root logical volume (which can happen when snapshotting
>     the root filesystem for example -- been there, done that).
>  
>  o  Shrinking filesystems, including the root filesystem.
>     The various resizers only allow for online growth.
>  
>  Having a simple method to create an initramfs image that includes tools plus
>  their load dependencies (pulling in shared libraries, etc.), whatever the size,
>  would be a win.  One could decide what is useful locally and put it in a
>  special initramfs image, e.g.,
>  
>  	/boot/initrd-$KVER-rescue.img
>  
>  With a remote filesystem mount (nfs, cifs, etc.) one could have access to
>  a full set of tools with a simple "mount."
>  
>  Booting with grub is a simple matter of including a stanza for rescue, or
>  editing the initrd line manually at the grub menu.
>   
Bill makes a good point here. This doesn't have to be something used on
every boot (though if we can clean out the initramfs then I don't see
the harm). We could potentially generate 2 initrd's -- one "normal" and
one "rescue", and then have separate grub entries for each.

If we want to go with this, it might be best to think about just
ditching nash and moving to using busybox instead. Most of the functions
that nash does could be handled by the script, and other stuff we'd need
(like switchroot) could be added to it.

-- Jeff

[Jeff Layton]

On Thu, 2005-06-02 at 17:04 -0400, Bill Rugolsky Jr. wrote:
>   
>  One is expected to delete files before doing the switchroot.  Unlike a ramdisk,
>  the ramfs pages are immediately reclaimed. 
Ok, I've whipped up a new patch. This adds a simple "unlink" function to
nash. The mkinitrd script now does an "rm /bin/fsck*" (using busybox's
rm), and then a "unlink /bin/busybox" after the rescue mode and prior to
the switchroot. That should free up any significant extra memory we
consume by adding these tools to the initramfs.

The new patch is attached to the BZ ticket:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=159287

I've done some looking around at whether there is some way to get at the
initramfs after switchrooting (or whether there is some mechanism to
cause the kernel to just free it), but haven't found any way to do this
yet.

The early-userspace docs seem to state that you can only have a single
cpio archive, but that multiple 'images' are allowed. I'm not clear at
this point on what these images are and how they're stored, however.

-- Jeff

[Jeff Layton]

On second thought, I changed the mkinitrd script to just use unlink to
remove the files individually. I wasn't sure how nash would handle
wildcards, and this should be slightly faster.

See BZ case for updated patch.

-- Jeff

[Bill Rugolsky Jr.]

On Sat, Jun 04, 2005 at 08:50:44AM -0400, Jeff Layton wrote:
>  The early-userspace docs seem to state that you can only have a
> single
>  cpio archive, but that multiple 'images' are allowed. I'm not clear at
>  this point on what these images are and how they're stored, however. 
 
I was referring explicitly to this:

   In human terms, the initramfs buffer contains a collection of
   compressed and/or uncompressed cpio archives (in the "newc" or
"crc"
   formats); arbitrary amounts zero bytes (for padding) can be added
   between members.

Back in 2001, Dave Cinege proposed the following patch for GRUB:

   http://lists.gnu.org/archive/html/bug-grub/2001-06/msg00141.html

   This patch fixes GRUB's initrd loading behavior. Each 'initrd' call is
   now loaded 'inline' instead of replacing the previous file.
   This allows one to load an initrd image that is split across several
   devices (or floppies, using 'pause'). It also allows kernel patches
   like initrd_dynamic to load multiple tgz archives to construct a
   root.
   ...

Some version of this appears to have been incorporated into GRUB.  In
any case, with an appropriately patched boot loader, one could
specify two initrd load lines, and they will be concatenated together.

There ought to be a base initrd with hooks that would automatically pick
up add-on functionality just by placing files into the appropriate ".d"
directories, just as Red Hat has wisely done all over the rest of the
distribution.

A sample GRUB stanza might look like:

title Fedora Core Rescue Mode (2.6.11-1.1369_FC4)
        kernel /vmlinuz-2.6.11-1.1369_FC4 ro root=LABEL=/ acpi=on
	initrd /initrd-boot.img
        initrd /initrd-2.6.11-1.1369_FC4.img
	initrd /initrd-rescue.img

My preference would be to place add-on functionality in
a directory under sysconfig, e.g., /etc/sysconfig/boot.d/*,
so that rpms for EVMS, net booting, ... , could simply
drop files there.


	-Bill

[Peter Jones]

On Mon, 2005-06-06 at 09:39 -0400, Bill Rugolsky Jr. wrote:

>  My preference would be to place add-on functionality in
>  a directory under sysconfig, e.g., /etc/sysconfig/boot.d/*,
>  so that rpms for EVMS, net booting, ... , could simply
>  drop files there. 
/etc/sysconfig is a total no-go there.  Something like /boot/initrd.d/
might work...

-- 
        Peter

[Peter Jones]

On Sat, 2005-06-04 at 08:50 -0400, Jeff Layton wrote:

>  Ok, I've whipped up a new patch. This adds a simple
> "unlink" function to
>  nash. The mkinitrd script now does an "rm /bin/fsck*" (using busybox's
>  rm), and then a "unlink /bin/busybox" after the rescue mode and prior to
>  the switchroot. That should free up any significant extra memory we
>  consume by adding these tools to the initramfs. 
That's really not the right solution, or rather not the whole solution,
for the memory problem.  See the patch and discussion in bz 153069 .

>  I've done some looking around at whether there is some way to get
> at the
>  initramfs after switchrooting (or whether there is some mechanism to
>  cause the kernel to just free it), but haven't found any way to do this
>  yet. 
You could bind mount it at another location before the MS_MOVE mount
(inside of "switchroot" itself).

-- 
        Peter

[Jeff Layton]

On Mon, 2005-06-06 at 14:52 -0400, Peter Jones wrote:

>  That's really not the right solution, or rather not the whole
> solution,
>  for the memory problem.  See the patch and discussion in bz 153069 . 
Agreed -- I've opened a new BZ today and posted a proposed patch to
recursively remove all files and directories under a given directory
(namely '/'), without crossing mountpoints. I didn't realize there was
already a BZ open on this, so please let me know if you'd like me to
close 159636, and post my patch to 153069.

Though, the more I think about it, I like your second suggestion of
bind-mounting initramfs inside the switchroot routine. Then we could
just use regular coreutils 'rm' to clean it out and unmount it fairly
early in the boot process.

That would also potentially give you the option to leave it intact for
troubleshooting problems inside it after switchrooting (if that were
ever needed). It also keeps us from having to hack a lot of new stuff
into nash.

Maybe if a /initrd directory exists on the "real" root, we could bind-
mount it there and do the cleanup?

-- 
Jeff Layton <jlayton(a)redhat.com&gt;

[Peter Jones]

On Mon, 2005-06-06 at 15:28 -0400, Jeff Layton wrote:
>  Agreed -- I've opened a new BZ today and posted a proposed patch
> to
>  recursively remove all files and directories under a given directory
>  (namely '/'), without crossing mountpoints. I didn't realize there was
>  already a BZ open on this, so please let me know if you'd like me to
>  close 159636, and post my patch to 153069. 
Already done, thanks.

>  Though, the more I think about it, I like your second suggestion of
>  bind-mounting initramfs inside the switchroot routine. Then we could
>  just use regular coreutils 'rm' to clean it out and unmount it fairly
>  early in the boot process. 
I actually like the recursiveRemove way far more, because it means
"init" is still the only thing we rely on from the mounted fs.  The
result is much more flexibility for the installed system.

>  That would also potentially give you the option to leave it intact
> for
>  troubleshooting problems inside it after switchrooting (if that were
>  ever needed). It also keeps us from having to hack a lot of new stuff
>  into nash. 
I'd rather see "call another initramfs script" stuff I mentioned in my
other email...

>  Maybe if a /initrd directory exists on the "real" root, we
> could bind-
>  mount it there and do the cleanup? 
Ew, please no.  Users already erroneously complain about, file bugs
regarding, etc. /initrd way too much, it's better just not to have it if
we can... (and it was recently removed, as well)

-- 
        Peter

[Peter Jones]

On Thu, 2005-06-02 at 13:06 -0400, Jeffrey Layton wrote:

>  This does increase the memory profile of the initramfs by about
> 1.5M.
>  Isn't that memory freed after the switchroot occurs, though? 
It isn't currently freed.  There's a plan for how to free it within the
switchroot command, but I haven't had the time to implement it.

>  If you update to a new kernel, and that kernel isn't detecting
> your 
>  drives correctly, then that is very difficult to troubleshoot once you
>  boot to a rescue kernel. 
How often is that a real problem?  I don't think I've seen any
significant number of bug reports on this in recent memory.
-- 
        Peter

[Jeff Layton]

On Mon, 2005-06-06 at 14:06 -0400, Peter Jones wrote:
>  How often is that a real problem?  I don't think I've seen
> any
>  significant number of bug reports on this in recent memory. 
That's just a "for instance". Most of them aren't actual bugs, and so
you won't see BZ tickets on them.

Problems booting after kernel upgrades are common, however. We deal with
them on a daily basis in production support. The problem for us is that
there is virtually no simple way to troubleshoot the case where the
rootfs isn't getting mounted for some reason, the switchroot fails, and
the kernel panics. The most helpful messages telling us what the problem
is usually have scrolled off the screen. We generally have to fix this
based on experience and guesswork.

You can have the user set up a serial console, but users who are savvy
enough to figure out how to do that are generally able to troubleshoot
their own booting problems. Being able to tell a user to add "rescue" to
the command line and then to walk them through some commands (like
dmesg) to try to determine the problem would be very helpful for a
number of different reasons. It would also give people the ability to
try to rescue corrupted root filesystems without needing special
infrastructure (like a PXE server) and without having to physically be
near the machine (with a CD boot).

Since we're discussing this, I posted a proposed patch this morning to
nash to clean out the initramfs prior to the switchroot:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=159636

If nash is able to clean out the initramfs before switching the root, is
there any reason _not_ to have some useful tools in it?

-- 
Jeffrey Layton <jlayton(a)redhat.com&gt;

[Peter Jones]

On Mon, 2005-06-06 at 14:27 -0400, Jeffrey Layton wrote:
>  On Mon, 2005-06-06 at 14:06 -0400, Peter Jones wrote:
>  > How often is that a real problem?  I don't think I've seen any
>  > significant number of bug reports on this in recent memory.
>  
>  That's just a "for instance". Most of them aren't actual bugs, and so
>  you won't see BZ tickets on them. 
I don't buy that at all.  If, as you state below, we're seeing lots of
cases where machines don't boot after upgrades, then there's a bug in
something.  Period.

>  Problems booting after kernel upgrades are common, however. We deal
> with
>  them on a daily basis in production support. The problem for us is that
>  there is virtually no simple way to troubleshoot the case where the
>  rootfs isn't getting mounted for some reason, the switchroot fails, and
>  the kernel panics. 
And if we're getting those, then there *is* a real mkinitrd bug, or a
real module_upgrade bug, or a real new-kernel-pkg bug.  If Engineering
isn't seeing BZ tickets on them, then Engineering and GPS both lose.  No
new nash/mkinitrd features will help that at all.

>  The most helpful messages telling us what the problem is usually have
>
>  scrolled off the screen. We generally have to fix this based on experience
>  and guesswork. 
One (much more acceptable) solution for that would be a much simpler
switchroot change -- make it look for a command line option "pause", and
if it finds it, wait for the user to hit "enter" before executing init.
For the overwhelmingly vast majority of boxes that'll get you enough
scrollback in shift+pageup to see everything since kernel started.

>  You can have the user set up a serial console, but users who are
> savvy
>  enough to figure out how to do that are generally able to troubleshoot
>  their own booting problems.  Being able to tell a user to add "rescue" 
>  to the command line and then to walk them through some commands (like 
>  dmesg) to try to determine the problem would be very helpful for a
>  number of different reasons. 
This is something of a contradiction -- you've just said the user isn't
savvy enough to debug boot problems, and then suggested a step that'd
make them only very marginally easier.  

>  It would also give people the ability to try to rescue corrupted
> root
>  filesystems without needing special infrastructure (like a PXE server) 
>  and without having to physically be near the machine (with a CD boot). 
This is a strawman -- your scenario is that they've just installed or
upgraded, in which case they've already set up this infrastructure or
are already close to the box.

>  Since we're discussing this, I posted a proposed patch this
> morning to
>  nash to clean out the initramfs prior to the switchroot:
>  
>  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=159636 
Thank you for doing this.

>  If nash is able to clean out the initramfs before switching the root,
> is
>  there any reason _not_ to have some useful tools in it? 
Added complexity is bad.  Sometimes we have to add some, and that sucks.
When we don't have to, in general the answer is "no".

So if you *really* think this is worth doing, I'm more likely to take a
change to add a command line argument which causes nash to execute
something on a second initramfs cpio ball in lieu of switchroot/init,
and then an entirely separate image (unrelated to mkinitrd) to do your
rescue stuff.

-- 
        Peter

[Jeff Layton]

On Mon, 2005-06-06 at 15:31 -0400, Peter Jones wrote:
>  And if we're getting those, then there *is* a real mkinitrd bug,
> or a
>  real module_upgrade bug, or a real new-kernel-pkg bug.  If Engineering
>  isn't seeing BZ tickets on them, then Engineering and GPS both lose.  No
>  new nash/mkinitrd features will help that at all.
>   
These problems are often configuration issues, or someone has removed
the /initrd directory (not such a problem with initramfs, but very
common with initrd problems).

On occasion this is a kernel issue, where a device is no longer being
detected correctly for some reason. In these cases, we _do_ open BZ
tickets and engineering gets them. Unfortunately, we it's often after a
lot of time working with the user before we can determine that this is
the case. Making this easier to troubleshoot would be helpful for users
and us.

It's tough to simply categorize all of the instances where we see
problems with booting. If I could then we could just eliminate them,
this is an attempt to get better visibility into an area where we are
generally flying blind.

>  > The most helpful messages telling us what the problem is usually
> have 
>  > scrolled off the screen. We generally have to fix this based on experience
>  > and guesswork.
>  
>  One (much more acceptable) solution for that would be a much simpler
>  switchroot change -- make it look for a command line option "pause", and
>  if it finds it, wait for the user to hit "enter" before executing init.
>  For the overwhelmingly vast majority of boxes that'll get you enough
>  scrollback in shift+pageup to see everything since kernel started.
>   
I'm open to this. I don't think it would be as helpful as a true rescue
mode, but it's better than nothing.

>  > You can have the user set up a serial console, but users who are
> savvy
>  > enough to figure out how to do that are generally able to troubleshoot
>  > their own booting problems.  Being able to tell a user to add "rescue" 
>  > to the command line and then to walk them through some commands (like 
>  > dmesg) to try to determine the problem would be very helpful for a
>  > number of different reasons.
>  
>  This is something of a contradiction -- you've just said the user isn't
>  savvy enough to debug boot problems, and then suggested a step that'd
>  make them only very marginally easier.   
True, but this gives us _some_ ability to see what the problem is. Being
able to tell them (for instance): Ok run "cat /proc/mounts" and tell me
if /dev/hda2 is listed would be very helpful. This tool wouldn't be
useful for users with no clue on their own, but would be helpful for
helping such users over the phone.

>  > It would also give people the ability to try to rescue corrupted
> root
>  > filesystems without needing special infrastructure (like a PXE server) 
>  > and without having to physically be near the machine (with a CD boot).
>  
>  This is a strawman -- your scenario is that they've just installed or
>  upgraded, in which case they've already set up this infrastructure or
>  are already close to the box.
>   
Not necessarily -- serial consoles are very common in datacenters.

>  > Since we're discussing this, I posted a proposed patch this
> morning to
>  > nash to clean out the initramfs prior to the switchroot:
>  > 
>  > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=159636
>  
>  Thank you for doing this.
>   
Gladly :-)

>  > If nash is able to clean out the initramfs before switching the
> root, is
>  > there any reason _not_ to have some useful tools in it?
>  
>  Added complexity is bad.  Sometimes we have to add some, and that sucks.
>  When we don't have to, in general the answer is "no".
>  
>  So if you *really* think this is worth doing, I'm more likely to take a
>  change to add a command line argument which causes nash to execute
>  something on a second initramfs cpio ball in lieu of switchroot/init,
>  and then an entirely separate image (unrelated to mkinitrd) to do your
>  rescue stuff.
>   
I'm with you -- extra complexity is generally bad, but in this case I
don't see where it's harmful. If you don't want to use it, don't add
"rescue" to the commandline (or generate your initrd images without
it). 

To make sure I understand what you're proposing as an alternative...

You're proposing having a secondary cpio image containing the "rescue"
tools. We'd then pass this as a secondary initrd image to GRUB. We could
then either use the rescue command line parameter (more or less as-is),
or could key off the presence of something from the rescue image to
enter rescue mode.

If so, that would be acceptable to me, but I'll have to see how multiple
cpio archives work in practice (I've never used them).

-- 
Jeff Layton <jlayton(a)redhat.com&gt;

[Peter Jones]

On Mon, 2005-06-06 at 15:56 -0400, Jeff Layton wrote:

>  > > It would also give people the ability to try to rescue
> corrupted root
>  > > filesystems without needing special infrastructure (like a PXE server) 
>  > > and without having to physically be near the machine (with a CD boot).
>  > 
>  > This is a strawman -- your scenario is that they've just installed or
>  > upgraded, in which case they've already set up this infrastructure or
>  > are already close to the box.
>  > 
>  
>  Not necessarily -- serial consoles are very common in datacenters. 
How does that matter?  They've still either got CDs and are standing in
front of the box, or they've set up the infrastructure for netboot and
whatnot.

>  I'm with you -- extra complexity is generally bad, but in this
> case I
>  don't see where it's harmful. If you don't want to use it, don't add
>  "rescue" to the commandline (or generate your initrd images without
>  it).  
I just don't think there's a significant amount of benefit.  Using the
rescue image just isn't a very difficult task.

>  To make sure I understand what you're proposing as an
> alternative...
>  
>  You're proposing having a secondary cpio image containing the "rescue"
>  tools. We'd then pass this as a secondary initrd image to GRUB. We could
>  then either use the rescue command line parameter (more or less as-is),
>  or could key off the presence of something from the rescue image to
>  enter rescue mode. 
Right.  And I like the second way better -- because once you've decided
you sometimes need to do debugging stuff, you can add the second initrd
and just keep on using it.  Then ignore it until you need it again.

>  If so, that would be acceptable to me, but I'll have to see how
> multiple
>  cpio archives work in practice (I've never used them). 
Me neither -- let me know how it works for you ;)
-- 
        Peter

[Bill Rugolsky Jr.]

On Mon, Jun 06, 2005 at 03:31:42PM -0400, Peter Jones wrote:
>  > It would also give people the ability to try to rescue corrupted
> root
>  > filesystems without needing special infrastructure (like a PXE server) 
>  > and without having to physically be near the machine (with a CD boot).
>  
>  This is a strawman -- your scenario is that they've just installed or
>  upgraded, in which case they've already set up this infrastructure or
>  are already close to the box. 
I don't understand why you think that everyone should be running a PXE
server, which can be a security nightmare as well as an administrative
hassle.

The following is a real-world, though perhaps ill-advised, configuration:

 o /boot on RAID1 across several /dev/sd?1
 o  swap on RAID1 across several other /dev/sd?1
 o Everything else on LVM2-over-RAID5 on /dev/sd?2

[As disks have gotten larger and cheaper, putting the root file system on
 RAID5 has become less attractive, but people still do it.]

A disk failure followed by a crash/outage requires manual intervention to
bring the RAID5 back online, since we do not have RAID5 journaling yet,
and it is still early days for RAID6.  Getting to a command prompt and
running mdadm to reassemble the array would certainly be quite useful.

	-Bill

[Peter Jones]

On Mon, 2005-06-06 at 16:59 -0400, Bill Rugolsky Jr. wrote:
>  On Mon, Jun 06, 2005 at 03:31:42PM -0400, Peter Jones wrote:
>  > > It would also give people the ability to try to rescue corrupted root
>  > > filesystems without needing special infrastructure (like a PXE server) 
>  > > and without having to physically be near the machine (with a CD boot).
>  > 
>  > This is a strawman -- your scenario is that they've just installed or
>  > upgraded, in which case they've already set up this infrastructure or
>  > are already close to the box.
>  
>  I don't understand why you think that everyone should be running a PXE
>  server, which can be a security nightmare as well as an administrative
>  hassle. 
I don't think that at all.  In the scenario we were discussing, the
system had just been installed or upgraded.  In that case, I'm 100% sure
you have things set up to boot either a CD, a PXE image, or some other
boot media that you can just as easily put the rescue image on.

>  The following is a real-world, though perhaps ill-advised,
> configuration:
>  
>   o /boot on RAID1 across several /dev/sd?1
>   o  swap on RAID1 across several other /dev/sd?1
>   o Everything else on LVM2-over-RAID5 on /dev/sd?2
>  
>  [As disks have gotten larger and cheaper, putting the root file system on
>   RAID5 has become less attractive, but people still do it.]
>  
>  A disk failure followed by a crash/outage requires manual intervention to
>  bring the RAID5 back online, since we do not have RAID5 journaling yet,
>  and it is still early days for RAID6.  Getting to a command prompt and
>  running mdadm to reassemble the array would certainly be quite useful. 
In this scenario you're replacing a drive.  I hope you've got physical
access to the machine.

Also, this isn't a very good example -- your random drive failure is
relatively likely to wedge the bus, etc.  In that case, this is the
exact same scenario as if /boot isn't available at all.

You have to go and touch the box here.  If you have to touch the box,
then I don't buy "I couldn't boot a CD".
-- 
        Peter

[PATCH] mkinitrd rescue mode - now crypto

[Michael H. Warfield]

Ok...

	Time for a variation on a theme.  I'm also in the middle of doing some
similar type stuff in initrd.  Modified the subject to reflect a change
in direction

On Mon, 2005-06-06 at 14:06 -0400, Peter Jones wrote:
>  On Thu, 2005-06-02 at 13:06 -0400, Jeffrey Layton wrote:
>  
>  > This does increase the memory profile of the initramfs by about 1.5M.
>  > Isn't that memory freed after the switchroot occurs, though? 
>  It isn't currently freed.  There's a plan for how to free it
> within the
>  switchroot command, but I haven't had the time to implement it. 
	Ouch!  I thought the kernel was suppose to be doing this already.
Wasn't that what the big discussion was on the kernel mailing list about
"Double free of initramfs" back in March?  So, if we add stuff to the
initrd, we just have to delete it all before calling switchroot?  Not
freeing that memory seems like a bug.

	I'm using this for a more sophisticated scenario.  Basically, I'm using
it to run with my laptop hard drives totally encrypted, including the
root and swap partitions, through dm_crypt and booted from a USB
pendrive/key/flash/whatever (possibly CD-Rom as well).  To accomplish
that, I needed to add cryptsetup and busybox to the initrd (plus any
modules for USB for any USB devices like USB keyboards and SCSI if I
want to mount the flash while in initrd space).  Busybox needed to be
rebuilt with a few different options, like adding losetup (so I can
decrypt a luks encrypted keyblob image) and switching from msh to ash
(msh is borked - gag).  Then just add a single line hook into the "init"
nash script to run "crytpsetup.sh" which was also added into bin (and
cryptsetup.sh has an optional drop to shell for doing maintenance on the
crypto from initrd).

	Only reason I seem to need nash at all, right now, is for the
"makerootdev" command and the "switchroot" command (busybox has
pivot_root, but not switchroot, and I can ignore the "setquiet" noise).
With those two functions, I could dump nash entirely and operate totally
from busybox with a minimal increase in size.  Which raises some other
questions regarding busybox...

	Originally, I was using the static busybox from the Adios bootable CD
flavor of FC (based on FC3).  That was built with ash and wasn't much
bigger than nash (about 10K larger).  If I could eliminate nash, the
size difference would have been relatively small.  But, busybox doesn't
have that "makerootdev" or mount by LABEL= (AFAICT) or switchroot, so
it's an add on, rather that a replace.  Sigh...  Then I tried switching
to the stock FC version of busybox.  Static version, built from the
SRPM, is over twice as large as the Adios version (gag!)...  I yet
haven't gone through it, option by option, to figure out just WHAT is
bloating busybox like that.  But I had to modify busybox anyways, since
neither version included losetup, which I definitely needed.

	Busybox in Fedora Core seems to be built with msh and not ash.  But
I've found several normal "sh" type things that work just fine under
"ash" but just flat out don't work properly in "msh".

	Example:

	cat /proc/partitions | while read major minor blocks name
	do
		:
	done

	and

	while read major minor blocks name
	do
		:
	done < /proc/partitions

	Under msh, the first iteration in the loop works and I get the correct
values.  All subsequent iterations return null in all the variables
although the number of iterations is correct.  That's not the only
instance of normal "sh" type structures that fail on msh but work with
ash.  I also had some troubles with some odd VARBLE=`command` type
substitutions that didn't seem to do the right thing.  So it seems like
msh is buggy.  So I rebuilt the FC flavor of busybox to dump msh and add
ash and add ash as sh.  And the scripts then work once again.  And
busybox got smaller (a bit smaller).  Ok...  So msh is busted for
certain legitimate sh constructs and it's fatter than ash.  Why is msh
being used instead of ash?  What does msh do (I can't find ANY doco on
msh) that ash doesn't?  There must be something good and useful about it
or why would it be there?

	Right now, I've got a script that runs the stock "mkinitrd" called to
add the appropriate modules to be loaded.  It then unpacks the resulting
initrd archive and adds cryptsetup (which is already static) and the
static busybox, plus my files and finally hooks init to call the crypto
script.  Then I repack the initrd file and it's done.  I'd definitely
like it to be smaller and I'm definitely going to add some rm's in there
to get rid of anything other than nash in bin before running
"switchroot" but it's now working rock solid.  Any idea when the
switchroot thing will be fixed to completely free up the initramfs
thing?

>  > If you update to a new kernel, and that kernel isn't
> detecting your 
>  > drives correctly, then that is very difficult to troubleshoot once you
>  > boot to a rescue kernel. 
>  How often is that a real problem?  I don't think I've seen
> any
>  significant number of bug reports on this in recent memory. 
	In my case, with encrypting the drives, I'm into this area every time
my laptop boots.  I travel internationally for speaking engagements and
we consider this an important feature.  I'm also working up a talk for
some Linux shows (maybe LinuxWorld where I regularly speak) and groups
on doing this.  There seems to be some significant interest, judging
from comments I've heard back from the Atlanta Linux Enthusiasts group,
especially after some of the headlines about stolen laptops and lost
hard drives resulting in confidential information (like customer and
client data) leaking out.  :-)

	I'll be making the various scripts publicly available once I get past a
few cleanups and write some maintenance scripts and get past my first
talk on the subject (July 14 in Atlanta as things stand right now).

>  -- 
>          Peter
>   
	Mike
-- 
 Michael H. Warfield    |  [phone]   |  mhw(a)WittsEnd.com  
  /\/\|=mhw=|\/\/       |  [phone]   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!

[PATCH] mkinitrd rescue mode - now crypto

[Jeff Layton]

On Wed, 2005-06-08 at 13:15 -0400, Michael H. Warfield wrote:
>  Any idea when the
>  switchroot thing will be fixed to completely free up the initramfs
>  thing? 
Looks like Peter incorporated a slightly modified version of my patch
into mkinitrd-4.2.16. I've been working with it today and that version
of nash seems to work, but it's tough to tell whether it's actually
cleaning things up or not.

Is there some way to tell whether it's actually cleaning things up?
Where would this memory allocation show up under /proc/meminfo? Perhaps
I can roll a large initramfs image, and see if there is a difference
with and without this nash patch in how memory is allocated...

-- 
Jeff Layton <jlayton(a)redhat.com&gt;

[PATCH] mkinitrd rescue mode - now crypto

[Peter Jones]

On Thu, 2005-06-09 at 16:29 -0400, Jeff Layton wrote:
>  On Wed, 2005-06-08 at 13:15 -0400, Michael H. Warfield wrote:
>  > Any idea when the
>  > switchroot thing will be fixed to completely free up the initramfs
>  > thing?
>  
>  Looks like Peter incorporated a slightly modified version of my patch
>  into mkinitrd-4.2.16. I've been working with it today and that version
>  of nash seems to work, but it's tough to tell whether it's actually
>  cleaning things up or not.
>  
>  Is there some way to tell whether it's actually cleaning things up?
>  Where would this memory allocation show up under /proc/meminfo? Perhaps
>  I can roll a large initramfs image, and see if there is a difference
>  with and without this nash patch in how memory is allocated... 
Well, there is the scary echo/cd hack patch that was attached to one of
those bugs...
-- 
        Peter

[PATCH] mkinitrd rescue mode - now crypto

[Jeff Layton]

On Thu, 2005-06-09 at 17:13 -0400, Peter Jones wrote:
>  
>  Well, there is the scary echo/cd hack patch that was attached to one of
>  those bugs... 
Ok, I did a little more testing today. Enabled -DDEBUG in the build
process and added this just after the recursiveRemove():

lsdir("/","");

After that, I booted to an initramfs with my test nash, and it showed
this:

Switching to new root
//proc/
//sys/
//sysroot/
unmounting old /proc
unmounting old /sys

So it looks like only /proc,/sys, and /sysroot were still in the root
directory. I suppose this means we can call the recursiveRemove() a
success!

As a side note, however, I've been unable so far to get an initramfs
that is a "merge" of more than one cpio image. I've tried passing more
than 1 initrd directive via GRUB, and have tried using an initrd command
line that has more than one arg. Neither way seems to work -- perhaps
this is an issue with GRUB in FC4?

-- 
Jeff Layton <jlayton(a)redhat.com&gt;

[PATCH] mkinitrd rescue mode - now crypto

[Gregory Maxwell]

On 6/8/05, Michael H. Warfield <mhw(a)wittsend.com&gt; wrote:
>          Only reason I seem to need nash at all, right now, is for
> the
>  "makerootdev" command and the "switchroot" command (busybox has
>  pivot_root, but not switchroot, and I can ignore the "setquiet" noise).
>  With those two functions, I could dump nash entirely and operate totally
>  from busybox with a minimal increase in size.  Which raises some other
>  questions regarding busybox... 
I too am running an entirely encrypted disk (LVM in dmcrypt no less,
so I get auto volume detection)... I pretty much solved it the same
way you did, I kept nash around for switchroot and change to ash for
everything else so I could interact with the password prompt.

[PATCH] mkinitrd rescue mode - now crypto

[W. Michael Petullo]

> >         Only reason I seem to need nash at all, right now, is for
> the
> > "makerootdev" command and the "switchroot" command (busybox has
> > pivot_root, but not switchroot, and I can ignore the "setquiet" noise).
> > With those two functions, I could dump nash entirely and operate totally
> > from busybox with a minimal increase in size.  Which raises some other
> > questions regarding busybox...
>
>  I too am running an entirely encrypted disk (LVM in dmcrypt no less,
>  so I get auto volume detection)... I pretty much solved it the same
>  way you did, I kept nash around for switchroot and change to ash for
>  everything else so I could interact with the password prompt. 
I have been maintaining a patch for Fedora's mkinitrd that adds support
for encrypted root filesystems.  The patch may be found at:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=124789

Currently, my system requires that the boot partition and encryption key
reside together on a removable disk.

--
Mike

[Jason L Tibbitts III]

> >>>> "JL" == Jeffrey Layton
> <jlayton(a)redhat.com&gt; writes: 
JL> Essentially, I'm sending this to solicit some feedback on
JL> it. Anyone have comments or concerns with this idea?

I really like the idea.  Often when there's a problem I don't want to
go messing with the DHCP server to get a PXE load going (due the the
screams from the hallway) and I have a couple of boxes on networks not
under my control so that there's no chance of getting DHCP properly
configured.

Many will complain about initramfs bloat and those complaints are
valid.  But isn't the initramfs dynamically generated at kernel
install time (or later if you call the right tool)?  So let me decide
(via /etc/sysconfig/mkinitrd or whatever) what stuff gets crammed into
my initramfs and give me a boot flag to drop into it and I'm quite
happy.

 - J<