3.11.1 kernel upgrade fails: any way to use former kernel?

[sean darcy]

Using Fedora-x86_64-19-20130627-sda (ami-10cce555) on us-west-1b, I 
updated to the recent kernel. Now it won't boot. It seems initrd is 
fried. Anyway to get in to make it to boot from a previous kernel?

Or is this another argument for making more frequent snapshots?

BTW, this kernel worked with another instance in asia. Perhaps it's an 
aws issue in us-west?

sean

syslog:

  [J  Booting 'Fedora (3.11.2-201.fc19.x86_64) 19 (Schrödinger’s Cat)'



root (hd0)

  Filesystem type is ext2fs, using whole disk

kernel /boot/vmlinuz-3.11.2-201.fc19.x86_64 ro 
root=UUID=e3577648-5703-459d-b55

7-b3947684f8ab serial=tty0 console=ttyS0,115200n8 console=hvc0 
console=tty1 LAN

G=en_US.UTF-8

initrd /boot/initramfs-3.11.2-201.fc19.x86_64.img

block error -1 for op 0


Error 3: Bad or corrupt data while decompressing file



Press any key to continue...Xen Minimal OS!
   start_info: 0xac4000(VA)
     nr_pages: 0x26700
   shared_inf: 0x8c534000(MA)
      pt_base: 0xac7000(VA)
nr_pt_frames: 0x9
     mfn_list: 0x990000(VA)
    mod_start: 0x0(VA)
      mod_len: 0
        flags: 0x0
     cmd_line: root=/dev/sda1 ro 4
   stack:      0x94f860-0x96f860
MM: Init
       _text: 0x0(VA)
      _etext: 0x5ff6d(VA)
    _erodata: 0x78000(VA)
      _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
        _end: 0x98fe68(VA)
   start_pfn: ad3
     max_pfn: 26700
Mapping memory range 0xc00000 - 0x26700000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for c01000(c01000)-26700000(26700000)
MM: done
Demand map pfns at 26701000-2026701000.
Heap resides at 2026702000-4026702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x26701000.
Initialising scheduler
Thread "Idle": pointer: 0x2026702010, stack: 0x26640000
Initialising xenbus
Thread "xenstore": pointer: 0x20267027c0, stack: 0x26650000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x2026702f70, stack: 0x26660000
"main" "root=/dev/sda1" "ro" "4"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********


backend at /local/domain/0/backend/vbd/544/2049
Failed to read /local/domain/0/backend/vbd/544/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/544/2049/feature-flush-cache.
4194304 sectors of 512 bytes
**************************
Thread "kbdfront": pointer: 0x2026830010, stack: 0x26680000
******************* FBFRONT for device/vfb/0 **********


******************* KBDFRONT for device/vkbd/0 **********


Failed to read device/vfb/0/backend-id.
Failed to read device/vkbd/0/backend-id.
Error ENOENT when reading the backend path device/vkbd/0/backend
Thread "kbdfront" exited.
Error ENOENT when reading the backend path device/vfb/0/backend

[Matthew Miller]

On Tue, Oct 01, 2013 at 01:56:01PM -0400, Sean Darcy wrote:
>  Using Fedora-x86_64-19-20130627-sda (ami-10cce555) on us-west-1b, I
>  updated to the recent kernel. Now it won't boot. It seems initrd is
>  fried. Anyway to get in to make it to boot from a previous kernel? 
So, here's what to do in ten actually-pretty-easy steps:

 1. Stop (but do not terminate) the broken instance,
 2. Launch a new temporary instance in the same zone. (Or use an existing
    one that's working.)
 3. Detach the root volume from the broken instance
 4. Attach it to the working instance as /dev/sdf (or whatever)
 5. In the working instance, mount the filesystem
 6. Find and fix the problem (possibly edit the /boot/grub/menu.lst?)
 7. Unmount the filesystem
 8. Detach from the working instance
 9. Reattach to the hopefully-fixed original instance
 10. Boot


As step four and a half, you may actually want to run fsck on the volume.

In general, you should store your data on a separate volume so that if this
happens you just replace the machine and go on without worrying about it.


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[David Duncan]

On Tue, Oct 1, 2013 at 12:08 PM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  happens you just replace the machine and go on without worrying about
> it. 
Another step, after that that may be helpful to you is an md5sum check
of the initrd

David Duncan
http://about.me/davdunc
Registered Linux User #279425
http://counter.li.org

[sean darcy]

On 10/01/2013 03:08 PM, Matthew Miller wrote:
>  On Tue, Oct 01, 2013 at 01:56:01PM -0400, Sean Darcy wrote:
> > Using Fedora-x86_64-19-20130627-sda (ami-10cce555) on us-west-1b, I
> > updated to the recent kernel. Now it won't boot. It seems initrd is
> > fried. Anyway to get in to make it to boot from a previous kernel?
>
>  So, here's what to do in ten actually-pretty-easy steps:
>
>    1. Stop (but do not terminate) the broken instance,
>    2. Launch a new temporary instance in the same zone. (Or use an existing
>       one that's working.)
>    3. Detach the root volume from the broken instance
>    4. Attach it to the working instance as /dev/sdf (or whatever)
>    5. In the working instance, mount the filesystem
>    6. Find and fix the problem (possibly edit the /boot/grub/menu.lst?)
>    7. Unmount the filesystem
>    8. Detach from the working instance
>    9. Reattach to the hopefully-fixed original instance
>    10. Boot
>
>
>  As step four and a half, you may actually want to run fsck on the volume.
>
>  In general, you should store your data on a separate volume so that if this
>  happens you just replace the machine and go on without worrying about it.
>
>  
Finally got around to try to fix this. Did steps 1-4, using an F19 cloud 
image. But then:


fdisk -l /dev/xvdf

Disk /dev/xvdf: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x20ac7dda

This doesn't look like a partition table
Probably you selected the wrong device.

     Device Boot      Start         End      Blocks   Id  System
/dev/xvdf1   ?  3224498923  3657370039   216435558+   7  HPFS/NTFS/exFAT
/dev/xvdf2   ?  3272020941  5225480974   976730017   16  Hidden FAT16
/dev/xvdf3   ?           0           0           0   6f  Unknown
/dev/xvdf4        50200576   974536369   462167897    0  Empty

Partition table entries are not in disk order

  fsck /dev/xvdf1
fsck from util-linux 2.23.2
e2fsck 1.42.7 (21-Jan-2013)
fsck.ext2: No such file or directory while trying to open /dev/xvdf1
Possibly non-existent device?


mount /dev/xvdf1 /mnt/xvdf1
mount: special device /dev/xvdf1 does not exist

Really odd.

sean





















  fdisk /dev/xvdf
Welcome to fdisk (util-linux 2.23.1).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/xvdf: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x20ac7dda

This doesn't look like a partition table
Probably you selected the wrong device.

     Device Boot      Start         End      Blocks   Id  System
/dev/xvdf1   ?  3224498923  3657370039   216435558+   7  HPFS/NTFS/exFAT
/dev/xvdf2   ?  3272020941  5225480974   976730017   16  Hidden FAT16
/dev/xvdf3   ?           0           0           0   6f  Unknown
/dev/xvdf4        50200576   974536369   462167897    0  Empty

fsck /dev/xvdf1
fsck from util-linux 2.23.1
e2fsck 1.42.7 (21-Jan-2013)
fsck.ext2: No such file or directory while trying to open /dev/xvdf1
Possibly non-existent device?

  mount  /dev/xvdf1 xvdf1
mount: special device /dev/xvdf1 does not exist

[Matthew Miller]

On Sun, Oct 13, 2013 at 01:54:37PM -0400, Sean Darcy wrote:
>  $ fdisk -l /dev/xvdf
>  Disk /dev/xvdf: 2147 MB, 2147483648 bytes, 4194304 sectors
>  Units = sectors of 1 * 512 = 512 bytes
>  Sector size (logical/physical): 512 bytes / 512 bytes
>  I/O size (minimum/optimal): 512 bytes / 512 bytes
>  Disk label type: dos
>  Disk identifier: 0x20ac7dda
>  This doesn't look like a partition table 
Try just mounting it, not as partitioned but as a "partition" itself.

>  mount /dev/xvdf1 /mnt/xvdf1
>  mount: special device /dev/xvdf1 does not exist 
 mount /dev/xvdf /mnt

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;