32 bit AMI registration boot problem: explaination and possible solutions

[Andy Grimm]

On Mon, Nov 3, 2014 at 2:43 PM, David Gay <dgay at redhat.com> wrote:
> Hi -- these beta AMIs need to be done today. 64 bit base AMIs have been successfully uploaded, but I *still* haven't nailed the problem with the 32 bit registration method. Or rather, I've isolated the problem, but haven't found a solution. Searching the net has not been going well with this particular issue, as most people don't seem to deal with 32 bit images much anymore, and barely any EC2 instance types even support the architecture anymore.
>
> I've written up a little problem statement with few solutions. If anyone has experience with this stuff or ideas/input, it's greatly welcomed.
>
> Problem
> -------
>
> 64 bit base image works 100% fine, 32 bit image fails to boot with a probable
> kernel issue.
>
> System Log
> ----------
>
> The below log likely indicates the attempted use of an invalid kernel. However,
> I _do_ properly specify the AKI of the proper architecture and region.
>
> http://ur1.ca/ioklt
>
> Givens
> ------
>
> 1.  I register AMIs based on an EBS volume snapshot.

The registration looks okay, although the AKI you were using doesn't
appaer to be the very latest.  The one I tried gives the same result.

> 2.  You cannot have an instance that is both 32 bit architecture and
>     EBS-optimized.

"EBS-optimized" is just a storage performance thing, and has nothing
to do with whether the image boots.

> 3.  I use a 64 bit "utility instance" (so that I can use EBS backing) to write
>     the 32-bit raw image file to a secondary volume.

You're just dd'ing bytes here, so that does not matter.

>     3a. That volume is necessarily EBS, as previously mentioned.

Again, no problem.  bytes are bytes.

> 4.  The registered 32 bit AMI is tested by spinning up a 32 bit instance
>     (non-EBS) based on the AMI I register from the snapshotted EBS volume. (I
>     assume this is where the problem stems from, AKA, maybe you can't register a 32
>     bit AMI based on an EBS-backed volume?)

Nah, this is fine.  You are limited as to which instance types will
boot it, but for example, an m1.medium should.

> 5.  64 bit cloud base AMI registers perfectly fine with this same setup (of
>     course, I use all 64 bit instances and kernel IDs for this process).

yep, and this backs up the idea that it really is a 32-bit kernel
issue, rather than something in your process.

Based on a couple of conversations I found about the Xen hypervisor
and this error message, I believe the issue may be PAE.  Have to do
some testing to confirm that.  have past versions used a PAE kernel?

> Hopeful Solution
> ----------------
>
> There _is_ some sort of way to get a 32 bit AMI registered and bootable with
> this EBS method.
>
> Less-nice Solution
> ------------------
>
> Do an instance store backed registration manually with a manifest file for the
> 32 bit images only. *I don't have experience with instance store registration,
> so help would be welcome, though I can probably figure it out*.
>
> Last-ditch Solution
> -------------------
>
> I've been lead to believe that other folks like kushal may have been able to
> register the 32 bit AMIs succesfully just by straight up using the ec2 command
> line tools. If you have a good method for this, send it my way, and I'll run it
> locally to ensure that we have 32 bit beta AMIs today, since this is sort of the
> hard deadline.
>
> Thanks for reading this longer email!
>
> -- oddshocks
> _______________________________________________
> cloud mailing list
> cloud at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/cloud
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct