Hi -- these beta AMIs need to be done today. 64 bit base AMIs have been successfully uploaded, but I *still* haven't nailed the problem with the 32 bit registration method. Or rather, I've isolated the problem, but haven't found a solution. Searching the net has not been going well with this particular issue, as most people don't seem to deal with 32 bit images much anymore, and barely any EC2 instance types even support the architecture anymore.
I've written up a little problem statement with few solutions. If anyone has experience with this stuff or ideas/input, it's greatly welcomed.
Problem -------
64 bit base image works 100% fine, 32 bit image fails to boot with a probable kernel issue.
System Log ----------
The below log likely indicates the attempted use of an invalid kernel. However, I _do_ properly specify the AKI of the proper architecture and region.
Givens ------
1. I register AMIs based on an EBS volume snapshot.
2. You cannot have an instance that is both 32 bit architecture and EBS-optimized.
3. I use a 64 bit "utility instance" (so that I can use EBS backing) to write the 32-bit raw image file to a secondary volume.
3a. That volume is necessarily EBS, as previously mentioned.
4. The registered 32 bit AMI is tested by spinning up a 32 bit instance (non-EBS) based on the AMI I register from the snapshotted EBS volume. (I assume this is where the problem stems from, AKA, maybe you can't register a 32 bit AMI based on an EBS-backed volume?)
5. 64 bit cloud base AMI registers perfectly fine with this same setup (of course, I use all 64 bit instances and kernel IDs for this process).
Hopeful Solution ----------------
There _is_ some sort of way to get a 32 bit AMI registered and bootable with this EBS method.
Less-nice Solution ------------------
Do an instance store backed registration manually with a manifest file for the 32 bit images only. *I don't have experience with instance store registration, so help would be welcome, though I can probably figure it out*.
Last-ditch Solution -------------------
I've been lead to believe that other folks like kushal may have been able to register the 32 bit AMIs succesfully just by straight up using the ec2 command line tools. If you have a good method for this, send it my way, and I'll run it locally to ensure that we have 32 bit beta AMIs today, since this is sort of the hard deadline.
Thanks for reading this longer email!
-- oddshocks
On Mon, Nov 3, 2014 at 2:43 PM, David Gay dgay@redhat.com wrote:
Hi -- these beta AMIs need to be done today. 64 bit base AMIs have been successfully uploaded, but I *still* haven't nailed the problem with the 32 bit registration method. Or rather, I've isolated the problem, but haven't found a solution. Searching the net has not been going well with this particular issue, as most people don't seem to deal with 32 bit images much anymore, and barely any EC2 instance types even support the architecture anymore.
I've written up a little problem statement with few solutions. If anyone has experience with this stuff or ideas/input, it's greatly welcomed.
Problem
64 bit base image works 100% fine, 32 bit image fails to boot with a probable kernel issue.
System Log
The below log likely indicates the attempted use of an invalid kernel. However, I _do_ properly specify the AKI of the proper architecture and region.
Givens
- I register AMIs based on an EBS volume snapshot.
The registration looks okay, although the AKI you were using doesn't appaer to be the very latest. The one I tried gives the same result.
- You cannot have an instance that is both 32 bit architecture and EBS-optimized.
"EBS-optimized" is just a storage performance thing, and has nothing to do with whether the image boots.
- I use a 64 bit "utility instance" (so that I can use EBS backing) to write the 32-bit raw image file to a secondary volume.
You're just dd'ing bytes here, so that does not matter.
3a. That volume is necessarily EBS, as previously mentioned.
Again, no problem. bytes are bytes.
- The registered 32 bit AMI is tested by spinning up a 32 bit instance (non-EBS) based on the AMI I register from the snapshotted EBS volume. (I assume this is where the problem stems from, AKA, maybe you can't register a 32 bit AMI based on an EBS-backed volume?)
Nah, this is fine. You are limited as to which instance types will boot it, but for example, an m1.medium should.
- 64 bit cloud base AMI registers perfectly fine with this same setup (of course, I use all 64 bit instances and kernel IDs for this process).
yep, and this backs up the idea that it really is a 32-bit kernel issue, rather than something in your process.
Based on a couple of conversations I found about the Xen hypervisor and this error message, I believe the issue may be PAE. Have to do some testing to confirm that. have past versions used a PAE kernel?
Hopeful Solution
There _is_ some sort of way to get a 32 bit AMI registered and bootable with this EBS method.
Less-nice Solution
Do an instance store backed registration manually with a manifest file for the 32 bit images only. *I don't have experience with instance store registration, so help would be welcome, though I can probably figure it out*.
Last-ditch Solution
I've been lead to believe that other folks like kushal may have been able to register the 32 bit AMIs succesfully just by straight up using the ec2 command line tools. If you have a good method for this, send it my way, and I'll run it locally to ensure that we have 32 bit beta AMIs today, since this is sort of the hard deadline.
Thanks for reading this longer email!
-- oddshocks _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On Mon, Nov 03, 2014 at 04:54:33PM -0500, Andy Grimm wrote:
Based on a couple of conversations I found about the Xen hypervisor and this error message, I believe the issue may be PAE. Have to do some testing to confirm that. have past versions used a PAE kernel?
Yes they have. Are we not getting PAE kernels on i386?
On Mon, Nov 03, 2014 at 05:18:11PM -0500, Matthew Miller wrote:
Based on a couple of conversations I found about the Xen hypervisor and this error message, I believe the issue may be PAE. Have to do some testing to confirm that. have past versions used a PAE kernel?
Yes they have. Are we not getting PAE kernels on i386?
Nice — we are not. So if this is the problem, I recommend we drop i386 from EC2 for the beta, and add this to the list of things we need test candidates for early-ish for final.
On Mon, Nov 03, 2014 at 05:22:36PM -0500, Matthew Miller wrote:
Nice — we are not. So if this is the problem, I recommend we drop i386 from EC2 for the beta, and add this to the list of things we need test candidates for early-ish for final.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader
The more I think about it - why do we need 32bit AMIs in the first place? EC2, Openstack, and company all support 64bit OOTB. Is there an instance where you would pick a 32bit cloud instance *over* a 64bit on? I can't think of any instances where that would be the case. But then again, I could be wrong and it's more common/needed than I know.
If there isn't a clearcut case for 32 bit cloud images, I say we just cut them all together.
Thoughts?
On Mon, Nov 03, 2014 at 03:32:29PM -0700, Mike Ruckman wrote:
The more I think about it - why do we need 32bit AMIs in the first place? EC2, Openstack, and company all support 64bit OOTB. Is there an instance where you would pick a 32bit cloud instance *over* a 64bit on? I can't think of any instances where that would be the case. But then again, I could be wrong and it's more common/needed than I know.
Feedback we've gotten so far is that there are still cases where the reduced memory usage is desired. I don't know if we want to go so far as only providing 32 bit on micro and small instances — the matrix gets complicated!
On Mon, Nov 3, 2014 at 5:22 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Nov 03, 2014 at 05:18:11PM -0500, Matthew Miller wrote:
Based on a couple of conversations I found about the Xen hypervisor and this error message, I believe the issue may be PAE. Have to do some testing to confirm that. have past versions used a PAE kernel?
Yes they have. Are we not getting PAE kernels on i386?
Nice -- we are not. So if this is the problem, I recommend we drop i386 from EC2 for the beta, and add this to the list of things we need test candidates for early-ish for final.
There's another issue with 32-bit kernels in Beta as well. If you have an AMD machine, it will oops when it tries to load the early microcode. Now, you think this wouldn't matter on Xen (or KVM) because it's the host kernel that does the loading, but the issue does show up there because of the nature of the bug.
This is fixed in rawhide and F21 now, but it didn't make the Beta kernel. How prevalent AMD host CPUs are in EC2, I have no idea.
josh
Andy:
- You cannot have an instance that is both 32 bit architecture and EBS-optimized.
"EBS-optimized" is just a storage performance thing, and has nothing to do with whether the image boots.
Yeah, that's fine. What I was saying is simply that there are no instance types in EC2 which support both EBS and 32 bit.
------------------------
Mike:
The more I think about it - why do we need 32bit AMIs in the first place? EC2, Openstack, and company all support 64bit OOTB. Is there an instance where you would pick a 32bit cloud instance *over* a 64bit on? I can't think of any instances where that would be the case. But then again, I could be wrong and it's more common/needed than I know.
If there isn't a clearcut case for 32 bit cloud images, I say we just cut them all together.
I'm with you on that. I don't know many people who use 32 bit cloud images. 64 bit is available for every EC2 instance type, and like you said, other services support it. I'm likely not the best person to ask why we provide 32 bit images, though. It could be a legacy thing, but it might also be habit.
------------------------
-- oddshocks
On 11/03/2014 04:43 PM, David Gay wrote:
I'm with you on that. I don't know many people who use 32 bit cloud images. 64 bit is available for every EC2 instance type, and like you said, other services support it. I'm likely not the best person to ask why we provide 32 bit images, though. It could be a legacy thing, but it might also be habit.
I'm +1 to dropping 32-bit unless someone has a good reason why we're doing them...
Best,
jzb
----- Original Message -----
From: "Joe Brockmeier" jzb@redhat.com To: cloud@lists.fedoraproject.org Sent: Monday, November 3, 2014 2:47:00 PM Subject: Re: 32 bit AMI registration boot problem: explaination and possible solutions
On 11/03/2014 04:43 PM, David Gay wrote:
I'm with you on that. I don't know many people who use 32 bit cloud images. 64 bit is available for every EC2 instance type, and like you said, other services support it. I'm likely not the best person to ask why we provide 32 bit images, though. It could be a legacy thing, but it might also be habit.
I'm +1 to dropping 32-bit unless someone has a good reason why we're doing them...
Best,
jzb
+1 from me, as well.
On Mon, Nov 03, 2014 at 05:53:21PM -0500, David Gay wrote:
----- Original Message -----
From: "Joe Brockmeier" jzb@redhat.com To: cloud@lists.fedoraproject.org Sent: Monday, November 3, 2014 2:47:00 PM Subject: Re: 32 bit AMI registration boot problem: explaination and possible solutions
On 11/03/2014 04:43 PM, David Gay wrote:
I'm with you on that. I don't know many people who use 32 bit cloud images. 64 bit is available for every EC2 instance type, and like you said, other services support it. I'm likely not the best person to ask why we provide 32 bit images, though. It could be a legacy thing, but it might also be habit.
I'm +1 to dropping 32-bit unless someone has a good reason why we're doing them...
Best,
jzb
+1 from me, as well.
I would love to get rid of the 32bit AMIs. That being said, is this a "knee jerk" reaction? Typically it's best to make changes like this with planning involved and sufficient notice of the change so the community can prepare.
Again, I'm all for getting rid of them. Just want to make sure we think about everything.
Dusty
On Mon, Nov 3, 2014 at 7:25 PM, Dusty Mabe dusty@dustymabe.com wrote:
On Mon, Nov 03, 2014 at 05:53:21PM -0500, David Gay wrote:
----- Original Message -----
From: "Joe Brockmeier" jzb@redhat.com To: cloud@lists.fedoraproject.org Sent: Monday, November 3, 2014 2:47:00 PM Subject: Re: 32 bit AMI registration boot problem: explaination and possible solutions
On 11/03/2014 04:43 PM, David Gay wrote:
I'm with you on that. I don't know many people who use 32 bit cloud images. 64 bit is available for every EC2 instance type, and like you said, other services support it. I'm likely not the best person to ask why we provide 32 bit images, though. It could be a legacy thing, but it might also be habit.
I'm +1 to dropping 32-bit unless someone has a good reason why we're doing them...
Best,
jzb
+1 from me, as well.
I would love to get rid of the 32bit AMIs. That being said, is this a "knee jerk" reaction? Typically it's best to make changes like this with planning involved and sufficient notice of the change so the community can prepare.
Again, I'm all for getting rid of them. Just want to make sure we think about everything.
My rambling thoughts on this:
Do we have _any_ analytics on usage? (launch stats from AWS, or unique IPs hitting Fedora mirrors inside AWS, or ...)
I don't have any strong reason to want to keep it; I don't run 32-bit anything. But given that we do still formally support the arch in the distro, I don't like knowing that it's completely broken. If it can be made to work (without undue burden on the kernel folks, who certainly have better things to do), we should. Blocking the beta on this if other 32-bit installs work does not seem necessary to me, though.
FWIW, there are only 4 AWS instance types where 32-bit is supported (t1.micro, m1.small, m1.medium, and c1.medium), and what's _really_ funny is that none of them have more than 3.7 GB or RAM (so PAE serves no purpose). It seems like the use case for these would be pretty narrow, given that the price/performance of m3 and t2/t3 are generally better. As Garrett likes to remind us, there's the large ephemeral disk factor on the m1/c1 types, but I don't know how many people care about that.
As for how to actually deal with the problem, since the system is booting and just doesn't appear to be getting a network connection, I suspect we can add some debugging info at boot-time to figure out what's going on. I'll try to come up with a script that'll dump some useful data to the console for us. That could be generally useful, anyway. (cloud-init already does some of this, except that it doesn't ever get run if the NIC isn't present)
--Andy
On Mon, Nov 03, 2014 at 08:41:19PM -0500, Andy Grimm wrote:
Do we have _any_ analytics on usage? (launch stats from AWS, or unique IPs hitting Fedora mirrors inside AWS, or ...)
We might be able to get some mirrormanager stats. But overall... we're pretty short here.
Anyway, thanks for looking into this!
On Mon, Nov 03, 2014 at 07:25:28PM -0500, Dusty Mabe wrote:
I would love to get rid of the 32bit AMIs. That being said, is this a "knee jerk" reaction? Typically it's best to make changes like this with planning involved and sufficient notice of the change so the community can prepare.
Again, I'm all for getting rid of them. Just want to make sure we think about everything.
Dusty
I wasn't meaning it to come across as a knee jerk reaction - I was just curious after thinking about who *uses* a 32bit image when a 64bit is available? Especially for cloud where 64bit is supported everywhere someone might be launching our image from.
The thing I thought about was perhaps there's some package people run on a regular basis that was only available in 32bit - but then again, if it was a commonly used package it's likely (if not for certain) there is a x86_64 version of it in the repos already.
All that being said, I just figured I'd ask if we still need a 32bit AMI. I was counting on everyone else having more background knowledge into this than I have :)
On 11/03/2014 11:51 PM, Mike Ruckman wrote:
On Mon, Nov 03, 2014 at 07:25:28PM -0500, Dusty Mabe wrote:
I would love to get rid of the 32bit AMIs. That being said, is this a "knee jerk" reaction? Typically it's best to make changes like this with planning involved and sufficient notice of the change so the community can prepare.
Again, I'm all for getting rid of them. Just want to make sure we think about everything.
Dusty
I wasn't meaning it to come across as a knee jerk reaction - I was just curious after thinking about who *uses* a 32bit image when a 64bit is available? Especially for cloud where 64bit is supported everywhere someone might be launching our image from.
Indeed. Given the way things work in Fedora, I didn't imagine that saying "yes, we should get rid of these if there's not a reason to keep them" meant they'd disappear immediately. (If only things happened that quickly...)
The thing I thought about was perhaps there's some package people run on a regular basis that was only available in 32bit - but then again, if it was a commonly used package it's likely (if not for certain) there is a x86_64 version of it in the repos already.
All that being said, I just figured I'd ask if we still need a 32bit AMI. I was counting on everyone else having more background knowledge into this than I have :)
Do we have any way of seeing how many people actually use the 32-bit AMI on AWS?
Best,
jzb
On Tue, Nov 04, 2014 at 07:41:27AM -0600, Joe Brockmeier wrote:
Do we have any way of seeing how many people actually use the 32-bit AMI on AWS?
Unfortunately, not really.
On Mon, Nov 03, 2014 at 04:54:33PM -0500, Andy Grimm wrote:
yep, and this backs up the idea that it really is a 32-bit kernel issue, rather than something in your process.
Based on a couple of conversations I found about the Xen hypervisor and this error message, I believe the issue may be PAE. Have to do some testing to confirm that. have past versions used a PAE kernel?
I did find some similar information when I was googling around last week about the PAE kernel. Is the necessity of the PAE kernel specific to paravirt instances? When I created the HVM 32bit AMI last week it did at least boot with whatever 32bit kernel was in RC4.
Dusty
-
Andy Grimm -
David Gay -
Dusty Mabe -
Joe Brockmeier -
Josh Boyer -
Matthew Miller -
Mike Ruckman