Kickstart a Fedora (Anaconda) Install on EC2

[Raphaël De GIUSTI]

Hi everyone,

I've been playing around with Amazon EC2, building my own Centos and Fedora
EBS backed AMI's without much trouble, following tutorials and other
practices I found on the internet.

But one thing I'd like to do, and I tried to do, is kickstarting an
installation using anaconda.

So I would have an minimal AMI that only contains a /boot directory with
vmlinuz and initrd + a /boot/grub/menu.lst file that would look like this :

default 0
timeout 3
title RH-Like-OS
        root (hd0,0)
        kernel /boot/vmlinuz ks=http://some.server/myks.cfg
        initrd /boot/initrd.img

And it would parse my ks, start anaconda and go on with the install.

The further I managed to go is to the partioning step with Centos55. If
someone's interested, there's my unanswered thread on amazon aws' forums

https://forums.aws.amazon.com/message.jspa?messageID=216575#216575

Same process with Fedora 14 only brings me to pvgrub reading the menu.lst,
then failing on a mmu_update.

So, here come my questions:

I was wondering what, technically speaking, prevents me from doing this.
And maybe I would understand why everyone is building the system from
scratch and why I did not find anyone who tried to do the same thing.

Note that I'm not an expert at linux kernels or boot processes, but I'm
quite curious, and I would really appreciate if you could help me understand
:-)

Thank you.

raphdg

[Brian LaMere]

Raphaël - I was thinking about this sort of thing myself a few weeks ago,
but just was too busy with other things to play with it.  Maybe I can trick
a few people to work on it with me this weekend at FUDConn ;)

Brian

2011/1/28 Raphaël De GIUSTI
> Hi everyone,
>
> I've been playing around with Amazon EC2, building my own Centos and Fedora
> EBS backed AMI's without much trouble, following tutorials and other
> practices I found on the internet.
>
> But one thing I'd like to do, and I tried to do, is kickstarting an
> installation using anaconda.
>
> So I would have an minimal AMI that only contains a /boot directory with
> vmlinuz and initrd + a /boot/grub/menu.lst file that would look like this :
>
> default 0
> timeout 3
> title RH-Like-OS
>         root (hd0,0)
>         kernel /boot/vmlinuz ks=http://some.server/myks.cfg
>         initrd /boot/initrd.img
>
> And it would parse my ks, start anaconda and go on with the install.
>
> The further I managed to go is to the partioning step with Centos55. If
> someone's interested, there's my unanswered thread on amazon aws' forums
>
> https://forums.aws.amazon.com/message.jspa?messageID=216575#216575
>
> Same process with Fedora 14 only brings me to pvgrub reading the menu.lst,
> then failing on a mmu_update.
>
> So, here come my questions:
>
> I was wondering what, technically speaking, prevents me from doing this.
> And maybe I would understand why everyone is building the system from
> scratch and why I did not find anyone who tried to do the same thing.
>
> Note that I'm not an expert at linux kernels or boot processes, but I'm
> quite curious, and I would really appreciate if you could help me understand
> :-)
>
> Thank you.
>
> raphdg
>
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

[Michael Howard]

2011/1/28 Raphaël De GIUSTI :
> Hi everyone,
> I've been playing around with Amazon EC2, building my own Centos and Fedora
> EBS backed AMI's without much trouble, following tutorials and other
> practices I found on the internet.

I have been looking for this kind of information, but not having much luck.

Any pointers / URLs to building EBS-backed AMIs for Fedora & CentOS
would be appreciated.

Thanks,
Michael

[Brian LaMere]

http://fedoraproject.org/wiki/Publishing_image_to_EC2 worked fine each of
the times I tried.  There's a big leap from just making an image, to making
a kickstart image that is smart enough to set itself up correctly during
install.  Then, using the "user-data" options, specify which type of
instance to build....

I just haven't gotten to that part yet.  one man, lots of things to do, etc.
 If a number of people are interested in it, perhaps someone should start a
project..?

Brian L
US2002021365

On Fri, Jan 28, 2011 at 11:58 AM, Michael Howard wrote:
> 2011/1/28 Raphaël De GIUSTI :
> > Hi everyone,
> > I've been playing around with Amazon EC2, building my own Centos and
>
> Fedora
> > EBS backed AMI's without much trouble, following tutorials and other
> > practices I found on the internet.
>
> I have been looking for this kind of information, but not having much luck.
>
> Any pointers / URLs to building EBS-backed AMIs for Fedora & CentOS
> would be appreciated.
>
> Thanks,
> Michael
> _______________________________________________
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

[Raphaël De GIUSTI]

Brian, I would appreciate it. Where can I get feedback from this ?

Michael,

Here's a tutorial for a Centos 5.5 install I found very useful :

http://www.ioncannon.net/system-administration/1205/installing-cent-os-5-5-o...

http://www.ioncannon.net/system-administration/1205/installing-cent-os-5-5-on-ec2-with-the-cent-os-5-5-kernel/As
for automating the building of Centos 55 and Fedora 14 instances, there's
the ami-creator from Jeremy Katz that's closer to what I'm looking for as
you put a kickstart in and an ami pops out :

Here is his blog post about it

http://velohacker.com/fedora-notes/announcing-ami-creator/

On Fri, Jan 28, 2011 at 8:58 PM, Michael Howard wrote:
> 2011/1/28 Raphaël De GIUSTI :
> > Hi everyone,
> > I've been playing around with Amazon EC2, building my own Centos and
>
> Fedora
> > EBS backed AMI's without much trouble, following tutorials and other
> > practices I found on the internet.
>
> I have been looking for this kind of information, but not having much luck.
>
> Any pointers / URLs to building EBS-backed AMIs for Fedora & CentOS
> would be appreciated.
>
> Thanks,
> Michael
> _______________________________________________
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

[Brian LaMere]

aye, ami-creator doesn't do EBS-backed AMIs though, right?  It makes a local
image which you then upload to s3, and then you're ephemeral.  if you're
good with that, then hey :)

An EBS-backed AMI that was just a shell looking for a spacewalk/satellite
server for guidance on what to become (or listening to an SQS queue for info
on what to build, or....etc etc) is the next matter.  The biggest question
is then that: where it is you want to get that info about the type of
install to then do?

lemme be more clear:

Need:  a functional replacement for the old use these two types of cases:
1) turning on a machine with PXE-boot that then builds itself magically to
what it is you happen to need
2) a machine that has a tiny bootable CD that knows to look on the network
at magical location X for either the entire kickstart file, or uses a
kickstart file on the CD that then sources to network locations to flesh out
contents of the kickstart.

So, having an AMI that works that way...
1)  hard code it to look for whatever happens to be sitting at location X
for a kickstart file?
2) Have it start up a basic environment that sources to X location for more
info?
3) Have the kickstart file have various logic, and the decisions for the
logic be based on:
3-1) "user-data" info passed during image creation?
3-2) info available at a url?  Ie, check "ec2master.guardis.com/nextec2type"
for a simple answer?
3-3) have kickstart join a queue system of some sort (SQS, AMQP, whatever)
and get info about what to do from the queue?
4) have a spacewalk/satellite server tell the bare-metal install-boot what
to do next?

Yeah, it's something I think I'll sit down and solve relatively soon; I
realized I'm not actually satisfied with making an AMI for each type of role
and then just launching that AMI.  That poses, in my mind, too many
problems...problems that prompted the solutions that have been in place
within datacenters for many years already - be it kickstart servers,
jumpstart servers, software depots, wsus servers...these centralized
solutions for taking a bare installation image and installing whatever you
need next are clearly out there for physical datacenters.  They weren't made
in an absence of need.

So...how then to do those same use cases "in the cloud?"  Do we....need to?
 It seems to me to be an effort worth exploring, but...maybe not?

Whatever the case, we haven't even defined a spec, much less has anyone
solved the problem yet - that I've seen, at least.  Though we might have
been able to get a POC during a few hour hackfest this past weekend, maybe
;)

Brian

2011/1/31 Raphaël De GIUSTI
> Brian, I would appreciate it. Where can I get feedback from this ?
>
> Michael,
>
> Here's a tutorial for a Centos 5.5 install I found very useful :
>
> http://www.ioncannon.net/system-administration/1205/installing-cent-os-5-5-o...
>
> http://www.ioncannon.net/system-administration/1205/installing-cent-os-5-5-on-ec2-with-the-cent-os-5-5-kernel/As
> for automating the building of Centos 55 and Fedora 14 instances, there's
> the ami-creator from Jeremy Katz that's closer to what I'm looking for as
> you put a kickstart in and an ami pops out :
>
> Here is his blog post about it
>
> http://velohacker.com/fedora-notes/announcing-ami-creator/
>
> On Fri, Jan 28, 2011 at 8:58 PM, Michael Howard wrote:
> > 2011/1/28 Raphaël De GIUSTI :
> > > Hi everyone,
> > > I've been playing around with Amazon EC2, building my own Centos and
> >
> > Fedora
> > > EBS backed AMI's without much trouble, following tutorials and other
> > > practices I found on the internet.
> >
> > I have been looking for this kind of information, but not having much
> > luck.
> >
> > Any pointers / URLs to building EBS-backed AMIs for Fedora & CentOS
> > would be appreciated.
> >
> > Thanks,
> > Michael
> > _______________________________________________
> > cloud mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/cloud
>
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

[Jeremy Katz]

On Mon, Jan 31, 2011 at 5:09 PM, Brian LaMere
 wrote:
> aye, ami-creator doesn't do EBS-backed AMIs though, right?  It makes a local
> image which you then upload to s3, and then you're ephemeral.  if you're
> good with that, then hey :)

Right, it's entirely ephemeral AMIs at this point.  Unfortunately,
doing more is somewhere between tough and impossible in a sane way as
there isn't really a good API for uploading an EBS backed AMI.  It's
all "dd onto a block device, snapshot, foo".
> An EBS-backed AMI that was just a shell looking for a spacewalk/satellite
> server for guidance on what to become (or listening to an SQS queue for info
> on what to build, or....etc etc) is the next matter.  The biggest question
> is then that: where it is you want to get that info about the type of
> install to then do?

Yep, would definitely be pretty cool.  And it should be relatively
straight-forward to do without requiring too much work on the anaconda
front.  In terms of keeping things relatively consistent with a normal
install path, the path of least resistance is probably just sticking
what would be kernel command line params in user data and then having
anaconda know to look there if it notices it's running in EC2.  Then
you can install to your EBS volume, reboot and voila.

If I get to where I'm doing more with EBS backed AMIs, I'll probably
sit down and look at it if no one has tackled it before then.  That
said, if someone wanted to tackle it before then and wanted some
pointers on anaconda code, etc, then let me know and I could
definitely try to be a helpful resource for that.

- Jeremy

[Brian LaMere]

> Right, it's entirely ephemeral AMIs at this point.  Unfortunately,
> doing more is somewhere between tough and impossible in a sane way as
> there isn't really a good API for uploading an EBS backed AMI.  It's
> all "dd onto a block device, snapshot, foo".

yeah, that's why although my silly script is ugly it still produces the
right result; an AMI that can be used for EBS-backed stores, of an image
that has never run and is free of such taint.  How one gets to there is
important, but considering it's not something that gets repeated often (once
you have an AMI, the AMI is done...) the real issue is what the end result
ended up being.

front.  In terms of keeping things relatively consistent with a normal
> install path, the path of least resistance is probably just sticking
> what would be kernel command line params in user data and then having
> anaconda know to look there if it notices it's running in EC2.  Then
> you can install to your EBS volume, reboot and voila.

Yup, no reason "curl 169.254.169.254/2009-0404/user-data/" couldn't respond
with "BUILD_ROLE=lamp" with kickstart doing logic based on such, or how ever
else one would want to do it.  Could even lists the package groups and
individual packages right there, as well as where to find root and such.
 The requisite tools all exist, they just need someone to put them all
together.  Sounds like Jeremy is volunteering!  ;)

Brian

[Garrett Holmstrom]

On Wed, Feb 2, 2011 at 10:19 AM, Brian LaMere
 wrote:
> > Right, it's entirely ephemeral AMIs at this point.  Unfortunately,
> > doing more is somewhere between tough and impossible in a sane way as
> > there isn't really a good API for uploading an EBS backed AMI.  It's
> > all "dd onto a block device, snapshot, foo".
>
> yeah, that's why although my silly script is ugly it still produces the
> right result; an AMI that can be used for EBS-backed stores, of an image
> that has never run and is free of such taint.  How one gets to there is
> important, but considering it's not something that gets repeated often (once
> you have an AMI, the AMI is done...) the real issue is what the end result
> ended up being.

At FUDCon I heard that rawhide's version of anaconda can install
directly to things like disk images and block devices, IIRC.  Maybe
that would be useful.

Maybe I'm missing something:  why would you ever want an instance to
kickstart at boot time?  You should create an image for every role you
care about and then boot the appropriate one for every instance you
need.

[Jeremy Katz]

On Wed, Feb 2, 2011 at 3:42 PM, Garrett Holmstrom
 wrote:
> On Wed, Feb 2, 2011 at 10:19 AM, Brian LaMere
>  wrote:
> > > Right, it's entirely ephemeral AMIs at this point.  Unfortunately,
> > > doing more is somewhere between tough and impossible in a sane way as
> > > there isn't really a good API for uploading an EBS backed AMI.  It's
> > > all "dd onto a block device, snapshot, foo".
> >
> > yeah, that's why although my silly script is ugly it still produces the
> > right result; an AMI that can be used for EBS-backed stores, of an image
> > that has never run and is free of such taint.  How one gets to there is
> > important, but considering it's not something that gets repeated often (once
> > you have an AMI, the AMI is done...) the real issue is what the end result
> > ended up being.
>
> At FUDCon I heard that rawhide's version of anaconda can install
> directly to things like disk images and block devices, IIRC.  Maybe
> that would be useful.

Unclear.  The pendulum is swaying back towards putting the pieces into
anaconda when they were pulled out from anaconda and into
python-imgcreate/livecd-creator ~ 5 years ago ;)  There are arguments
both ways and I've even argued both of them over time :-)
> Maybe I'm missing something:  why would you ever want an instance to
> kickstart at boot time?  You should create an image for every role you
> care about and then boot the appropriate one for every instance you
> need.

For the exact same reason that you ever kickstart a machine vs using
some imaging technology -- flexibility.  You can generate a kickstart
config by a CGI and have a virtually infinite number of combinations.
You can't store an infinite number of images (obviously) and even
storing a lot of images becomes costly.  Even at EC2 prices.

- Jeremy

[Brian LaMere]

> Maybe I'm missing something:  why would you ever want an instance to
> kickstart at boot time?  You should create an image for every role you
> care about and then boot the appropriate one for every instance you
> need.

roles change, updates happen frequently, and I'd rather a machine spin up
with the latest packages.  I've always found that updating a pre-built
machine is slower, sometimes substantially so, than just building a fresh
image with the newest rpms.

That said, some roles can (and often should) be fairly rigid and slow to be
updated.  But there's not much less of a need for flexible, dynamic builds
in the cloud than there is in a local server room; do you build all new
local servers based on a pre-built image that you just replicate?  Would
seem to negate the purpose of a kickstart server ;)

Brian

[Raphaël De GIUSTI]

So if we put aside how the kickstart would be generated, what would be the
starting point in all of this ?

I was working with Centos55 when I made it to the partitioning phase (and I
don't really know why it went wrong), but I couldn't even get the F14 kernel
to load.

In other words :

- Which fedora kernel / initrd combination should I use in my grub.conf ?
- Should any of those two be altered in any way ?

I'm also willing to put some time in it if I may be useful... but like I
said I'm no expert.

On Wed, Feb 2, 2011 at 9:48 PM, Brian LaMere wrote:
> Maybe I'm missing something:  why would you ever want an instance to
> > kickstart at boot time?  You should create an image for every role you
> > care about and then boot the appropriate one for every instance you
> > need.
>
> roles change, updates happen frequently, and I'd rather a machine spin up
> with the latest packages.  I've always found that updating a pre-built
> machine is slower, sometimes substantially so, than just building a fresh
> image with the newest rpms.
>
> That said, some roles can (and often should) be fairly rigid and slow to be
> updated.  But there's not much less of a need for flexible, dynamic builds
> in the cloud than there is in a local server room; do you build all new
> local servers based on a pre-built image that you just replicate?  Would
> seem to negate the purpose of a kickstart server ;)
>
> Brian
>
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

[Raphaël De GIUSTI]

Hi,

I simply tried to boot on the fedora 15 alpha kernel from there :
http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...
http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/releases/test/15-Alpha/Fedora/i386/os/images/pxeboot/and
I'm getting this error :

root (hd0,0)
> Filesystem type is ext2fs, partition type 0x83
> kernel /boot/vmlinuz-PAE
> initrd /boot/initrd-PAE.img
> ERROR: mmu_update failed with rc=-22
> Do_exit called!
> base is 0x26617ba8 caller is 0x45c87
> base is 0x26617bd8 caller is 0x50336
> base is 0x26617c38 caller is 0x504c0
> base is 0x26617c88 caller is 0x506a3
> base is 0x2661fd08 caller is 0x479d7
> base is 0x2661fd48 caller is 0x5613b
> base is 0x2661fd58 caller is 0x54698
> base is 0x2661fd98 caller is 0x55a79
> base is 0x2661fde8 caller is 0x55811
> base is 0x2661fe08 caller is 0x3b0c
> base is 0x2661fe38 caller is 0x3bc4
> base is 0x2661fe48 caller is 0x7af7
> base is 0x2661fe58 caller is 0xa243
> base is 0x2661fe78 caller is 0xfe69
> base is 0x2661fef8 caller is 0x10489
> base is 0x2661ff68 caller is 0x3eb2
> base is 0x2661ff78 caller is 0x4729d
> base is 0x2661fff0 caller is 0x31ad

I just wondered where it does come from ?
Any idea ?

2011/2/8 Raphaël De GIUSTI
> So if we put aside how the kickstart would be generated, what would be the
> starting point in all of this ?
>
> I was working with Centos55 when I made it to the partitioning phase (and I
> don't really know why it went wrong), but I couldn't even get the F14 kernel
> to load.
>
> In other words :
> Which fedora kernel / initrd combination should I use in my grub.conf ?
> Should any of those two be altered in any way ?
>
> I'm also willing to put some time in it if I may be useful... but like I
> said I'm no expert.
>
> On Wed, Feb 2, 2011 at 9:48 PM, Brian LaMere wrote:
> > Maybe I'm missing something:  why would you ever want an instance to
> > > kickstart at boot time?  You should create an image for every role you
> > > care about and then boot the appropriate one for every instance you
> > > need.
> >
> > roles change, updates happen frequently, and I'd rather a machine spin up
> > with the latest packages.  I've always found that updating a pre-built
> > machine is slower, sometimes substantially so, than just building a fresh
> > image with the newest rpms.
> >
> > That said, some roles can (and often should) be fairly rigid and slow to
> > be updated.  But there's not much less of a need for flexible, dynamic
> > builds in the cloud than there is in a local server room; do you build all
> > new local servers based on a pre-built image that you just replicate?  Would
> > seem to negate the purpose of a kickstart server ;)
> >
> > Brian
> >
> > cloud mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/cloud

[Raphaël De GIUSTI]

It seems to be happening when launching micro instances, as memory is low
(about 600M)
Launching it as a small instance, anaconda is executed, but there seems to
be a problem related with dbus :

Greetings.
anaconda installer init version 15.20.1 starting
mounting /proc filesystem... done

(process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
`hash_table != NULL' failed
creating /dev filesystem... done
starting udev...done
mounting /dev/pts (unix98 pty) filesystem... done
mounting /sys filesystem... done

(process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
`hash_table != NULL' failed
anaconda installer init version 15.20.1 using /dev/hvc0 as console
trying to remount root filesystem read write... done
mounting /tmp as tmpfs... done

(process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
`hash_table != NULL' failed
running install...
running /sbin/loader

%Gdetecting hardware...
waiting for hardware to initialize...
detecting hardware...
waiting for hardware to initialize...

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or directory

2011/3/10 Raphaël De GIUSTI
> Hi,
> I simply tried to boot on the fedora 15 alpha kernel from there :

http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...
> and I'm getting this error :
> > root (hd0,0)
> >
> > Filesystem type is ext2fs, partition type 0x83
> > kernel /boot/vmlinuz-PAE
> > initrd /boot/initrd-PAE.img
> > ERROR: mmu_update failed with rc=-22
> > Do_exit called!
> > base is 0x26617ba8 caller is 0x45c87
> > base is 0x26617bd8 caller is 0x50336
> > base is 0x26617c38 caller is 0x504c0
> > base is 0x26617c88 caller is 0x506a3
> > base is 0x2661fd08 caller is 0x479d7
> > base is 0x2661fd48 caller is 0x5613b
> > base is 0x2661fd58 caller is 0x54698
> > base is 0x2661fd98 caller is 0x55a79
> > base is 0x2661fde8 caller is 0x55811
> > base is 0x2661fe08 caller is 0x3b0c
> > base is 0x2661fe38 caller is 0x3bc4
> > base is 0x2661fe48 caller is 0x7af7
> > base is 0x2661fe58 caller is 0xa243
> > base is 0x2661fe78 caller is 0xfe69
> > base is 0x2661fef8 caller is 0x10489
> > base is 0x2661ff68 caller is 0x3eb2
> > base is 0x2661ff78 caller is 0x4729d
> > base is 0x2661fff0 caller is 0x31ad
>
> I just wondered where it does come from ?
> Any idea ?
>
> 2011/2/8 Raphaël De GIUSTI
> > So if we put aside how the kickstart would be generated, what would be

the starting point in all of this ?
> > I was working with Centos55 when I made it to the partitioning phase (and

I don't really know why it went wrong), but I couldn't even get the F14
kernel to load.
> > In other words :
> > Which fedora kernel / initrd combination should I use in my grub.conf ?
> > Should any of those two be altered in any way ?
> >
> > I'm also willing to put some time in it if I may be useful... but like I

said I'm no expert.
> > On Wed, Feb 2, 2011 at 9:48 PM, Brian LaMere

wrote:
> > > > Maybe I'm missing something:  why would you ever want an instance to
> > > > kickstart at boot time?  You should create an image for every role you
> > > > care about and then boot the appropriate one for every instance you
> > > > need.
> > >
> > > roles change, updates happen frequently, and I'd rather a machine spin

up with the latest packages.  I've always found that updating a pre-built
machine is slower, sometimes substantially so, than just building a fresh
image with the newest rpms.
> > > That said, some roles can (and often should) be fairly rigid and slow to

be updated.  But there's not much less of a need for flexible, dynamic
builds in the cloud than there is in a local server room; do you build all
new local servers based on a pre-built image that you just replicate?  Would
seem to negate the purpose of a kickstart server ;)
> > > Brian
> > > _______________________________________________
> > > cloud mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/cloud

[Brian LaMere]

Raphael - there's more than just a memory difference between micro and
small; micros are 64bit, smalls are 32bit.  If you try the same thing
on a large, do you have the same problem?  Small and medium are both
32bit, micro, large, and every other size are all 64bit.

The initial error of "base is 0x26617ba8 caller is 0x45c87" looks like
something calling a 32 against a 64.  Someone smarter than me
suggested this means that XSAVE isn't disabled in the kernel; that
would mean the anaconda kernel itself, which is pre-built.  Maybe get
with the anaconda folks to verify whether the CD kernel has the XSAVE
patch?  The kernel in the distro does, otherwise we wouldn't be able
to use it ;)  But that doesn't mean the kernel in CDHOME/isolinux
does.

Does what I'm asking/suggesting make sense?

Brian

2011/3/18 Raphaël De GIUSTI :
> It seems to be happening when launching micro instances, as memory is low
> (about 600M)
> Launching it as a small instance, anaconda is executed, but there seems to
> be a problem related with dbus :
> Greetings.
> anaconda installer init version 15.20.1 starting
> mounting /proc filesystem... done
>
> (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> `hash_table != NULL' failed
> creating /dev filesystem... done
> starting udev...done
> mounting /dev/pts (unix98 pty) filesystem... done
> mounting /sys filesystem... done
>
> (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> `hash_table != NULL' failed
> anaconda installer init version 15.20.1 using /dev/hvc0 as console
> trying to remount root filesystem read write... done
> mounting /tmp as tmpfs... done
>
> (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> `hash_table != NULL' failed
> running install...
> running /sbin/loader
>
> %Gdetecting hardware...
> waiting for hardware to initialize...
> detecting hardware...
> waiting for hardware to initialize...
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or directory
>
> 2011/3/10 Raphaël De GIUSTI
> > Hi,
> > I simply tried to boot on the fedora 15 alpha kernel from there :
> >
> > http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...
> > and I'm getting this error :
> > > root (hd0,0)
> > >
> > > Filesystem type is ext2fs, partition type 0x83
> > > kernel /boot/vmlinuz-PAE
> > > initrd /boot/initrd-PAE.img
> > > ERROR: mmu_update failed with rc=-22
> > > Do_exit called!
> > > base is 0x26617ba8 caller is 0x45c87
> > > base is 0x26617bd8 caller is 0x50336
> > > base is 0x26617c38 caller is 0x504c0
> > > base is 0x26617c88 caller is 0x506a3
> > > base is 0x2661fd08 caller is 0x479d7
> > > base is 0x2661fd48 caller is 0x5613b
> > > base is 0x2661fd58 caller is 0x54698
> > > base is 0x2661fd98 caller is 0x55a79
> > > base is 0x2661fde8 caller is 0x55811
> > > base is 0x2661fe08 caller is 0x3b0c
> > > base is 0x2661fe38 caller is 0x3bc4
> > > base is 0x2661fe48 caller is 0x7af7
> > > base is 0x2661fe58 caller is 0xa243
> > > base is 0x2661fe78 caller is 0xfe69
> > > base is 0x2661fef8 caller is 0x10489
> > > base is 0x2661ff68 caller is 0x3eb2
> > > base is 0x2661ff78 caller is 0x4729d
> > > base is 0x2661fff0 caller is 0x31ad
> >
> > I just wondered where it does come from ?
> > Any idea ?
> >
> > 2011/2/8 Raphaël De GIUSTI
> > > So if we put aside how the kickstart would be generated, what would be
> > > the starting point in all of this ?
> > > I was working with Centos55 when I made it to the partitioning phase (and
> > > I don't really know why it went wrong), but I couldn't even get the F14
> > > kernel to load.
> > > In other words :
> > > Which fedora kernel / initrd combination should I use in my grub.conf ?
> > > Should any of those two be altered in any way ?
> > >
> > > I'm also willing to put some time in it if I may be useful... but like I
> > > said I'm no expert.
> > > On Wed, Feb 2, 2011 at 9:48 PM, Brian LaMere 
> > > wrote:
> > > > > Maybe I'm missing something:  why would you ever want an instance to
> > > > > kickstart at boot time?  You should create an image for every role you
> > > > > care about and then boot the appropriate one for every instance you
> > > > > need.
> > > >
> > > > roles change, updates happen frequently, and I'd rather a machine spin
> > > > up with the latest packages.  I've always found that updating a pre-built
> > > > machine is slower, sometimes substantially so, than just building a fresh
> > > > image with the newest rpms.
> > > > That said, some roles can (and often should) be fairly rigid and slow to
> > > > be updated.  But there's not much less of a need for flexible, dynamic
> > > > builds in the cloud than there is in a local server room; do you build all
> > > > new local servers based on a pre-built image that you just replicate?  Would
> > > > seem to negate the purpose of a kickstart server ;)
> > > > Brian
> > > > _______________________________________________
> > > > cloud mailing list
> > > > [e-mail]
> > > > https://admin.fedoraproject.org/mailman/listinfo/cloud
>
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

[Brian LaMere]

Apologies Raphael - just waking up, didn't notice you said where you
got the kernel.  Yes, it probably means that the kernel at:

http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...

doesn't have the XSAVE patch.  It probably doesn't work in to what
they consider their target audience for that kernel ;)  Whether they'd
take a bug report on that...I don't know.

On Fri, Mar 18, 2011 at 9:31 AM, Brian LaMere
 wrote:
> Raphael - there's more than just a memory difference between micro and
> small; micros are 64bit, smalls are 32bit.  If you try the same thing
> on a large, do you have the same problem?  Small and medium are both
> 32bit, micro, large, and every other size are all 64bit.
>
> The initial error of "base is 0x26617ba8 caller is 0x45c87" looks like
> something calling a 32 against a 64.  Someone smarter than me
> suggested this means that XSAVE isn't disabled in the kernel; that
> would mean the anaconda kernel itself, which is pre-built.  Maybe get
> with the anaconda folks to verify whether the CD kernel has the XSAVE
> patch?  The kernel in the distro does, otherwise we wouldn't be able
> to use it ;)  But that doesn't mean the kernel in CDHOME/isolinux
> does.
>
> Does what I'm asking/suggesting make sense?
>
> Brian
>
> 2011/3/18 Raphaël De GIUSTI :
> > It seems to be happening when launching micro instances, as memory is low
> > (about 600M)
> > Launching it as a small instance, anaconda is executed, but there seems to
> > be a problem related with dbus :
> > Greetings.
> > anaconda installer init version 15.20.1 starting
> > mounting /proc filesystem... done
> >
> > (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> > `hash_table != NULL' failed
> > creating /dev filesystem... done
> > starting udev...done
> > mounting /dev/pts (unix98 pty) filesystem... done
> > mounting /sys filesystem... done
> >
> > (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> > `hash_table != NULL' failed
> > anaconda installer init version 15.20.1 using /dev/hvc0 as console
> > trying to remount root filesystem read write... done
> > mounting /tmp as tmpfs... done
> >
> > (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> > `hash_table != NULL' failed
> > running install...
> > running /sbin/loader
> >
> > %Gdetecting hardware...
> > waiting for hardware to initialize...
> > detecting hardware...
> > waiting for hardware to initialize...
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> >
> > 2011/3/10 Raphaël De GIUSTI
> > > Hi,
> > > I simply tried to boot on the fedora 15 alpha kernel from there :
> > >
> > > http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...
> > > and I'm getting this error :
> > > > root (hd0,0)
> > > >
> > > > Filesystem type is ext2fs, partition type 0x83
> > > > kernel /boot/vmlinuz-PAE
> > > > initrd /boot/initrd-PAE.img
> > > > ERROR: mmu_update failed with rc=-22
> > > > Do_exit called!
> > > > base is 0x26617ba8 caller is 0x45c87
> > > > base is 0x26617bd8 caller is 0x50336
> > > > base is 0x26617c38 caller is 0x504c0
> > > > base is 0x26617c88 caller is 0x506a3
> > > > base is 0x2661fd08 caller is 0x479d7
> > > > base is 0x2661fd48 caller is 0x5613b
> > > > base is 0x2661fd58 caller is 0x54698
> > > > base is 0x2661fd98 caller is 0x55a79
> > > > base is 0x2661fde8 caller is 0x55811
> > > > base is 0x2661fe08 caller is 0x3b0c
> > > > base is 0x2661fe38 caller is 0x3bc4
> > > > base is 0x2661fe48 caller is 0x7af7
> > > > base is 0x2661fe58 caller is 0xa243
> > > > base is 0x2661fe78 caller is 0xfe69
> > > > base is 0x2661fef8 caller is 0x10489
> > > > base is 0x2661ff68 caller is 0x3eb2
> > > > base is 0x2661ff78 caller is 0x4729d
> > > > base is 0x2661fff0 caller is 0x31ad
> > >
> > > I just wondered where it does come from ?
> > > Any idea ?
> > >
> > > 2011/2/8 Raphaël De GIUSTI
> > > > So if we put aside how the kickstart would be generated, what would be
> > > > the starting point in all of this ?
> > > > I was working with Centos55 when I made it to the partitioning phase (and
> > > > I don't really know why it went wrong), but I couldn't even get the F14
> > > > kernel to load.
> > > > In other words :
> > > > Which fedora kernel / initrd combination should I use in my grub.conf ?
> > > > Should any of those two be altered in any way ?
> > > >
> > > > I'm also willing to put some time in it if I may be useful... but like I
> > > > said I'm no expert.
> > > > On Wed, Feb 2, 2011 at 9:48 PM, Brian LaMere 
> > > > wrote:
> > > > > > Maybe I'm missing something:  why would you ever want an instance to
> > > > > > kickstart at boot time?  You should create an image for every role you
> > > > > > care about and then boot the appropriate one for every instance you
> > > > > > need.
> > > > >
> > > > > roles change, updates happen frequently, and I'd rather a machine spin
> > > > > up with the latest packages.  I've always found that updating a pre-built
> > > > > machine is slower, sometimes substantially so, than just building a fresh
> > > > > image with the newest rpms.
> > > > > That said, some roles can (and often should) be fairly rigid and slow to
> > > > > be updated.  But there's not much less of a need for flexible, dynamic
> > > > > builds in the cloud than there is in a local server room; do you build all
> > > > > new local servers based on a pre-built image that you just replicate?  Would
> > > > > seem to negate the purpose of a kickstart server ;)
> > > > > Brian
> > > > > _______________________________________________
> > > > > cloud mailing list
> > > > > [e-mail]
> > > > > https://admin.fedoraproject.org/mailman/listinfo/cloud
> >
> > cloud mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/cloud

[Raphaël De GIUSTI]

Well, it seems that the "mmu_update" issue is related to instances memory
size . Micro instances' memory is too low to handle that kernel. Same kernel
boots on a small (or greater) instance. One of the AWS folks suggested to
prune down the kernel so it works on low memory instances.

So XSAVE is probably disabled in recent kernels.

The dbus issue seems to be Xen related with i386 kernels : EC2 small
instance uses the version 3.0.3-rc5-8.1.14.f and sometimes randomly uses the
good one, which is 3.4.3-2.6.18 (preserve-AD) that does fail on dbus.

So, the only one that's working is F14 x86_64 kernel on large instances and
anaconda is now loading properly (but hangs on kickstart's package
installation).

On Fri, Mar 18, 2011 at 5:45 PM, Brian LaMere wrote:
> Apologies Raphael - just waking up, didn't notice you said where you
> got the kernel.  Yes, it probably means that the kernel at:
>
> http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...
>
> doesn't have the XSAVE patch.  It probably doesn't work in to what
> they consider their target audience for that kernel ;)  Whether they'd
> take a bug report on that...I don't know.
>
> On Fri, Mar 18, 2011 at 9:31 AM, Brian LaMere
>  wrote:
> > Raphael - there's more than just a memory difference between micro and
> > small; micros are 64bit, smalls are 32bit.  If you try the same thing
> > on a large, do you have the same problem?  Small and medium are both
> > 32bit, micro, large, and every other size are all 64bit.
> >
> > The initial error of "base is 0x26617ba8 caller is 0x45c87" looks like
> > something calling a 32 against a 64.  Someone smarter than me
> > suggested this means that XSAVE isn't disabled in the kernel; that
> > would mean the anaconda kernel itself, which is pre-built.  Maybe get
> > with the anaconda folks to verify whether the CD kernel has the XSAVE
> > patch?  The kernel in the distro does, otherwise we wouldn't be able
> > to use it ;)  But that doesn't mean the kernel in CDHOME/isolinux
> > does.
> >
> > Does what I'm asking/suggesting make sense?
> >
> > Brian
> >
> > 2011/3/18 Raphaël De GIUSTI :
> > > It seems to be happening when launching micro instances, as memory is
>
> low
> > > (about 600M)
> > > Launching it as a small instance, anaconda is executed, but there seems
>
> to
> > > be a problem related with dbus :
> > > Greetings.
> > > anaconda installer init version 15.20.1 starting
> > > mounting /proc filesystem... done
> > >
> > > (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> > > `hash_table != NULL' failed
> > > creating /dev filesystem... done
> > > starting udev...done
> > > mounting /dev/pts (unix98 pty) filesystem... done
> > > mounting /sys filesystem... done
> > >
> > > (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> > > `hash_table != NULL' failed
> > > anaconda installer init version 15.20.1 using /dev/hvc0 as console
> > > trying to remount root filesystem read write... done
> > > mounting /tmp as tmpfs... done
> > >
> > > (process:1): GLib-CRITICAL **: g_hash_table_lookup_extended: assertion
> > > `hash_table != NULL' failed
> > > running install...
> > > running /sbin/loader
> > >
> > > %Gdetecting hardware...
> > > waiting for hardware to initialize...
> > > detecting hardware...
> > > waiting for hardware to initialize...
> > >
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > ** (loader:49): WARNING **: Couldn't connect to system bus: Failed to
> > > connect to socket /var/run/dbus/system_bus_socket: No such file or
>
> directory
> > > 2011/3/10 Raphaël De GIUSTI
> > > > Hi,
> > > > I simply tried to boot on the fedora 15 alpha kernel from there :
>
> http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/rele...
> > > > and I'm getting this error :
> > > > > root (hd0,0)
> > > > >
> > > > > Filesystem type is ext2fs, partition type 0x83
> > > > > kernel /boot/vmlinuz-PAE
> > > > > initrd /boot/initrd-PAE.img
> > > > > ERROR: mmu_update failed with rc=-22
> > > > > Do_exit called!
> > > > > base is 0x26617ba8 caller is 0x45c87
> > > > > base is 0x26617bd8 caller is 0x50336
> > > > > base is 0x26617c38 caller is 0x504c0
> > > > > base is 0x26617c88 caller is 0x506a3
> > > > > base is 0x2661fd08 caller is 0x479d7
> > > > > base is 0x2661fd48 caller is 0x5613b
> > > > > base is 0x2661fd58 caller is 0x54698
> > > > > base is 0x2661fd98 caller is 0x55a79
> > > > > base is 0x2661fde8 caller is 0x55811
> > > > > base is 0x2661fe08 caller is 0x3b0c
> > > > > base is 0x2661fe38 caller is 0x3bc4
> > > > > base is 0x2661fe48 caller is 0x7af7
> > > > > base is 0x2661fe58 caller is 0xa243
> > > > > base is 0x2661fe78 caller is 0xfe69
> > > > > base is 0x2661fef8 caller is 0x10489
> > > > > base is 0x2661ff68 caller is 0x3eb2
> > > > > base is 0x2661ff78 caller is 0x4729d
> > > > > base is 0x2661fff0 caller is 0x31ad
> > > >
> > > > I just wondered where it does come from ?
> > > > Any idea ?
> > > >
> > > > 2011/2/8 Raphaël De GIUSTI
> > > > > So if we put aside how the kickstart would be generated, what would be
> > > > > the starting point in all of this ?
> > > > > I was working with Centos55 when I made it to the partitioning phase
>
> (and
> > > > > I don't really know why it went wrong), but I couldn't even get the
>
> F14
> > > > > kernel to load.
> > > > > In other words :
> > > > > Which fedora kernel / initrd combination should I use in my
>
> grub.conf ?
> > > > > Should any of those two be altered in any way ?
> > > > >
> > > > > I'm also willing to put some time in it if I may be useful... but like
>
> I
> > > > > said I'm no expert.
> > > > > On Wed, Feb 2, 2011 at 9:48 PM, Brian LaMere <
>
> [e-mail]>
> > > > > wrote:
> > > > > > >
> > > > > > > Maybe I'm missing something:  why would you ever want an instance to
> > > > > > > kickstart at boot time?  You should create an image for every role
>
> you
> > > > > > > care about and then boot the appropriate one for every instance you
> > > > > > > need.
> > > > > >
> > > > > > roles change, updates happen frequently, and I'd rather a machine
>
> spin
> > > > > > up with the latest packages.  I've always found that updating a
>
> pre-built
> > > > > > machine is slower, sometimes substantially so, than just building a
>
> fresh
> > > > > > image with the newest rpms.
> > > > > > That said, some roles can (and often should) be fairly rigid and slow
>
> to
> > > > > > be updated.  But there's not much less of a need for flexible,
>
> dynamic
> > > > > > builds in the cloud than there is in a local server room; do you
>
> build all
> > > > > > new local servers based on a pre-built image that you just replicate?
>
> Would
> > > > > > seem to negate the purpose of a kickstart server ;)
> > > > > > Brian
> > > > > > _______________________________________________
> > > > > > cloud mailing list
> > > > > > [e-mail]
> > > > > > https://admin.fedoraproject.org/mailman/listinfo/cloud
> > >
> > > cloud mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/cloud
>
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud