Anaconda Light for Servers

[Sean M]

Hi all,

Firstly, thank you for the work on Anaconda/kickstart, it's very reliable and is pleasant to work with.

Like many others, I've been busy upgrading several hundred machines from EL7 to EL9 and in doing this I've had to work around a size/RAM limitation in Anaconda.

Hundreds of machines I've been upgrading are physical, outside the country I reside in, and do not have an OOB interface (effectively only accessible over ssh) making the upgrade process a bit perilous.

When the machine has enough RAM we can boot into the pxeboot/images/[vmlinuz/initrd.img] pair and source the stage2 and the rest of the things we need from the network. 
When the machine _does not_ have enough RAM, we have to repartition the disk (a combination of XFS/LVM prevent shrinking, the /boot partition is too small, and sourcing stage2 from a disk targeted to be wiped and repartitioned is marked as protected stopping the installer) with a separate partition labeled "installer", download the vmlinuz/initrd.img/install.img there, and booting with `inst.stage2=hd:LABEL=installer`.

This "works", but the extra step to boot the installer off the disk instead of the network seems like it could be avoided. I know there are many ways to solve this problem (cat an image over the network and dd it from a rescue environment, or something along those lines), but in this case the wide variety of network configurations, limited access, and time/experience constraints didn't really allow for much experimenting. Anaconda and the kickstart work reliably, and we have the most experience with that.

I know this is asking a lot, but as I do not have the expertise to go down this road myself, is it possible to have a build of Anaconda that does not contain the graphical interface to keep the size down? I think this could save a lot of time / resources / e-waste in the server/sysadmin world.

Thanks,
Sean

[Brian C. Lane]

On Tue, Aug 13, 2024 at 03:12:16PM -0000, Sean M wrote:
> I know this is asking a lot, but as I do not have the expertise to go down this road myself, is it possible to have a build of Anaconda that does not contain the graphical interface to keep the size down? I think this could save a lot of time / resources / e-waste in the server/sysadmin world.

The last time I looked into that it didn't actually save that much
space. FYI another option for you is to setup an nfs server that's
available to the systems you are booting and run stage2 from there
instead of having to load it all into memory.

Brian

[Sean Mottles]

Gotcha, that's unfortunate. As for NFS, these machines are with hundreds 
of different hosts/networks, and getting NFS allowed/through each one is 
untenable, thank you for the suggestion though!

Sean

On 8/13/24 5:55 PM, Brian C. Lane wrote:
> On Tue, Aug 13, 2024 at 03:12:16PM -0000, Sean M wrote:
> > I know this is asking a lot, but as I do not have the expertise to go down this road myself, is it possible to have a build of Anaconda that does not contain the graphical interface to keep the size down? I think this could save a lot of time / resources / e-waste in the server/sysadmin world.
>
> The last time I looked into that it didn't actually save that much
> space. FYI another option for you is to setup an nfs server that's
> available to the systems you are booting and run stage2 from there
> instead of having to load it all into memory.
>
> Brian

On Tue, 2024-08-13 at 15:12 +0000, Sean M wrote:
> Hi all,
>
> Firstly, thank you for the work on Anaconda/kickstart, it's very
> reliable and is pleasant to work with.
>
> Like many others, I've been busy upgrading several hundred machines
> from EL7 to EL9 and in doing this I've had to work around a size/RAM
> limitation in Anaconda.

Just making sure - are you running the installation in text mode ?
I it is doing an automated install driven by kickstart, the TUI should
be sufficient & end result the same. By not loading all the GUI stuff
to memory, the peak memory usage should be lower.

Also some other things to try:
- make sure there is a swap partition (IIRC, at least when we create
one, we will use it during the installation), which should also reduce
peak memory pressure
- reduce number of packages being installed, possibly install the rest
after rebooting to the new system (should reduce peak memory pressure
caused by depsolving)
- create a custom minimal image via Image Builder:
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-sin...
-> not sure if it can actually make a smaller image, but possibly still
worth checking out/asking around for options/submitting a RFE
-> I think it might be also able to build a non-RPM payload (eq. rootfs
tarball), which could reduce peak RAM pressure (no depsolving) at
installation time
> Hundreds of machines I've been upgrading are physical, outside the
> country I reside in, and do not have an OOB interface (effectively
> only accessible over ssh) making the upgrade process a bit perilous.
>
> When the machine has enough RAM we can boot into the
> pxeboot/images/[vmlinuz/initrd.img] pair and source the stage2 and
> the rest of the things we need from the network. 
> When the machine _does not_ have enough RAM, we have to repartition
> the disk (a combination of XFS/LVM prevent shrinking, the /boot
> partition is too small, and sourcing stage2 from a disk targeted to
> be wiped and repartitioned is marked as protected stopping the
> installer) with a separate partition labeled "installer", download
> the vmlinuz/initrd.img/install.img there, and booting with
> `inst.stage2=hd:LABEL=installer`.
>
> This "works", but the extra step to boot the installer off the disk
> instead of the network seems like it could be avoided. I know there
> are many ways to solve this problem (cat an image over the network
> and dd it from a rescue environment, or something along those lines),
> but in this case the wide variety of network configurations, limited
> access, and time/experience constraints didn't really allow for much
> experimenting. Anaconda and the kickstart work reliably, and we have
> the most experience with that.
>
> I know this is asking a lot, but as I do not have the expertise to go
> down this road myself, is it possible to have a build of Anaconda
> that does not contain the graphical interface to keep the size down?
> I think this could save a lot of time / resources / e-waste in the
> server/sysadmin world.
>
> Thanks,
> Sean

Best Wishes
Martin

[Sean Mottles]

Yep, it's in text mode. The issue arises early, when it fetches the 
stage2 over the network it fails with "no space left on device", so 
before Anaconda starts up/the kickstart is fetched. I was hoping the 
removal of the graphical portions of Anaconda would at least allow the 
installer to start up/avoid that problem, but it doesn't sound like it 
from Brian's reply.

Thanks for your suggestions! How would I submit an RFE? I have worked 
around this issue with our unique constraints, but a slimmed down 
"kickstart/tui exclusive" version of Anaconda still seems like it could 
be a useful thing for something like the Fedora Server SIG or even the 
greater "EL" community who face similar constraints. These machines can 
run EL9 without issue, but are just unable to install it over the network.

Best,

Sean

On 8/14/24 11:09 AM, [e-mail] wrote:
> On Tue, 2024-08-13 at 15:12 +0000, Sean M wrote:
> > Hi all,
> >
> > Firstly, thank you for the work on Anaconda/kickstart, it's very
> > reliable and is pleasant to work with.
> >
> > Like many others, I've been busy upgrading several hundred machines
> > from EL7 to EL9 and in doing this I've had to work around a size/RAM
> > limitation in Anaconda.
>
> Just making sure - are you running the installation in text mode ?
> I it is doing an automated install driven by kickstart, the TUI should
> be sufficient & end result the same. By not loading all the GUI stuff
> to memory, the peak memory usage should be lower.
>
> Also some other things to try:
> make sure there is a swap partition (IIRC, at least when we create
>
> one, we will use it during the installation), which should also reduce
> peak memory pressure
> reduce number of packages being installed, possibly install the rest
>
> after rebooting to the new system (should reduce peak memory pressure
> caused by depsolving)
> create a custom minimal image via Image Builder:
>
> https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-sin...
> -> not sure if it can actually make a smaller image, but possibly still
> worth checking out/asking around for options/submitting a RFE
> -> I think it might be also able to build a non-RPM payload (eq. rootfs
> tarball), which could reduce peak RAM pressure (no depsolving) at
> installation time
> > Hundreds of machines I've been upgrading are physical, outside the
> > country I reside in, and do not have an OOB interface (effectively
> > only accessible over ssh) making the upgrade process a bit perilous.
> >
> > When the machine has enough RAM we can boot into the
> > pxeboot/images/[vmlinuz/initrd.img] pair and source the stage2 and
> > the rest of the things we need from the network.
> > When the machine _does not_ have enough RAM, we have to repartition
> > the disk (a combination of XFS/LVM prevent shrinking, the /boot
> > partition is too small, and sourcing stage2 from a disk targeted to
> > be wiped and repartitioned is marked as protected stopping the
> > installer) with a separate partition labeled "installer", download
> > the vmlinuz/initrd.img/install.img there, and booting with
> > `inst.stage2=hd:LABEL=installer`.
> >
> > This "works", but the extra step to boot the installer off the disk
> > instead of the network seems like it could be avoided. I know there
> > are many ways to solve this problem (cat an image over the network
> > and dd it from a rescue environment, or something along those lines),
> > but in this case the wide variety of network configurations, limited
> > access, and time/experience constraints didn't really allow for much
> > experimenting. Anaconda and the kickstart work reliably, and we have
> > the most experience with that.
> >
> > I know this is asking a lot, but as I do not have the expertise to go
> > down this road myself, is it possible to have a build of Anaconda
> > that does not contain the graphical interface to keep the size down?
> > I think this could save a lot of time / resources / e-waste in the
> > server/sysadmin world.
> >
> > Thanks,
> > Sean
>
> Best Wishes
> Martin

[Jiri Konecny]

Hi,

feel free to file an RFE here: https://issues.redhat.com/projects/RHEL/

Unfortunately, not sure how can I help you in this case. I guess you can 
clone the disk image to the system if machines are the same and avoid 
installation step.

Just from curiosity, how much RAM do the machines have?

Best Regards,
Jirka

On 14. 08. 24 14:48, Sean Mottles wrote:
> Yep, it's in text mode. The issue arises early, when it fetches the 
> stage2 over the network it fails with "no space left on device", so 
> before Anaconda starts up/the kickstart is fetched. I was hoping the 
> removal of the graphical portions of Anaconda would at least allow the 
> installer to start up/avoid that problem, but it doesn't sound like it 
> from Brian's reply.
>
> Thanks for your suggestions! How would I submit an RFE? I have worked 
> around this issue with our unique constraints, but a slimmed down 
> "kickstart/tui exclusive" version of Anaconda still seems like it 
> could be a useful thing for something like the Fedora Server SIG or 
> even the greater "EL" community who face similar constraints. These 
> machines can run EL9 without issue, but are just unable to install it 
> over the network.
>
> Best,
>
> Sean
>
> On 8/14/24 11:09 AM, [e-mail] wrote:
> > On Tue, 2024-08-13 at 15:12 +0000, Sean M wrote:
> > > Hi all,
> > >
> > > Firstly, thank you for the work on Anaconda/kickstart, it's very
> > > reliable and is pleasant to work with.
> > >
> > > Like many others, I've been busy upgrading several hundred machines
> > > from EL7 to EL9 and in doing this I've had to work around a size/RAM
> > > limitation in Anaconda.
> >
> > Just making sure - are you running the installation in text mode ?
> > I it is doing an automated install driven by kickstart, the TUI should
> > be sufficient & end result the same. By not loading all the GUI stuff
> > to memory, the peak memory usage should be lower.
> >
> > Also some other things to try:
> > make sure there is a swap partition (IIRC, at least when we create
> >
> > one, we will use it during the installation), which should also reduce
> > peak memory pressure
> > reduce number of packages being installed, possibly install the rest
> >
> > after rebooting to the new system (should reduce peak memory pressure
> > caused by depsolving)
> > create a custom minimal image via Image Builder:
> >
> > https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-sin...
> >
> > -> not sure if it can actually make a smaller image, but possibly still
> > worth checking out/asking around for options/submitting a RFE
> > -> I think it might be also able to build a non-RPM payload (eq. rootfs
> > tarball), which could reduce peak RAM pressure (no depsolving) at
> > installation time
> > > Hundreds of machines I've been upgrading are physical, outside the
> > > country I reside in, and do not have an OOB interface (effectively
> > > only accessible over ssh) making the upgrade process a bit perilous.
> > >
> > > When the machine has enough RAM we can boot into the
> > > pxeboot/images/[vmlinuz/initrd.img] pair and source the stage2 and
> > > the rest of the things we need from the network.
> > > When the machine _does not_ have enough RAM, we have to repartition
> > > the disk (a combination of XFS/LVM prevent shrinking, the /boot
> > > partition is too small, and sourcing stage2 from a disk targeted to
> > > be wiped and repartitioned is marked as protected stopping the
> > > installer) with a separate partition labeled "installer", download
> > > the vmlinuz/initrd.img/install.img there, and booting with
> > > `inst.stage2=hd:LABEL=installer`.
> > >
> > > This "works", but the extra step to boot the installer off the disk
> > > instead of the network seems like it could be avoided. I know there
> > > are many ways to solve this problem (cat an image over the network
> > > and dd it from a rescue environment, or something along those lines),
> > > but in this case the wide variety of network configurations, limited
> > > access, and time/experience constraints didn't really allow for much
> > > experimenting. Anaconda and the kickstart work reliably, and we have
> > > the most experience with that.
> > >
> > > I know this is asking a lot, but as I do not have the expertise to go
> > > down this road myself, is it possible to have a build of Anaconda
> > > that does not contain the graphical interface to keep the size down?
> > > I think this could save a lot of time / resources / e-waste in the
> > > server/sysadmin world.
> > >
> > > Thanks,
> > > Sean
> >
> > Best Wishes
> > Martin

[Sean M]

Hi,

Thanks!

Unfortunately we're not in possession of them, but we have been providing custom ISO's just in case things go wrong. The hope was that we could netinstall the machines ourselves without needing the host to do anything.

The majority of these physical machines have 1426132 kB, but I've noticed that the pure netinstall fails to download the install.img file with less than 2265508 kB no matter the platform.

Best,

Sean

[Jan Stodola]

Hi,
it's possible to try upgrades RHEL-7 -> 8 -> 9:

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/upg...
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/upg...

Another option might be creating your own RHEL-9 installation images using
lorax with modified templates (from the lorax-templates-rhel package) and
removing more kernel drivers or firmware files (or other files) that you do
not need for the installation process. Obviously, this is an unsupported
path.

Regards,
Jan

On Tue, Aug 13, 2024 at 5:12 PM Sean M  wrote:
> Hi all,
>
> Firstly, thank you for the work on Anaconda/kickstart, it's very reliable
> and is pleasant to work with.
>
> Like many others, I've been busy upgrading several hundred machines from
> EL7 to EL9 and in doing this I've had to work around a size/RAM limitation
> in Anaconda.
>
> Hundreds of machines I've been upgrading are physical, outside the country
> I reside in, and do not have an OOB interface (effectively only accessible
> over ssh) making the upgrade process a bit perilous.
>
> When the machine has enough RAM we can boot into the
> pxeboot/images/[vmlinuz/initrd.img] pair and source the stage2 and the rest
> of the things we need from the network.
> When the machine _does not_ have enough RAM, we have to repartition the
> disk (a combination of XFS/LVM prevent shrinking, the /boot partition is
> too small, and sourcing stage2 from a disk targeted to be wiped and
> repartitioned is marked as protected stopping the installer) with a
> separate partition labeled "installer", download the
> vmlinuz/initrd.img/install.img there, and booting with
> `inst.stage2=hd:LABEL=installer`.
>
> This "works", but the extra step to boot the installer off the disk
> instead of the network seems like it could be avoided. I know there are
> many ways to solve this problem (cat an image over the network and dd it
> from a rescue environment, or something along those lines), but in this
> case the wide variety of network configurations, limited access, and
> time/experience constraints didn't really allow for much experimenting.
> Anaconda and the kickstart work reliably, and we have the most experience
> with that.
>
> I know this is asking a lot, but as I do not have the expertise to go down
> this road myself, is it possible to have a build of Anaconda that does not
> contain the graphical interface to keep the size down? I think this could
> save a lot of time / resources / e-waste in the server/sysadmin world.
>
> Thanks,
> Sean
> --
> _______________________________________________
> Anaconda-devel mailing list -- [e-mail]
> To unsubscribe send an email to
> [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Sean Mottles]

Hi Jan,

Thanks for the suggestions! We changed so much throughout the stack we 
felt it would be better to start each machine off with a fresh installation.

Admittedly I've never used lorax, but would the removal of 
drivers/firmware reduce the size of the install.img file? That's the 
part that can't fit over the network if using a machine with less than 
~3G of RAM.

Best,

Sean

On 8/15/24 2:46 PM, Jan Stodola wrote:
> Another option might be creating your own RHEL-9 installation images 
> using lorax with modified templates (from the lorax-templates-rhel 
> package) and removing more kernel drivers or firmware files (or other 
> files) that you do not need for the installation process. Obviously, 
> this is an unsupported path.

[Brian C. Lane]

On Thu, Aug 15, 2024 at 03:09:27PM +0000, Sean Mottles wrote:
> Hi Jan,
>
> Thanks for the suggestions! We changed so much throughout the stack we felt
> it would be better to start each machine off with a fresh installation.
>
> Admittedly I've never used lorax, but would the removal of drivers/firmware
> reduce the size of the install.img file? That's the part that can't fit over
> the network if using a machine with less than ~3G of RAM.

Yes, uncompressed they currently take about 400M and they don't compress
very well so if you know what you don't need you could save a bunch of
space.

I have an older blog post that goes over how lorax works:
https://www.brianlane.com/post/creating-the-anaconda-bootiso-with-lorax/
some things have changed since then but it should give you an idea of
how things work.

The basics are that you need a RHEL 9 VM and access to the RHEL9 BaseOS
and AppStream repositories. Install lorax and lorax-templates-rhel.

Make a copy of /usr/share/lorax/templates.d/80-rhel/
This contains the templates used to control how Lorax installs packages
and removes files.

Edit the runtime-install.tmpl to exclude more firmware, or change it to
only install specific firmware. Edit runtime-cleanup.tmpl to remove
parts of firmware packages that aren't needed.

Run lorax with your custom templates (pass --sharedir to lorax to point
to the directory with your changes, otherwise it will use the ones from
80-rhel).

Good luck!

Brian

[Sean M]

Ah! Wonderful! I've managed to get our setup working in 1800MB of RAM using lorax, thanks so much. I'll need to do some more testing as I've ripped out many things (all the wifi firmware I could find, all the fonts/themes, all graphical bits, replaced the locale archive with glibc-langpack-en), but this looks promising.