Estimate the size of iso file based on a kickstart file

[Kalpa Welivitigoda]

Hi all,

Given a kickstart file (flatten) and we intend to make an iso using
it, is there a tool or service by which we can estimate the size of
the final iso (based on the packages defined in the kickstart file)
before actually creating the iso?

-- 
Best Regards,

Kalpa Welivitigoda
http://about.me/callkalpa

[Nico Kadel-Garcia]

On Thu, Sep 19, 2019 at 8:11 AM Kalpa Welivitigoda <callkalpa(a)gmail.com&gt; wrote:
>  Hi all,
>
>  Given a kickstart file (flatten) and we intend to make an iso using
>  it, is there a tool or service by which we can estimate the size of
>  the final iso (based on the packages defined in the kickstart file)
>  before actually creating the iso? 
Since a kickstart file can refer to RPM's stored on the ISO file, or
on a separate website, and those RPM's are the majority of the bulk of
an installation DVD, it's not clear you can gracefully do this. A
single RPM file can drag in a *lot* of dependencies that may change
based on upstream updates. Why try to guess rather than simply running
the tool, which will be pretty fast if you work from a local yum
mirror?

[John Florian]

On 9/19/19 8:25 AM, Nico Kadel-Garcia wrote:
>  On Thu, Sep 19, 2019 at 8:11 AM Kalpa Welivitigoda
> <callkalpa(a)gmail.com&gt; wrote:
> > Hi all,
> >
> > Given a kickstart file (flatten) and we intend to make an iso using
> > it, is there a tool or service by which we can estimate the size of
> > the final iso (based on the packages defined in the kickstart file)
> > before actually creating the iso?
>  Since a kickstart file can refer to RPM's stored on the ISO file, or
>  on a separate website, and those RPM's are the majority of the bulk of
>  an installation DVD, it's not clear you can gracefully do this. A
>  single RPM file can drag in a *lot* of dependencies that may change
>  based on upstream updates. Why try to guess rather than simply running
>  the tool, which will be pretty fast if you work from a local yum
>  mirror? 

Doesn't anaconda do a test to see if the install will fit?  I vaguely
recall (or imagine, with age it seems there's little difference
sometimes) seeing such a message.  If so, that would handle all the
dependency recursion and from there a statistical factor for the
expected compression should get a reasonably close estimate, no?

-- 
John Florian

[Brian C. Lane]

On Thu, Sep 19, 2019 at 08:10:21PM +0800, Kalpa Welivitigoda wrote:
>  Hi all,
>  
>  Given a kickstart file (flatten) and we intend to make an iso using
>  it, is there a tool or service by which we can estimate the size of
>  the final iso (based on the packages defined in the kickstart file)
>  before actually creating the iso? 
AFAIK we don't have any stand-alone tools to do that, which is probably
a good thing since trying to estimate the installed size usually ends up
being very much not exact. In lorax-composer I have this:

https://github.com/weldr/lorax/blob/master/src/pylorax/api/projects.py#L288

which does the raw estimation. But then there is this:

https://github.com/weldr/lorax/blob/master/src/pylorax/api/compose.py#L738

Which adds 20%

But wait, that's not all! Anaconda then adds more:

https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/payload/dn...

The reason behind all of this fudging is that we really don't know the
exact sizes of the installed packages, and it can be influenced by block
size, filesystem type, etc.

Even with all this added extra space I have occasionally seen it run out
of space (usually when trying to do a smaller install than a bigger one,
where the room for error is smaller).

The 'best' way to do what you want is to put together your kickstart,
give it a big / partition, do a build and see how much free space there
is, and trim it down. But not too far :)

Brian

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart

[Martin Kolman]

On Tue, 2019-09-24 at 11:32 -0700, Brian C. Lane wrote:
>  On Thu, Sep 19, 2019 at 08:10:21PM +0800, Kalpa Welivitigoda wrote:
>  > Hi all,
>  > 
>  > Given a kickstart file (flatten) and we intend to make an iso using
>  > it, is there a tool or service by which we can estimate the size of
>  > the final iso (based on the packages defined in the kickstart file)
>  > before actually creating the iso?
>  
>  AFAIK we don't have any stand-alone tools to do that, which is probably
>  a good thing since trying to estimate the installed size usually ends up
>  being very much not exact. In lorax-composer I have this:
>  
>  https://github.com/weldr/lorax/blob/master/src/pylorax/api/projects.py#L288
>  
>  which does the raw estimation. But then there is this:
>  
>  https://github.com/weldr/lorax/blob/master/src/pylorax/api/compose.py#L738
>  
>  Which adds 20%
>  
>  But wait, that's not all! Anaconda then adds more:
>  
>
> https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/payload/dn...
>  
>  The reason behind all of this fudging is that we really don't know the
>  exact sizes of the installed packages, and it can be influenced by block
>  size, filesystem type, etc. 
Exactly, just for fun I'll try to enumerate the
factors I can remember:
- even an empty filesystem has to store it's metadata inside itself, so there will
always be less
  useable space on a filesystem than it's raw size, this value depends on the
filesystem and possibly
  other factors
- some older/simpler filesystems (such as FAT32) are very inefficient at storing many
small files
  which can further increase actual needed ammount of storage space
- while RPMs do contain unpacked size of all the files inside, it does not list size per
file,
  so if you have a separate mountpoint for say var, you can't know how much data will
be installed
  in the given location by a RPM (but this is likely not a problem for image size)
- RPMs can do arbitrary stuff in their scriptlets, including cache generation, that can
take an arbitrary ammount o
  space that simply can't be estimated beforehand
- when configuring the system Anaconda writes out config files, enables services and
similar, which also uses up
  some space, though very very minor amount
- kickstart describing the installation and installaiton logs are stored on the system by
default, those could
  eat up some more space - a couple MB
- AFAIK images generally use squashfs, so the content of its filesystem will be
compressed, reducing it size,
  but again in a hard to tell beforehadn manner, based on how well the filesystem content
compresses with the
  given algorithm used

So there is indeed quite a few variables, many with values uknown until things are already
in place,
which is why the various percentual & absolute size adjustments are done.

Technically, I guess it might be possible to just run the size estimation code for a
kickstart file
and then just output the estimated number instead of starting an installation/image
generation if someone
finds that useful & wirtes the needed patches. :)

>  
>  Even with all this added extra space I have occasionally seen it run out
>  of space (usually when trying to do a smaller install than a bigger one,
>  where the room for error is smaller).
>  
>  The 'best' way to do what you want is to put together your kickstart,
>  give it a big / partition, do a build and see how much free space there
>  is, and trim it down. But not too far :) 
Definitely, this is the only real way to
get real numbers with the way our seoftware & its installation process is
built. Actually, I would suggest this to a regular thing, done automatically say nightly,
with the results being
measured & notification being sent when when a given size threashold is reached.

AFAIK nothing like this is done currently and we basically only catch any signifficant
size increases once they break
things, which is kinda late & makes fixing things harder.


>  
>  Brian
>  
>  -- 
>  Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]