Performance increase for Koji builders

[Miroslav Suchý]

I have been doing some test for past two days and experimenting with
TmpFS plugin of Mock.
I summarized my findings here:

http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performan...

The performance increase is huge and I believe Koji can do the same.
I.e. enable tmpfs plugin, set the tmpfs size to some big number and
create create big swap - if there is not such big partition available,
it can be done as a file in some data partition.

Mirek

[Jason L Tibbitts III]

> >>>> "MS" == Miroslav Suchy
> <msuchy(a)redhat.com&gt; writes: 
MS> The performance increase is huge and I believe Koji can do the same.

I have always done my local mock builds in tmpfs and indeed the
performance increase is huge.  There were, however, odd problems in the
past with tmpfs not supporting some necessary features (like
capabilities, I think) which would get in the way of using it for
everything.  I don't know if that's solved now, and particularly if it's
solved on whatever OS our builders run currently.

 - J<

[Mike McLean]

On 05/28/2015 03:41 PM, Jason L Tibbitts III wrote:
> >>>>> "MS" == Miroslav Suchy
> <msuchy(a)redhat.com&gt; writes:
>
>  MS> The performance increase is huge and I believe Koji can do the same.
>
>  I have always done my local mock builds in tmpfs and indeed the
>  performance increase is huge.  There were, however, odd problems in the
>  past with tmpfs not supporting some necessary features (like
>  capabilities, I think) which would get in the way of using it for
>  everything.  I don't know if that's solved now, and particularly if it's
>  solved on whatever OS our builders run currently. 
Also, some builders do not necessarily have a lot of ram. Some 
buildroots are large, some sources are extremely large (e.g. texlive). 
If we do this, it will need to be configurable, probably per-package or 
per-tag.


from https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt
>  size:      The limit of allocated bytes for this tmpfs instance. The
>             default is half of your physical RAM without swap. If you
>             oversize your tmpfs instances the machine will deadlock
>             since the OOM handler will not be able to free that memory. 
"machine will deadlock" does not sound promising


Note that it should be possible to enable the tmpfs plugin in 
default.cfg on koji builders if folks want to play around with this. The 
mock configs that koji writes do not say anything about the tmpfs plugin.

[Miroslav Suchý]

Dne 28.5.2015 v 21:41 Jason L Tibbitts III napsal(a):
>  There were, however, odd problems in the
>  past with tmpfs not supporting some necessary features (like
>  capabilities, I think) 
I believe that this one is fixed since Fedora 15:
https://bugzilla.redhat.com/show_bug.cgi?id=648653

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Miroslav Suchý]

Dne 31.5.2015 v 21:50 Mike McLean napsal(a):
>  Also, some builders do not necessarily have a lot of ram. Some
> buildroots are large, some sources are extremely large
>  (e.g. texlive). If we do this, it will need to be configurable, probably per-package or
> per-tag. 
The devils in detail is that you do *not* need to have a lot of ram. Just a lot of swap
disk.

E.g. in Copr we had reserved 50 GB for mock chroot. And now instead of that, we now have
50 GB swap.
Therefore you never ever can have oversized tmpfs, because size of tmpfs is 50 GB, but
space for that is 5GB RAM + 50 GB
swap.
So you consume the same resource. And you get big performance increase.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Peter Robinson]

On Mon, Jun 1, 2015 at 1:08 PM, Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 31.5.2015 v 21:50 Mike McLean napsal(a):
> > Also, some builders do not necessarily have a lot of ram. Some buildroots are large,
> some sources are extremely large
> > (e.g. texlive). If we do this, it will need to be configurable, probably per-package
> or per-tag.
>
>  The devils in detail is that you do *not* need to have a lot of ram. Just a lot of swap
> disk. 
And a lot of swap disk can cause the IO issues that you mention and
ultimately lead to performance problems.

>  E.g. in Copr we had reserved 50 GB for mock chroot. And now instead
> of that, we now have 50 GB swap.
>  Therefore you never ever can have oversized tmpfs, because size of tmpfs is 50 GB, but
> space for that is 5GB RAM + 50 GB
>  swap. 
Yes, but it still filters down to IOPs on the disk one way or the other.

>  So you consume the same resource. And you get big performance
> increase. 
What package set did you test this against?

[Miroslav Suchý]

Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
>  And a lot of swap disk can cause the IO issues that you mention and
>  ultimately lead to performance problems. 
...
>  What package set did you test this against? 
http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performan...
6th paragraph - answer to both questions.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Peter Robinson]

On Mon, Jun 1, 2015 at 4:59 PM, Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
> > And a lot of swap disk can cause the IO issues that you mention and
> > ultimately lead to performance problems.
>  ...
> > What package set did you test this against?
>
>
> http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performan...
>  6th paragraph - answer to both questions. 
Blogs are great but it's also TL;DR .... sometimes bits here are better.

Kernel isn't a major issue compared to others. I'd like to see how
well it works for things like java, eclipse, libreoffice.

Peter

[Kevin Fenzi]

On Mon, 1 Jun 2015 17:15:42 +0100
Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:

>  On Mon, Jun 1, 2015 at 4:59 PM, Miroslav Suchý
> <msuchy(a)redhat.com&gt;
>  wrote:
>  > Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
>  >> And a lot of swap disk can cause the IO issues that you mention and
>  >> ultimately lead to performance problems.
>  > ...
>  >> What package set did you test this against?
>  >
>  >
> http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performan...
>  > 6th paragraph - answer to both questions.
>  
>  Blogs are great but it's also TL;DR .... sometimes bits here are
>  better.
>  
>  Kernel isn't a major issue compared to others. I'd like to see how
>  well it works for things like java, eclipse, libreoffice. 
To add some info, current configurations: 

Type / Memory / swap: 

arm / 4gb / 4gb
buildvm / 10gb / 2gb 
buildhw / 20gb / 0gb

The arm builders have 300GB drives. 
The buildvm's are on an iscsi lun and each has 150GB allocated. The lun
is 4.5TB, and there's 650gb or so free on it. 
The buildhw's have 2 300GB drives in a raid, so 300 usable. 

So, on the arm and buildhw's we could add a 50GB swap probibly at the
cost of the base / being down to 240-250gb. 
On the buildvm's we could only add 25GB or so, unless we increased the
space (which we may be able to do down the road, but cannot now). 

The thing I find odd about this is that the linux kernel caching
doesn't seem to be a win. Shouldn't it cache buildroot and such in
memory anyhow? Or why is tmpfs performing so much better?

We could also look at testing in staging and gather more data. 

kevin

[Josh Boyer]

On Mon, Jun 1, 2015 at 6:16 PM, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Mon, 1 Jun 2015 17:15:42 +0100
>  Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>
> > On Mon, Jun 1, 2015 at 4:59 PM, Miroslav Suchý <msuchy(a)redhat.com&gt;
> > wrote:
> > > Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
> > >> And a lot of swap disk can cause the IO issues that you mention and
> > >> ultimately lead to performance problems.
> > > ...
> > >> What package set did you test this against?
> > >
> > >
> http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performan...
> > > 6th paragraph - answer to both questions.
> >
> > Blogs are great but it's also TL;DR .... sometimes bits here are
> > better.
> >
> > Kernel isn't a major issue compared to others. I'd like to see how
> > well it works for things like java, eclipse, libreoffice.
>
>  To add some info, current configurations:
>
>  Type / Memory / swap:
>
>  arm / 4gb / 4gb
>  buildvm / 10gb / 2gb
>  buildhw / 20gb / 0gb
>
>  The arm builders have 300GB drives.
>  The buildvm's are on an iscsi lun and each has 150GB allocated. The lun
>  is 4.5TB, and there's 650gb or so free on it.
>  The buildhw's have 2 300GB drives in a raid, so 300 usable.
>
>  So, on the arm and buildhw's we could add a 50GB swap probibly at the
>  cost of the base / being down to 240-250gb.
>  On the buildvm's we could only add 25GB or so, unless we increased the
>  space (which we may be able to do down the road, but cannot now).
>
>  The thing I find odd about this is that the linux kernel caching
>  doesn't seem to be a win. Shouldn't it cache buildroot and such in
>  memory anyhow? Or why is tmpfs performing so much better? 
The page cache will cache pages as they are read and keep them in an
LRU fashion, yes.  But if workload is primarily I/O bound, it's a
combination of both reads and writes.  I suspect the majority of the
gain of tmpfs is the write out to memory instead of disk.  The page
cache will help with frequently used files (like the toolchain
executables), but all of output of the compiler is brand new and being
written frequently.

If we're talking about things happening in a VM, there are other
things in play as well.  Like interaction from other guests on the
physical disks, whether KSM is running on the host to share common
pages between guests, etc.  And if the guests are frequently being
created and torn down, that isn't going to help at all.

I will say that I'm surprised the kernel was that much of a win.
Particularly since it is bound to hit swap even with a 5GB tmpfs.  I
suppose some of that makes sense, as the kernel has massive output
that isn't read a lot.  The swap hit might be minimal in that case
because we aren't going to be paging a bunch of stuff back in from
disk for most of the build duration.  Instrumenting it would be still
be fun to see what all is in play.

josh

[Miroslav Suchý]

Dne 2.6.2015 v 00:16 Kevin Fenzi napsal(a):
>  The thing I find odd about this is that the linux kernel caching
>  doesn't seem to be a win. Shouldn't it cache buildroot and such in
>  memory anyhow? Or why is tmpfs performing so much better? 
When building mock.src.rpm it allocated about 1.5GB in tmpfs. Never done
any disk IO. When you rely on kernel caching it will still sync() every
few seconds (is it still 30 secs?). So it will write those 1.5GB to
disk. However I even seen a lot of reading in dstat output.

Mirek

[Miroslav Suchý]

Dne 1.6.2015 v 18:15 Peter Robinson napsal(a):
>  Kernel isn't a major issue compared to others. I'd like to
> see how
>  well it works for things like java, eclipse, libreoffice. 
It is quite time intensive. But I just run test for libreoffice
Normal time: 579m 21s (that is 9 hours, 39 minutes !!!)
With TmpFSL: 63m 49s

Mirek