I have been doing some test for past two days and experimenting with TmpFS plugin of Mock. I summarized my findings here:
http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_...
The performance increase is huge and I believe Koji can do the same. I.e. enable tmpfs plugin, set the tmpfs size to some big number and create create big swap - if there is not such big partition available, it can be done as a file in some data partition.
Mirek
"MS" == Miroslav Suchy msuchy@redhat.com writes:
MS> The performance increase is huge and I believe Koji can do the same.
I have always done my local mock builds in tmpfs and indeed the performance increase is huge. There were, however, odd problems in the past with tmpfs not supporting some necessary features (like capabilities, I think) which would get in the way of using it for everything. I don't know if that's solved now, and particularly if it's solved on whatever OS our builders run currently.
- J<
On 05/28/2015 03:41 PM, Jason L Tibbitts III wrote:
"MS" == Miroslav Suchy msuchy@redhat.com writes:
MS> The performance increase is huge and I believe Koji can do the same.
I have always done my local mock builds in tmpfs and indeed the performance increase is huge. There were, however, odd problems in the past with tmpfs not supporting some necessary features (like capabilities, I think) which would get in the way of using it for everything. I don't know if that's solved now, and particularly if it's solved on whatever OS our builders run currently.
Also, some builders do not necessarily have a lot of ram. Some buildroots are large, some sources are extremely large (e.g. texlive). If we do this, it will need to be configurable, probably per-package or per-tag.
from https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt
size: The limit of allocated bytes for this tmpfs instance. The default is half of your physical RAM without swap. If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory.
"machine will deadlock" does not sound promising
Note that it should be possible to enable the tmpfs plugin in default.cfg on koji builders if folks want to play around with this. The mock configs that koji writes do not say anything about the tmpfs plugin.
Dne 31.5.2015 v 21:50 Mike McLean napsal(a):
Also, some builders do not necessarily have a lot of ram. Some buildroots are large, some sources are extremely large (e.g. texlive). If we do this, it will need to be configurable, probably per-package or per-tag.
The devils in detail is that you do *not* need to have a lot of ram. Just a lot of swap disk.
E.g. in Copr we had reserved 50 GB for mock chroot. And now instead of that, we now have 50 GB swap. Therefore you never ever can have oversized tmpfs, because size of tmpfs is 50 GB, but space for that is 5GB RAM + 50 GB swap. So you consume the same resource. And you get big performance increase.
On Mon, Jun 1, 2015 at 1:08 PM, Miroslav Suchý msuchy@redhat.com wrote:
Dne 31.5.2015 v 21:50 Mike McLean napsal(a):
Also, some builders do not necessarily have a lot of ram. Some buildroots are large, some sources are extremely large (e.g. texlive). If we do this, it will need to be configurable, probably per-package or per-tag.
The devils in detail is that you do *not* need to have a lot of ram. Just a lot of swap disk.
And a lot of swap disk can cause the IO issues that you mention and ultimately lead to performance problems.
E.g. in Copr we had reserved 50 GB for mock chroot. And now instead of that, we now have 50 GB swap. Therefore you never ever can have oversized tmpfs, because size of tmpfs is 50 GB, but space for that is 5GB RAM + 50 GB swap.
Yes, but it still filters down to IOPs on the disk one way or the other.
So you consume the same resource. And you get big performance increase.
What package set did you test this against?
Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
And a lot of swap disk can cause the IO issues that you mention and ultimately lead to performance problems.
...
What package set did you test this against?
http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_... 6th paragraph - answer to both questions.
On Mon, Jun 1, 2015 at 4:59 PM, Miroslav Suchý msuchy@redhat.com wrote:
Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
And a lot of swap disk can cause the IO issues that you mention and ultimately lead to performance problems.
...
What package set did you test this against?
http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_... 6th paragraph - answer to both questions.
Blogs are great but it's also TL;DR .... sometimes bits here are better.
Kernel isn't a major issue compared to others. I'd like to see how well it works for things like java, eclipse, libreoffice.
Peter
On Mon, 1 Jun 2015 17:15:42 +0100 Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Jun 1, 2015 at 4:59 PM, Miroslav Suchý msuchy@redhat.com wrote:
Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
And a lot of swap disk can cause the IO issues that you mention and ultimately lead to performance problems.
...
What package set did you test this against?
http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_... 6th paragraph - answer to both questions.
Blogs are great but it's also TL;DR .... sometimes bits here are better.
Kernel isn't a major issue compared to others. I'd like to see how well it works for things like java, eclipse, libreoffice.
To add some info, current configurations:
Type / Memory / swap:
arm / 4gb / 4gb buildvm / 10gb / 2gb buildhw / 20gb / 0gb
The arm builders have 300GB drives. The buildvm's are on an iscsi lun and each has 150GB allocated. The lun is 4.5TB, and there's 650gb or so free on it. The buildhw's have 2 300GB drives in a raid, so 300 usable.
So, on the arm and buildhw's we could add a 50GB swap probibly at the cost of the base / being down to 240-250gb. On the buildvm's we could only add 25GB or so, unless we increased the space (which we may be able to do down the road, but cannot now).
The thing I find odd about this is that the linux kernel caching doesn't seem to be a win. Shouldn't it cache buildroot and such in memory anyhow? Or why is tmpfs performing so much better?
We could also look at testing in staging and gather more data.
kevin
On Mon, Jun 1, 2015 at 6:16 PM, Kevin Fenzi kevin@scrye.com wrote:
On Mon, 1 Jun 2015 17:15:42 +0100 Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Jun 1, 2015 at 4:59 PM, Miroslav Suchý msuchy@redhat.com wrote:
Dne 1.6.2015 v 16:58 Peter Robinson napsal(a):
And a lot of swap disk can cause the IO issues that you mention and ultimately lead to performance problems.
...
What package set did you test this against?
http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_... 6th paragraph - answer to both questions.
Blogs are great but it's also TL;DR .... sometimes bits here are better.
Kernel isn't a major issue compared to others. I'd like to see how well it works for things like java, eclipse, libreoffice.
To add some info, current configurations:
Type / Memory / swap:
arm / 4gb / 4gb buildvm / 10gb / 2gb buildhw / 20gb / 0gb
The arm builders have 300GB drives. The buildvm's are on an iscsi lun and each has 150GB allocated. The lun is 4.5TB, and there's 650gb or so free on it. The buildhw's have 2 300GB drives in a raid, so 300 usable.
So, on the arm and buildhw's we could add a 50GB swap probibly at the cost of the base / being down to 240-250gb. On the buildvm's we could only add 25GB or so, unless we increased the space (which we may be able to do down the road, but cannot now).
The thing I find odd about this is that the linux kernel caching doesn't seem to be a win. Shouldn't it cache buildroot and such in memory anyhow? Or why is tmpfs performing so much better?
The page cache will cache pages as they are read and keep them in an LRU fashion, yes. But if workload is primarily I/O bound, it's a combination of both reads and writes. I suspect the majority of the gain of tmpfs is the write out to memory instead of disk. The page cache will help with frequently used files (like the toolchain executables), but all of output of the compiler is brand new and being written frequently.
If we're talking about things happening in a VM, there are other things in play as well. Like interaction from other guests on the physical disks, whether KSM is running on the host to share common pages between guests, etc. And if the guests are frequently being created and torn down, that isn't going to help at all.
I will say that I'm surprised the kernel was that much of a win. Particularly since it is bound to hit swap even with a 5GB tmpfs. I suppose some of that makes sense, as the kernel has massive output that isn't read a lot. The swap hit might be minimal in that case because we aren't going to be paging a bunch of stuff back in from disk for most of the build duration. Instrumenting it would be still be fun to see what all is in play.
josh
Dne 2.6.2015 v 00:16 Kevin Fenzi napsal(a):
The thing I find odd about this is that the linux kernel caching doesn't seem to be a win. Shouldn't it cache buildroot and such in memory anyhow? Or why is tmpfs performing so much better?
When building mock.src.rpm it allocated about 1.5GB in tmpfs. Never done any disk IO. When you rely on kernel caching it will still sync() every few seconds (is it still 30 secs?). So it will write those 1.5GB to disk. However I even seen a lot of reading in dstat output.
Mirek
Dne 1.6.2015 v 18:15 Peter Robinson napsal(a):
Kernel isn't a major issue compared to others. I'd like to see how well it works for things like java, eclipse, libreoffice.
It is quite time intensive. But I just run test for libreoffice Normal time: 579m 21s (that is 9 hours, 39 minutes !!!) With TmpFSL: 63m 49s
Mirek
Dne 28.5.2015 v 21:41 Jason L Tibbitts III napsal(a):
There were, however, odd problems in the past with tmpfs not supporting some necessary features (like capabilities, I think)
I believe that this one is fixed since Fedora 15: https://bugzilla.redhat.com/show_bug.cgi?id=648653
buildsys@lists.fedoraproject.org
-
Jason L Tibbitts III -
Josh Boyer -
Kevin Fenzi -
Mike McLean -
Miroslav Suchý -
Peter Robinson