Koji build root caching

[Dennis Gilmore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 14 Jan 2015 16:40:35 +0100
Vít Ondruch <vondruch at redhat.com> wrote:

> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Dne 14.1.2015 v 16:00 Dennis Gilmore napsal(a):
> > On Wed, 14 Jan 2015 14:57:59 +0100
> > Miroslav Suchý <msuchy at redhat.com> wrote:
> >
> > > On 01/13/2015 06:01 PM, Dennis Gilmore wrote:
> > >> that all being said. koji doesn't use any caching and will not
> > >> use the lvm plugin. we make every buildroot from scratch using a
> > >> fully clean environment to help with ensuring reproducability.
> >
> > > You can cache and still preserve reproducability. What I'm
> > > planning for Copr is to do (every week/month) for chroot in
> > > fedora-20-x86_64 fedora-21_86_64 ... ; do mock --init $chroot
> > >   done
> > > take snapshot of that. I plan to do that on VM level.
> > > And when new task come, I will just restore from that snapshot.
> > > And mock will start with already populated cache. So I will have
> > > better caching and yet reproducability.
> >
> > you really can't.  you would need to make a new cache any time one
> > of the packages in the minimal buildroot changes.
> 
> Actually this is not anytime. newRepo has to be run, which is not run
> more then 4 times per hour I'd say. If the new snapshot is prepared as
> part of newRepo task, the mock could reuse the snapshot.

ok I have a data point here https://ausil.fedorapeople.org/buildroots
the file in the link is the output of a sql query on koji's db it gives
the number of times a unique repo was used on a builder to do a build.
so basically it tells you if we cached the buildroots somehow  how many
times they would get used. 

$ grep "^ 19 |" buildroots|wc -l
1
$ grep "^ 18 |" buildroots|wc -l
0
$ grep "^ 17 |" buildroots|wc -l
0
$ grep "^ 16 |" buildroots|wc -l
2
$ grep "^ 15 |" buildroots|wc -l
1
$ grep "^ 14 |" buildroots|wc -l
0
$ grep "^ 13 |" buildroots|wc -l
8
$ grep "^ 12 |" buildroots|wc -l
13
$ grep "^ 11 |" buildroots|wc -l
14
$ grep "^ 10 |" buildroots|wc -l
15
$ grep "^  9 |" buildroots|wc -l
20
$ grep "^  8 |" buildroots|wc -l
24
$ grep "^  7 |" buildroots|wc -l
40
$ grep "^  6 |" buildroots|wc -l
59
$ grep "^  5 |" buildroots|wc -l
84
$ grep "^  4 |" buildroots|wc -l
155
$ grep "^  3 |" buildroots|wc -l
409
$ grep "^  2 |" buildroots|wc -l
1446
$ grep "^  1 |" buildroots|wc -l
4794

What I get from this is that in the majority of cases caching won't
help at all.  The cost of creating the cache is higher than the benefit
we would get.

Dennis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJUtvHgAAoJEH7ltONmPFDRP/sP/1iteUyvThkZLQGH4o+MY4uX
oIwdgwQQwkkjvpCyjFP1yyO5aJa/BTiOGSMtT0vQtnfAl+qYs7IiqGH6VJkNt+Io
Lmqr0yItXLr6ZyTwuKha2XYa88g+q657VmaJT/OmXGqi0Ezx8Rj/6NLXcV6twgKW
LD3YjxtPmyxX79p7RrkrI5+iJ+Iyw48HLepZup+XOfydOJ0RWZccRVlm0c0MYzt1
P6H0M+iiDikLl2NE1/jCTx+/P9yQNSQ8zhMVCjhevGiHSHmhIZmcpPib/IYGfRBN
OmgTaQ9XSaePNNrE4iqq6bQQ9fKqpA1FDlj1U2IaEHTh0CQ9zLCqKBuOl0ky98zY
/m3obzNSYu441tl/1Ig6imzRXga/iQ0jo8WnQoW8Grh3JtGpNM6NVPEUkkyKWCxE
XuagkPGWqP7x8YT/czn0Y9ZGjA+jIijzhdcAJ8R9R94wuhy+wvY5F69z6FKSycX6
Mho6yGj6cOrIB+F3Vs2+zn4rmtsU8jT7uhYc81CHJ0pEilcnINvFW9VDw7gYKq1E
LFzxMeixQugtGO39PqVFdG7WErqX2pjInJLu/CZIVz0bofQ17RuE006yl1pS4+jG
HoF6pCbZlapsRWS6TqXiMPhTSruszbZ733zihOgbDNSB6kIXZgI9nqj/jN8s8Dw1
ikGLTDN8nrnFxWMSt9pY
=/MGg
-----END PGP SIGNATURE-----