Mash slowness update

[Dennis Gilmore]

Hi all,

So in debugging why mash is so slow. we had a realisation, one thing it was 
supposed to always be doing never worked.  and that is hardlinking the 
packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning 
everything under /mnt/koji/packages and the compose processes all running as 
masher. the hardlink failed due to permission errors

the rawhide mash today took ~3 hours to put all the files into place in 
/mnt/koji/mash as shown by some snippets from the mash.log
2015-05-05 05:25:51 mash: Writing out files for 
/mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
2015-05-05 08:11:43 mash: createrepo: starting 
/mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...

when run as apache in a test putting all the files in place in /mnt/koji/mash 
took ~4 minutes
2015-05-05 19:27:07 mash: Writing out files for 
/mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages...
2015-05-05 19:30:54 mash: createrepo: starting 
/mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages... 

So that is a massive massive savings, both in the time it will take to mash 
the tree, but also the disk we use on /mnt/koji  right now we are using 27T of 
disk, there will be large chunks of duplicated data.  so now we need to adjust 
a bunch of processes. we need to have bodhi run mash as apache, we need to do 
the nightly branched and rawhide as apache, and we need to change how we do 
TC's and RC's so that they run as apache also.  however for TC's and RC's I 
will propose we make the change for F23 so as to not shake up things too much 
for F22 final. we do not want to be responsible for any slip.

There is likely other areas of mash that can do with improvement.  we need to 
run some mashes with debugging enabled to see how exactly drpm creation goes 
as It seems to be a big slowdown point.

Dennis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.fedoraproject.org/pipermail/rel-eng/attachments/20150505/3be2715a/attachment.sig>