Mash slowness update

[Ralph Bean]

On Tue, May 05, 2015 at 04:16:55PM -0500, Dennis Gilmore wrote:
> Hi all,
> 
> So in debugging why mash is so slow. we had a realisation, one thing it was 
> supposed to always be doing never worked.  and that is hardlinking the 
> packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning 
> everything under /mnt/koji/packages and the compose processes all running as 
> masher. the hardlink failed due to permission errors
> 
> the rawhide mash today took ~3 hours to put all the files into place in 
> /mnt/koji/mash as shown by some snippets from the mash.log
> 2015-05-05 05:25:51 mash: Writing out files for 
> /mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
> 2015-05-05 08:11:43 mash: createrepo: starting 
> /mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
> 
> when run as apache in a test putting all the files in place in /mnt/koji/mash 
> took ~4 minutes
> 2015-05-05 19:27:07 mash: Writing out files for 
> /mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages...
> 2015-05-05 19:30:54 mash: createrepo: starting 
> /mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages... 
> 
> So that is a massive massive savings, both in the time it will take to mash 
> the tree, but also the disk we use on /mnt/koji  right now we are using 27T of 
> disk, there will be large chunks of duplicated data.  so now we need to adjust 
> a bunch of processes. we need to have bodhi run mash as apache, we need to do 
> the nightly branched and rawhide as apache, and we need to change how we do 
> TC's and RC's so that they run as apache also.  however for TC's and RC's I 
> will propose we make the change for F23 so as to not shake up things too much 
> for F22 final. we do not want to be responsible for any slip.
> 
> There is likely other areas of mash that can do with improvement.  we need to 
> run some mashes with debugging enabled to see how exactly drpm creation goes 
> as It seems to be a big slowdown point.
> 
> Dennis

Excellent find!

Changing everything to run as apache raises a flag for me, though. If
we just add the masher user to the apache group, would that give us
the same benefit without all the re-organization work?  Then we could:

1) continue to isolate things from the apache user that only the
   masher user should access
2) while the masher user could still (potentially) carry out that hard
   linking in an apache owned directory
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/rel-eng/attachments/20150506/8295d0c5/attachment.sig>