Hi all,
So in debugging why mash is so slow. we had a realisation, one thing it was supposed to always be doing never worked. and that is hardlinking the packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning everything under /mnt/koji/packages and the compose processes all running as masher. the hardlink failed due to permission errors
the rawhide mash today took ~3 hours to put all the files into place in /mnt/koji/mash as shown by some snippets from the mash.log 2015-05-05 05:25:51 mash: Writing out files for /mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages... 2015-05-05 08:11:43 mash: createrepo: starting /mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
when run as apache in a test putting all the files in place in /mnt/koji/mash took ~4 minutes 2015-05-05 19:27:07 mash: Writing out files for /mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages... 2015-05-05 19:30:54 mash: createrepo: starting /mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages...
So that is a massive massive savings, both in the time it will take to mash the tree, but also the disk we use on /mnt/koji right now we are using 27T of disk, there will be large chunks of duplicated data. so now we need to adjust a bunch of processes. we need to have bodhi run mash as apache, we need to do the nightly branched and rawhide as apache, and we need to change how we do TC's and RC's so that they run as apache also. however for TC's and RC's I will propose we make the change for F23 so as to not shake up things too much for F22 final. we do not want to be responsible for any slip.
There is likely other areas of mash that can do with improvement. we need to run some mashes with debugging enabled to see how exactly drpm creation goes as It seems to be a big slowdown point.
Dennis
On Tue, May 05, 2015 at 04:16:55PM -0500, Dennis Gilmore wrote:
So in debugging why mash is so slow. we had a realisation, one thing it was supposed to always be doing never worked. and that is hardlinking the packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning everything under /mnt/koji/packages and the compose processes all running as masher. the hardlink failed due to permission errors
[…]
the rawhide mash today took ~3 hours to put all the files into place in
[…]
when run as apache in a test putting all the files in place in /mnt/koji/mash took ~4 minutes
Nice find, this is huge.
we need to have bodhi run mash as apache
Untested patch attached. It should be simple enough to test in staging to see if it causes any SELinux explosions.
luke
On Tue, May 05, 2015 at 04:28:27PM -0600, Luke Macken wrote:
On Tue, May 05, 2015 at 04:16:55PM -0500, Dennis Gilmore wrote:
So in debugging why mash is so slow. we had a realisation, one thing it was supposed to always be doing never worked. and that is hardlinking the packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning everything under /mnt/koji/packages and the compose processes all running as masher. the hardlink failed due to permission errors
[…]
the rawhide mash today took ~3 hours to put all the files into place in
[…]
when run as apache in a test putting all the files in place in /mnt/koji/mash took ~4 minutes
Nice find, this is huge.
+1, every bit counts and this is a BIG bit.
On Tue, May 05, 2015 at 04:16:55PM -0500, Dennis Gilmore wrote:
the rawhide mash today took ~3 hours to put all the files into place in
[...]
when run as apache in a test putting all the files in place in /mnt/koji/mash took ~4 minutes
Wow! A few more finds like that, and it'll finish before it starts!
On Tue, May 05, 2015 at 04:16:55PM -0500, Dennis Gilmore wrote:
Hi all,
So in debugging why mash is so slow. we had a realisation, one thing it was supposed to always be doing never worked. and that is hardlinking the packages in /mnt/koji/mash to /mnt/koji/packages/ this is due to apache owning everything under /mnt/koji/packages and the compose processes all running as masher. the hardlink failed due to permission errors
the rawhide mash today took ~3 hours to put all the files into place in /mnt/koji/mash as shown by some snippets from the mash.log 2015-05-05 05:25:51 mash: Writing out files for /mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages... 2015-05-05 08:11:43 mash: createrepo: starting /mnt/koji/mash/rawhide-20150505/rawhide/x86_64/os/Packages...
when run as apache in a test putting all the files in place in /mnt/koji/mash took ~4 minutes 2015-05-05 19:27:07 mash: Writing out files for /mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages... 2015-05-05 19:30:54 mash: createrepo: starting /mnt/koji/mash/rawhide-20150505.01/rawhide/x86_64/os/Packages...
So that is a massive massive savings, both in the time it will take to mash the tree, but also the disk we use on /mnt/koji right now we are using 27T of disk, there will be large chunks of duplicated data. so now we need to adjust a bunch of processes. we need to have bodhi run mash as apache, we need to do the nightly branched and rawhide as apache, and we need to change how we do TC's and RC's so that they run as apache also. however for TC's and RC's I will propose we make the change for F23 so as to not shake up things too much for F22 final. we do not want to be responsible for any slip.
There is likely other areas of mash that can do with improvement. we need to run some mashes with debugging enabled to see how exactly drpm creation goes as It seems to be a big slowdown point.
Dennis
Excellent find!
Changing everything to run as apache raises a flag for me, though. If we just add the masher user to the apache group, would that give us the same benefit without all the re-organization work? Then we could:
1) continue to isolate things from the apache user that only the masher user should access 2) while the masher user could still (potentially) carry out that hard linking in an apache owned directory
On Wed, 2015-05-06 at 09:52 -0400, Ralph Bean wrote:
Changing everything to run as apache raises a flag for me, though. If we just add the masher user to the apache group, would that give us the same benefit without all the re-organization work?
Not really.
Making a hardlink requires having write permissions on the original source file.
We talked about that yesterday, and Dennis found that the file have the following permissions:
-rw-r--r--. 1 apache apache 42K Apr 30 19:24 /mnt/koji/packages/.../mutter-tests-3.17.1-1.fc23.i686.rpm
So adding the masher user to the apache group wouldn't give it the write permission.
Unless we change Koji to make sure it writes those packages with the write permission granted to the group?
On Wed, May 6, 2015 at 6:52 AM, Ralph Bean rbean@redhat.com wrote:
Excellent find!
Changing everything to run as apache raises a flag for me, though. If we just add the masher user to the apache group, would that give us the same benefit without all the re-organization work? Then we could:
- continue to isolate things from the apache user that only the masher user should access
For the isolation portion, it might be that koji should be run as a different user so that it writes things out as a different user than apache. We do that for most of our other web apps.
-Toshio
rel-eng@lists.fedoraproject.org
-
Dennis Gilmore -
Luke Macken -
Mathieu Bridon -
Matthew Miller -
Paul W. Frields -
Ralph Bean -
Toshio Kuratomi