Running mash in Koji

[Kevin Fenzi]

On Tue, 24 Sep 2013 12:38:44 +0800
Mathieu Bridon <bochecha at fedoraproject.org> wrote:

> Hi,
> 
> Dennis Gilmore has been talking for some time about how he'd like to
> run Mash in Koji tasks, instead of the set of scripts releng
> currently uses.
> 
> So I started working on this, and I got to the point where it actually
> works, but I have a few questions before I can finish it and submit
> the patches.
> 
> 1. What would be an appropriate weight for this new task?
> 
> At the moment I'm just using the default, but given that mashing can
> be quite intensive, it might be better to increase that. I have no
> idea on what would be the right number, though.

Yeah, not sure either... but I wonder if it wouldn't make sense to make
this it's own channel so it could go to specific builders. 

In the Fedora case we would want that as our mashes are very memory
intensive (mostly due to deltarpm generation), so if somehow an arm
builder got it, not sure how long it would take it to finish (if ever). 

> 2. What would be the appropriate permissions to run this task?
> 
> Right now, the task requires the user to be an admin. Should I
> introduce a new 'mash' permission? Is there an existing 'releng'
> permission I should use?

Admin would probibly be ok for a first cut. 

> 3. What to do with the output?
> 
> Mash will create the repositories for all configured architectures,
> corresponding to a certain Koji tag.
> 
> As such, its output is much bigger that, say, a build task.
> 
> The task could upload everything back to the Hub, but that could take
> a significant time and might hammer the Hub?
> 
> Dennis was saying that another possibility was to mount /mnt/koji as
> read-write on the builder, and inside the chroot, so that the task
> could write its output directly there.
> 
> What would be the preferred option?

Not sure. For some data to toss in tho, a current rawhide mash takes
up about 163GB. Branched f20 is currently around 153GB. 

So, in addition to the question of if it should try and upload to the
hub or not, it needs to make sure and have enough local working space
to hold all that (unless it's writing to a rw /mnt/koji for that). 

kevin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20130926/c9496e57/attachment.sig>