[RFR #4562] Koschei - continuous integration in Koji

[Kevin Fenzi]

On Thu, 16 Oct 2014 11:14:01 +0200
Mikolaj Izdebski <mizdebsk at redhat.com> wrote:

> On 10/15/2014 09:31 PM, Kevin Fenzi wrote:
> > ok, some general questions, please excuse me if they are dumb. ;) 
> > 
> > high level: 
> > 
> > * How well does it keep up currently? I know you are careful not to
> >   overload koji, but I wonder if that means things like perl builds
> > are often behind because there are so many of them? 
> 
> Koji has more than enough resources to sustain current Koschei load
> (~3000 packages).  Storage might become problematic if more packages
> are added (scratch build results are kept for some period of time),
> but we have a solution for that (see [2] or [3]).  If more Koji
> builders are ever need then I think it sould be quite easy to add
> them, as long as there is budget for that.

Great. I wasn't sure if it was keeping up or not. ;) 
Interesting idea on the immediate purge.

> > * right now the service is opt-in right? Someone adds a group and
> >   packages in that group and then when one of them changes it
> > scratch rebuilds the rest. Do you see a time/case when we could
> > just make it operate on all builds? that is, build foo is made, and
> > it just does all the things that buildrequire foo? 
> 
> For now only some subset of all packages is tracked by Koschei, but
> the ultimate goal is to track all packages - they would be added
> automatically after first build appears on Koji and removed when they
> are blocked. What would be up to individuals is maintainig package
> groups.  (One package can be in any number of groups.)

ok. Groups are only managed manually by maintainers? 

And deps are then checked when something updates in a group and the
rest of the group rebuilt? or can you explain when a scratch build is
fired?

> > * The notifications of failed builds currently are via fedmsg? We
> >   should investigate adding this to FMN if it's not already there,
> > so anyone interested could be notified via that. 
> 
> fedmsg publishing is already operational as can be seen on [1]. FMN
> rule has been recently added. The new FMN is not yet in production,
> but in (hopefully near) future users will be able to enable email or
> IRC notifications for buildability status of packages they are
> interested in.

Great!

> > todo's/ideas: 
> > 
> > * Could this ever be a koji plugin? Or does it do too much on top of
> >   that to ever be a plugin? 
> 
> Koschei has its own architecture and converting it to Koji plugin
> would require substantial amount of work.  In other words, it should
> be possible, but I don't see any good reason to do so.

Fair enough. 

> > * Might it be possible to run on all the broken deps packages in
> >   rawhide/branched? This would depend I guess on the compose process
> >   generating fedmsgs with those package names, but if so it could
> > tell maintainers "hey, your package is broken in rawhide, but a
> > simple rebuild will fix it" (or any other group that just wants to
> > go fix them). 
> 
> This is an interesting idea.
> 
> A simillar feature was planned for future. The idea was that Koschei
> could be resolving runtime dependencies of all packages besides just
> build dependencies. Users would be able to see whether package is
> installable and if yes, see its installation size with dependencies
> (in terms of MB to download, MB installed size and package count).
> There would be graphs showing how this changes in time. (We had a
> simillar POC service runnig for a few months, see [4].)
> 
> We could extend this and make Koschei resolve runtime dependencies of
> successful scratch builds it runs.  In case scratch build would fix
> broken package in offcial repo, a fedmsg would be emited.

Yeah, something to consider... 

> > * boost is another group of packages I could see this being useful
> > for. Perhaps it woul<d be worth reaching out to the boost
> > maintainers?
> 
> I don't know specifics of boost packages, but we'll cosider any
> feature request.

ok. Boost often updates once a cycle or so, and lots of dependent
packages need rebuilding. If we could see which of those fail it could
be helpfull. But this is up to boost maintainers I suppose. 
 
> > * Could this be used to scratch build packages that are
> >   ExcludeArch/ExclusiveArch with that removed? ie, to tell
> > maintainers, "hey, you exclude arm, but it builds ok, are you sure
> > thats fine?"
> 
> This would require generating a new SRPM with
> ExcludeArch/ExclusiveArch removed, which requires installing all
> build dependencies, so it should be done by Koji as buildSRPMfromSCM
> task. This in turn requires Koschei having ability to push to some
> branch in SCM or maintaining separate git repo and changing Koji
> policy to allow scratch builds from it. And of course this would have
> to be implemented in Koschei. Not impossible, but looks like a lot of
> work for something that could be done manually by running some script
> from time to time.

Yeah, agreed. 

> > technical: 
> > 
> > * Can this application be load balanced any? Ie, if we have two of
> > them could they operate against the same db at the same time? 
> 
> To answer this question I need to elaborate more about Koschei
> architecture. tl;dr yes, it can be load balanced well.
 ...snip...

> To sum up, all components either don't need load balancing or can be
> load-balanced.

ok. And I suspect this service would be ok just being one instance as
well, since it's not critical. ie, if we had to update/reboot it we
could just do so and it would go on, no need to keep everything up 100%?

> 
> > * Are there any common sysadmin tasks we need to know about with the
> >   instance? Is there any special process to start/stop/reinstall
> > it? 
> 
> Installing Koschei is done by installing RPM package from Fedora or
> EPEL repositories and coping a single config file.
> Managing all services (starting, stopping, viewing logs etc.) is done
> using standard system tools (systemctl, journalctl and so on).
> There is nothing special to be done besides standard sysadmin stuff
> (updating packages, viewing logs, backing up database and so on).

Excellent. 

> > * When there is koji maint, should we stop this service? How do we
> >   gracefully do that and start it again? 
> 
> In this case you can stop Koschei services that communicate with Koji
> (using systemctl stop koschei-<servicename>) and start them when
> maintenance is over. Web UI will remain functional during that time,
> but there will be no new build scheduled.
> 
> > Thats all I can think of right now. :) 
> 
> I hope this pretty long email answers your questions.

It does indeed. Thanks for taking the time to do so. :) 

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20141016/15cc972f/attachment.sig>