[RFR #4562] Koschei - continuous integration in Koji

Thu Oct 16 17:09:24 UTC 2014

----- Original Message -----
> From: "Kevin Fenzi" <kevin at scrye.com>
> To: infrastructure at lists.fedoraproject.org
> Sent: Thursday, 16 October, 2014 5:51:19 PM
> Subject: Re: [RFR #4562] Koschei - continuous integration in Koji
> 
> On Thu, 16 Oct 2014 11:14:01 +0200
> Mikolaj Izdebski <mizdebsk at redhat.com> wrote:
> 
> > On 10/15/2014 09:31 PM, Kevin Fenzi wrote:
> > > ok, some general questions, please excuse me if they are dumb. ;)
> > > 
> > > high level:
> > > 
> > > * How well does it keep up currently? I know you are careful not to
> > >   overload koji, but I wonder if that means things like perl builds
> > > are often behind because there are so many of them?
> > 
> > Koji has more than enough resources to sustain current Koschei load
> > (~3000 packages).  Storage might become problematic if more packages
> > are added (scratch build results are kept for some period of time),
> > but we have a solution for that (see [2] or [3]).  If more Koji
> > builders are ever need then I think it sould be quite easy to add
> > them, as long as there is budget for that.
> 
> Great. I wasn't sure if it was keeping up or not. ;)
> Interesting idea on the immediate purge.
> 
> > > * right now the service is opt-in right? Someone adds a group and
> > >   packages in that group and then when one of them changes it
> > > scratch rebuilds the rest. Do you see a time/case when we could
> > > just make it operate on all builds? that is, build foo is made, and
> > > it just does all the things that buildrequire foo?
> > 
> > For now only some subset of all packages is tracked by Koschei, but
> > the ultimate goal is to track all packages - they would be added
> > automatically after first build appears on Koji and removed when they
> > are blocked. What would be up to individuals is maintainig package
> > groups.  (One package can be in any number of groups.)
> 
> ok. Groups are only managed manually by maintainers?

Currently yes. There is a web interface for modifying them by users, but it's
not enabled in production, because we haven't decided who should have the
permission. If we allow everyone to manage groups, it might become quite
chaotic.  My idea was to have global groups representing language stacks and
other types of "natural" grouping which will be managed by corresponding SIGs
and then have groups created by regular users which will be namespaced with
their account name and displayed separately in the UI. What do you think about
that?

> 
> And deps are then checked when something updates in a group and the
> rest of the group rebuilt? or can you explain when a scratch build is
> fired?

No, groups are just convenience for displaying packages and filtering messages,
but they don't affect scheduling. Packages are rebuilt when their build
dependencies (also transitive) change, regardless of their groups. So when there
is a package foo which depends on bar, when bar is updated, we detect it after
Koji repo is generated and raise foo's priority. Then we submit scratch-builds
for packages with priority higher than X, ordered by the priority, so packages
with most updates will be scheduled first. X is configurable and currently set
that one direct dependency update is enough to trigger a scratch-build. Priority
is also slowly raised over time.

> 
> > > * The notifications of failed builds currently are via fedmsg? We
> > >   should investigate adding this to FMN if it's not already there,
> > > so anyone interested could be notified via that.
> > 
> > fedmsg publishing is already operational as can be seen on [1]. FMN
> > rule has been recently added. The new FMN is not yet in production,
> > but in (hopefully near) future users will be able to enable email or
> > IRC notifications for buildability status of packages they are
> > interested in.
> 
> Great!
> 
> > > todo's/ideas:
> > > 
> > > * Could this ever be a koji plugin? Or does it do too much on top of
> > >   that to ever be a plugin?
> > 
> > Koschei has its own architecture and converting it to Koji plugin
> > would require substantial amount of work.  In other words, it should
> > be possible, but I don't see any good reason to do so.
> 
> Fair enough.
> 
> > > * Might it be possible to run on all the broken deps packages in
> > >   rawhide/branched? This would depend I guess on the compose process
> > >   generating fedmsgs with those package names, but if so it could
> > > tell maintainers "hey, your package is broken in rawhide, but a
> > > simple rebuild will fix it" (or any other group that just wants to
> > > go fix them).
> > 
> > This is an interesting idea.
> > 
> > A simillar feature was planned for future. The idea was that Koschei
> > could be resolving runtime dependencies of all packages besides just
> > build dependencies. Users would be able to see whether package is
> > installable and if yes, see its installation size with dependencies
> > (in terms of MB to download, MB installed size and package count).
> > There would be graphs showing how this changes in time. (We had a
> > simillar POC service runnig for a few months, see [4].)
> > 
> > We could extend this and make Koschei resolve runtime dependencies of
> > successful scratch builds it runs.  In case scratch build would fix
> > broken package in offcial repo, a fedmsg would be emited.
> 
> Yeah, something to consider...
> 
> > > * boost is another group of packages I could see this being useful
> > > for. Perhaps it woul<d be worth reaching out to the boost
> > > maintainers?
> > 
> > I don't know specifics of boost packages, but we'll cosider any
> > feature request.
> 
> ok. Boost often updates once a cycle or so, and lots of dependent
> packages need rebuilding. If we could see which of those fail it could
> be helpfull. But this is up to boost maintainers I suppose.
>  
> > > * Could this be used to scratch build packages that are
> > >   ExcludeArch/ExclusiveArch with that removed? ie, to tell
> > > maintainers, "hey, you exclude arm, but it builds ok, are you sure
> > > thats fine?"
> > 
> > This would require generating a new SRPM with
> > ExcludeArch/ExclusiveArch removed, which requires installing all
> > build dependencies, so it should be done by Koji as buildSRPMfromSCM
> > task. This in turn requires Koschei having ability to push to some
> > branch in SCM or maintaining separate git repo and changing Koji
> > policy to allow scratch builds from it. And of course this would have
> > to be implemented in Koschei. Not impossible, but looks like a lot of
> > work for something that could be done manually by running some script
> > from time to time.
> 
> Yeah, agreed.
> 
> > > technical:
> > > 
> > > * Can this application be load balanced any? Ie, if we have two of
> > > them could they operate against the same db at the same time?
> > 
> > To answer this question I need to elaborate more about Koschei
> > architecture. tl;dr yes, it can be load balanced well.
>  ...snip...
> 
> > To sum up, all components either don't need load balancing or can be
> > load-balanced.
> 
> ok. And I suspect this service would be ok just being one instance as
> well, since it's not critical. ie, if we had to update/reboot it we
> could just do so and it would go on, no need to keep everything up 100%?

Yes, stopping the resolver will just delay builds a bit. So even if we stop
it for few hours, nothing bad will happen.

> 
> > 
> > > * Are there any common sysadmin tasks we need to know about with the
> > >   instance? Is there any special process to start/stop/reinstall
> > > it?
> > 
> > Installing Koschei is done by installing RPM package from Fedora or
> > EPEL repositories and coping a single config file.
> > Managing all services (starting, stopping, viewing logs etc.) is done
> > using standard system tools (systemctl, journalctl and so on).
> > There is nothing special to be done besides standard sysadmin stuff
> > (updating packages, viewing logs, backing up database and so on).
> 
> Excellent.
> 
> > > * When there is koji maint, should we stop this service? How do we
> > >   gracefully do that and start it again?
> > 
> > In this case you can stop Koschei services that communicate with Koji
> > (using systemctl stop koschei-<servicename>) and start them when
> > maintenance is over. Web UI will remain functional during that time,
> > but there will be no new build scheduled.
> > 
> > > Thats all I can think of right now. :)
> > 
> > I hope this pretty long email answers your questions.
> 
> It does indeed. Thanks for taking the time to do so. :)
> 
> kevin
> 
> _______________________________________________
> infrastructure mailing list
> infrastructure at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Michael Simacek