as a Copr contributor, I am missing a _standard_ design for integrating
our cool infrastructure into the _upstream_ work-flows. We have a lot of
teams trying to implement the same thing:
1. Catch some upstream event (like push, pull-request, release, tag, ...)
2. React to the event. Do something with this event, (e.g. do CI build,
Copr build, do some scratch build using Jenkins, etc.)
3. Ideally report back upstream, somehow, the results.
By _standard_ I mean that (a) it should be absolutely trivial to start the
Fedora Infra <-> Upstream interaction (to integrate) from the package
contributor POV, and (b) that it should be _fairly simple_ to implement
the reaction (point 2 and 3) inside our infra.
Namely, consider that we in Copr want to trigger builds by the events from
GitHub (PRs, tags, pushes, ...), GitLab (MRs, ...), Pagure, BitBucket,
etc. We sort of are able to do so (it is far from complete support!), but
for this to happen -- we had to implement pretty complicated webhook
interface on Copr side for all the services, and for Pagure we have to
listen on fedora-messaging bus.
The real challenge though is _how to notify_ upstream projects that
something inside our infra (eg Copr) happened.
Nothing dramatic happened in Copr so far. We "only" have support for
notifying Pagure ATM (we are able to set "commit status" for PRs and
pushes). To be able to communicate with other forges, it would be an
equivalent amount of repetitive work.. (we'd have to store somewhere the
access "tokens", know how to/where contact the upstream, etc.). Quite
some work, ... worth doing only once in whole Fedora?
What I think we could have is something like a central gateway between
Fedora and Upstream, let's say "Fedora Integrator"? That thing should cut
out and provide the part of Copr code which does this integration steps:
- It should "listen" on several forges (webhooks, on our bus, atm.),
and translate the variously formatted event payloads into well-defined,
uniform events. Say "change", "change request", etc.
- Such an event would be dumped (back) on the fedora-messaging bus,
through well-defined (schema) message. The bus part is important because
it is (a) asynchronous and (b) even services behind corporate VPN can
react on them.
- Each such event would have its own ID, so **any** Fedora service can react
on it by posting ReactsOn=EventID (again well-defined) "reaction" on
- The Integrator service would be able to translate the "reactions"
on bus to forge responses -- e.g., when CI passes, notify
Pagure/GitHub/GitLab/... commit status, send an email to upstream, etc.
I sketched a simple component graph how that could look like . Feel
free to comment here or there... the webhook/message payload translation
mechanism should be somehow wrapped into a library, so we could still get
the unified format of events without the integrator service (iff needed).
This way we would build a single place for collaboration on such things
(parsing the webhooks, forge interaction, etc.). But we would have a
single orientation points for people who use our services for
CI/CD (single "Fedora Integrator" GitHub app, GitLab.com
"Fedora Integration", etc.).
Not only that would be super easy to find -- such forge apps are also
pretty privileged things (they can do various stuff with the
repositories), so people might not be very confident they should enable
them. If we had one blessed Fedora place to enable those things per
forge, I think it would be more "trusted".
Where to start? E.g. by using parts of github2fedmsg, copr and Packit
We need to find a way to get the events on our bus easily, so I e.g. asked
GitLab people  how to easily implement this.
Do you think such initiative makes sense?
On Thursday, June 25, 2020 9:27:24 PM CEST Pierre-Yves Chibon wrote:
Good Morning Everyone,
Just like every team we have technical debt in our work.
I would like your help to try to define what it is for us.
So far, I've come up with the following:
- python3 support/migration
- fedora-messaging schema
- OpenID Connect
What else would we want in there?
Looking forward to your thoughts,
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines