On Mon, 2020-11-23 at 19:29 +0100, Clement Verna wrote:
On Wed, 18 Nov 2020 at 18:30, Fabio Valentini decathorpe@gmail.com wrote:
On Mon, Nov 9, 2020 at 3:05 PM Aoife Moloney amoloney@redhat.com wrote:
Hi everyone,
I hope you enjoyed the F33 release party this weekend! Getting
back to
the GitLab topic mail threads, this weeks topic from the GitLab
AMA
session on September 10th is on Message Bus. As always, here are
some
links to the resources I have been pulling content from as well:
- Questions and Answers hackmd link
https://hackmd.io/RW8HahOeR7OJPON1dwuo3w
- Chat log from session
https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_wi...
- AMA Blog post
https://communityblog.fedoraproject.org/gitlab-ama-follow-up/#more-9346
- Here is this email in hackmd if you wish to view it there:
Sorry for taking so long to respond, the past weeks have been quite busy.
Sorry for taking even longer to answer, you know the feeling :P
(snip)
## Topic: Message Bus
- Question: Fedora uses a message bus to integrate different
parts of
its infrastructure. How should we onboard GitLab into this
message
bus? - Answer: Currently we would need to have a service that
proxies
GitLab’s events to fedora-messaging something similar to github2fedmsg. There were some concerns raised about the order of events sent by GitLab’s webhooks, these will need to be looked after during a
Proof
of Concept stage.
Do we know if such a proxy would even be theoretically possible for GitLab? IIRC, some doubts were raised during the AMA that getting a chronologically consistent stream of events out of GitLab would be ∈ [hard, impossible[. What would that mean for fedora? Do services relying on fedora-messaging events related to dist-git need them to be consistent / chronological? What would be the effect of those services not having a reliable stream of events from dist-git?
I personally don't have a good answer to these questions, and I don't think we will be able to have without actually doing a Proof of Concept and see how that would work and scale.
Regarding the order or messages, I believe that anything related to CI testing might need to have the chronological order of messages consistent. I am not sure if there are any other use cases ?
If any service relies on chronological ordering for messaging, I've got bad news on that front. There's plenty of scenarios where messages can arrive out-of-order now, regardless of whether they were queued up sequentially (which, in a distributed system, with multiple nodes accepting messages for publication...). There's also no guarantees about messages only being delivered once. Never mind that GitLab's webhooks may well fire multiple times if a response is not received in a timely manner.
I would highly recommend not creating message consumers that rely on any particular message ordering because they're not going to work properly, GitLab or not.
- Jeremy