GitLab AMA Topic: Message Bus

[Aoife Moloney]

Hi everyone,

I hope you enjoyed the F33 release party this weekend! Getting back to
the GitLab topic mail threads, this weeks topic from the GitLab AMA
session on September 10th is on Message Bus. As always, here are some
links to the resources I have been pulling content from as well:
* Questions and Answers hackmd link https://hackmd.io/RW8HahOeR7OJPON1dwuo3w
* Chat log from session
https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_wi...
* AMA Blog post
https://communityblog.fedoraproject.org/gitlab-ama-follow-up/#more-9346
* Here is this email in hackmd if you wish to view it there:
https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view

## Topic: Message Bus
- Question: Fedora uses a message bus to integrate different parts of
its infrastructure. How should we onboard GitLab into this message
bus?
    - Answer: Currently we would need to have a service that proxies
GitLab’s events to fedora-messaging something similar to
github2fedmsg.
There were some concerns raised about the order of events sent by
GitLab’s webhooks, these will need to be looked after during a Proof
of Concept stage.
- Question: How would git push over http work with GitLab? (assuming
gitlab does not have Fedora's password since they are stored in FAS)
    - Answer: GitLab has a good number of authentication options and
integrations where the "best" solution usually depends on a team's
specific needs and use case. As such,  the best way to know and meet
Fedora's needs and use cases is to have a conversation and discuss the
options with GitLab. How does git push over HTTP work with FAS right
now, and what git push (over HTTP) auth requirements/flow would you
like to have for your projects in GitLab?

These are the only two questions and answers I could gather relating
to message bus from the AMA question sheet, however I know there was a
lot of discussion regarding this topic during the AMA session itself,
so if you would like to resume/kick off  that conversation again,
please feel free to use this email to discuss.

A personal note and for full transparency: the weeks seem to be
passing in the blink of an eye lately, I assume it's because I'm
busy(?) but it might be just the weird 2020 vibe the world is on
nowadays. I really don't know. Whatever the reason, there has been no
further discussion with GitLab since early October-ish, but we will be
returning to the conversation of how this migration could be
technically possible soon, so sincerely thank you all again for
engaging with us/me here as I found reading the discussion on
permission and access much easier to follow and have been taking notes
on your expectations to use that feedback in conversations with GitLab
when we pick the discussion back up.

I hope you all had a good weekend and will talk to you all next week
when the topic of Namespace & Issue Tracking is sent!

Kindest regards,
Aoife

[Fabio Valentini]

On Mon, Nov 9, 2020 at 3:05 PM Aoife Moloney  wrote:
> Hi everyone,
>
> I hope you enjoyed the F33 release party this weekend! Getting back to
> the GitLab topic mail threads, this weeks topic from the GitLab AMA
> session on September 10th is on Message Bus. As always, here are some
> links to the resources I have been pulling content from as well:
> Questions and Answers hackmd link https://hackmd.io/RW8HahOeR7OJPON1dwuo3w
> Chat log from session
>
> https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_wi...
> AMA Blog post
>
> https://communityblog.fedoraproject.org/gitlab-ama-follow-up/#more-9346
> Here is this email in hackmd if you wish to view it there:
>
> https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view

Sorry for taking so long to respond, the past weeks have been quite busy.

(snip)
> ## Topic: Message Bus
> Question: Fedora uses a message bus to integrate different parts of
>
> its infrastructure. How should we onboard GitLab into this message
> bus?
>     - Answer: Currently we would need to have a service that proxies
> GitLab’s events to fedora-messaging something similar to
> github2fedmsg.
> There were some concerns raised about the order of events sent by
> GitLab’s webhooks, these will need to be looked after during a Proof
> of Concept stage.

Do we know if such a proxy would even be theoretically possible for GitLab?
IIRC, some doubts were raised during the AMA that getting a
chronologically consistent stream of events out of GitLab would be ∈
[hard, impossible[.
What would that mean for fedora? Do services relying on
fedora-messaging events related to dist-git need them to be consistent
/ chronological?
What would be the effect of those services not having a reliable
stream of events from dist-git?

Fabio
> Question: How would git push over http work with GitLab? (assuming
>
> gitlab does not have Fedora's password since they are stored in FAS)
>     - Answer: GitLab has a good number of authentication options and
> integrations where the "best" solution usually depends on a team's
> specific needs and use case. As such,  the best way to know and meet
> Fedora's needs and use cases is to have a conversation and discuss the
> options with GitLab. How does git push over HTTP work with FAS right
> now, and what git push (over HTTP) auth requirements/flow would you
> like to have for your projects in GitLab?
>
> These are the only two questions and answers I could gather relating
> to message bus from the AMA question sheet, however I know there was a
> lot of discussion regarding this topic during the AMA session itself,
> so if you would like to resume/kick off  that conversation again,
> please feel free to use this email to discuss.
>
> A personal note and for full transparency: the weeks seem to be
> passing in the blink of an eye lately, I assume it's because I'm
> busy(?) but it might be just the weird 2020 vibe the world is on
> nowadays. I really don't know. Whatever the reason, there has been no
> further discussion with GitLab since early October-ish, but we will be
> returning to the conversation of how this migration could be
> technically possible soon, so sincerely thank you all again for
> engaging with us/me here as I found reading the discussion on
> permission and access much easier to follow and have been taking notes
> on your expectations to use that feedback in conversations with GitLab
> when we pick the discussion back up.
>
> I hope you all had a good weekend and will talk to you all next week
> when the topic of Namespace & Issue Tracking is sent!
>
> Kindest regards,
> Aoife
> --
> Aoife Moloney
> Product Owner
> Community Platform Engineering Team
> Red Hat EMEA
> Communications House
> Cork Road
> Waterford
> _______________________________________________
> devel-announce mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Clement Verna]

On Wed, 18 Nov 2020 at 18:30, Fabio Valentini  wrote:
> On Mon, Nov 9, 2020 at 3:05 PM Aoife Moloney  wrote:
> > Hi everyone,
> >
> > I hope you enjoyed the F33 release party this weekend! Getting back to
> > the GitLab topic mail threads, this weeks topic from the GitLab AMA
> > session on September 10th is on Message Bus. As always, here are some
> > links to the resources I have been pulling content from as well:
> > Questions and Answers hackmd link
>
> https://hackmd.io/RW8HahOeR7OJPON1dwuo3w
> > Chat log from session
>
> https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_wi...
> > AMA Blog post
> >
> > https://communityblog.fedoraproject.org/gitlab-ama-follow-up/#more-9346
> > Here is this email in hackmd if you wish to view it there:
> >
> > https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view
>
> Sorry for taking so long to respond, the past weeks have been quite busy.

Sorry for taking even longer to answer, you know the feeling :P
> (snip)
> > ## Topic: Message Bus
> > Question: Fedora uses a message bus to integrate different parts of
> >
> > its infrastructure. How should we onboard GitLab into this message
> > bus?
> >     - Answer: Currently we would need to have a service that proxies
> > GitLab’s events to fedora-messaging something similar to
> > github2fedmsg.
> > There were some concerns raised about the order of events sent by
> > GitLab’s webhooks, these will need to be looked after during a Proof
> > of Concept stage.
>
> Do we know if such a proxy would even be theoretically possible for GitLab?
> IIRC, some doubts were raised during the AMA that getting a
> chronologically consistent stream of events out of GitLab would be ∈
> [hard, impossible[.
> What would that mean for fedora? Do services relying on
> fedora-messaging events related to dist-git need them to be consistent
> / chronological?
> What would be the effect of those services not having a reliable
> stream of events from dist-git?

I personally don't have a good answer to these questions, and I don't think
we will be able to
have without actually doing a Proof of Concept and see how that would work
and scale.

Regarding the order or messages, I believe that anything related to CI
testing might need to
have the chronological order of messages consistent.
I am not sure if there are any other use cases ?
> Fabio
> > Question: How would git push over http work with GitLab? (assuming
> >
> > gitlab does not have Fedora's password since they are stored in FAS)
> >     - Answer: GitLab has a good number of authentication options and
> > integrations where the "best" solution usually depends on a team's
> > specific needs and use case. As such,  the best way to know and meet
> > Fedora's needs and use cases is to have a conversation and discuss the
> > options with GitLab. How does git push over HTTP work with FAS right
> > now, and what git push (over HTTP) auth requirements/flow would you
> > like to have for your projects in GitLab?
> >
> > These are the only two questions and answers I could gather relating
> > to message bus from the AMA question sheet, however I know there was a
> > lot of discussion regarding this topic during the AMA session itself,
> > so if you would like to resume/kick off  that conversation again,
> > please feel free to use this email to discuss.
> >
> > A personal note and for full transparency: the weeks seem to be
> > passing in the blink of an eye lately, I assume it's because I'm
> > busy(?) but it might be just the weird 2020 vibe the world is on
> > nowadays. I really don't know. Whatever the reason, there has been no
> > further discussion with GitLab since early October-ish, but we will be
> > returning to the conversation of how this migration could be
> > technically possible soon, so sincerely thank you all again for
> > engaging with us/me here as I found reading the discussion on
> > permission and access much easier to follow and have been taking notes
> > on your expectations to use that feedback in conversations with GitLab
> > when we pick the discussion back up.
> >
> > I hope you all had a good weekend and will talk to you all next week
> > when the topic of Namespace & Issue Tracking is sent!
> >
> > Kindest regards,
> > Aoife
> > --
> > Aoife Moloney
> > Product Owner
> > Community Platform Engineering Team
> > Red Hat EMEA
> > Communications House
> > Cork Road
> > Waterford
> > _______________________________________________
> > devel-announce mailing list -- [e-mail]
> > To unsubscribe send an email to
>
> [e-mail]
> > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Jeremy Cline]

On Mon, 2020-11-23 at 19:29 +0100, Clement Verna wrote:
> On Wed, 18 Nov 2020 at 18:30, Fabio Valentini 
> wrote:
> > On Mon, Nov 9, 2020 at 3:05 PM Aoife Moloney 
> > wrote:
> > > Hi everyone,
> > >
> > > I hope you enjoyed the F33 release party this weekend! Getting
> >
> > back to
> > > the GitLab topic mail threads, this weeks topic from the GitLab
> >
> > AMA
> > > session on September 10th is on Message Bus. As always, here are
> >
> > some
> > > links to the resources I have been pulling content from as well:
> > > Questions and Answers hackmd link
> >
> > https://hackmd.io/RW8HahOeR7OJPON1dwuo3w
> > > Chat log from session
> >
> > https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_wi...
> > > AMA Blog post
> >
> > https://communityblog.fedoraproject.org/gitlab-ama-follow-up/#more-9346
> > > Here is this email in hackmd if you wish to view it there:
> > >
> > > https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view
> >
> > Sorry for taking so long to respond, the past weeks have been quite
> > busy.
>
> Sorry for taking even longer to answer, you know the feeling :P
> > (snip)
> > > ## Topic: Message Bus
> > > Question: Fedora uses a message bus to integrate different
> >
> > parts of
> > > its infrastructure. How should we onboard GitLab into this
> >
> > message
> > > bus?
> > >      - Answer: Currently we would need to have a service that
> >
> > proxies
> > > GitLab’s events to fedora-messaging something similar to
> > > github2fedmsg.
> > > There were some concerns raised about the order of events sent by
> > > GitLab’s webhooks, these will need to be looked after during a
> >
> > Proof
> > > of Concept stage.
> >
> > Do we know if such a proxy would even be theoretically possible for
> > GitLab?
> > IIRC, some doubts were raised during the AMA that getting a
> > chronologically consistent stream of events out of GitLab would be
> > ∈
> > [hard, impossible[.
> > What would that mean for fedora? Do services relying on
> > fedora-messaging events related to dist-git need them to be
> > consistent
> > / chronological?
> > What would be the effect of those services not having a reliable
> > stream of events from dist-git?
>
> I personally don't have a good answer to these questions, and I don't
> think we will be able to 
> have without actually doing a Proof of Concept and see how that would
> work and scale.
>
> Regarding the order or messages, I believe that anything related to
> CI testing might need to 
> have the chronological order of messages consistent. 
> I am not sure if there are any other use cases ?

If any service relies on chronological ordering for messaging, I've got
bad news on that front. There's plenty of scenarios where messages can
arrive out-of-order now, regardless of whether they were queued up
sequentially (which, in a distributed system, with multiple nodes
accepting messages for publication...). There's also no guarantees
about messages only being delivered once. Never mind that GitLab's
webhooks may well fire multiple times if a response is not received in
a timely manner.

I would highly recommend not creating message consumers that rely on
any particular message ordering because they're not going to work
properly, GitLab or not.

- Jeremy

[Neal Gompa]

On Mon, Nov 23, 2020 at 3:00 PM Jeremy Cline  wrote:
> On Mon, 2020-11-23 at 19:29 +0100, Clement Verna wrote:
> > On Wed, 18 Nov 2020 at 18:30, Fabio Valentini 
> > wrote:
> > > On Mon, Nov 9, 2020 at 3:05 PM Aoife Moloney 
> > > wrote:
> > > > Hi everyone,
> > > >
> > > > I hope you enjoyed the F33 release party this weekend! Getting
> > >
> > > back to
> > > > the GitLab topic mail threads, this weeks topic from the GitLab
> > >
> > > AMA
> > > > session on September 10th is on Message Bus. As always, here are
> > >
> > > some
> > > > links to the resources I have been pulling content from as well:
> > > > Questions and Answers hackmd link
> > >
> > > https://hackmd.io/RW8HahOeR7OJPON1dwuo3w
> > > > Chat log from session
> > >
> > > https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_wi...
> > > > AMA Blog post
> > >
> > > https://communityblog.fedoraproject.org/gitlab-ama-follow-up/#more-9346
> > > > Here is this email in hackmd if you wish to view it there:
> > > >
> > > > https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view
> > >
> > > Sorry for taking so long to respond, the past weeks have been quite
> > > busy.
> >
> > Sorry for taking even longer to answer, you know the feeling :P
> > > (snip)
> > > > ## Topic: Message Bus
> > > > Question: Fedora uses a message bus to integrate different
> > >
> > > parts of
> > > > its infrastructure. How should we onboard GitLab into this
> > >
> > > message
> > > > bus?
> > > >      - Answer: Currently we would need to have a service that
> > >
> > > proxies
> > > > GitLab’s events to fedora-messaging something similar to
> > > > github2fedmsg.
> > > > There were some concerns raised about the order of events sent by
> > > > GitLab’s webhooks, these will need to be looked after during a
> > >
> > > Proof
> > > > of Concept stage.
> > >
> > > Do we know if such a proxy would even be theoretically possible for
> > > GitLab?
> > > IIRC, some doubts were raised during the AMA that getting a
> > > chronologically consistent stream of events out of GitLab would be
> > > ∈
> > > [hard, impossible[.
> > > What would that mean for fedora? Do services relying on
> > > fedora-messaging events related to dist-git need them to be
> > > consistent
> > > / chronological?
> > > What would be the effect of those services not having a reliable
> > > stream of events from dist-git?
> >
> > I personally don't have a good answer to these questions, and I don't
> > think we will be able to
> > have without actually doing a Proof of Concept and see how that would
> > work and scale.
> >
> > Regarding the order or messages, I believe that anything related to
> > CI testing might need to
> > have the chronological order of messages consistent.
> > I am not sure if there are any other use cases ?
>
> If any service relies on chronological ordering for messaging, I've got
> bad news on that front. There's plenty of scenarios where messages can
> arrive out-of-order now, regardless of whether they were queued up
> sequentially (which, in a distributed system, with multiple nodes
> accepting messages for publication...). There's also no guarantees
> about messages only being delivered once. Never mind that GitLab's
> webhooks may well fire multiple times if a response is not received in
> a timely manner.
>
> I would highly recommend not creating message consumers that rely on
> any particular message ordering because they're not going to work
> properly, GitLab or not.

Too late, pretty much every consumer I'm aware of relies on having
chronological order or at least some way to sort them chronologically
for processing for messages.

[Adam Williamson]

On Mon, 2020-11-23 at 15:11 -0500, Neal Gompa wrote:
> > I would highly recommend not creating message consumers that rely on
> > any particular message ordering because they're not going to work
> > properly, GitLab or not.
>
> Too late, pretty much every consumer I'm aware of relies on having
> chronological order or at least some way to sort them chronologically
> for processing for messages.

I don't think any consumer I've written does. They all just work on the
basis "a thing happened; do some things relevant to the thing that
happened".

[Clement Verna]

On Mon, 23 Nov 2020 at 21:31, Adam Williamson 
wrote:
> On Mon, 2020-11-23 at 15:11 -0500, Neal Gompa wrote:
> > > I would highly recommend not creating message consumers that rely on
> > > any particular message ordering because they're not going to work
> > > properly, GitLab or not.
> >
> > Too late, pretty much every consumer I'm aware of relies on having
> > chronological order or at least some way to sort them chronologically
> > for processing for messages.
>
> I don't think any consumer I've written does. They all just work on the
> basis "a thing happened; do some things relevant to the thing that
> happened".

Honestly my feeling is that most of our consumers fall into that category ,
I can't think of a use case where the chronological order matters (I have
been told CI might, but I don't know the exact use case).
If anyone has a concrete example, I would welcome it very much so that we
can use it as a test case with GitLab.
> --
> Adam Williamson
> Fedora QA
> IRC: adamw | Twitter: adamw_ha
> https://www.happyassassin.net
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Jeremy Cline]

On Tue, 2020-11-24 at 14:57 +0100, Clement Verna wrote:
> On Mon, 23 Nov 2020 at 21:31, Adam Williamson <
> [e-mail]> wrote:
> > On Mon, 2020-11-23 at 15:11 -0500, Neal Gompa wrote:
> > > > I would highly recommend not creating message consumers that
> >
> > rely on
> > > > any particular message ordering because they're not going to
> >
> > work
> > > > properly, GitLab or not.
> > >
> > > Too late, pretty much every consumer I'm aware of relies on
> >
> > having
> > > chronological order or at least some way to sort them
> >
> > chronologically
> > > for processing for messages.
> >
> > I don't think any consumer I've written does. They all just work on
> > the
> > basis "a thing happened; do some things relevant to the thing that
> > happened".
>
> Honestly my feeling is that most of our consumers fall into that
> category , I can't think of a use case where the chronological order
> matters (I have been told CI might, but I don't know the exact use
> case). 
> If anyone has a concrete example, I would welcome it very much so
> that we can use it as a test case with GitLab.

Well CI needs to be told it can't whatever the use case because
chronological ordering of messages isn't a guarantee and it'll break if
that's assumed. That's true of any consumer now, so if Neal is aware of
any that fall into that category (I'm not) they should be fixed. It's
not a safe assumption with AMQP and it wasn't a safe assumption with
ZeroMQ either.