Migrating fedmsg to AMQP: a proposal
by Aurelien Bompard
Hey folks!
Jeremy and I have been working on a proposal to migrate fedmsg from our
current brokerless architecture to a broker-based architecture.
The overview and reasons for the migration are described on this page:
https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/overvie...
Head there if you want the details. The plan has the following requirements:
* No flag day.
* Don't disrupt any services or applications.
* Don't break any services outside of Fedora's infrastructure relying on
these messages.
The first step is to deploy a broker in Fedora to use. In order to avoid a
flag day, bridges from AMQP to ZeroMQ and ZeroMQ to AMQP have been
implemented and will be deployed.
In order to validate that the bridges are functioning, a small service will
be run during the transition period that connects to fedmsg and to the AMQP
queues to compare messages.
After the bridges are running, applications are free to migrate. There are
several options when migrating and each has advantages and disadvantages.
We have written a new library called fedora-messaging that has the
following features:
* A method to define message schemas and offer automatic validation of
messages using those schemas.
* Boilerplate for typical publishers and consumers.
Head over to the document for a demo!
What do you think of this proposal? Any blind spots?
Thanks!
Aurélien
5 years, 10 months
Re: infrastructure Digest, Vol 144, Issue 5
by Aditya Mukherjee
Please remove me from this list. Thanks.
On 5 Jun 2018 5:10 am, <infrastructure-request(a)lists.fedoraproject.org>
wrote:
> Send infrastructure mailing list submissions to
> infrastructure(a)lists.fedoraproject.org
>
> To subscribe or unsubscribe via email, send a message with subject or
> body 'help' to
> infrastructure-request(a)lists.fedoraproject.org
>
> You can reach the person managing the list at
> infrastructure-owner(a)lists.fedoraproject.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of infrastructure digest..."
>
> Today's Topics:
>
> 1. Re: Migrating fedmsg to AMQP: a proposal (Jeremy Cline)
> 2. Meeting Agenda Item: Introduction Innocent Asiimwe
> (Innocent Asiimwe)
> 3. Re: Migrating fedmsg to AMQP: a proposal (Ken Dreyer)
>
>
> ----------------------------------------------------------------------
>
> Date: Mon, 4 Jun 2018 14:30:59 +0000
> From: Jeremy Cline <jeremy(a)jcline.org>
> Subject: Re: Migrating fedmsg to AMQP: a proposal
> To: Fedora Infrastructure <infrastructure(a)lists.fedoraproject.org>,
> Michael Bonnet <mikeb(a)redhat.com>
> Message-ID: <01000163cb3553f7-dc122a05-66bb-45c7-8363-23ebe8590585-000
> 000(a)email.amazonses.com>
> Content-Type: multipart/signed; micalg=pgp-sha256;
> protocol="application/pgp-signature";
> boundary="jHoSkhHphgVOG8vWarTggGMP9aTb87Ggr"
>
> This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
> --jHoSkhHphgVOG8vWarTggGMP9aTb87Ggr
> Content-Type: multipart/mixed; boundary="vipAwbxR7XhIe2JI5JsHFy2yg6HjAy
> ukS";
> protected-headers="v1"
> From: Jeremy Cline <jeremy(a)jcline.org>
> To: Fedora Infrastructure <infrastructure(a)lists.fedoraproject.org>,
> Michael Bonnet <mikeb(a)redhat.com>
> Message-ID: <2757f424-a91c-b7f4-bb85-9720306d69c5(a)jcline.org>
> Subject: Re: Migrating fedmsg to AMQP: a proposal
> References: <CAHmoxtXHdgHwbGrRqFanJC5iLacDezO1SwEf_b=_0zFBt3mqOQ@mail.
> gmail.com>
> <CAD-DQxS8gm8jofrXX8oMGrqFZKQ44_KEkCwWTbh6Pwo3p0oONw(a)mail.gmail.com>
> <01000163ad746b7e-5fb1fc93-2117-4b1e-bd8b-d8f1d602b8fc-
> 000000(a)email.amazonses.com>
> <CACZP5mtW3c1Ot+2vkdPwunPs0Ve9dj2EVxpr5=c3se9KuzPEhg(a)mail.gmail.com>
> In-Reply-To: <CACZP5mtW3c1Ot+2vkdPwunPs0Ve9dj2EVxpr5=c3se9K
> uzPEhg(a)mail.gmail.com>
>
> --vipAwbxR7XhIe2JI5JsHFy2yg6HjAyukS
> Content-Type: text/plain; charset=utf-8
> Content-Language: en-US
> Content-Transfer-Encoding: quoted-printable
>
> On 06/01/2018 05:45 PM, Michael Bonnet wrote:
> > On Tue, May 29, 2018 at 12:51 PM, Jeremy Cline <jeremy(a)jcline.org> wrot=
> e:
> >=20
> >> Hi,
> >>
> >> On 05/29/2018 09:31 AM, Jeffrey Ollie wrote:
> >>> On Thu, May 24, 2018 at 11:16 AM, Aurelien Bompard <
> >>> abompard(a)fedoraproject.org> wrote:
> >>>
> >>>>
> >>>> What do you think of this proposal? Any blind spots?
> >>>>
> >>>
> >>> Not that I disagree, but please add/expand a section as to why AMQP (=
> and
> >>> RabbitMQ) was chosen over other messaging technologies.
> >>
> >> Thanks for the feedback, I've added a small section[0]. It is, perhaps=
> ,
> >> a little wishy-washy. I don't want to give the impression that we
> >> couldn't implement this with a different messaging protocol or a
> >> different broker. We definitely could. AMQP has short-comings, to be
> >> sure, but the RabbitMQ extensions (mainly pulisher acks) cover the mos=
> t
> >> important ones in my opinion.
> >>
> >> I did some research, but I'd definitely welcome feedback on protocols
> >> and brokers. I've read all or nearly all of the AMQP 0.9, ZeroMQ, and
> >> STOMP protocols, and I skimmed through the MQTT protocol, but I've not=
>
> >> looked closely at the AMQP 1.0 protocol and I'm by no means a message
> >> protocol expert.
> >>
> >=20
> > I think moving to a broker-based architecture is a great idea! Your
> > document does a great job explaining the advantages it brings, and it c=
> ould
> > help increase the adoption of event-based workflows.
> >=20
> > Regarding protocols, my preference would be for STOMP. It's has very wi=
> de
> > support, with libraries in pretty much every language, and being entire=
> ly
> > text-based makes it *much* easier to debug than other protocols. The
> > message delivery semantics are well-defined, and the protocol spec has =
> the
> > nice property of being readable in one sitting. Some brokers provide th=
> e
> > ability to translate between protocols, so it may not be difficult to
> > support more than one, but I would suggest STOMP as the reference proto=
> col.
>
> I had a hard time justifying choosing STOMP over AMQP because most
> brokers just map the other protocol they focus on onto STOMP. It's true
> the the spec is short, but it leaves a lot up to individual
> implementations as far as I can tell (like how topic matching works, for
> example).
>
> While debuggability is important, I'm not certain we'll ever need to dig
> into the wire protocol. In the unlikely event that we need to, I'd go
> about it the same way (e.g. tcpdump/wireshark) and Wireshark knows how
> to parse the AMQP protocol. Based on my super simple test (capture a
> single message being published) it seems very easy to inspect. Have you
> had a different experience here?
>
>
> --=20
> Jeremy Cline
> XMPP: jeremy(a)jcline.org
> IRC: jcline
>
>
> --vipAwbxR7XhIe2JI5JsHFy2yg6HjAyukS--
>
> --jHoSkhHphgVOG8vWarTggGMP9aTb87Ggr
> Content-Type: application/pgp-signature; name="signature.asc"
> Content-Description: OpenPGP digital signature
> Content-Disposition: attachment; filename="signature.asc"
>
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEfjR9J//2iFvH1Gi7kiMwj6myRtsFAlsVTR0ACgkQkiMwj6my
> Rtv/eQ//fU41kjzEXd7AVVO263VxBVGkk5cR85bKXk7mqF7wGevDh2URoiHfBaMd
> Qs638uc4Iskxrpzk+JsPjeH4Y1jI69zIzHTK/CnsFPBdnRuIwlG+PFB+jjxerf8a
> g0cyyKkitKbBIbKDpXvFumVQNeSFVlr4RdICyDgshXzayxifKOJN5OtoqgwiaAj8
> 2SoqnySMaMGzhji84dlUXknmEZm7Ilz9jBsJzA7uhNowQLQxov6AZCdvp4JlWV1n
> Y12NjABFsk0rrzw3A4XjGDGQqCRNUtkscXlBVd3dqU9+TccSpUU0Bxzz6Kjnd9xj
> UEsRcjwygXrDa0ZXz6XYeqlYiQi0xczC98gr1LOngB/VY8yBGhCVEhQ8dGdH+BJS
> BvlbUzDUt2MfieXtHOp8PwEPjEaI3Ornq1/FGzouh+4uBaeSZgyvFbtNAsNv//Vh
> jBQARSFICg0kLvuQUp1adXYtnAr56XXJzsmSeCJAV9I0l2yuMuXMszDzHzUHNRDU
> GKY5Tks8AxogEVJugX7SbpMgsWd58+astSF8nl5aqi49gdprAY7/ivghQ/tHS4A1
> gk7nhq8rfzhoA1uVauh2EZa+TAYN9tySyqJKWsKYD6NkMZC7AiSXgBPSR4GvPDlU
> Gmaitp1AgpoaOZf/6bMcWlEZvN5tB7aXpRnSu7YPSo7AriR6mY4=
> =Pwwm
> -----END PGP SIGNATURE-----
>
> --jHoSkhHphgVOG8vWarTggGMP9aTb87Ggr--
>
> ------------------------------
>
> Date: Mon, 4 Jun 2018 22:02:47 +0300
> From: Innocent Asiimwe <innocent.asiimwe(a)andela.com>
> Subject: Meeting Agenda Item: Introduction Innocent Asiimwe
> To: infrastructure(a)lists.fedoraproject.org
> Message-ID:
> <CAD8GFgYVnoeFFp6davtpav3L1Y-=-aCgt04D=-_6=qM+7oKMOQ@mail.
> gmail.com>
> Content-Type: multipart/alternative;
> boundary="0000000000005924f1056dd59470"
>
> --0000000000005924f1056dd59470
> Content-Type: text/plain; charset="UTF-8"
>
> Dear all,
>
> IRC handle: asiimwe
>
> skills I possess:
>
> - python programming
> - bash scripting
> - Using ansible
> - Using packer
> - Using terraform
> - Using jenkins, travis and circleci for CI/CD
> - linux and windows server system administration
>
> skills I want to learn
>
> - go programming language
> - new devops and system administration workflows
> - Building tools used in devops
> - maintaining large infrastructures
> - Effectively contributing to open source
>
> I am currently working at andela(andela.com) as a software developer and
> devops engineer. (entry level)
>
> I would like to work on maintaining the Fedora infrastructure and building
> tools used in the same.
>
> regards,
> Innocent
>
> --0000000000005924f1056dd59470
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><span style=3D"color:rgb(34,34,34);
> font-family:arial,sans-=
> serif;font-size:12.8px;font-style:normal;font-variant-
> ligatures:normal;font=
> -variant-caps:normal;font-weight:400;letter-spacing:
> normal;text-align:start=
> ;text-indent:0px;text-transform:none;white-space:
> normal;word-spacing:0px;ba=
> ckground-color:rgb(255,255,255);text-decoration-style:
> initial;text-decorati=
> on-color:initial;float:none;display:inline">Dear all,</span><div
> style=3D"c=
> olor:rgb(34,34,34);font-family:arial,sans-serif;font-
> size:12.8px;font-style=
> :normal;font-variant-ligatures:normal;font-variant-
> caps:normal;font-weight:=
> 400;letter-spacing:normal;text-align:start;text-indent:
> 0px;text-transform:n=
> one;white-space:normal;word-spacing:0px;background-color:
> rgb(255,255,255);t=
> ext-decoration-style:initial;text-decoration-color:initial"><br></div><div
> =
> style=3D"color:rgb(34,34,34);font-family:arial,sans-serif;
> font-size:12.8px;=
> font-style:normal;font-variant-ligatures:normal;font-
> variant-caps:normal;fo=
> nt-weight:400;letter-spacing:normal;text-align:start;text-
> indent:0px;text-t=
> ransform:none;white-space:normal;word-spacing:0px;
> background-color:rgb(255,=
> 255,255);text-decoration-style:initial;text-decoration-color:initial">IRC
> h=
> andle: asiimwe</div><div style=3D"color:rgb(34,34,34);
> font-family:arial,san=
> s-serif;font-size:12.8px;font-style:normal;font-variant-
> ligatures:normal;fo=
> nt-variant-caps:normal;font-weight:400;letter-spacing:
> normal;text-align:sta=
> rt;text-indent:0px;text-transform:none;white-space:
> normal;word-spacing:0px;=
> background-color:rgb(255,255,255);text-decoration-style:
> initial;text-decora=
> tion-color:initial"><br></div><div style=3D"color:rgb(34,34,34);
> font-family=
> :arial,sans-serif;font-size:12.8px;font-style:normal;font-
> variant-ligatures=
> :normal;font-variant-caps:normal;font-weight:400;letter-
> spacing:normal;text=
> -align:start;text-indent:0px;text-transform:none;white-
> space:normal;word-sp=
> acing:0px;background-color:rgb(255,255,255);text-
> decoration-style:initial;t=
> ext-decoration-color:initial">skills I possess:</div><div
> style=3D"color:rg=
> b(34,34,34);font-family:arial,sans-serif;font-size:12.8px;
> font-style:normal=
> ;font-variant-ligatures:normal;font-variant-caps:
> normal;font-weight:400;let=
> ter-spacing:normal;text-align:start;text-indent:0px;text-
> transform:none;whi=
> te-space:normal;word-spacing:0px;background-color:rgb(255,
> 255,255);text-dec=
> oration-style:initial;text-decoration-color:initial"><ul><li
> style=3D"margi=
> n-left:15px">python programming</li><li style=3D"margin-left:15px">bash
> scr=
> ipting</li><li style=3D"margin-left:15px">Using ansible</li><li
> style=3D"ma=
> rgin-left:15px">Using packer</li><li style=3D"margin-left:15px">Using
> terra=
> form</li><li style=3D"margin-left:15px">Using jenkins, travis and
> circleci =
> for CI/CD</li><li style=3D"margin-left:15px">linux and windows server
> syste=
> m administration</li></ul><div>skills I want to learn</div></div><div
> style=
> =3D"color:rgb(34,34,34);font-family:arial,sans-serif;font-
> size:12.8px;font-=
> style:normal;font-variant-ligatures:normal;font-variant-
> caps:normal;font-we=
> ight:400;letter-spacing:normal;text-align:start;text-
> indent:0px;text-transf=
> orm:none;white-space:normal;word-spacing:0px;background-
> color:rgb(255,255,2=
> 55);text-decoration-style:initial;text-decoration-color:initial"><ul><li
> st=
> yle=3D"margin-left:15px">go programming language</li><li
> style=3D"margin-le=
> ft:15px">new devops=C2=A0and system administration workflows</li><li style=
> =3D"margin-left:15px">Building tools used in devops</li><li
> style=3D"margin=
> -left:15px">maintaining large infrastructures=C2=A0</li><li
> style=3D"margin=
> -left:15px">Effectively contributing to open source</li></ul><div>I am
> curr=
> ently working at andela(<a href=3D"http://andela.com/" target=3D"_blank"
> st=
> yle=3D"color:rgb(17,85,204)">andela.com</a>) as a software developer and
> de=
> vops=C2=A0engineer. (entry level)</div></div><div
> style=3D"color:rgb(34,34,=
> 34);font-family:arial,sans-serif;font-size:12.8px;font-
> style:normal;font-va=
> riant-ligatures:normal;font-variant-caps:normal;font-
> weight:400;letter-spac=
> ing:normal;text-align:start;text-indent:0px;text-
> transform:none;white-space=
> :normal;word-spacing:0px;background-color:rgb(255,255,
> 255);text-decoration-=
> style:initial;text-decoration-color:initial"><br></div><div
> style=3D"color:=
> rgb(34,34,34);font-family:arial,sans-serif;font-size:12.
> 8px;font-style:norm=
> al;font-variant-ligatures:normal;font-variant-caps:
> normal;font-weight:400;l=
> etter-spacing:normal;text-align:start;text-indent:0px;
> text-transform:none;w=
> hite-space:normal;word-spacing:0px;background-color:
> rgb(255,255,255);text-d=
> ecoration-style:initial;text-decoration-color:initial">I would like to
> work=
> on maintaining the Fedora infrastructure and building tools used in the
> sa=
> me.</div><div style=3D"color:rgb(34,34,34);font-family:arial,sans-serif;
> fon=
> t-size:12.8px;font-style:normal;font-variant-ligatures:
> normal;font-variant-=
> caps:normal;font-weight:400;letter-spacing:normal;text-
> align:start;text-ind=
> ent:0px;text-transform:none;white-space:normal;word-
> spacing:0px;background-=
> color:rgb(255,255,255);text-decoration-style:initial;text-
> decoration-color:=
> initial"><br></div><div style=3D"color:rgb(34,34,34);
> font-family:arial,sans=
> -serif;font-size:12.8px;font-style:normal;font-variant-
> ligatures:normal;fon=
> t-variant-caps:normal;font-weight:400;letter-spacing:
> normal;text-align:star=
> t;text-indent:0px;text-transform:none;white-space:
> normal;word-spacing:0px;b=
> ackground-color:rgb(255,255,255);text-decoration-style:
> initial;text-decorat=
> ion-color:initial">regards,</div><div style=3D"color:rgb(34,34,34);
> font-fam=
> ily:arial,sans-serif;font-size:12.8px;font-style:normal;
> font-variant-ligatu=
> res:normal;font-variant-caps:normal;font-weight:400;letter-
> spacing:normal;t=
> ext-align:start;text-indent:0px;text-transform:none;white-
> space:normal;word=
> -spacing:0px;background-color:rgb(255,255,255);text-
> decoration-style:initia=
> l;text-decoration-color:initial">Innocent</div><br></div>
>
> --0000000000005924f1056dd59470--
>
> ------------------------------
>
> Date: Mon, 4 Jun 2018 16:59:25 -0600
> From: Ken Dreyer <ktdreyer(a)ktdreyer.com>
> Subject: Re: Migrating fedmsg to AMQP: a proposal
> To: Fedora Infrastructure <infrastructure(a)lists.fedoraproject.org>
> Cc: Michael Bonnet <mikeb(a)redhat.com>
> Message-ID:
> <CAD3FbMX=A+FO_Cc6wqXqjWr9DhjNdu_7mcUvPai=tUK
> tVmxBTQ(a)mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
>
> On Mon, Jun 4, 2018 at 8:30 AM, Jeremy Cline <jeremy(a)jcline.org> wrote:
> > I had a hard time justifying choosing STOMP over AMQP because most
> > brokers just map the other protocol they focus on onto STOMP. It's true
> > the the spec is short, but it leaves a lot up to individual
> > implementations as far as I can tell (like how topic matching works, for
> > example).
>
> It's nice to give the flexibility to clients by exposing both. I
> haven't seen a problem with topic matching in my experience so far.
>
> One thing I found with AMQP vs STOMP is that it's possible for AMQP
> clients to (accidentally) emit "binary" message bodies, and then
> ActiveMQ does not translate or expose these as plaintext JSON for
> STOMP clients. It just looks like an empty message body to STOMP
> clients, or possibly garbage. The solution was for clients to
> translate the messages to text/json prior to sending. (Of course if
> you never enable STOMP on your broker at all, maybe this won't be a
> problem :)
>
> - Ken
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-leave@lists.
> fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: ${hyperkitty_url}
>
>
> ------------------------------
>
> End of infrastructure Digest, Vol 144, Issue 5
> **********************************************
>
5 years, 10 months
Meeting Agenda Item: Introduction Innocent Asiimwe
by Innocent Asiimwe
Dear all,
IRC handle: asiimwe
skills I possess:
- python programming
- bash scripting
- Using ansible
- Using packer
- Using terraform
- Using jenkins, travis and circleci for CI/CD
- linux and windows server system administration
skills I want to learn
- go programming language
- new devops and system administration workflows
- Building tools used in devops
- maintaining large infrastructures
- Effectively contributing to open source
I am currently working at andela(andela.com) as a software developer and
devops engineer. (entry level)
I would like to work on maintaining the Fedora infrastructure and building
tools used in the same.
regards,
Innocent
5 years, 10 months
Updates for AtomicHost artifacts in bodhi pungi runs
by Dusty Mabe
[PATCH 1/5] bodhi-pungi: f29 will be new branched target
[PATCH 2/5] bodhi-pungi: convert image-build to use non branched
[PATCH 3/5] bodhi-pungi: only provide one repo to lorax
[PATCH 4/5] bodhi-pungi: s/Atomic/AtomicHost/
[PATCH 5/5] bodhi-pungi: add vagrant box creation
The commit messages should give a good idea of what is related to each
one. The biggest thing is working around a lorax/dnf bug for now in
patch 3.
5 years, 10 months
Old deltarpms are being thrown away on each compose (was Re: dnf
and deltarpm)
by Jonathan Dieter
On Thu, 2018-05-31 at 22:34 +0100, Tomasz Kłoczko wrote:
> Just checked on few mirrors usual location of f28 updates
> (/pub/linux/dist/fedora/linux/updates/28/Everything/x86_64/drpms) and
> in this directory there are at the moment only 56 files from May 31
> and nothing older. So not two drpm per RPM package but only generated
> files out of last batch of updates.
(CC'ing the Fedora infrastructure list)
This is a bug. It looks like we're throwing away any drpms not
generated in this compose, when we should be keeping them.
I've just created:
https://pagure.io/fedora-infrastructure/issue/7008
Jonathan
5 years, 10 months
