11:28 a.m.
Hi everyone!
The Websites & Apps team is currently working on rewriting all major Fedora websites
(such as getfedora, spins & alt) and I believe this is a good opportunity to revisit
the current deployment workflow and try to make it simpler.
Currently, websites are being built in Openshift, with a cronjob running every hour that
fetches code from git, builds it, then saves it to an NFS share.
That same NFS share is also mounted on sundries01, which exposes it through rsync for
proxies to sync it, then serve it to the world.
I have a few solutions in mind to replace that, and I would like your input on them.
A) Full Openshift
We can build, and deploy the websites directly in Openshift, and serve it from here.
Just like silverblue.fp-o.
While this is probably the most straightforward solution, it has one major downside:
if Openshift is unavailable for any reason, our major websites become also unavailable.
I believe this is why we are still using our proxies to host them, as such a scenario
is unlikely to happen on every single one of them at the same time.
B) Same as before, with a twist
We build on Openshift, but instead of going through NFS and sundries with rsync, we
store the websites on S3 storage provided by Openshift, then we sync the proxies using
`s3cmd sync`.
C) Same as B, but with an external builder
We already build the new websites on Gitlab CI, and since the S3 gateway is accessible
from the outside, we could just push the build artifacts to s3 directly from GitLab CI.
Then sync the proxies from it.
D) Keep using the Openshift->Sundries->proxies workflow
E) Your solution here.
We could also improve B and C by adding fedora-messaging to the mix to trigger a proxy
resync as soon as a new build is available instead of doing so every hour.
What do you all think?
-darknao
11:58 a.m.
What is likelyhood of Openshift going down? A would be best solution if stable enough.
copperi
________________________________
From: Francois Andrieu <darknao(a)fedoraproject.org>
Sent: Thursday, November 24, 2022 7:28 PM
To: infrastructure(a)lists.fedoraproject.org <infrastructure(a)lists.fedoraproject.org>
Subject: Rethinking fedora websites deployment
Hi everyone!
The Websites & Apps team is currently working on rewriting all major Fedora websites
(such as getfedora, spins & alt) and I believe this is a good opportunity to revisit
the current deployment workflow and try to make it simpler.
Currently, websites are being built in Openshift, with a cronjob running every hour that
fetches code from git, builds it, then saves it to an NFS share.
That same NFS share is also mounted on sundries01, which exposes it through rsync for
proxies to sync it, then serve it to the world.
I have a few solutions in mind to replace that, and I would like your input on them.
A) Full Openshift
We can build, and deploy the websites directly in Openshift, and serve it from here.
Just like silverblue.fp-o.
While this is probably the most straightforward solution, it has one major downside:
if Openshift is unavailable for any reason, our major websites become also unavailable.
I believe this is why we are still using our proxies to host them, as such a scenario
is unlikely to happen on every single one of them at the same time.
B) Same as before, with a twist
We build on Openshift, but instead of going through NFS and sundries with rsync, we
store the websites on S3 storage provided by Openshift, then we sync the proxies using
`s3cmd sync`.
C) Same as B, but with an external builder
We already build the new websites on Gitlab CI, and since the S3 gateway is accessible
from the outside, we could just push the build artifacts to s3 directly from GitLab CI.
Then sync the proxies from it.
D) Keep using the Openshift->Sundries->proxies workflow
E) Your solution here.
We could also improve B and C by adding fedora-messaging to the mix to trigger a proxy
resync as soon as a new build is available instead of doing so every hour.
What do you all think?
-darknao
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
11:59 a.m.
My vote is for C. If we can reduce the amount of steps required to
implement service redundancy. Deployment to S3 provides out of the box
access to the public domain and is straightforward.
Regards,
Ahmed Al-meleh
Fedora QA Contributor
On Thu, 24 Nov 2022, 17:28 Francois Andrieu, <darknao(a)fedoraproject.org>
wrote:
Hi everyone!
The Websites & Apps team is currently working on rewriting all major
Fedora websites (such as getfedora, spins & alt) and I believe this is a
good opportunity to revisit the current deployment workflow and try to make
it simpler.
Currently, websites are being built in Openshift, with a cronjob running
every hour that fetches code from git, builds it, then saves it to an NFS
share.
That same NFS share is also mounted on sundries01, which exposes it
through rsync for proxies to sync it, then serve it to the world.
I have a few solutions in mind to replace that, and I would like your
input on them.
A) Full Openshift
We can build, and deploy the websites directly in Openshift, and serve
it from here. Just like silverblue.fp-o.
While this is probably the most straightforward solution, it has one
major downside: if Openshift is unavailable for any reason, our major
websites become also unavailable.
I believe this is why we are still using our proxies to host them, as
such a scenario is unlikely to happen on every single one of them at the
same time.
B) Same as before, with a twist
We build on Openshift, but instead of going through NFS and sundries
with rsync, we store the websites on S3 storage provided by Openshift, then
we sync the proxies using `s3cmd sync`.
C) Same as B, but with an external builder
We already build the new websites on Gitlab CI, and since the S3
gateway is accessible from the outside, we could just push the build
artifacts to s3 directly from GitLab CI. Then sync the proxies from it.
D) Keep using the Openshift->Sundries->proxies workflow
E) Your solution here.
We could also improve B and C by adding fedora-messaging to the mix to
trigger a proxy resync as soon as a new build is available instead of doing
so every hour.
What do you all think?
-darknao
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to
infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
6:07 a.m.
I like C too.
Currently, when something breaks on the websites (the most common issue
is outdated content), the websites team needs to reach out to infra to
understand what's happening and ask them to check the build logs.
Using Openshift is not a widespread skill, and it can be a bit difficult
to debug anything when you don't know where to look, how to trigger a
new build, or just don't have any access.
With C, we are offloading that task to the website team, who are then
able to use the tools they know to deploy and solve any issues related
to the build process that may arise.
Now, the real question is, are we going to allow that?
Giving access to such s3 storage to a third party (I'm talking about
GitLab here since the s3 access key will be stored on their platform)
can be a potential security concern.
If this key gets stolen, it basically gives direct access to our
proxies.
I feel like I've somewhat answered my own question, but I would love
your opinion on this :)
-darknao
On 2022-11-24 18:59, Ahmed Almeleh wrote:
> My vote is for C. If we can reduce the amount of steps required to
> implement service redundancy. Deployment to S3 provides out of the
> box access to the public domain and is straightforward.
>
> Regards,
> Ahmed Al-meleh
> Fedora QA Contributor
>
> On Thu, 24 Nov 2022, 17:28 Francois Andrieu,
> <darknao(a)fedoraproject.org> wrote:
>
>> B) Same as before, with a twist
>> We build on Openshift, but instead of going through NFS and
>> sundries with rsync, we store the websites on S3 storage provided by
>> Openshift, then we sync the proxies using `s3cmd sync`.
>>
>> C) Same as B, but with an external builder
>> We already build the new websites on Gitlab CI, and since the S3
>> gateway is accessible from the outside, we could just push the build
>> artifacts to s3 directly from GitLab CI. Then sync the proxies from
>> it.
>>
>> -darknao
6:32 p.m.
On Thu, Nov 24, 2022 at 05:28:22PM -0000, Francois Andrieu wrote:
Hi everyone!
Hello.
The Websites & Apps team is currently working on rewriting all
major Fedora websites (such as getfedora, spins & alt) and I believe this is a good
opportunity to revisit the current deployment workflow and try to make it simpler.
Indeed. I have often thought about this workflow. ;)
Currently, websites are being built in Openshift, with a cronjob
running every hour that fetches code from git, builds it, then saves it to an NFS share.
That same NFS share is also mounted on sundries01, which exposes it through rsync for
proxies to sync it, then serve it to the world.
I have a few solutions in mind to replace that, and I would like your input on them.
A) Full Openshift
We can build, and deploy the websites directly in Openshift, and serve it from here.
Just like silverblue.fp-o.
While this is probably the most straightforward solution, it has one major downside:
if Openshift is unavailable for any reason, our major websites become also unavailable.
I believe this is why we are still using our proxies to host them, as such a scenario
is unlikely to happen on every single one of them at the same time.
Yep. That and also proxies are usually 'closer' to the end user and
serving static content, so it's much faster network wise. ie, say a user
in germany would just hit a german proxy and get the content fast,
while if we moved it to openshift they would have to transit all the way
over to the us and back to get that content.
B) Same as before, with a twist
We build on Openshift, but instead of going through NFS and sundries with rsync, we
store the websites on S3 storage provided by Openshift, then we sync the proxies using
`s3cmd sync`.
I think this is definitely a improvement over the current setup.
C) Same as B, but with an external builder
We already build the new websites on Gitlab CI, and since the S3 gateway is
accessible from the outside, we could just push the build artifacts to s3 directly from
GitLab CI. Then sync the proxies from it.
Or just pull from gitlab directly? Or does it expose that data in a way
we could sync from?
D) Keep using the Openshift->Sundries->proxies workflow
I'd like to move to something better once we decide whats worth trying.
:)
E) Your solution here.
Some more I have thought on:
E) a twist on A. We build and serve in openshift, but we stick
cloudfront in front of it. This would solve the speed problems, but
still would have the openshift down issue.
F) (this is a fun one :) How about looking into FCOS or RHEL for edge?
In this model we would install ostree based vm's in the places we have
proxies now and we would build the web content as a ostree ref and pull
it from our registry (or quay.io). I think this would be fun, but
probibly overkill/too much effort for just static content, but I thought
I would throw it out there.
We could also improve B and C by adding fedora-messaging to the mix
to trigger a proxy resync as soon as a new build is available instead of doing so every
hour.
+1 (although we should make sure not to thundering herd the endpoint if
all proxies decided to sync at the same instant).
What do you all think?
I like B... but possibly could be talked into C.
The thing I don't like about C is that it has less visibility, if there
was a problem, it would require someone from websites to fix, rather
than possibly being something that anyone with access to openshift could
fix.
kevin
6:35 p.m.
On Fri, Nov 25, 2022 at 01:07:50PM +0100, darknao wrote:
I like C too.
Currently, when something breaks on the websites (the most common issue is
outdated content), the websites team needs to reach out to infra to
understand what's happening and ask them to check the build logs.
Using Openshift is not a widespread skill, and it can be a bit difficult to
debug anything when you don't know where to look, how to trigger a new
build, or just don't have any access.
Yeah, although it works the other way too... if we move to gitlab it
would need someone who understands that setup to debug and fix.
With C, we are offloading that task to the website team, who are then
able
to use the tools they know to deploy and solve any issues related to the
build process that may arise.
Sure, but we are then saying that there would be someone available to
fix things for... all the time we are still running things there. ;)
Now, the real question is, are we going to allow that?
Giving access to such s3 storage to a third party (I'm talking about GitLab
here since the s3 access key will be stored on their platform) can be a
potential security concern.
Sure, but we should hopefully be able to make sure nothing else would be
accessable to that key.
If this key gets stolen, it basically gives direct access to our
proxies.
True, we would need to make sure it was as secure as we could make it.
I feel like I've somewhat answered my own question, but I would
love your
opinion on this :)
kevin
--
-darknao
On 2022-11-24 18:59, Ahmed Almeleh wrote:
> My vote is for C. If we can reduce the amount of steps required to
> implement service redundancy. Deployment to S3 provides out of the
> box access to the public domain and is straightforward.
>
> Regards,
> Ahmed Al-meleh
> Fedora QA Contributor
>
> On Thu, 24 Nov 2022, 17:28 Francois Andrieu,
> <darknao(a)fedoraproject.org> wrote:
>
> > B) Same as before, with a twist
> > We build on Openshift, but instead of going through NFS and
> > sundries with rsync, we store the websites on S3 storage provided by
> > Openshift, then we sync the proxies using `s3cmd sync`.
> >
> > C) Same as B, but with an external builder
> > We already build the new websites on Gitlab CI, and since the S3
> > gateway is accessible from the outside, we could just push the build
> > artifacts to s3 directly from GitLab CI. Then sync the proxies from
> > it.
> >
> > -darknao
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
7:24 a.m.
On 2022-11-28 01:32, Kevin Fenzi wrote:
Some more I have thought on:
E) a twist on A. We build and serve in openshift, but we stick
cloudfront in front of it. This would solve the speed problems, but
still would have the openshift down issue.
I'm not familiar with Cloudfront, so I can't really comment on that one.
F) (this is a fun one :) How about looking into FCOS or RHEL for
edge?
In this model we would install ostree based vm's in the places we have
proxies now and we would build the web content as a ostree ref and pull
it from our registry (or quay.io). I think this would be fun, but
probibly overkill/too much effort for just static content, but I
thought
I would throw it out there.
That does sounds a bit overkill yes :) I'm not expert on the subject,
but I believe applying a new ostree ref requires a reboot to use it,
right?
I was considering something similar in the past: Build the websites into
a container image, then pull & extract the content on the proxies.
One downside to that is every build creates a new image (or at least a
new layer), that proxies will have to pull every time.
With rsync (or s3 sync) we only download what actually changes and save
bandwidth in the long run.
I like B... but possibly could be talked into C.
The thing I don't like about C is that it has less visibility, if there
was a problem, it would require someone from websites to fix, rather
than possibly being something that anyone with access to openshift
could
fix.
That's a valid point.
I think I'll go with B as a starting point. We could always build from
there at a later time, as we see fit.
-darknao
9:39 a.m.
On Mon, Nov 28, 2022 at 02:24:14PM +0100, darknao wrote:
On 2022-11-28 01:32, Kevin Fenzi wrote:
> Some more I have thought on:
>
> E) a twist on A. We build and serve in openshift, but we stick
> cloudfront in front of it. This would solve the speed problems, but
> still would have the openshift down issue.
I'm not familiar with Cloudfront, so I can't really comment on that one.
> F) (this is a fun one :) How about looking into FCOS or RHEL for edge?
> In this model we would install ostree based vm's in the places we have
> proxies now and we would build the web content as a ostree ref and pull
> it from our registry (or quay.io). I think this would be fun, but
> probibly overkill/too much effort for just static content, but I thought
> I would throw it out there.
That does sounds a bit overkill yes :) I'm not expert on the subject, but I
believe applying a new ostree ref requires a reboot to use it, right?
yep.
I was considering something similar in the past: Build the websites
into a
container image, then pull & extract the content on the proxies.
One downside to that is every build creates a new image (or at least a new
layer), that proxies will have to pull every time.
With rsync (or s3 sync) we only download what actually changes and save
bandwidth in the long run.
Yeah, and if we run in a pod or a ostree setup, we have to duplicate the
proxy stack (ie, httpd, varnish, etc), where as if we just copy the
content we can leverage the existing software stack.
> I like B... but possibly could be talked into C.
>
> The thing I don't like about C is that it has less visibility, if there
> was a problem, it would require someone from websites to fix, rather
> than possibly being something that anyone with access to openshift could
> fix.
That's a valid point.
I think I'll go with B as a starting point. We could always build from there
at a later time, as we see fit.
Yeah, thats sounds good. We can always adjust, and B will still be a
nice improvement, especially if we get it to sync on message (although
we should be sure to have some manual way to force a 'sync now' too).
kevin
midnight
On Sun, Nov 27, 2022 at 5:32 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
> C) Same as B, but with an external builder
> We already build the new websites on Gitlab CI, and since the S3 gateway is
accessible from the outside, we could just push the build artifacts to s3 directly from
GitLab CI. Then sync the proxies from it.
Or just pull from gitlab directly? Or does it expose that data in a way
we could sync from?
Gitlab CI supports downloadable artifacts. A tarball seems most
appropriate here but I often build RPMs myself. :-D
Here's a simple CI script that build and tars the website up:
https://gitlab.com/tchollingsworth/fedora-websites/-/blob/6787d86f6e81c84...
And a stable link by branch and commit to the tarball it generated:
https://gitlab.com/api/v4/projects/tchollingsworth%2Ffedora-websites/jobs...
https://gitlab.com/api/v4/projects/tchollingsworth%2Ffedora-websites/jobs...
> We could also improve B and C by adding fedora-messaging to the
mix to trigger a proxy resync as soon as a new build is available instead of doing so
every hour.
+1 (although we should make sure not to thundering herd the endpoint if
all proxies decided to sync at the same instant).
Gitlab can call a webhook when the build is completed, which could
trigger (a bus message that fires off) a sync. I see CentOS developed
webhooks that forward commits, issues, etc. to the fedora-messaging
bus, not sure if they were ever put into production or if they include
CI successes and failures, but maybe half the work needed for this is
already done.
Yeah, although it works the other way too... if we move to gitlab it
would need someone who understands that setup to debug and fix.
I spent more time messing around with stuff like rolling back to
Fedora 34 because the flask-assets dependency it needed was retired in
the current release than I did with anything that was gitlab's fault.
;-)
On Mon, Nov 28, 2022 at 6:24 AM darknao <darknao(a)fedoraproject.org> wrote:
One downside to that is every build creates a new image (or at least
a
new layer), that proxies will have to pull every time.
With rsync (or s3 sync) we only download what actually changes and save
bandwidth in the long run.
Maybe my build isn't working all the way but tarball above is 11MB
gzipped and 30MB uncompressed so probably not a big burden, especially
if you only build when someone pushes.
508
days inactive
518
days old
infrastructure@lists.fedoraproject.org
8 comments
6 participants
participants (6)
-
Ahmed Almeleh -
darknao -
Francois Andrieu
-
Jan K -
Kevin Fenzi -
T.C. Hollingsworth