Shared Redis instance

[Aurelien Bompard]

Hey folks!

The new version of FMN will run in OpenShift and will use Redis as a
cache backends (we chose it over memcached because it can do native
"is-this-string-in-this-set" operations).

I can deploy redis inside my openshift project easily enough , but I
was wondering if it would be worthwhile to have a shared Redis
instance, like we have a shared PostgreSQL instance.
It's not just for ease of use, but I expect to store quite a bit of
data in our Redis instance, and since we don't attach persistent
storage to OpenShift that means that it will live in the pod's memory.
So I'm being conscious of the memory hog it can become.
Unless I'm mistaken there can be several databases in the same Redis
instance, so we could share it between projects without stepping on
each other's toes.

What do you think?

[darknao]

Hey!

I'm not sure it's worth doing. While I don't think this is a bad idea, I 
can't find any real gains to a shared instance in an Openshift world.
I do see a few downsides though:
- You'll need to share the same redis password across several projects. 
I see that as a potential security issue.
- Since you'll use an emptyDir (in-memory storage), every restart will 
flush the cache for all connected applications.
- Applications owners lose control over the redis instance in case they 
want to do some fancy stuff with it, or just general debugging.

The resource footprint of a redis container is quite low.
We have a couple of redis pods deployed already, and the memory usage 
for some of them is as low as 14Mb.

[Kevin Fenzi]

On Thu, Nov 24, 2022 at 10:56:57AM +0100, Aurelien Bompard wrote:
>  Hey folks!
>  
>  The new version of FMN will run in OpenShift and will use Redis as a
>  cache backends (we chose it over memcached because it can do native
>  "is-this-string-in-this-set" operations).
>  
>  I can deploy redis inside my openshift project easily enough , but I
>  was wondering if it would be worthwhile to have a shared Redis
>  instance, like we have a shared PostgreSQL instance.
>  It's not just for ease of use, but I expect to store quite a bit of
>  data in our Redis instance, and since we don't attach persistent
>  storage to OpenShift that means that it will live in the pod's memory.
>  So I'm being conscious of the memory hog it can become.
>  Unless I'm mistaken there can be several databases in the same Redis
>  instance, so we could share it between projects without stepping on
>  each other's toes.
>  
>  What do you think? 
We have talked about it before, but I think the tradeoffs come down on
the side of seperate instances. They aren't too hard to spin up in
openshift. 

It avoids a single point of failure for a bunch of services. 

Contention/resource problems. (ie, one app is hammering the shared
instance and starving other apps for resources). 

etc.

So, I would say we should do seperate ones per service...

kevin

[Aurelien Bompard]

>  - You'll need to share the same redis password across several
> projects. 
Redis does have users and permissions, at least from a quick look at
their docs:
https://docs.redis.com/latest/rc/security/database-security/passwords-use...

>  - Since you'll use an emptyDir (in-memory storage), every restart
> will flush the cache for all connected applications. 
I was thinking of running Redis in a VM, not in OpenShift. Sorry if
that wasn't clear in my initial message.

>  - Applications owners lose control over the redis instance in case
> they want to do some fancy stuff with it, or just general debugging. 
True.

>  It avoids a single point of failure for a bunch of services. 
Right, but that's what our PostgreSQL host is at the moment already.

>  Contention/resource problems. (ie, one app is hammering the shared
> instance and starving other apps for resources). 
True, true.

OK, that makes sense. The good thing about having a central Redis DB
was, in my mind, to have persistent storage. What happens if I store a
lot of data in the Redis Openshift pod? Won't that hit a memory limit?
I think our current usage of Redis has been pubsub and light cache,
but we haven't stored a lot of data in there yet.

Le lun. 28 nov. 2022 à 01:17, Kevin Fenzi <kevin(a)scrye.com&gt; a écrit :
>
> On Thu, Nov 24, 2022 at 10:56:57AM +0100, Aurelien Bompard wrote:
> > Hey folks!
> >
> > The new version of FMN will run in OpenShift and will use Redis as a
> > cache backends (we chose it over memcached because it can do native
> > "is-this-string-in-this-set" operations).
> >
> > I can deploy redis inside my openshift project easily enough , but I
> > was wondering if it would be worthwhile to have a shared Redis
> > instance, like we have a shared PostgreSQL instance.
> > It's not just for ease of use, but I expect to store quite a bit of
> > data in our Redis instance, and since we don't attach persistent
> > storage to OpenShift that means that it will live in the pod's memory.
> > So I'm being conscious of the memory hog it can become.
> > Unless I'm mistaken there can be several databases in the same Redis
> > instance, so we could share it between projects without stepping on
> > each other's toes.
> >
> > What do you think?
>
> We have talked about it before, but I think the tradeoffs come down on
> the side of seperate instances. They aren't too hard to spin up in
> openshift.
>
>  It avoids a single point of failure for a bunch of services.
>
> Contention/resource problems. (ie, one app is hammering the shared
> instance and starving other apps for resources).
>
> etc.
>
> So, I would say we should do seperate ones per service...
>
> kevin
> _______________________________________________
> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Kevin Fenzi]

On Mon, Nov 28, 2022 at 10:49:28AM +0100, Aurelien Bompard wrote:
>  > - You'll need to share the same redis password across
> several projects.
>  
>  Redis does have users and permissions, at least from a quick look at
>  their docs:
> https://docs.redis.com/latest/rc/security/database-security/passwords-use...
>  
>  > - Since you'll use an emptyDir (in-memory storage), every restart will flush the
> cache for all connected applications.
>  
>  I was thinking of running Redis in a VM, not in OpenShift. Sorry if
>  that wasn't clear in my initial message.
>  
>  > - Applications owners lose control over the redis instance in case they want to do
> some fancy stuff with it, or just general debugging.
>  
>  True.
>  
>  > It avoids a single point of failure for a bunch of services.
>  
>  Right, but that's what our PostgreSQL host is at the moment already. 
Yeah, true. I have thought about splitting that out too, but I am not
sure if I think it's a good idea to have databases in openshift and
making them vm's adds overhead of more vm's.

>  > Contention/resource problems. (ie, one app is hammering the
> shared instance and starving other apps for resources).
>  
>  True, true.
>  
>  OK, that makes sense. The good thing about having a central Redis DB
>  was, in my mind, to have persistent storage. What happens if I store a
>  lot of data in the Redis Openshift pod? Won't that hit a memory limit?
>  I think our current usage of Redis has been pubsub and light cache,
>  but we haven't stored a lot of data in there yet. 
Well, we can actually do persistent storage in the ocp4 cluster. ;) 

There's nfs volumes, but also there's a local ceph storage (using disk
on the compute nodes). I'm not sure how slow/fast it might be, but it is
there...

kevin

[Aurelien Bompard]

>  Well, we can actually do persistent storage in the ocp4 cluster. ;)
Oh, that's interesting! Are we using it already in one of our
ansible-deployed apps?

>  I'm not sure how slow/fast it might be, but it is there...
I think it's fine, Redis will use memory first and snapshot to disk
periodically, so disk speed should not be an issue.
That said, now that I look more into the docs it means that if I try
to store more data than what the memory allows, it'll evict data and
not use the disk for that. So having persistent storage for Redis only
helps with pod restarts. Which is still useful.

A.

[darknao]

On 2022-11-29 15:55, Aurelien Bompard wrote:
> > Well, we can actually do persistent storage in the ocp4 cluster.
> ;)
>  
>  Oh, that's interesting! Are we using it already in one of our
>  ansible-deployed apps? 
You can take the meetbot app as an example:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/m...

There are 4 types of persistent storage available you can use:
- NFS (no storageClass): poor performance, need to be provisioned 
beforehand on Netapp. Can be shared by multiple pods and outside of 
Openshift.
- RBD (storageClass: ocs-storagecluster-ceph-rbd): Block storage. 
Openshift will create an ext4 FS on top of it for you by default. 
Provide fast performance, but can only be accessed by one node at a 
time.
- CephFS (storageClass: ocs-storagecluster-cephfs): Shared Filesystem 
storage. It's pretty much like NFS. Can be accessed by several pods 
simultaneously.
- S3 (storageClass: openshift-storage.noobaa.io): Object storage. 
Require specific support from the application to use it. I don't think 
Redis support that kind of storage.

Basically, if you want more than 1 replica/pod to access your storage, 
pick CephFS. If you need max perf and single access (like a PSQL 
database), use RBD. If you want your data accessible from outside 
Openshift (i.e: on another VM), use NFS.
With the exception of S3, all other storage types are used the same way. 
Only the PVC definition is different.

-darknao