openshift applications best practices

[Kevin Fenzi]

So, when moving openshift apps this week, I really came to the
conclusion we should look at writing up some guidelines for openshift
apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
'SUGGESTION' items. 

I know I am coming at this from the operations side of things, so please
do share other viewpoints, but let me mention a few issues I see: 

It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is
  running depends on when it was last built/deployed. (unless: see
  below)
- A number of apps use images elsewhere (quay.io, etc). We have very
  little way to tell whats in those or how they are made or if anyone is
  still making them.

We have a few apps that rebuild/deploy on git changes, which is fine,
but doesn't do anything to keep their container env up to date/secure. 

We have a number of apps that tigger on imagestream changes, but as far
as I can tell thats only changes in their image ('bodhi-base') not the
base image that it was built from ('fedora:36'). I'd like to see us
track base images and build/deploy when one changes. If we do that then
:latest might be ok to use, since we know automation took care of
updating it for us.

There's a number of applications using ubi (rhel's base image), which I
think might be worth recommending, especially if we start build/deploy
on base image changes. Likely Fedora base images will change more
quickly. 

Currently we have some apps that were setup and probibly have not
changes in years, happily running the same old container image since
there was no reason to redeploy them.

Similarly, python versions are all over the place... python3.6, 3.8,
3.9, etc. Of course different apps will target different versions
sometimes, but it might be nice to make sure we are using a supported
version by whatever container image the app is using. (ie, gets security
fixes).

Our naming conventions are a bit all over the place too. Some apps have
$appname-build for buildconfig, some just $appname, some seperate
configs for seperate containers ($app-web, $app-frontend). Same for
deploymentconfigs. 

Anyhow, I'll stop there... but hopefully everyone would be ok with such
a doc and then changing things to match it?

kevin

[Andrew Heath]

+1 to this. What I would like to know is their away we can do
approval/check if certain requirements are met before a build kicks off,
that may help enforce requirements.

On Fri, Jun 17, 2022, 4:56 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  So, when moving openshift apps this week, I really came to the
>  conclusion we should look at writing up some guidelines for openshift
>  apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
>  'SUGGESTION' items.
>
>  I know I am coming at this from the operations side of things, so please
>  do share other viewpoints, but let me mention a few issues I see:
>
>  It's very difficult to tell what an application is running on:
>  - We should probibly forbid 'X:latest' because what the application is
>    running depends on when it was last built/deployed. (unless: see
>    below)
>  - A number of apps use images elsewhere (quay.io, etc). We have very
>    little way to tell whats in those or how they are made or if anyone is
>    still making them.
>
>  We have a few apps that rebuild/deploy on git changes, which is fine,
>  but doesn't do anything to keep their container env up to date/secure.
>
>  We have a number of apps that tigger on imagestream changes, but as far
>  as I can tell thats only changes in their image ('bodhi-base') not the
>  base image that it was built from ('fedora:36'). I'd like to see us
>  track base images and build/deploy when one changes. If we do that then
>  :latest might be ok to use, since we know automation took care of
>  updating it for us.
>
>  There's a number of applications using ubi (rhel's base image), which I
>  think might be worth recommending, especially if we start build/deploy
>  on base image changes. Likely Fedora base images will change more
>  quickly.
>
>  Currently we have some apps that were setup and probibly have not
>  changes in years, happily running the same old container image since
>  there was no reason to redeploy them.
>
>  Similarly, python versions are all over the place... python3.6, 3.8,
>  3.9, etc. Of course different apps will target different versions
>  sometimes, but it might be nice to make sure we are using a supported
>  version by whatever container image the app is using. (ie, gets security
>  fixes).
>
>  Our naming conventions are a bit all over the place too. Some apps have
>  $appname-build for buildconfig, some just $appname, some seperate
>  configs for seperate containers ($app-web, $app-frontend). Same for
>  deploymentconfigs.
>
>  Anyhow, I'll stop there... but hopefully everyone would be ok with such
>  a doc and then changing things to match it?
>
>  kevin
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Jon Stanley]

Welcome to the hell at $DAYJOB :). Some comments below.....

On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  So, when moving openshift apps this week, I really came to the
>  conclusion we should look at writing up some guidelines for openshift
>  apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
>  'SUGGESTION' items. 
Yep, sounds good. We have something that we call the "big rules" which
fit some of the things here.
>  I know I am coming at this from the operations side of things, so please
>  do share other viewpoints, but let me mention a few issues I see:
>
>  It's very difficult to tell what an application is running on:
>  - We should probibly forbid 'X:latest' because what the application is
>    running depends on when it was last built/deployed. (unless: see
>    below) 
Yep. It's a bad practice to use "latest", especially for an image that
you didn't build. This leaves you with zero control over potentially
breaking changes

>  - A number of apps use images elsewhere (quay.io, etc). We have very
>    little way to tell whats in those or how they are made or if anyone is
>    still making them. 
This is an advantage that we have being a large enterprise - our
clusters have no Internet access and therefore all images come from a
registry that we control. Perhaps something similar would be valuable
here.
>  We have a few apps that rebuild/deploy on git changes, which is fine,
>  but doesn't do anything to keep their container env up to date/secure. 
One thing that's possible to fix this is to use vulnerability scanning
products. We use a proprietary one, but just recently Red Hat
open-sourced Stackrox (a company that was acquired a bit ago -
https://stackrox.io). I haven't used either the upstream or downstream
(Advanced Cluster Security for Kubernetes) version of it (though I did
build the upstream version just to see that it worked, it's somewhat
difficult to build and AFAIK there's no distribution
mechanism/prebuilt images for it - they threw some source code over
the wall AFAICT and are looking to improve the build process "soon" so
that they can ship prebuilt images - i.e., the build currently
requires Docker and doesn't seem to work with podman-docker (I
tried.....)

>  We have a number of apps that tigger on imagestream changes, but as
> far
>  as I can tell thats only changes in their image ('bodhi-base') not the
>  base image that it was built from ('fedora:36'). I'd like to see us
>  track base images and build/deploy when one changes. If we do that then
>  :latest might be ok to use, since we know automation took care of
>  updating it for us. 
Once you figure that out, let me know :). Bonus points if it doesn't
use imagestreams. You still don't want :latest though :).

>  There's a number of applications using ubi (rhel's base
> image), which I
>  think might be worth recommending, especially if we start build/deploy
>  on base image changes. Likely Fedora base images will change more
>  quickly. 
Yep, this is one of our "big rules" that we prefer ubi-based images.
For internally built code, we require it and for vendor apps we prefer
it. Since I imagine that we don't have many "vendor" apps (except
maybe stuff like Redis, which already exists as a Red Hat packaged
downstream image which should likely be preferred), the amount of
non-ubi images should be minimal I would think.

>  Currently we have some apps that were setup and probibly have not
>  changes in years, happily running the same old container image since
>  there was no reason to redeploy them. 
Yep :(

One of our requirements is that a pod lifetime is no longer than 2
weeks (we'll delete any pod older than 2 weeks, and the underlying
controller will recreate it). We also won't allow images with a
certain vulnerability rating to run, so you're pretty assured that
you're running current stuff. This might be a bit draconian for infra,
but something to consider/think about.

>  Similarly, python versions are all over the place... python3.6, 3.8,
>  3.9, etc. Of course different apps will target different versions
>  sometimes, but it might be nice to make sure we are using a supported
>  version by whatever container image the app is using. (ie, gets security
>  fixes). 
See above, once you have unpatched vulnerabilities you can't run, so
this is effectively mitigated.

>  Anyhow, I'll stop there... but hopefully everyone would be ok
> with such
>  a doc and then changing things to match it? 
Doc makes sense, and I might steal some of it :).

[Alex Senges]

Hi everyone can you give us a good free training on Docker, Kubernetes,
podman and Openshift. Thanks in advance!

On Fri, Jun 17, 2022, 23:59 Jon Stanley <jonstanley(a)gmail.com&gt; wrote:

>  Welcome to the hell at $DAYJOB :). Some comments below.....
>
>  On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  >
>  > So, when moving openshift apps this week, I really came to the
>  > conclusion we should look at writing up some guidelines for openshift
>  > apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
>  > 'SUGGESTION' items.
>
>  Yep, sounds good. We have something that we call the "big rules" which
>  fit some of the things here.
>  >
>  > I know I am coming at this from the operations side of things, so please
>  > do share other viewpoints, but let me mention a few issues I see:
>  >
>  > It's very difficult to tell what an application is running on:
>  > - We should probibly forbid 'X:latest' because what the application is
>  >   running depends on when it was last built/deployed. (unless: see
>  >   below)
>
>  Yep. It's a bad practice to use "latest", especially for an image that
>  you didn't build. This leaves you with zero control over potentially
>  breaking changes
>
>  > - A number of apps use images elsewhere (quay.io, etc). We have very
>  >   little way to tell whats in those or how they are made or if anyone is
>  >   still making them.
>
>  This is an advantage that we have being a large enterprise - our
>  clusters have no Internet access and therefore all images come from a
>  registry that we control. Perhaps something similar would be valuable
>  here.
>  >
>  > We have a few apps that rebuild/deploy on git changes, which is fine,
>  > but doesn't do anything to keep their container env up to date/secure.
>
>  One thing that's possible to fix this is to use vulnerability scanning
>  products. We use a proprietary one, but just recently Red Hat
>  open-sourced Stackrox (a company that was acquired a bit ago -
>  https://stackrox.io). I haven't used either the upstream or downstream
>  (Advanced Cluster Security for Kubernetes) version of it (though I did
>  build the upstream version just to see that it worked, it's somewhat
>  difficult to build and AFAIK there's no distribution
>  mechanism/prebuilt images for it - they threw some source code over
>  the wall AFAICT and are looking to improve the build process "soon" so
>  that they can ship prebuilt images - i.e., the build currently
>  requires Docker and doesn't seem to work with podman-docker (I
>  tried.....)
>
>  > We have a number of apps that tigger on imagestream changes, but as far
>  > as I can tell thats only changes in their image ('bodhi-base') not the
>  > base image that it was built from ('fedora:36'). I'd like to see us
>  > track base images and build/deploy when one changes. If we do that then
>  > :latest might be ok to use, since we know automation took care of
>  > updating it for us.
>
>  Once you figure that out, let me know :). Bonus points if it doesn't
>  use imagestreams. You still don't want :latest though :).
>
>  > There's a number of applications using ubi (rhel's base image), which I
>  > think might be worth recommending, especially if we start build/deploy
>  > on base image changes. Likely Fedora base images will change more
>  > quickly.
>
>  Yep, this is one of our "big rules" that we prefer ubi-based images.
>  For internally built code, we require it and for vendor apps we prefer
>  it. Since I imagine that we don't have many "vendor" apps (except
>  maybe stuff like Redis, which already exists as a Red Hat packaged
>  downstream image which should likely be preferred), the amount of
>  non-ubi images should be minimal I would think.
>
>  > Currently we have some apps that were setup and probibly have not
>  > changes in years, happily running the same old container image since
>  > there was no reason to redeploy them.
>
>  Yep :(
>
>  One of our requirements is that a pod lifetime is no longer than 2
>  weeks (we'll delete any pod older than 2 weeks, and the underlying
>  controller will recreate it). We also won't allow images with a
>  certain vulnerability rating to run, so you're pretty assured that
>  you're running current stuff. This might be a bit draconian for infra,
>  but something to consider/think about.
>
>  > Similarly, python versions are all over the place... python3.6, 3.8,
>  > 3.9, etc. Of course different apps will target different versions
>  > sometimes, but it might be nice to make sure we are using a supported
>  > version by whatever container image the app is using. (ie, gets security
>  > fixes).
>
>  See above, once you have unpatched vulnerabilities you can't run, so
>  this is effectively mitigated.
>
>  > Anyhow, I'll stop there... but hopefully everyone would be ok with such
>  > a doc and then changing things to match it?
>
>  Doc makes sense, and I might steal some of it :).
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Kevin Fenzi]

On Sat, Jun 18, 2022 at 08:47:44AM -0300, Alex Senges wrote:
>  Hi everyone can you give us a good free training on Docker,
> Kubernetes,
>  podman and Openshift. Thanks in advance! 
Well, we are not any kind of training org, but luckily all those things
(well, ok, except docker) are open source/free and you can play around
with them on your own machine. ;) 

I'm sure there's tons of tutorials and self paced training stuff out
there for containers. :)

kevin

[Alex Senges]

Hello Kevin, I asked this because I made a course on Red Hat about the
Openshift architecture and the course covered some topics of Docker,
Kubernetes and podman, and since this is the infrastructure Fedora team
which is essentially part of the Redhat culture, I thought that it was
important to have access to those kind of resources. That being said, I
apologize if my question was unclear.


Alex

*--*

*Just a human among others in the Uncanny valley.*


On Sat, Jun 18, 2022 at 1:53 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Sat, Jun 18, 2022 at 08:47:44AM -0300, Alex Senges wrote:
>  > Hi everyone can you give us a good free training on Docker, Kubernetes,
>  > podman and Openshift. Thanks in advance!
>
>  Well, we are not any kind of training org, but luckily all those things
>  (well, ok, except docker) are open source/free and you can play around
>  with them on your own machine. ;)
>
>  I'm sure there's tons of tutorials and self paced training stuff out
>  there for containers. :)
>
>  kevin
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Michal Konecny]

I'm +1 to have this documented in developer guidelines 
https://docs.fedoraproject.org/en-US/infra/developer_guide/

More replies inline.

On 18. 06. 22 4:58, Jon Stanley wrote:
>  Welcome to the hell at $DAYJOB :). Some comments below.....
>
>  On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
> > So, when moving openshift apps this week, I really came to the
> > conclusion we should look at writing up some guidelines for openshift
> > apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
> > 'SUGGESTION' items.
>  Yep, sounds good. We have something that we call the "big rules" which
>  fit some of the things here.
> > I know I am coming at this from the operations side of things, so please
> > do share other viewpoints, but let me mention a few issues I see:
> >
> > It's very difficult to tell what an application is running on:
> > - We should probibly forbid 'X:latest' because what the application is
> >    running depends on when it was last built/deployed. (unless: see
> >    below)
>  Yep. It's a bad practice to use "latest", especially for an image that
>  you didn't build. This leaves you with zero control over potentially
>  breaking changes 
When I last checked, I noticed that we have plenty of apps where
you are 
not even able to find what they are using, because it's not specified in 
the buildconfig. They are instead using some Dockerfile in the upstream 
project, which
isn't bad, but it could cause confusion and it's hard to track.
> > - A number of apps use images elsewhere (quay.io, etc). We have very
> >    little way to tell whats in those or how they are made or if anyone is
> >    still making them.
>  This is an advantage that we have being a large enterprise - our
>  clusters have no Internet access and therefore all images come from a
>  registry that we control. Perhaps something similar would be valuable
>  here.
> > We have a few apps that rebuild/deploy on git changes, which is fine,
> > but doesn't do anything to keep their container env up to date/secure.
>  One thing that's possible to fix this is to use vulnerability scanning
>  products. We use a proprietary one, but just recently Red Hat
>  open-sourced Stackrox (a company that was acquired a bit ago -
>  https://stackrox.io). I haven't used either the upstream or downstream
>  (Advanced Cluster Security for Kubernetes) version of it (though I did
>  build the upstream version just to see that it worked, it's somewhat
>  difficult to build and AFAIK there's no distribution
>  mechanism/prebuilt images for it - they threw some source code over
>  the wall AFAICT and are looking to improve the build process "soon" so
>  that they can ship prebuilt images - i.e., the build currently
>  requires Docker and doesn't seem to work with podman-docker (I
>  tried.....)
>
> > We have a number of apps that tigger on imagestream changes, but as far
> > as I can tell thats only changes in their image ('bodhi-base') not the
> > base image that it was built from ('fedora:36'). I'd like to see us
> > track base images and build/deploy when one changes. If we do that then
> > :latest might be ok to use, since we know automation took care of
> > updating it for us.
>  Once you figure that out, let me know :). Bonus points if it doesn't
>  use imagestreams. You still don't want :latest though :).
>
> > There's a number of applications using ubi (rhel's base image), which I
> > think might be worth recommending, especially if we start build/deploy
> > on base image changes. Likely Fedora base images will change more
> > quickly.
>  Yep, this is one of our "big rules" that we prefer ubi-based images.
>  For internally built code, we require it and for vendor apps we prefer
>  it. Since I imagine that we don't have many "vendor" apps (except
>  maybe stuff like Redis, which already exists as a Red Hat packaged
>  downstream image which should likely be preferred), the amount of
>  non-ubi images should be minimal I would think.
>
> > Currently we have some apps that were setup and probibly have not
> > changes in years, happily running the same old container image since
> > there was no reason to redeploy them.
>  Yep :(
>
>  One of our requirements is that a pod lifetime is no longer than 2
>  weeks (we'll delete any pod older than 2 weeks, and the underlying
>  controller will recreate it). We also won't allow images with a
>  certain vulnerability rating to run, so you're pretty assured that
>  you're running current stuff. This might be a bit draconian for infra,
>  but something to consider/think about. 
This sounds like a really good rule, how are
you addressing if anything 
gets broken during this recreation?
> > Similarly, python versions are all over the place... python3.6, 3.8,
> > 3.9, etc. Of course different apps will target different versions
> > sometimes, but it might be nice to make sure we are using a supported
> > version by whatever container image the app is using. (ie, gets security
> > fixes).
>  See above, once you have unpatched vulnerabilities you can't run, so
>  this is effectively mitigated.
>
> > Anyhow, I'll stop there... but hopefully everyone would be ok with such
> > a doc and then changing things to match it?
>  Doc makes sense, and I might steal some of it :).
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Neal Gompa]

On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  So, when moving openshift apps this week, I really came to the
>  conclusion we should look at writing up some guidelines for openshift
>  apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
>  'SUGGESTION' items.
>
>  I know I am coming at this from the operations side of things, so please
>  do share other viewpoints, but let me mention a few issues I see:
>
>  It's very difficult to tell what an application is running on:
>  - We should probibly forbid 'X:latest' because what the application is
>    running depends on when it was last built/deployed. (unless: see
>    below)
>  - A number of apps use images elsewhere (quay.io, etc). We have very
>    little way to tell whats in those or how they are made or if anyone is
>    still making them.
>
>  We have a few apps that rebuild/deploy on git changes, which is fine,
>  but doesn't do anything to keep their container env up to date/secure.
>
>  We have a number of apps that tigger on imagestream changes, but as far
>  as I can tell thats only changes in their image ('bodhi-base') not the
>  base image that it was built from ('fedora:36'). I'd like to see us
>  track base images and build/deploy when one changes. If we do that then
>  :latest might be ok to use, since we know automation took care of
>  updating it for us.
>
>  There's a number of applications using ubi (rhel's base image), which I
>  think might be worth recommending, especially if we start build/deploy
>  on base image changes. Likely Fedora base images will change more
>  quickly.
>  
Fedora's images churning more often is a good thing, not a bad thing.
And frankly, I would not recommend RHEL UBI for a few reasons:

1. It's hard to get things fixed in a timely fashion
2. The content set is too reduced from RHEL
3. We already control the Fedora content, so we should use those

Dogfooding is important, as is self-sufficiency. Inability to do both
makes it much more difficult for others to reuse our stuff.

-- 
真実はいつも一つ！/ Always, there's only one truth!

[darknao]

On 2022-06-17 22:56, Kevin Fenzi wrote:
>  We have a few apps that rebuild/deploy on git changes, which is
> fine,
>  but doesn't do anything to keep their container env up to date/secure.
>  
>  We have a number of apps that tigger on imagestream changes, but as far
>  as I can tell thats only changes in their image ('bodhi-base') not the
>  base image that it was built from ('fedora:36'). I'd like to see us
>  track base images and build/deploy when one changes. If we do that then
>  :latest might be ok to use, since we know automation took care of
>  updating it for us.
>  
>  There's a number of applications using ubi (rhel's base image), which I
>  think might be worth recommending, especially if we start build/deploy
>  on base image changes. Likely Fedora base images will change more
>  quickly.
>   
We should use imagestream for base image too.
One good way to achieve that is to have all these commonly used 
imagestreams
available in a shared namespace (just like the openshift namespace for 
images catalog).
We can configure them to automatically track remote images (ubi, fedora, 
...),
then it's really easy for developers to just add the required trigger to 
their buildconfig to keep their own images up-to-date.

IMHO using :latest is fine but require proper monitoring. Currently,
if a build fail, the application owner is not notified, and will most 
likely never fix it.

[Michal Konecny]

On 20. 06. 22 11:21, darknao wrote:
>  On 2022-06-17 22:56, Kevin Fenzi wrote:
> > We have a few apps that rebuild/deploy on git changes, which is fine,
> > but doesn't do anything to keep their container env up to date/secure.
> >
> > We have a number of apps that tigger on imagestream changes, but as far
> > as I can tell thats only changes in their image ('bodhi-base') not the
> > base image that it was built from ('fedora:36'). I'd like to see us
> > track base images and build/deploy when one changes. If we do that then
> > :latest might be ok to use, since we know automation took care of
> > updating it for us.
> >
> > There's a number of applications using ubi (rhel's base image), which I
> > think might be worth recommending, especially if we start build/deploy
> > on base image changes. Likely Fedora base images will change more
> > quickly.
> >
>
>  We should use imagestream for base image too.
>  One good way to achieve that is to have all these commonly used 
>  imagestreams
>  available in a shared namespace (just like the openshift namespace for 
>  images catalog).
>  We can configure them to automatically track remote images (ubi, 
>  fedora, ...),
>  then it's really easy for developers to just add the required trigger 
>  to their buildconfig to keep their own images up-to-date.
>
>  IMHO using :latest is fine but require proper monitoring. Currently,
>  if a build fail, the application owner is not notified, and will most 
>  likely never fix it. 
Notifications about failures would be awesome. I'm usually
not aware if 
there is something like that happening till the service is unavailable 
for users and somebody reports it on application tracker.
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to 
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: 
>  https://pagure.io/fedora-infrastructure