On Fri, 26 Jun 2020 09:43:54 -0700
Kevin Fenzi <kevin(a)scrye.com> wrote:
> I'd be very much in favour of having an Infra managed
> instance (+ grafana and alertmanager on Openshift), its something I
> hoped to work on within CPE sustaining infact.
You know, I'm not in love with that stack. It could well be that I
just haven't used it enough or know enough about it, but it seems just
needlessly complex. ;(
I'd prefer we start out at a lower level... what are our requirements?
Then, see how we can setup something to meet those.
Off the top of my head (I'm sure I can think of more):
* Ability to collect/gather rsyslog output from all our machines.
* Ability to generate reports of 'variances' from all that (ie, what
odd messages should a human look at?)
* Handle all the logs from openshift, possibly multiple clusters?
* Ability to easily drill down and look at some specifc historical
logs (ie, show me the logs for the bodhi-web pods from last week when
there was a issue).
Perhaps prometheus/graphana/alertmanager is the solution, but there's
also tons of other open source projects out there too that we might
I appreciate this reply because it contains very relevant technical
points, but it highlights that we are looking at different problems
at different levels.
Your list of requirements basically describes an "ingestion + storage +
anomaly inference for logs" solution, where the infrastructure and the
applications running on top of it are blended together.
This would be handy to have and it's indeed a good problem to work on
(I don't have specific answers/suggestions for this).
But that's not the space I'm looking at, nor the one that Prometheus
applies to. Instead, the gap/usecase is narrower and more akin to "SNMP
counters for containerized web services".
Logs are surely useful to drill-down into problems and investigate
root-causes, but that comes after being able to answer "is any of those
web services experiencing non-transient issues".
Instrumented applications and a metrics-gathering system enable this
kind of observation before sorting through logs, in the same way that
SNMP counters allow you to gauge if any switch port on your network is
seeing an eyebrow-raising amount of traffic, without having to log every
single frame-processing event and having to infer traffic levels from
So, for a developer/operator of an instrumented service running as
"openshift-apps", the requirements would be:
* Existence of an in-cluster component which periodically collects
* Short-term retention of those metrics into time-series (2 to 4 weeks)
* Ability to interactively query those metrics, at a point in time
* Ability to interactively graph those metrics, over a recent timespan
* (optional) Ability to persist the timespan-graphs and organize them
* (optional²) Ability to define thresholds for anomalous metrics and
getting alerts on such events
That is, most of the value is in collecting/recording/querying metrics
to answer "is everything fine today" (Prometheus).
Dashboarding helps in cutting down the amount of manual interactive
queries (optional, Grafana).
Alerting on metrics helps skipping active observations, but formalizing
the right anomaly thresholds is hard (optional, AlertManager; but for
my personal needs I honestly wouldn't venture into that now).
As a real world example, this is a buggy client behavior we noticed
by observing metrics exposed from one of our services running under
Detecting and adjusting the behavior in turn removed a potential
thundering-herd effect on the service, before actually cascading onto
other infra components.
Possibly this was a lucky case, but not a single line of logs was
involved into this investigation, as the clients are instrumented and
can be observed in the same way too.