Freeze break request: forward openshift logs to log01

[Kevin Fenzi]

I'm not really sure this needs a freeze break (we haven't really
established a formal policy about if our openshift cluster is frozen or
just some applications on it, or none of it) but I thought I would run
it by everyone just in case.

I'd like to setup log forwarding on our production cluster to log all
application level logs to log01. I've already done this for staging to
test everything out and it seems to be working fine.

Now of course this means a bunch more logs on log01, but on the other
hand, it means we can actually look back and see what happened when say
a update didn't process as we expected, but the bodhi web pod has been
restarted. ;)

Thoughts?

kevin

[Stephen Smoogen]

On Mon, 27 Feb 2023 at 17:22, Kevin Fenzi  wrote:
> I'm not really sure this needs a freeze break (we haven't really
> established a formal policy about if our openshift cluster is frozen or
> just some applications on it, or none of it) but I thought I would run
> it by everyone just in case.
>
> I'd like to setup log forwarding on our production cluster to log all
> application level logs to log01. I've already done this for staging to
> test everything out and it seems to be working fine.
>
> Now of course this means a bunch more logs on log01, but on the other
> hand, it means we can actually look back and see what happened when say
> a update didn't process as we expected, but the bodhi web pod has been
> restarted. ;)
>
> Thoughts?

I think it is a good idea. The only feedback (which you have
already covered in IRC) I have is that for a high load of logs we should
make sure we are using TCP versus UDP. We are also going to need to change
our archiving amount. We currently have to compress every 10-20 days as we
run out of space on the 1.5 TB / partition. I would expect we would need to
make a daily compression of the previous days and then move every month off
to long term storage somewhere.
> kevin
> _______________________________________________
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to
> [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Aurelien Bompard]

> I'd like to setup log forwarding on our production cluster to log all
> application level logs to log01.

+1 to that, it would be very useful to the folks developing apps as
well, as we all know that no bug ever shows up when we deploy
something to production.

Thanks!

Aurélien

[Kevin Fenzi]

On Wed, Mar 01, 2023 at 05:58:57PM +0100, Aurelien Bompard wrote:
> > I'd like to setup log forwarding on our production cluster to log all
> > application level logs to log01.
>
> +1 to that, it would be very useful to the folks developing apps as
> well, as we all know that no bug ever shows up when we deploy
> something to production.

ok. This is now done.

You can find logs on log01.iad2.fedoraproject.org in
/var/log/merged/messages.log (for the current day) or
/var/log/hosts/worker*.ocp.fedoraproject.org/YYYY/MM/DD

Logs look like:

Mar  1 19:39:43 worker01.ocp.stg.iad2.fedoraproject.org fluentd:
namespace_name=bodhi, container_name=bodhi-web, pod_name=bodhi-web-80-zr4ph,
message=2023-03-01 19:39:42,722 DEBUG [MARKDOWN][ThreadPoolExecutor-0_0] Successfully loaded extension "markdown.extensions.fenced_code.FencedCodeExtension".

So you can search for the namespace, container, or specific pod.

Happy log viewing.

kevin