rsyslogd suddenly went crazy (120% cpu)

[Clemens Eisserer]

Hi,

I am running Fedora 19 + updates-testing and suddenly rsyslogd went crazy,
and is now consuming constantly 120% cpu (and is also constantly expanding
allocated memory).
I first thought it could have something to do with updates I've applied
recently, so I rolled back to an older btrfs-snapshot - which however
didn't improve the situation.

I've recorded some stack-traces: http://pastebin.com/ikwC3h4g

When executing "tail -f /var/log/messages" I get the same messages from 16.
and 17. May over and over again.

Any ideas what could cause these problems?

Thank you in advance, Clemens

[Reindl Harald]

Am 14.06.2013 23:10, schrieb Clemens Eisserer:
> I am running Fedora 19 + updates-testing and suddenly rsyslogd went crazy, and is now consuming constantly 120% cpu
> (and is also constantly expanding allocated memory).
> I first thought it could have something to do with updates I've applied recently, so I rolled back to an older
> btrfs-snapshot - which however didn't improve the situation.
>
> I've recorded some stack-traces: http://pastebin.com/ikwC3h4g
>
> When executing "tail -f /var/log/messages" I get the same messages from 16. and 17. May over and over again.
>
> Any ideas what could cause these problems?

wrong mailing-list because F19 is *not* released now and
so your question belongs to the testing-list, but however

-------- Original-Nachricht --------
Betreff: Re: rsyslog-7.4.0-1 is bad
Datum: Fri, 14 Jun 2013 00:24:24 -0700
Von: Adam Williamson 
Antwort an: Development discussions related to Fedora 
Organisation: Red Hat
An: For testing and quality assurance of Fedora releases 
Kopie (CC): [e-mail]

On Thu, 2013-06-13 at 23:48 -0700, Adam Williamson wrote:
> Just a heads up for F19 / F20 users - rsyslog-7.4.0-1 is a bad update,
> many users (inc. me on two systems) are seeing it
> spamming /var/log/messages with old logs and sucking up 100% CPU time.
> You might not notice immediately if you're not running something that
> spins its fans up. I recommend downgrading to 7.2.6-1 ASAP. Bug is
> https://bugzilla.redhat.com/show_bug.cgi?id=974335 .

Better bug link:

https://bugzilla.redhat.com/show_bug.cgi?id=974132

Turns out this is ultimately a journald bug (and explains why journalctl
has been busted for me lately). But still, if you're hit by it, you're
going to want to nerf rsyslogd ASAP, because until you do, it will
spam /var/log/messages until your disk is full. Mine had reached 8GB by
the time I noticed. Some other poor bugger's got to 30GB.

[Clemens Eisserer]

Hi Harald,

Better bug link:
> https://bugzilla.redhat.com/show_bug.cgi?id=974132
>
> Turns out this is ultimately a journald bug (and explains why journalctl
> has been busted for me lately). But still, if you're hit by it, you're
> going to want to nerf rsyslogd ASAP, because until you do, it will
> spam /var/log/messages until your disk is full. Mine had reached 8GB by
> the time I noticed. Some other poor bugger's got to 30GB

Thanks for the hint & pointer - by deleting the journal I was able to solve
the effects to my system.

Thanks, Clemens

[Kevin Fenzi]

On Fri, 14 Jun 2013 23:10:08 +0200
Clemens Eisserer  wrote:
> Hi,
>
> I am running Fedora 19 + updates-testing and suddenly rsyslogd went
> crazy, and is now consuming constantly 120% cpu (and is also
> constantly expanding allocated memory).
> I first thought it could have something to do with updates I've
> applied recently, so I rolled back to an older btrfs-snapshot - which
> however didn't improve the situation.
>
> I've recorded some stack-traces: http://pastebin.com/ikwC3h4g
>
> When executing "tail -f /var/log/messages" I get the same messages
> from 16. and 17. May over and over again.
>
> Any ideas what could cause these problems?
>
> Thank you in advance, Clemens

http://www.happyassassin.net/2013/06/14/fedora-1920-logfile-explosions/

kevin

[Sam Varshavchik]

Kevin Fenzi writes:
> http://www.happyassassin.net/2013/06/14/fedora-1920-logfile-explosions/

Did I read this right? /var/log/messages have been replaced by binary goo?

[Kevin Fenzi]

On Fri, 14 Jun 2013 18:51:53 -0400
Sam Varshavchik  wrote:
> Kevin Fenzi writes:
> > http://www.happyassassin.net/2013/06/14/fedora-1920-logfile-explosions/
>
> Did I read this right? /var/log/messages have been replaced by binary
> goo?

No, you didn't read it right.

rsyslog now has the ability to pull from journald and log (as it always
has) in text to /var/log/messages. Nothing changes aside from more
information in /var/log/messages from journald.

kevin

[Tom Horsley]

On Fri, 14 Jun 2013 18:51:53 -0400
Sam Varshavchik wrote:
> Did I read this right? /var/log/messages have been replaced by binary goo?

Nah, just the systemd crap which no one wants to read anyway
(I think). I certainly still have a /var/log/messages with lots
of junk in it (in fact, lots of systemd junk as well).

P.S. A good trick here is to add this line to /etc/yum.conf

exclude=rsyslog-7.4.0-1.fc19.x86_64

That way, yum won't wind up reinstalling the crazy version
of rsyslog.