cloud images and journald/rsyslog

[Matthew Miller]

I'm putting this in a separate thread so it doesn't get buried in the
enthusiasm over the other one. :)

Here's our dilemma (Or trilemma?) in the Fedora Cloud SIG.

1) Double-logging is a significant waste of scarce resources

2) If we disable persistent journald (the f19 approach), we lose the
   nice features that journalctl offers. And, they really are nice.

3) If we leave out rsyslog, we lose text /var/log/messages ...
   ... which is less of a big deal from the server-room emergency-analysis 
       angle that might apply in the general case ...
   ... but it's a pretty big change from the rest of Fedora, and we'd
       prefer not to do that just for the cloud image.

So, it'd be nice to at least have rsyslogd moved from @core to @standard
(from minimal to default).

I'm also pretty well sold on the idea that journald is better on the
desktop. But that's not my thing.

On the server, I don't think I've _ever_ run a serious system where I didn't
have a custom syslog configuration (be it sysklogd, rsyslog, or syslog-ng).
I absolutely don't think that will go away. (And on those systems, generally
I don't look in /var/log/messages anymore anyway.)



-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Stephen John Smoogen]

On 15 July 2013 16:39, Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:

>  I'm putting this in a separate thread so it doesn't get
> buried in the
>  enthusiasm over the other one. :)
>
>  Here's our dilemma (Or trilemma?) in the Fedora Cloud SIG.
>
>  1) Double-logging is a significant waste of scarce resources
>
>  
Well I can see IO resources as being scarse but how scarse are we really
talking? [There comes a point where cutting down the size of an image isn't
going to help a bit because it takes up the same number of sectors on a
disk whether its X Mb or Y Mb.


>  2) If we disable persistent journald (the f19 approach), we lose the
>     nice features that journalctl offers. And, they really are nice.
>
>  3) If we leave out rsyslog, we lose text /var/log/messages ...
>     ... which is less of a big deal from the server-room emergency-analysis
>         angle that might apply in the general case ...
>     ... but it's a pretty big change from the rest of Fedora, and we'd
>         prefer not to do that just for the cloud image.
>
>  
So if I am going to spin up a lot of boxes.. I am going to want those logs
not on one box but transmitted to a central location. Does journald have
remote logging capabilities (will it?)

As it is, I don't have a problem with it not being in minimal but being in
the @standard setting.



-- 
Stephen J Smoogen.

[Matthew Miller]

On Mon, Jul 15, 2013 at 05:21:02PM -0600, Stephen John Smoogen wrote:
>  > 1) Double-logging is a significant waste of scarce resources
>  Well I can see IO resources as being scarse but how scarse are we really
>  talking? [There comes a point where cutting down the size of an image isn't
>  going to help a bit because it takes up the same number of sectors on a
>  disk whether its X Mb or Y Mb. 
I think "enough scarce that we should attempt to minimize the default system
use of them so that people can make their own decisions".

I'm also not particularly keen on just the plain space for storing the logs
themselves.


>  So if I am going to spin up a lot of boxes.. I am going to want those
> logs
>  not on one box but transmitted to a central location. Does journald have
>  remote logging capabilities (will it?) 
Right, exactly. In that case, you'd install rsyslog, but not configure it to
write a thing locally -- just have it forward everything you want forwarded.

journald doesn't have remote logging capabilities and it's my understanding
that it's a non-goal. Personally, I'd be interested in seeing a lightweight
forwarder which integrates with, say, Logstash.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Jóhann B. Guðmundsson]

On 07/15/2013 11:34 PM, Matthew Miller wrote:
>  journald doesn't have remote logging capabilities and it's my
> understanding
>  that it's a non-goal. Personally, I'd be interested in seeing a lightweight
>  forwarder which integrates with, say, Logstash. 
Well let's not forget and easily dismiss the journal gateway...

JBG

[Matthew Miller]

On Mon, Jul 15, 2013 at 11:35:48PM +0000, "Jóhann B. Guðmundsson" wrote:
>  On 07/15/2013 11:34 PM, Matthew Miller wrote:
>  >journald doesn't have remote logging capabilities and it's my understanding
>  >that it's a non-goal. Personally, I'd be interested in seeing a lightweight
>  >forwarder which integrates with, say, Logstash.
>  Well let's not forget and easily dismiss the journal gateway... 
I haven't forgotten it. It would be interesting were that to grow into
something useful. And like rsyslog, I'd prefer it not to be installed by
default. *cough* *cough* https://bugzilla.redhat.com/show_bug.cgi?id=908081




-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Miloslav Trmač]

On Tue, Jul 16, 2013 at 1:34 AM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  Personally, I'd be interested in seeing a lightweight
>  forwarder which integrates with, say, Logstash. 
More lightweight than running
rsyslog with local logging turned off?
Is the overhead of rsyslog in such a configuration really a concern?
    Mirek

[Matthew Miller]

On Wed, Jul 17, 2013 at 07:42:49PM +0200, Miloslav Trmač wrote:
>  > Personally, I'd be interested in seeing a lightweight
>  > forwarder which integrates with, say, Logstash.
>  More lightweight than running rsyslog with local logging turned off?
>  Is the overhead of rsyslog in such a configuration really a concern? 
It would be nice to retain the field delimination and extra data the journal
collects. If rsyslog can do that, then it effectively can be that service. I
think rsyslog can with imjournal, but I haven't investigated in depth.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Miloslav Trmač]

On Wed, Jul 17, 2013 at 7:52 PM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Wed, Jul 17, 2013 at 07:42:49PM +0200, Miloslav Trmač wrote:
> > > Personally, I'd be interested in seeing a lightweight
> > > forwarder which integrates with, say, Logstash.
> > More lightweight than running rsyslog with local logging turned off?
> > Is the overhead of rsyslog in such a configuration really a concern?
>
>  It would be nice to retain the field delimination and extra data the journal
>  collects. If rsyslog can do that, then it effectively can be that service. I
>  think rsyslog can with imjournal, but I haven't investigated in depth. 
rsyslog has facilities to read from journal, send the full data in
text, receive and read it back, and even write it back to journal at
the destination.  (Full disclosure I haven't actually tried such a
chain up, and I wouldn't be surprised if there were some differences
between this and just copying the journal binary logs.)
     Mirek

[Matthew Miller]

On Wed, Jul 17, 2013 at 07:54:17PM +0200, Miloslav Trmač wrote:
>  rsyslog has facilities to read from journal, send the full data in
>  text, receive and read it back, and even write it back to journal at
>  the destination.  (Full disclosure I haven't actually tried such a
>  chain up, and I wouldn't be surprised if there were some differences
>  between this and just copying the journal binary logs.) 
That would probably do it, then. In the future we could look at separating
the rsyslog package from the default logging configuration (as I a few
messages ago suggested to Bill as a possible solution for the
depends-on-/var/log/messages issue). Then, one could easily install the
rsyslog package and drop in a forwarder configuration without that
immediately turning on local logging. (Can't do this immediately without
disruption on upgrades, though).

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Kay Sievers]

On Wed, Jul 17, 2013 at 7:54 PM, Miloslav Trmač <mitr(a)volny.cz&gt; wrote:

>  rsyslog has facilities to read from journal, send the full data in
>  text, receive and read it back, and even write it back to journal at
>  the destination.  (Full disclosure I haven't actually tried such a
>  chain up, and I wouldn't be surprised if there were some differences
>  between this and just copying the journal binary logs.) 
"External" (already logged) data passed to the journal again will
record the time stamps of the copying not the ones from the original
source. Most of the data would be there, but it would be something
different.

Kay

[Lennart Poettering]

On Wed, 17.07.13 19:42, Miloslav Trmač (mitr(a)volny.cz) wrote:

>  On Tue, Jul 16, 2013 at 1:34 AM, Matthew Miller
>  <mattdm(a)fedoraproject.org&gt; wrote:
>  > Personally, I'd be interested in seeing a lightweight
>  > forwarder which integrates with, say, Logstash.
>  More lightweight than running rsyslog with local logging turned off?
>  Is the overhead of rsyslog in such a configuration really a concern? 
You could just mount the same /dev/log into all your containers.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Lennart Poettering]

On Mon, 15.07.13 17:21, Stephen John Smoogen (smooge(a)gmail.com) wrote:

>  On 15 July 2013 16:39, Matthew Miller
> <mattdm(a)fedoraproject.org&gt; wrote:
>  
>  > I'm putting this in a separate thread so it doesn't get buried in the
>  > enthusiasm over the other one. :)
>  >
>  > Here's our dilemma (Or trilemma?) in the Fedora Cloud SIG.
>  >
>  > 1) Double-logging is a significant waste of scarce resources
>  >
>  >
>  Well I can see IO resources as being scarse but how scarse are we really
>  talking? [There comes a point where cutting down the size of an image isn't
>  going to help a bit because it takes up the same number of sectors on a
>  disk whether its X Mb or Y Mb. 
Well, the duplicate log files will be accounted for every instance of a
container/VM. The more containers you run, the more often you pay for
it. This is different than just having one package installed too much in
the image, which can be shared among instances. Written log files cannot.

>  > 3) If we leave out rsyslog, we lose text /var/log/messages ...
>  >    ... which is less of a big deal from the server-room emergency-analysis
>  >        angle that might apply in the general case ...
>  >    ... but it's a pretty big change from the rest of Fedora, and we'd
>  >        prefer not to do that just for the cloud image.
>  >
>  >
>  So if I am going to spin up a lot of boxes.. I am going to want those logs
>  not on one box but transmitted to a central location. Does journald have
>  remote logging capabilities (will it?) 
Well, it currently doesn't have full featured solution.

However, it has some bits and pieces around to make it work nicely for
some usecases: if you share /var/log/journal between multiple hosts or
containers, then you can use "journalctl -m" to view them all at the
same time and nicely interleaved. 

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Chris Adams]

Once upon a time, Lennart Poettering <mzerqung(a)0pointer.de&gt; said:
>  Well, the duplicate log files will be accounted for every instance of
> a
>  container/VM. The more containers you run, the more often you pay for
>  it. This is different than just having one package installed too much in
>  the image, which can be shared among instances. Written log files cannot. 
If it isn't too much of a thread-drift - this brings to mind another
question: how are the journal files rotated, archived, etc.?  I don't
see anything in the man page.
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jul 15, 2013 at 07:27:57PM -0500, Chris Adams wrote:
>  Once upon a time, Lennart Poettering <mzerqung(a)0pointer.de&gt;
> said:
>  > Well, the duplicate log files will be accounted for every instance of a
>  > container/VM. The more containers you run, the more often you pay for
>  > it. This is different than just having one package installed too much in
>  > the image, which can be shared among instances. Written log files cannot.
>  
>  If it isn't too much of a thread-drift - this brings to mind another
>  question: how are the journal files rotated, archived, etc.?  I don't
>  see anything in the man page. 
The total space taken by journal is limited by size
and percantage of
free space on the /var/log/journal filesystem [1], and age
[2]. Individual journal files are limited by size and age [3,4].

[1] http://www.freedesktop.org/software/systemd/man/journald.conf.html#System...
[2] http://www.freedesktop.org/software/systemd/man/journald.conf.html#MaxRet...
[3] http://www.freedesktop.org/software/systemd/man/journald.conf.html#MaxFil...
[4] http://www.freedesktop.org/software/systemd/man/journald.conf.html#System...
(SystemMaxFileSize setting).

Zbyszek

-- 
they are not broken. they are refucktored
                           -- alxchk

[Chris Adams]

Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  The total space taken by journal is limited by size and percantage
> of
>  free space on the /var/log/journal filesystem [1], and age
>  [2]. Individual journal files are limited by size and age [3,4]. 
Thanks, I see there's a journald.conf now.  Another thing that would be
nice to add to the man page.  Now that I know about it, I see the SEE
ALSO, but most things have a FILES section for that type stuff.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jul 15, 2013 at 08:46:55PM -0500, Chris Adams wrote:
>  Once upon a time, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; said:
>  > The total space taken by journal is limited by size and percantage of
>  > free space on the /var/log/journal filesystem [1], and age
>  > [2]. Individual journal files are limited by size and age [3,4].
>  
>  Thanks, I see there's a journald.conf now.  Another thing that would be
>  nice to add to the man page.  Now that I know about it, I see the SEE
>  ALSO, but most things have a FILES section for that type stuff. 
Yes, people
occasionally complain that systemd has too much documentation.
That's why there's systemd.index(7) and systemd.directives(7) to make
it easier to search for pages and settings.

systemd-jouranld.service(8) now has a FILES section.

Zbyszek
-- 
they are not broken. they are refucktored
                           -- alxchk

[Matthew Miller]

On Mon, Jul 15, 2013 at 07:27:57PM -0500, Chris Adams wrote:
>  If it isn't too much of a thread-drift - this brings to mind
> another
>  question: how are the journal files rotated, archived, etc.?  I don't
>  see anything in the man page. 
You can also use time-based rotation policies, but it's not very granular.
If you're in an environment where data retention and lifecycle policies are
important, you should install rsyslog or syslog-ng.


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Matthias Clasen]

On Mon, 2013-07-15 at 18:39 -0400, Matthew Miller wrote:

>  I'm also pretty well sold on the idea that journald is better on
> the
>  desktop. But that's not my thing. 
I agree. I would also like to stop the double-logging on the desktop.
We're working on a graphical journal frontend now, and we will make use
of the indexing and matching capabilities of the journal there.

Matthias