lumberjack cookie

[Rainer Gerhards]

Hi all,

at least some of us seem to share a concern that CEE will depart from what it was early this year into a big monster effort that needs complete software overhaul. That would obviously be quite incompatible with what we are trying to do.

With the lumberjack-enabled software not yet rolled out widely, I think we have a final time to make a decision on the cookie. Currently we use "@cee" to detect lumberjack-enabled message content. With the potential fork of the efforts, I think it would probably be wiser to change that to something like "@lj:" (or similar).

This change would hurt rsyslog (and our Windows products) a little, but I get the impression that this may be a wise step in the long term.

Comments (especially from other implementers)?

Rainer

[William Heinbockel]

Our focus needs to be implementation. I was hoping the CEE was a bit
closer to being finalized, but it seems like there is still some
unanswered questions regarding the spec.

For lumberjack implementatons, we have the ceelog (rename?),
libumberlog, rsyslog and syslogNG.

I agree with Rainer that we need to step carefully and probably want to
view lumberjack as a fork of CEE, with the goal of feeding back issues
to the CEE project and hopefully a merger of the two in the not too
distant future.

On Tue, 2012-10-09 at 12:42 +0000, Rainer Gerhards wrote:
> Hi all,
>
> at least some of us seem to share a concern that CEE will depart from what it was early this year into a big monster effort that needs complete software overhaul. That would obviously be quite incompatible with what we are trying to do.
>
> With the lumberjack-enabled software not yet rolled out widely, I think we have a final time to make a decision on the cookie. Currently we use "@cee" to detect lumberjack-enabled message content. With the potential fork of the efforts, I think it would probably be wiser to change that to something like "@lj:" (or similar).
>
> This change would hurt rsyslog (and our Windows products) a little, but I get the impression that this may be a wise step in the long term.
>
> Comments (especially from other implementers)?
>
> Rainer
> _______________________________________________
> lumberjack-developers mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/lumberjack-developers

[Botond Botyanszki]

So far every CEE specification version was significantly different from
the previous. Now the number 1.0 creates an impression that it is stable
and can be used. In reality it has never really been tried in practice and
there isn't even a proper reference implementation available. The CEE
discussions indicate that this will evolve a lot more.
Now that "lumberjack is a fork of CEE" this makes it even more
questionable from the point of view of users and implementers.
Personally I still don't understand why this is necessary instead of
having a single effort which would look more serious.
People will just keep shipping plain JSON without any further standard
enforced , or create there own. They can get more productive by ignoring
all of cee/lj instead of trying to follow whether to use @cee, @lj, or
@WTF. Same with fields and other design decisions which keep changing
every other day.
For this reason we decided it was better to stop at being able to
parse/produce json/xml (plain or syslog encapsulated) in nxlog and
not push it further until a consensus is made about the rest of the
format.

Regards,
Botond

On Tue, 09 Oct 2012 08:55:00 -0400
William Heinbockel  wrote:
> Our focus needs to be implementation. I was hoping the CEE was a bit
> closer to being finalized, but it seems like there is still some
> unanswered questions regarding the spec.
>
> For lumberjack implementatons, we have the ceelog (rename?),
> libumberlog, rsyslog and syslogNG.
>
> I agree with Rainer that we need to step carefully and probably want to
> view lumberjack as a fork of CEE, with the goal of feeding back issues
> to the CEE project and hopefully a merger of the two in the not too
> distant future.
>
> On Tue, 2012-10-09 at 12:42 +0000, Rainer Gerhards wrote:
> > Hi all,
> >
> > at least some of us seem to share a concern that CEE will depart from what it was early this year into a big monster effort that needs complete software overhaul. That would obviously be quite incompatible with what we are trying to do.
> >
> > With the lumberjack-enabled software not yet rolled out widely, I think we have a final time to make a decision on the cookie. Currently we use "@cee" to detect lumberjack-enabled message content. With the potential fork of the efforts, I think it would probably be wiser to change that to something like "@lj:" (or similar).
> >
> > This change would hurt rsyslog (and our Windows products) a little, but I get the impression that this may be a wise step in the long term.
> >
> > Comments (especially from other implementers)?
> >
> > Rainer
> > _______________________________________________
> > lumberjack-developers mailing list
> > [e-mail]
> > https://lists.fedorahosted.org/mailman/listinfo/lumberjack-developers
>
> lumberjack-developers mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/lumberjack-developers

[Gergely Nagy]

Botond Botyanszki  writes:
> People will just keep shipping plain JSON without any further standard
> enforced , or create there own. They can get more productive by ignoring
> all of cee/lj instead of trying to follow whether to use @cee, @lj, or
> @WTF.

That's what I did too.

With my application developer hat on, both CEE and LJ would've made my
life far more difficult than it needs be, so I stopped caring about
both.

(I did forget to unsubscribe from this list, though, that'll be
corrected shortly, please CC me on replies if you want me to see it.)

[David Lang]

On Tue, 9 Oct 2012, Gergely Nagy wrote:
> Botond Botyanszki  writes:
> > People will just keep shipping plain JSON without any further standard
> > enforced , or create there own. They can get more productive by ignoring
> > all of cee/lj instead of trying to follow whether to use @cee, @lj, or
> > @WTF.
>
> That's what I did too.
>
> With my application developer hat on, both CEE and LJ would've made my
> life far more difficult than it needs be, so I stopped caring about
> both.

I think there are two different issues here that are getting mixed up.

1. how to transport (and filter) structured logs

2. standardizing the format of such logs so that programs that consume the 
logs no longer have to have a different parser for each application.

I understand that Gergely and Botond are focused on #1, and with that, 
just accepting JSON and XML and being able to process them is 'good 
enough'

I think that #2 is a bit of a pipe dream, and I'm not sure it's even 
desirable to try and specify things completely enough for the approach to 
work.

But I do think there is room for a little bit of standardization.

Syslog has four standard fields, pri, timestamp, hostname, programname. 
Even that little bit of standardization has enabled a huge amount of 
compatibility between programs. I think we need to be careful about trying 
to extend this. But that being said, there is a lot of interest in being 
able to gather the SCM_CREDENTIALS and have those in the logs. Having this 
information in the logs in a way that makes it impossible for the consumer 
of the logs to tell if the data was created by a rogue application or was 
gathered by the logging program defeats the purpose of this data, so I 
think it's worth extending the standardization just a little bit to 
reserve a prefix for this sort of data and name these fields.

going much beyond that, I expect the standardization to either break down, 
be ignored, or go the cee route where they spend so much time trying to 
standardize things that it doesn't matter.

David Lang