lumberjack-developers March 2012

lumberjack-developers@lists.fedorahosted.org

16 participants
49 discussions

quoting of mail...
by J.O. Aho 08 Mar '12

08 Mar '12

Hi All, I have just joined this maillist, not that I think I have much to contribute, but more to keep myself up to date on this interesting project. I have run into a big issue while reading mails, some of you on this list don't make a proper quoting of the previous mail, which makes it really difficult to know who wrote what when you use pine. So for those of you who do favour the microsoft life, please could you see to have something that shows you are quoting the previous mail with something like "> " as the first characters of a quote row. Colors and other HTML don't work that well for all us oldies who like a plain text mail. Thanks in advance for your help to make it a bit easier for me. -- //Aho ------------------------------------------------------------------------ E-Mail: trizt(a)iname.com URL: http://www.kotiaho.net/~trizt/ ICQ: 13696780 Skype: jo_aho System: Linux System (PPC7447/1000 AMD K8/2500) ------------------------------------------------------------------------ EU forbids you to send spam without my permission ------------------------------------------------------------------------

2 2

support in nxlog
by Botond Botyanszki 08 Mar '12

08 Mar '12

Hi, I'm the leader behind the nxlog project and I'm glad to see this lumberjack project gaining some momentum. Some of you may or may not be familiar with nxlog as it is a relatively new player on the field. See http://nxlog.org if you fall into the latter category. There was already a fair amount of protocol/format support in nxlog, including Syslog (legacy and RFC5424-5426), CSV, Windows eventlog etc. Now I'm happy to announce that JSON and XML format support has been added to nxlog as of version 1.4.615. This makes it possible to send logs in JSON and XML format over various protocols and this can be even encapsulated into syslog as defined by the CEE spec. Nxlog can not only generate but it can also parse these formats. Probably nxlog is the first open source project to support both JSON and XML on several platforms (including but not limited to RedHat) and we are happy to see others follow. Here is an example of a syslog encapsulated JSON event coming from Windows EventLog: <14>Mar 8 14:40:11 WIN-OUNNPISDHIG Service_Control_Manager: {"EventTime":"2012-03-08 14:40:11","EventTimeWritten":"2012-03-08 14:40:11","Hostname":"WIN-OUNNPISDHIG","EventType":"INFO","SeverityValue":2,"Severity":"INFO", "SourceName":"Service Control Manager","FileName":"System","EventID":7036, "CategoryNumber":0, "RecordNumber":6788, "Message":"The nxlog service entered the running state.","EventReceivedTime":"2012-03-08 14:40:12"} See the relevant docs at the following url for more examples: http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html#xm_json http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html#xm_xml I should note that the full CEE spec is not supported, so nxlog is still at the stage what William recommended: "Just focus on just getting things into structured content, name-value pairs. Whatever internal naming structure you use is great." Making nxlog output CEE conformant logs can be probably achieved with some nxlog.conf tweaks by renaming the required fields and it would be probably not that difficult to add native support, but we are reluctant to develop CEE format support this point. See my next email for the reasons. Regards, Botond

2 4

Alpha support in rsyslog? Syslog-NG?
by William Heinbockel 08 Mar '12

08 Mar '12

I have some time and resources to dedicate to the testing and development of the lumberjack libraries and tooling. What I would like to do is get a Fedora box going with the auditd, rsyslog, and Syslog-NG using the structured json output. I will draft tutorials for these and get them posted to the lumberjack wiki. >From there, we can work with Dmitri and Keith to develop the API, log4j/log4c/etc. bindings, and supporting XML Schemas. Rainer: I will most likely need your assistance to get the latest rsyslog source running properly. Is there anything special I need to do to configure the cee/json support for rsyslog? Right now, I'm most interested in structured json output. But will soon need the input processing side as well. Peter, Balaks: What is the status of cee/json-support in the latest Syslog-NG codebase? Thanks

6 8

Please say no to CEE.
by Jordan Sissel 08 Mar '12

08 Mar '12

Howdy! So, I read over CEE, and frankly it's a pretty horrible thing. It's quite poorly written and designed; the goal of CEE seems anything other than improve the state and practices of logging. I have a rough summary of the problems here https://gist.github.com/1983121 I'm not trying to derail any progress on improving event tooling. In fact, I am quite aligned with such improvements (see my many log-related projects: logstash, grok, ruby-cabin, etc). The summary link above includes recommendations for where to go next, specifically choosing *one* serialization and *one* transportation implementation and additionally resolving the wild inconsistencies in CEE's field naming. I've seen mentions of JSON on this list, awesome, keep doing that, but pick a schema that suits the project's needs and uses consistent style while also not being overly verbose (example of overly verbose: CEE's 'custom profile' stuff) -Jordan

6 12

Re: [lumberjack] Alpha support in rsyslog? Syslog-NG?
by Rainer Gerhards 07 Mar '12

07 Mar '12

As i said in brno, i has held some implementation effort. However, lots has been done since then :-). You need the current git master branch, probably i'll do a release of it tomorrow. But it would make sense if you work with the git version, as this eases the task when i change things based on your experuence. I will gladly help get you going, i am super-interested in your testing. Rainer William Heinbockel <wheinbockel(a)gmail.com> hat geschrieben:I have some time and resources to dedicate to the testing and development of the lumberjack libraries and tooling. What I would like to do is get a Fedora box going with the auditd, rsyslog, and Syslog-NG using the structured json output. I will draft tutorials for these and get them posted to the lumberjack wiki. >From there, we can work with Dmitri and Keith to develop the API, log4j/log4c/etc. bindings, and supporting XML Schemas. Rainer: I will most likely need your assistance to get the latest rsyslog source running properly. Is there anything special I need to do to configure the cee/json support for rsyslog? Right now, I'm most interested in structured json output. But will soon need the input processing side as well. Peter, Balaks: What is the status of cee/json-support in the latest Syslog-NG codebase? Thanks _______________________________________________ lumberjack-developers mailing list lumberjack-developers(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers

1 0

Please say no to CEE
by kroberts＠redhat.com 07 Mar '12

07 Mar '12

In response to... [snip] > 3) XSD schema, IMO is just a way to describe a structure. Correct, I used XSD to define the structure of lumberjack because... - it is strongly typed - it is relatively easy to define hierachical relationships - it enforces structure. > As a person who actually closely involved with API definition and proposed internals I still have some concerns about how the profile expressed in XSD would be actually used to construct or validate the messages. I would envision a consumer of lumberjack XML events [1] using the schema to generate bindings (e.g. JAXB, Python (GenerateDS, PyXB), etc.). With these bindings the consumer can simply feed the stream of events into their parser and get objects in return. The Eclipse project in the GIT repo demonstrates this behavior. For JSON style events [2] there are various parsers available, and I'll try to post some code that demonstrates this behavior later. > I agree that XML processing is too heavy. Maybe maybe not. IBM uses the TMS standard [3] which can produce XML output for each event. It is quite readable and *very* easy to parse. In any case, I think that the lumberjack standard should facilitate various forms of output XML, JSON, and plain text/CSV. In my opinion, the logging libraries bear the burden of making it easy to select a particular format. > IMO it would be application developers responsibility to actually follow the profile specification and construct the structured logs according to specific. See [4] for an example of how an application would extend the basic lumberjack profile with something specific. [1] https://fedorahosted.org/lumberjack/wiki/sampleXml#ExamplelumberjackLogEven… [2] https://fedorahosted.org/lumberjack/wiki/sampleJson#ExamplelumberjackLogEve… [3] http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=%2Fc… [4] https://fedorahosted.org/lumberjack/wiki/sampleProfile#ExamplelumberjackPro… Cheers, Keith Robertson

1 0

[lumberjack 12] Test message
by kroberts＠redhat.com 07 Mar '12

07 Mar '12

This is a test message from the administrator. I am testing to see if the lists prepents [lumberjack] to the subject line. Cheers, Keith Robertson

1 0

cee-enhanced syslog presentation
by Rainer Gerhards 02 Mar '12

02 Mar '12

Hi all, if someone is interested: I have created a small online presentation on cee-enhanced syslog in the hope that this helps spread the idea. Far from being fancy, but hopefully useful. It can be found here: http://youtu.be/C_mVwQM_oS0 Comments, feedback, corrections happily accepted. If you like it, please help spreading. I am planning to do some more 3 to 5 minute presentations, so feedback at this stage is very useful. Thanks, Rainer

1 0

Windows Event Log & cee enhanced syslog
by Rainer Gerhards 01 Mar '12

01 Mar '12

Hi all, we have just completed a first implementation of cee-enhanced syslog for our Windows products: http://blog.gerhards.net/2012/03/cee-enhanced-event-log-to-syslog.html I will make sure that this works well together with rsyslog. The important thing I wanted to share is that the "lumberjack momentum" now has reached the Windows world (well, via us as proxy, currently ;)). Rainer

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

lumberjack-developers March 2012