lumberjack-developers March 2012

lumberjack-developers@lists.fedorahosted.org

16 participants
49 discussions

Re: [lumberjack] syslog-like API for structured messages
by Rainer Gerhards 19 Mar '12

19 Mar '12

IMHO the context of this thread is plain c. Rainer ps: sry, no decent mailer at hand Keith Robertson <kroberts(a)redhat.com> hat geschrieben:On 03/19/2012 03:54 PM, david(a)lang.hm wrote: > On Mon, 19 Mar 2012, Keith Robertson wrote: > >> On 03/19/2012 03:43 PM, david(a)lang.hm wrote: >>> On Mon, 19 Mar 2012, Keith Robertson wrote: >>> >>> If the data is trivially available to the code, then yes, it should >>> be included. If it takes doing system calls to find it out, then you >>> need to think a bit harder about the value of the data and if you >>> want to pay that system call overhead for every log message. >>> >>> If the data is mandatory (like a timestamp), you have to pay the >>> overhead, but things that are useful, but not mandatory (UID, GID, >>> TID, etc) should only be included all the time if they are trivial >>> to get. If any of them need a syscall to find out, then it either >>> needs to be a configurable item, or it needs to be something that >>> can be done once, at the initialization of the syslog connection, >>> and cached for the life of the program (which UID, GID, TID cannot >>> be, since they may change over time) > >> I agree here also, and I'd envision that this could be controlled >> through a property/conf file. > > please no. > > Either have this data be added by the syslog daemon (which would use > it's config file), or have it be part of the initialization of the > syslog connection call openlog() You aren't considering the python, ruby, Java use cases where syslog may/may not be involved. > > This may need to be tuned on a per-application level, so you aren't > just talking about one simple config file, you are talking about one > per app, or one complex file that can have per-app settings. > > where possible, having this data added by the syslog daemon that the > syslog() call is writing to would be the best bet. > > David Lang _______________________________________________ lumberjack-developers mailing list lumberjack-developers(a)lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers

1 0

Libcollection and JSON
by Dmitri Pal 19 Mar '12

19 Mar '12

Hello, Yesterday I spend quite some time making the collection be able to serialize data into JSON. Well... though I managed to do it and it was not that hard I realized that there are some ambiguities that need to be discussed. So we were saying that structured event would be represented by JSON and we also mentioned that though we want to provide a way to create hierarchical events we would lean towards flat events in general and discourage but not prohibit the use of the hierarchical events . So the general assumption is still that an event is a set of potentially hierarchical KVPs. I was playing with libcollection as the container for this set. I wanted to make sure the serialization works correctly in the general case. A month ago I did a quick prototype and it worked fine for the flat collections so now I focused on hierarchical collections and proper escaping. So here are the questions: Does the collection i.e. the event have a name? Let us not go into the detail what acts as a name it can be event namespace or event id or something else. Let us just ask this first question. And then ask how it is expressed i.e. it is just another attribute or it is an object name. Let me illustrate what I mean. Let us say we have: event with name "foo" "key1" with value "bar" subcollection with name "baz" key1 in baz with value "xyz" key2 in baz with value "abc" No let us see how this can be expressed with JSON Option 1: (name is the top level) { "foo" : { "key1" : "bar", "baz" : { "key1" : "xyz", "key2" : "abc" }} Option2: (name is the top level attribute but not on sub - "name" is a predefined literal in this case. We can use other literal - does not matter this is just for illustration of the structure ) { "name : "foo", "key1" : "bar", "baz" : { "key1" : "xyz", "key2" : "abc" }} Option3: (name is the always an attribute, "name" and "subset" are predefined literals but we can use different literals. ) { "name : "foo", "key1" : "bar", "subset" : { "name : "baz", "key1" : "xyz", "key2" : "abc" }} Option4: (name is ignored and not used) { "key1" : "bar", "baz" : { "key1" : "xyz", "key2" : "abc" }} Option5: (name is ignored on top level but subsets are clearly defined) { "key1" : "bar", "subset" : { "name : "baz", "key1" : "xyz", "key2" : "abc" }} Option6: (name is used on top level but subsets are flattened) { "name" : "foo", "key1" : "bar", "baz.key1" : "xyz", "baz.key2" : "abc" }} Option7: (name is not used and subsets are flattened) { "key1" : "bar", "baz.key1" : "xyz", "baz.key2" : "abc" } There are probably some other options but I hope you see the point. I am leaning to support two formats: This for generic JSON serialization for the libcollection as a library: { "foo" : { "key1" : "bar", "baz" : { "key1" : "xyz", "key2" : "abc" }} This one for use with the logging API (ELAPI) which is a wrapper around collection: { "key1" : "bar", "baz.key1" : "xyz", "baz.key2" : "abc" } based on the assumption that events do not have names. If they do, I would prefer the first option. Thoughts? -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/

4 6

Teleconference problems
by kroberts＠redhat.com 19 Mar '12

19 Mar '12

All, I seem to be having some difficulties with the teleconference line. If I cannot get it resolved in the next 5 minutes I'll reschedule for another time this week. I'll send a note to the list with an update in a minute. Keith

3 2

Try this conference code:
by kroberts＠redhat.com 19 Mar '12

19 Mar '12

2216377974

2 1

Monday Teleconference Information
by kroberts＠redhat.com 19 Mar '12

19 Mar '12

All, Please find the information for the teleconference attached. When: 8:30 A.M. ET which is 1:30 P.M. Berlin time (for reference) Conference Access Code: 504085 Call-in lines are below: Toll-free dial-in number (U.S. and Canada): (800) 451-8679 Global Network Access Toll Dial-in Numbers Australia, Adelaide : 61870020130 Australia, Brisbane : 61730870178 Australia, Melbourne : 0382561740 Australia, Perth : 61861884572 Australia, Sydney : 0289852326 Austria, Vienna : 012534978196 Belgium, Brussels : 027920405 China, All Cities Domestic : 8008190132 China, All Cities Domestic : 4006205013 Czech Republic, Prague : 239014984 Denmark, Copenhagen : 32729215 Finland, Helsinki : 0923194436 France, Paris : 0170377140 Germany, Berlin : 030300190579 Germany, Frankfurt : 06922222594 Hong Kong, Hong Kong : 85230730429 Hungary, Budapest : 7789030 India, Bangalore : 08039417180 India, Chennai : 4430061276 India, Hyderabad : 4030644055 India, Mumbai : 02230985358 India, New Delhi : 01139417180 Ireland, Dublin : 014367793 Italy, Milan : 0236269529 Japan, Tokyo : 0345807897 Korea (South), Seoul : 0234837408 Lithuania, Vilnius : 52054226 Luxembourg, Luxembourg : 24871157 Malaysia, Kuala Lumpur : 0348190012 Netherlands, Amsterdam : 0207975872 Norway, Oslo : 21033188 Poland, Warsaw : 222120148 Romania, Bucharest : 0318103711 Russian Federation, Moscow : 4999221989 Singapore, All Cities : 64840858 Singapore, All Cities : 64840858 Slovak Republic, Bratislava : 0233456338 Slovenia, Ljubljana : 016003991 Spain, Barcelona : 935452328 Spain, Madrid : 914146284 Sweden, Stockholm : 0850513770 Switzerland, Geneva : 0225927881 Switzerland, Zurich : 0445803463 United Kingdom, All Cities : 02078970515 United Kingdom, LocalCall : 08445790678 United States, All Cities : 2127295016 Vietnam, Ho Chi Minh : 84838012421 Global Network Access Toll-Free Dial-in Numbers Argentina : 08004441016 Australia : 1800337169 Austria : 0800005898 Bahamas : 18002054778 Bahrain : 80004377 Barbados : 18668556594 Belarus : 882000110160 Belgium : 080048325 Bolivia : 800100768 Brazil : 08008921002 Bulgaria : 008001100236 Chile : 800370228 Colombia : 018005182186 Costa Rica : 08000131048 Croatia (Hrvatska) : 0800222320 Cyprus : 80095297 Czech Republic : 800701035 Denmark : 80887114 Dominican Republic : 18007519076 Ecuador : 1800020545 Egypt : 08000000188 El Salvador : 8006699 Estonia : 8000100232 Fiji : 008002539 Finland : 0800117116 France : 0805632867 Germany : 08006647541 Greece : 00800127562 Guam : 18773010136 Hong Kong : 800930349 Hungary : 0680014726 Iceland : 8008967 India : 180030104350 Indonesia, PT Indosat only : 0018030179162 Indonesia, PT Telkom only : 0078030179162 Ireland : 1800932401 Israel : 1809462557 Italy : 800985897 Jamaica : 18002050328 Japan : 00531250120 Japan : 0120934453 Kazakhstan : 88003337376 Korea (South) : 007986517393 Latvia : 80003339 Lithuania : 880031223 Luxembourg : 80026595 Malaysia : 1800814451 Malta : 80062176 Mexico : 018009269658 Monaco : 80093642 Netherlands : 08000222329 New Zealand : 0800888167 Nicaragua : 0018002202067 Norway : 80013504 Panama : 0018002043574 Peru : 080052972 Philippines : 180011100991 Poland : 008001210187 Portugal : 800814625 Romania : 0800895537 Russian Federation : 81080028341012 Saint Kitts and Nevis : 18002059252 Saudi Arabia : 8008445917 Singapore : 8006162235 Slovak Republic : 0800001441 Slovenia : 080080471 South Africa : 0800982957 Spain : 800300524 Sweden : 0200896860 Switzerland : 0800650077 Taiwan : 00801127141 Thailand : 001800656966 Trinidad and Tobago : 18002024615 Turkey : 0080044632093 Turks and Caicos Islands : 18772780472 Ukraine : 0800500152 United Arab Emirates : 8000440163 United Kingdom : 08006948057 United States : 8004518679 Uruguay : 00040190315 Venezuela : 8001627182 Vietnam : 12011346 Virgin Islands (U.S.) : 8773007428**

3 2

Example structured logs
by William Heinbockel 19 Mar '12

19 Mar '12

Modified the audisp plugin to provide compatible output to the proposal from earlier. This is not an official interface, API, audisp plugin, etc. It is only to show a set of logs that have been converted into the CEE Structured Text format. I have attached the plugin code for those of you who are interested. Right now it uses json-c and just prints to stdout. All of the "auditd" field names are now members of the auditd namespace. A "namespace.name" field naming convention allows for easy translation between JSON, XML, and other formats. The logs are prepended with either a legacy or RFC5424 Syslog header. 1 2012-03-16T20:37:04 localhost.localdomain cee-plugin 9011 - - @cee:{"time":"2012-03-09T14:01:01.347-05:00","serial":"188","id":"1105","p_host":[],"p_app":"auditd","file":"stdin","line":"4806","auditd.type":"USER_START","auditd.pid":"0","auditd.uid":["0","root"],"auditd.auid":["0","root"],"auditd.ses":"20","auditd.subj":"system_u:system_r:crond_t:s0-s0:c0.c1023","auditd.op":"PAM:session_open","auditd.acct":"root","auditd.exe":"/usr/sbin/crond","auditd.hostname":[],"auditd.addr":[],"auditd.terminal":"cron","auditd.res":"success"} Mar 16 20:37:04 localhost.localdomain cee-plugin[9011]: @cee:{"time":"2012-03-09T14:01:01.371-05:00","serial":"191","id":"1106","p_host":[],"p_app":"auditd","file":"stdin","line":"4809","auditd.type":"USER_END","auditd.pid":"0","auditd.uid":["0","root"],"auditd.auid":["0","root"],"auditd.ses":"20","auditd.subj":"system_u:system_r:crond_t:s0-s0:c0.c1023","auditd.op":"PAM:session_close","auditd.acct":"root","auditd.exe":"/usr/sbin/crond","auditd.hostname":[],"auditd.addr":[],"auditd.terminal":"cron","auditd.res":"success"} 1 2012-03-16T20:37:04 localhost.localdomain cee-plugin 9011 - - @cee:{"time":"2012-03-09T14:56:32.359-05:00","serial":"192","id":"1100","p_host":[],"p_app":"auditd","file":"stdin","line":"4810","auditd.type":"USER_AUTH","auditd.pid":"0","auditd.uid":["0","root"],"auditd.auid":["1000","bockel"],"auditd.ses":"1","auditd.subj":"unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023","auditd.op":"PAM:authentication","auditd.acct":"root","auditd.exe":"/bin/su","auditd.hostname":[],"auditd.addr":[],"auditd.terminal":"pts/1","auditd.res":"success"} Mar 16 20:37:04 localhost.localdomain cee-plugin[9011]: @cee:{"time":"2012-03-09T14:56:32.445-05:00","serial":"195","id":"1105","p_host":[],"p_app":"auditd","file":"stdin","line":"4813","auditd.type":"USER_START","auditd.pid":"0","auditd.uid":["0","root"],"auditd.auid":["1000","bockel"],"auditd.ses":"1","auditd.subj":"unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023","auditd.op":"PAM:session_open","auditd.acct":"root","auditd.exe":"/bin/su","auditd.hostname":[],"auditd.addr":[],"auditd.terminal":"pts/1","auditd.res":"success"} 1 2012-03-16T20:37:04 localhost.localdomain cee-plugin 9011 - - @cee:{"time":"2012-03-09T15:01:01.400-05:00","serial":"198","id":"1006","p_host":[],"p_app":"auditd","file":"stdin","line":"4816","auditd.type":"LOGIN","auditd.pid":"13370","auditd.uid":["0","root"],"auditd.auid_old":["4294967295","unset"],"auditd.auid":["0","root"],"auditd.ses_old":["4294967295","unset"],"auditd.ses":"21"} Mar 16 20:37:04 localhost.localdomain cee-plugin[9011]: @cee:{"time":"2012-03-09T15:01:01.411-05:00","serial":"199","id":"1105","p_host":[],"p_app":"auditd","file":"stdin","line":"4817","auditd.type":"USER_START","auditd.pid":"0","auditd.uid":["0","root"],"auditd.auid":["0","root"],"auditd.ses":"21","auditd.subj":"system_u:system_r:crond_t:s0-s0:c0.c1023","auditd.op":"PAM:session_open","auditd.acct":"root","auditd.exe":"/usr/sbin/crond","auditd.hostname":[],"auditd.addr":[],"auditd.terminal":"cron","auditd.res":"success"} 1 2012-03-16T20:37:04 localhost.localdomain cee-plugin 9011 - - @cee:{"time":"2012-03-09T15:01:01.435-05:00","serial":"202","id":"1106","p_host":[],"p_app":"auditd","file":"stdin","line":"4820","auditd.type":"USER_END","auditd.pid":"0","auditd.uid":["0","root"],"auditd.auid":["0","root"],"auditd.ses":"21","auditd.subj":"system_u:system_r:crond_t:s0-s0:c0.c1023","auditd.op":"PAM:session_close","auditd.acct":"root","auditd.exe":"/usr/sbin/crond","auditd.hostname":[],"auditd.addr":[],"auditd.terminal":"cron","auditd.res":"success"}

4 3

thoughts on CEE
by Botond Botyanszki 19 Mar '12

19 Mar '12

Hi, As a follow-up to my previous email, here are some thoughts and opinions about the CEE spec and the effort. I have read the complaints of some other fellow list members and I must admit that I agree with some of them. First of all I must stress that I'm also happy to see the CEE and lumberjack effort to finally have proper structured logging. A couple months ago I repeatedly tried to subscribe to the CEE mailing list with no success for unknown reasons and the CEE "standardization process" seemed to me not happaning in a truly open fashion but rather "behind the scenes". I hope that with project lumberjack this will change and everybody can add his few cents in a truly open way. Here are mine: * JSON/XML formatted CEE payload over RFC 5424 I see no reason for this syslog encapsulation other than support for legacy systems. But RFC 5424 is hardly supported at all. I had mixed feelings about this protocol when we implemented it in nxlog and now that its creators have indirectly admitted its failure, I'm pretty surprised that CEE wants to build upon such a protocol. As others have noted this before, all the fields in the syslog header would be duplicated as structured field data anyway, so why bother sending XML/JSON encapsulated instead of sending it on its own (i.e. leave out syslog and just send the plaing JSON/XML on a single line)? While nxlog supports this encapsulation, this is from the fact that it is flexible enough, and not to show that I'm supporting the idea. RFC5424 has it's own STRUCTURED-DATA format. Probably I'm not the only one who feels that adding XML/JSON formatted data in the message part instead of using the SD-DATA format defined in the RFC is an abuse of the protocol. When we try to do this JSON/XML RFC 5424 encapsulation in nxlog, the end result is that the fields get included in both the STRUCTURED-DATA part and in the JSON/XML. * CEE field names It looks to me that the 'p_sys', 'id' and similar short names are there to save space and bandwidth. Use compression for that. Otherwise these short names will just make people say "WTF is 'id'"? While GUI configuration and log analysis tools can probably map these to a sane and human readable name, this will just lead to double naming. For the tool configuration (as in nxlog.conf, syslog-ng.conf) you'd have to use these short names anyway. We have considered the switch to this CEE naming scheme in nxlog, but are reluctant do so. The current field naming used in nxlog is descriptive and short enough and we believe it is better for our users. If interested, see the nxlog reference manual for the field names we use. * Nested structure While the idea of being able to extend the CEE field list with custom fields is nice, I think this nested structure only adds burden for the implementors and app develpers. A plain flat list with a set of recommended (required/optional) fields would be much easier to use for everyone. Grouping the fields related to the taxonomization under the Type block may hold these releted fields together, but I think this should be done on the view layer, not on the data model. Any log analysis tool can take 'action', 'domain' etc and display it differently to make these stand out from the rest. Mapping a plain field list to SQL databases is trivial. Now with nested structures you need to flatten this out and then must be able convert this back again somehow when producing SQL->CEE. Nxlog, syslog-ng, rsyslog can access or modify fields from the configuration. Now with this nested structure we tool vendors have to come up with a solution to allow access to these nested fields, e.g. in nxlog it would probably be something like $Type.action = 'backup'. While this may look easy, this adds a lot of complexity to the code (both GUI configurator and the log processors) and a possible performance penalty. I think it would be better and easiler for all if CEE only required a flat list of fields and would avoid using the nested structure. The required/optional field names can be still defined and it would be possible to validate even if a tool vendor extends with custom fields. Take a look at the structure used in GELF. This format is getting increasingly popular and there are now many bindings for various languages. This also has a plain flat field list in JSON: easy, simple and great for everybody. (The chunked gzipped part of the spec is another issue which might not be the best.) * "Event":{...} The CEE JSON format has this outer encapsulation. The only reason I see for this is to be in parallel with the XML version, as XML requires a mandatory root tag. But this one in the JSON version just adds space and bandwidth waste. Why not use ony CEE for both, i.e: "CEE":{..} and <CEE></CEE> and get rid of 'Event'. After all we all know that the data is an event. So I'm sorry to say, but I'm also in the "Please say no to CEE" camp as things stand now. No wonder why nxlog, rsyslog and syslog-ng don't have full CEE support yet. People will just use easier protocols instead, or will only partially support CEE as we all do now. While Microsoft was in a position to be able to force their developers use that log horror which their EventLog API gives, users and developers in the Linux/Unix world will just chose another alternative (e.g. GELF) instead of struggling with CEE and being unproductive. So please reconsider otherwise CEE may never be able to gain momentum. Regards, Botond

10 37

dateTime type
by Rainer Gerhards 19 Mar '12

19 Mar '12

Hi all, I think the dateTime type is not sufficient, as it seems not to support subsecord resolution. Or did I simply misread the type & usage samples? Rainer

1 0

syslog/lumberjack mapping
by Rainer Gerhards 19 Mar '12

19 Mar '12

Hi, I have done some mapping of lumberjack to syslog in the rsyslog context. I'd appreciate comments and suggestions. Note that I also tried to do a translation of syslog severity to lumberjack level. Not sure if that's the best way to do it. Ideally, we'd come up with a standard translation so that all syslog implementations do the same... http://blog.gerhards.net/2012/03/rsyslog-cee-base-schema-mapping.html Thanks, Rainer

1 0

Structured Event Format
by William Heinbockel 16 Mar '12

16 Mar '12

Here is my proposal for structured event format. I have removed all of the extraneous structure and have simplified everything down to the most basic requirements. There are no header or required fields, no optional structures, and only one variable type. Other syntaxes, such as XML, CSV, etc. can be easily transformed from this simple structure. This is the basis of CEE and Lumberjack. Everything else should be built on top of this format. CEE requirements will be built on top of this structure in terms of compliance -- such as the required header fields and syntax. E.g., a CEE-compliant event will be encoded using the Structured Event Format and MUST contain the following fields: time, p_host, p_app... Other external tools, such as search tools, transformations (patterndb, liblognorm), etc. should also be build on top of this structure. # Structured Events Format Structured Events are expressed as structured content using a flat list of name-value pairs. This structure provides the minimally viable text-based syntax necessary to support structured logging. ## Model A **log** consists of multiple **events**. An **event** consists of many **fields**. Each **field** has a **name** and zero or more atomic **values**. This is described in the following EBNF grammar: LOG ::= EVENT* EVENT ::= FIELD* FIELD ::= NAME VALUE* ## Representation The Structured Event model is encoded using UTF-8 and represented according to the following grammar: LOG ::= EVENT ( '\r'? '\n' EVENT )* EVENT ::= '{' FIELD ( ',' FIELD )* '}' FIELD ::= NAME ':' VALUES NAME ::= '"' [0-9a-zA-Z_\.]+ '"' VALUES ::= VALUE | '[' ( VALUE ( ',' VALUE )* )? ']' VALUE ::= '"' UTF8CHAR* '"' You may notice that this encoding is a formal subset of [JSON] and is compatible with associative array representations in several programming languages, including Python's `dict` and JavaScript's `dictionary` object. ## Usage * Events are encoded in UTF-8 * Events are separated by LF (or optionally CRLF) * No extraneous whitespace should be present * All field names should be short (< 32 characters), but readable ## Examples Representation of a simple event: {"time":"2012-03-01T12:00:00.123Z","p_app":"vendor.app.version","p_host":"host.example.com","id":"event123","dstport":"12345","srcport":"80","dstip":"10.10.0.1","srcip":"127.0.01"} An event with fields having multiple values: {"time":"2012-03-01T07:01:32-05:00","id":"47","p_app":"httpd","p_host":"localhost.localdomain","one_value":"value1","no_values":[],"many_values":["val1","val2","val3"]} Event using field names with schemas (borrowed from Syslog-NG *patterndb*): {"time":"2012-03-01T13:21:02.005+01:00","p_app":"myapp","p_host":"localhost.localdomain","login.username":"bob","login.client_ip":"127.0.0.1"} ## Embedding in Syslog structured event representation is designed to be embedded into a Syslog *message* and is compatible with both legacy Syslog implementations (e.g., [RFC3164]) as well as the newer Syslog [RFC5424] syntax. In order to accurately parse the entire Syslog message, the following rules apply: 1. The event representation MUST appear after the Syslog header 2. The event representation MUST be immediately preceded by the `@cee:` flag indicating that everything following is a structured event representation 3. The event representation MUST be at the end of the Syslog message. No additional data can be present after the end of the event representation ## Examples using Syslog Representation of a simple event using Syslog RFC3164: Mar 3 12:00:00 host app[123]: @cee:{"time":"2012-03-01T12:00:00.123Z","p_app":"vendor.app.version","p_host":"host.example.com","dstport":"12345","srcport":"80","dstip":"10.10.0.1","srcip":"127.0.01","id":"event123"} An event with fields having multiple values in Syslog RFC5424: 1 2012-03-01T12:01:44Z localhost.localdomain evtslog - ID47 - Some message @cee:{"time":"2012-03-01T07:01:32-05:00","id":"47",p_app":"httpd","p_host":"localhost.localdomain","one_value":"value1","no_values":[],"many_values":["val1","val2","val3"]} [JSON]: http://json.org [RFC3164]: http://www.ietf.org/rfc/rfc3164.txt [RFC5424]: http://www.ietf.org/rfc/rfc5424.txt

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

lumberjack-developers March 2012