New Syntax Spec and updated Field list
by William Heinbockel
Folks,
I'm prepping the CEE 1.0beta release and as part of that, I've tried
to summarize some of the discussions surrounding the field naming and
syntaxes. The plan for CEE to basically adopt this Lumberjack syntax.
An update field list, based on David's suggestions and borrowing from
Syslog & libumberlog, is posted here:
https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList
The Syntax spec is mostly complete, but is still a bit rough around the edges:
https://fedorahosted.org/lumberjack/wiki/SyntaxFormats
What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
Then made a basic conversion from JSON --> XML and tried to align the
libumberlog syntaxes with Keith's XML.
I also added some notes about the "structured" vs. "inline" field naming.
Basically, most of this should line up pretty well with the
discussions we've been having.
The only real delta is the difference between Keith's XML and the
proposed syntax -- not everything aligns, but it is fairly close.
Keith: I will contact you directly with my schema proposals.
12 years
Syslog priority & facility
by William Heinbockel
What are people's opinions on the Syslog priority and facility being
supported in the field list?
I see that they are currently in libumberlog and am not sure that they
should be there.
As used, the *priority* and *facility* have no real meaning outside of
the Syslog protocol.
Priority has some relation to severity, so I can of understand
supporting one/both of those -- though I think a numeric value instead
of a string might make more sense.
Facility however seems very specific to Syslog and we might want to
think of a way to "modernize" the concept when putting the information
into structured data (if the concept belongs there at all)
After all, when you encode the event data into structured text, the
priority/facility are still present in the Syslog header. Carrying
them over into the event data as is makes little sense being that they
are specific to the Syslog protocol.
Thoughts?
12 years
Re: [lumberjack] [ANN] libumberlog 0.2.0 released!
by Rainer Gerhards
Thx for all your hard work and congrats for what has been achieved already :-)
Rainer
Peter Czanik <czanik(a)balabit.hu> hat geschrieben:On 04/16/2012 09:19 PM, Gergely Nagy wrote:
> Hi!
>
> I'll skip the formal release announcement this time, as the new stuff
> the 0.2.0 release brings are fairly small: there was another release
> inbetween, 0.1.2, which fixed a silly mistake I made by not copying
> va_lists.
>
> The 0.2.0 release drops the json-c dependency, all the library needs now
> is libc. This has multiple advantages, ranging from not depending on
> anything outside of /lib on a default install, through noticably faster
> speed, to a more compact JSON payload - one with no unnecessary
> whitespace.
It is now available in FreeBSD ports:
http://www.freshports.org/sysutils/libumberlog/ But as it does not have
ld.so.preload, and even LD_PRELOAD has more limitations, it's not that
much useful, as on Linux. The good news is, that it is going to be fixed
(loader & libmap.conf extended). Libumberlog is the perfect use case to
add this functionality to FreeBSD :-)
It is also part of devel:libraries:c_c++ development project on
openSUSE, which is the stepping stone to the distribution. I still need
to fix some packaging bugs before submitting it to openSUSE factory.
Not yet tested on Fedora, but as it's a very simple spec file, it should
also work there with minor modifications.
The one at
https://build.opensuse.org/package/show?package=libumberlog&project=devel...
still installs to /usr/lib, which is problematic, if used in
/etc/ld.so.preload and /usr is a separate partition. The one in
https://build.opensuse.org/package/show?package=libumberlog&project=home%...
installs to /lib, but I still need to fix the 64bit package. Once it's
fixed, I can push to devel:libraries:c_c++ and to openSUSE factory.
Bye,
--
Peter Czanik (CzP)<czanik(a)balabit.hu>
BalaBit IT Security / syslog-ng upstream
http://czanik.blogs.balabit.com/
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
12 years
[ANN] libumberlog 0.2.0 released!
by Gergely Nagy
Hi!
I'll skip the formal release announcement this time, as the new stuff
the 0.2.0 release brings are fairly small: there was another release
inbetween, 0.1.2, which fixed a silly mistake I made by not copying
va_lists.
The 0.2.0 release drops the json-c dependency, all the library needs now
is libc. This has multiple advantages, ranging from not depending on
anything outside of /lib on a default install, through noticably faster
speed, to a more compact JSON payload - one with no unnecessary
whitespace.
Functionality remains the same, but the dependencies are lighter, and
the output tighter. For the test suite though, one still needs json-c,
but it's completely optional, and not required for the runtime library.
As usual, it is available on github, both via git[1], and as a
tarball[2].
[1]: git://github.com/algernon/libumberlog.git
[2]: https://github.com/downloads/algernon/libumberlog/libumberlog-0.2.0.tar.gz
--
|8]
12 years
Re: [lumberjack] field names and taxonomy
by William Heinbockel
I just updated the Lumberjack Field List on the wiki to start unifying
the field names based on the nxlog, libumberlog, and some of the
auditd fields.
I tried to keep the list minimal and have at least enough support for
syslog compatibility.
<https://fedorahosted.org/lumberjack/wiki/FieldList>
> On Mon, Apr 2, 2012 at 11:51 AM, Jan Zelený <jzeleny(a)redhat.com> wrote:
>>> >> What we should figure out is whether it is better to prefer one field
>>> >> that might have a range of values depending on the producer, or we want
>>> >> to separate these across multiple fields.
>>> >
>>> > I thought the whole point of defining common set of names is to unify
>>> > what is produced by producers, isn't it? If yes then I don't think we
>>> > want a range of multiple fields.
>>>
>>> Correct.
>>>
>>> But there are two ways to approach the problem, either have fields be
>>> unique to the concept or unique to the (concept, value type).
>>
>> Ah, ok. Thanks for clarification
>>
>>> For example, I could argue to have something like
>>>
>>> { "dst":["10.10.0.2","localhost.localdomain"] }
>>>
>>> instead of
>>>
>>> { "dst_ipv4":"10.10.0.2", "dst_host":"localhost.localdomain" }
>>>
>>> While the latter option might be more intuitive, the former is more
>>> flexible. This loops back to practice of whether the datatype should
>>> be associated with an array or the array values.
>>>
>>> >From an end-user standpoint, I would push for the latter
>>
>> I agree with the latter approach. Among other things, it will probably be
>> easier to read and less error-prone when it comes to optional values. (for
>> example only hostname being set for dst)
>>
>> Jan
12 years, 1 month
Field naming structure
by William Heinbockel
Before we dive too deep into which fields to support and how to name
them, we should first discuss the structure and organization of the
names.
I propose we roughly follow the proposal from Balazs:
1. Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
2. Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus,
firewalls, etc.)
We should focus on scoping and defining #1, probably combining the
libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the
best place to start is with the `auditd` list
<http://people.redhat.com/sgrubb/audit/audit-parse.txt> and abstract
linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the
structure should infer the namespace and the namespace should either
be unique to the vendor/product or developed by the community.
{"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to
{"red_hat|auditd|time":"..."}
(feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an
organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from
organization/functional ones.
12 years, 1 month
Re: [lumberjack] [ANN] libumberlog 0.1.1 released!
by Rainer Gerhards
Congrats, very useful :-)
Rainer
Gergely Nagy <algernon(a)balabit.hu> hat geschrieben:---------------------------------------------------------------
PACKAGE : libumberlog
VERSION : 0.1.1
SUMMARY : First (public) stable release
DATE : 2012 April 2
HOMEPAGE : http://algernon.github.com/libumberlog/
---------------------------------------------------------------
DESCRIPTION:
The libumberlog library is a a thin layer over the traditional
syslog() function that turns your legacy syslog messages into
structured logs, so that the message part becomes a JSON string (see
the example below), with a couple of extra fields added by default: a
high-precision timestamp, uid, gid, and pid, to name a few.
The library can be used as a system-wide LD_PRELOAD-ed library,
turning every syslog() message into the improved format, or it can be
LD_PRELOAD-ed on a case-by-case basis, or even linked to, in which
case it provides a few other functions, mentioned below.
EXAMPLE:
Mar 24 12:01:34 localhost sshd[12590]: @cee:{
"msg": "Accepted publickey for algernon from 127.0.0.1 port 55519 ssh2",
"pid": "12590", "facility": "auth", "priority": "info",
"program": "sshd", "uid": "0", "gid": "0",
"host": "hadhodrond", "timestamp": "2012-03-24T12:01:34.236987887+0100" }
FEATURES:
The library supports LD_PRELOAD (and has been used in such capacity on
production systems), turning legacy messages into something little bit
more structured, yet, backwards compatible.
It also introduces a few new functions, namely ul_format() and
ul_syslog(), that allow one to make use of structured logging even
better, by allowing them to include arbitrary key-value pairs in the
message.
Credits
=======
The library has been written by Gergely Nagy <algernon(a)balabit.hu>,
with invaluable feedback from members of Project Lumberjack, an
open-source initiative to update and enhance the event log
architecture.
DOWNLOADS:
The source is available from the git repository at github:
git://github.com/algernon/libumberlog.git
A source tarball has also been released, and is available via github
aswell, at:
https://github.com/downloads/algernon/libumberlog/libumberlog-0.1.1.tar.gz
Documentation and more information about the library is available on
its homepage at http://algernon.github.com/libumberlog/. For more
information about Project Lumberjack, please see their homepage at
https://fedorahosted.org/lumberjack/, and the mailing list at
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
--
|8]
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
12 years, 1 month
[ANN] libumberlog 0.1.1 released!
by Gergely Nagy
---------------------------------------------------------------
PACKAGE : libumberlog
VERSION : 0.1.1
SUMMARY : First (public) stable release
DATE : 2012 April 2
HOMEPAGE : http://algernon.github.com/libumberlog/
---------------------------------------------------------------
DESCRIPTION:
The libumberlog library is a a thin layer over the traditional
syslog() function that turns your legacy syslog messages into
structured logs, so that the message part becomes a JSON string (see
the example below), with a couple of extra fields added by default: a
high-precision timestamp, uid, gid, and pid, to name a few.
The library can be used as a system-wide LD_PRELOAD-ed library,
turning every syslog() message into the improved format, or it can be
LD_PRELOAD-ed on a case-by-case basis, or even linked to, in which
case it provides a few other functions, mentioned below.
EXAMPLE:
Mar 24 12:01:34 localhost sshd[12590]: @cee:{
"msg": "Accepted publickey for algernon from 127.0.0.1 port 55519 ssh2",
"pid": "12590", "facility": "auth", "priority": "info",
"program": "sshd", "uid": "0", "gid": "0",
"host": "hadhodrond", "timestamp": "2012-03-24T12:01:34.236987887+0100" }
FEATURES:
The library supports LD_PRELOAD (and has been used in such capacity on
production systems), turning legacy messages into something little bit
more structured, yet, backwards compatible.
It also introduces a few new functions, namely ul_format() and
ul_syslog(), that allow one to make use of structured logging even
better, by allowing them to include arbitrary key-value pairs in the
message.
Credits
=======
The library has been written by Gergely Nagy <algernon(a)balabit.hu>,
with invaluable feedback from members of Project Lumberjack, an
open-source initiative to update and enhance the event log
architecture.
DOWNLOADS:
The source is available from the git repository at github:
git://github.com/algernon/libumberlog.git
A source tarball has also been released, and is available via github
aswell, at:
https://github.com/downloads/algernon/libumberlog/libumberlog-0.1.1.tar.gz
Documentation and more information about the library is available on
its homepage at http://algernon.github.com/libumberlog/. For more
information about Project Lumberjack, please see their homepage at
https://fedorahosted.org/lumberjack/, and the mailing list at
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
--
|8]
12 years, 1 month
