1:01 p.m.
Folks,
I'm prepping the CEE 1.0beta release and as part of that, I've tried
to summarize some of the discussions surrounding the field naming and
syntaxes. The plan for CEE to basically adopt this Lumberjack syntax.
An update field list, based on David's suggestions and borrowing from
Syslog & libumberlog, is posted here:
https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList
The Syntax spec is mostly complete, but is still a bit rough around the edges:
https://fedorahosted.org/lumberjack/wiki/SyntaxFormats
What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
Then made a basic conversion from JSON --> XML and tried to align the
libumberlog syntaxes with Keith's XML.
I also added some notes about the "structured" vs. "inline" field
naming.
Basically, most of this should line up pretty well with the
discussions we've been having.
The only real delta is the difference between Keith's XML and the
proposed syntax -- not everything aligns, but it is fairly close.
Keith: I will contact you directly with my schema proposals.
3:28 p.m.
Thanks for the update William!
Brian
On Thu, Apr 19, 2012 at 2:01 PM, William Heinbockel
<wheinbockel(a)gmail.com>wrote:
Folks,
I'm prepping the CEE 1.0beta release and as part of that, I've tried
to summarize some of the discussions surrounding the field naming and
syntaxes. The plan for CEE to basically adopt this Lumberjack syntax.
An update field list, based on David's suggestions and borrowing from
Syslog & libumberlog, is posted here:
https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList
The Syntax spec is mostly complete, but is still a bit rough around the
edges:
https://fedorahosted.org/lumberjack/wiki/SyntaxFormats
What I did was take the JSON field types and add datetime, ipv4, and ipv6
types.
Then made a basic conversion from JSON --> XML and tried to align the
libumberlog syntaxes with Keith's XML.
I also added some notes about the "structured" vs. "inline" field
naming.
Basically, most of this should line up pretty well with the
discussions we've been having.
The only real delta is the difference between Keith's XML and the
proposed syntax -- not everything aligns, but it is fairly close.
Keith: I will contact you directly with my schema proposals.
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
6:40 p.m.
On Thu, 19 Apr 2012, William Heinbockel wrote:
Folks,
I'm prepping the CEE 1.0beta release and as part of that, I've tried
to summarize some of the discussions surrounding the field naming and
syntaxes. The plan for CEE to basically adopt this Lumberjack syntax.
An update field list, based on David's suggestions and borrowing from
Syslog & libumberlog, is posted here:
https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList
Overall this looks good. The only quibble I have is where you talk about
Objects and Names. This implies that any hierarchy that exists is only two
levels deep (object!name), and I don't see a good reason to limit it that
way.
David Lang
The Syntax spec is mostly complete, but is still a bit rough around
the edges:
https://fedorahosted.org/lumberjack/wiki/SyntaxFormats
What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
Then made a basic conversion from JSON --> XML and tried to align the
libumberlog syntaxes with Keith's XML.
I also added some notes about the "structured" vs. "inline" field
naming.
Basically, most of this should line up pretty well with the
discussions we've been having.
The only real delta is the difference between Keith's XML and the
proposed syntax -- not everything aligns, but it is fairly close.
Keith: I will contact you directly with my schema proposals.
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
11:34 a.m.
On Thu, Apr 19, 2012 at 7:40 PM, <david(a)lang.hm> wrote:
On Thu, 19 Apr 2012, William Heinbockel wrote:
> Folks,
>
> I'm prepping the CEE 1.0beta release and as part of that, I've tried
> to summarize some of the discussions surrounding the field naming and
> syntaxes. The plan for CEE to basically adopt this Lumberjack syntax.
>
> An update field list, based on David's suggestions and borrowing from
> Syslog & libumberlog, is posted here:
> https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList
Overall this looks good. The only quibble I have is where you talk about
Objects and Names. This implies that any hierarchy that exists is only two
levels deep (object!name), and I don't see a good reason to limit it that
way.
That was not my intention. I will update the wording and add some
additional examples that show how multiple layers work.
I.e., "foo!bar!obj!field" -->
"foo":{"bar":{"obj":{"field":...}}} or
<foo><bar><obj><field>...</field></obj></bar></foo>
That said, I think we should limit the depth to 2 (or maybe 3) for all
of our fields. Of course, we have no control over the depth of the
**native** field contents.
1:44 p.m.
On Mon, 23 Apr 2012, William Heinbockel wrote:
On Thu, Apr 19, 2012 at 7:40 PM, <david(a)lang.hm> wrote:
> On Thu, 19 Apr 2012, William Heinbockel wrote:
>
>> Folks,
>>
>> I'm prepping the CEE 1.0beta release and as part of that, I've tried
>> to summarize some of the discussions surrounding the field naming and
>> syntaxes. The plan for CEE to basically adopt this Lumberjack syntax.
>>
>> An update field list, based on David's suggestions and borrowing from
>> Syslog & libumberlog, is posted here:
>> https://fedorahosted.org/lumberjack/wiki/FieldList#UnifiedFieldList
>
>
> Overall this looks good. The only quibble I have is where you talk about
> Objects and Names. This implies that any hierarchy that exists is only two
> levels deep (object!name), and I don't see a good reason to limit it that
> way.
>
That was not my intention. I will update the wording and add some
additional examples that show how multiple layers work.
I.e., "foo!bar!obj!field" -->
"foo":{"bar":{"obj":{"field":...}}} or
<foo><bar><obj><field>...</field></obj></bar></foo>
That said, I think we should limit the depth to 2 (or maybe 3) for all
of our fields. Of course, we have no control over the depth of the
**native** field contents.
I agree that 2-3 layers is probably a good place to target, but there will
be cases where more make sense (and as you say, in the native structure it
could be really messy)
2 layers is probably a reasonable default to think of using.
David Lang
3:21 a.m.
William Heinbockel <wheinbockel(a)gmail.com> writes:
What I did was take the JSON field types and add datetime, ipv4, and
ipv6 types.
Then made a basic conversion from JSON --> XML and tried to align the
libumberlog syntaxes with Keith's XML.
I also added some notes about the "structured" vs. "inline" field
naming.
Sorry for the late reply, I've been swamped as of late. Nevertheless, my
former opinion that the inline representation is a very, very bad thing
still stands:
First of all, JSON is perfectly capable of representing structured data
(and the syntax format already describes that too!), bolting something
like the inline representation on it is both counter-intuitive for those
who worked with JSON, it's incompatible with anything that is not
explicitly aware of it: you can't just forward it to a database that
happens to accept JSON as-is, you'll have to translate the inline
representation into proper JSON structure. (For example, if I had a
producer that spew out CEE-JSON on one end, and an ElasticSearch indexer
on the other end, I'd need to convert inline-syntax, which would pretty
much destroy the whole idea of having a thin layer.)
It adds quite a bit of burden on the implementation side, and I still
can't see what benefit it would bring, apart from saving a few bytes on
the wire. I'm not entirely sure it would save bytes, either, by the
way.. For example:
{"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
{"foo":{"field1":1,"field2":2,"field3":3}}
Proper structuring is shorteer pretty much the moment you have more than
two fields with the same 'object' prefix.
Furthermore, as mentioned above already, anything that speaks JSON,
understands the structures. None of them understand the inline
syntax. It's alien to JSON as much as trying to bolt on new types were.
It complicates both the sender side (do I want to send stuff as proper
JSON, or with inline representation?), it complicates things on the
consumer side too (do I need to expand inline syntax? if so, it will
likely turn into dotted-notation, and then how do I reassemble it back
if it needs to be forwarded?), and adds extra work on every level.
Granted, parsing nested structures is a bit more code, but seriously,
there are hundreds of libraries out there to work with JSON, multiple
libraries for pretty much every single language, and all of them can
handle nested structures just fine. There is hardly any extra work that
needs be done.
I don't buy the 'compatibility between syntaxes' thing, either. That's
what dot-notation is for, and it needs not be present on the wire.
Another thing that I don't like is that field names are much more
limited than JSON's limitations, and while allowing the full range JSON
allows is probably not a good idea, I'd at least add "-" to the allowed
set.
--
|8]
8:18 a.m.
Hello,
My concerns are the same what Gergely expressed below, thanks for taking
the time to write this.
Other than that, the field list at
https://fedorahosted.org/lumberjack/wiki/FieldList#FieldNames contains
appname and app!name (app.name to put it right;). Same with proc and
possibly others. This is ambiguous, or I'm not understanding something.
I'm happy to see the effort that is being put in the project. On the
other hand I have more general problems with the lumberjack/cee project
as a whole. Instead of listing issues like the above concretely here,
these are my general concerns:
1. Conclusions drawn from the discussions.
There were hundreds of mails sent to the mailing list about the pros and
cons of each solution/method (e.g. to use a flat namespace or not).
The spec in the wiki is evolving. According to William this will be
CEE 1.0 at some point.
Unless it is documented and summarized somewhere (e.g. FAQ section in the
wiki), people will keep repeating the same questions again what we
already discussed and imho not agreed on. I fail to see how the
discussions have influenced the design of the specification.
2. Stance of RedHat in this effort.
It's nice that some folks at RedHat took the lead and organized this
lumberjack project. This makes people think that they (RedHat) are fully
supporting the standardization effort. Yet the following makes it pretty
questionable:
http://lwn.net/Articles/492125/
http://lwn.net/Articles/490690/
http://lwn.net/Articles/464276/
AFAIK the authors of journal both work for RedHat. Instead of striving
for standardization, they are writing code to be later forced upon
millions of users (including us) which perfectly despises the effort of
those individuals (including me) who took their time to participate and
discuss here.
I thought by now RedHat would be a lot more organized company than this
to allow a bunch of hackers playing around. Even BSD can do better in
this regard.
Mainly these are the reasons why I have been silent in the past couple
weeks and decided not to reply because I felt that I'm just wasting my
time. I'll go waste it on something else instead.
Regards,
Botond
On Wed, 02 May 2012 10:21:53 +0200
Gergely Nagy <algernon(a)balabit.hu> wrote:
William Heinbockel <wheinbockel(a)gmail.com> writes:
> What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
> Then made a basic conversion from JSON --> XML and tried to align the
> libumberlog syntaxes with Keith's XML.
> I also added some notes about the "structured" vs. "inline"
field
> naming.
Sorry for the late reply, I've been swamped as of late. Nevertheless, my
former opinion that the inline representation is a very, very bad thing
still stands:
First of all, JSON is perfectly capable of representing structured data
(and the syntax format already describes that too!), bolting something
like the inline representation on it is both counter-intuitive for those
who worked with JSON, it's incompatible with anything that is not
explicitly aware of it: you can't just forward it to a database that
happens to accept JSON as-is, you'll have to translate the inline
representation into proper JSON structure. (For example, if I had a
producer that spew out CEE-JSON on one end, and an ElasticSearch indexer
on the other end, I'd need to convert inline-syntax, which would pretty
much destroy the whole idea of having a thin layer.)
It adds quite a bit of burden on the implementation side, and I still
can't see what benefit it would bring, apart from saving a few bytes on
the wire. I'm not entirely sure it would save bytes, either, by the
way.. For example:
{"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
{"foo":{"field1":1,"field2":2,"field3":3}}
Proper structuring is shorteer pretty much the moment you have more than
two fields with the same 'object' prefix.
Furthermore, as mentioned above already, anything that speaks JSON,
understands the structures. None of them understand the inline
syntax. It's alien to JSON as much as trying to bolt on new types were.
It complicates both the sender side (do I want to send stuff as proper
JSON, or with inline representation?), it complicates things on the
consumer side too (do I need to expand inline syntax? if so, it will
likely turn into dotted-notation, and then how do I reassemble it back
if it needs to be forwarded?), and adds extra work on every level.
Granted, parsing nested structures is a bit more code, but seriously,
there are hundreds of libraries out there to work with JSON, multiple
libraries for pretty much every single language, and all of them can
handle nested structures just fine. There is hardly any extra work that
needs be done.
I don't buy the 'compatibility between syntaxes' thing, either. That's
what dot-notation is for, and it needs not be present on the wire.
Another thing that I don't like is that field names are much more
limited than JSON's limitations, and while allowing the full range JSON
allows is probably not a good idea, I'd at least add "-" to the allowed
set.
--
|8]
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
7:18 a.m.
Botond Botyanszki <boti(a)nxlog.org> writes:
2. Stance of RedHat in this effort.
It's nice that some folks at RedHat took the lead and organized this
lumberjack project. This makes people think that they (RedHat) are fully
supporting the standardization effort. Yet the following makes it pretty
questionable:
http://lwn.net/Articles/492125/
http://lwn.net/Articles/490690/
http://lwn.net/Articles/464276/
AFAIK the authors of journal both work for RedHat. Instead of striving
for standardization, they are writing code to be later forced upon
millions of users (including us) which perfectly despises the effort of
those individuals (including me) who took their time to participate and
discuss here.
Note, I haven't read the lwn articles yet, but I've been talking with
the systemd guys (Lennart in particular), and they showed interest in
normalizing their naming to something that would work for CEE too.
Even enhancing the journald to emit CEE-JSON payload was mentioned, and
on-and-off, I've been poking around the journal code to do just that
(sadly, I have very little time to do that, so nothing came out of it
yet).
They're not working *against* standardisation efforts. They work fast,
and don't want to wait on us. I don't see anything inherently wrong with
that, to be honest.
In any case, I've been lurking on their IRC channel, and talked with
some of them on occassion, I expect that this will continue, and in the
end, we'll end up being compatible both in goals, naming and structure.
They certainly showed interest in that, but at the time, we didn't have
a reasonable consensus, so they went with their own for the time being.
--
|8]
3:18 p.m.
On Wed, 2 May 2012, Gergely Nagy wrote:
William Heinbockel <wheinbockel(a)gmail.com> writes:
> What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
> Then made a basic conversion from JSON --> XML and tried to align the
> libumberlog syntaxes with Keith's XML.
> I also added some notes about the "structured" vs. "inline"
field
> naming.
Sorry for the late reply, I've been swamped as of late. Nevertheless, my
former opinion that the inline representation is a very, very bad thing
still stands:
First of all, JSON is perfectly capable of representing structured data
(and the syntax format already describes that too!), bolting something
like the inline representation on it is both counter-intuitive for those
who worked with JSON, it's incompatible with anything that is not
explicitly aware of it: you can't just forward it to a database that
happens to accept JSON as-is, you'll have to translate the inline
representation into proper JSON structure. (For example, if I had a
producer that spew out CEE-JSON on one end, and an ElasticSearch indexer
on the other end, I'd need to convert inline-syntax, which would pretty
much destroy the whole idea of having a thin layer.)
It adds quite a bit of burden on the implementation side, and I still
can't see what benefit it would bring, apart from saving a few bytes on
the wire. I'm not entirely sure it would save bytes, either, by the
way.. For example:
{"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
{"foo":{"field1":1,"field2":2,"field3":3}}
Proper structuring is shorteer pretty much the moment you have more than
two fields with the same 'object' prefix.
Furthermore, as mentioned above already, anything that speaks JSON,
understands the structures. None of them understand the inline
syntax. It's alien to JSON as much as trying to bolt on new types were.
It complicates both the sender side (do I want to send stuff as proper
JSON, or with inline representation?), it complicates things on the
consumer side too (do I need to expand inline syntax? if so, it will
likely turn into dotted-notation, and then how do I reassemble it back
if it needs to be forwarded?), and adds extra work on every level.
I think that the only good reason for doing having the inline stuff is for
a template based API. Over the wire it would be far better for everything
to be nested JSON so that all recipients only need to know JSON.
But when creating an API to generate the logs, figuring out how to specify
structure is non-trivial and I can see the desire to be able to specify
the fully qualified name of a field.
David Lang
3:26 p.m.
On 05/02/2012 04:18 PM, david(a)lang.hm wrote:
On Wed, 2 May 2012, Gergely Nagy wrote:
> William Heinbockel <wheinbockel(a)gmail.com> writes:
>
>> What I did was take the JSON field types and add datetime, ipv4, and
>> ipv6 types.
>> Then made a basic conversion from JSON --> XML and tried to align the
>> libumberlog syntaxes with Keith's XML.
>> I also added some notes about the "structured" vs. "inline"
field
>> naming.
>
> Sorry for the late reply, I've been swamped as of late. Nevertheless, my
> former opinion that the inline representation is a very, very bad thing
> still stands:
>
> First of all, JSON is perfectly capable of representing structured data
> (and the syntax format already describes that too!), bolting something
> like the inline representation on it is both counter-intuitive for those
> who worked with JSON, it's incompatible with anything that is not
> explicitly aware of it: you can't just forward it to a database that
> happens to accept JSON as-is, you'll have to translate the inline
> representation into proper JSON structure. (For example, if I had a
> producer that spew out CEE-JSON on one end, and an ElasticSearch indexer
> on the other end, I'd need to convert inline-syntax, which would pretty
> much destroy the whole idea of having a thin layer.)
>
> It adds quite a bit of burden on the implementation side, and I still
> can't see what benefit it would bring, apart from saving a few bytes on
> the wire. I'm not entirely sure it would save bytes, either, by the
> way.. For example:
>
> {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
> {"foo":{"field1":1,"field2":2,"field3":3}}
>
> Proper structuring is shorteer pretty much the moment you have more than
> two fields with the same 'object' prefix.
>
> Furthermore, as mentioned above already, anything that speaks JSON,
> understands the structures. None of them understand the inline
> syntax. It's alien to JSON as much as trying to bolt on new types were.
>
> It complicates both the sender side (do I want to send stuff as proper
> JSON, or with inline representation?), it complicates things on the
> consumer side too (do I need to expand inline syntax? if so, it will
> likely turn into dotted-notation, and then how do I reassemble it back
> if it needs to be forwarded?), and adds extra work on every level.
I think that the only good reason for doing having the inline stuff is
for a template based API. Over the wire it would be far better for
everything to be nested JSON so that all recipients only need to know
JSON.
But when creating an API to generate the logs, figuring out how to
specify structure is non-trivial and I can see the desire to be able
to specify the fully qualified name of a field.
I see no problem in having "foo!field1" at the API layer and proposed
nested structure below and on the wire. Seems very reasonable.
David Lang
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
--
Thank you,
Dmitri Pal
Sr. Engineering Manager IPA project,
Red Hat Inc.
-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/
4:04 p.m.
----- Original message -----
On 05/02/2012 04:18 PM, david(a)lang.hm wrote:
> On Wed, 2 May 2012, Gergely Nagy wrote:
>
> > William Heinbockel <wheinbockel(a)gmail.com> writes:
> >
> > > What I did was take the JSON field types and add datetime, ipv4,
> > > and ipv6 types.
> > > Then made a basic conversion from JSON --> XML and tried to align
> > > the libumberlog syntaxes with Keith's XML.
> > > I also added some notes about the "structured" vs.
"inline" field
> > > naming.
> >
> > Sorry for the late reply, I've been swamped as of late.
> > Nevertheless, my former opinion that the inline representation is a
> > very, very bad thing still stands:
> >
> > First of all, JSON is perfectly capable of representing structured
> > data (and the syntax format already describes that too!), bolting
> > something like the inline representation on it is both
> > counter-intuitive for those who worked with JSON, it's incompatible
> > with anything that is not explicitly aware of it: you can't just
> > forward it to a database that happens to accept JSON as-is, you'll
> > have to translate the inline representation into proper JSON
> > structure. (For example, if I had a producer that spew out CEE-JSON
> > on one end, and an ElasticSearch indexer on the other end, I'd need
> > to convert inline-syntax, which would pretty much destroy the whole
> > idea of having a thin layer.)
> >
> > It adds quite a bit of burden on the implementation side, and I still
> > can't see what benefit it would bring, apart from saving a few bytes
> > on the wire. I'm not entirely sure it would save bytes, either, by
> > the way.. For example:
> >
> > {"foo!field1":1,"foo!field2":2,"foo!field3":3}
vs
> >
{"foo":{"field1":1,"field2":2,"field3":3}}
> >
> > Proper structuring is shorteer pretty much the moment you have more
> > than two fields with the same 'object' prefix.
> >
> > Furthermore, as mentioned above already, anything that speaks JSON,
> > understands the structures. None of them understand the inline
> > syntax. It's alien to JSON as much as trying to bolt on new types
> > were.
> >
> > It complicates both the sender side (do I want to send stuff as
> > proper JSON, or with inline representation?), it complicates things
> > on the consumer side too (do I need to expand inline syntax? if so,
> > it will likely turn into dotted-notation, and then how do I
> > reassemble it back if it needs to be forwarded?), and adds extra
> > work on every level.
>
> I think that the only good reason for doing having the inline stuff is
> for a template based API. Over the wire it would be far better for
> everything to be nested JSON so that all recipients only need to know
> JSON.
>
> But when creating an API to generate the logs, figuring out how to
> specify structure is non-trivial and I can see the desire to be able
> to specify the fully qualified name of a field.
I see no problem in having "foo!field1" at the API layer and proposed
nested structure below and on the wire. Seems very reasonable.
cool, we have a conclusion then?
I'd recommend to use json dotted notation for fields though. It's more standard
than the exclamation mark stuff. but this is certainly an API territory.
4:02 p.m.
----- Original message -----
On Wed, 2 May 2012, Gergely Nagy wrote:
> William Heinbockel <wheinbockel(a)gmail.com> writes:
>
> > What I did was take the JSON field types and add datetime, ipv4, and
> > ipv6 types. Then made a basic conversion from JSON --> XML and tried
> > to align the libumberlog syntaxes with Keith's XML.
> > I also added some notes about the "structured" vs. "inline"
field
> > naming.
>
> Sorry for the late reply, I've been swamped as of late. Nevertheless,
> my former opinion that the inline representation is a very, very bad
> thing still stands:
>
> First of all, JSON is perfectly capable of representing structured data
> (and the syntax format already describes that too!), bolting something
> like the inline representation on it is both counter-intuitive for
> those who worked with JSON, it's incompatible with anything that is not
> explicitly aware of it: you can't just forward it to a database that
> happens to accept JSON as-is, you'll have to translate the inline
> representation into proper JSON structure. (For example, if I had a
> producer that spew out CEE-JSON on one end, and an ElasticSearch
> indexer on the other end, I'd need to convert inline-syntax, which
> would pretty much destroy the whole idea of having a thin layer.)
>
> It adds quite a bit of burden on the implementation side, and I still
> can't see what benefit it would bring, apart from saving a few bytes on
> the wire. I'm not entirely sure it would save bytes, either, by the
> way.. For example:
>
> {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
> {"foo":{"field1":1,"field2":2,"field3":3}}
>
> Proper structuring is shorteer pretty much the moment you have more
> than two fields with the same 'object' prefix.
>
> Furthermore, as mentioned above already, anything that speaks JSON,
> understands the structures. None of them understand the inline
> syntax. It's alien to JSON as much as trying to bolt on new types were.
>
> It complicates both the sender side (do I want to send stuff as proper
> JSON, or with inline representation?), it complicates things on the
> consumer side too (do I need to expand inline syntax? if so, it will
> likely turn into dotted-notation, and then how do I reassemble it back
> if it needs to be forwarded?), and adds extra work on every level.
I think that the only good reason for doing having the inline stuff is
for a template based API. Over the wire it would be far better for
everything to be nested JSON so that all recipients only need to know
JSON.
But when creating an API to generate the logs, figuring out how to
specify structure is non-trivial and I can see the desire to be able to
specify the fully qualified name of a field.
I can understand this desire, however this is what the dotted notation is good for. E.g.
if it's simpler to use a flat namespace in an API, use that. the mapping between the
json structure is quite standard.
In syslog-ng I'm planning to use dotted notation when working with fields as syslog-ng
has a flat structure in its event model, but then generate nested json at the end.
The same can be done in any logging API.
4:17 p.m.
On Wed, 2 May 2012, Balazs Scheidler wrote:
----- Original message -----
> On Wed, 2 May 2012, Gergely Nagy wrote:
>
>> William Heinbockel <wheinbockel(a)gmail.com> writes:
>>
>>> What I did was take the JSON field types and add datetime, ipv4, and
>>> ipv6 types. Then made a basic conversion from JSON --> XML and tried
>>> to align the libumberlog syntaxes with Keith's XML.
>>> I also added some notes about the "structured" vs.
"inline" field
>>> naming.
>>
>> Sorry for the late reply, I've been swamped as of late. Nevertheless,
>> my former opinion that the inline representation is a very, very bad
>> thing still stands:
>>
>> First of all, JSON is perfectly capable of representing structured data
>> (and the syntax format already describes that too!), bolting something
>> like the inline representation on it is both counter-intuitive for
>> those who worked with JSON, it's incompatible with anything that is not
>> explicitly aware of it: you can't just forward it to a database that
>> happens to accept JSON as-is, you'll have to translate the inline
>> representation into proper JSON structure. (For example, if I had a
>> producer that spew out CEE-JSON on one end, and an ElasticSearch
>> indexer on the other end, I'd need to convert inline-syntax, which
>> would pretty much destroy the whole idea of having a thin layer.)
>>
>> It adds quite a bit of burden on the implementation side, and I still
>> can't see what benefit it would bring, apart from saving a few bytes on
>> the wire. I'm not entirely sure it would save bytes, either, by the
>> way.. For example:
>>
>> {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
>>
{"foo":{"field1":1,"field2":2,"field3":3}}
>>
>> Proper structuring is shorteer pretty much the moment you have more
>> than two fields with the same 'object' prefix.
>>
>> Furthermore, as mentioned above already, anything that speaks JSON,
>> understands the structures. None of them understand the inline
>> syntax. It's alien to JSON as much as trying to bolt on new types were.
>>
>> It complicates both the sender side (do I want to send stuff as proper
>> JSON, or with inline representation?), it complicates things on the
>> consumer side too (do I need to expand inline syntax? if so, it will
>> likely turn into dotted-notation, and then how do I reassemble it back
>> if it needs to be forwarded?), and adds extra work on every level.
>
> I think that the only good reason for doing having the inline stuff is
> for a template based API. Over the wire it would be far better for
> everything to be nested JSON so that all recipients only need to know
> JSON.
>
> But when creating an API to generate the logs, figuring out how to
> specify structure is non-trivial and I can see the desire to be able to
> specify the fully qualified name of a field.
>
I can understand this desire, however this is what the dotted notation is good for. E.g.
if it's simpler to use a flat namespace in an API, use that. the mapping between the
json structure is quite standard.
In syslog-ng I'm planning to use dotted notation when working with fields as
syslog-ng has a flat structure in its event model, but then generate nested json at the
end.
The same can be done in any logging API.
is the only difference between the two notations the fact that the
serparator character is '!' instead of '.'?
If so, I don't have any strong preference betwen them, but I would think
that '.' would be more likley to conflict with other uses.
David Lang
5:02 p.m.
On 05/02/2012 05:17 PM, david(a)lang.hm wrote:
On Wed, 2 May 2012, Balazs Scheidler wrote:
> ----- Original message -----
>> On Wed, 2 May 2012, Gergely Nagy wrote:
>>
>>> William Heinbockel <wheinbockel(a)gmail.com> writes:
>>>
>>>> What I did was take the JSON field types and add datetime, ipv4, and
>>>> ipv6 types. Then made a basic conversion from JSON --> XML and tried
>>>> to align the libumberlog syntaxes with Keith's XML.
>>>> I also added some notes about the "structured" vs.
"inline" field
>>>> naming.
>>>
>>> Sorry for the late reply, I've been swamped as of late. Nevertheless,
>>> my former opinion that the inline representation is a very, very bad
>>> thing still stands:
>>>
>>> First of all, JSON is perfectly capable of representing structured
>>> data
>>> (and the syntax format already describes that too!), bolting something
>>> like the inline representation on it is both counter-intuitive for
>>> those who worked with JSON, it's incompatible with anything that is
>>> not
>>> explicitly aware of it: you can't just forward it to a database that
>>> happens to accept JSON as-is, you'll have to translate the inline
>>> representation into proper JSON structure. (For example, if I had a
>>> producer that spew out CEE-JSON on one end, and an ElasticSearch
>>> indexer on the other end, I'd need to convert inline-syntax, which
>>> would pretty much destroy the whole idea of having a thin layer.)
>>>
>>> It adds quite a bit of burden on the implementation side, and I still
>>> can't see what benefit it would bring, apart from saving a few
>>> bytes on
>>> the wire. I'm not entirely sure it would save bytes, either, by the
>>> way.. For example:
>>>
>>> {"foo!field1":1,"foo!field2":2,"foo!field3":3}
vs
>>>
{"foo":{"field1":1,"field2":2,"field3":3}}
>>>
>>> Proper structuring is shorteer pretty much the moment you have more
>>> than two fields with the same 'object' prefix.
>>>
>>> Furthermore, as mentioned above already, anything that speaks JSON,
>>> understands the structures. None of them understand the inline
>>> syntax. It's alien to JSON as much as trying to bolt on new types
>>> were.
>>>
>>> It complicates both the sender side (do I want to send stuff as proper
>>> JSON, or with inline representation?), it complicates things on the
>>> consumer side too (do I need to expand inline syntax? if so, it will
>>> likely turn into dotted-notation, and then how do I reassemble it back
>>> if it needs to be forwarded?), and adds extra work on every level.
>>
>> I think that the only good reason for doing having the inline stuff is
>> for a template based API. Over the wire it would be far better for
>> everything to be nested JSON so that all recipients only need to know
>> JSON.
>>
>> But when creating an API to generate the logs, figuring out how to
>> specify structure is non-trivial and I can see the desire to be
>> able to
>> specify the fully qualified name of a field.
>>
>
> I can understand this desire, however this is what the dotted
> notation is good for. E.g. if it's simpler to use a flat namespace in
> an API, use that. the mapping between the json structure is quite
> standard.
>
> In syslog-ng I'm planning to use dotted notation when working with
> fields as syslog-ng has a flat structure in its event model, but then
> generate nested json at the end.
>
> The same can be done in any logging API.
is the only difference between the two notations the fact that the
serparator character is '!' instead of '.'?
If so, I don't have any strong preference betwen them, but I would
think that '.' would be more likley to conflict with other uses.
Yes that was the only reason for picking '!'. I came across conflicts
when field name has to have '.'.
I might envision that host name might become a field name in some case
for example.
'!" looks much safer to me.
David Lang
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
--
Thank you,
Dmitri Pal
Sr. Engineering Manager IPA project,
Red Hat Inc.
-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/
6:58 p.m.
On Wed, 2 May 2012, Dmitri Pal wrote:
On 05/02/2012 05:17 PM, david(a)lang.hm wrote:
> On Wed, 2 May 2012, Balazs Scheidler wrote:
>
>> ----- Original message -----
>>> On Wed, 2 May 2012, Gergely Nagy wrote:
>>>
>>>>
>>>> It complicates both the sender side (do I want to send stuff as proper
>>>> JSON, or with inline representation?), it complicates things on the
>>>> consumer side too (do I need to expand inline syntax? if so, it will
>>>> likely turn into dotted-notation, and then how do I reassemble it back
>>>> if it needs to be forwarded?), and adds extra work on every level.
>>>
>>> I think that the only good reason for doing having the inline stuff is
>>> for a template based API. Over the wire it would be far better for
>>> everything to be nested JSON so that all recipients only need to know
>>> JSON.
>>>
>>> But when creating an API to generate the logs, figuring out how to
>>> specify structure is non-trivial and I can see the desire to be
>>> able to
>>> specify the fully qualified name of a field.
>>>
>>
>> I can understand this desire, however this is what the dotted
>> notation is good for. E.g. if it's simpler to use a flat namespace in
>> an API, use that. the mapping between the json structure is quite
>> standard.
>>
>> In syslog-ng I'm planning to use dotted notation when working with
>> fields as syslog-ng has a flat structure in its event model, but then
>> generate nested json at the end.
>>
>> The same can be done in any logging API.
>
> is the only difference between the two notations the fact that the
> serparator character is '!' instead of '.'?
>
> If so, I don't have any strong preference betwen them, but I would
> think that '.' would be more likley to conflict with other uses.
Yes that was the only reason for picking '!'. I came across conflicts
when field name has to have '.'.
I might envision that host name might become a field name in some case
for example.
'!" looks much safer to me.
we need to differentiate between the data (which may have any text in it)
and the field names (where we are saying we want to eliminate a few
characters, one for the hierarchy indicator)
if JSON already has a standard we should use it, if it doesn't we should
try to use something that minimizes conflicts.
David Lang
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
7:06 a.m.
Dmitri Pal <dpal(a)redhat.com> writes:
> If so, I don't have any strong preference betwen them, but I
would
> think that '.' would be more likley to conflict with other uses.
Yes that was the only reason for picking '!'. I came across conflicts
when field name has to have '.'.
I might envision that host name might become a field name in some case
for example.
'!" looks much safer to me.
Safer, perhaps, but at the cost of being different from everyone else
who uses JSON, and putting additional burden on both producers and
consumers to recognise this extended syntax.
For the record, the current Event Syntax on the wiki[1] does not allow
neither '!' nor '.' in field names, so hostnames can't become key
names
and still be valid if we follow the current syntax: they would need to
be rewritten anyway. And if they get rewritten, we might aswell do that
so we can still use dot-notation, something that is well known,
understood and expected when working with JSON.
[1]: https://fedorahosted.org/lumberjack/wiki/SyntaxFormats
Nevertheless, whether '!' or '.', I wouldn't want either to be seen
on
the wire. In a template or property-access API, sure! That is where it
belongs, but not on the wire.
--
|8]
6:56 a.m.
david(a)lang.hm writes:
is the only difference between the two notations the fact that the
serparator character is '!' instead of '.'?
Pretty much, yes, that is the only difference.
If so, I don't have any strong preference betwen them, but I
would
think that '.' would be more likley to conflict with other uses.
On the other hand, dot-notation already exist, and is used by pretty
much anyone who's dealing with JSON or JavaScript in general. I think it
is unlikely that '.' would cause any conflict, as it's primary use in
the JSON world (in key names) is as a separator.
--
|8]
6:52 a.m.
david(a)lang.hm writes:
I think that the only good reason for doing having the inline stuff
is
for a template based API. Over the wire it would be far better for
everything to be nested JSON so that all recipients only need to know
JSON.
A template based API can use dot-notation. That is the standard way in
javascript to reach into objects anyway, and people dealing with JSON
are most probably already familiar with it.
You can use dot-notation in the API, and behind the courtain, it will
turn it back to structured data on the wire.
So there is an existing, already understood and wide-spread way to reach
into objects and pretty much accomplish everything the object!field
stuff does: dot-notation. But it should not appear on the wire.
I understand the need to standardize how a structure can be 'flattened'
and how one can access the various parts of it using a template-ish API,
but this should be separate from the on-the-wire format, and should - in
my opinion - be dot-notation, as that's something already well
understood.
--
|8]
4401
days inactive
4415
days old
lumberjack-developers@lists.fedorahosted.org
17 comments
7 participants
participants (7)
-
Balazs Scheidler -
Botond Botyanszki -
Brian Knox -
David Lang -
Dmitri Pal -
Gergely Nagy -
William Heinbockel