I posted some of this at the bottom of a long message yesterday that nobody replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with the only feedback being that the write could block.
With the new formats and new syslog replacement library, does there need to be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no changes (which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to have a way to tell what format to use when writing the log.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
David Lang
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
Regards, Botond
On Tue, 27 Mar 2012, Botond Botyanszki wrote:
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
other than recompiling things, how would the sysadmin change the configuration system wide?
There's a significant overhead in searching for a config file and parsing it, so I wouldn't want to go that route.
David Lang
On Tue, 27 Mar 2012 13:07:51 -0700 (PDT) david@lang.hm wrote:
On Tue, 27 Mar 2012, Botond Botyanszki wrote:
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
other than recompiling things, how would the sysadmin change the configuration system wide?
There's a significant overhead in searching for a config file and parsing it, so I wouldn't want to go that route.
There are other options too: * environment variables * loadable modules Both of these are already used within glibc.
Regards, Botond
On Tue, 27 Mar 2012, Botond Botyanszki wrote:
On Tue, 27 Mar 2012 13:07:51 -0700 (PDT) david@lang.hm wrote:
On Tue, 27 Mar 2012, Botond Botyanszki wrote:
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
other than recompiling things, how would the sysadmin change the configuration system wide?
There's a significant overhead in searching for a config file and parsing it, so I wouldn't want to go that route.
There are other options too:
- environment variables
- loadable modules
Both of these are already used within glibc.
environment variables are not system-wide (especially for things fired off at system start)
loadable modules require code changes by the application programmer.
David Lang
On 03/27/2012 04:29 PM, Botond Botyanszki wrote:
On Tue, 27 Mar 2012 13:07:51 -0700 (PDT) david@lang.hm wrote:
On Tue, 27 Mar 2012, Botond Botyanszki wrote:
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
other than recompiling things, how would the sysadmin change the configuration system wide?
There's a significant overhead in searching for a config file and parsing it, so I wouldn't want to go that route.
There are other options too:
- environment variables
This is a bit ugly to require every app to set env variables. This is close to the config file hassle.
- loadable modules
Yes. I want a small module that has one function
const char *syslog_capabilities(void);
that will return a commas separated list of the structured log formats in a priority order: "JSON,XML,XMLT,BSON" or something like. The the library will figure out what format to use base on what it understands.
No config files, no environment varibles. The module comes with syslog implementation. I do not see any drawback in such approach.
Both of these are already used within glibc.
Regards, Botond _______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
There's a significant overhead in searching for a config file and
parsing
it, so I wouldn't want to go that route.
There are other options too:
- environment variables
+1 ;)
Rainer
- loadable modules
Both of these are already used within glibc.
Regards, Botond _______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
----- Original message -----
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
+1
Either just assume that the server supports it or use /dev/slog or something where you can assume support for this. I wouldn't open a negotiation phase unless that's also available on the network which opens the scope considerably: define a new protocol for syslog. wasn't an easy thing with 3164 or 5424 either.
'logger' uses a custom implementation for submitting log messages for example.
On Tue, 27 Mar 2012, Balazs Scheidler wrote:
----- Original message -----
On Tue, 27 Mar 2012 12:18:56 -0700 (PDT) david@lang.hm wrote:
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
If you want a bidirectional protocol, you should create a new interface for that instead of breaking compatibility with older tools and doing it over the good old /dev/log unix domain socket. But I think this is an overkill if it is only designed to query capabilities locally and not over the network. If the local admin can configure the syslogd, he could surely configure the syslog interface (inside or outside glibc) so that they understand each other. This would be the job of the distro maintainers anyway.
+1
Either just assume that the server supports it or use /dev/slog or something where you can assume support for this. I wouldn't open a negotiation phase unless that's also available on the network which opens the scope considerably: define a new protocol for syslog. wasn't an easy thing with 3164 or 5424 either.
'logger' uses a custom implementation for submitting log messages for example.
I don't understand your point about logger.
The idea is that if the sender ignores the capabilities string, things would just work the same way they do today.
The problem with doing slog is that it means that the apps need to be changed to open /dev/slog instead of /dev/log, and it breaks backwards compatibility if the syslog daemon doesn't support /dev/slog
At the network layer, this is less of a problem as the admin can configure all the systems, and can do upgrades from the central server out.
David Lang
_______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that nobody replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with the only feedback being that the write could block.
With the new formats and new syslog replacement library, does there need to be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no changes (which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to have a way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
1. Backwards-compatible structured log implementation that works with the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
2. Implement a new event API frontend and improved logging capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that nobody replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with the only feedback being that the write could block.
With the new formats and new syslog replacement library, does there need to be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no changes (which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to have a way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they will never send structured logs at all?
I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
David Lang
On Mar 27, 2012 4:17 PM, david@lang.hm wrote:
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that
nobody
replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with
the
only feedback being that the write could block.
With the new formats and new syslog replacement library, does there
need to
be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no
changes
(which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to
have a
way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they will
never send structured logs at all?
There are only two ways to have applications log with structured data: 1. Update the app to send name-value pairs 2. Use something like patterndb to convert the unstructured data
If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
I see there being value in having the cee-syslog replacement library that
sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be
determined
automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities
message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send
data
to the server.
This approach lets the format be anything that the library supports,
under
the control of the sysadmin of the system (by configuring what
capabilities
the syslog daemon advertises) without the application programmer
needing to
know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a good
idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.
William Heinbockel
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Mar 27, 2012 4:17 PM, david@lang.hm wrote:
But does this mean that unless applications are re-written, they will
never send structured logs at all?
There are only two ways to have applications log with structured data:
- Update the app to send name-value pairs
- Use something like patterndb to convert the unstructured data
If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
I don't expect the applications that are writing the logs to care about structured data, they generally don't need it (as it's just as easy for them to do their own thing, they know how to understand it)
The big benifit of structured logs is to the people consuming the logs, the organizations running the applications.
If they cannot benefit until the programs are re-written, then I think that a lot of the value of structured logging is lost.
I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This is continuing on this subject.
Today the syslog messages have almost no structure, most of the data in the log is generated by the application (and so things like uid, pid, etc cannot be trusted as they are just what the app claims that they are)
going to structured logging without any app changes makes the metadata in the resulting logs more trustworthy, and allows for a lot of additional metadata to be put in the logs with no application changes.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.
these are almost exclusivly additional log sources that generate logs, not consume them.
David Lang
_______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Mar 27, 2012 4:17 PM, david@lang.hm wrote:
Adding a new application API to generate more structured logs is a good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Ok, breaking this into a separate sub-thread.
I agree that it's a good idea to know what the ideal situation is before defineing how to get there (as this lets you avoid false starts and wasted effort)
so, if /dev/log is not the best interface to passing log messages to a system daemon for delivery, what would the ideal interface look like?
I believe the following requiremsnts will apply.
There will be multiple logging formats (some people love XML, some hate it, some people will want the speed of a binary logging protocol, some people will want the debugging ease of a human-readable protocol, etc)
There needs to be a way to run multiple protocols over one interface, having to have multiple interfaces (one for each protocol) gets really messy.
The format used for the message should be able to be set by the system administrator without requireing code changes in the applications, and without depending that all applications are using the same logging library.
What other requirements do people see?
David Lang
On 03/27/2012 06:17 PM, William Heinbockel wrote:
On Mar 27, 2012 4:17 PM, <david@lang.hm mailto:david@lang.hm> wrote:
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, <david@lang.hm
mailto:david@lang.hm> wrote:
I posted some of this at the bottom of a long message yesterday
that nobody
replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface,
with the
only feedback being that the write could block.
With the new formats and new syslog replacement library, does
there need to
be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with
no changes
(which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need
to have a
way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they
will never send structured logs at all?
There are only two ways to have applications log with structured data:
- Update the app to send name-value pairs
- Use something like patterndb to convert the unstructured data
If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
I see there being value in having the cee-syslog replacement library
that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be
determined
automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities
message.
The client can ignore this message and send traditional formatted
syslog
data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to
send data
to the server.
This approach lets the format be anything that the library
supports, under
the control of the sysadmin of the system (by configuring what
capabilities
the syslog daemon advertises) without the application programmer
needing to
know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a
good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.
William Heinbockel
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard. IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On Tue, 27 Mar 2012, Dmitri Pal wrote:
On 03/27/2012 06:17 PM, William Heinbockel wrote:
On Mar 27, 2012 4:17 PM, <david@lang.hm mailto:david@lang.hm> wrote:
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, <david@lang.hm
mailto:david@lang.hm> wrote:
I posted some of this at the bottom of a long message yesterday
that nobody
replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface,
with the
only feedback being that the write could block.
With the new formats and new syslog replacement library, does
there need to
be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with
no changes
(which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need
to have a
way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they
will never send structured logs at all?
There are only two ways to have applications log with structured data:
- Update the app to send name-value pairs
- Use something like patterndb to convert the unstructured data
If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
I see there being value in having the cee-syslog replacement library
that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be
determined
automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities
message.
The client can ignore this message and send traditional formatted
syslog
data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to
send data
to the server.
This approach lets the format be anything that the library
supports, under
the control of the sysadmin of the system (by configuring what
capabilities
the syslog daemon advertises) without the application programmer
needing to
know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a
good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.
William Heinbockel
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard.
as I am understanding it, the primary value of umberlog is that it gets us structured metadata (and more metadata) with no application changes.
I agree that we need to define additional functions to use for adding application structured data to the log.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
I think we differ in details, but not in overall strategy.
David Lang
_______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On 03/27/2012 08:49 PM, david@lang.hm wrote:
On Tue, 27 Mar 2012, Dmitri Pal wrote:
On 03/27/2012 06:17 PM, William Heinbockel wrote:
On Mar 27, 2012 4:17 PM, <david@lang.hm mailto:david@lang.hm> wrote:
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, <david@lang.hm
mailto:david@lang.hm> wrote:
I posted some of this at the bottom of a long message yesterday
that nobody
replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface,
with the
only feedback being that the write could block.
With the new formats and new syslog replacement library, does
there need to
be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with
no changes
(which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need
to have a
way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they
will never send structured logs at all?
There are only two ways to have applications log with structured data:
- Update the app to send name-value pairs
- Use something like patterndb to convert the unstructured data
If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
I see there being value in having the cee-syslog replacement library
that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be
determined
automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities
message.
The client can ignore this message and send traditional formatted
syslog
data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to
send data
to the server.
This approach lets the format be anything that the library
supports, under
the control of the sysadmin of the system (by configuring what
capabilities
the syslog daemon advertises) without the application programmer
needing to
know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a
good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.
William Heinbockel
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard.
as I am understanding it, the primary value of umberlog is that it gets us structured metadata (and more metadata) with no application changes.
I agree that we need to define additional functions to use for adding application structured data to the log.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
I think we differ in details, but not in overall strategy.
May be I am missing something but I fail to understand how changing from syslog() function to ul_syslog() is not an application change.
David Lang
lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On Tue, 27 Mar 2012, Dmitri Pal wrote:
On 03/27/2012 08:49 PM, david@lang.hm wrote:
as I am understanding it, the primary value of umberlog is that it gets us structured metadata (and more metadata) with no application changes.
I agree that we need to define additional functions to use for adding application structured data to the log.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
I think we differ in details, but not in overall strategy.
May be I am missing something but I fail to understand how changing from syslog() function to ul_syslog() is not an application change.
it could be that I am misunderstanding things, but as I understand it, if you just do a LD_PRELOAD of umberlog with no application changes the result is log messages with structured metadata around the raw log message.
If the programmer wants to add additional name/value pairs they can change individual syslog() calls to ul_syslog() and pass in parameters.
I actually don't expect many people to make a lot of use of this option, but in many cases something as simple as this is really going to be the right thing to do.
As I see it, the end result in terms of C is going to be four possible logging calls
1. syslog()
traditional interface, no app changes. structured metadata with one message field.
2. ul_syslog()
syslog() with some additional name/value pairs added
3. template based (ELAPI or similar)
a template that lists the names, followed by the types and values
4. memory structure based
a tree structure is created pointing to the names and values. a pointer to this structure is passed to the logging call.
This last one could be a variation or replacement for ul_syslog(), and I think it would be ideal if we could hash things out so that it can be just a newer variation of ul_syslog()
I think that all of these variations are appropriate to have (with the possible combination of #2 and #4)
David Lang
it could be that I am misunderstanding things, but as I understand it, if you just do a LD_PRELOAD of umberlog with no application changes the result is log messages with structured metadata around the raw log message.
While I find this a notable effort, I personally do not see much value in these "extra" nv pairs. After all, this is metadata that I can also get hold of via SCM_CREDENTIALS, which is also a superior way in regard to trust. So to me this is just a "nice little gimmick" (not meant to upset something, I see some value in this, just not very much - *my* *personal* opinion).
If the programmer wants to add additional name/value pairs they can change individual syslog() calls to ul_syslog() and pass in parameters.
That *IMO* is the prime reason for having umberlog. We want an interface that permits apps to submit "real" nv pairs.
I actually don't expect many people to make a lot of use of this option,
then we have already lost... because everything else we (almost) already have. Remember that I hacked a PoC to get metadata via SCM_CREDENTIALS into rsyslog and it is extremely easy to extend the metadata set. No lib needed for that. I am very sure syslog-ng will take no longer than a day if it does not already have the capability. Expect the same for nxlog and...
Rainer
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
it could be that I am misunderstanding things, but as I understand it, if you just do a LD_PRELOAD of umberlog with no application changes the result is log messages with structured metadata around the raw log message.
While I find this a notable effort, I personally do not see much value in these "extra" nv pairs. After all, this is metadata that I can also get hold of via SCM_CREDENTIALS, which is also a superior way in regard to trust. So to me this is just a "nice little gimmick" (not meant to upset something, I see some value in this, just not very much - *my* *personal* opinion).
If the programmer wants to add additional name/value pairs they can change individual syslog() calls to ul_syslog() and pass in parameters.
That *IMO* is the prime reason for having umberlog. We want an interface that permits apps to submit "real" nv pairs.
I actually don't expect many people to make a lot of use of this option,
then we have already lost... because everything else we (almost) already have. Remember that I hacked a PoC to get metadata via SCM_CREDENTIALS into rsyslog and it is extremely easy to extend the metadata set. No lib needed for that. I am very sure syslog-ng will take no longer than a day if it does not already have the capability. Expect the same for nxlog and...
Unless you are passing a structured log message around, how do you get the SCM_CREDENTIALS into the log message in a way that doesn't let them get confused with data that the application is writing?
That's one problem with the PoC that you did for this, by putting the data as text at the end of the message, it makes it hard for parsers to grab it and be sure they are getitng the data from rsyslog, not something provided by an app.
David Lang
-----Original Message----- From: lumberjack-developers-bounces@lists.fedorahosted.org [mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf Of david@lang.hm Sent: Wednesday, March 28, 2012 6:56 PM To: lumberjack logging Subject: Re: [lumberjack] connection to syslog (or equivalent)
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
it could be that I am misunderstanding things, but as I understand
it,
if you just do a LD_PRELOAD of umberlog with no application changes the result is log messages with structured metadata around the raw log message.
While I find this a notable effort, I personally do not see much
value in
these "extra" nv pairs. After all, this is metadata that I can also
get hold
of via SCM_CREDENTIALS, which is also a superior way in regard to
trust. So
to me this is just a "nice little gimmick" (not meant to upset
something, I
see some value in this, just not very much - *my* *personal*
opinion).
If the programmer wants to add additional name/value pairs they can change individual syslog() calls to ul_syslog() and pass in parameters.
That *IMO* is the prime reason for having umberlog. We want an
interface that
permits apps to submit "real" nv pairs.
I actually don't expect many people to make a lot of use of this option,
then we have already lost... because everything else we (almost)
already
have. Remember that I hacked a PoC to get metadata via
SCM_CREDENTIALS into
rsyslog and it is extremely easy to extend the metadata set. No lib
needed
for that. I am very sure syslog-ng will take no longer than a day if
it does
not already have the capability. Expect the same for nxlog and...
Unless you are passing a structured log message around, how do you get the SCM_CREDENTIALS into the log message in a way that doesn't let them get confused with data that the application is writing?
I'd still put them into wherever umberlog puts them (so let's say JSON content). My key point is that I can add this information without any help from a lib.
That's one problem with the PoC that you did for this, by putting the data as text at the end of the message, it makes it hard for parsers to grab it and be sure they are getitng the data from rsyslog, not something provided by an app.
Sure, there was no standard to follow at that time and the main point was to prove that it is doable (as journald folks said its impossible).
Rainer
On 03/28/2012 01:07 AM, david@lang.hm wrote:
On Tue, 27 Mar 2012, Dmitri Pal wrote:
On 03/27/2012 08:49 PM, david@lang.hm wrote:
as I am understanding it, the primary value of umberlog is that it gets us structured metadata (and more metadata) with no application changes.
I agree that we need to define additional functions to use for adding application structured data to the log.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
I think we differ in details, but not in overall strategy.
May be I am missing something but I fail to understand how changing from syslog() function to ul_syslog() is not an application change.
it could be that I am misunderstanding things, but as I understand it, if you just do a LD_PRELOAD of umberlog with no application changes the result is log messages with structured metadata around the raw log message.
If the programmer wants to add additional name/value pairs they can change individual syslog() calls to ul_syslog() and pass in parameters.
I actually don't expect many people to make a lot of use of this option, but in many cases something as simple as this is really going to be the right thing to do.
As I see it, the end result in terms of C is going to be four possible logging calls
- syslog()
traditional interface, no app changes. structured metadata with one message field.
- ul_syslog()
syslog() with some additional name/value pairs added
- template based (ELAPI or similar)
a template that lists the names, followed by the types and values
- memory structure based
a tree structure is created pointing to the names and values. a pointer to this structure is passed to the logging call.
This last one could be a variation or replacement for ul_syslog(), and I think it would be ideal if we could hash things out so that it can be just a newer variation of ul_syslog()
I think that all of these variations are appropriate to have (with the possible combination of #2 and #4)
David Lang
I see 3 & 4 as being the same thing as you can't do templates without memory structure.
Dmitri Pal dpal@redhat.com writes:
- template based (ELAPI or similar)
a template that lists the names, followed by the types and values
- memory structure based
a tree structure is created pointing to the names and values. a pointer to this structure is passed to the logging call.
This last one could be a variation or replacement for ul_syslog(), and I think it would be ideal if we could hash things out so that it can be just a newer variation of ul_syslog()
I think that all of these variations are appropriate to have (with the possible combination of #2 and #4)
David Lang
I see 3 & 4 as being the same thing as you can't do templates without memory structure.
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
On 03/28/2012 11:37 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
- template based (ELAPI or similar)
a template that lists the names, followed by the types and values
- memory structure based
a tree structure is created pointing to the names and values. a pointer to this structure is passed to the logging call.
This last one could be a variation or replacement for ul_syslog(), and I think it would be ideal if we could hash things out so that it can be just a newer variation of ul_syslog()
I think that all of these variations are appropriate to have (with the possible combination of #2 and #4)
David Lang
I see 3 & 4 as being the same thing as you can't do templates without memory structure.
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 11:37 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
the resulting logs should be the same, but I don't think that overloading the same function call for such very different approaches makes sense.
yes, you can always add enough flags and extra paramters to combine function calls, but that makes it more complicated to use any of them.
I see it more like
foo_logf("format string",type,value,type,value...)'
foo_logs( *tree);
David Lang
On 03/28/2012 03:26 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 11:37 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
the resulting logs should be the same, but I don't think that overloading the same function call for such very different approaches makes sense.
yes, you can always add enough flags and extra paramters to combine function calls, but that makes it more complicated to use any of them.
I see it more like
foo_logf("format string",type,value,type,value...)'
foo_logs( *tree);
IMO they are just parts of the same interface. From the common variant that I propose (which is more complex than each of yours) you can create two convenince wrppers of the two unctions that you propose. It is the developer choice to sue the call tha better fits his needs in a specific place in his code.
David Lang
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 03:26 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 11:37 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
the resulting logs should be the same, but I don't think that overloading the same function call for such very different approaches makes sense.
yes, you can always add enough flags and extra paramters to combine function calls, but that makes it more complicated to use any of them.
I see it more like
foo_logf("format string",type,value,type,value...)'
foo_logs( *tree);
IMO they are just parts of the same interface. From the common variant that I propose (which is more complex than each of yours) you can create two convenince wrppers of the two unctions that you propose. It is the developer choice to sue the call tha better fits his needs in a specific place in his code.
Ok, that's a reasonable way to look at it. I wouldn't ever expose the 'full' variation to the programmers, it will just scare them off, but having this inside the library and exposing the convienience wrappers makes good sense.
David Lang
On 03/28/2012 05:01 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 03:26 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 11:37 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
the resulting logs should be the same, but I don't think that overloading the same function call for such very different approaches makes sense.
yes, you can always add enough flags and extra paramters to combine function calls, but that makes it more complicated to use any of them.
I see it more like
foo_logf("format string",type,value,type,value...)'
foo_logs( *tree);
IMO they are just parts of the same interface. From the common variant that I propose (which is more complex than each of yours) you can create two convenince wrppers of the two unctions that you propose. It is the developer choice to sue the call tha better fits his needs in a specific place in his code.
Ok, that's a reasonable way to look at it. I wouldn't ever expose the 'full' variation to the programmers, it will just scare them off, but having this inside the library and exposing the convienience wrappers makes good sense.
For that there are two headers: standard like selog.h and then selog_adv.h where the scary functions like this would be exposed. This will be the place for all sorts of weird extensions that people would want to add to the library for their own use cases over time.
David Lang
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 05:01 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 03:26 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 11:37 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
the resulting logs should be the same, but I don't think that overloading the same function call for such very different approaches makes sense.
yes, you can always add enough flags and extra paramters to combine function calls, but that makes it more complicated to use any of them.
I see it more like
foo_logf("format string",type,value,type,value...)'
foo_logs( *tree);
IMO they are just parts of the same interface. From the common variant that I propose (which is more complex than each of yours) you can create two convenince wrppers of the two unctions that you propose. It is the developer choice to sue the call tha better fits his needs in a specific place in his code.
Ok, that's a reasonable way to look at it. I wouldn't ever expose the 'full' variation to the programmers, it will just scare them off, but having this inside the library and exposing the convienience wrappers makes good sense.
For that there are two headers: standard like selog.h and then selog_adv.h where the scary functions like this would be exposed. This will be the place for all sorts of weird extensions that people would want to add to the library for their own use cases over time.
the counter-argument to this approach is that the log(e); approach doesn't require a format parser and so a library that supports it but doesn't support a logf("format",type,data..) format shoudl be able to be a smaller library.
David Lang
On 03/28/2012 05:34 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 05:01 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 03:26 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 11:37 AM, Gergely Nagy wrote: > Dmitri Pal dpal@redhat.com writes: > > The difference is in the API, not in the underlying structures in > which > you store the result. > > Template-based would be something like: > foo_log ("%(message) %(whatever)", msg, whatever); > > While the same thing with 4 would look somewhat like: > foo_log ("message", STRING, msg, > "whatever", STRING, whatever, > NULL); > > The structure you end up with might very well be the same > underneath, > but the API is much different. > IMO they should be a part of the same interface
foo_log ("message", STRING, msg, "whatever", STRING, whatever, "readable_message" STRING "%(message) %(whatever)" NULL);
Or
foo_log ("readable_message" STRING "%(message) %(whatever)" "message", STRING, msg, "whatever", STRING, whatever, NULL);
Should be possible and produce the same result
the resulting logs should be the same, but I don't think that overloading the same function call for such very different approaches makes sense.
yes, you can always add enough flags and extra paramters to combine function calls, but that makes it more complicated to use any of them.
I see it more like
foo_logf("format string",type,value,type,value...)'
foo_logs( *tree);
IMO they are just parts of the same interface. From the common variant that I propose (which is more complex than each of yours) you can create two convenince wrppers of the two unctions that you propose. It is the developer choice to sue the call tha better fits his needs in a specific place in his code.
Ok, that's a reasonable way to look at it. I wouldn't ever expose the 'full' variation to the programmers, it will just scare them off, but having this inside the library and exposing the convienience wrappers makes good sense.
For that there are two headers: standard like selog.h and then selog_adv.h where the scary functions like this would be exposed. This will be the place for all sorts of weird extensions that people would want to add to the library for their own use cases over time.
the counter-argument to this approach is that the log(e); approach doesn't require a format parser and so a library that supports it but doesn't support a logf("format",type,data..) format shoudl be able to be a smaller library.
David Lang
Really not much of a gain.
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 05:34 PM, david@lang.hm wrote:
the counter-argument to this approach is that the log(e); approach doesn't require a format parser and so a library that supports it but doesn't support a logf("format",type,data..) format shoudl be able to be a smaller library.
David Lang
Really not much of a gain.
It depends on the environment and language. for a general purpose PC, not a big deal, for an embedded device (think router or access point), it may make more of a difference.
David Lang
On 03/28/2012 07:02 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 05:34 PM, david@lang.hm wrote:
the counter-argument to this approach is that the log(e); approach doesn't require a format parser and so a library that supports it but doesn't support a logf("format",type,data..) format shoudl be able to be a smaller library.
David Lang
Really not much of a gain.
It depends on the environment and language. for a general purpose PC, not a big deal, for an embedded device (think router or access point), it may make more of a difference.
David Lang
We are talking about a difference in couple KB. With the current memory sizes of GBs on devices couple KB are within a margin of error. The code that I have in mind based on collection is lighter than any C++ or Java for example and Mobile devices run Java or other object oriented languages. Though it is a valid concern it is not IMO a primary concern. You can always compile pieces out if the code is built in the right way.
On Mar 28, 2012 8:08 PM, "Dmitri Pal" dpal@redhat.com wrote:
On 03/28/2012 07:02 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 05:34 PM, david@lang.hm wrote:
the counter-argument to this approach is that the log(e); approach doesn't require a format parser and so a library that supports it but doesn't support a logf("format",type,data..) format shoudl be able to be a smaller library.
David Lang
Really not much of a gain.
It depends on the environment and language. for a general purpose PC, not a big deal, for an embedded device (think router or access point), it may make more of a difference.
David Lang
We are talking about a difference in couple KB. With the current memory sizes of GBs on devices couple KB are within a margin of error. The code that I have in mind based on collection is lighter than any C++ or Java for example and Mobile devices run Java or other object oriented languages. Though it is a valid concern it is not IMO a primary concern. You can always compile pieces out if the code is built in the right way.
-- Thank you, Dmitri Pal
Let us not get ahead of ourselves.
More libraries with differing requirements and features can and will be written. As long as the underlying event model and output formats are consistent, we should not be concerning ourselves too much with implementation specific details for specific use cases.
On Wed, 28 Mar 2012, Gergely Nagy wrote:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
actually, I was seeing the structure one being more like
e=create_structure(); various commands to add nodes to the tree pointed at by 'e' foo_log(e);
the point being that e is not just a list of name/type/value tuples, but it is a hierarchical tree structure.
David Lang
On 03/28/2012 03:22 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Gergely Nagy wrote:
The difference is in the API, not in the underlying structures in which you store the result.
Template-based would be something like: foo_log ("%(message) %(whatever)", msg, whatever);
While the same thing with 4 would look somewhat like: foo_log ("message", STRING, msg, "whatever", STRING, whatever, NULL);
The structure you end up with might very well be the same underneath, but the API is much different.
actually, I was seeing the structure one being more like
e=create_structure(); various commands to add nodes to the tree pointed at by 'e' foo_log(e);
the point being that e is not just a list of name/type/value tuples, but it is a hierarchical tree structure.
Agree but one does not preclude another.
David Lang _______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
david@lang.hm writes:
May be I am missing something but I fail to understand how changing from syslog() function to ul_syslog() is not an application change.
it could be that I am misunderstanding things, but as I understand it, if you just do a LD_PRELOAD of umberlog with no application changes the result is log messages with structured metadata around the raw log message.
That is correct. A syslog() message with umberlog preloaded will have extra metadata, and the original message in the 'msg' field.
If the programmer wants to add additional name/value pairs they can change individual syslog() calls to ul_syslog() and pass in parameters.
Yup, correct.
I actually don't expect many people to make a lot of use of this option, but in many cases something as simple as this is really going to be the right thing to do.
Agreed.
As I see it, the end result in terms of C is going to be four possible logging calls
- syslog()
traditional interface, no app changes. structured metadata with one message field.
- ul_syslog()
syslog() with some additional name/value pairs added
- template based (ELAPI or similar)
a template that lists the names, followed by the types and values
- memory structure based
a tree structure is created pointing to the names and values. a pointer to this structure is passed to the logging call.
This last one could be a variation or replacement for ul_syslog(), and I think it would be ideal if we could hash things out so that it can be just a newer variation of ul_syslog()
I think that all of these variations are appropriate to have (with the possible combination of #2 and #4)
Agreed, with the addition that the ul_syslog() API can easily be reimplemented on top of either 3 or 4, so once 3 and/or 4 is available, the ul_syslog() part of umberlog can be provided by that library too, so anyone who used ul_syslog() until that point can easily migrate without code change (or with as minimal as changing an include), just like syslog()->umberlog works now.
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard. IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
This definitely is a valid argument, but I think we need to weigh the pros and cons. Too much change in a single instant is very hard to sell. To small steps don't make sense, either.
My personal opinion (and experience) is that smaller steps work better than larger ones. There are obviously people in the absolute opposite camp. I still think that evolution works better than revolution.
So my *personal* ballot is that I am ready to except the cost and risk involved in requiring app developers to change code twice. I find it far too hard to do it all perfect in a single iteration. Evolve, see results, improve what was bad, start next iteration...
I think we are currently trying to much at once. Been there, done that, failed with it. Let's focus on small steps, get them accepted, build on that. The discussion reminds me on extreme programming vs. traditional software engineering approach... I tend to prefer quick refactoring of code build to be good enough for the job at hand...
Again, I do not go as far as to object another approach, but I don't like the other one either.
Rainer
"Rainer Gerhards" rgerhards@hq.adiscon.com writes:
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard. IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
We should take selog or like, polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
This definitely is a valid argument, but I think we need to weigh the pros and cons. Too much change in a single instant is very hard to sell. To small steps don't make sense, either.
My personal opinion (and experience) is that smaller steps work better than larger ones. There are obviously people in the absolute opposite camp. I still think that evolution works better than revolution.
There's one more thing I would like to add, regarding the discussion about handshake and how to change/replace/extend/whatever /dev/log: why care?
I don't think there must be a single interface to get logs from one application to another, nor should we care about the transportation issue, unless in context of libc (but more on that later, in another thread).
Why? Because there's no single size that fits all. We want to be able to *work with* logs, so we need a format, or representation that makes sense, is easy to work with, and is pretty much a standard. How it arrives from the producer to the consumer, is none of our business, in my opinion.
Like Rainer, I'm not against anyone trying to go down this route, but that's a route that I, personally, do not wish to tread.
On Wed, 28 Mar 2012, Gergely Nagy wrote:
There's one more thing I would like to add, regarding the discussion about handshake and how to change/replace/extend/whatever /dev/log: why care?
I don't think there must be a single interface to get logs from one application to another, nor should we care about the transportation issue, unless in context of libc (but more on that later, in another thread).
Why? Because there's no single size that fits all. We want to be able to *work with* logs, so we need a format, or representation that makes sense, is easy to work with, and is pretty much a standard. How it arrives from the producer to the consumer, is none of our business, in my opinion.
There are a few points where I believe that there needs to be widespread standardization, and the passing of log messages from applications to the system is the most important of them in my opinion.
If this is not standardized, then the result will be that some applications can only write logs to some destinations. This defeats a huge amount of the value of defining standards for logging.
looking at things from a high level, I don't care what the function calls look like that a programmer uses to generate a log message, but I care tremendously about how those functions communicate to a Dispatcher daemon (which will be based on the existing syslog daemons, at least for now since they provide so much of the desired functionality already)
I agree that there will be differing needs. As a result, I believe that there will need to be multiple protocols. (more on this below)
If there are multiple protocols possible, the standard needs to define a few things.
1. what protocols are mandatory
2. how to figure out what protocol is being used
We could just say that JSON is the only mandatory protocol, but I think we do want more than that.
We could just say that it's up to the system administrators and programmers to configure everything to know that the other side is useing, but if we do that we will have justly earned the hate of system administrators for life (and beyond), especially if changing the sending protocol requires changes to the openlog call in the application.
I'm not saying that we have to define all the possible protocols now, but we have some ideas of a few that we want to do, so we should start with them and think about how we would add others.
As for what protocols are desirable, we have
JSON simple, human readable, works without any changes to existing log mechanisms
XML human readable, supports data types, message types, strong mindshare in the fields of structured data. many existing tools
BSON relativly simple, more efficient than human readable text in some cases, supports data types
David Lang
On 03/28/2012 03:45 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Gergely Nagy wrote:
There's one more thing I would like to add, regarding the discussion about handshake and how to change/replace/extend/whatever /dev/log: why care?
I don't think there must be a single interface to get logs from one application to another, nor should we care about the transportation issue, unless in context of libc (but more on that later, in another thread).
Why? Because there's no single size that fits all. We want to be able to *work with* logs, so we need a format, or representation that makes sense, is easy to work with, and is pretty much a standard. How it arrives from the producer to the consumer, is none of our business, in my opinion.
There are a few points where I believe that there needs to be widespread standardization, and the passing of log messages from applications to the system is the most important of them in my opinion.
If this is not standardized, then the result will be that some applications can only write logs to some destinations. This defeats a huge amount of the value of defining standards for logging.
looking at things from a high level, I don't care what the function calls look like that a programmer uses to generate a log message, but I care tremendously about how those functions communicate to a Dispatcher daemon (which will be based on the existing syslog daemons, at least for now since they provide so much of the desired functionality already)
I agree that there will be differing needs. As a result, I believe that there will need to be multiple protocols. (more on this below)
If there are multiple protocols possible, the standard needs to define a few things.
what protocols are mandatory
how to figure out what protocol is being used
We could just say that JSON is the only mandatory protocol, but I think we do want more than that.
We could just say that it's up to the system administrators and programmers to configure everything to know that the other side is useing, but if we do that we will have justly earned the hate of system administrators for life (and beyond), especially if changing the sending protocol requires changes to the openlog call in the application.
I'm not saying that we have to define all the possible protocols now, but we have some ideas of a few that we want to do, so we should start with them and think about how we would add others.
As for what protocols are desirable, we have
JSON simple, human readable, works without any changes to existing log mechanisms
XML human readable, supports data types, message types, strong mindshare in the fields of structured data. many existing tools
BSON relativly simple, more efficient than human readable text in some cases, supports data types
I think I agree with you more than with the rest. But IMO the standardization of the top level API much more important as it affects much more people. You are bringing right points and I agree with you but it is easier to evolve under the API that the API itself.
David Lang _______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 03:45 PM, david@lang.hm wrote:
I agree that there will be differing needs. As a result, I believe that there will need to be multiple protocols. (more on this below)
If there are multiple protocols possible, the standard needs to define a few things.
what protocols are mandatory
how to figure out what protocol is being used
We could just say that JSON is the only mandatory protocol, but I think we do want more than that.
We could just say that it's up to the system administrators and programmers to configure everything to know that the other side is useing, but if we do that we will have justly earned the hate of system administrators for life (and beyond), especially if changing the sending protocol requires changes to the openlog call in the application.
I'm not saying that we have to define all the possible protocols now, but we have some ideas of a few that we want to do, so we should start with them and think about how we would add others.
As for what protocols are desirable, we have
JSON simple, human readable, works without any changes to existing log mechanisms
XML human readable, supports data types, message types, strong mindshare in the fields of structured data. many existing tools
BSON relativly simple, more efficient than human readable text in some cases, supports data types
I think I agree with you more than with the rest. But IMO the standardization of the top level API much more important as it affects much more people. You are bringing right points and I agree with you but it is easier to evolve under the API that the API itself.
Here I disagree with you a bit.
I fully expect that there will be many different APIs available. Many of them may end up being implemented as convienience functions wrapped around a more complex API, but from the programmers point of view that doesn't matter. Different languages will have different APIs as well.
What needs to be standardized is the communication between the different components. Define that and everything else can evolve and the resulting ecosystem continues to work.
Even the communication can evolve, but the protocols need to be built with evolution and backwards compatibility in mind.
David Lang
On 03/28/2012 05:06 PM, david@lang.hm wrote:
On Wed, 28 Mar 2012, Dmitri Pal wrote:
On 03/28/2012 03:45 PM, david@lang.hm wrote:
I agree that there will be differing needs. As a result, I believe that there will need to be multiple protocols. (more on this below)
If there are multiple protocols possible, the standard needs to define a few things.
what protocols are mandatory
how to figure out what protocol is being used
We could just say that JSON is the only mandatory protocol, but I think we do want more than that.
We could just say that it's up to the system administrators and programmers to configure everything to know that the other side is useing, but if we do that we will have justly earned the hate of system administrators for life (and beyond), especially if changing the sending protocol requires changes to the openlog call in the application.
I'm not saying that we have to define all the possible protocols now, but we have some ideas of a few that we want to do, so we should start with them and think about how we would add others.
As for what protocols are desirable, we have
JSON simple, human readable, works without any changes to existing log mechanisms
XML human readable, supports data types, message types, strong mindshare in the fields of structured data. many existing tools
BSON relativly simple, more efficient than human readable text in some cases, supports data types
I think I agree with you more than with the rest. But IMO the standardization of the top level API much more important as it affects much more people. You are bringing right points and I agree with you but it is easier to evolve under the API that the API itself.
Here I disagree with you a bit.
I fully expect that there will be many different APIs available. Many of them may end up being implemented as convienience functions wrapped around a more complex API, but from the programmers point of view that doesn't matter. Different languages will have different APIs as well.
What needs to be standardized is the communication between the different components. Define that and everything else can evolve and the resulting ecosystem continues to work.
Even the communication can evolve, but the protocols need to be built with evolution and backwards compatibility in mind.
David Lang
We need to start somewhere. At this point I see enough support of the selog.h effort to start cranking code when I can. I hope this would switch discussion from a theoretical into a more concrete one.
Dmitri Pal dpal@redhat.com writes:
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard.
I don't see it that way. First of all, umberlog can be entirely skipped, one can go - and in my opinion, should - go straight to a more flexible solution.
Second, the main purpose of umberlog is the LD_PRELOAD thing. That it has a few extras, is a side-effect, which I personally find convenient. But the main reason for the library's existence is its ability to change how syslog() works, with neither source changes, nor recompilation.
It works out of the box, it's simple, it's available already. That's what umberlog is about.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
That's a commendable goal, but it takes time, and coming up with an interface that most like, is non-trivial at best. That also does not help legacy applications..
Consider the scenario, where we have an amazing library that makes structured logging the most intuitive thing ever, after the nipple, and more and more applications begin to embrace and use it. But! There are still legacy applications out there that use plain old syslog(). We gotta do something about those, too, otherwise it becomes a bit of a pain to process both kind of logs.
Umberlog fills that void. It's a help for legacy apps first and foremost.
Second, the main purpose of umberlog is the LD_PRELOAD thing. That it has a few extras, is a side-effect, which I personally find convenient. But the main reason for the library's existence is its ability to change how syslog() works, with neither source changes, nor recompilation.
Is it really? I thought the idea was initiated when we discussed providing the ability to an app to pass name/value pairs, something that the existing syslog api cannot do. The addition of metadata nv pairs, IIRC, came at a somewhat later stage.
I don't see much value in these metadata over SCM_CREDENTIALS-obtained metadata. I know umberlog can do this without a syslogd, but with the major implementations participating in this effort is that really a big point?
So IMHO the extended ul_syslog() API is the really important part.
Rainer
"Rainer Gerhards" rgerhards@hq.adiscon.com writes:
Second, the main purpose of umberlog is the LD_PRELOAD thing. That it has a few extras, is a side-effect, which I personally find convenient. But the main reason for the library's existence is its ability to change how syslog() works, with neither source changes, nor recompilation.
Is it really? I thought the idea was initiated when we discussed providing the ability to an app to pass name/value pairs, something that the existing syslog api cannot do. The addition of metadata nv pairs, IIRC, came at a somewhat later stage.
Well.. the original plan was to extend syslog() with the ability to add nv-pairs, while remaining backwards compatible. When that failed, ul_syslog() came, and syslog() became a @cee{"message":...}. The came the extra metadata.
But the original plan was to make syslog() emit JSON, for legacy apps. Secondary to that, to add an API (that does require code change) to add nvpairs. Sadly, that could only be done with a separate function.
I don't see much value in these metadata over SCM_CREDENTIALS-obtained metadata. I know umberlog can do this without a syslogd, but with the major implementations participating in this effort is that really a big point?
Nope, it isn't. But turning legacy apps syslog() messages into a JSON-payloaded thing is already a win, imo, as the syslogds don't have to go to extra lengths to accept both legacy and @cee logs.
The metadata is a nice bonus, but it's just that, a bonus.
So IMHO the extended ul_syslog() API is the really important part.
It's not negligible, but that can be provided by something else, too (which can do better in that regard). Legacy syslog() => structured thingy is somthing that's better done in a thin lib than in something a lot more complicated.
-----Original Message----- From: lumberjack-developers-bounces@lists.fedorahosted.org [mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf Of Gergely Nagy Sent: Wednesday, March 28, 2012 1:56 PM To: lumberjack-developers@lists.fedorahosted.org Subject: Re: [lumberjack] connection to syslog (or equivalent)
"Rainer Gerhards" rgerhards@hq.adiscon.com writes:
Second, the main purpose of umberlog is the LD_PRELOAD thing. That
it
has a few extras, is a side-effect, which I personally find convenient. But the main reason for the library's existence is its ability to change how syslog() works, with neither source changes,
nor
recompilation.
Is it really? I thought the idea was initiated when we discussed
providing
the ability to an app to pass name/value pairs, something that the
existing
syslog api cannot do. The addition of metadata nv pairs, IIRC, came
at a
somewhat later stage.
Well.. the original plan was to extend syslog() with the ability to add nv-pairs, while remaining backwards compatible.
Yes, but... As you say, the plan was to extend syslog(). So in order to make use of that extension, app changes would have been necessary. Right? So even if the name syslog() had remained, apps would have to be upgraded to emit "real" nv pairs.
In that sense, I do not see any difference between syslog(,,nam1,val1,...,nameN,valN,NULL) and ul_syslog().
The core intent was to make passing nv pairs *from the app* possible.
When that failed, ul_syslog() came, and syslog() became a @cee{"message":...}. The came the extra metadata.
But the original plan was to make syslog() emit JSON, for legacy apps Secondary to that, to add an API (that does require code change) to add nvpairs. Sadly, that could only be done with a separate function.
I don't see much value in these metadata over SCM_CREDENTIALS-
obtained
metadata. I know umberlog can do this without a syslogd, but with the
major
implementations participating in this effort is that really a big point?
Nope, it isn't. But turning legacy apps syslog() messages into a JSON-payloaded thing is already a win, imo, as the syslogds don't have to go to extra lengths to accept both legacy and @cee logs.
Where ist he win? Besides, we always must process legacy logs as they may come in via the network. And cee-enhanced syslog also comes in via the network. So cookie check and appropriate action is required in any way, isn’t it?
The metadata is a nice bonus, but it's just that, a bonus.
So IMHO the extended ul_syslog() API is the really important part.
It's not negligible, but that can be provided by something else, too (which can do better in that regard).
Sure, but to *me* this capability is what I am really after. I think we would have won almost nothing if we end up with just metadata (that we can obtain anyways) being part of JSON - and nothing else.
That's NOT what *I* envisioned when we talked about the API on the teleconference.
Rainer
Legacy syslog() => structured thingy is somthing that's better done in a thin lib than in something a lot more complicated.
-- |8]
lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
"Rainer Gerhards" rgerhards@hq.adiscon.com writes:
Well.. the original plan was to extend syslog() with the ability to add nv-pairs, while remaining backwards compatible.
Yes, but... As you say, the plan was to extend syslog(). So in order to make use of that extension, app changes would have been necessary. Right? So even if the name syslog() had remained, apps would have to be upgraded to emit "real" nv pairs.
Correct.
In that sense, I do not see any difference between syslog(,,nam1,val1,...,nameN,valN,NULL) and ul_syslog().
That is also correct.
The core intent was to make passing nv pairs *from the app* possible.
..while maintaining backwards compatibility with legacy syslog(), which still is the main selling point of umberlog, imo. The rest is cool and useful, but imo less so.
If you like and think of other features as more important, that's also fine! I find them useful aswell, just our priorities seem to differ. The end result is pretty much the same =)
Nope, it isn't. But turning legacy apps syslog() messages into a JSON-payloaded thing is already a win, imo, as the syslogds don't have to go to extra lengths to accept both legacy and @cee logs.
Where ist he win? Besides, we always must process legacy logs as they may come in via the network. And cee-enhanced syslog also comes in via the network. So cookie check and appropriate action is required in any way, isn’t it?
Not if you LD_PRELOAD the lib on the whole network (which I've done on my systems). In that case, you can get rid of the cookie check.
For me, that's a win, because I can just shoehorn everything into Solr pretty much as-is, legacy logs and properly structured logs alike. My config became simpler, just by LD_PRELOADing a lib.
The metadata is a nice bonus, but it's just that, a bonus.
So IMHO the extended ul_syslog() API is the really important part.
It's not negligible, but that can be provided by something else, too (which can do better in that regard).
Sure, but to *me* this capability is what I am really after. I think we would have won almost nothing if we end up with just metadata (that we can obtain anyways) being part of JSON - and nothing else.
That's NOT what *I* envisioned when we talked about the API on the teleconference.
Lovely! Then we hit two birds with the same stone, and we won't even have to argue about which one each of us eats, as we both choose a different one! :D
-----Original Message----- From: lumberjack-developers-bounces@lists.fedorahosted.org [mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf Of Gergely Nagy Sent: Wednesday, March 28, 2012 5:49 PM To: lumberjack-developers@lists.fedorahosted.org Subject: Re: [lumberjack] connection to syslog (or equivalent)
"Rainer Gerhards" rgerhards@hq.adiscon.com writes:
Well.. the original plan was to extend syslog() with the ability to
add
nv-pairs, while remaining backwards compatible.
Yes, but... As you say, the plan was to extend syslog(). So in order
to make
use of that extension, app changes would have been necessary. Right?
So even
if the name syslog() had remained, apps would have to be upgraded to
emit
"real" nv pairs.
Correct.
In that sense, I do not see any difference between syslog(,,nam1,val1,...,nameN,valN,NULL) and ul_syslog().
That is also correct.
The core intent was to make passing nv pairs *from the app* possible.
..while maintaining backwards compatibility with legacy syslog(), which still is the main selling point of umberlog, imo. The rest is cool and useful, but imo less so.
If you like and think of other features as more important, that's also fine! I find them useful aswell, just our priorities seem to differ. The end result is pretty much the same =)
Nope, it isn't. But turning legacy apps syslog() messages into a JSON-payloaded thing is already a win, imo, as the syslogds don't
have
to go to extra lengths to accept both legacy and @cee logs.
Where ist he win? Besides, we always must process legacy logs as they
may
come in via the network. And cee-enhanced syslog also comes in via
the
network. So cookie check and appropriate action is required in any
way, isn’t
it?
Not if you LD_PRELOAD the lib on the whole network (which I've done on my systems). In that case, you can get rid of the cookie check.
I know you have Zorp, but... what if there is a single PIX, Cisco Switch or other gear? ;)
For me, that's a win, because I can just shoehorn everything into Solr pretty much as-is, legacy logs and properly structured logs alike. My config became simpler, just by LD_PRELOADing a lib.
The metadata is a nice bonus, but it's just that, a bonus.
So IMHO the extended ul_syslog() API is the really important part.
It's not negligible, but that can be provided by something else, too (which can do better in that regard).
Sure, but to *me* this capability is what I am really after. I think
we would
have won almost nothing if we end up with just metadata (that we can
obtain
anyways) being part of JSON - and nothing else.
That's NOT what *I* envisioned when we talked about the API on the teleconference.
Lovely! Then we hit two birds with the same stone, and we won't even have to argue about which one each of us eats, as we both choose a different one! :D
Jup, can perfectly live with that. But I have to admit had I correctly understood the intent in the first place, I'd probably not withdrawn my initial idea. Anyhow, umberlog still has everything that's needed, we just lock from a different angle. But depending on how it is sold, it will probably not achieve what *I* had hoped for, that is being real structured app-generated logs (vs. metadata, which I already have).
Nevertheless, still very useful work and thanks for putting the effort into it :-)
Rainer
On Wed, 28 Mar 2012, Gergely Nagy wrote:
"Rainer Gerhards" rgerhards@hq.adiscon.com writes:
Nope, it isn't. But turning legacy apps syslog() messages into a JSON-payloaded thing is already a win, imo, as the syslogds don't have to go to extra lengths to accept both legacy and @cee logs.
Where ist he win? Besides, we always must process legacy logs as they may come in via the network. And cee-enhanced syslog also comes in via the network. So cookie check and appropriate action is required in any way, isn?t it?
Not if you LD_PRELOAD the lib on the whole network (which I've done on my systems). In that case, you can get rid of the cookie check.
you can't LD_PRELOAD on a Cisco switch.
but that said, you can configure your syslog server in such a way that the non-structured sources go to a different IP/port than your structured sources, and then have the one syslog server change them to all be structured before they go to processing.
David Lang
On 03/28/2012 06:39 AM, Gergely Nagy wrote:
Dmitri Pal dpal@redhat.com writes:
With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard.
I don't see it that way. First of all, umberlog can be entirely skipped, one can go - and in my opinion, should - go straight to a more flexible solution.
Second, the main purpose of umberlog is the LD_PRELOAD thing. That it has a few extras, is a side-effect, which I personally find convenient. But the main reason for the library's existence is its ability to change how syslog() works, with neither source changes, nor recompilation.
It works out of the box, it's simple, it's available already. That's what umberlog is about.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.
That's a commendable goal, but it takes time, and coming up with an interface that most like, is non-trivial at best. That also does not help legacy applications..
Consider the scenario, where we have an amazing library that makes structured logging the most intuitive thing ever, after the nipple, and more and more applications begin to embrace and use it. But! There are still legacy applications out there that use plain old syslog(). We gotta do something about those, too, otherwise it becomes a bit of a pain to process both kind of logs.
Umberlog fills that void. It's a help for legacy apps first and foremost.
OK this statement even more prooves that we need to move on both fronts in parallel. And deliver two options at the same time or close so that people have a choice and leap if they want to bite a bullet or take a slow approach if it fits their needs.
Dmitri Pal dpal@redhat.com writes:
OK this statement even more prooves that we need to move on both fronts in parallel. And deliver two options at the same time or close so that people have a choice and leap if they want to bite a bullet or take a slow approach if it fits their needs.
Indeed.
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
Rainer
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
I agree that we need to not break any existing applications. What I am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications. They will never try to read anything from it.
So I was thinking that if the syslog daemon writes one string to each thing that connects to it, existing software will never read this string and everything will work as-is, new software could read the string and adjust it's behavior accordingly.
This isn't the only way to support being able to talk multiple protocols to the syslog daemon. Other options are
1. different interfaces for legacy syslog() compatible messages vs new messages (/dev/log vs /dev/slog)
2. depend on the syslog server auto-detecting the type of message
3. depend on the sysadmin and programmers to configure everything to talk the same protocol.
I happen to think that the the long-term benifits of a handshake are worth the up-front costs of defining it.
David Lang
On Wed, 28 Mar 2012 12:57:06 -0700 (PDT) david@lang.hm wrote:
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
I agree that we need to not break any existing applications. What I am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications. They will never try to read anything from it.
This breaks assumptions, notably that the original protocol is unidirectional. The writer is not prepared to receive data, and if it does, weird things may happen. The writer may conclude that the syslog server has gone berserk. On the other hand if the unix domain socket is SOCK_STREAM and the data (capability packet) is not read by the client, the socket buffer can get full and the write may block, stalling the syslogd.
There are so many unforeseen problems to this approach that it's best not to try hacking it. There should be a new interface, like we said earlier.
Another reason why I think it is better to create another interface and a new protocol is that the current syslog encapsulated JSON passing over /dev/log is pretty inefficient. To pass the structured data over to the syslogd locally, some other format (binary) should be used. Having syslogd parse the locally received syslog+JSON is pretty inefficient.
Regards, Botond
On Wed, 28 Mar 2012, Botond Botyanszki wrote:
On Wed, 28 Mar 2012 12:57:06 -0700 (PDT) david@lang.hm wrote:
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
I agree that we need to not break any existing applications. What I am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications. They will never try to read anything from it.
This breaks assumptions, notably that the original protocol is unidirectional. The writer is not prepared to receive data, and if it does, weird things may happen. The writer may conclude that the syslog server has gone berserk.
I guess I am not seeing how the writer would detect that there is data available to be read.
I am also expecting that 99.999+% of writers will be doing so via the glibc syslog() call, so if it behaves sanely there is very little to worry about.
On the other hand if the unix domain socket is SOCK_STREAM and the data (capability packet) is not read by the client, the socket buffer can get full and the write may block, stalling the syslogd.
I agree that this could be a problem, but this is why I specified a one-time short write. The expectation being that in the short term when most writers are traditional syslog, this message will never be read and will sit in the socket buffer until it is closed.
There are so many unforeseen problems to this approach that it's best not to try hacking it. There should be a new interface, like we said earlier.
Another reason why I think it is better to create another interface and a new protocol is that the current syslog encapsulated JSON passing over /dev/log is pretty inefficient. To pass the structured data over to the syslogd locally, some other format (binary) should be used. Having syslogd parse the locally received syslog+JSON is pretty inefficient.
I agree that detecting it at parse time is inefficient, and I agree that a binary protocol would be better, but a binary protocol is no longer JSON.
I don't think that pointing structured logs at /bin/slog instead of /bin/log is a good idea because /bin/slog won't exist on systems that don't have an 'advanced' transport, and in those cases plain JSON to /dev/log is the right thing to do.
also, putting structured logs through /dev/slog instead of /dev/log means a change in openlog, which I don't think is a great idea.
David Lang
----- Original message -----
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
I agree that we need to not break any existing applications. What I am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications. They will never try to read anything from it.
So I was thinking that if the syslog daemon writes one string to each thing that connects to it, existing software will never read this string and everything will work as-is, new software could read the string and adjust it's behavior accordingly.
/dev/log is currently a SOCK_DGRAM socket, no need to connect, and no need to accept, only sendto and recvfrom.
Earlier, it used to be SOCK_STREAM, but that was 10 years ago.
On Thu, 29 Mar 2012, Balazs Scheidler wrote:
----- Original message -----
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
I agree that we need to not break any existing applications. What I am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications. They will never try to read anything from it.
So I was thinking that if the syslog daemon writes one string to each thing that connects to it, existing software will never read this string and everything will work as-is, new software could read the string and adjust it's behavior accordingly.
/dev/log is currently a SOCK_DGRAM socket, no need to connect, and no need to accept, only sendto and recvfrom.
Earlier, it used to be SOCK_STREAM, but that was 10 years ago.
so I think that this means that there can only be one reader, all data written is seen by that one reader.
Is this correct? (if so, it kills my capabilities idea)
David Lang
----- Original message -----
On Thu, 29 Mar 2012, Balazs Scheidler wrote:
----- Original message -----
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
I am strongly against changing the existing interface. This will break so many things that nobody will use it...
I agree that we need to not break any existing applications. What I am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications. They will never try to read anything from it.
So I was thinking that if the syslog daemon writes one string to each thing that connects to it, existing software will never read this string and everything will work as-is, new software could read the string and adjust it's behavior accordingly.
/dev/log is currently a SOCK_DGRAM socket, no need to connect, and no need to accept, only sendto and recvfrom.
Earlier, it used to be SOCK_STREAM, but that was 10 years ago.
so I think that this means that there can only be one reader, all data written is seen by that one reader.
Is this correct? (if so, it kills my capabilities idea)
yeah, pretty much. senders have an anonymous address automatically created by the kernel. syslogd could respond to that address, but not before the first datagram.
it's about the same as udp. there's only one process that can bind to a given port, and although it is possible to create a more specific 'connection' by using SO_REUSEADDR, then bind() and then connect() to the client address, you have to know the client port in advance, which is automatically allocated.
-----Original Message----- From: lumberjack-developers-bounces@lists.fedorahosted.org [mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf Of Balazs Scheidler Sent: Friday, March 30, 2012 6:43 AM To: david@lang.hm Cc: lumberjack logging Subject: Re: [lumberjack] connection to syslog (or equivalent)
----- Original message -----
On Thu, 29 Mar 2012, Balazs Scheidler wrote:
----- Original message -----
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it
it
might be best to redo it entirely.
I am strongly against changing the existing interface. This
will
break so many things that nobody will use it...
I agree that we need to not break any existing applications. What
I
am hoping for is that we can extend the interface in a way that's fully backwards compatible.
In this case, /dev/log is write-only for existing applications.
They
will never try to read anything from it.
So I was thinking that if the syslog daemon writes one string to
each
thing that connects to it, existing software will never read this string and everything will work as-is, new software could read
the
string and adjust it's behavior accordingly.
/dev/log is currently a SOCK_DGRAM socket, no need to connect, and
no
need to accept, only sendto and recvfrom.
Earlier, it used to be SOCK_STREAM, but that was 10 years ago.
so I think that this means that there can only be one reader, all
data
written is seen by that one reader.
Is this correct? (if so, it kills my capabilities idea)
yeah, pretty much. senders have an anonymous address automatically created by the kernel. syslogd could respond to that address, but not before the first datagram.
it's about the same as udp. there's only one process that can bind to a given port, and although it is possible to create a more specific 'connection' by using SO_REUSEADDR, then bind() and then connect() to the client address, you have to know the client port in advance, which is automatically allocated.
For the capabilities idea, a new local protocol (stream based) needs to be defined. Sorry, I assumed that was part of the suggestion.
Rainer
On Thu, 29 Mar 2012 23:26:03 +0200 Balazs Scheidler bazsi77@gmail.com wrote:
/dev/log is currently a SOCK_DGRAM socket, no need to connect, and no need to accept, only sendto and recvfrom.
On Linux and BSD. How about the rest? Solaris /dev/log is a streams device and afair HPUX was also something special.
----- Original message -----
On Thu, 29 Mar 2012 23:26:03 +0200 Balazs Scheidler bazsi77@gmail.com wrote:
/dev/log is currently a SOCK_DGRAM socket, no need to connect, and no need to accept, only sendto and recvfrom.
On Linux and BSD. How about the rest? Solaris /dev/log is a streams device and afair HPUX was also something special.
solaris has a STREAMS device augmented with a door. HP-UX has a named pipe, AIX has a socket, Tru64 is a pipe or also a socket.
Some Linux platforms might still use SOCK_STREAM.
On 03/27/2012 04:17 PM, david@lang.hm wrote:
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that nobody replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with the only feedback being that the write could block.
With the new formats and new syslog replacement library, does there need to be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no changes (which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to have a way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they will never send structured logs at all?
I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
+1 For now we just need one call to get the capabilities. It is a local call and has nothing to do with network protocols.
David Lang _______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
For now we just need one call to get the capabilities. It is a local call and has nothing to do with network protocols.
If that's the main desire, so why not define a "capabilities file" that the syslogd writes on startup and that the lib can access? That's the simplest possible interface and, yes, it is imperfect...
Rainer
On 03/28/2012 05:02 AM, Rainer Gerhards wrote:
For now we just need one call to get the capabilities. It is a local call and has nothing to do with network protocols.
If that's the main desire, so why not define a "capabilities file" that the syslogd writes on startup and that the lib can access? That's the simplest possible interface and, yes, it is imperfect...
Rainer
That would work too.
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
For now we just need one call to get the capabilities. It is a local call and has nothing to do with network protocols.
If that's the main desire, so why not define a "capabilities file" that the syslogd writes on startup and that the lib can access? That's the simplest possible interface and, yes, it is imperfect...
It will work in many cases, but as you say it's imperfect. Is there a better option?
David Lang
-----Original Message----- From: lumberjack-developers-bounces@lists.fedorahosted.org [mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf Of david@lang.hm Sent: Tuesday, March 27, 2012 10:17 PM To: lumberjack logging Subject: Re: [lumberjack] connection to syslog (or equivalent)
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that
nobody
replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface,
with the
only feedback being that the write could block.
With the new formats and new syslog replacement library, does there
need to
be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no
changes
(which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need
to have a
way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a
new
logging interface, probably maintaining at least minimal
compatibility
with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
But does this mean that unless applications are re-written, they will never send structured logs at all?
I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
Well, I think if you want just this, it can be obtained right now without een a new lib or so. Actually, this is the focus of liblognorm and syslog-ng's patterndb, to extract nv pairs from semistructured data. I think this *is* pretty important stuff. Other metadata you can get via SCM_CREDENTIALS (and I still think this is the preferred way from a trust perspective).
Rainer
On Wed, 28 Mar 2012, Rainer Gerhards wrote:
[mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf
I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
Well, I think if you want just this, it can be obtained right now without een a new lib or so. Actually, this is the focus of liblognorm and syslog-ng's patterndb, to extract nv pairs from semistructured data. I think this *is* pretty important stuff. Other metadata you can get via SCM_CREDENTIALS (and I still think this is the preferred way from a trust perspective).
First off, I agree with you about SCM_CREDENTIALS
other than that, trying to guess the structure after the fact, especially with unstructured log messages from whatever source, is significantly harder to do right all the time than dealing with structured data.
just look at all the guesswork (and how frequently it goes wrong) that is in the syslog parser to figure out the date, hostname, and syslogtag fields. If we can avoid that with a library change, I think it's well worth it.
David Lang
On 03/27/2012 04:09 PM, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that nobody replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with the only feedback being that the write could block.
With the new formats and new syslog replacement library, does there need to be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no changes (which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to have a way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
I do not see a reason why we can't start defining it now as we see a need. This is not all or nothing, we can grow a new syslog interface gradually with the ELAPI/selog on top being able to gradually take advantage of the new interfaces as they emerge. But to do this the capability call is essential.
lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
-----Original Message----- From: lumberjack-developers-bounces@lists.fedorahosted.org [mailto:lumberjack-developers-bounces@lists.fedorahosted.org] On Behalf Of William Heinbockel Sent: Tuesday, March 27, 2012 10:09 PM To: lumberjack logging Subject: Re: [lumberjack] connection to syslog (or equivalent)
On Tue, Mar 27, 2012 at 3:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that
nobody
replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface,
with the
only feedback being that the write could block.
With the new formats and new syslog replacement library, does there
need to
be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no
changes
(which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to
have a
way to tell what format to use when writing the log.
I don't think this is right approach. The end goal is to support structured log messages. The first step is to introduce the capability to sent structured events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would introduce a new logging interface, probably maintaining at least minimal compatibility with the syslog daemon (not the syslog API or syslog protocol). This would allow us to define and support more feature-rich log protocols and other features.
I am with Botond, Bazsi and Bill here: let's keep it simplex for now. Duplex syslog is not really standardized yet and requires a new protocol/effort. IMHO the main point why all previous tries at standardizing structured logging failed was that they were to ambituous. Let's go for the low-hanging fruit, get experience and do the rest in the next iteration. If we do right this time, I guess there will be enough momentum left, based on success, for the next iteration. Evolve, not revolve ;)
My 2 cts...
Rainer
On 03/27/2012 03:18 PM, david@lang.hm wrote:
I posted some of this at the bottom of a long message yesterday that nobody replied to, so I broke it out to a separate thread
The traditional syslog interface has been a write-only interface, with the only feedback being that the write could block.
With the new formats and new syslog replacement library, does there need to be some rethinking of this interface?
JSON can be pushed through the traditional syslog interface with no changes (which is one of the attractions of using JSON), but when we support additional formats, especially ones with type information, we need to have a way to tell what format to use when writing the log.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
This kind of handshake is exactly what I was talking about in the other thread. I do not know the syslog internal well enough to propose the right solution. I was thinking about a special call exposed by the syslog implementation that would return the capability but I am open to anything that makes sense.
David Lang
lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
lumberjack-developers@lists.fedorahosted.org
-
Balazs Scheidler -
Botond Botyanszki -
David Lang -
Dmitri Pal -
Gergely Nagy -
Rainer Gerhards -
William Heinbockel