Handling of "trusted" properties coming from untrusted inputs

[Miloslav Trmač]

Hello,
while working on the Lumberjack message processing "pipeline", a design question
has come up that probably must be a known issue for logging in general, so I'd like to
ask before we start implementing.

What to do about "trusted" properties coming from untrusted input sources?

In particular, we are aiming at an rsyslog configuration where anything passed to /dev/log
is automatically annotated with "pid", "uid" and other fields; at that
point the information is 100% reliable.

Similarly, syslog messages can come from remote hosts over TCP, and if those hosts are
similarly configured by the same administrators, the data is reliable as well.

OTOH, if syslog messages come from hosts that are not under the same control (e.g. data
coming from user-administered machines), this data is not "reliable" - what to
do about it?


We have come up with three possibilities:

a) Just leave it in, and let the user filter the data out based on other properties (e.g.
host name) at the time of log searching (or just let the user notice when reading the log
message ad hoc) - some queries and statistics may have misleading results.

b) Delete the untrusted fields - drops data.

c) Somehow mark the fields as "untrusted" - preserves all data, and allows
queries that ignore it.
   c1) Is it any really any better than just filtering on host names as in a)?
   c2) How can we mark the fields as "untrusted", in particular in the context
of Lumberjack?  


I'm leaning towards a) as the UNIX-tradition "worse is better" solution, but
I really don't have that much experience in logging, so any feedback would be
appreciated.
    Mirek

[William Heinbockel]

I will toss in my $0.02:

There is nothing you can or should do (or your option (a))

If we have logs written to files and transmitted over a network unprotected,
placing any reliance on any of the values is absurd

Until we have integrity protected (signed) event logs and event streams, I do
not see any value in distinguishing "trusted" properties or even
"untrusted"
sources. If/when we do ever get signed events, we will probably have better
alternatives to reflect trust

On 08/21/2012 03:16 PM, Miloslav Trmac wrote:
>  Hello,
>  while working on the Lumberjack message processing "pipeline", a design
> question has come up that probably must be a known issue for logging in general, so
> I'd like to ask before we start implementing.
>  
>  What to do about "trusted" properties coming from untrusted input sources?
>  
>  In particular, we are aiming at an rsyslog configuration where anything passed to
> /dev/log is automatically annotated with "pid", "uid" and other
> fields; at that point the information is 100% reliable.
>  
>  Similarly, syslog messages can come from remote hosts over TCP, and if those hosts are
> similarly configured by the same administrators, the data is reliable as well.
>  
>  OTOH, if syslog messages come from hosts that are not under the same control (e.g. data
> coming from user-administered machines), this data is not "reliable" - what to
> do about it?
>  
>  
>  We have come up with three possibilities:
>  
>  a) Just leave it in, and let the user filter the data out based on other properties (e.g.
> host name) at the time of log searching (or just let the user notice when reading the log
> message ad hoc) - some queries and statistics may have misleading results.
>  
>  b) Delete the untrusted fields - drops data.
>  
>  c) Somehow mark the fields as "untrusted" - preserves all data, and allows
> queries that ignore it.
>     c1) Is it any really any better than just filtering on host names as in a)?
>     c2) How can we mark the fields as "untrusted", in particular in the context
> of Lumberjack?  
>  
>  
>  I'm leaning towards a) as the UNIX-tradition "worse is better" solution,
> but I really don't have that much experience in logging, so any feedback would be
> appreciated.
>      Mirek
>  _______________________________________________
>  lumberjack-developers mailing list
>  lumberjack-developers(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/lumberjack-developers
>   
-- 
William Heinbockel
Product Security Team, Red Hat
heinbockel(a)redhat.com

[David Lang]

On Tue, 21 Aug 2012, Miloslav Trmac wrote:

>  Hello,
>  while working on the Lumberjack message processing "pipeline", a design
> question has come up that probably must be a known issue for logging in general, so
> I'd like to ask before we start implementing.
>
>  What to do about "trusted" properties coming from untrusted input sources?
>
>  In particular, we are aiming at an rsyslog configuration where anything passed to
> /dev/log is automatically annotated with "pid", "uid" and other
> fields; at that point the information is 100% reliable.
>
>  Similarly, syslog messages can come from remote hosts over TCP, and if those hosts are
> similarly configured by the same administrators, the data is reliable as well.
>
>  OTOH, if syslog messages come from hosts that are not under the same control (e.g. data
> coming from user-administered machines), this data is not "reliable" - what to
> do about it? 
how do you know if the remote hosts are not under the same control?

how do you know that the remote hosts connecting to you via TCP really are 
who you think they are? (how do you know they haven't hijacked the IP 
address for example)

how do you know the machine hasn't been comprimised and all the data it 
sends is false?

I could go on.

At the software level, you really don't know these things

This is up to the sysadmins of that network to decide.

They could setup a filter based on the source IP to then write files with 
a different template (rsyslog could really use the ability for a filter to 
modify properties, but it doesn't currently have this ability) or to drop 
the message.

so this is really a case-by-case decision to be made, and with the 
exception of modifying the properties, rsyslog already gives you the 
ability to implement whatever choice you want.

David Lang

>  We have come up with three possibilities:
>
>  a) Just leave it in, and let the user filter the data out based on other properties (e.g.
> host name) at the time of log searching (or just let the user notice when reading the log
> message ad hoc) - some queries and statistics may have misleading results.
>
>  b) Delete the untrusted fields - drops data.
>
>  c) Somehow mark the fields as "untrusted" - preserves all data, and allows
> queries that ignore it.
>    c1) Is it any really any better than just filtering on host names as in a)?
>    c2) How can we mark the fields as "untrusted", in particular in the context
> of Lumberjack? 
>
>
>  I'm leaning towards a) as the UNIX-tradition "worse is better" solution,
> but I really don't have that much experience in logging, so any feedback would be
> appreciated.
>     Mirek
>  _______________________________________________
>  lumberjack-developers mailing list
>  lumberjack-developers(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/lumberjack-developers

[Dmitri Pal]

On 08/21/2012 03:16 PM, Miloslav Trmac wrote:
>  Hello,
>  while working on the Lumberjack message processing "pipeline", a design
> question has come up that probably must be a known issue for logging in general, so
> I'd like to ask before we start implementing.
>
>  What to do about "trusted" properties coming from untrusted input sources?
>
>  In particular, we are aiming at an rsyslog configuration where anything passed to
> /dev/log is automatically annotated with "pid", "uid" and other
> fields; at that point the information is 100% reliable.
>
>  Similarly, syslog messages can come from remote hosts over TCP, and if those hosts are
> similarly configured by the same administrators, the data is reliable as well.
>
>  OTOH, if syslog messages come from hosts that are not under the same control (e.g. data
> coming from user-administered machines), this data is not "reliable" - what to
> do about it?
>
>
>  We have come up with three possibilities:
>
>  a) Just leave it in, and let the user filter the data out based on other properties (e.g.
> host name) at the time of log searching (or just let the user notice when reading the log
> message ad hoc) - some queries and statistics may have misleading results.
>  
You do not own data so do not touch it.
You might provide some heuristics about its trustworthiness if you get
some metadata later but it is out of scope of the logging problem ATM.
I vote - a)

>  b) Delete the untrusted fields - drops data.
>
>  c) Somehow mark the fields as "untrusted" - preserves all data, and allows
> queries that ignore it.
>     c1) Is it any really any better than just filtering on host names as in a)?
>     c2) How can we mark the fields as "untrusted", in particular in the context
> of Lumberjack?  
>
>
>  I'm leaning towards a) as the UNIX-tradition "worse is better" solution,
> but I really don't have that much experience in logging, so any feedback would be
> appreciated.
>      Mirek
>  _______________________________________________
>  lumberjack-developers mailing list
>  lumberjack-developers(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/lumberjack-developers 

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Rainer Gerhards]

Short answer: a)

Long answer below:

>  In particular, we are aiming at an rsyslog configuration where
> anything
>  passed to /dev/log is automatically annotated with "pid", "uid" and
>  other fields; at that point the information is 100% reliable.
>  
Even that is not 100% true: for example, systemd fakes this information when it
"forwards" log messages from the real log socket to the one systemd provides. If
systemd can fake SCM_CREDENTIALS info, other apps can do as well. So there is no complete
trust even at this level.

>  Similarly, syslog messages can come from remote hosts over TCP, and
> if
>  those hosts are similarly configured by the same administrators, the
>  data is reliable as well.
>  
Ignoring the local problem mentioned above for the moment: if RFC5425 mutual TLS
cert-based auth is in place, you can be sure that the remote system is actually what it
tells it is. Unfortunately, rsyslog does not currently expose the remote cert name. This
is on the todo list and waits for the new hierarchical property engine which I hope to
begin "these days" (blog post coming up).

Note that there is a solution described in RFC5848, "signed syslog messages".
This permits signing of messages and thus provides ultimate proof of authenticy.
Unfortunately, the RFC does not support the dynamic filtering usually found in practice.
This is the prime reason I was so far hesitant to implement it. But it may be worth having
another look at that issue. The second problem is that the authenticy goes back to the
signer, which usually is the syslogd but not the log-emitting application. If the syslogd
receives faked information (see first point above), these signatures provide a false sense
of security. Even if we ignore that problem, it would probably be best to provide
signature capability in the logging library vs. the syslogd.

Rainer