Obscure rsyslog question on remote logging

[Tom Horsley]

I've got an old PC running Truenas as a local server for backups,
videos, etc.

I've got another PC I consider my "main" system running rsyslog.
On this PC I have tried (mostly successfully) to direct remote
syslog messages from the Truenas system to logfiles in a subdirectory
on my main system. This gibberish in my main system's rsyslog.conf
seems to do the trick:

# Log incoming message from remote systems here
$template DynamicFile,"/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log"
:hostname, contains, "nast"
*.*    -?DynamicFile

Most remote log message do indeed get redirected to the subdirectory
as expected, but SMART messages like these still wind up in
/var/log/messages on my main system rather than in the subdirectory.

Example:

Nov 16 16:16:51 nast 1 2023-11-16T16:16:51.564170-05:00 nast.my.lan smartd 1501 - - Device: /dev/ada0, SMART Usage Attribute: 194 Temperature_Celsius changed from 58 to 57

Most other messages wind up down in the subdirectory as expected, like this:

Nov 16 19:15:00 nast 1 2023-11-16T19:15:00.012359-05:00 nast.my.lan /usr/sbin/cron 16239 - - (root) CMD (/usr/libexec/atrun)

Anyone understand remote logging enough to provide any hints why this
is happening? It is clearly not all that critical, but it confuses me.

[Go Canes]

On Thu, Nov 16, 2023 at 7:51 PM Tom Horsley  wrote:
> # Log incoming message from remote systems here
> $template DynamicFile,"/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log"
> :hostname, contains, "nast"
> *.*    -?DynamicFile
>
> Most remote log message do indeed get redirected to the subdirectory
> as expected, but SMART messages like these still wind up in
> /var/log/messages on my main system rather than in the subdirectory.
>
> Example:
>
> Nov 16 16:16:51 nast 1 2023-11-16T16:16:51.564170-05:00 nast.my.lan smartd 1501 - - Device: /dev/ada0, SMART Usage Attribute: 194 Temperature_Celsius changed from 58 to 57
>
> Most other messages wind up down in the subdirectory as expected, like this:
>
> Nov 16 19:15:00 nast 1 2023-11-16T19:15:00.012359-05:00 nast.my.lan /usr/sbin/cron 16239 - - (root) CMD (/usr/libexec/atrun)

Any chance there is a rule for logging SMART messages that is taking precedence?

[Tom Horsley]

On Thu, 16 Nov 2023 20:09:26 -0500
Go Canes wrote:
> Any chance there is a rule for logging SMART messages that is taking precedence?

A good thing to check. I'll see if I can decrypt the .conf file
and figure that out. Thanks.

[Todd Zullinger]

Tom Horsley wrote:
> I've got an old PC running Truenas as a local server for backups,
> videos, etc.
>
> I've got another PC I consider my "main" system running rsyslog.
> On this PC I have tried (mostly successfully) to direct remote
> syslog messages from the Truenas system to logfiles in a subdirectory
> on my main system. This gibberish in my main system's rsyslog.conf
> seems to do the trick:
>
> # Log incoming message from remote systems here
> $template DynamicFile,"/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log"
> :hostname, contains, "nast"
> *.*    -?DynamicFile
>
> Most remote log message do indeed get redirected to the subdirectory
> as expected, but SMART messages like these still wind up in
> /var/log/messages on my main system rather than in the subdirectory.
>
> Example:
>
> Nov 16 16:16:51 nast 1 2023-11-16T16:16:51.564170-05:00 nast.my.lan smartd 1501 - - Device: /dev/ada0, SMART Usage Attribute: 194 Temperature_Celsius changed from 58 to 57
>
> Most other messages wind up down in the subdirectory as expected, like this:
>
> Nov 16 19:15:00 nast 1 2023-11-16T19:15:00.012359-05:00 nast.my.lan /usr/sbin/cron 16239 - - (root) CMD (/usr/libexec/atrun)
>
> Anyone understand remote logging enough to provide any hints why this
> is happening? It is clearly not all that critical, but it confuses me.

You probably need a stop action in your config to cease
further processing of messages which come from the remote
system.

I'd probably do this in the newer "advanced" format (AKA
RainerScript) instead of the "obsolete legacy" format.  I
believe this would do what you want:

# Log incoming message from remote systems here
template (name="DynamicFile" type="string" 
          string="/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log")
if ($hostname != $myhostname) then {
    action(type="omfile" dynaFile="DynamicFile")
    stop
}

With the legacy format, I _think_ adding '& stop' after the
'*.*    -?DynamicFile' line would work, but that format is
harder to read and far less clear so it's worth avoiding
when you need anything other than dirt-simple rules.

The comment and the condition don't really match in the
original.  The rule doesn't handle _any_ remote system, only
those whose hostname matches the string "nast" would be
handled.  If that's the only remote host you have sending
logs, then for all intents and purposes, it's the same
thing.  But it should work to use the "!=" condition just as
well (I say without having tested -- what could go wrong?).

To keep the conditional as you have it (tightened up
slightly, since your host appears to match 'nast' exactly so
an equals is better than contains):

if ($hostname == "nast") { ... }

Hopefully this helps more than it hurts. ;)

[Tom Horsley]

On Thu, 16 Nov 2023 20:46:33 -0500
Todd Zullinger wrote:
> I'd probably do this in the newer "advanced" format (AKA
> RainerScript) instead of the "obsolete legacy" format.  I
> believe this would do what you want:

I'm just adding a bit to the .conf file that ships with the
rpm, and it is using the old style, so I try to conform. Plus
when I first heard about the new style I tried and failed
to find the slightest particle of documentation and gave up :-).

I do have my added code at the end. Perhaps if I add the stop
then move it to the beginning of the rules, the SMART messages
would move.

[Tom Horsley]

On Thu, 16 Nov 2023 20:46:33 -0500
Todd Zullinger wrote:
> if ($hostname != $myhostname) then {

I tried this just running it through the rsyslogd syntax check (-N1 opt)
and it complained about using "myhostname" and asked if I didn't
actually mean "$myhostname" :-).

The documentation on rsyslog.com is still incredibly skimpy, but
possibly I need to use $$myhostname (at least the syntax check likes
that).

I found some obscure text about needing to use a $ in front of
system variables and since $myhostname is a system variable
I need to type it as $$myhostname in the condition (I think :-)

I don't suppose there is a totally different alternative to
rsyslog out there with sensible syntax? (Something like postfix
instead of sendmail.)

I haven't yet worked up the courage to actually try this for
real...

[Chris Adams]

Once upon a time, Tom Horsley  said:
> I found some obscure text about needing to use a $ in front of
> system variables and since $myhostname is a system variable
> I need to type it as $$myhostname in the condition (I think :-)

So an alternate approach that might help... you can assign a ruleset to
an input, and the UDP input is separate from the local input.  I put
this in /etc/rsyslog.d/net.conf:

########################################################################
template(name="Remote" type="list") {
        constant(value="/var/log/remote/")
        property(name="hostname")
        constant(value=".log")
}

module(load="imudp")

$umask 0022
ruleset(name="remote") {
        *.*     action(type="omfile" DynaFile="Remote" fileCreateMode="0644")
}
input(type="imudp" port="514" ruleset="remote")
########################################################################

My template is a simple /var/log/remote/<hostname>.log (I use the really
redundant .log extension so I can have a logrotate config to get
/var/log/remote/*.log).  I set the umask to 0022 and the file create
mode to 0644 because I have the group permissions on the /var/log/remote
directory to allow certain non-root users to read these logs.

[Roger Heflin]

I have seen syslog messages over 4096 chars have the first part go to the
right file and the overflow go to the default.  Is smart sending a large
message?  The other file would have a long smart message at the same time
and this would be overflow.

On Thu, Nov 16, 2023, 6:51 PM Tom Horsley  wrote:
> I've got an old PC running Truenas as a local server for backups,
> videos, etc.
>
> I've got another PC I consider my "main" system running rsyslog.
> On this PC I have tried (mostly successfully) to direct remote
> syslog messages from the Truenas system to logfiles in a subdirectory
> on my main system. This gibberish in my main system's rsyslog.conf
> seems to do the trick:
>
> # Log incoming message from remote systems here
> $template
> DynamicFile,"/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log"
> :hostname, contains, "nast"
> *.*    -?DynamicFile
>
> Most remote log message do indeed get redirected to the subdirectory
> as expected, but SMART messages like these still wind up in
> /var/log/messages on my main system rather than in the subdirectory.
>
> Example:
>
> Nov 16 16:16:51 nast 1 2023-11-16T16:16:51.564170-05:00 nast.my.lan smartd
> 1501 - - Device: /dev/ada0, SMART Usage Attribute: 194 Temperature_Celsius
> changed from 58 to 57
>
> Most other messages wind up down in the subdirectory as expected, like
> this:
>
> Nov 16 19:15:00 nast 1 2023-11-16T19:15:00.012359-05:00 nast.my.lan
> /usr/sbin/cron 16239 - - (root) CMD (/usr/libexec/atrun)
>
> Anyone understand remote logging enough to provide any hints why this
> is happening? It is clearly not all that critical, but it confuses me.
> --
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue