RHEL7: Configure a system to log to a remote system.

Presentation

Sometimes you don’t only want to locally store syslog messages on your server but you also want to send them to a remote server.

This presents several advantages:

• all your logs are in a central location,
• they can be analyzed in a easy way,
• they can be backed up in a unique operation.

Configuration Procedure

Edit the /etc/rsyslog.conf file and uncomment the following lines at the end of the file where remote-host is the name of the rsyslog server:

#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
#*.* @@remote-host:514

Note: The two @ characters mean use of the TCP protocol. To use the UDP protocol, put only one @.

Check the file syntax:

# rsyslogd -N 1

Restart the rsyslog service:

# systemctl restart rsyslog

After setting up the rsyslog server, test the configuration:

# logger -p local0.notice -t TEST "Test"

On the rsyslog server, check the TEST string in the /var/log/messages file:

# grep "TEST" /var/log/messages

Advanced Configuration

With the previous configuration, you’ve got a spof (single point of failure) with your central rsyslog server. What happens if it is not available (failure, maintenance)? You temporarily loose messages.
You can avoid this situation by building two or more rsyslog servers (here remote-host1, remote-host2, remote-host3) and put the following configuration into your /etc/rsyslog.conf file:

$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
*.* @@remote-host1:514
$ActionExecOnlyWhenPreviousIsSuspended on
& @@remote-host2:514
& @@remote-host3:514
$ActionExecOnlyWhenPreviousIsSuspended off

Caution: The $ActionQueueType LinkedList and $ActionResumeRetryCount -1 lines have to be commented, otherwise this doesn’t work.

You can then set up an ELK (Elasticsearch, Logstash, Kibana) cluster with the 3 rsyslog servers to group the logs together again.

Additional Resources

You can go to the rsyslog website to get more information about the rsyslog features.