Note: This is an RHCSA 7 exam objective.
Presentation
Most of system log files are located in the /var/log directory due to SYSLOG default configuration (see /etc/rsyslog.conf file).
In addition, all SELinux events are written into the /var/log/audit/audit.log file.
With Systemd, new commands have been created to analyse logs at boot time and later.
Boot Process
Systemd primary task is to manage the boot process and provides informations about it.
To get the boot process duration, type:
# systemd-analyze Startup finished in 422ms (kernel) + 2.722s (initrd) + 9.674s (userspace) = 12.820s
To get the time spent by each task during the boot process, type:
# systemd-analyze blame 7.029s network.service 2.241s plymouth-start.service 1.293s kdump.service 1.156s plymouth-quit-wait.service 1.048s firewalld.service 632ms postfix.service 621ms tuned.service 460ms iprupdate.service 446ms iprinit.service 344ms accounts-daemon.service ... 7ms systemd-update-utmp-runlevel.service 5ms systemd-random-seed.service 5ms sys-kernel-config.mount
Note: You will find additional information on this point in the Lennart Poettering’s blog.
Journal Analysis
In addition, Systemd handles the system event log, a syslog daemon is not mandatory any more.
The reasons behind Journald creation are explained in this Introduction to Journald.
To get the content of the Systemd journal, type:
# journalctl
To get all the events related to the crond process in the journal, type:
# journalctl /sbin/crond
Note: You can replace /sbin/crond by `which crond`.
Altenatively, to get all the events related to the crond process, you can also type:
# journalctl -u crond
To get all the events since the last boot, type:
# journalctl -b
To get all the events that appeared today in the journal, type:
# journalctl --since=today
To get all the events with a syslog priority of err, type:
# journalctl -p err
To get the 10 last events and wait for any new one (like tail -f /var/log/messages), type:
# journalctl -f
Permanent Storage
By default, Journald logs are stored in the /run/log/journal directory and disappear after a reboot.
To store Journald logs in a more permanent way, type:
# mkdir /var/log/journal # echo "SystemMaxUse=50M" >> /etc/systemd/journald.conf # systemctl restart systemd-journald
Note: Setting the SystemMaxUse variable is necessary because otherwise 10% of the filesystem where the /var/log/journal directory is stored may be used at maximum by the journal.
Additional Resources
There is a tutorial dedicated to Systemd that presents additional information about the Systemd journal.
The rsyslog.com website provides some interesting information about rsyslogd (features, modules, plugins, etc).
Fedora Magazine also published an interesting article, Systemd: Using the journal.
Rainer Gerhards, rsyslog main author, discusses the arguments around the creation of Journald.
Nikolai Bezroukov wrote an authoritative page about Syslog.
The FreeIPA project (Identity Management with Kerberos, LDAP and Apache) provides several useful resources about Centralized Logging or Logging with Logstash/ElasticSearch/Kibana.
Logs can get flooded by some Systemd messages (starting sessions, etc). Red Hat provides a way to get rid of useless Systemd messages.
To get all the events related to a unit I prefer to do (instead of the example above):
# journalctl -u crond
In general, -u for any other systemd unit.
Interesting.
I updated the tutorial with your very good comment.
Thanks.
“syslog daemon is not mandatory any more”, maybe during the exam, but during daily admin work, you still check /var/log/messages because not all logs are adjusted to the journal.
I’m slightly surprised. I thought the systemd journal got more messages than the syslog daemon: it gets all boot and shutdown messages, when the syslog daemon doesn’t.
Option journalctl -b in the case when your logs are not permanently keep, in my opinion does not make sense. Please expand your article about a thread how to keep journal logs permanent, and then describe options journalctl -b -1,-2 etc. And as always thank you for a great job.
Yes, I will.
Permanent storage for systemd journal as per Red Hat training, they recommend the following steps:
# mkdir -p /var/log/journal
# chown root:systemd-journal /var/log/journal
# chmod 2755 /var/log/journal
and the last step is to reboot the system or issue the following special signal: killall -USR1 systemd-journal
can someone please explain what the main difference is and compare to the steps posted earlier:
# mkdir /var/log/journal
# echo “SystemMaxUse=50M” >> /etc/systemd/journald.conf
# systemctl restart systemd-journald
The first one sets the folder owner, group and setgid permissions while the second one does not, therefore in the latter case you end up with a folder that has the root group.
rsyslogd – In sander’s book the chapter on Configuring Logging has 70% of material on rsyslogd and only a couple pages on journald. He also says in the chapter “In the current state of RHEL 7, journald is not a replacement for rsyslog; it is just another way of logging information.” But since I don’t see much on rsyslogd here on this site, does it mean I don’t have to know it for the RHCSA/RHCE as long as I know journald? Trying to just focus on exams stuff for now.
The Syslog topic was removed from the RHCEv7 objectives (http://www.pmsas.pr.gov.br/wp-content/?id=certdepot-EX200&exam=rhel7-rhce-whats-new/).
You can still find some information through the ‘Locate and interpret system log files and journals’ RHCSA objective (http://www.pmsas.pr.gov.br/wp-content/?id=certdepot-EX200&exam=rhel7-interpret-system-log-files/).
In addition, you can find additional information through these two links: http://www.pmsas.pr.gov.br/wp-content/?id=certdepot-EX200&exam=rhel7-configure-system-log-remote-system/ and http://www.pmsas.pr.gov.br/wp-content/?id=certdepot-EX200&exam=rhel7-configure-system-accept-logging-remote-system/.
Maybe it is worth of mentioning that we can find log paths of different types of messages in configuration file of /etc/rsyslog.conf – this daemon is modular and it accepts messages from journald daemon, but not detailed like in journal.