Presentation
As the Systemd now replaces SysVinit, it is time to get familiar with it and learn new commands. You can use this page as a Systemd tutorial. For the story behind Systemd, you can read Lennart Poettering on systemd’s Tumultuous Ascendancy.
Systemd makes a server boot quicker because it uses fewer scripts and tries to run more tasks in parallel, Systemd calls them units (to get a better understanding about the problems Systemd solves, read this thread from an ArchLinux maintainer).
The global Systemd configuration is stored in the /etc/systemd directory.
Service configuration files are located in the /usr/lib/systemd/system directory.
Custom service configuration files are stored in the /etc/systemd/system directory.
The /run/systemd/system directory is also used.
To get the current release of Systemd, type:
# systemctl --version systemd 208 +PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ
Note: The RHEL 7.2 release brings an upgrade of Systemd to version 219 (already available for pre-RHEL 7.2 here).
# systemctl --version systemd 219 +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
Boot Process
Systemd primary task is to manage the boot process and provides information about it.
To get the boot process duration, type:
# systemd-analyze Startup finished in 422ms (kernel) + 2.722s (initrd) + 9.674s (userspace) = 12.820s
To get the time spent by each task during the boot process, type:
# systemd-analyze blame 7.029s network.service 2.241s plymouth-start.service 1.293s kdump.service 1.156s plymouth-quit-wait.service 1.048s firewalld.service 632ms postfix.service 621ms tuned.service 460ms iprupdate.service 446ms iprinit.service 344ms accounts-daemon.service ... 7ms systemd-update-utmp-runlevel.service 5ms systemd-random-seed.service 5ms sys-kernel-config.mount
Seth Jennings created a short video about the systemd-analyze blame command.
To get the list of the critical chain of tasks during the boot process (any additional delay of these tasks would cause an increase of the overall boot time), type:
# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
multi-user.target @6.167s
└─mariadb.service @2.661s +3.505s
└─network.target @2.649s
└─network.service @2.168s +478ms
└─NetworkManager.service @1.993s +174ms
└─firewalld.service @826ms +1.162s
└─basic.target @822ms
└─sockets.target @821ms
└─dbus.socket @820ms
└─sysinit.target @813ms
└─systemd-update-utmp.service @806ms +6ms
└─auditd.service @747ms +58ms
└─local-fs.target @742ms
└─boot.mount @709ms +32ms
└─systemd-fsck@dev-disk-by\x2duuid-497a43ab\x2d33b0\x2
└─dev-disk-by\x2duuid-497a43ab\x2d33b0\x2d4153\x2d9f
To get the list of the critical chain for a particular service (here firewalld), type:
# systemd-analyze critical-chain firewalld.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
firewalld.service +1.162s
└─basic.target @822ms
└─sockets.target @821ms
└─dbus.socket @820ms
└─sysinit.target @813ms
└─systemd-update-utmp.service @806ms +6ms
└─auditd.service @747ms +58ms
└─local-fs.target @742ms
└─boot.mount @709ms +32ms
└─systemd-fsck@dev-disk-by\x2duuid-497a43ab\x2d33b0\x2d4153\x2d9
└─dev-disk-by\x2duuid-497a43ab\x2d33b0\x2d4153\x2d9f8d\x2d7788
To get the list of the critical chain for a particular target (here basic.target), type:
# systemd-analyze critical-chain basic.target | grep target
basic.target @822ms
└─sockets.target @821ms
└─sysinit.target @813ms
└─local-fs.target @742ms
To get the list of the dependencies, type:
# systemctl list-dependencies default.target ├─abrt-ccpp.service ├─abrt-oops.service ... ├─tuned.service ├─basic.target │ ├─firewalld.service │ ├─microcode.service ... ├─getty.target │ ├─getty@tty1.service │ └─serial-getty@ttyS0.service └─remote-fs.target
To get the list of dependencies for a particular service (here sshd), type:
# systemctl list-dependencies sshd.service
To get the list of dependencies for a particular target (here graphical.target), type:
# systemctl list-dependencies graphical.target | grep target graphical.target └─multi-user.target ├─basic.target │ ├─paths.target │ ├─slices.target │ ├─sockets.target │ ├─sysinit.target │ │ ├─cryptsetup.target │ │ ├─local-fs.target │ │ └─swap.target │ └─timers.target ├─getty.target └─remote-fs.target
To get the list of targets that need a particular target (here multi-user.target), type:
# systemctl list-dependencies --reverse multi-user.target multi-user.target └─graphical.target
You can get other information about dependencies by typing the following commands (here for sshd):
# systemctl show sshd | grep Requires Requires=basic.target # systemctl show sshd | grep Wants Wants=sshd-keygen.service system.slice
You can find some additional information about dependencies in Lennart Poettering’s blog.
In case you get boot problems, the Systemd debugging page may also help you.
Journal Analysis
In addition, Systemd handles the system event log through Journald.
The reasons behind Journald creation are explained in this Introduction to Journald.
The rsyslogd daemon, although present by default, is not mandatory any more.
To get the content of the Systemd journal, type:
# journalctl
Note: It is possible to add a specific user to the systemd-journal group to view journals without switching to root.
To get all the events related to the crond process in the journal, type:
# journalctl /sbin/crond
Note: You can replace /sbin/crond by `which crond`.
To get all the events related to a specific service (here chronyd), type:
# journalctl --unit=chronyd
Note1: You can also type: # journalctl -u chronyd
Note2: Starting with RHEL 7.2, you can now use a command like journalctl –unit=c*.
To get all the events since the last boot, type:
# journalctl -b
To get all the events that appeared today in the journal, type:
# journalctl --since=today
To get all the events with a syslog priority of err, type:
# journalctl -p err
To get all the events concerning the kernel, type:
# journalctl -k
To get the 10 last events and wait for any new one (like “tail -f /var/log/messages“), type:
# journalctl -f
With the RHEL 7.2 release, to get the list of the recent boots, type:
# journalctl --list-boots
By default, Journald logs are stored in the /run/log/journal directory and disappear after a reboot.
To store Journald logs in a more permanent way, type:
# mkdir /var/log/journal # systemd-tmpfiles --create --prefix /var/log/journal # echo "SystemMaxUse=50M" >> /etc/systemd/journald.conf # systemctl restart systemd-journald
Note1: The systemd-tmpfiles command is required to correctly set up permissions on the /var/log/journal directory (see details here).
Note2: Setting the SystemMaxUse variable is necessary because otherwise 10% of the filesystem where the /var/log/journal directory is stored may be used at maximum by the journal. Besides the space used, this will slow down the boot process and potentially the whole system (see details here). Furthermore, when using small VMs, the size of the journal can become critical (see this article about the journal size and the Storage parameter).
Note3: When changing the content of the /etc/systemd/journald.conf file, it is advisable to reboot (see details here).
Note4: Starting with Systemd v219 and RHEL 7.2, the filesystem where the /var/log/journal directory is located requires ACL support (see details here)
To display the disk space used by Journald at any time, type:
# journalctl --disk-usage Journals take up 8.0M on disk.
Note: With RHEL 7.2 and the 219 Systemd version, two new options have been added: –vacuum-size=1M and –vacuum-time=1day. They reduce the space used by archived journal files respectively to 1 MB or to the last 24 hours. These restrictions may be combined. They only operate on archived, not on active journal files.
Additional information about Journald is available in the Lennart Poettering’s blog or Lennart Poettering’s video (44min: the first ten minutes are very interesting concerning security issues).
Fedora Magazine also published an interesting article, Systemd: Using the journal.
Rainer Gerhards, rsyslog main author, discusses the arguments around the creation of Journald.
Nikolai Bezroukov wrote an authoritative page about Syslog.
If your journal gets corrupt, you could be interested in this article about Repairing CentOS 7 Journal Corruption.
Daniel Aleksandersen wrote an article about Feeding log files to the systemd journal.
Control Groups
Systemd organizes processes in control groups. For example, all the processes started by an apache webserver will be in the same control group, CGI scripts included.
To get the full hierarchy of control groups, type:
# systemd-cgls ├─user.slice │ └─user-1000.slice │ └─session-1.scope │ ├─2889 gdm-session-worker [pam/gdm-password] │ ├─2899 /usr/bin/gnome-keyring-daemon --daemonize --login │ ├─2901 gnome-session --session gnome-classic . . └─iprupdate.service └─785 /sbin/iprupdate --daemon
To get the list of control group ordered by CPU, memory and disk I/O load, type:
# systemd-cgtop Path Tasks %CPU Memory Input/s Output/s / 213 3.9 829.7M - - /system.slice 1 - - - - /system.slice/ModemManager.service 1 - - - -
To kill all the processes associated with an apache server (CGI scripts included), type:
# systemctl kill httpd
To put resource limits on a service (here 500 CPUShares), type:
# systemctl set-property httpd.service CPUShares=500
Note1: The change is written into the service unit file. Use the –runtime option to avoid this behavior.
Note2: By default, each service owns 1024 CPUShares. Nothing prevents you from giving a value smaller or bigger.
To get the current CPUShares service value, type:
# systemctl show -p CPUShares httpd.service CPUShares=500
Or:
# systemctl show httpd.service | grep CPUShares CPUShares=500
Go to the CGroup page for a more detailed discussion on this topic.
Sources: New control group interface, Systemd 205 announcement.
Service Management
Systemd deals with all the aspects of the service management. The systemctl command replaces the chkconfig and the service commands. The old commands are now a link to the systemctl command.
To activate the NTP service at boot, type:
# systemctl enable ntpd
Note1: You should specify ntpd.service but by default the .service suffix will be added.
Note2: If you specify a path, the .mount suffix will be added.
Note3: If you mention a device, the .device suffix will be added.
To deactivate it, start it, stop it, restart it, reload it, type:
# systemctl disable ntpd # systemctl start ntpd # systemctl stop ntpd # systemctl restart ntpd # systemctl reload ntpd
Note1: The reload option is not available for all the services.
Note2: It is also possible to mask and unmask a service. Masking a service prevents it from being started manually, by another service or by D-Bus.
Note3: With RHEL 7.2, you get the –now option to combine several actions at the same time: # systemctl enable –now ntpd httpd command enable and start the ntpd and httpd services (this also works with the disable and mask options).
To know if the NTP service is activated at boot, type:
# systemctl is-enabled ntpd enabled
To know if the NTP service is running, type:
# systemctl is-active ntpd inactive
To get the status of the Apache service, type:
# systemctl status httpd httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled) Active: active (running) since Tue 2014-08-21 11:48:23 CEST; 44s ago Main PID: 2446 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" CGroup: /system.slice/httpd.service └─2446 /usr/sbin/httpd -DFOREGROUND └─2447 /usr/sbin/httpd -DFOREGROUND └─2448 /usr/sbin/httpd -DFOREGROUND └─2449 /usr/sbin/httpd -DFOREGROUND └─2450 /usr/sbin/httpd -DFOREGROUND └─2451 /usr/sbin/httpd -DFOREGROUND └─2452 /usr/sbin/httpd -DFOREGROUND Aug 21 11:48:23 server.example.com systemd[1]: Starting The Apache HTTP Ser.... Aug 21 11:48:23 server.example.com systemd[1]: Started The Apache HTTP Server. Hint: Some lines were ellipsized, use -l to show in full.
Note1: In the line starting with “Loaded:”, you learn that the unit configuration file named /usr/lib/systemd/system/httpd.service has been loaded by Systemd. You also learn that the service isn’t started at boot time (disabled).
Note2: In the line beginning by “Active:”, you are informed that the service has been started for 44 seconds and is thus currently running (active).
Note3: The “Status:” line is specific to the Apache service and gives you a quick report of the service activity.
Note4: The line starting with “CGroup:” gives you all the information about the CGroup that the Apache service is part of (here /system.slice/httpd.service).
Note5: With the RHEL 7.3 release, you can now turn off any color output by assigning false to the SYSTEMD_COLORS environment variable in case you use a non-black background (for example: # export SYSTEMD_COLORS=false)(BZ#1265749).
If you change a service configuration, you will need to reload it:
# systemctl daemon-reload
Note: The systemd-delta command displays the list of changes in the service configuration files.
To get the list of all the units (services, mount points, devices) with their status and description, type:
# systemctl
Note: You can use the arrow keys to move inside the unit list.
To get a more readable list, type:
# systemctl list-unit-files
To get the list of services that failed at boot, type:
# systemctl --failed
To get the status of a service (here httpd) on a remote server (here rhel7.example.com), type:
# systemctl -H root@rhel7.example.com status httpd.service
The RHEL 7.2 release brings the ability to connect as root to a container inside a host.
To get the status of a service (here httpd) in a container (here httpsrv) on a remote server (here rhel72.example.com), type:
# systemctl -H root@rhel72.example.com:httpsrv status httpd.service
Note: This feature is only available for the root user, as entering containers is a privileged operation.
To get all the configuration details about a service (here httpd), type:
# systemctl show httpd Id=httpd.service Names=httpd.service Requires=basic.target Wants=system.slice Conflicts=shutdown.target Before=shutdown.target After=network.target remote-fs.target nss-lookup.target systemd-journald.socket Description=The Apache HTTP Server LoadState=loaded ActiveState=inactive ... ExecMainPID=0 ExecMainCode=0 ExecMainStatus=0
Digital Ocean provides an excellent article about Systemd unit files.
Alexander Patrakov‘s blog also provides a very good introduction for writing Systemd service files.
Vidar Hokstad‘s tutorial describes numerous options when writing Systemd unit files.
In addition, the FedoraProject website can also help you if you need to write Systemd unit files.
If you want to run several instances of the same service, the nodesource website offers a tutorial about configuring Systemd to run multiple instances.
If you need to use environment variables in an unit file, this article can help you (see also here and here).
Finally, FedoraMagazine published a great article about systemd unit dependencies and order.
Official RedHat documentation has recently (2017) been improved about Creating and modifying Systemd unit files.
Systemd doesn’t include /sbin and /bin in the PATH variable by default. If you need it, you will have to add it in your unit file like this (source):
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
It seems that Systemd doesn’t take the ulimit values into account, you have to use a LimitNOFILE statement for this purpose (see this thread).
The RHEL 7.2 release brings the systemctl cat command to display a unit file.
To display a service configuration file (here rngd), type:
# systemctl cat rngd # /usr/lib/systemd/system/rngd.service [Unit] Description=Hardware RNG Entropy Gatherer Daemon [Service] ExecStart=/sbin/rngd -f [Install] WantedBy=multi-user.target
Note: The systemctl cat rngd command carries several advantages compared to a simple cat rngd.service command:
- You don’t need to know which version is executed (the default version in /usr/lib/system/system/rngd.service or a customized version in /etc/systemd/system/rngd.service),
- You get the current version starting with a # character at the top,
- If there are “drop-in” unit file snippets (here in /etc/systemd/system/rngd.service.d), they are inserted at the right place.
The RHEL 7.2 release also adds the systemctl edit command.
By default, this command edits a drop-in snippet called override.conf or creates it if it doesn’t already exist.
After the snippet has been edited, systemd configuration is reloaded.
For example, to edit/create a drop-in file for the httpd service, type:
# systemctl edit httpd
Note1: The /etc/systemd/system/httpd.service.d directory is automatically created if necessary.
Note2: Editing is done with an editor specified by the SYSTEMD_EDITOR, EDITOR or VISUAL environment variables in this order. By default, the nano, vim, and vi editors will be tried one after the other. If like me you prefer vim to nano, don’t forget to type export EDITOR=vim or to set up your environment before executing systemctl edit.
Note3: No need to type systemctl daemon-reload, systemctl edit takes care of it.
If you add the –full option, it is no longer a drop-in snippet that is edited or created, it’s the unit file itself.
To modify the content of the httpd unit file, type:
# systemctl edit httpd --full
Note1: Be careful when editing this way, your changes will be active right after exiting the editor.
Note2: You don’t need to specify where the unit file is located.
Note3: Instead of using the –full option, you can use the –runtime option to edit a unit file if no /etc/systemd/system unit file exists for a given service and you don’t want the change to survive a reboot.
Targets/Run Levels
Systemd also deals with targets. A target is a grouping mechanism allowing several services to start at the same time. The list of services associated with a particular target is stored in a directory named with the target name and the suffix “.wants”. In some way, targets replace run levels but they are more general. To move to maintenance mode, type:
# systemctl rescue
Note: There is also an emergency target only available when set up in the kernel boot line (systemd.unit=emergency.target) and for critical situations.
To move to the level 3 (equivalent to the previous level 3), type:
# systemctl isolate runlevel3.target
Or
# systemctl isolate multi-user.target
To move to the graphical level (equivalent to the previous level 5), type:
# systemctl isolate graphical.target
To set the default run level to non-graphical mode, type:
# systemctl set-default multi-user.target
To set the default run level to graphical mode, type:
# systemctl set-default graphical.target
To get the current default run level, type:
# systemctl get-default graphical.target
To stop a server, type:
# systemctl poweroff
Note: You can still use the poweroff command, a link to the systemctl command has been created (the same thing is true for the halt and reboot commands).
To reboot a server, suspend it or put it into hibernation, type:
# systemctl reboot # systemctl suspend # systemctl hibernate
In case you get boot or shutdown problems, the Systemd debugging page may help you (see also debug mode below).
Debug Mode
Systemd can be put into debug mode by adding the following options to the kernel line:
systemd.log_level=debug systemd.log_target=kmsg
Caution: This seriously increases the amount of logs written to disks. This would affect any central logging system you have with lots of unnecessary traffic, amplified if you have many machines running on a VM host. This is definitively not for production servers (see discussion here).
Linux Standardization
Systemd‘s authors have decided to help Linux standardization among distributions. Through Systemd, changes happen in the localization of some configuration files.
Miscellaneous
To get the server hostnames, type:
# hostnamectl
Static hostname: rhel7.example.com
Icon name: computer-laptop
Chassis: laptop
Machine ID: bcdc71f1943f4d859aa37e54a422938d
Boot ID: f84556924b4e4bbf9c4a82fef4ac26d0
Operating System: Red Hat Enterprise Linux Everything 7.0 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.0:beta:everything
Kernel: Linux 3.10.0-54.0.1.el7.x86_64
Architecture: x86_64
Note: There are three kinds of hostnames: static, pretty, and transient.
“The static host name is the traditional hostname, which can be chosen by the user, and is stored in the /etc/hostname file. The transient hostname is a dynamic host name maintained by the kernel. It is initialized to the static host name by default, whose value defaults to localhost. It can be changed by DHCP or mDNS at runtime. The pretty hostname is a free-form UTF8 host name for presentation to the user.” Source: RHEL 7 Networking Guide.
To assign the rhel7 hostname permanently to the server, type:
# hostnamectl set-hostname rhel7
Note: With this syntax all three hostnames (static, pretty, and transient) take the rhel7 value at the same time. However, it is possible to set the three hostnames separately by using the –pretty, –static, and –transient options.
To get the current locale, virtual console keymap and X11 layout, type:
# localectl System Locale: LANG=en_US.UTF-8 VC Keymap: en_US X11 Layout: en_US
To assign the en_GB.utf8 value to the locale, type:
# localectl set-locale LANG=en_GB.utf8
To assign the en_GB value to the virtual console keymap, type:
# localectl set-keymap en_GB
To assign the en_GB value to the X11 layout, type:
# localectl set-x11-keymap en_GB
To get the current date and time, type:
# timedatectl Local time: Fri 2014-01-24 22:34:05 CET Universal time: Fri 2014-01-24 21:34:05 UTC RTC time: Fri 2014-01-24 21:34:05 Timezone: Europe/Madrid (CET, +0100) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2013-10-27 02:59:59 CEST Sun 2013-10-27 02:00:00 CET Next DST change: DST begins (the clock jumps one hour forward) at Sun 2014-03-30 01:59:59 CET Sun 2014-03-30 03:00:00 CEST
To set the current date, type:
# timedatectl set-time YYYY-MM-DD
To set the current time, type:
# timedatectl set-time HH:MM:SS
To get the list of time zones, type:
# timedatectl list-timezones
To change the time zone to America/New_York, type:
# timedatectl set-timezone America/New_York
To get the users’ list, type:
# loginctl list-users UID USER 42 gdm 1000 tom 0 root
To get the list of all current user sessions, type:
# loginctl list-sessions SESSION UID USER SEAT 1 1000 tom seat0 1 sessions listed.
To get the properties of the user tom, type:
# loginctl show-user tom UID=1000 GID=1000 Name=tom Timestamp=Fri 2014-01-24 21:53:43 CET TimestampMonotonic=160754102 RuntimePath=/run/user/1000 Slice=user-1000.slice Display=1 State=active Sessions=1 IdleHint=no IdleSinceHint=0 IdleSinceHintMonotonic=0
Additional Resources
For a better understanding, you can additionally read:
- a detailed introduction to Systemd,
- Bob Cromwell’s blog about Systemd,
- an overview of Systemd by Red Hat,
- a first dive into Systemd by Etsuji Nakai,
- a Systemd presentation by Ben Breard,
- an interview of Lennart Poettering explaining the story behind Systemd,
- the 2015 Red Hat Summit Systemd presentation talking about version 219 new features,
- an article about converting Sysvinit scripts into Systemd unit files,
- a description of the Systemd bug fix and enhancement updates coming with the version 219,
- the Systemd changelog,
- the softpanorama website provides a lot of resources about Systemd,
- a tutorial about Systemd,
- an in-depth presentation to Understand Systemd.
You can also watch a video about Working with systemd by Sander van Vugt (43min/2015).
Sources: Archlinux wiki, Freedesktop wiki, Gentoo wiki, RHEL 7 System Administration Guide, Fedora wiki.
Leave a Reply
You must be logged in to post a comment.