Systemd service debugging tips.

Troubleshooting a systemd service can be tricky.

Because the Restart=always directive is sometimes set in the unit file, you don’t know if a service is running fine or if it is stopping and restarting all the time.

The systemctl status command doesn’t help you much:

# systemctl status myservice
● myservice.service
Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2016-09-09 20:20:30 CEST; 44s ago
Main PID: 21366 (node)
CGroup: /system.slice/myservice.service
├─21366 /opt/nodejs/current/bin/node /opt/myapp/index.js -config=myconf.js
└─21376 /opt/nodejs/node-v4.5.0-linux-x64/bin/node index.js -config=myconf.js

Sep 09 20:20:30 myvm systemd[1]: Started myservice.service.
Sep 09 20:20:30 myvm systemd[1]: Starting myservice.service...
Sep 09 20:20:31 myvm nodejs[21366]: index.js: configurationFilename=myconf.js

First, you can add the -l/–full and -n 20/–lines 20 options to the systemctl status command. They respectively stop truncating the journal output and display 20 lines instead of 10 by default.

But if the service is regularly stopping and starting, these options won’t help you.

Hopefully, you can use the -u option and specify the service name in the journalctl command:

# journalctl -u myservice
Sep 09 16:52:47 myvm systemd[1]: Started myservice.service.
Sep 09 16:52:47 myvm systemd[1]: Starting myservice.service...
Sep 09 16:52:47 myvm nodejs[4076]: index.js: configurationFilename=myconf.js
Sep 09 16:53:47 myvm nodejs[4076]: can't access database
Sep 09 16:53:47 myvm nodejs[4076]: exited
Sep 09 16:53:48 myvm systemd[1]: myservice.service holdoff time over, scheduling restart.
Sep 09 16:53:48 myvm systemd[1]: Started myservice.service.
Sep 09 16:53:48 myvm systemd[1]: Starting myservice.service...
Sep 09 16:53:48 myvm nodejs[4261]: index.js: configurationFilename=myconf.js
Sep 09 16:54:48 myvm nodejs[4261]: can't access database
Sep 09 16:54:48 myvm nodejs[4261]: exited
Sep 09 16:54:48 myvm systemd[1]: myservice.service holdoff time over, scheduling restart.

Due to an error (“can’t access database”), it is now clear that the service is stopping and starting.
If it were not the case, adding Environment=SYSTEMD_LOG_LEVEL=debug in the service stanza of the unit file could provide some useful messages.