Monitor systemd journals by way of e mail

Modern Linux methods usually use systemd as their init system and supervisor for jobs and plenty of different features. Services managed by systemd typically ship their output (of all types: warnings, errors, informational messages, and extra) to the systemd journal, to not conventional logging methods like syslog.

In addition to companies, Linux methods usually have many scheduled jobs (historically referred to as cron jobs, even when the system does not use cron to run them), and these jobs could both ship their output to the logging system or permit the job scheduler to seize the output and ship it by way of e mail.

When managing a number of methods, you’ll be able to set up and configure a centralized log-capture system to watch their habits, however the complexity of centralized methods could make them exhausting to handle.

A less complicated resolution is to have every system immediately ship “interesting” output to the administrator(s) by e mail. For methods utilizing systemd, this may be carried out utilizing Tim Waugh’s journal-brief device. This device nearly served my wants after I found it just lately, so, in typical open supply style, I contributed varied patches so as to add e mail help to the mission. Tim labored with me to get them merged, and now I can use the device to watch the 20-plus methods I handle as merely as potential.

Now, early every morning, I obtain between 20 and 23 e mail messages: most of them comprise a filtered view of every machine’s whole systemd journal (with warnings or extra critical messages), however a number of are logs generated by scheduled ZFS snapshot-replication jobs that I exploit for backups. In this text, I will present you tips on how to arrange comparable messages.

Install journal-brief

Although journal-brief is on the market in lots of Linux bundle repositories, the packaged variations won’t embody e mail help as a result of that was simply added just lately. That means you will want to put in it from PyPI; I will present you tips on how to manually set up it right into a Python digital setting to keep away from interfering with different elements of the put in system. If you will have a favourite device for doing this, be at liberty to make use of it.

Choose a location for the digital setting; on this article, I will use /decide/journal-brief for simplicity.

Nearly all the instructions on this tutorial have to be executed with root permissions or the equal (famous by the # immediate). However, it’s potential to put in the software program in a user-owned listing, grant that person permission to learn from the journal, and set up the required items as systemd person items, however that isn’t coated on this article.

Execute the next to create the digital setting and set up journal-brief and its dependencies:

$ python3 -m venv /decide/journal-brief
$ supply /decide/journal-brief/bin/activate
$ pip set up ‘journal-brief>=1.1.7’
$ deactivate

In order, these instructions will:

1. Create /decide/journal-brief and arrange a Python three.x digital setting there
2. Activate the digital setting in order that subsequent Python instructions will use it
3. Install journal-brief; be aware that the single-quotes are essential to maintain the shell from deciphering the > character as a redirection
4. Deactivate the digital setting, returning the shell again to the unique Python set up

Also, create some directories to retailer journal-brief configuration and state recordsdata with:

$ mkdir /and many others/journal-brief
$ mkdir /var/lib/journal-brief

Configure e mail necessities

While configuring e mail shoppers and servers is outdoors the scope of this text, for journal-brief to ship e mail, you’ll need to have one of many two supported mechanisms configured and operational.

Option 1: The mail command

Many methods have a mail command that can be utilized to ship (and skim) e mail. If such a command is put in in your system, you’ll be able to confirm that it’s configured correctly by executing a command like:

$ echo "Message body" | mail --subject="Test message" 

If the message arrives in your mailbox, you are able to proceed utilizing this kind of mail supply in journal-brief. If not, you’ll be able to both troubleshoot and proper the configuration or use SMTP supply.

To management the generated e mail messages’ attributes (e.g., From handle, To handle, Subject) with the mail command methodology, you have to use the command-line choices in your system’s mailer program: journal-brief will solely assemble a message’s physique and pipe it to the mailer.

Option 2: SMTP supply

If you will have an SMTP server out there that may settle for e mail and ahead it to your mailbox, journal-brief can talk immediately with it. In addition to plain SMTP, journal-brief helps Transport Layer Security (TLS) connections and authentication, which suggests it may be used with many hosted e mail companies (like Fastmail, Gmail, Pobox, and others). You might want to acquire a number of items of data to configure this supply mode:

• SMTP server hostname
• Port quantity for use for message submission (it defaults to port 25, however port 587 is often used)
• TLS help (non-obligatory or required)
• Authentication info (username and password/token, if required)

When utilizing this supply mode, journal-brief will assemble the whole message earlier than submitting it to the SMTP server, so the From handle, To handle, and Subject shall be equipped in journal-brief’s configuration.

Set up configuration and cursor recordsdata

Journal-brief makes use of YAML-formatted configuration recordsdata; it makes use of one file per desired mixture of filtering parameters, supply choices, and output codecs. For this text, these recordsdata are saved in /and many others/journal-brief, however you’ll be able to retailer them in any location you want.

In addition to the configuration recordsdata, journal-brief creates and manages cursor recordsdata, which permit it to maintain observe of the final message in its output. Using one cursor file for every configuration file ensures that no journal messages shall be misplaced, in distinction to a time-based log-delivery system, which could miss messages if a scheduled supply job cannot run to completion. For this text, the cursor recordsdata shall be saved in /var/lib/journal-brief (you’ll be able to retailer the cursor recordsdata in any location you want, however be sure to not retailer them in any kind of momentary filesystem, or they’re going to be misplaced).

Finally, journal-brief has intensive filtering and formatting capabilities; I will describe solely essentially the most primary choices, and you’ll study extra about its capabilities within the documentation for journal-brief and systemd.journal-fields.

Configure a day by day e mail with fascinating journal entries

This instance will arrange a day by day e mail to a system administrator named Robin at [email protected] from a server named storage. Robin’s mail supplier gives SMTP message submission via port 587 on a server named mail.server.invalid however doesn’t require authentication or TLS. The e mail shall be despatched from [email protected], so Robin can simply filter the incoming messages or generate alerts from them.

Robin has the great fortune to dwell in Fiji, the place the workday begins reasonably late (round 10:00am), so there’s loads of time each morning to learn emails of fascinating journal entries. This instance will collect the entries and ship them at eight:30am within the native time zone (Pacific/Fiji).

Step 1: Configure journal-brief

Create a textual content file at /and many others/journal-brief/daily-journal-email.yml with these contents:

cursor-file: '/var/lib/journal-brief/daily-journal-email'
output:
  - 'quick'
  - ‘systemd’
inclusions:
  - PRIORITY: 'warning'
e mail:
  suppress_empty: false
  smtp:
    to: '”Robin” <[email protected]>'
    from: '"Storage Server" <[email protected]>'
    topic: 'day by day journal'
    host: 'mail.server.invalid'
    port: 587

This configuration causes journal-brief to:

• Store the cursor on the path configured as cursor-file
• Format journal entries utilizing the quick format (one line per entry) and supply a listing of any systemd items which can be within the failed state
• Include journal entries from any service unit (even the Linux kernel) with a precedence of warning, error, or emergency
• Send an e mail even when there aren’t any matching journal entries, so Robin can make sure that the storage server continues to be working and has connectivity
• Send the e-mail utilizing SMTP

You can take a look at this configuration file by executing a journal-brief command:

$ journal-brief --conf /and many others/journal-brief/daily-journal-email

Journal-brief will scan the systemd journal for all new messages (sure, all of the messages it has by no means seen earlier than), establish any that match the precedence filter, and format them into an e mail that it sends to Robin. If the storage server has been operational for months (or years) and the systemd journal has by no means been purged, this might produce a really giant e mail message. In addition to Robin not appreciating such a big message, Robin’s e mail supplier is probably not prepared to simply accept it, so you’ll be able to generate a shorter message by executing this command:

$ journal-brief -b --conf /and many others/journal-brief/daily-journal-email

Adding the -b argument tells journal-brief to examine solely the systemd journal entries from the latest system boot and ignore any which can be older.

After journal-brief sends the e-mail to the SMTP server, it writes a string into the cursor file in order that the subsequent time it runs utilizing the identical cursor file, it’s going to know the place to begin within the journal. If the method fails for any motive (e.g., journal entry gathering, entry formatting, or SMTP supply), the cursor file will not be up to date, which suggests the subsequent time it makes use of the cursor file, the entries that might have been within the failed e mail shall be included within the subsequent e mail as a substitute.

Step 2: Set up the systemd service unit

Create a textual content file at /and many others/systemd/system/daily-journal-email.service with:

[Unit]

Description=Send day by day journal report
[Service]
ExecStart=/decide/journal-brief/bin/journal-brief --conf /and many others/journal-brief/%N.yml
Type=oneshot

This service unit will run journal-brief and specify a configuration file with the identical title because the unit file with the suffix eliminated, which is what %N provides. Since this service shall be began by a timer (see step three), there isn’t a must allow or manually begin it.

Step three: Set up the systemd timer unit

Create a textual content file at /and many others/systemd/system/daily-journal-email.timer with:

[Unit]

Description=Trigger day by day journal e mail report
[Timer]
OnCalendar=*-*-* 08:30:00 Pacific/Fiji
[Install]
WantedBy=multi-user.goal

This timer will begin the daily-journal-email service unit (as a result of its title matches the timer title) daily at eight:30am within the Pacific/Fiji time zone. If the time zone was not specified, the timer would set off the service at eight:30am within the system time zone configured on the storage server.

To make this timer begin each time the system boots, it’s WantedBy by the multi-user goal. To allow and begin the timer:

$ systemctl allow daily-journal-email.timer
$ systemctl begin daily-journal-email.timer
$ systemctl list-timers daily-journal-email.timer

The final command will show the timer’s standing, and the NEXT column will point out the subsequent time the timer will begin the service.

To study extra about systemd timers and constructing schedules for them, learn Use systemd timers instead of cronjobs.

Now the configuration is full, and Robin will obtain a day by day e mail of fascinating journal entries.

Monitor the output of a selected service

The storage server has some filesystems on solid-state storage units (SSD) and runs Fedora Linux. Fedora has an fstrim service that’s scheduled to run as soon as per week (utilizing a systemd timer, as within the instance above). Robin want to see the output generated by this service, even when it does not generate any warnings or errors. While this output shall be included within the day by day journal e mail, it will likely be intermingled with different journal entries, and Robin would like to have the output in its personal e mail message.

Step 1: Configure journal-brief

Create a textual content file at /and many others/journal-brief/fstrim.yml with:

cursor-file: '/var/lib/journal-brief/fstrim'
output: 'quick'
inclusions:
  - _SYSTEMD_UNIT:
    - ‘fstrim.service’
e mail:
  suppress_empty: false
  smtp:
    to: '”Robin” <[email protected]>'
    from: '"Storage Server" <[email protected]>'
    topic: 'weekly fstrim'
    host: 'mail.server.invalid'
    port: 587

This configuration is much like the earlier instance, besides that it’ll embody all entries associated to a systemd unit named fstrim.service, no matter their precedence ranges, and can embody solely entries associated to that service.

Step 2: Modify the systemd service unit

Unlike within the earlier instance, you needn’t create a systemd service unit or timer, since they exist already. Instead, you need to add habits to the prevailing service unit through the use of the systemd “drop-in file” mechanism (to keep away from modifying the system-provided unit file).

First, make sure that the EDITOR setting variable is ready to your most well-liked textual content editor (in any other case you will get the default editor in your system), and execute:

$ systemctl edit fstrim.service

Note that this doesn’t edit the prevailing service unit file; as a substitute, it opens an editor session to create a drop-in file (situated at /and many others/systemd/system/fstrim.service.d/override.conf).

Paste these contents into the editor and save the file:

[Service]
ExecStopPost=/decide/journal-brief/bin/journal-brief --conf /and many others/journal-brief/%N.yml

After you exit the editor, the systemd configuration will reload routinely (which is one good thing about utilizing systemctl edit as a substitute of making the file immediately). Like within the earlier instance, this drop-in makes use of %N to keep away from duplicating the service title; because of this the drop-in contents may be utilized to any service on the system, so long as the suitable configuration file is created in /and many others/journal-brief.

Using ExecStopPost will make journal-brief run after any try and run the fstrim.service, whether or not or not it is profitable. This is sort of helpful, as the e-mail shall be generated even when the fstrim.service can’t be began (for instance, if the fstrim command is lacking or not executable).

Please be aware that this method is primarily relevant to systemd companies that run to completion earlier than exiting (in different phrases, not background or daemon processes). If the Type within the Service part of the service’s unit file is forking, then journal-brief won’t execute till the desired service has stopped (both manually or by a system goal change, like shutdown).

The configuration is full; Robin will obtain an e mail after each try to begin the fstrim service; if the try is profitable, then the e-mail will embody the output generated by the service.

With this setup, you’ll be able to monitor the well being of your Linux methods that use systemd without having to arrange any centralized monitoring or logging instruments. I discover this monitoring methodology fairly efficient, because it attracts my consideration to uncommon occasions on the servers I preserve with out requiring any further effort.

Special because of Tim Waugh for creating the journal-brief device and being prepared to simply accept a reasonably giant patch so as to add direct e mail help reasonably than working journal-brief via cron.