Understanding systemd at startup on Linux

In Learning to love systemd, the primary article on this sequence, I checked out systemd’s capabilities and structure and the controversy round its function as a alternative for the previous SystemV init program and startup scripts. In this second article, I am going to begin exploring the information and instruments that handle the Linux startup sequence. I am going to clarify the systemd startup sequence, tips on how to change the default startup goal (runlevel in SystemV phrases), and tips on how to manually swap to a special goal with out going by way of a reboot.

I am going to additionally take a look at two essential systemd instruments. The first is the systemctl command, which is the first technique of interacting with and sending instructions to systemd. The second is journalctl, which supplies entry to the systemd journals that include large quantities of system historical past information resembling kernel and repair messages (each informational and error messages).

Be positive to make use of a non-production system for testing and experimentation on this and future articles. Your take a look at system must have a GUI desktop (resembling Xfce, LXDE, Gnome, KDE, or one other) put in.

I wrote in my earlier article that I deliberate to have a look at making a systemd unit and including it to the startup sequence on this article. Because this text grew to become longer than I anticipated, I’ll maintain that for the following article on this sequence.

Exploring Linux startup with systemd

Before you possibly can observe the startup sequence, you have to do a few issues to make the boot and startup sequences open and visual. Normally, most distributions use a startup animation or splash display to cover the detailed messages that will in any other case be displayed throughout a Linux host’s startup and shutdown. This known as the Plymouth boot display on Red Hat-based distros. Those hidden messages can present quite a lot of details about startup and shutdown to a sysadmin on the lookout for data to troubleshoot a bug or to only be taught concerning the startup sequence. You can change this utilizing the GRUB (Grand Unified Boot Loader) configuration.

The primary GRUB configuration file is /boot/grub2/grub.cfg, however, as a result of this file will be overwritten when the kernel model is up to date, you don’t want to vary it. Instead, modify the /and so forth/default/grub file, which is used to switch the default settings of grub.cfg.

Start by wanting on the present, unmodified model of the /and so forth/default/grub file:

[root@testvm1 ~]# cd /and so forth/default ; cat grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, launch .*$,,g' /and so forth/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora_testvm1-swap rd.lvm.
lv=fedora_testvm1/root rd.lvm.lv=fedora_testvm1/swap rd.lvm.lv=fedora_
testvm1/usr rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
[root@testvm1 default]#

Chapter 6 of the GRUB documentation incorporates an inventory of all of the potential entries within the /and so forth/default/grub file, however I deal with the next:

• I modify GRUB_TIMEOUT, the variety of seconds for the GRUB menu countdown, from 5 to 10 to present a bit extra time to reply to the GRUB menu earlier than the countdown hits zero.
• I delete the final two parameters on GRUB_CMDLINE_LINUX, which lists the command-line parameters which can be handed to the kernel at boot time. One of those parameters, rhgb stands for Red Hat Graphical Boot, and it shows the little Fedora icon animation in the course of the kernel initialization as an alternative of displaying boot-time messages. The different, the quiet parameter, prevents displaying the startup messages that doc the progress of the startup and any errors that happen. I delete each rhgb and quiet as a result of sysadmins must see these messages. If one thing goes mistaken throughout boot, the messages displayed on the display can level to the reason for the issue.

After you make these adjustments, your GRUB file will appear to be:

[root@testvm1 default]# cat grub
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, launch .*$,,g' /and so forth/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora_testvm1-swap rd.lvm.
lv=fedora_testvm1/root rd.lvm.lv=fedora_testvm1/swap rd.lvm.lv=fedora_
testvm1/usr"
GRUB_DISABLE_RECOVERY="false"
[root@testvm1 default]#

The grub2-mkconfig program generates the grub.cfg configuration file utilizing the contents of the /and so forth/default/grub file to switch a few of the default GRUB settings. The grub2-mkconfig program sends its output to STDOUT. It has a -o possibility that permits you to specify a file to ship the datastream to, however it’s simply as straightforward to make use of redirection. Run the next command to replace the /boot/grub2/grub.cfg configuration file:

[root@testvm1 grub2]# grub2-mkconfig > /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux picture: /boot/vmlinuz-Four.18.9-200.fc28.x86_64
Found initrd picture: /boot/initramfs-Four.18.9-200.fc28.x86_64.img
Found linux picture: /boot/vmlinuz-Four.17.14-202.fc28.x86_64
Found initrd picture: /boot/initramfs-Four.17.14-202.fc28.x86_64.img
Found linux picture: /boot/vmlinuz-Four.16.Three-301.fc28.x86_64
Found initrd picture: /boot/initramfs-Four.16.Three-301.fc28.x86_64.img
Found linux picture: /boot/vmlinuz-Zero-rescue-7f12524278bd40e9b10a085bc82dc504
Found initrd picture: /boot/initramfs-Zero-rescue-7f12524278bd40e9b10a085bc82dc504.img
achieved
[root@testvm1 grub2]#

Reboot your take a look at system to view the startup messages that will in any other case be hidden behind the Plymouth boot animation. But what if you have to view the startup messages and haven’t disabled the Plymouth boot animation? Or you have got, however the messages stream by too quick to learn? (Which they do.)

There are a few choices, and each contain log information and systemd journals—that are your mates. You can use the much less command to view the contents of the /var/log/messages file. This file incorporates boot and startup messages in addition to messages generated by the working system throughout regular operation. You may also use the journalctl command with none choices to view the systemd journal, which incorporates basically the identical data:

[root@testvm1 grub2]# journalctl
-- Logs start at Sat 2020-01-11 21:48:08 EST, finish at Fri 2020-04-03 08:54:30 EDT. --
Jan 11 21:48:08 f31vm.each.org kernel: Linux model 5.Three.7-301.fc31.x86_64 (mockbuild@bkernel03.phx2.fedoraproject.org) (gcc model 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #1 SMP Mon Oct >
Jan 11 21:48:08 f31vm.each.org kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.Three.7-301.fc31.x86_64 root=/dev/mapper/VG01-root ro resume=/dev/mapper/VG01-swap rd.lvm.lv=VG01/root rd>
Jan 11 21:48:08 f31vm.each.org kernel: x86/fpu: Supporting XSAVE characteristic 0x001: 'x87 floating level registers'
Jan 11 21:48:08 f31vm.each.org kernel: x86/fpu: Supporting XSAVE characteristic 0x002: 'SSE registers'
Jan 11 21:48:08 f31vm.each.org kernel: x86/fpu: Supporting XSAVE characteristic 0x004: 'AVX registers'
Jan 11 21:48:08 f31vm.each.org kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Jan 11 21:48:08 f31vm.each.org kernel: x86/fpu: Enabled xstate options 0x7, context measurement is 832 bytes, utilizing 'normal' format.
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-provided bodily RAM map:
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000dffeffff] usable
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x00000000dfff0000-0x00000000dfffffff] ACPI information
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x00000000fec0000Zero-0x00000000fec00fff] reserved
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x00000000charge0000Zero-0x00000000charge00fff] reserved
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Jan 11 21:48:08 f31vm.each.org kernel: BIOS-e820: [mem 0x0000000100000000-0x000000041fffffff] usable
Jan 11 21:48:08 f31vm.each.org kernel: NX (Execute Disable) safety: lively
Jan 11 21:48:08 f31vm.each.org kernel: SMBIOS 2.5 current.
Jan 11 21:48:08 f31vm.each.org kernel: DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jan 11 21:48:08 f31vm.each.org kernel: Hypervisor detected: KVM
Jan 11 21:48:08 f31vm.each.org kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00
Jan 11 21:48:08 f31vm.each.org kernel: kvm-clock: cpu Zero, msr 30ae01001, main cpu clock
Jan 11 21:48:08 f31vm.each.org kernel: kvm-clock: utilizing sched offset of 8250734066 cycles
Jan 11 21:48:08 f31vm.each.org kernel: clocksource: kvm-clock: masks: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
Jan 11 21:48:08 f31vm.each.org kernel: tsc: Detected 2807.992 MHz processor
Jan 11 21:48:08 f31vm.each.org kernel: e820: replace [mem 0x00000000-0x00000fff] usable ==> reserved
Jan 11 21:48:08 f31vm.each.org kernel: e820: take away [mem 0x000a0000-0x000fffff] usable
<snip>

I truncated this datastream as a result of it may be tons of of 1000’s and even tens of millions of traces lengthy. (The journal itemizing on my main workstation is 1,188,482 traces lengthy.) Be positive to do this in your take a look at system. If it has been operating for a while—even when it has been rebooted many occasions—large quantities of knowledge can be displayed. Explore this journal information as a result of it incorporates quite a lot of data that may be very helpful when doing downside willpower. Knowing what this information appears to be like like for a traditional boot and startup might help you find issues once they happen.

I’ll focus on systemd journals, the journalctl command, and tips on how to kind by way of all of that information to search out what you need in additional element in a future article on this sequence.

After GRUB hundreds the kernel into reminiscence, it should first extract itself from the compressed model of the file earlier than it will possibly carry out any helpful work. After the kernel has extracted itself and began operating, it hundreds systemd and turns management over to it.

This is the tip of the boot course of. At this level, the Linux kernel and systemd are operating however unable to carry out any productive duties for the tip consumer as a result of nothing else is operating, there is not any shell to supply a command line, no background processes to handle the community or different communication hyperlinks, and nothing that permits the pc to carry out any productive perform.

Systemd can now load the useful items required to deliver the system as much as a particular goal run state.

Targets

A systemd goal represents a Linux system’s present or desired run state. Much like SystemV begin scripts, targets outline the providers that should be current for the system to run and be lively in that state. Figure 1 exhibits the potential run-state targets of a Linux system utilizing systemd. As seen within the first article of this sequence and within the systemd bootup man web page (man bootup), there are different intermediate targets which can be required to allow numerous crucial providers. These can embrace swap.goal, timers.goal, local-fs.goal, and extra. Some targets (like fundamental.goal) are used as checkpoints to make sure that all of the required providers are up and operating earlier than transferring on to the next-higher stage goal.

Unless in any other case modified at boot time within the GRUB menu, systemd at all times begins the default.goal. The default.goal file is a symbolic hyperlink to the true goal file. For a desktop workstation, that is sometimes going to be the graphical.goal, which is equal to runlevel 5 in SystemV. For a server, the default is extra more likely to be the multi-user.goal, which is like runlevel Three in SystemV. The emergency.goal file is much like single-user mode. Targets and providers are systemd items.

The following desk, which I included within the earlier article on this sequence, compares the systemd targets with the previous SystemV startup runlevels. The systemd goal aliases are offered by systemd for backward compatibility. The goal aliases enable scripts—and sysadmins—to make use of SystemV instructions like init Three to vary runlevels. Of course, the SystemV instructions are forwarded to systemd for interpretation and execution.

^{Fig. 1: Comparison of SystemV runlevels with systemd targets and goal aliases.}

Each goal has a set of dependencies described in its configuration file. systemd begins the required dependencies, that are the providers required to run the Linux host at a particular stage of performance. When all the dependencies listed within the goal configuration information are loaded and operating, the system is operating at that concentrate on stage. If you need, you possibly can overview the systemd startup sequence and runtime targets within the first article on this sequence, Learning to love systemd.

Exploring the present goal

Many Linux distributions default to putting in a GUI desktop interface in order that the put in methods can be utilized as workstations. I at all times set up from a Fedora Live boot USB drive with an Xfce or LXDE desktop. Even once I’m putting in a server or different infrastructure sort of host (resembling those I take advantage of for routers and firewalls), I take advantage of considered one of these installations that installs a GUI desktop.

I might set up a server with out a desktop (and that will be typical for information facilities), however that doesn’t meet my wants. It shouldn’t be that I want the GUI desktop itself, however the LXDE set up contains most of the different instruments I take advantage of that aren’t in a default server set up. This means much less work for me after the preliminary set up.

But simply because I’ve a GUI desktop doesn’t imply it is smart to make use of it. I’ve a 16-port KVM that I can use to entry the KVM interfaces of most of my Linux methods, however the overwhelming majority of my interplay with them is through a distant SSH connection from my main workstation. This approach is safer and makes use of fewer system sources to run multi-user.goal in comparison with graphical.goal.

To start, test the default goal to confirm that it’s the graphical.goal:

[root@testvm1 ~]# systemctl get-default
graphical.goal
[root@testvm1 ~]#

Now confirm the at present operating goal. It needs to be the identical because the default goal. You can nonetheless use the previous technique, which shows the previous SystemV runlevels. Note that the earlier runlevel is on the left; it’s N (which implies None), indicating that the runlevel has not modified for the reason that host was booted. The quantity 5 signifies the present goal, as outlined within the previous SystemV terminology:

[root@testvm1 ~]# runlevel
N 5
[root@testvm1 ~]#

Note that the runlevel man web page signifies that runlevels are out of date and supplies a conversion desk.

You may also use the systemd technique. There is not any one-line reply right here, nevertheless it does present the reply in systemd phrases:

[root@testvm1 ~]# systemctl list-units --type goal

UNIT                   LOAD   ACTIVE SUB    DESCRIPTION                

fundamental.goal           loaded lively lively Basic System               

cryptsetup.goal      loaded lively lively Local Encrypted Volumes    

getty.goal           loaded lively lively Login Prompts              

graphical.goal       loaded lively lively Graphical Interface        

local-fs-pre.goal    loaded lively lively Local File Systems (Pre)   

local-fs.goal        loaded lively lively Local File Systems         

multi-user.goal      loaded lively lively Multi-User System          

network-online.goal  loaded lively lively Network is Online          

community.goal         loaded lively lively Network                    

nfs-client.goal      loaded lively lively NFS shopper providers        

nss-user-lookup.goal loaded lively lively User and Group Name Lookups

paths.goal           loaded lively lively Paths                      

remote-fs-pre.goal   loaded lively lively Remote File Systems (Pre)  

remote-fs.goal       loaded lively lively Remote File Systems        

rpc_pipefs.goal      loaded lively lively rpc_pipefs.goal          

slices.goal          loaded lively lively Slices                     

sockets.goal         loaded lively lively Sockets                    

sshd-keygen.goal     loaded lively lively sshd-keygen.goal         

swap.goal            loaded lively lively Swap                       

sysinit.goal         loaded lively lively System Initialization      

timers.goal          loaded lively lively Timers                     
LOAD   = Reflects whether or not the unit definition was correctly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values rely on unit sort.
21 loaded items listed. Pass --all to see loaded however inactive items, too.
To present all put in unit information use 'systemctl list-unit-files'.

This exhibits all the at present loaded and lively targets. You may also see the graphical.goal and the multi-user.goal. The multi-user.goal is required earlier than the graphical.goal will be loaded. In this instance, the graphical.goal is lively.

Switching to a special goal

Making the swap to the multi-user.goal is simple:

[root@testvm1 ~]# systemctl isolate multi-user.goal

The show ought to now change from the GUI desktop or login display to a digital console. Log in and record the at present lively systemd items to confirm that graphical.goal is now not operating:

[root@testvm1 ~]# systemctl list-units --type goal

Be positive to make use of the runlevel command to confirm that it exhibits each earlier and present “runlevels”:

[root@testvm1 ~]# runlevel
5 Three

Changing the default goal

Now, change the default goal to the multi-user.goal so that it’s going to at all times boot into the multi-user.goal for a console command-line interface quite than a GUI desktop interface. As the basis consumer in your take a look at host, change to the listing the place the systemd configuration is maintained and do a fast itemizing:

[root@testvm1 ~]# cd /and so forth/systemd/system/ ; ll
drwxr-xr-x. 2 root root 4096 Apr 25  2018  fundamental.goal.needs
<snip>
lrwxrwxrwx. 1 root root   36 Aug 13 16:23  default.goal -> /lib/systemd/system/graphical.goal
lrwxrwxrwx. 1 root root   39 Apr 25  2018  display-manager.service -> /usr/lib/systemd/system/lightdm.service
drwxr-xr-x. 2 root root 4096 Apr 25  2018  getty.goal.needs
drwxr-xr-x. 2 root root 4096 Aug 18 10:16  graphical.goal.needs
drwxr-xr-x. 2 root root 4096 Apr 25  2018  local-fs.goal.needs
drwxr-xr-x. 2 root root 4096 Oct 30 16:54  multi-user.goal.needs
<snip>
[root@testvm1 system]#

I shortened this itemizing to focus on a number of essential issues that can assist clarify how systemd manages the boot course of. You ought to be capable of see the complete record of directories and hyperlinks in your digital machine.

The default.goal entry is a symbolic hyperlink (symlink, gentle hyperlink) to the listing /lib/systemd/system/graphical.goal. List that listing to see what else is there:

[root@testvm1 system]# ll /lib/systemd/system/ | much less

You ought to see information, directories, and extra hyperlinks on this itemizing, however look particularly for multi-user.goal and graphical.goal. Now show the contents of default.goal, which is a hyperlink to /lib/systemd/system/graphical.goal:

[root@testvm1 system]# cat default.goal 
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is a part of systemd.
#
#  systemd is free software program; you possibly can redistribute it and/or modify it
#  underneath the phrases of the GNU Lesser General Public License as revealed by
#  the Free Software Foundation; both model 2.1 of the License, or
#  (at your possibility) any later model.
[Unit]
Description=Graphical Interface
Documentation=man:systemd.particular(7)
Requires=multi-user.goal
Wants=display-manager.service
Conflicts=rescue.service rescue.goal
After=multi-user.goal rescue.service rescue.goal display-manager.service
AllowIsolate=sure
[root@testvm1 system]#

This hyperlink to the graphical.goal file describes all the stipulations and necessities that the graphical consumer interface requires. I’ll discover a minimum of a few of these choices within the subsequent article on this sequence.

To allow the host besides to multi-user mode, you have to delete the present hyperlink and create a brand new one which factors to the proper goal. Make the PWD /and so forth/systemd/system, if it isn’t already:

[root@testvm1 system]# rm -f default.goal 
[root@testvm1 system]# ln -s /lib/systemd/system/multi-user.goal default.goal

List the default.goal hyperlink to confirm that it hyperlinks to the proper file:

[root@testvm1 system]# ll default.goal 
lrwxrwxrwx 1 root root 37 Nov 28 16:08 default.goal -> /lib/systemd/system/multi-user.goal
[root@testvm1 system]#

If your hyperlink doesn’t look precisely like this, delete it and take a look at once more. List the content material of the default.goal hyperlink:

[root@testvm1 system]# cat default.goal 
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is a part of systemd.
#
#  systemd is free software program; you possibly can redistribute it and/or modify it
#  underneath the phrases of the GNU Lesser General Public License as revealed by
#  the Free Software Foundation; both model 2.1 of the License, or
#  (at your possibility) any later model.
[Unit]
Description=Multi-User System
Documentation=man:systemd.particular(7)
Requires=fundamental.goal
Conflicts=rescue.service rescue.goal
After=fundamental.goal rescue.service rescue.goal
AllowIsolate=sure
[root@testvm1 system]#

The default.goal—which is mostly a hyperlink to the multi-user.goal at this level—now has totally different necessities within the [Unit] part. It doesn’t require the graphical show supervisor.

Reboot. Your digital machine ought to boot to the console login for digital console 1, which is recognized on the show as tty1. Now that you know the way to vary the default goal, change it again to the graphical.goal utilizing a command designed for the aim.

First, test the present default goal:

[root@testvm1 ~]# systemctl get-default 
multi-user.goal
[root@testvm1 ~]# systemctl set-default graphical.goal
Removed /and so forth/systemd/system/default.goal.
Created symlink /and so forth/systemd/system/default.goal → /usr/lib/systemd/system/graphical.goal.
[root@testvm1 ~]#

Enter the next command to go on to the graphical.goal and the show supervisor login web page with out having to reboot:

[root@testvm1 system]# systemctl isolate default.goal

I have no idea why the time period “isolate” was chosen for this sub-command by systemd’s builders. My analysis signifies that it might discuss with operating the required goal however “isolating” and terminating all different targets that aren’t required to help the goal. However, the impact is to modify targets from one run goal to a different—on this case, from the emergency goal to the graphical goal. The command above is equal to the previous init 5 command in SystemV begin scripts and the init program.

Log into the GUI desktop, and confirm that it’s working because it ought to.

Summing up

This article explored the Linux systemd startup sequence and began to discover two essential systemd instruments, systemctl and journalctl. It additionally defined tips on how to swap from one goal to a different and to vary the default goal.

The subsequent article on this sequence will create a brand new systemd unit and configure it to run throughout startup. It may also take a look at a few of the configuration choices that assist decide the place within the sequence a specific unit will begin, for instance, after networking is up and operating.

Resources

There is quite a lot of details about systemd obtainable on the web, however a lot is terse, obtuse, and even deceptive. In addition to the sources talked about on this article, the next webpages supply extra detailed and dependable details about systemd startup.

• The Fedora Project has an excellent, sensible guide to systemd. It has just about every part you have to know with a purpose to configure, handle, and preserve a Fedora laptop utilizing systemd.
• The Fedora Project additionally has an excellent cheat sheet that cross-references the previous SystemV instructions to comparable systemd ones.
• For detailed technical details about systemd and the explanations for creating it, try Freedesktop.org‘s description of systemd.
• Linux.com‘s “More systemd fun” gives extra superior systemd information and tips.

There can be a sequence of deeply technical articles for Linux sysadmins by Lennart Poettering, the designer and first developer of systemd. These articles have been written between April 2010 and September 2011, however they’re simply as related now as they have been then. Much of every part else good that has been written about systemd and its ecosystem is predicated on these papers.

• Rethinking PID 1
• systemd for Administrators, Part I
• systemd for Administrators, Part II
• systemd for Administrators, Part III
• systemd for Administrators, Part IV
• systemd for Administrators, Part V
• systemd for Administrators, Part VI
• systemd for Administrators, Part VII
• systemd for Administrators, Part VIII
• systemd for Administrators, Part IX
• systemd for Administrators, Part X
• systemd for Administrators, Part XI