Creating and debugging Linux dump recordsdata

Crash dump, reminiscence dump, core dump, system dump … all produce the identical end result: a file containing the state of an utility’s reminiscence at a particular time—normally when the applying crashes.

Knowing find out how to cope with these recordsdata might help you discover the foundation trigger(s) of a failure. Even in case you are not a developer, dump recordsdata created in your system may be very useful (in addition to approachable) in understanding software program.

This is a hands-on article, and may you comply with together with the instance by cloning the pattern utility repository with:

git clone https://github.com/hANSIc99/core_dump_example.git

How alerts relate to dumps

Signals are a type of interprocess communication between the working system and the person functions. Linux makes use of the alerts outlined within the POSIX standard. On your system, you will discover the usual alerts outlined in /usr/embrace/bits/signum-generic.h. There can be an informative man signal web page if you need extra on utilizing alerts in your utility. Put merely, Linux makes use of alerts to set off additional actions primarily based on whether or not they have been anticipated or sudden.

When you stop a working utility, the applying will normally obtain the SIGTERM sign. Because such a exit sign is predicted, this motion is not going to create a reminiscence dump.

The following alerts will trigger a dump file to be created (supply: GNU C Library):

• SIGFPE: Erroneous arithmetic operation
• SIGILL: Illegal instruction
• SIGSEGV: Invalid entry to storage
• SIGBUS: Bus error
• SIGABRT: An error detected by this system and reported by calling abort
• SIGIOT: Labeled archaic on Fedora, this sign used to set off on abort() on a PDP-11 and now maps to SIGABRT

Creating dump recordsdata

Navigate to the core_dump_example listing, run make, and execute the pattern with the -c1 change:

./coredump -c1

The utility ought to exit in state four with an error:

“Abgebrochen (Speicherabzug geschrieben)” roughly interprets to “Segmentation fault (core dumped).”

Whether it creates a core dump or not is set by the useful resource restrict of the person working the method. You can modify the useful resource limits with the ulimit command.

Check the present setting for core dump creation:

ulimit -c

If it outputs limitless, then it’s utilizing the (beneficial) default. Otherwise, appropriate the restrict with:

ulimit -c limitless

To disable creating core dumps’ sort:

ulimit -c zero

The quantity specifies the useful resource in kilobytes.

What are core dumps?

The manner the kernel handles core dumps is outlined in:

/proc/sys/kernel/core_pattern

I am working Fedora 31, and on my system, the file comprises:

/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

This reveals core dumps are forwarded to the systemd-coredump utility. The contents of core_pattern can range broadly between the completely different flavors of Linux distributions. When systemd-coredump is in use, the dump recordsdata are saved compressed beneath /var/lib/systemd/coredump. You needn’t contact the recordsdata immediately; as a substitute, you should use coredumpctl. For instance:

coredumpctl listing

reveals all accessible dump recordsdata saved in your system.

With coredumpctl dump, you’ll be able to retrieve info from the final dump file saved:

[stephan@localhost core_dump_example]$ ./coredump 

Application began…
(…….)
Message: Process 4598 (coredump) of person 1000 dumped core.
Stack hint of thread 4598:
#zero 0x00007f4bbaf22625 __GI_raise (libc.so.6)
#1 0x00007f4bbaf0b8d9 __GI_abort (libc.so.6)
#2 0x00007f4bbaf664af __libc_message (libc.so.6)
#three 0x00007f4bbaf6da9c malloc_printerr (libc.so.6)
#four 0x00007f4bbaf6f49c _int_free (libc.so.6)
#5 0x000000000040120e n/a (/house/stephan/Dokumente/core_dump_example/coredump)
#6 0x00000000004013b1 n/a (/house/stephan/Dokumente/core_dump_example/coredump)
#7 0x00007f4bbaf0d1a3 __libc_start_main (libc.so.6)
#eight 0x000000000040113e n/a (/house/stephan/Dokumente/core_dump_example/coredump)
Refusing to dump core to tty (use shell redirection or specify — output).

This reveals that the method was stopped by SIGABRT. The stack hint on this view isn’t very detailed as a result of it doesn’t embrace operate names. However, with coredumpctl debug, you’ll be able to merely open the dump file with a debugger (GDB by default). Type bt (quick for backtrace) to get a extra detailed view:

Core was generated by `./coredump -c1'.
Program terminated with sign SIGABRT, Aborted.
#zero  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/increase.c:50
50  return ret;
(gdb) bt
#zero  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/increase.c:50
#1  0x00007fc37a9aa8d9 in __GI_abort () at abort.c:79
#2  0x00007fc37aa054af in __libc_message (motion=motion@entry=do_abort, fmt=fmt@entry=0x7fc37ab14f4b "%sn") at ../sysdeps/posix/libc_fatal.c:181
#three  0x00007fc37aa0ca9c in malloc_printerr (str=str@entry=0x7fc37ab130e0 "free(): invalid pointer") at malloc.c:5339
#four  0x00007fc37aa0e49c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=zero) at malloc.c:4173
#5  0x000000000040120e in freeSomething(void*) ()
#6  0x0000000000401401 in fundamental ()

The reminiscence addresses: fundamental() and freeSomething() are fairly low in comparison with subsequent frames. Due to the truth that shared objects are mapped to an space on the finish of the digital handle house, you’ll be able to assume that the SIGABRT was brought on by a name in a shared library. Memory addresses of shared objects usually are not fixed between invocations, so it’s completely fantastic whenever you see various addresses between calls.

The stack hint reveals that subsequent calls originate from malloc.c, which signifies that one thing with reminiscence (de-)allocation might have gone improper.

In the supply code, you’ll be able to see (even with none information of C++) that it tried to free a pointer, which was not returned by a reminiscence administration operate. This leads to undefined habits and causes the SIGABRT:

void freeSomething(void *ptr)
int nTmp = 5;
int *ptrNull = &nTmp;

freeSomething(ptrNull);

The systemd coredump utility may be configured beneath /and many others/systemd/coredump.conf. Rotation of dump file cleansing may be configured in /and many others/systemd/system/systemd-tmpfiles-clean.timer.

You can discover extra details about coredumpctl on its man page.

Compiling with debug symbols

Open the Makefile and remark out the final a part of line 9. It ought to now seem like:

CFLAGS =-Wall -Werror -std=c++11 -g

The -g change allows the compiler to create debug info. Start the applying, this time with the -c2 change:

./coredump -c2

You will get a floating-point exception. Open the dump in GDB with:

coredumpctl debug

This time, you might be pointed on to the road within the supply code that precipitated the error:

Reading symbols from /house/stephan/Dokumente/core_dump_example/coredump…
[New LWP 6218]
Core was generated by `./coredump -c2'.
Program terminated with sign SIGFPE, Arithmetic exception.
#zero 0x0000000000401233 in zeroDivide () at fundamental.cpp:29
29 nRes = 5 / nDivider;
(gdb)

Type listing to get a greater overview of the supply code:

(gdb) listing
24      int zeroDivide()
25          int nDivider = 5;
26          int nRes = zero;
27          whereas(nDivider > zero)
31          return nRes;
32      

Use the command information locals to retrieve the values of the native variables from the cut-off date when the applying failed:

(gdb) information locals
nDivider = zero
nRes = 5

In mixture with the supply code, you’ll be able to see that you simply ran right into a division by zero:

nRes = 5 / zero

Conclusion

Knowing find out how to cope with dump recordsdata will aid you discover and repair hard-to-reproduce random bugs in an utility. And if it’s not your utility, forwarding a core dump to the developer will assist him or her discover and repair the issue.