Microsoft’s rising appreciation for Linux and open supply isn’t any secret. The firm has steadily elevated its contributions to open supply within the final a number of years, together with porting a few of its software program and instruments to Linux. In late 2018, Microsoft announced it was porting a few of its Sysinternals instruments to Linux as open supply, and ProcDump for Linux was the primary such launch.
If you may have labored on Windows in debugging or troubleshooting, you may have in all probability heard of Sysinternals. It is a “Swiss Army knife” toolset that helps system directors, builders, and IT safety professionals monitor and troubleshoot Windows environments.
One of Sysinternals’ hottest instruments is ProcDump. As its title suggests, it’s used for dumping the reminiscence of a working course of right into a core file on disk. This core file can then be analyzed utilizing a debugger to know the method’ state when the dump was taken. Having used Sysinternals beforehand, I used to be curious to check out the Linux port of ProcDump.
Get began with ProcDump for Linux
To strive ProcDump for Linux, you might want to obtain the device and compile it. (I’m utilizing Red Hat Enterprise Linux, although these directions ought to work the identical on different Linux distros):
$ cat /and many others/redhat-release
Red Hat Enterprise Linux launch eight.2 (Ootpa)
$
$ uname -r
four.18.Zero-193.el8.x86_64
$
First, clone the ProcDump for Linux repository:
$ git clone https://github.com/microsoft/ProcDump-for-Linux.git
Cloning into 'ProcDump-for-Linux'...
distant: Enumerating objects: 40, achieved.
distant: Counting objects: 100% (40/40), achieved.
distant: Compressing objects: 100% (33/33), achieved.
distant: Total 414 (delta 14), reused 14 (delta 6), pack-reused 374
Receiving objects: 100% (414/414), 335.28 KiB | 265.00 KiB/s, achieved.
Resolving deltas: 100% (232/232), achieved.
$
$ cd ProcDump-for-Linux/
$
$ ls
azure-pipelines.yml CONTRIBUTING.md docs INSTALL.md Makefile procdump.gif src
CODE_OF_CONDUCT.md dist embody LICENSE procdump.1 README.md exams
$
Next, construct this system utilizing make. It prints out the precise GCC command-line interface wanted to compile the supply recordsdata:
$ make
rm -rf obj
rm -rf bin
rm -rf /root/ProcDump-for-Linux/pkgbuild
gcc -c -g -o obj/Logging.o src/Logging.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/Events.o src/Events.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/ProcDumpConfiguration.o src/ProcDumpConfiguration.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/Handle.o src/Handle.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/Process.o src/Process.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/Procdump.o src/Procdump.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/TriggerThreadProcs.o src/TriggerThreadProcs.c -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/CoreDumpWriter.o src/CoreDumpWriter.c -Wall -I ./embody -pthread -std=gnu99
gcc -o bin/procdump obj/Logging.o obj/Events.o obj/ProcDumpConfiguration.o obj/Handle.o obj/Process.o obj/Procdump.o obj/TriggerThreadProcs.o obj/CoreDumpWriter.o -Wall -I ./embody -pthread -std=gnu99
gcc -c -g -o obj/ProcDumpTestApplication.o exams/integration/ProcDumpTestApplication.c -Wall -I ./embody -pthread -std=gnu99
gcc -o bin/ProcDumpTestApplication obj/ProcDumpTestApplication.o -Wall -I ./embody -pthread -std=gnu99
$
The compilation creates two new directories. First is an obj/ listing, which holds the item recordsdata created throughout compilation. The second (and extra essential) listing is bin/, which is the place the compiled procdump program is saved. It additionally compiles one other take a look at binary referred to as ProcDumpTestApplication:
$ ls obj/
CoreDumpWriter.o Handle.o ProcDumpConfiguration.o ProcDumpTestApplication.o TriggerThreadProcs.o
Events.o Logging.o Procdump.o Process.o
$
$
$ ls bin/
procdump ProcDumpTestApplication
$
$ file bin/procdump
bin/procdump: ELF 64-bit LSB executable, x86-64, model 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux three.2.Zero, BuildID[sha1]=6e8827db64835ea0d1f0941ac3ecff9ee8c06e6b, with debug_info, not stripped
$
$ file bin/ProcDumpTestApplication
bin/ProcDumpTestApplication: ELF 64-bit LSB executable, x86-64, model 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux three.2.Zero, BuildID[sha1]=c8fd86f53c07df142e52518815b2573d1c690e4e, with debug_info, not stripped
$
With this setup, each time you run the procdump utility, it’s essential to transfer into the bin/ folder. To make it accessible from anyplace inside the system, run make set up. This copies the binary into the same old bin/ listing, which is a part of your shell’s $PATH:
$ which procdump
/usr/bin/which: no procdump in (/usr/native/sbin:/usr/native/bin:/usr/sbin:/usr/bin)
$
$ make set up
mkdir -p //usr/bin
cp bin/procdump //usr/bin
mkdir -p //usr/share/man/man1
cp procdump.1 //usr/share/man/man1
$
$ which procdump
/usr/bin/procdump
$
With set up, ProcDump offers a person web page, which you’ll entry with man procdump:
Run ProcDump
To dump a course of’ reminiscence, you might want to present its course of ID (PID) to ProcDump. You can use any of the working applications or daemons in your machine. For this instance, I’ll use a tiny C program that loops ceaselessly. Compile this system and run it (to exit this system, hit Ctrl+C, or if it is working within the background, use the kill command with the PID):
$ cat progxyz.c
#embody <stdio.h>int predominant()
for (;;)
return Zero;
$
$ gcc progxyz.c -o progxyz
$
$ ./progxyz &
[1] 350498
$
By working this system, you could find its PID utilizing both pgrep or ps. Make observe of the PID:
$ pgrep progxyz
350498
$
$ ps -ef | grep progxyz
root 350498 345445 Zero 03:29 pts/1 00:00:00 ./progxyz
root 350508 347350 Zero 03:29 pts/Zero 00:00:00 grep --color=auto progxyz
$
While the take a look at course of is working, invoke procdump and supply the PID. The output states the title of the method and the PID, experiences that a Core dump was generated, and exhibits its file title:
$ procdump -p 350498ProcDump v1.1.1 - Sysinternals course of dump utility
Copyright (C) 2020 Microsoft Corporation. All rights reserved. Licensed beneath the MIT license.
Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi
Monitors a course of and writes a dump file when the method exceeds the
specified standards.Process: progxyz (350498)
CPU Threshold: n/a
Commit Threshold: n/a
Polling interval (ms): 1000
Threshold (s): 10
Number of Dumps: 1Press Ctrl-C to finish monitoring with out terminating the method.
[03:30:00 - INFO]: Timed:
[03:30:01 - INFO]: Core dump Zero generated: progxyz_time_2020-06-24_03:30:00.350498
$
List the contents of the present listing, and it is best to see the brand new core file. The file title matches the one proven by the procdump command, and the date, time, and PID are appended to it:
$ ls -l progxyz_time_2020-06-24_03:30:00.350498
-rw-r--r--. 1 root root 356848 Jun 24 03:30 progxyz_time_2020-06-24_03:30:00.350498
$
$ file progxyz_time_2020-06-24_03:30:00.350498
progxyz_time_2020-06-24_03:30:00.350498: ELF 64-bit LSB core file, x86-64, model 1 (SYSV), SVR4-style, from './progxyz', actual uid: Zero, efficient uid: Zero, actual gid: Zero, efficient gid: Zero, execfn: './progxyz', platform: 'x86_64'
$
Analyze the core file with the GNU Project Debugger
To see if you happen to can learn the proc file, invoke the GNU Project Debugger (gdb). Remember to supply the take a look at binary’s path so you may see all of the perform names on the stack. Here, bt (backtrace) exhibits that the sleep() perform was being executed when the dump was taken:
$ gdb -q ./progxyz ./progxyz_time_2020-06-24_03:30:00.350498
Reading symbols from ./progxyz...(no debugging symbols discovered)...achieved.
[New LWP 350498]
Core was generated by `./progxyz'.
#Zero 0x00Zero07fb6947e9208 in nanosleep () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-101.el8.x86_64
(gdb) bt
#Zero 0x00Zero07fb6947e9208 in nanosleep () from /lib64/libc.so.6
#1 0x00Zero07fb6947e913e in sleep () from /lib64/libc.so.6
#2 0x00000000004005f3 in predominant ()
(gdb)
What about gcore?
Linux customers might be fast to level out that Linux already has a command referred to as gcore, which ships with most Linux distros and does the very same factor as ProcDump. This is a sound argument. If you may have by no means used it, strive the next to dump a course of’ core with gcore. Run the take a look at program once more, then run gcore, and supply the PID as an argument:
$ ./progxyz &
[1] 350664
$
$
$ pgrep progxyz
350664
$
$
$ gcore 350664
0x00Zero07fefd3be2208 in nanosleep () from /lib64/libc.so.6
Saved corefile core.350664
[Inferior 1 (course of 350664) indifferent]
$
gcore prints a message saying it has saved the core to a particular file. Check the present listing to seek out this core file, and use gdb once more to load it:
$
$ ls -l core.350664
-rw-r--r--. 1 root root 356848 Jun 24 03:34 core.350664
$
$
$ file core.350664
core.350664: ELF 64-bit LSB core file, x86-64, model 1 (SYSV), SVR4-style, from './progxyz', actual uid: Zero, efficient uid: Zero, actual gid: Zero, efficient gid: Zero, execfn: './progxyz', platform: 'x86_64'
$
$ gdb -q ./progxyz ./core.350664
Reading symbols from ./progxyz...(no debugging symbols discovered)...achieved.
[New LWP 350664]
Core was generated by `./progxyz'.
#Zero 0x00Zero07fefd3be2208 in nanosleep () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-101.el8.x86_64
(gdb) bt
#Zero 0x00Zero07fefd3be2208 in nanosleep () from /lib64/libc.so.6
#1 0x00Zero07fefd3be213e in sleep () from /lib64/libc.so.6
#2 0x00000000004005f3 in predominant ()
(gdb) q
$
For gcore to work, you might want to ensure the next settings are in place. First, make sure the ulimit is about for core recordsdata; whether it is set to Zero, core recordsdata will not be generated. Second, be sure that /proc/sys/kernel/core_pattern has the right settings to specify the core sample:
Should you utilize ProcDump or gcore?
There are a number of instances the place you may desire utilizing ProcDump as an alternative of gcore, and ProcDump has just a few built-in options that is likely to be helpful usually.
Waiting for a take a look at binary to execute
Whether you utilize ProcDump or gcore, the take a look at course of should be executed and in a working state so that you could present a PID to generate a core file. But ProcDump has a characteristic that waits till a particular binary runs; as soon as it finds a take a look at binary working that matches that given title, it generates a core file for that take a look at binary. It might be enabled utilizing the -w argument and this system’s title as an alternative of a PID. This characteristic might be helpful in situations the place the take a look at program exits shortly.
Here’s the way it works. In this instance, there isn’t any course of named progxyz working:
Invoke procdump with the -w command to maintain it ready. From one other terminal, invoke the take a look at binary progxyz:
$ procdump -w progxyzProcDump v1.1.1 - Sysinternals course of dump utility
Copyright (C) 2020 Microsoft Corporation. All rights reserved. Licensed beneath the MIT license.
Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi
Monitors a course of and writes a dump file when the method exceeds the
specified standards.Process: progxyz (pending)
CPU Threshold: n/a
Commit Threshold: n/a
Polling interval (ms): 1000
Threshold (s): 10
Number of Dumps: 1Press Ctrl-C to finish monitoring with out terminating the method.
[03:39:23 - INFO]: Waiting for course of 'progxyz' to launch...
Then, from one other terminal, invoke the take a look at binary progxyz:
ProcDump instantly detects that the binary is working and dumps the core file for this binary:
[03:39:23 - INFO]: Waiting for course of 'progxyz' to launch...
[03:43:22 - INFO]: Found course of with PID 350951
[03:43:22 - INFO]: Timed:
[03:43:23 - INFO]: Core dump Zero generated: progxyz_time_2020-06-24_03:43:22.350951
$$ ls -l progxyz_time_2020-06-24_03:43:22.350951
-rw-r--r--. 1 root root 356848 Jun 24 03:43 progxyz_time_2020-06-24_03:43:22.350951
$
$ file progxyz_time_2020-06-24_03:43:22.350951
progxyz_time_2020-06-24_03:43:22.350951: ELF 64-bit LSB core file, x86-64, model 1 (SYSV), SVR4-style, from './progxyz', actual uid: Zero, efficient uid: Zero, actual gid: Zero, efficient gid: Zero, execfn: './progxyz', platform: 'x86_64'
$
Multiple core dumps
Another essential ProcDump characteristic is which you could specify what number of core recordsdata to generate through the use of the command-line argument -n <rely>. The default time hole between the core dumps is 10 seconds, however you may modify this utilizing the -s <sec> argument. This instance makes use of ProcDump to take three core dumps of the take a look at binary:
$ ./progxyz &
[1] 351014
$
$ procdump -n three -p 351014ProcDump v1.1.1 - Sysinternals course of dump utility
Copyright (C) 2020 Microsoft Corporation. All rights reserved. Licensed beneath the MIT license.
Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi
Monitors a course of and writes a dump file when the method exceeds the
specified standards.Process: progxyz (351014)
CPU Threshold: n/a
Commit Threshold: n/a
Polling interval (ms): 1000
Threshold (s): 10
Number of Dumps: threePress Ctrl-C to finish monitoring with out terminating the method.
[03:45:20 - INFO]: Timed:
[03:45:21 - INFO]: Core dump Zero generated: progxyz_time_2020-06-24_03:45:20.351014
[03:45:31 - INFO]: Timed:
[03:45:32 - INFO]: Core dump 1 generated: progxyz_time_2020-06-24_03:45:31.351014
[03:45:42 - INFO]: Timed:
[03:45:44 - INFO]: Core dump 2 generated: progxyz_time_2020-06-24_03:45:42.351014
$
$ ls -l progxyz_time_2020-06-24_03:45:*
-rw-r--r--. 1 root root 356848 Jun 24 03:45 progxyz_time_2020-06-24_03:45:20.351014
-rw-r--r--. 1 root root 356848 Jun 24 03:45 progxyz_time_2020-06-24_03:45:31.351014
-rw-r--r--. 1 root root 356848 Jun 24 03:45 progxyz_time_2020-06-24_03:45:42.351014
$
Core dump based mostly on CPU and reminiscence utilization
ProcDump additionally allows you to set off a core dump when a take a look at binary or course of reaches a sure CPU or reminiscence threshold. ProcDump’s man web page exhibits the command-line arguments to make use of when invoking ProcDump:
-C Trigger core dump technology when CPU exceeds or equals specified worth (Zero to 100 * nCPU)
-c Trigger core dump technology when CPU is lower than specified worth (Zero to 100 * nCPU)
-M Trigger core dump technology when reminiscence commit exceeds or equals specified worth (MB)
-m Trigger core dump technology when when reminiscence commit is lower than specified worth (MB)
-T Trigger when thread rely exceeds or equals specified worth.
-F Trigger when filedescriptor rely exceeds or equals specified worth.
-I Polling frequency in milliseconds (default is 1000)
For instance, you may ask ProcDump to dump the core when the given PID’s CPU utilization exceeds 70%:
procdump -C 70 -n three -p 351014Conclusion
ProcDump is an fascinating addition to the lengthy checklist of Windows applications being ported to Linux. Not solely does it present further tooling choices to Linux customers, however it will probably additionally make Windows customers really feel extra at residence when engaged on Linux.