In my previous article, I defined how one can use ftrace to hint kernel features. Using ftrace by writing and studying from information can get tedious, so I used a wrapper round it to run instructions with choices to allow and disable tracing, set filters, view output, clear output, and extra.
The trace-cmd command is a utility that helps you just do this. In this text, I take advantage of trace-cmd to carry out the identical duties I did in my ftrace article. Since I refer again to that article regularly, I like to recommend you learn it earlier than you learn this one.
Install trace-cmd
I run the instructions on this article as the basis person.
The ftrace mechanism is constructed into the kernel, and you may confirm it’s enabled with:
# mount | grep tracefs
none on /sys/kernel/tracing sort tracefs (rw,relatime,seclabel)
However, it’s essential to set up the trace-cmd utility manually.
# dnf set up trace-cmd -yList accessible tracers
When utilizing ftrace, you could view a file’s contents to see what tracers can be found. But with trace-cmd, you will get this data with:
# trace-cmd listing -t
hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup perform nop
Enable the perform tracer
In my earlier article, I used two tracers, and I am going to do the identical right here. Enable your first tracer, perform, with:
$ trace-cmd begin -p perform
plugin 'perform'
View the hint output
Once the tracer is enabled, you possibly can view the output through the use of the present arguments. This exhibits solely the primary 20 traces to maintain the instance quick (see my earlier article for a proof of the output):
# trace-cmd present | head -20
## tracer: perform
#
# entries-in-buffer/entries-written: 410142/3380032 #P:eight
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
gdbus-2606 [004] ..s. 10520.538759: __msecs_to_jiffies <-rebalance_domains
gdbus-2606 [004] ..s. 10520.538760: load_balance <-rebalance_domains
gdbus-2606 [004] ..s. 10520.538761: idle_cpu <-load_balance
gdbus-2606 [004] ..s. 10520.538762: group_balance_cpu <-load_balance
gdbus-2606 [004] ..s. 10520.538762: find_busiest_group <-load_balance
gdbus-2606 [004] ..s. 10520.538763: update_group_capacity <-update_sd_lb_stats.constprop.zero
gdbus-2606 [004] ..s. 10520.538763: __msecs_to_jiffies <-update_group_capacity
gdbus-2606 [004] ..s. 10520.538765: idle_cpu <-update_sd_lb_stats.constprop.zero
gdbus-2606 [004] ..s. 10520.538766: __msecs_to_jiffies <-rebalance_domains
Stop tracing and clear the buffer
Tracing continues to run within the background, and you may hold viewing the output utilizing present.
To cease tracing, run trace-cmd with the cease argument:
# trace-cmd ceaseTo clear the buffer, run it with the clear argument:
# trace-cmd clearEnable the function_graph tracer
Enable the second tracer, function_graph, by working:
# trace-cmd begin -p function_graph
plugin 'function_graph'
Once once more, view the output utilizing the present argument. As anticipated, the output is barely completely different from the primary hint output. This time it features a perform calls chain:
# trace-cmd present | head -20
## tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
four) zero.079 us | } /* rcu_all_qs */
four) zero.327 us | } /* __cond_resched */
four) zero.081 us | rcu_read_unlock_strict();
four) | __cond_resched()
four) zero.080 us | rcu_read_unlock_strict();
four) | __cond_resched()
four) zero.080 us | rcu_read_unlock_strict();
four) | __cond_resched() rcu_all_qs();
four) zero.235 us
four) zero.095 us | rcu_read_unlock_strict();
four) | __cond_resched() /* __fdget */
6) zero.547 us | eventfd_poll();
6) zero.535 us | fput();
6) | __fdget()
6) | sock_poll() unix_poll();
6) 1.905 us
6) zero.475 us | fput();
6) | __fdget() {
Learn accessible features to hint
If you need to hint solely sure features and ignore the remaining, it’s essential to know the precise perform names. You can get them with the listing argument adopted by -f. This instance searches for the widespread kernel perform kmalloc, which is used to allocate reminiscence within the kernel:
# trace-cmd listing -f | grep kmalloc
bpf_map_kmalloc_node
mempool_kmalloc
__traceiter_kmalloc
__traceiter_kmalloc_node
kmalloc_slab
kmalloc_order
kmalloc_order_trace
kmalloc_large_node
__kmalloc
__kmalloc_track_caller
__kmalloc_node
__kmalloc_node_track_caller
[...]
Here’s the whole depend of features accessible on my take a look at system:
# trace-cmd listing -f | wc -l
63165
You may also hint features associated to a selected kernel module. Imagine you need to hint kvm kernel module-related features. Ensure the module is loaded:
# lsmod | grep kvm_intel
kvm_intel 335872 zero
kvm 987136 1 kvm_intel
Run trace-cmd once more with the listing argument, and from the output, grep for traces that finish in ]. This will filter out the kernel modules. Then grep the kernel module kvm_intel, and it’s best to see all of the features associated to that kernel module:
# trace-cmd listing -f | grep ]$ | grep kvm_intel
vmx_can_emulate_instruction [kvm_intel]
vmx_update_emulated_instruction [kvm_intel]
vmx_setup_uret_msr [kvm_intel]
vmx_set_identity_map_addr [kvm_intel]
handle_machine_check [kvm_intel]
handle_triple_fault [kvm_intel]
vmx_patch_hypercall [kvm_intel][...]
vmx_dump_dtsel [kvm_intel]
vmx_dump_sel [kvm_intel]
Trace particular features
Now that you know the way to search out features of curiosity, put that data to work with an instance. As within the earlier article, attempt to hint filesystem-related features. The filesystem I had on my take a look at system was ext4.
This process is barely completely different; as an alternative of begin, you run the command with the document argument adopted by the “pattern” of the features you need to hint. You additionally must specify the tracer you need; on this case, that is function_graph. The command continues recording the hint till you cease it with Ctrl+C. So after a number of seconds, hit Ctrl+C to cease tracing:
# trace-cmd listing -f | grep ^ext4_# trace-cmd document -l ext4_* -p function_graph
plugin 'function_graph'
Hit Ctrl^C to cease recording
^C
CPU0 knowledge recorded at offset=0x856000
8192 bytes in measurement
[...]
View the recorded hint
To view the hint you recorded earlier, run the command with the report argument. From the output, it is clear that the filter labored, and also you see solely the ext4-related perform hint:
# trace-cmd report | head -20
[...]
cpus=eight
trace-cmd-12697 [000] 11303.928103: funcgraph_entry: | ext4_show_options()
trace-cmd-12697 [000] 11303.928122: funcgraph_entry: | ext4_create() {
trace-cmd-12697 [000] 11303.928122: funcgraph_entry: | ext4_alloc_inode() ext4_init_pending_tree();
trace-cmd-12697 [000] 11303.928123: funcgraph_entry: zero.141 us
trace-cmd-12697 [000] 11303.928124: funcgraph_entry: zero.081 us | ext4_get_dummy_policy();
trace-cmd-12697 [000] 11303.928124: funcgraph_entry: zero.133 us | ext4_get_group_desc();
trace-cmd-12697 [000] 11303.928124: funcgraph_entry: zero.115 us | ext4_free_inodes_count();
trace-cmd-12697 [000] 11303.928124: funcgraph_entry: zero.114 us | ext4_get_group_desc();
Trace a selected PID
Say you need to hint features associated to a selected persistent identifier (PID). Open one other terminal and notice the PID of the working shell:
Run the document command once more and go the PID utilizing the -P possibility. This time, let the terminal run (i.e., don’t press Ctrl+C but):
# trace-cmd document -P 10885 -p function_graph
plugin 'function_graph'
Hit Ctrl^C to cease recording
Run some exercise on the shell
Move again to the opposite terminal the place you had a shell working with a selected PID and run any command, e.g., ls to listing information:
# ls
Temp-9b61f280-fdc1-4512-9211-5c60f764d702
tracker-extract-three-files.1000
v8-compile-cache-1000
[...]
Move again to the terminal the place you enabled tracing and hit Ctrl+C to cease tracing:
# trace-cmd document -P 10885 -p function_graph
plugin 'function_graph'
Hit Ctrl^C to cease recording
^C
CPU1 knowledge recorded at offset=0x856000
618496 bytes in measurement
[...]
In the hint’s output, you possibly can see the PID and the Bash shell on the left and the perform calls associated to it on the fitting. This may be fairly useful to slim down your tracing:
# trace-cmd report | head -20cpus=eight
<idle>-zero [001] 11555.380581: funcgraph_entry: | switch_mm_irqs_off() switch_ldt();
<idle>-zero [001] 11555.380587: funcgraph_exit: 7.235 us
bash-10885 [001] 11555.380589: funcgraph_entry: 1.046 us | finish_task_switch.isra.zero();
bash-10885 [001] 11555.380591: funcgraph_entry: | __fdget()
bash-10885 [001] 11555.380595: funcgraph_entry: | tty_poll() {
bash-10885 [001] 11555.380597: funcgraph_entry: | tty_ldisc_ref_wait() {
bash-10885 [001] 11555.380598: funcgraph_entry: | ldsem_down_read() {
bash-10885 [001] 11555.380598: funcgraph_entry: | __cond_resched() {
Give it a attempt
These temporary examples present how utilizing trace-cmd as an alternative of the underlying ftrace mechanism is each simple to make use of and wealthy in options, together with many I did not cowl right here. To be taught extra and get higher at it, seek the advice of its man web page and check out its different helpful instructions.
