Linux debugging core dump - (Part 2)

A more modern crash reporting system

1. systemd-coredump

systemd-coredump can collect and display kernel core dumps for analyzing application crash problems. When a process crashes, it can automatically collect and save the corresponding core dumps. At the same time, you can use the client tool coredumpctl to manage core dumps and view them. and analysis.

• Install systemd-coredump
  sudo apt install systemd-coredump
  • core dumps are stored in /var/lib/systemd/coredump
  • Automatically install the coredumpctl tool
• Configuration
  • /etc/systemd/coredump.conf
    • systemd-coredump will automatically delete core dumps that exceed configurable size limits (2 GB by default).
    • It also deletes core dumps if your free disk space falls below a configurable threshold (15% free by default).
    • Additionally, systemd-tmpfiles will delete core dumps automatically after some time has passed (three days by default).

[Coredump]
#Storage=external
#Compress=yes
#ProcessSizeMax=2G
#ExternalSizeMax=2G
#JournalSizeMax=767M
#MaxUse=
#KeepFree=

• Common commands of the client tool coredumpctl
  • coredumpctl [List all core dumps]
  • coredumpctl gdb [Open recent core dumps]
  • coredumpctl gdb 1234 [Open the latest core dumps of the pid=1234 process]

2. Examples

• test environment
  • ubuntu 18.04
  • g++ 7.5.0
• Test code【segfault.cpp】

#include <iostream>

void test_crash(int a, int b)
{
	std::cout << __func__ << " enter" << std::endl;
	int *ptr = nullptr;
	*ptr = 0;
	return ;
}

int main(void)
{
	std::cout << "hello crash" << std::endl;
	test_crash(0, 0);

	return 0;
}

• Compilation command
  g++ -g segfault.cpp

• run
  ./a.out

• coredumpctl gdbView crash stacktrace

PID: 27711 (a.out)
           UID: 1000 (ubuntu)
           GID: 1000 (ubuntu)
        Signal: 11 (SEGV)
     Timestamp: Mon 2022-08-22 19:06:31 CST (20min ago)
  Command Line: ./a.out
    Executable: /home/ubuntu/work/Linux_4.6.131/lava/sdk/a.out
 Control Group: /user.slice/user-1000.slice/session-1236.scope
          Unit: session-1236.scope
         Slice: user-1000.slice
       Session: 1236
     Owner UID: 1000 (ubuntu)
       Boot ID: 6488216f2d4c465eaff663978d90786a
    Machine ID: 350062f635cd4e9a95051d7f498230c9
      Hostname: ubuntu-Precision-3630-Tower
       Storage: /var/lib/systemd/coredump/core.a\x2eout.1000.6488216f2d4c465eaff663978d90786a.27711.1661166391000000.lz4
       Message: Process 27711 (a.out) of user 1000 dumped core.

                Stack trace of thread 27711:
                #0  0x0000556a6dc0290b n/a (/home/ubuntu/work/Linux_4.6.131/lava/sdk/a.out)
                #1  0x0000556a6dc0294f n/a (/home/ubuntu/work/Linux_4.6.131/lava/sdk/a.out)
                #2  0x00007f254a935c87 __libc_start_main (libc.so.6)
                #3  0x0000556a6dc027da n/a (/home/ubuntu/work/Linux_4.6.131/lava/sdk/a.out)

GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/ubuntu/work/Linux_4.6.131/lava/sdk/a.out...done.
[New LWP 27711]
Core was generated by `./a.out'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000556a6dc0290b in test_crash (a=0, b=0) at segfault.cpp:7
7               *ptr = 0;
(gdb)

3. Conclusion

• Stop using manual coredump, use systemd-coredump instead
  • Manually set ulimit -c unlimited, only valid for the current shell
  • Manually close apport (ubuntu) to generate coredump

4. Reference

1. Creating Quality Backtraces for Crash Reports
2.systemd-coredump, systemd-coredump.socket, [email protected] — Acquire, save and process core dumps