Analyzing vmcore with crash

In the article linux kernel crash simulation using kdump, i gave a brief idea as to how to generate a vmcore file during a crash or hangs. On this article, i will emphasize on the analysis of a vmcore which has been generated and the tool 'crash' which can be used for advance analysis. In a future article, i will elaborate on how to decode the detailed information given with the crash tool. Lets see how to use the crash utility first.

tux-logo

1.Download the package kernel-debuginfo and kernel-debuginfo-common. You will notice a vmlinux file has been created just after the installation under /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux

Screenshot from 2015-11-02 12:49:34

yum install kernel-debuginfo kernel-debuginfo-common -y

2. Now, we will launch the crash utility which can be used for live debugging. By default, it will give you the info from the available vmcore.

crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686

3. However, you can specify a specific vmcore file with the following command by mentioning the location of the vmcore

crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686 /var/crash/127.0.0.1-2015-10-30-00\:12\:34/vmcore

Screenshot from 2015-11-02 13:52:46

4. You will have several information related to the kernel as well as the most interesting stuff is what have cause the panic that is the warning message. In this case it is a "SysRq". If you remember from the last article we had fired an echo c > /proc/sysrq-trigger. Under the state tab it also gave an indication of the task SYSRQ running.

5. We can also check the process running on the crash utility using the PID given.

Screenshot from 2015-11-02 14:03:396. Another interesting command is the bt which enable us to see execution history of the process

Screenshot from 2015-11-02 14:05:22

7. The sys command will give you an idea of the system. ps | grep ">" - will show you running processes during time of the crash. mount command will show you partitions mounted etc..  h command for the history.

Tips:

  • A good crash utility manual page can be found at people.redhat.com/anderson. Almost all info is available there.
  • To be able to dowload the kernel-debuginfo package, you will need to activate the repo located at /etc/yum.repos.d
  • The version of the kernel of the machine should corroborate with that of the kernel-debug-info otherwise it will not work.