In the article Linux kernel crash simulation using kdump, I gave a brief idea as to how to generate a vmcore file during a crash or hangs. On this article, I will emphasize the analysis of a vmcore which has been generated and the tool ‘crash’ which can be used for advanced analysis. In a future article, I will elaborate on how to decode the detailed information given with the crash tool. Let’s see how to use the crash utility first.
1.Download the package kernel-debuginfo and kernel-debuginfo-common. You will notice a vmlinux file has been created just after the installation under /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux
yum install kernel-debuginfo kernel-debuginfo-common -y
2. Now, we will launch the crash utility which can be used for live debugging. By default, it will give you the info from the available vmcore.
crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686
3. However, you can specify a specific vmcore file with the following command by mentioning the location of the vmcore
crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686 /var/crash/127.0.0.1-2015-10-30-00\:12\:34/vmcore
4. You will have several pieces of information related to the kernel as well as the most interesting stuff is what has caused the panic that is the warning message. In this case, it is a “SysRq”. If you remember from the last article we had fired an echo c > /proc/sysrq-trigger. Under the state tab, it also gave an indication of the task SYSRQ running.
5. We can also check the process running on the crash utility using the PID given.
6. Another interesting command is the bt which enable us to see execution history of the process
7. The sys command will give you an idea of the system. ps | grep “>” – will show you running processes during the time of the crash. mount command will show you partitions mounted etc..
Tips:
-
- A good crash utility manual page can be found at people.redhat.com/anderson[hyperlink removed in 2023 as subdomain is dead]. Almost all info is available there.
-
- To be able to download the kernel-debuginfo package, you will need to activate the repo located at /etc/yum.repos.d
- The version of the kernel of the machine should corroborate with that of the kernel-debug-info otherwise it will not work.
