Tag: linux

Analyzing vmcore with crash

In the article Linux kernel crash simulation using kdump, I gave a brief idea as to how to generate a vmcore file during a crash or hangs. On this article, I will emphasize the analysis of a vmcore which has been generated and the tool ‘crash’ which can be used for advanced analysis. In a future article, I will elaborate on how to decode the detailed information given with the crash tool. Let’s see how to use the crash utility first.


tux-logo

1.Download the package kernel-debuginfo and kernel-debuginfo-common. You will notice a vmlinux file has been created just after the installation under /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux

Screenshot from 2015-11-02 12:49:34

yum install kernel-debuginfo kernel-debuginfo-common -y

2. Now, we will launch the crash utility which can be used for live debugging. By default, it will give you the info from the available vmcore.


crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686

3. However, you can specify a specific vmcore file with the following command by mentioning the location of the vmcore

crash /usr/lib/debug/lib/modules/2.6.32-573.7.1.el6.centos.plus.i686/vmlinux /boot/System.map-2.6.32-573.7.1.el6.i686 /var/crash/127.0.0.1-2015-10-30-00\:12\:34/vmcore

Screenshot from 2015-11-02 13:52:46

4. You will have several pieces of information related to the kernel as well as the most interesting stuff is what has caused the panic that is the warning message. In this case, it is a “SysRq”. If you remember from the last article we had fired an echo c > /proc/sysrq-trigger. Under the state tab, it also gave an indication of the task SYSRQ running.


5. We can also check the process running on the crash utility using the PID given.

Screenshot from 2015-11-02 14:03:396. Another interesting command is the bt which enable us to see execution history of the process

Screenshot from 2015-11-02 14:05:22

7. The sys command will give you an idea of the system. ps | grep “>” – will show you running processes during the time of the crash. mount command will show you partitions mounted etc.. 

Tips:


    • To be able to download the kernel-debuginfo package, you will need to activate the repo located at /etc/yum.repos.d
  • The version of the kernel of the machine should corroborate with that of the kernel-debug-info otherwise it will not work.

Linux Kernel-4.3 Compilation from source

The Linux Kernel 4.3 has been released today, Monday the 2nd of November 2015. I have compiled it from source on a Virtual Box CentOS 7 minimum install the virtual machine for some further testing. I have also used my same old configuration file. You can also view detailed packages and commits on the git repo. Here, is a brief idea how to compile it from source.


Linux_kernel_map
Linux Kernel Map – Photo credits Wikipedia

1. You will need to download all the pre-requirements if you are on a minimum install.


yum groupinstall "Development Tools"
yum install ncurses-devel bc hmaccalc zlib-devel elfutils-libelf-devel binutils-devel qt-devel

2. Download the wget tools to download the Kernel itself.

yum install wget

3.Download and untar the kernel directory

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.3.tar.gz
tar -xvzf linux-4.3.tar.gz

4. You will need to ensure that the decompressed directory is in the /usr/src/kernels directory. If you have untar it at a location other than this one move the Linux-4.3 directory in the /usr/src/kernels

5. Choose your default kernel configuration options


make menuconfig

6. To use the old config file

make oldconfig

7. Compiling the kernel

make

8. Installing the kernel

make modules_install install

Tips:

    • Be sure to get rid of too many old kernels files in the /boot directory to do not get confused.
    • You can also use the command make olddefconfig to set the default values without prompting anew for configuration.
    • To set different boot options use the command sudo grub2-set-default 0 – 0, in this case, is the default kernel.
    • The command make usually take lots of time. If you have 4 vCPU, you can use make like this: make -j 4 where j stands for jobs and 4 for all the 4 CPUs


  • uname -r allows you to find your kernel version. Example uname -r gives me 3.19.0-25-generic; i.e the letter 3 is the major, number 19 is the minor (developmental stage) and 0 is the revision number

Linux Kernel crash simulation using Kdump

There are several reasons for a Linux Kernel Crash which may include hangs, hardware and software errors. We usually consider a “Kernel hangs” and a “Kernel crash” as just a ‘crash’. In fact, these are totally two different issues; a “hang” occurs due to a time-consuming operation whilst a “crash” occurs instantaneously leading to a reboot. However, during the crash process prior to the reboot, the kernel will register “oops” messages.


In this article, I will lay emphasis on the installation of the tools for analyzing Linux Kernel crash. I will elaborate more on Linux Kernel errors in a future article. Right now, we will look at the installation of Kdump – Kernel dump, a Linux kernel dumping mechanism which uses a ‘kexec mechanism‘ to enable us to collect a ‘dump’ of the Linux kernel called “vmcore” (virtual memory core). Whatever event occurred during the time of the crash is registered in the “vmcore” for future analysis.


tux-logo

“Kdump uses kexec to quickly boot to a dump-capture kernel whenever a dump of the system kernel’s memory needs to be taken (for example, when the system panics). The system kernel’s memory image is preserved across the reboot and is accessible to the dump-capture kernel.”Kernel.org

Follow the steps below:

1. On both CentOS 6/7, you will need to install the kexec package using the command yum install kexec-tools

2.vim /boot/grub/grub.conf and for the kernel you are actually running edit the parameter crashkernel = auto and replace it with crashkernel= 128M (I tested it on a virtual machine with 1024MB)

3. Start the Kdump service using the command service kdump start

4. Save this parameter and verify it using the command cat /proc/cmdline. Here is a screenshot of how it should look

Screenshot from 2015-10-29 23:57:42

5. You would notice that the Kdump have the following configuration files using the command rpm -qc kexec-tools

  • /etc/kdump.conf
  • /etc/rc.d/init.d/kdump
  • /etc/sysconfig/kdump
  • /etc/udev/rules.d/98-kexec.rules

6. You can also choose the location to save your vmcore. By default, it will be saved in /var/crash/. However, if your /var directory is assigned to a different partition with low disk space, you can choose exactly where you want to generate your vmcore by modifying the parameter path /var/crash in the /etc/kdump.conf file.

7. After modification, you will need to restart the kdump service using the command service kdump restart.

8. Now the last step is to crash the machine thus creating a vmcore. Use the command echo c > /proc/sysrq-trigger. You would notice that this will take some time and the server will reboot by itself. A crash simulation has been done.

9. You will notice now after the reboot that a vmcore file has been created in the /var/crash directory.

Screenshot from 2015-10-30 00:15:18

10. The size of the vmcore depends on the consequence of the crash. In this simulation its just 19M. It also depends on the kernel activity during the time of the crash.

Tips:

  • You can also specify crashkernel = auto on a 64-bit machine. However, you can calculate it as follows:
  • If your RAM is greater than 0 GB  and less than 2 GB use 128 MB
  • If your RAM is greater than 2 GB and less than 6 GB use 256 MB
  • If your RAM is greater than 6 GB and less than 8 GB use 512 MB and so on
  • You can also test with less than 128 MB, it may work but the reliability and consistency is cautioned
  • If the kdump service does not start after a fresh installation, you might need to reboot your machine.
  • Since you have allocated a portion of the memory to the kdump, you might need to reboot your machine again and test it with a free -m


Move your /home to a new LVM partition

To have better control and security over your Linux OS, you might want to move your say /home or/var to another LVM partition. The advantage is that you can easily increase/decrease the size at a future stage.


On this article, I will take an example of the /home directory and we will move it to a fresh disk on Virtual Box. Here is an example of a df -h on my Virtual Machine.

We will now add a new disk [sdc] by creating another VG.

Screenshot from 2015-10-21 19:22:05

Start with the following steps:

    • pvcreate /dev/sdc
    • vgcreate vghome /dev/sdc
    • lvcreate -l 100%FREE vghome
  • lvrename /dev/vghome/lvol0 /dev/vghome/lvhome

Screenshot from 2015-10-21 19:27:38

Once you have successfully created the lvhome, you will need to edit your /etc/fstab before mounting the partition so that each time you reboot your machine, it will render the same configuration.



I have inserted the following parameters:

  • /dev/mapper/vghome-lvhome /home ext4 defaults,noatime 1 2

Screenshot from 2015-10-21 19:32:28



You will also need to format the partition before mounting the disk with this command :

  • mkfs.ext4 /dev/mapper/vghome-lvhome
Screenshot from 2015-10-21 19:35:49
  • You now need to mount the partition by using the following command mount /home
  • Here is now the results from a df -h
Screenshot from 2015-10-21 19:39:55
Tips:
  • You can also format your partition with ext3 when your /etc/fstab has been specified with the ext4 format as ext4 support ext3
    • You can also specify the name of the lv directly whilst creating it by using the command lvcreate -l 100%FREE -n lvhome vghome so that you don’t need to rename anew the lv. By default on Vbox and VMware, it uses lvol0
  • More articles I have posted on LVM are :

Add and extend disk on Virtual Box through LVM

Managing LVM with Pvmove – Part 1


Managing LVM with Pvmove – Part 2

Repair your Kernel Panic with Dracut

If you have encountered a Kernel Panic which usually happens after a major change in the Linux System, you can follow these procedures to rebuild the Kernel files with Dracut tools.

  1. Boot the server on rescue mode or simply through a live CD or ISO.
  2. To boot the server on rescue mode login on the Vsphere Interface and look for a live CD. In case of Kernel Panic on your own machine, you can boot your machine with a live CD.
  3. Once booted, create a directory in the folder /mnt
    mkdir /mnt/sysimage
  4. Use fdisk –l to find where is the /boot. However, you can also create another directory in mnt to mount different partitions. [sysimage is just a name given]
  5. Mount the disk into sysimage with the aim to mount the boot file. In my case, the sda1 is the boot partition
    mount /dev/sda2 /mnt/sysimage
    %MINIFYHTML72a44eecd2538f434531bc607606e0dc40%
    %MINIFYHTML72a44eecd2538f434531bc607606e0dc41%
    mount/dev/sda1 /mnt/sysimage/boot
  6. Once the disks are mounted mount the proc/dev/ and sys folders. Here are the commands:
    mount - -bind /proc /mnt/sysimage/proc
    
    mount - -bind /dev /mnt/sysimage/dev
    
    mount - -bind/sys /mnt/sysimage/sys
  7. After the mount operations have been carried out, you need to access the directory by chrooting into it.
    chroot /mnt/sysimage
  8. Get into the directory sysimage 
  9. You can back up the /boot to another location and use the command Dracut to regenerate anew the file initramfs. An example is as follows: 
    dracut -f /boot/initramfs-2.6.32-358.el6.x86_64.img 2.6.32-358.el6.x86_64
  10. You can umount all partitions and /or simply reboot the machine.
 

Repair your Kernel Panic with Dracut 1



Tips:

    • On Vcenter, you may need to boot go through the BIOS interface first before being able to boot through the ISO and force the BIOS screen to appear on your screen.
    • You may also use the Finnix ISO which is usually compatible with all UNIX system.
    • When firing the dracut command make sure you only paste the kernel version with the architecture. Do not use the file .img extension, otherwise, it won’t work – Step9
    • The last part ‘2.6.32-358.el6.x86_64’ is just the same version which needs to be regenerated. -Step9
    • To know which kernel version your machine is actually using, you need to get into the grub folder and look for the grub.conf. The first option is usually the kernel used by default.
    • Sometimes, you need to try with the same version of the OS, it may happen after you have boot your machine with a live CD, the ISO which you have used do not detect your disk or the data store. You may, for example, think the disk is not good or there is a problem in the SAN.
    • However, without doing a root cause analysis, you cannot be certain if by repairing the initrd the Kernel Panic might be the unique solution. There are circumstances where a mounted NFS is not the same version with the actual machine which can result in Kernel Panic. The Dracut solution is not a definite solution.
  • Always investigate on the Dmesg log if possible or the crash dump if same has been set up.