Topic: Kernel panic after update

[turandot]

Today I manually updated one server via remote SSH connection. During update, the SSH tunnel collapsed, the update was not complete. Logging in again, I tried to continue the update, but I dismissed to clean pending transactions i.e. to run yum-complete-transaction .

After the update, the server tried to boot the brand new kernel vmlinuz-2.6.32-696.10.2.el6.i686, and this resulted in a kernel panic:

Code: [Select]

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Pid: 1, comm: swapper Not tainted 2.6.32-696.10.2.el6.i686 #1
Call Trace:
Using the Grub boot menu, I tried an old kernel, which started without problems. Since I had to bring back the system up and running, I have decided to manually modify the grub.conf setting default=1 (instead of default=0):

Code: [Select]

[root@mysystem grub]# cat grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro rd_NO_PLYMOUTH root=/dev/mapper/main-root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=1
timeout=5
splashimage=(hd0,0)/grub/smeserver.xpm.gz
foreground 000000
background 4E95D3
title SME Server 9.2 (2.6.32-696.10.2.el6.i686)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-696.10.2.el6.i686 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS rd_LVM_LV=main/root  KEYBOARDTYPE=pc KEYTABLE=de-latin1-nodeadkeys nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 rd_MD_UUID=88c49d56:3eb48d5f:62af662b:99f835e1 LANG=de_DE.UTF-8 rd_NO_DM rhgb quiet crashkernel=auto
title SME Server 9.2 (2.6.32-696.6.3.el6.i686)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-696.6.3.el6.i686 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS rd_LVM_LV=main/root  KEYBOARDTYPE=pc KEYTABLE=de-latin1-nodeadkeys nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 rd_MD_UUID=88c49d56:3eb48d5f:62af662b:99f835e1 LANG=de_DE.UTF-8 rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-696.6.3.el6.i686.img
title SME Server 9.2 (2.6.32-696.3.2.el6.i686)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-696.3.2.el6.i686 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS rd_LVM_LV=main/root  KEYBOARDTYPE=pc KEYTABLE=de-latin1-nodeadkeys nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 rd_MD_UUID=88c49d56:3eb48d5f:62af662b:99f835e1 LANG=de_DE.UTF-8 rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-696.3.2.el6.i686.img
[root@mysystem grub]#


I do understand that this is not the best way, but at least the system is up and running again at the moment.

I am not sure whether this issue is linked to the problems during update, or whether the kernel does not work on my system. It is an Intel NUC, which is officially not supported. Therefore I get the the message

Code: [Select]

Detected CPU family 6 model 76
Warning: Intel CPU model - this hardware has not undergone testing by Red Hat an
d might not be certified. Please consult https://hardware.redhat.com for certified hardware.while booting. This message is also shown when older kernels boot successfully.

So where do I now start best with troubleshooting: is this a problem linked to the kernel itself, or did I create issues while updating the system?

While writing this up, I have just recognized that grub.conf does not point to initrd /initramfs-2.6.32-696.10.2.el6.i686.img. This file is also missing:

Code: [Select]

[root@mysystem boot]# ls -l
insgesamt 52861
-rw-r--r-- 1 root root   112821 12. Sep 15:59 config-2.6.32-696.10.2.el6.i686
-rw-r--r-- 1 root root   112820 20. Jun 02:53 config-2.6.32-696.3.2.el6.i686
-rw-r--r-- 1 root root   112820 12. Jul 15:44 config-2.6.32-696.6.3.el6.i686
drwxr-xr-x 3 root root     1024  3. Dez 2016  efi
drwxr-xr-x 2 root root     1024 22. Sep 21:20 grub
-rw------- 1 root root 17265279 21. Jun 20:26 initramfs-2.6.32-696.3.2.el6.i686.img
-rw------- 1 root root 17261064 18. Jul 22:14 initramfs-2.6.32-696.6.3.el6.i686.img
drwx------ 2 root root    12288  3. Dez 2016  lost+found
-rw-r--r-- 1 root root   211993 12. Sep 16:00 symvers-2.6.32-696.10.2.el6.i686.gz
-rw-r--r-- 1 root root   211993 20. Jun 02:54 symvers-2.6.32-696.3.2.el6.i686.gz
-rw-r--r-- 1 root root   211993 12. Jul 15:44 symvers-2.6.32-696.6.3.el6.i686.gz
-rw-r--r-- 1 root root  2064350 12. Sep 15:59 System.map-2.6.32-696.10.2.el6.i686
-rw-r--r-- 1 root root  2064268 20. Jun 02:53 System.map-2.6.32-696.3.2.el6.i686
-rw-r--r-- 1 root root  2064411 12. Jul 15:44 System.map-2.6.32-696.6.3.el6.i686
-rwxr-xr-x 1 root root  4137952 12. Sep 15:59 vmlinuz-2.6.32-696.10.2.el6.i686
-rwxr-xr-x 1 root root  4137600 20. Jun 02:53 vmlinuz-2.6.32-696.3.2.el6.i686
-rwxr-xr-x 1 root root  4137632 12. Jul 15:44 vmlinuz-2.6.32-696.6.3.el6.i686
[root@mysystem boot]#

One of the problems is that it is a remote system. To manually change the used kernel on the boot menu, I need to sit down at the local konsole...

What should I do?

Thank you,

turandot

[guest22]

As a tip, when working on a remote system via ssh, ALWAYS use screen so the processing continues even when the connection is lost. You can pick up on the same session when the connection is restored.

[Jean-Philippe Pialasse]

As a tip, when working on a remote system via ssh, ALWAYS use screen so the processing continues even when the connection is lost. You can pick up on the same session when the connection is restored.

excellent advice.
or at least if you do not manage to get the connection back, the process you started will finish cleanly (e.g. yum update...)

see screen man page

Code: [Select]

man screen

[ReetP]

HSF tip years back has saved me many a glorious disaster

[guest22]

So where do I now start best with troubleshooting: is this a problem linked to the kernel itself, or did I create issues while updating the system?

You probably did. Please see https://wiki.centos.org/TipsAndTricks/CreateNewInitrd on how to create the missing initrmfs


HTH

[Stefano]

I had such an issue in the past..

to solve it, even form remote:
- connect via ssh
- use screen
- use yum-complete-transaction to check if you still have an uncomplete yum transaction
- identify your current kernel with

Code: [Select]

uname -a
- identify the last kernel installed (but not usable) with

Code: [Select]

rpm -qa | grep kernel

you'd have something like

dracut-kernel-004-409.el6_8.2.noarch
kernel-firmware-2.6.32-696.10.2.el6.noarch
kernel-headers-2.6.32-696.10.2.el6.x86_64
kernel-2.6.32-696.10.1.el6.x86_64
kernel-2.6.32-696.10.2.el6.x86_64
kernel-2.6.32-696.1.1.el6.x86_64

in your case, you'd be using 10.1 and having issues with 10.2 (for example)

- remove the failing kernel

Code: [Select]

yum remove kernel-2.6.32-696.10.2.el6.x86_64

- try to update again with

Code: [Select]

yum update

it'd find a newer kernel and install all the needed files

- check if you have the correct kernel-headers and kernel-firmware files.. if not, update them too

I can't, right now, check if without removing the failing kernel you can simply do a

Code: [Select]

yum reinstall kernel kernel-firmware kernel-headers

HTH

[turandot]

Stefano,

many thanks for your advice. Removing and performing a new install kernel-2.6.32-696.10.2.el6.i686 and the kernel firmware as well as a reinstall for kernel header solved the issue. I also performed yet another update to now kernel-2.6.32-696.10.3.el6.i686, now all three installed kernels boot properly Naturally the update also removed my manual entry in grub.conf pointing to the older kernel.

RequestedDeletion,

thanks for pointing to screen. I knew about screen, but I have rarely used it up to now. Sometimes you have to learn it the hard way 

turandot