Topic: Kernel Panic on new kernels

[Jáder]

Hi,

I´m with a kernel panic error in a very stable server (HP ML110).

edit: add screens to ilustrate the problem: (I´ll remove them later)

Kernel panic with 2.6.32-754-9.1
https://radio.antinsect.com.br/SME92-kernelPanic-2.6.32-754-9.1.el6.x86_64.jpg

Kernel panic with 2.6.32-754-10.1
https://radio.antinsect.com.br/SME92-kernelPanic-2.6.32-754-10.1.el6.x86_64.jpg

Boot Menu
https://radio.antinsect.com.br/SME92-kernelPanic-BootMenu.jpg



This is my boot menu:

Code: [Select]

[root@andorinha ~]# cat /boot/grub/menu.lst.ajm
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro rd_NO_PLYMOUTH root=/dev/mapper/main-root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/smeserver.xpm.gz
foreground 000000
background 4E95D3
title SME Server 9.2 (2.6.32-754.10.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-754.10.1.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-754.10.1.el6.x86_64.img
title SME Server 9.2 (2.6.32-754.9.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-754.9.1.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-754.9.1.el6.x86_64.img
title SME Server 9.2 (2.6.32-754.6.3.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-754.6.3.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-754.6.3.el6.x86_64.img

I´ll had to remove the first 2 entries to avoid problems (last reboots , I´ve to choose manually the last option) and also will remove kernel package because all RPM keep only 3 versions ... so I´m about to loose last option and it ´s the only working.

I have these packages installed:

Code: [Select]

[root@andorinha ~]# rpm -qa|grep kernel
kernel-2.6.32-754.9.1.el6.x86_64
kernel-devel-2.6.32-754.6.3.el6.x86_64
kernel-2.6.32-754.10.1.el6.x86_64
dracut-kernel-004-411.el6.noarch
kernel-2.6.32-754.6.3.el6.x86_64
kernel-firmware-2.6.32-754.10.1.el6.noarch
kernel-headers-2.6.32-754.10.1.el6.x86_64
kernel-devel-2.6.32-754.9.1.el6.x86_64
kernel-devel-2.6.32-754.10.1.el6.x86_64
and now removindo olde kernels -9 (to keep on safe side and avoid any other RPM package remove my working kernel package)

Code: [Select]

[root@andorinha ~]# rpm -e kernel-devel-2.6.32-754.9.1.el6.x86_64 kernel-2.6.32-754.9.1.el6.x86_64
warning:    erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/updates failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.order failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.networking failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.modesetting failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.drm failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.block failed: No such file or directory
and verifying the remaining kernel packages:

Code: [Select]

[root@andorinha ~]# rpm -qa|grep kernel
kernel-devel-2.6.32-754.6.3.el6.x86_64
kernel-2.6.32-754.10.1.el6.x86_64
dracut-kernel-004-411.el6.noarch
kernel-2.6.32-754.6.3.el6.x86_64
kernel-firmware-2.6.32-754.10.1.el6.noarch
kernel-headers-2.6.32-754.10.1.el6.x86_64
kernel-devel-2.6.32-754.10.1.el6.x86_64
and now updating menu.lst

Code: [Select]

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro rd_NO_PLYMOUTH root=/dev/mapper/main-root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/smeserver.xpm.gz
foreground 000000
background 4E95D3
title SME Server 9.2 (2.6.32-754.6.3.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-754.6.3.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-754.6.3.el6.x86_64.img
title Jm diz RUIM - SME Server 9.2 (2.6.32-754.10.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-754.10.1.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto
        initrd /initramfs-2.6.32-754.10.1.el6.x86_64.img

Note I´ve increased timeout to allow user to see menu and choose right one (the oldest) if necessary and removed the 754-9 option, because I removed it as package using RPM.

Anyone had same problem ?
How can I diagnose the kernel panic error ?

Thanks,

Jáder

[ReetP]

First question I can think of is why have you got kernel-devel packages installed??

Have you been rolling your own stuff?

Next, have you had a read of some links on the interwbs?

Have a search for say:

kernel 2.6.32-754 panic RAMDISK incomplete write

This is a pain.... behind their firewall as usual.

https://access.redhat.com/solutions/715833

Insufficent /boot space?

https://www.centos.org/forums/viewtopic.php?t=3864

Maybe causing corruption of the intramfs image?

http://www.thesmartadmin.com/ramdisk-corruption-causes-new-rhel-or-centos-to-fail

[TerryF]

Anyone had same problem ?
How can I diagnose the kernel panic error ?

No issues, HP N54L
[root@xxxxxx ~]# uname -r
2.6.32-754.10.1.el6.x86_64

[root@xxxxxx ~]# rpm -qa|grep kernel
dracut-kernel-004-411.el6.noarch
kernel-2.6.32-754.6.3.el6.x86_64
kernel-firmware-2.6.32-754.10.1.el6.noarch
kernel-2.6.32-754.9.1.el6.x86_64
kernel-2.6.32-754.10.1.el6.x86_64
kernel-headers-2.6.32-754.10.1.el6.x86_64

What ReetP said...

[Jáder]

First question I can think of is why have you got kernel-devel packages installed??

Have you been rolling your own stuff?

I'm really not sure. I'm not developing nothing!
EDIT: I think I've found out: I'm using VirtualBox and most of time it need to be compiled to load when new kernel is installed. Anyway that would not represent a threat or failure point se early on boot process.

Next, have you had a read of some links on the interwbs?
Have a search for say:

kernel 2.6.32-754 panic RAMDISK incomplete write

This is a pain.... behind their firewall as usual.

https://access.redhat.com/solutions/715833

I'll start reading now

Insufficent /boot space?
https://www.centos.org/forums/viewtopic.php?t=3864
Maybe causing corruption of the intramfs image?
http://www.thesmartadmin.com/ramdisk-corruption-causes-new-rhel-or-centos-to-fail

No
There are lot's of boot space , at least now I removed the other kernel.

Code: [Select]

[root@andorinha ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                      1.8T  746G  988G  44% /
tmpfs                 3.9G  666M  3.2G  18% /dev/shm
/dev/md0              239M   53M  174M  24% /boot


[Jáder]

No issues, HP N54L
[root@xxxxxx ~]# uname -r
2.6.32-754.10.1.el6.x86_64

[root@xxxxxx ~]# rpm -qa|grep kernel
dracut-kernel-004-411.el6.noarch
kernel-2.6.32-754.6.3.el6.x86_64
kernel-firmware-2.6.32-754.10.1.el6.noarch
kernel-2.6.32-754.9.1.el6.x86_64
kernel-2.6.32-754.10.1.el6.x86_64
kernel-headers-2.6.32-754.10.1.el6.x86_64

What ReetP said...

I have several N40L not showing problem at all. Regards...

[Jean-Philippe Pialasse]

I remember having seen something similar here or on CentOs forums. Was a failure compiling the initrd on the update before reboot.
Strange it has happen on two kernel in a row.
as far as i remember the quick fix was to rebuild the initrd manually

[Jean-Philippe Pialasse]

No
There are lot's of boot space , at least now I removed the other kernel.

Code: [Select]

[root@andorinha ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                      1.8T  746G  988G  44% /
tmpfs                 3.9G  666M  3.2G  18% /dev/shm
/dev/md0              239M   53M  174M  24% /boot


I would head that direction
53M  is not a lot, and even less sufficient if you had you just removed things to get that.
each new kernel will need at least 25Mo
SO this is probable the last two kernels had an issue for space.

[Jáder]

I would head that direction
53M  is not a lot, and even less sufficient if you had you just removed things to get that.
each new kernel will need at least 25Mo
SO this is probable the last two kernels had an issue for space.

I have 174MB free...anyways I'd like to try the rebuild of initrd ... can you help me about this or just following standard linux procedure would be fine ?

EDIT: moved forward : Moved old bad copy (754-10) as .bad and run:

Code: [Select]

dracut -f /boot/initramfs-2.6.32-754.10.1.el6.x86_64.img 2.6.32-754.10.1.el6.x86_64
Now I have:

Code: [Select]

[root@andorinha ~]# ls /boot/initramfs-2.6.32-754.10* -lart
 -rw------- 1 root root 18727102 Jan 19 16:11 /boot/initramfs-2.6.32-754.10.1.el6.x86_64.img.bad
 -rw------- 1 root root 18726924 Feb 17 20:34 /boot/initramfs-2.6.32-754.10.1.el6.x86_64.img
 [root@andorinha ~]#

So I may be victim of a lack of free space, and that would be verified by installer.
I'll try if this new initram works and open a NFR to verify this lack of free space.

[ReetP]

It is probably enough to allow for a couple of kernels with older ones being auto removed, but not enough when you start adding your own kernels.

It isn't a bug per se with a stock install but may need notes adding to the virtualbox wiki page.