Koozali.org: home of the SME Server

Kernel wont boot after RAID5 problem

Kernel wont boot after RAID5 problem
« on: September 01, 2011, 11:32:49 PM »
Hi, I need some advice here, I am kind of stuck. LVM wont recognize my RAID disks during kernel boot.

SME 8.beta6 (kernel 2.6.18-238)
MOBO: Atom + ICH9R, SuperMicro 7XSPA-H-D525
RAM: 4G
DISK: 4 x 1T, RAID5 (and RAID1 for the boot  section)

My 4 disk RAID5 lost a disk due to a bad power cable, but continued nicely on 3 disks.

After putting back the power cable the box wont boot, but gives kernel panic. I had no problem booting the rescue CD, and the RAID5 (and RAID 1) was recognized and I could add the missing "sdd1" and "sdd2" to the RAIDs, now mdadm tells both are clean. Furthermore, LVM will recognize the volumes and a fsck says the volumes are clean. I can see all data on the disks, booting with the rescue CD.

I have traced the problem to the command "lvm vgscan --ignorelockingfailure" in the "init" in the initrd run by the kernel during boot. (I did this by unpacking the existing initrd by using the rescue CD, adding debug arguments in the "init" script and repacking the initrd. During boot I sent the kernel console to ttyS0, and logged the debug output).

I have attached the console log (below can't attach as file) during kernel boot with "-vvv" aded to "lvm vgscan --ignorelockingfailure". As you can see it seems like LVM cant see the correct size of md2 (my 3T RAID5 for "/" and the swap volumes). It doesn't see md1 either, but I guess thats because its actually booting from that RAID1.

The central error message as I see it is:
Opened /dev/md2 RO O_DIRECT
      /dev/md2: size is 0 sectors
        /dev/md2: Skipping: Too small to hold a PV
        Closed /dev/md2


So why doesn't LVM recognize my RAID's during boot, when there is no problems using the rescue CD? Any clues, ideas, hints are appreciated.

Thanks, Michael


Booting:
---- snip -----
..
..
---- snip -----
Loading jbd.ko module
Loading ext3.ko module
Loading xor.ko module
raid5: automatically using best checksumming function: pIII_sse
   pIII_sse  :  5140.000 MB/sec
raid5: using function: pIII_sse (5140.000 MB/sec)
Loading raid456.ko module
raid6: int32x1    171 MB/s
raid6: int32x2    246 MB/s
raid6: int32x4    265 MB/s
raid6: int32x8    300 MB/s
raid6: mmxx1      375 MB/s
raid6: mmxx2      734 MB/s
raid6: sse1x1     308 MB/s
raid6: sse1x2     558 MB/s
raid6: sse2x1     613 MB/s
raid6: sse2x2    1109 MB/s
raid6: using algorithm sse2x2 (1109 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
insmod: error inserting '/lib/raid456.ko': -1 File exists
Loading scsi_mod.ko module
SCSI subsystem initialized
Loading sd_mod.ko module
Loading libata.ko module
Loading ahci.ko module
Loading dm-mod.ko module
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
Loading dm-log.ko module
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Loading dm-mem-cache.ko module
Loading dm-region_hash.ko module
Loading dm-message.ko module
Loading dm-raid45.ko module
device-mapper: dm-raid45: initialized v0.2594l
Waiting for driver initialization.
Scanning and conmd: Autodetecting RAID arrays.
figuring dmraid md: autorun ...
md: ... autorun DONE.
supported devices
Scanning logical volumes
    Logging initialised at Thu Sep  1 20:04:38 2011
    Set umask to 0077
        Processing: vgscan --ignorelockingfailure -vvv
        O_DIRECT will be used
      Setting global/locking_type to 1
      Setting global/wait_for_locks to 1
      File-based locking selected.
      Setting global/locking_dir to /var/lock/lvm
    Creating directory "/var/lock/lvm"
      Locking /var/lock/lvm/P_global WB
        _do_flock /var/lock/lvm/P_global:aux WB
        _do_flock /var/lock/lvm/P_global WB
        _undo_flock /var/lock/lvm/P_global:aux
    Wiping cache of LVM-capable devices
        /dev/console: Not a block device
        /dev/md2: Added to device cache
        /dev/null: Not a block device
        /dev/ptmx: Not a block device
        /dev/ram0: Added to device cache
        /dev/ram1: Added to device cache
        /dev/ram10: Added to device cache
        /dev/ram11: Added to device cache
        /dev/ram12: Added to device cache
        /dev/ram13: Added to device cache
        /dev/ram14: Added to device cache
        /dev/ram15: Added to device cache
        /dev/ram2: Added to device cache
        /dev/ram3: Added to device cache
        /dev/ram4: Added to device cache
        /dev/ram5: Added to device cache
        /dev/ram6: Added to device cache
        /dev/ram7: Added to device cache
        /dev/ram8: Added to device cache
        /dev/ram9: Added to device cache
        /dev/rtc: Not a block device
        /dev/systty: Not a block device
        /dev/tty: Not a block device
        /dev/tty0: Not a block device
        /dev/tty1: Not a block device
        /dev/tty10: Not a block device
        /dev/tty11: Not a block device
        /dev/tty12: Not a block device
        /dev/tty2: Not a block device
        /dev/tty3: Not a block device
        /dev/tty4: Not a block device
        /dev/tty5: Not a block device
        /dev/tty6: Not a block device
        /dev/tty7: Not a block device
        /dev/tty8: Not a block device
        /dev/tty9: Not a block device
        /dev/ttyS0: Not a block device
        /dev/ttyS1: Not a block device
        /dev/ttyS2: Not a block device
        /dev/ttyS3: Not a block device
        /dev/urandom: Not a block device
        /dev/zero: Not a block device
    Wiping internal VG cache
        lvmcache: initialised VG #orphans_lvm1
        lvmcache: initialised VG #orphans_pool
        lvmcache: initialised VG #orphans_lvm2
  Reading all physical volumes.  This may take a while...
    Finding all volume groups
        Opened /dev/ram0 RO O_DIRECT
      /dev/ram0: size is 32768 sectors
        Closed /dev/ram0
      /dev/ram0: size is 32768 sectors
        Opened /dev/ram0 RO O_DIRECT
        /dev/ram0: block size is 4096 bytes
        Closed /dev/ram0
        Using /dev/ram0
        Opened /dev/ram0 RO O_DIRECT
        /dev/ram0: block size is 4096 bytes
      /dev/ram0: No label detected
        Closed /dev/ram0
        Opened /dev/ram1 RO O_DIRECT
      /dev/ram1: size is 32768 sectors
        Closed /dev/ram1
      /dev/ram1: size is 32768 sectors
        Opened /dev/ram1 RO O_DIRECT
        /dev/ram1: block size is 4096 bytes
        Closed /dev/ram1
        Using /dev/ram1
        Opened /dev/ram1 RO O_DIRECT
        /dev/ram1: block size is 4096 bytes
      /dev/ram1: No label detected
        Closed /dev/ram1
        Opened /dev/ram2 RO O_DIRECT
      /dev/ram2: size is 32768 sectors
        Closed /dev/ram2
      /dev/ram2: size is 32768 sectors
        Opened /dev/ram2 RO O_DIRECT
        /dev/ram2: block size is 4096 bytes
        Closed /dev/ram2
        Using /dev/ram2
        Opened /dev/ram2 RO O_DIRECT
        /dev/ram2: block size is 4096 bytes
      /dev/ram2: No label detected
        Closed /dev/ram2
        Opened /dev/md2 RO O_DIRECT
      /dev/md2: size is 0 sectors
        /dev/md2: Skipping: Too small to hold a PV
        Closed /dev/md2
        Opened /dev/ram3 RO O_DIRECT
      /dev/ram3: size is 32768 sectors
        Closed /dev/ram3
      /dev/ram3: size is 32768 sectors
        Opened /dev/ram3 RO O_DIRECT
        /dev/ram3: block size is 4096 bytes
        Closed /dev/ram3
        Using /dev/ram3
        Opened /dev/ram3 RO O_DIRECT
        /dev/ram3: block size is 4096 bytes
      /dev/ram3: No label detected
        Closed /dev/ram3
        Opened /dev/ram4 RO O_DIRECT
      /dev/ram4: size is 32768 sectors
        Closed /dev/ram4
      /dev/ram4: size is 32768 sectors
        Opened /dev/ram4 RO O_DIRECT
        /dev/ram4: block size is 4096 bytes
        Closed /dev/ram4
        Using /dev/ram4
        Opened /dev/ram4 RO O_DIRECT
        /dev/ram4: block size is 4096 bytes
      /dev/ram4: No label detected
        Closed /dev/ram4
        Opened /dev/ram5 RO O_DIRECT
      /dev/ram5: size is 32768 sectors
        Closed /dev/ram5
      /dev/ram5: size is 32768 sectors
        Opened /dev/ram5 RO O_DIRECT
        /dev/ram5: block size is 4096 bytes
        Closed /dev/ram5
        Using /dev/ram5
        Opened /dev/ram5 RO O_DIRECT
        /dev/ram5: block size is 4096 bytes
      /dev/ram5: No label detected
        Closed /dev/ram5
        Opened /dev/ram6 RO O_DIRECT
      /dev/ram6: size is 32768 sectors
        Closed /dev/ram6
      /dev/ram6: size is 32768 sectors
        Opened /dev/ram6 RO O_DIRECT
        /dev/ram6: block size is 4096 bytes
        Closed /dev/ram6
        Using /dev/ram6
        Opened /dev/ram6 RO O_DIRECT
        /dev/ram6: block size is 4096 bytes
      /dev/ram6: No label detected
        Closed /dev/ram6
        Opened /dev/ram7 RO O_DIRECT
      /dev/ram7: size is 32768 sectors
        Closed /dev/ram7
      /dev/ram7: size is 32768 sectors
        Opened /dev/ram7 RO O_DIRECT
        /dev/ram7: block size is 4096 bytes
        Closed /dev/ram7
        Using /dev/ram7
        Opened /dev/ram7 RO O_DIRECT
        /dev/ram7: block size is 4096 bytes
      /dev/ram7: No label detected
        Closed /dev/ram7
        Opened /dev/ram8 RO O_DIRECT
      /dev/ram8: size is 32768 sectors
        Closed /dev/ram8
      /dev/ram8: size is 32768 sectors
        Opened /dev/ram8 RO O_DIRECT
        /dev/ram8: block size is 4096 bytes
        Closed /dev/ram8
        Using /dev/ram8
        Opened /dev/ram8 RO O_DIRECT
        /dev/ram8: block size is 4096 bytes
      /dev/ram8: No label detected
        Closed /dev/ram8
        Opened /dev/ram9 RO O_DIRECT
      /dev/ram9: size is 32768 sectors
        Closed /dev/ram9
      /dev/ram9: size is 32768 sectors
        Opened /dev/ram9 RO O_DIRECT
        /dev/ram9: block size is 4096 bytes
        Closed /dev/ram9
        Using /dev/ram9
        Opened /dev/ram9 RO O_DIRECT
        /dev/ram9: block size is 4096 bytes
      /dev/ram9: No label detected
        Closed /dev/ram9
        Opened /dev/ram10 RO O_DIRECT
      /dev/ram10: size is 32768 sectors
        Closed /dev/ram10
      /dev/ram10: size is 32768 sectors
        Opened /dev/ram10 RO O_DIRECT
        /dev/ram10: block size is 4096 bytes
        Closed /dev/ram10
        Using /dev/ram10
        Opened /dev/ram10 RO O_DIRECT
        /dev/ram10: block size is 4096 bytes
      /dev/ram10: No label detected
        Closed /dev/ram10
        Opened /dev/ram11 RO O_DIRECT
      /dev/ram11: size is 32768 sectors
        Closed /dev/ram11
      /dev/ram11: size is 32768 sectors
        Opened /dev/ram11 RO O_DIRECT
        /dev/ram11: block size is 4096 bytes
        Closed /dev/ram11
        Using /dev/ram11
        Opened /dev/ram11 RO O_DIRECT
        /dev/ram11: block size is 4096 bytes
      /dev/ram11: No label Kernel panic - not syncing: Attempted to kill init!
detected
         Closed /dev/ra