Hi, I need some advice here, I am kind of stuck. LVM wont recognize my RAID disks during kernel boot.
SME 8.beta6 (kernel 2.6.18-238)
MOBO: Atom + ICH9R, SuperMicro 7XSPA-H-D525
RAM: 4G
DISK: 4 x 1T, RAID5 (and RAID1 for the boot section)
My 4 disk RAID5 lost a disk due to a bad power cable, but continued nicely on 3 disks.
After putting back the power cable the box wont boot, but gives kernel panic. I had no problem booting the rescue CD, and the RAID5 (and RAID 1) was recognized and I could add the missing "sdd1" and "sdd2" to the RAIDs, now mdadm tells both are clean. Furthermore, LVM will recognize the volumes and a fsck says the volumes are clean. I can see all data on the disks, booting with the rescue CD.
I have traced the problem to the command "lvm vgscan --ignorelockingfailure" in the "init" in the initrd run by the kernel during boot. (I did this by unpacking the existing initrd by using the rescue CD, adding debug arguments in the "init" script and repacking the initrd. During boot I sent the kernel console to ttyS0, and logged the debug output).
I have attached the console log (below can't attach as file) during kernel boot with "-vvv" aded to "lvm vgscan --ignorelockingfailure". As you can see it seems like LVM cant see the correct size of md2 (my 3T RAID5 for "/" and the swap volumes). It doesn't see md1 either, but I guess thats because its actually booting from that RAID1.
The central error message as I see it is:
Opened /dev/md2 RO O_DIRECT
/dev/md2: size is 0 sectors
/dev/md2: Skipping: Too small to hold a PV
Closed /dev/md2
So why doesn't LVM recognize my RAID's during boot, when there is no problems using the rescue CD? Any clues, ideas, hints are appreciated.
Thanks, Michael
Booting:
---- snip -----
..
..
---- snip -----
Loading jbd.ko module
Loading ext3.ko module
Loading xor.ko module
raid5: automatically using best checksumming function: pIII_sse
pIII_sse : 5140.000 MB/sec
raid5: using function: pIII_sse (5140.000 MB/sec)
Loading raid456.ko module
raid6: int32x1 171 MB/s
raid6: int32x2 246 MB/s
raid6: int32x4 265 MB/s
raid6: int32x8 300 MB/s
raid6: mmxx1 375 MB/s
raid6: mmxx2 734 MB/s
raid6: sse1x1 308 MB/s
raid6: sse1x2 558 MB/s
raid6: sse2x1 613 MB/s
raid6: sse2x2 1109 MB/s
raid6: using algorithm sse2x2 (1109 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
insmod: error inserting '/lib/raid456.ko': -1 File exists
Loading scsi_mod.ko module
SCSI subsystem initialized
Loading sd_mod.ko module
Loading libata.ko module
Loading ahci.ko module
Loading dm-mod.ko module
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
Loading dm-log.ko module
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Loading dm-mem-cache.ko module
Loading dm-region_hash.ko module
Loading dm-message.ko module
Loading dm-raid45.ko module
device-mapper: dm-raid45: initialized v0.2594l
Waiting for driver initialization.
Scanning and conmd: Autodetecting RAID arrays.
figuring dmraid md: autorun ...
md: ... autorun DONE.
supported devices
Scanning logical volumes
Logging initialised at Thu Sep 1 20:04:38 2011
Set umask to 0077
Processing: vgscan --ignorelockingfailure -vvv
O_DIRECT will be used
Setting global/locking_type to 1
Setting global/wait_for_locks to 1
File-based locking selected.
Setting global/locking_dir to /var/lock/lvm
Creating directory "/var/lock/lvm"
Locking /var/lock/lvm/P_global WB
_do_flock /var/lock/lvm/P_global:aux WB
_do_flock /var/lock/lvm/P_global WB
_undo_flock /var/lock/lvm/P_global:aux
Wiping cache of LVM-capable devices
/dev/console: Not a block device
/dev/md2: Added to device cache
/dev/null: Not a block device
/dev/ptmx: Not a block device
/dev/ram0: Added to device cache
/dev/ram1: Added to device cache
/dev/ram10: Added to device cache
/dev/ram11: Added to device cache
/dev/ram12: Added to device cache
/dev/ram13: Added to device cache
/dev/ram14: Added to device cache
/dev/ram15: Added to device cache
/dev/ram2: Added to device cache
/dev/ram3: Added to device cache
/dev/ram4: Added to device cache
/dev/ram5: Added to device cache
/dev/ram6: Added to device cache
/dev/ram7: Added to device cache
/dev/ram8: Added to device cache
/dev/ram9: Added to device cache
/dev/rtc: Not a block device
/dev/systty: Not a block device
/dev/tty: Not a block device
/dev/tty0: Not a block device
/dev/tty1: Not a block device
/dev/tty10: Not a block device
/dev/tty11: Not a block device
/dev/tty12: Not a block device
/dev/tty2: Not a block device
/dev/tty3: Not a block device
/dev/tty4: Not a block device
/dev/tty5: Not a block device
/dev/tty6: Not a block device
/dev/tty7: Not a block device
/dev/tty8: Not a block device
/dev/tty9: Not a block device
/dev/ttyS0: Not a block device
/dev/ttyS1: Not a block device
/dev/ttyS2: Not a block device
/dev/ttyS3: Not a block device
/dev/urandom: Not a block device
/dev/zero: Not a block device
Wiping internal VG cache
lvmcache: initialised VG #orphans_lvm1
lvmcache: initialised VG #orphans_pool
lvmcache: initialised VG #orphans_lvm2
Reading all physical volumes. This may take a while...
Finding all volume groups
Opened /dev/ram0 RO O_DIRECT
/dev/ram0: size is 32768 sectors
Closed /dev/ram0
/dev/ram0: size is 32768 sectors
Opened /dev/ram0 RO O_DIRECT
/dev/ram0: block size is 4096 bytes
Closed /dev/ram0
Using /dev/ram0
Opened /dev/ram0 RO O_DIRECT
/dev/ram0: block size is 4096 bytes
/dev/ram0: No label detected
Closed /dev/ram0
Opened /dev/ram1 RO O_DIRECT
/dev/ram1: size is 32768 sectors
Closed /dev/ram1
/dev/ram1: size is 32768 sectors
Opened /dev/ram1 RO O_DIRECT
/dev/ram1: block size is 4096 bytes
Closed /dev/ram1
Using /dev/ram1
Opened /dev/ram1 RO O_DIRECT
/dev/ram1: block size is 4096 bytes
/dev/ram1: No label detected
Closed /dev/ram1
Opened /dev/ram2 RO O_DIRECT
/dev/ram2: size is 32768 sectors
Closed /dev/ram2
/dev/ram2: size is 32768 sectors
Opened /dev/ram2 RO O_DIRECT
/dev/ram2: block size is 4096 bytes
Closed /dev/ram2
Using /dev/ram2
Opened /dev/ram2 RO O_DIRECT
/dev/ram2: block size is 4096 bytes
/dev/ram2: No label detected
Closed /dev/ram2
Opened /dev/md2 RO O_DIRECT
/dev/md2: size is 0 sectors
/dev/md2: Skipping: Too small to hold a PV
Closed /dev/md2
Opened /dev/ram3 RO O_DIRECT
/dev/ram3: size is 32768 sectors
Closed /dev/ram3
/dev/ram3: size is 32768 sectors
Opened /dev/ram3 RO O_DIRECT
/dev/ram3: block size is 4096 bytes
Closed /dev/ram3
Using /dev/ram3
Opened /dev/ram3 RO O_DIRECT
/dev/ram3: block size is 4096 bytes
/dev/ram3: No label detected
Closed /dev/ram3
Opened /dev/ram4 RO O_DIRECT
/dev/ram4: size is 32768 sectors
Closed /dev/ram4
/dev/ram4: size is 32768 sectors
Opened /dev/ram4 RO O_DIRECT
/dev/ram4: block size is 4096 bytes
Closed /dev/ram4
Using /dev/ram4
Opened /dev/ram4 RO O_DIRECT
/dev/ram4: block size is 4096 bytes
/dev/ram4: No label detected
Closed /dev/ram4
Opened /dev/ram5 RO O_DIRECT
/dev/ram5: size is 32768 sectors
Closed /dev/ram5
/dev/ram5: size is 32768 sectors
Opened /dev/ram5 RO O_DIRECT
/dev/ram5: block size is 4096 bytes
Closed /dev/ram5
Using /dev/ram5
Opened /dev/ram5 RO O_DIRECT
/dev/ram5: block size is 4096 bytes
/dev/ram5: No label detected
Closed /dev/ram5
Opened /dev/ram6 RO O_DIRECT
/dev/ram6: size is 32768 sectors
Closed /dev/ram6
/dev/ram6: size is 32768 sectors
Opened /dev/ram6 RO O_DIRECT
/dev/ram6: block size is 4096 bytes
Closed /dev/ram6
Using /dev/ram6
Opened /dev/ram6 RO O_DIRECT
/dev/ram6: block size is 4096 bytes
/dev/ram6: No label detected
Closed /dev/ram6
Opened /dev/ram7 RO O_DIRECT
/dev/ram7: size is 32768 sectors
Closed /dev/ram7
/dev/ram7: size is 32768 sectors
Opened /dev/ram7 RO O_DIRECT
/dev/ram7: block size is 4096 bytes
Closed /dev/ram7
Using /dev/ram7
Opened /dev/ram7 RO O_DIRECT
/dev/ram7: block size is 4096 bytes
/dev/ram7: No label detected
Closed /dev/ram7
Opened /dev/ram8 RO O_DIRECT
/dev/ram8: size is 32768 sectors
Closed /dev/ram8
/dev/ram8: size is 32768 sectors
Opened /dev/ram8 RO O_DIRECT
/dev/ram8: block size is 4096 bytes
Closed /dev/ram8
Using /dev/ram8
Opened /dev/ram8 RO O_DIRECT
/dev/ram8: block size is 4096 bytes
/dev/ram8: No label detected
Closed /dev/ram8
Opened /dev/ram9 RO O_DIRECT
/dev/ram9: size is 32768 sectors
Closed /dev/ram9
/dev/ram9: size is 32768 sectors
Opened /dev/ram9 RO O_DIRECT
/dev/ram9: block size is 4096 bytes
Closed /dev/ram9
Using /dev/ram9
Opened /dev/ram9 RO O_DIRECT
/dev/ram9: block size is 4096 bytes
/dev/ram9: No label detected
Closed /dev/ram9
Opened /dev/ram10 RO O_DIRECT
/dev/ram10: size is 32768 sectors
Closed /dev/ram10
/dev/ram10: size is 32768 sectors
Opened /dev/ram10 RO O_DIRECT
/dev/ram10: block size is 4096 bytes
Closed /dev/ram10
Using /dev/ram10
Opened /dev/ram10 RO O_DIRECT
/dev/ram10: block size is 4096 bytes
/dev/ram10: No label detected
Closed /dev/ram10
Opened /dev/ram11 RO O_DIRECT
/dev/ram11: size is 32768 sectors
Closed /dev/ram11
/dev/ram11: size is 32768 sectors
Opened /dev/ram11 RO O_DIRECT
/dev/ram11: block size is 4096 bytes
Closed /dev/ram11
Using /dev/ram11
Opened /dev/ram11 RO O_DIRECT
/dev/ram11: block size is 4096 bytes
/dev/ram11: No label Kernel panic - not syncing: Attempted to kill init!
detected
Closed /dev/ra