Topic: Server Crashes at Midnight

[ephraims]

I have a sme that crashes at midnigt. The server was recently reloaded and a backup restored. It worked for 2 days great then crashed at midnight then it worked another day and crashed again. This is the log just before it crashes.
Unable to handle kernel paging request at virtual address 00400010
Mar 13 00:13:40 smeserver kernel:  printing eip:
Mar 13 00:13:40 smeserver kernel: c014ed21
Mar 13 00:13:40 smeserver kernel: *pde = 00000000
Mar 13 00:13:40 smeserver kernel: Oops: 0000
Mar 13 00:13:40 smeserver kernel: ppp_mppe smbfs appletalk sch_ingress ppp_synctty ppp_async ppp_generic slhc 8139too mii e1000 ipt_LOG ipt_MASQUERADE ipt_state ipt_TOS ip_conntrack_ftp ip_nat
Mar 13 00:13:40 smeserver kernel: CPU:    0
Mar 13 00:13:40 smeserver kernel: EIP:    0010:[prune_dcache+241/352]    Tainted: P
Mar 13 00:13:40 smeserver kernel: EIP:    0010:[<c014ed21>]    Tainted: P
Mar 13 00:13:40 smeserver kernel: EFLAGS: 00010206
Mar 13 00:13:40 smeserver kernel:
Mar 13 00:13:40 smeserver kernel: EIP is at prune_dcache [kernel] 0xf1 (2.4.20-18.7)
Mar 13 00:13:40 smeserver kernel: eax: 00400000   ebx: d1de2900   ecx: f79288a0   edx: f7928920
Mar 13 00:13:40 smeserver kernel: esi: f7928880   edi: 00000000   ebp: 0000018b   esp: c36bdf60
Mar 13 00:13:40 smeserver kernel: ds: 0018   es: 0018   ss: 0018
Mar 13 00:13:40 smeserver kernel: Process kswapd (pid: 5, stackpage=c36bd000)
Mar 13 00:13:40 smeserver kernel: Stack: 0000001f 00000000 c033f100 c36bdf84 00000246 c36bdf84 c0115594 c36bdf84
Mar 13 00:13:40 smeserver kernel:        c36bdf84 00000000 00000000 0143938d 00000100 c02deb00 000001d0 c36bc000
Mar 13 00:13:40 smeserver kernel:        c014f080 000002d9 c0134913 00000006 000001d0 00000000 c36bc000 00000000
Mar 13 00:13:40 smeserver kernel: Call Trace:   [schedule_timeout+132/160] schedule_timeout [kernel] 0x84 (0xc36bdf78))
Mar 13 00:13:40 smeserver kernel: Call Trace:   [<c0115594>] schedule_timeout [kernel] 0x84 (0xc36bdf78))
Mar 13 00:13:40 smeserver kernel: [shrink_dcache_memory+32/48] shrink_dcache_memory [kernel] 0x20 (0xc36bdfa0))
Mar 13 00:13:40 smeserver kernel: [<c014f080>] shrink_dcache_memory [kernel] 0x20 (0xc36bdfa0))
Mar 13 00:13:40 smeserver kernel: [do_try_to_free_pages_kswapd+19/784] do_try_to_free_pages_kswapd [kernel] 0x13 (0xc36bdfa8))
Mar 13 00:13:40 smeserver kernel: [<c0134913>] do_try_to_free_pages_kswapd [kernel] 0x13 (0xc36bdfa8))
Mar 13 00:13:40 smeserver kernel: [kswapd+321/1248] kswapd [kernel] 0x141 (0xc36bdfd4))
Mar 13 00:13:40 smeserver kernel: [<c0134de1>] kswapd [kernel] 0x141 (0xc36bdfd4))
Mar 13 00:13:40 smeserver kernel: [_stext+0/48] stext [kernel] 0x0 (0xc36bdfe8))
Mar 13 00:13:40 smeserver kernel: [<c0105000>] stext [kernel] 0x0 (0xc36bdfe8))
Mar 13 00:13:40 smeserver kernel: [arch_kernel_thread+38/48] arch_kernel_thread [kernel] 0x26 (0xc36bdff0))
Mar 13 00:13:40 smeserver kernel: [<c0107146>] arch_kernel_thread [kernel] 0x26 (0xc36bdff0))
Mar 13 00:13:40 smeserver kernel: [kswapd+0/1248] kswapd [kernel] 0x0 (0xc36bdff8))
Mar 13 00:13:40 smeserver kernel: [<c0134ca0>] kswapd [kernel] 0x0 (0xc36bdff8))
Mar 13 00:13:40 smeserver kernel:
Mar 13 00:13:40 smeserver kernel:
Mar 13 00:13:40 smeserver kernel: Code: 8b 40 10 85 c0 74 04 56 ff d0 58 8b 56 3c 8d 46 60 39 c2 74

Any Ideas

[dsemuk]

I would report this in the bug tracker.

Best place for developers to see your problem.

Dave

[smeghead]

.. be sure to document your hardware setup when posting to the bug tracker

[Stefano]

HI all..

Me too..

Mar 11 02:12:22 fileserver kernel: EXT3-fs error (device md(9,1)):
ext3_readdir: bad entry in directory #13385814: rec_len %% 4 != 0 -
offset=0, inode=3287794679, rec_len=50167, name_len=0
Mar 11 02:19:17 fileserver kernel: Unable to handle kernel paging request at
virtual address 102444b7
Mar 11 02:19:17 fileserver kernel:  printing eip:
Mar 11 02:19:17 fileserver kernel: c0129277
Mar 11 02:19:17 fileserver kernel: *pde = 00000000
Mar 11 02:19:17 fileserver kernel: Oops: 0000
Mar 11 02:19:17 fileserver kernel: nls_iso8859-1 sr_mod ppp_mppe ppp_async
ppp_generic slhc 8139too mii st ide-scsi ide-cd cdrom hid input ehci-hcd
usb-uhci usbcore ext3 jbd raid1 a100u2w sd_mo
Mar 11 02:19:17 fileserver kernel: CPU:    0
Mar 11 02:19:17 fileserver kernel: EIP:    0010:[find_vma+55/96]    Tainted:
P
Mar 11 02:19:17 fileserver kernel: EIP:    0010:[<c0129277>]    Tainted: P
Mar 11 02:19:17 fileserver kernel: EFLAGS: 00010202
Mar 11 02:19:17 fileserver kernel:
Mar 11 02:19:17 fileserver kernel: EIP is at find_vma [kernel] 0x37
(2.4.20-43.7.legacy)
Mar 11 02:19:17 fileserver kernel: eax: c0143978   ebx: 01b97005   ecx:
c0143978   edx: 102444c7
Mar 11 02:19:17 fileserver kernel: esi: f7c35528   edi: 01a80005   ebp:
d658a45c   esp: c36bfea0
Mar 11 02:19:17 fileserver kernel: ds: 0018   es: 0018   ss: 0018
Mar 11 02:19:17 fileserver kernel: Process kswapd (pid: 4,
stackpage=c36bf000)
Mar 11 02:19:17 fileserver kernel: Stack: f7c35528 01b97005 c013b88d
f7c35528 01b97005 002f2200 f7c35528 01b97005
Mar 11 02:19:17 fileserver kernel:        c15a15e8 c15a15e8 00000006
c15a15e8 c15a15e8 000001d0 c013b98f 002f2200
Mar 11 02:19:17 fileserver kernel:        c15a15e8 c013735f ffffffff
00000000 c02e0888 c15a15e8 c15a15e8 00000000
Mar 11 02:19:17 fileserver kernel: Call Trace:   [try_to_unmap_one+317/464]
try_to_unmap_one [kernel] 0x13d (0xc36bfea8))
Mar 11 02:19:17 fileserver kernel: Call Trace:   [<c013b88d>]
try_to_unmap_one [kernel] 0x13d (0xc36bfea8))
Mar 11 02:19:17 fileserver kernel: [try_to_unmap+111/448] try_to_unmap
[kernel] 0x6f (0xc36bfed8))
Mar 11 02:19:17 fileserver kernel: [<c013b98f>] try_to_unmap [kernel] 0x6f
(0xc36bfed8))
Mar 11 02:19:17 fileserver kernel: [add_to_swap+95/128] add_to_swap [kernel]
0x5f (0xc36bfee4))
Mar 11 02:19:17 fileserver kernel: [<c013735f>] add_to_swap [kernel] 0x5f
(0xc36bfee4))
Mar 11 02:19:17 fileserver kernel: [launder_page+1191/1760] launder_page
[kernel] 0x4a7 (0xc36bff04))
Mar 11 02:19:17 fileserver kernel: [<c01328f7>] launder_page [kernel] 0x4a7
(0xc36bff04))
Mar 11 02:19:17 fileserver kernel: [rebalance_dirty_zone+90/144]
rebalance_dirty_zone [kernel] 0x5a (0xc36bff1c))
Mar 11 02:19:17 fileserver kernel: [<c0134b3a>] rebalance_dirty_zone
[kernel] 0x5a (0xc36bff1c))
Mar 11 02:19:17 fileserver kernel: [rebalance_inactive_zone+542/848]
rebalance_inactive_zone [kernel] 0x21e (0xc36bff3c))
Mar 11 02:19:17 fileserver kernel: [<c0134d8e>] rebalance_inactive_zone
[kernel] 0x21e (0xc36bff3c))
Mar 11 02:19:17 fileserver kernel: [rebalance_inactive+61/128]
rebalance_inactive [kernel] 0x3d (0xc36bff6c))
Mar 11 02:19:17 fileserver kernel: [<c0134efd>] rebalance_inactive [kernel]
0x3d (0xc36bff6c))
Mar 11 02:19:17 fileserver kernel: [do_try_to_free_pages_kswapd+49/864]
do_try_to_free_pages_kswapd [kernel] 0x31 (0xc36bff90))
Mar 11 02:19:17 fileserver kernel: [<c0135041>] do_try_to_free_pages_kswapd
[kernel] 0x31 (0xc36bff90))
Mar 11 02:19:17 fileserver kernel: [kswapd+321/1248] kswapd [kernel] 0x141
(0xc36bffd4))
Mar 11 02:19:17 fileserver kernel: [<c0135541>] kswapd [kernel] 0x141
(0xc36bffd4))
Mar 11 02:19:17 fileserver kernel: [rest_init+0/48] stext [kernel] 0x0
(0xc36bffe8))
Mar 11 02:19:17 fileserver kernel: [<c0105000>] stext [kernel] 0x0
(0xc36bffe8))
Mar 11 02:19:17 fileserver kernel: [arch_kernel_thread+38/48]
arch_kernel_thread [kernel] 0x26 (0xc36bfff0))
Mar 11 02:19:17 fileserver kernel: [<c0107146>] arch_kernel_thread [kernel]
0x26 (0xc36bfff0))
Mar 11 02:19:17 fileserver kernel: [kswapd+0/1248] kswapd [kernel] 0x0
(0xc36bfff8))
Mar 11 02:19:17 fileserver kernel: [<c0135400>] kswapd [kernel] 0x0
(0xc36bfff8))
Mar 11 02:19:17 fileserver kernel:
Mar 11 02:19:17 fileserver kernel:
Mar 11 02:19:17 fileserver kernel: Code: 39 5a f0 8d 42 e8 76 f1 39 5a ec 89
c1 77 e2 85 c9 74 03 89

kernel  2.4.20-43.7.legacy

ciao

Stefano

[marsa_matruh]

kernel  2.4.20-43.7.legacy

Where does it come from?

[marsa_matruh]

The server was recently reloaded and a backup restored.

Any Ideas

Did you applied updates to SME (by using yum or some other method)?

[Stefano]

kernel  2.4.20-43.7.legacy

Where does it come from?

www.fedoralegacy.org

HTH

Stefano

[marsa_matruh]

As kernel 2.4.20-43.7.legacy is not part of SME 6.x updates, I would suggest returning to an official kernel of SME.

[Stefano]

As kernel 2.4.20-43.7.legacy is not part of SME 6.x updates, I would suggest returning to an official kernel of SME.

AFAIK,SME's kernel is the same as redhat 7.3 (as most of rpms)

so I think this is not a problem...

ciao
Stefano

[marsa_matruh]

kernel 2.4.20-43.7.legacy is from fedoralegacy i.e. a doozen of people trying to maintane RH7.3 (and other) during there freetime. It is not from RedHat inc.

Recently, I had problem with my website on sme 6.0. After last php update, php crashes when visiting some pages. I went back to the previous php rpm and problem was gone. As php is modified in sme, I also tried rpm from fedoralegacy. I discovered that problem come with php-4.1.2-7.3.17.legacy and it was ok with php-4.1.2-7.3.18.legacy.

So, my suggestion ...

[ephraims]

After a bit of adjusting of the backups i have found that the server had stop crashing. I beleived that because two backups were overlapping each other was causing the problem. So i adjusted the backups and everything was good for two weeks.

So i thought Yea

But no it has crashed again same error on sunday night. Curiosly enough the last time it crashed was also midnight on a sunday

What happens at midnight on sunday that would cause a
Unable to handle kernel paging request at virtual address
Error

[cactus]

After a bit of adjusting of the backups i have found that the server had stop crashing. I beleived that because two backups were overlapping each other was causing the problem. So i adjusted the backups and everything was good for two weeks.

So i thought Yea

But no it has crashed again same error on sunday night. Curiosly enough the last time it crashed was also midnight on a sunday

What happens at midnight on sunday that would cause a
Unable to handle kernel paging request at virtual address
Error

Perhaps a two weekly/monthly backup?