Koozali.org: home of the SME Server

[SOLVED] Please help - local mail queue not being processed

Offline ktenbrook

  • 14
  • +0/-0
I hope a SME expert can help me here.  My mail server suddenly stopped processing the delivery of local emails.  My internal users can send out mail, and external recipients get the mail, however, no one can receive mail.  If a local user sends mail to another local user, the recipient does not get the mail.  No one receives any error messages.

Everything was working fine yesterday.  The problem began sometime last night.

Checking the mail log analysis - status of of the queue, shows 500+ (and growing) messages in the local queue, and none in the remote queue.

This was version 7.4.  When I noticed the problem do to not getting any mail overnight, I checked if there were updates, and of course, 7.5 was available.  So I installed that hoping it would cure the problem.  No luck.

Next I installed qmHandle and ran the program to force delivery of the queue.  When run from the command line, (qmHandle -a), there is no return message.  Run from the Server manager panel, an error is returned saying that qmail is not running.  Running /etc/init.d/qmail restart says OK, but the problem continues.

Please help!  Does anyone have any idea what to do?  This just came out of the blue - no changes were made to the system until after the problem started, and nothing so far is working.  I'm at a loss.

If I can't fix this in the next few hours, I'm just going to have to re-install.  Arggh!  I thought that only happened in Windows!

Thanks for your help in advance.

Offline larieu

  • ****
  • 214
  • +0/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #1 on: June 04, 2010, 07:00:55 PM »
please issue this commands into shell

Code: [Select]
tail -f /var/log/messages
try to send an email from one user to other one
(to stop it just use CTRL+C)

Code: [Select]
tail -f /var/log/qmail/current
try another one time to send one email to other user

and send some logs (please be careful with "sensible" informations)

if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline ktenbrook

  • 14
  • +0/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #2 on: June 04, 2010, 07:15:27 PM »
Thank you for the quick response. 

Output from tail -f /var/log/messages:
...
Jun  4 11:44:10 mail esmith::event[5367]: S50clear-pptp-interfaces=action|Event|local|Action|S50clear-pptp-interfaces|Start|1275669850 634865|End|1275669850 725899|Elapsed|0.091034
Jun  4 11:44:10 mail esmith::event[5367]: Running event handler: /etc/e-smith/events/actions/adjust-services
Jun  4 11:44:10 mail esmith::event[5367]: adjusting supervised yum (once)
Jun  4 11:44:10 mail esmith::event[5367]: adjust-services=action|Event|local|Action|adjust-services|Start|1275669850 726368|End|1275669850 945809|Elapsed|0.219441
Jun  4 11:51:51 mail sshd(pam_unix)[5979]: session opened for user root by (uid=0)
Jun  4 11:54:11 mail sshd(pam_unix)[6085]: session opened for user root by (uid=0)

Nothing really relevant there. 

Output from tail -f /var/log/qmail/current after sending email
[root@mail ~]# tail -f /var/log/qmail/current
@400000004c09336b2e91881c new msg 9785921
@400000004c09336b2e919ba4 info msg 9785921: bytes 1212 from <ktenbrook@pwarch.com> qp 7195 uid 453
...
The mail was sent from me, ktenbrook@pwarch.com to another local account, crothberg@pwarch.com

Offline ktenbrook

  • 14
  • +0/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #3 on: June 04, 2010, 07:20:17 PM »
More info.

After sending the email and capturing the above two log outputs, I also captured this from qpsmtpd/current (filtering on the message number from above):

2010-06-04 12:09:53.769415500 7195 queue::qmail_2dqueue plugin (queue): (for 7192 ) Queuing qp 7195 to /var/qmail/bin/qmail-queue

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #4 on: June 04, 2010, 07:56:17 PM »
Checking the mail log analysis - status of of the queue, shows 500+ (and growing) messages in the local queue, and none in the remote queue.

"mail log analysis" also includes "reason for non-delivery". Your answer will be there. Your answer will also be in the qmail log files  (and only in the qmail logs files) - that's where the mail log analysis gets its information.

The only time the reason for non-delivery will not be in the log files is if your disk is full - but in that case you wouldn't be able to add new messages to the queue.

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #5 on: June 04, 2010, 08:00:02 PM »
If I can't fix this in the next few hours, I'm just going to have to re-install.  Arggh!  I thought that only happened in Windows!

Only Windows administrators need to resort to re-install. You will not have to re-install. Re-install will be vastly worse for your users than just taking a little longer than a few hours to diagnose and fix the problem. Do not panic. Do not lose your users email messages.

I hope you saw:

Don't report problems here - Please report bugs and potential bugs in the bug tracker

This is a potential bug. Local delivery of messages should not stop. Report a bug. If it's not a bug, we will tell you.

Quote
Next I installed qmHandle and ran the program to force delivery of the queue.

DId you perhaps stop qmail and forget to start it again?

Offline ktenbrook

  • 14
  • +0/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #6 on: June 04, 2010, 08:08:34 PM »
Hello Charlie,

Thanks for your reply.  I am going to post all this in the bug tracker.  This thread was more about seeing if someone could help me clear this problem before my whole company has an email withdrawal meltdown.  People are walking up to my desk right now every few minutes complaining they have "critical" messages they are expecting. 

Anyway, I will be posting this in the bug tracker - hopefully with more info as to what is going on/what happened/what it took to fix.  I've been using SME for years with no real problems, so this is a very strange anomaly for me.  With no error messages, etc., I'm at a loss as to how to restore service to my users.  600 messages now in the queue, and growing.

Offline ktenbrook

  • 14
  • +0/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #7 on: June 04, 2010, 08:11:33 PM »
Oops, I forgot to answer your question.

I have issued this command:  sv u /service/qmail multiple times, so I think qmail is running.

I also issue this:

[root@mail ~]# sv s /service/qmail
run: /service/qmail: (pid 4701) 5201s, normally down; run: log: (pid 3159) 5427s

Which I understand is the normal response, correct?

Offline ktenbrook

  • 14
  • +0/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #8 on: June 04, 2010, 10:20:35 PM »
I have cleared the problem.  I first stopped qmail and qpsmtpd.  Then I manually deleted all the messages from the qmail queue from yesterday. 

This is actually a tricky and tedious operation. I deleted them from:
/var/qmail/queue/mess/##
/var/qmail/queue/info/##
/var/qmail/queue/local/##

Where ## is a number between 1 and 22.  (qmail puts the messages and information in directories based on the remainder of the message # divided by 23).  There were about 10 messages in total that I had to delete.

Then I restarted the server, and the remaining 600+ messages in the queue were delivered!

I still have no idea what started this in the first place.  It seems that a single message landed in the queue and caused qmail to stop processing the local queue. 

I've submitted this as a bug in the bug tracker and will provide as much additional information as I can there.

Thanks all you SME developers for your hard work on a great server.

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #9 on: June 05, 2010, 01:36:41 AM »
I still have no idea what started this in the first place.  It seems that a single message landed in the queue and caused qmail to stop processing the local queue. 

Your problem is a bug in the vacation program you are using. You deleted a large number of your users messages unnecessarily (I presume they will be angry with you) - but at least you did not re-install the system.

In future, please do not panic, and just follow the instructions you are given.

Offline chris burnat

  • ****
  • 1,135
  • +2/-0
    • http://www.burnat.com
Re: [SOLVED] Please help - local mail queue not being processed
« Reply #10 on: June 05, 2010, 01:46:19 AM »
Your problem is a bug in the vacation program you are using.

Moving to Contribs section of the Forums (vacation program is not core SME package) and adding [solved] to subject as requested.
« Last Edit: June 05, 2010, 01:48:05 AM by chris burnat »
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.