Topic: Problem using RBLList with smtpfront

[rickjones]

After discovering the excellent HowTo, I enabled RBL checking in smtpfront. Results are very good, with a major reduction in spam reaching the server.

It's been running a couple of weeks, but since then I've noticed significant delays when sending outgoing email from clients, and also from Horde.

I suspect that smtpfront is trying resolve RBL results for local addresses, which it's obviously not going to find. Is this in fact what's happening, and is there a way to configure smtpfront not to do this?

TIA

[raem]

rick

What type of Internet connection do you have, dial up, cable, ADSL etc ?

Is your server busy ie handling large volumes of mail & web requests ?

How many RBL lists did you enable ?

[rickjones]

Ray

I'm on ADSL (576/144 kbps), the server is not busy most of the time, but is only an old PII 400. It's essentially a domestic hub and gateway with 3 users.

I have 9 RBL lists, being most of the ones on your recommended list IIRC.

I'm sure outgoing SMTP was not this slow before I enabled RBL checking, so I figured it's needlessly doing lookups for the local IPs - is that to be expected?

Thanks
Rick

[raem]

Rick

I have similar hardware and volume and don't see any significant problem.

Perhaps you recently (or at the same time you set up RBL) configured another feature that may be having an effect ?

You could try disabling RBL lists one by one and seeing if that changes the situation There was a problem with the contacts.abuse.net list last year that stopped ALL incoming & outgoing mail if I remember correctly.

It's possible that other list servers could get busy and slow things down a little.


> but since then I've noticed significant delays when sending outgoing email from clients...

What type of delay (time) is it that you are experiencing ?

I'm not sure what you mean about looking up local addresses/IPs.

[raem]

Rick

Any feedback available on this issue ?

Have you been able to relate your slowness to the use of certain RBL lists ?

[rickjones]

Ray

I've only just had time to experiment a bit further.

In terms of numbers, I was finding the delay could be from about 7 secs up to about 20 secs. A longer delay occurred if there had been no SMTP activity for a while. I put this down to caching and paging, in that a rapid re-invocation of smtp would involve less disk activity. OTOH it could be the result of DNS caching of the addresses of the RBL-list servers.

I've now cut the RBL lookups to just spamhaus (which the logs showed account for over 90% of rejections anyway), and the response is much quicker - i.e within 2 secs and quite acceptable. This confirms that the RBL lookups were taking the time.

The point I was trying to make initially is that rblsmtpd should not even be invoked for emails originating on the LAN. There's clearly no point doing an RBL lookup for a local address. This suggests to me that $RBLSMTPD should be set in tcprules rather than in runenv.

I'll try configuring tcprules to invoke rblsmtpd selectively, and let you know how I get on.

Cheers

[CharlieBrady]

There's clearly no point doing an RBL lookup for a local address. This suggests to me that $RBLSMTPD should be set in tcprules rather than in runenv.

Correct.

I'll try configuring tcprules to invoke rblsmtpd selectively, and let you know how I get on.

You could try my e-smith-mailfront-1.5.0-14cb4.

[rickjones]

Charlie

Thanks for the tip. I tried installing e-smith-mailfront-1.5.0-14cb5, but it says it needs ipsvd. I haven't seen that anywhere - is your rpm compatible with 6.0, or is it for 6.5? I'm on 6.0.1.

Thanks

[CharlieBrady]

I tried installing e-smith-mailfront-1.5.0-14cb5, but it says it needs ipsvd. I haven't seen that anywhere - is your rpm compatible with 6.0, or is it for 6.5? I'm on 6.0.1.

Sorry about the missing RPM. Look in the mirrors in a few hours.

I think it should work on any 6.x. But you'll test it carefully, right?

[raem]

Hullo Charlie

Looking at the Changelog for e-smith-mailfront-1.5.0-14cb5.noarch.rpm, I don't see this change made by Gordon re fake HELO/EHLO. Did that make it in to your rpm ?

Part of changelog from e-smith-mailfront-1.5.0-13gr07.noarch.rpm

* Wed Mar  9 09:00:00 2005 Gordon Rowell <gordonr@gormand.com.au>
- [1.5.0-13gr07]
- Add /var/qmail/control/badhelo to reject SMTP connections which
  says HELO/EHLO yahoo.com/aoo.com/$ExternalIP
- Updated to depend on new mailfront


Changelog from e-smith-mailfront-1.5.0-14cb5.noarch.rpm (only seems to include changes up to [1.5.0-13gr06])

File: CHANGELOG         Col 0              21822 bytes   [grow]               0%
* Tue Dec 28 09:00:00 2004 Charlie Brady <charlieb@e-smith.com> 1.5.0-14cb5
- Add ssmtpfront-qmail service
- Combine mailfront-conf-env and mailfront-conf-control into
  mailfront-conf
- Add TCPPort property for smtp and ssmtp services.
- Replace sed mangling of filelist with params to genfilelist.

* Fri Oct  1 08:00:00 2004 Charlie Brady <charlieb@e-smith.com> 1.5.0-14cb4
- Set MAILRULES environment variable for external connections.
- Make missing peers subdirectory of smtpfront-qmail service
  directory

* Fri Oct  1 08:00:00 2004 Charlie Brady <charlieb@e-smith.com> 1.5.0-14cb3
- Fix inversion of filters for internal and external connections in ipsvd
  peers config files.
- Create empty template-begin files in templates for ipsvd peers config
  files (to prevent log noise).

* Thu Aug 12 08:00:00 2004 Charlie Brady <charlieb@e-smith.com> - 1.5.0-14cb2
- Complete conversion to tcpsvd. /etc/tcprules/tcp.smtp/ is obsoleted
  in favour of /service/smtpfront-qmail/peers/*.
- Add recipient verification.

* Thu Aug 12 08:00:00 2004 Charlie Brady <charlieb@e-smith.com> - 1.5.0-14cb1
- Switch to tcpsvd rather than tcpserver - add per IP concurrency control
- Don't apply RBL lookup to local addresses
- Remove smtpfront-qmail/env directory - use exports in runenv instead.

* Fri Jun 25 08:00:00 2004 Gordon Rowell <gordonr@e-smith.com>
- [1.5.0-13gr06]
- Put Body pattern code back into SMTP error message [gordonr MN00024867]

* Thu Jun 24 08:00:00 2004 Gordon Rowell <gordonr@e-smith.com>
- [1.5.0-13gr05]
- Allow the Body property of each pattern to have multiple
  patterns, separated by commas. This allows all executables
  to be in one pattern set.
- Add EXEFILES defaults which groups all the previous EXE01..EXE12
- Remove EXE01..EXE12 [gordonr MN00024867]


Regards

[CharlieBrady]

Looking at the Changelog for e-smith-mailfront-1.5.0-14cb5.noarch.rpm, I don't see this change made by Gordon re fake HELO/EHLO. Did that make it in to your rpm ?

Check the dates. Do you think I time travel?

[raem]

I seem to be having a problem with dates this year. I keep thinking it's still 2004 !

> Do you think I time travel?

It would be a handy tool to have !

Thanks for the reminder

[rickjones]

Charlie

Your rpm is clearly a significant change to the SMTP service. If I install it then find I want to rewind to the tcpserver-based one, will rpm -e of yours put everything back, or will I need to re-install the previous rpm.

Just trying to anticipate the worst!

Thanks

[mbachmann]

I really have missed to announce this one on New Development Announcements. Good, that you have brought this up again.

[CharlieBrady]

Your rpm is clearly a significant change to the SMTP service. If I install it then find I want to rewind to the tcpserver-based one, will rpm -e of yours put everything back, or will I need to re-install the previous rpm.

"rpm -e" just uninstalls an rpm. It doesn't do rollback.

You'll need "rpm -Uhv --oldpackage ..." with the previous rpm.

[edb]

I cannot find the e-smith-mailfront-1.5.0-14cb5.noarch.rpm that is being refered to here?
Is the other missing rpm there now and what is it called so I can download both of these files for my SME 6.0.1 box that has the same slow sending of messages problem.

Thanks

Ed

[raem]

Get e-smith-mailfront-1.5.0-14cb5.noarch.rpm  from
ftp://ftp.ibiblio.org/pub/linux/distributions/e-smith/contrib/CharlieBrady/RPMS/noarch/

I think the other rpm is
ipsvd-0.10.1-01.i386.rpm
from
ftp://ftp.ibiblio.org/pub/linux/distributions/e-smith/contrib/CharlieBrady/RPMS/i386/

[raem]

Charlie

I know you said:

> I think it should work on any 6.x. But you'll test it carefully, right?

I installed e-smith-mailfront-1.5.0-14cb5 & ipsvd-0.10.1-01.i386.rpm
(using rpm -Uvh)
and appeare dto install OK,
did
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart
/etc/init.d/smtpfront-qmail restart

but cannot send mail, I get cannot connect errors from my email client of a generic nature.

/var/log.smtpfront-qmail/current shows

2005-04-11 07:47:47.874316500 tcpsvd: info: status 1/40
2005-04-11 07:47:47.874467500 tcpsvd: info: pid 28133 from 192.168.X.XX
2005-04-11 07:47:47.874897500 tcpsvd: info: concurrency 28133 192.168.X.XX 1/4
2005-04-11 07:47:47.874907500 tcpsvd: info: start 28133 0:192.168.2.1 ::192.168.X.XX:2036 ./peers/192.168.X
2005-04-11 07:47:47.875185500 tcpsvd: drop: unable to run: ./smtpfront-qmail: file does not exist
2005-04-11 07:47:47.875702500 tcpsvd: info: end 28133 exit 111
2005-04-11 07:47:47.875710500 tcpsvd: info: status 0/40


I reverted to e-smith-mailfront-1.5.0-13gr07.noarch.rpm that was installed previously
& all working OK again.

Is there something else that needs to be done to get this working ?

Regards & Thanks

[CharlieBrady]

...
2005-04-11 07:47:47.874897500 tcpsvd: info: concurrency 28133 192.168.X.XX 1/4
2005-04-11 07:47:47.874907500 tcpsvd: info: start 28133 0:192.168.2.1 ::192.168.X.XX:2036 ./peers/192.168.X
2005-04-11 07:47:47.875185500 tcpsvd: drop: unable to run: ./smtpfront-qmail: file does not exist
2005-04-11 07:47:47.875702500 tcpsvd: info: end 28133 exit 111
2005-04-11 07:47:47.875710500 tcpsvd: info: status 0/40
...

Yep, quite right, it does not exist:
[charlieb@sme60build noarch]$ rpm -qlvp e-smith-mailfront-1.5.0-14cb5.noarch.rpm | grep 'smtpfront-qmail$'
drwxr-xr-x    2 root    root                0 Dec 27 19:40 /etc/e-smith/db/configuration/defaults/smtpfront-qmail
drwxr-xr-x    2 root    root                0 Dec 27 19:40 /etc/e-smith/db/configuration/defaults/ssmtpfront-qmail
drwxr-xr-x    2 root    root                0 Dec 27 19:40 /etc/e-smith/templates.metadata/var/service/smtpfront-qmail
drwxr-xr-x    2 root    root                0 Dec 27 19:40 /etc/e-smith/templates/var/service/smtpfront-qmail
lrwxr-xr-x    1 root    root               28 Dec 27 19:39 /service/smtpfront-qmail -> /var/service/smtpfront-qmail
lrwxrwxrwx    1 root    root               29 Dec 27 19:39 /service/ssmtpfront-qmail -> /var/service/ssmtpfront-qmail
drwxr-s---    2 qmaill  nofiles             0 Dec 27 19:40 /var/log/smtpfront-qmail
drwxr-s---    2 smelog  root                0 Dec 27 19:40 /var/log/ssmtpfront-qmail
drwxr-xr-t    2 root    root                0 Dec 27 19:40 /var/service/smtpfront-qmail
-rw-r--r--    1 root    root               49 Dec 27 19:39 /var/service/smtpfront-qmail/ssmtpfront-qmail
drwxr-xr-t    2 root    root                0 Dec 27 19:40 /var/service/ssmtpfront-qmail
-rwxr-xr-x    1 root    root              175 Dec 27 19:39 /var/service/ssmtpfront-qmail/ssmtpfront-qmail
[charlieb@sme60build noarch]$

Two problems there - script is called ssmtpfront-qmail, and it's not executable!

I'd better build a better one, eh? In the meantime, you could try renaming and chmoding the file that's there.

[edb]

Stupid question but will this work in conjunction with Knuddi's Spam Filter RBL list?
I'm still running e-smith-mailfront-1.3.0-11 so what are the prerequisites to installing the new e-smith-mailfront-1.5.0-14cb5.noarch.rpm?

Thanks

[CharlieBrady]

Stupid question but will this work in conjunction with Knuddi's Spam Filter RBL list?

It's not a stupid question, but it'd be silly to expect anyone here to know the answer.

[edb]

I take it your not running his Spamassassin which adds a server manager panel where you can add or remove RBL lists?

[raem]

edb

I have Knuddi's spam filter installed on my own server and it works OK with
e-smith-mailfront-1.5.0-13gr07.noarch.rpm

The RBL settings in Knuddi's server manager spam filter panel are using the functionality of the rpm, not the other way around.

Whether the spam filter panel works OK with Charlie's version of e-smith-mailfront is yet to be determined, but if the missing file issue is fixed, then I expect it would probably work OK.

After all Charlie does know what he's doing, even if he can't time travel just yet.

[edb]

Thanks Ray

That's what I thought but I just wanted to make sure.
Thanks for the reply.

Ed

[raem]

Charlie

Thanks for the feedback.

> -rw-r--r--    1 root    root               49 Dec 27 19:39 /var/service/smtpfront-qmail/ssmtpfront-qmail
> ....script is called ssmtpfront-qmail, and it's not executable!
> ....try renaming and chmoding the file that's there.

I did what you suggested and email appears to be working now. Not extensivley tested though.
I notice that sending email from the client is a little quicker also, I guess because local IP RBL lookups are not being done.


cd /var/service/smtpfront-qmail
ls -al

total 36
drwxr-xr-t    6 root     root         4096 Apr 12 11:29 .
drwxr-xr-x   16 root     root         4096 Dec 28 11:40 ..
-rw-r--r--    1 root     root            0 Dec 28 11:40 down
drwxr-xr-x    2 root     root         4096 Apr 12 11:22 env
drwxr-xr-x    3 root     root         4096 Apr 12 11:21 log
drwxr-xr-x    2 root     root         4096 Apr 12 11:23 peers
-rwxr-xr-x    1 root     root         1338 Dec 28 11:39 run
-rw-r--r--    1 root     root         1065 Mar 30 19:45 runenv
-rwxr-xr-x    1 root     root           49 Dec 28 11:39 smtpfront-qmail
drwx------    2 root     root         4096 Apr 12 11:23 supervise


> I'd better build a better one, eh?

I think myself and others would appreciate that.
Is there anything else I should look for and/or test when using your e-smith-mailfrontt rpm ?

Thanks & Regards

[edb]

Any word on a revised rpm yet? Just wondering if it's in the works  

[raem]

edb

> Any word on a revised rpm yet?

Charlie advised he won't be spending any time fixing it, he's busy doing sme 7.0 development work.

There are still some problems with that rpm, I don't think it's ready for production use.

[raem]

I was prompted to try the sme 7.0alpha6 rpm
e-smith-mailfront-1.5.2-03.noarch.rpm
on a 6.0 box with Mitel updates to get the later functionality.
Anyone else interested should investigate the feature set (see the changelogs) & thoroughly test to satisfy themselves.

After a bit of sorting out dependencies & versions the following rpms appear to install OK.

rpm -Uvh *.rpm
e-smith-lib-1.15.1-36.noarch.rpm
e-smith-mailfront-1.5.2-03.noarch.rpm
ipsvd-0.11.0-02.i386.rpm
perl-Authen-PAM-0.14-1.0.rh7.dag.i386.rpm
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart
/etc/init.d/smtpfront-qmail restart

You can get the correct perl-Authen-PAM from
http://www.rpmfind.net/linux/rpm2html/search.php?query=perl-Authen-PAM&submi
(Without the correct version of this you will break admin console, server manager & a few other things).

You can get the other 3 rpms from the v7.0alpha6 CD.

WARNING: This is an experimental installation of these rpms. Basic functionality appears to work but they have not been extensively tested or used on a production server. If your are brave and server breakage is tolerable then they could be installed on a non critical production server but make sure you have tested thoroughly first.

The developers have advised that mail system functionality may change with newer versions of rpms as the v7.0alpha releases are further developed.
It's possible that by using rpms from versions after v7.0alpha6, other as yet unknown dependency issues may arise.

Use at your own risk & only if you know how to get yourself out of trouble.

If you get into trouble using these, reinstall original rpm versions using
rpm -Uvh --replacepkgs --oldpackage *.rpm
and uninstall ipsvd
rpm -e ipsvd

It's a good idea to check the versions you currently have installed (& also have a copy of those rpms to reinstall if needed)
rpm -q e-smith-lib e-smith-mailfront ipsvd
perl-Authen-PAM

After reinstalling any of the above do
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart
/etc/init.d/smtpfront-qmail restart

Good luck.

[CharlieBrady]

If you get into trouble using these, reinstall original rpm versions using
rpm -Uvh --replacepkgs --oldpackage *.rpm
and uninstall ipsvd
rpm -e ipsvd

There's no need to uninstall ipsvd - it won't cause any problems if it's not used (as it won't be by earlier (i.e. current) e-smith-mailfront).

[DexterUS]

Hi Ray,

Sorry but I am a newbee trying to learn what I can.

I have only 5 RBLs installed in my SME Spam Filter Panel and as you already know the SME version 6.0 sends outgoing emails through the list of RBLs installed and that is a bug that you have fixed in the above post dated April 20th, 2005. In your fix above you mention that I would need "3 RPMs from the v7.0alpha6 CD" and a per-Authen-PAM, can you write me a How To that will automatically get the RPMs needed (with the http:// paths) and install them so all I have to do is just copy and paste your How To steps on my command line?  I will certainly understand if this is too basic for you and you do not want to spend the time but I really would appreciate your expert help. Please

Thanks a Bunch!
DexterUS@Hcm3.US

[raem]

DexterUS

I think there is enough informationm in my post.
Here are some additional tips of one way to do it.

create ibay accessible by your windows w/s (I assume you have a win w/s)
go to site mentioned using a browser, locate the file and download it saving to the ibay
http://www.rpmfind.net/linux/rpm2html/search.php?query=perl-Authen-PAM&submi
perl-Authen-PAM-0.14-1.0.rh7.dag.i386.rpm

Insert sme alpha CD into win w/s and copy files from rpms folder to the ibay
e-smith-lib-1.15.1-36.noarch.rpm
e-smith-mailfront-1.5.2-03.noarch.rpm
ipsvd-0.11.0-02.i386.rpm

log in to server as root
cd /home/e-smith/files/ibayname/files or ../html depending how your ibay is set up an dwhere you copied the files to.

rpm -Uvh *.rpm
and issue the other command mentioned previously
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart
/etc/init.d/smtpfront-qmail restart

[DexterUS]

Now I understand better how it is supposed to work!

Thanks!
Dexter

[firstbishop]

Picking up on something that Charlie suggested earlier in this thread, I installed his e-smith-mailfront-1.5.0-14cb4 contrib to ensure that internal mail was not being checked against the RBL list. I understand that this contrib rejects mail that is not addressed to valid users. From I can see though, this seems to include any pseudonyms that I had set up.

Is there a way to configure the new mailfront contrib to allow mail addressed to pseudonyms to be delivered? Or can I tweak the older contrib that I have now re-installed (e-smith-mailfront-1.5.0-13) to ensure that internal mail is not checked against the RBL list. I saw mention earlier in this thread of tcprules - can anyone give me some pointers if this is where I need to start and what sort of changes would need to be made?

Thanks

[raem]

firstbishop

>...I installed his e-smith-mailfront-1.5.0-14cb4 contrib...

As mentioned earlier I was still having problems with that rpm even after the manual fixes that Charlie suggested (and some more as well).
In my opinion it needs more work doing on it to be ready for production use, and it won't be Charlie who will be fixing it.

You may be better off trying the rpms from sme7alpha that were mentioned earlier. I have had them working on a 6.0 test box.

[firstbishop]

I should have paid a little more attention to your earlier post, Ray. I installed the 4 rpms you refer to and they did break the server console, server manager etc. (Although I did have the correct version of perl-Authen-PAM contrib).

Have now managed to backtrack and reinstalled the original contribs (wouldn't really describe myself as 'someone who can get myself out of trouble' - but have learned a bit in the process!).

Is there another way to do this?

Mike

[raem]

firstbishop

> I installed the 4 rpms you refer to and they did
> break the server console, server manager etc.

Now you have got me confused. You were originally referring to installing
e-smith-mailfront-1.5.0-14cb4

but now you seem to be referring to

rpm -Uvh *.rpm
e-smith-lib-1.15.1-36.noarch.rpm
e-smith-mailfront-1.5.2-03.noarch.rpm
ipsvd-0.11.0-02.i386.rpm
perl-Authen-PAM-0.14-1.0.rh7.dag.i386.rpm
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart
/etc/init.d/smtpfront-qmail restart


Which rpm(s) are you trying to install ?
Follow the steps again, carefully.

[firstbishop]

I originally installed Charlie's contrib which you mentioned still has some issues. Because it bounced mail addressed to pseudonyms, I then re-installed e-smith-mailfront-1.5.0-13.

You suggested that I try the SME7 contribs and I installed the four rpms (perl-Authen-PAM-0.14-1.0.rh7.dag.i386.rpm, e-smith-lib-1.15.1-36.noarch.rpm,
e-smith-mailfront-1.5.2-03.noarch.rpm and
ipsvd-0.11.0-02.i386.rpm) which I have now uninstalled as they broke my server.

Sorry for any confusion!

Is there another way to ensure that internal mail is not checked against the RBL list?

Mike

[raem]

firstbishop

> I installed the 4 rpms you refer to and they did
> break the server console, server manager etc.
> (Although I did have the correct version of perl-
> Authen-PAM contrib).

I suspect you grabbed the perl-Authen-PAM rpm from the 7 alpha, which is the wrong one.
Check your system with
rpm -q perl-Authen-PAM
to see which version is really installed.

You need
perl-Authen-PAM-0.14-1.0.rh7.dag.i386.rpm
You can get the correct perl-Authen-PAM from
http://www.rpmfind.net/linux/rpm2html/search.php?query=perl-Authen-PAM&submi

[raem]

Has anyone reading this thread tried the sme7alpha e-smith-mailfront-1.5.2-03.noarch.rpm on a 6.x production box and actually got the mail system working.

eg
rpm -Uvh *.rpm
e-smith-lib-1.15.1-36.noarch.rpm
e-smith-mailfront-1.5.2-03.noarch.rpm
ipsvd-0.11.0-02.i386.rpm
perl-Authen-PAM-0.14-1.0.rh7.dag.i386.rpm
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart
/etc/init.d/smtpfront-qmail restart

I just tried myself on a production box. The install runs OK but afterwards I can send mail internally or externally but cannot receive mail from external sources. There is nothing in the logs to indicate what the problem is as no mail is coming in.
Outgoing mail logs look OK.

On a remote server I can see mail sent to myself from that server (using webmail or forwarded back to me) sitting in the mail queue and not getting delivered (in the short term).
Mail delivery from & via that server is usually instantaneous.
Mail from a free online account does not get received either.

Mail sent locally to myself does get delivered OK.

I couldn't keep my mail server not working for long, so reverted to the originally installed rpms and all is OK now, so it's something to do with the new rpm/tcpsvd (?), but I don't know what.

Can anyone shed some light on this please, Charlie or Gordon perhaps ?

Regards
Ray