Koozali.org: home of the SME Server

Email Whitelist-Blacklist Control

Offline Fumetto

  • *
  • 874
  • +1/-0
Email Whitelist-Blacklist Control
« on: November 04, 2021, 06:20:39 PM »
In relation to the contrib in question, have some doubts. Can it be used in production?
It is not in the list of supported contrib on 10 (and not even on 9) ...
But there is the page on the wiki ... but with a open bug (actually on 10rc) that talks about MULTIPLE_RPM_OWNERS.

Now ... if I understand correctly, the contrib may "interfere" with the normal functioning of the SME.

I would like to use it to block, from a panel manager (Black Lists: REJECT), the arrival of "persistent" spam from certain IPs ...

At what point is it? It can be used? If still "buggy" Can I help out? Like?

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #1 on: November 04, 2021, 06:34:48 PM »
Now ... if I understand correctly, the contrib may "interfere" with the normal functioning of the SME.
At what point is it? It can be used? If still "buggy" Can I help out? Like?

Nope it is released. The instructions do not mention dev or test.

Quote
The latest version of smeserver-wbl is available in the SME repository, click on the version number(s) for more information.

However, like any software there may be bugs or other issues. It is the nature of the beast.

As a member of the community it is *your* software and *your* problem as just much as anyone else. So please, get a test machine and test the software and look at the bugs. Help us confirm/verify them. Satisfy yourself that you can use it on your server.

Talk to the devs and see how you can help - you do NOT have to be a coder to help us.

Thanks.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #2 on: November 04, 2021, 06:48:43 PM »
Ok. So I have to "just" make sure it works and to do that I have to check /var/log/qpsmtpd/current until I find a line with the offending IP and "see" if it is correctly "trashed" (dispatching EHLO ....). Correct?

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #3 on: November 04, 2021, 07:09:53 PM »
Ok. So I have to "just" make sure it works and to do that I have to check /var/log/qpsmtpd/current until I find a line with the offending IP and "see" if it is correctly "trashed" (dispatching EHLO ....). Correct?

Yes I would keep a watch on the mail logs and see what comes and goes.

Check the bugs.
11678 is not a blocker - a RPM annoyance we need to fix
10472 - a NFR
10117 - a NFR
9276 - might be part done - worth a look and comment - may want moving to v10 but not sure where it is at.
9275 - might be fixed? Please test/check
4664 - move to future.....

You can see all related bugs here - needs a bit of search wrangling to do this sort of thing. Practice.....

https://bugs.koozali.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=NEEDINFO&bug_status=IN_PROGRESS&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&component=smeserver-wbl&list_id=96012&product=SME%20Contribs&query_format=advanced

Ask me for a Rocket account and come and talk to us if you want to test. We can help.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Email Whitelist-Blacklist Control
« Reply #4 on: November 04, 2021, 09:43:58 PM »
This bug https://bugs.koozali.org/show_bug.cgi?id=11687 was the main issue, James Walsh doing some dirty work to help it through :-)
--
qui scribit bis legit

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #5 on: November 05, 2021, 12:55:13 PM »
Think I can help for 11687 and the others, i will do a couple of tests this weekend and update the bugs.

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #6 on: November 05, 2021, 02:50:00 PM »
Think I can help for 11687 and the others, i will do a couple of tests this weekend and update the bugs.

Fab. Ask if you need help.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #7 on: November 05, 2021, 10:54:49 PM »
I did some testing. And I think I did some "bullshit", because I gave a useless "signal-event email-update" since bug 11687 is "Status: CLOSED FIXED".

But ... I discovered that if I blacklist the server IP it doesn't work, if I put the host name the mail is correctly "trashed".

Quote from: With IP in Blacklist qpsmtpd badhelo
2021-11-05 22:23:02.552956500 17480 dispatching EHLO hostingweb67-116.smarthostingprovider.net
2021-11-05 22:23:02.609705500 17480 (ehlo) helo: pass
2021-11-05 22:23:02.610746500 17480 250-my_sme.it Hi hostingweb67-116.smarthostingprovider.net [89.xx.xx.116]
Quote from: =With hostname in Blacklist qpsmtpd badhelo
2021-11-05 22:25:50.878341500 17647 dispatching EHLO hostingweb67-116.smarthostingprovider.net
2021-11-05 22:25:50.879624500 17647 (ehlo) helo: karma -1 (-1)
2021-11-05 22:25:50.879650500 17647 (ehlo) helo: fail, NAUGHTY, in badhelo
2021-11-05 22:25:50.880641500 17647 250-my_sme.it Hi hostingweb67-116.smarthostingprovider.net [89.xx.xx.116]
...
2021-11-05 22:25:51.245320500 17647 550 (helo) I do not believe you are hostingweb67-116.smarthostingprovider.net.
2021-11-05 22:25:51.245604500 17647 click, disconnecting

So the wording in the wiki page  "Alternatively you may use the ip address." is wrong. Need hostname (or I found another bug).

...and an email was received by the sending server...
Quote from: Mail delivery failed: returning message to sender
This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed:

  admin@sending_sme.it
    host server.sending_sme.it [79.xx.xx.129]
    SMTP error from remote mail server after pipelined MAIL FROM:<admin@sending_sme.it> SIZE=4939:
    550-(helo) I do not believe you are hostingweb67-116.smarthostingprovider.net.
    550 (helo) I do not believe you are hostingweb67-116.smarthostingprovider.net.
« Last Edit: November 05, 2021, 11:17:00 PM by Fumetto »

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Email Whitelist-Blacklist Control
« Reply #8 on: November 05, 2021, 11:41:59 PM »
I did some testing. And I think I did some "bullshit", because I gave a useless "signal-event email-update" since bug 11687 is "Status: CLOSED FIXED".

But ... I discovered that if I blacklist the server IP it doesn't work, if I put the host name the mail is correctly "trashed".

So the wording in the wiki page  "Alternatively you may use the ip address." is wrong. Need hostname (or I found another bug).

...and an email was received by the sending server...

Lodge a new bug, better to document something than have it slip into the mist and be forgotton about..good work
--
qui scribit bis legit

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #9 on: November 05, 2021, 11:50:21 PM »
This is the second time since the release of SME10 that I start "playing" with SME as I did when I started (from version 7.4 if I remember correctly) and it is the second "bug" that I find (not a major bug in my opinion, actually wrong documentation) ... I guess I have to "play" more often ... and I think need an account on Rocketchat ... I'm not a coder but, as a bug finder I have no rivals ... :-D .
« Last Edit: November 05, 2021, 11:54:01 PM by Fumetto »

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #10 on: November 06, 2021, 12:28:24 AM »
First, document carefully exactly what you did to create the issue so others can try to duplicate & find it.

"I set IP address in box X
I ran email-update
I observed Y"

Then message me here your real name and an email adress for notifications and I'll set up an account for you.

It's my work test server so I don't allow general access to keep the spam down!
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #11 on: November 06, 2021, 12:28:51 AM »
PS well done and thanks for helping!!
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #12 on: November 06, 2021, 12:59:02 AM »
First, document carefully exactly what you did to create the issue so others can try to duplicate & find it.
Done

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Email Whitelist-Blacklist Control
« Reply #13 on: November 06, 2021, 01:00:22 AM »
This is the second time since the release of SME10 that I start "playing" with SME as I did when I started (from version 7.4 if I remember correctly) and it is the second "bug" that I find (not a major bug in my opinion, actually wrong documentation) ... I guess I have to "play" more often ... and I think need an account on Rocketchat ... I'm not a coder but, as a bug finder I have no rivals ... :-D .

always fun, always learning, keep poking the dragon with the blunt stick :-)
--
qui scribit bis legit

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #14 on: November 06, 2021, 12:56:18 PM »
Done

Just moved it to the correct section as it is a SME Contribs issue with the smeserver-wbl contribution :-)
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #15 on: November 06, 2021, 06:58:39 PM »
Can yo check that the host IP starts with @?

As per the wiki page:

https://wiki.koozali.org/Email_Whitelist-Blacklist_Control#Black_Lists:_REJECT

Badmailfrom
@host or user@host


Also you can read the relevant plugin

cat /usr/share/qpsmtpd/badmailfrom


Quote
#!perl -w

=head1 NAME

check_badmailfrom - checks the badmailfrom config, with per-line reasons

=head1 DESCRIPTION

Reads the "badmailfrom" configuration like qmail-smtpd does.  From the
qmail-smtpd docs:

"Unacceptable envelope sender addresses. qmail-smtpd will reject every
recipient address for a message if the envelope sender address is
listed in badmailfrom. A line in badmailfrom may be of the form
@host, meaning every address at host."

You may include an optional message after the sender address (leave a space),
to be used when rejecting the sender.

=head1 CONFIGURATION

=head2 reject

  badmailfrom reject [ 0 | 1 | naughty ]

I<0> will not reject any connections.

I<1> will reject naughty senders.

I<connect> is the most efficient setting. It's also the default.

To reject at any other connection hook, use the I<naughty> setting and the
B<naughty> plugin.

=head1 PATTERNS

This plugin also supports regular expression matches. This allows
special patterns to be denied (e.g. FQDN-VERP, percent hack, bangs,
double ats).

Patterns are stored in the format pattern(\s+)response, where pattern
is a Perl pattern expression. Don't forget to anchor the pattern
(front ^ and back $) if you want to restrict it from matching
anywhere in the string.

 ^streamsendbouncer@.*\.mailengine1\.com$    Your right-hand side VERP doesn't fool me
 ^return.*@.*\.pidplate\.biz$                I don't want it regardless of subdomain
 ^admin.*\.ppoonn400\.com$



And here is the bit of code that does the matching.

Quote
sub is_match {
    my ($self, $from, $bad, $host) = @_;

    if ($bad =~ /[\/\^\$\*\+\!\%\?\\]/) {    # it's a regexp
        if ($from =~ /$bad/) {
            $self->log(LOGDEBUG, "badmailfrom pattern ($bad) match for $from");
            return 1;
        }
        return;
    }

    $bad = lc $bad;
    if ($bad !~ m/\@/) {
        $self->log(LOGWARN, "badmailfrom: bad config: no \@ sign in $bad");
        return;
    }
    if (substr($bad, 0, 1) eq '@') {             ############# <<<<<<<<<<< Note the @ here!!!!
        return 1 if $bad eq "\@$host";
        return;
    }
    return if $bad ne $from;
    return 1;
}


I think I did some test code for whitelists to check what was happening. When I get 5 minutes I can try and do a bit to test this as well but I think the key is adding the @ so try @1.2.3.4

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #16 on: November 07, 2021, 10:21:57 PM »
I tried with "@1.2.3.4" on "qpsmtpd badhelo" and "*@1.2.3.4" on "qmail badmailfrom" and "spamassassin blacklist_from" but but the email arrived anyway.

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Email Whitelist-Blacklist Control
« Reply #17 on: November 08, 2021, 02:43:51 AM »
you are free to tell me to go and do something abnormal if I am asking a kindergarten question BUT, did you add *@1.2.3.4 or a real ip address?
--
qui scribit bis legit

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #18 on: November 08, 2021, 02:47:57 AM »
Real...real... I'm noob, but not at this point... ^_^
Only "@123.123.123.123", no "*".

Offline Fumetto

  • *
  • 874
  • +1/-0
Re: Email Whitelist-Blacklist Control
« Reply #19 on: November 08, 2021, 03:20:43 AM »
The file to which something probably needs to be retouched is /usr/share/qpsmtpd/plugin/helo.
I notice a msg "I do not believe you are $host" under it, and this message is what I can see in the log when a email is blocked

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Email Whitelist-Blacklist Control
« Reply #20 on: November 08, 2021, 03:25:46 AM »
you are free to tell me to go and do something abnormal if I am asking a kindergarten question BUT, did you add *@1.2.3.4 or a real ip address?

:-) always best to ask
--
qui scribit bis legit

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #21 on: November 08, 2021, 10:49:09 AM »
I can't look today but I will tomorrow - I did test all this a long while ago when fixing the whitelist.

Quote
"I do not believe you are $host"

Don't conflate things. You are going down rabbit holes here. You really need to read each plugin,  see when it is used and the messages it throws.

I think this an error from whitelist helo plugin which is detecting a bad helo name. Not part of the black
list check. You can affect this with the whitelisthelo setting.

Note. The blacklist IP format does not use *

If you read the code in the plugin you will see it checks for a couple of things as I have mentioned. Read the errors & grep match them to the plugin files.

The code looks for an IP indicated by an @ symbol.

This bit below.

Quote
  return 1 if $bad eq "\@$host";

So far I don't see an error.

The only thing that possibly could be modified is the better help and syntax checking in the panel, but IIRC syntax checking is tricky.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Email Whitelist-Blacklist Control
« Reply #22 on: November 11, 2021, 10:41:48 AM »
OK, as per the bug once I finally remembered what is what then it was clear the wiki was in error, or misleading.

The Helo plugin is not designed to block specific IPs. It is the nature of the beast, and as per various RFCs on what you can and cannot accept in Helo information. The plugin is designed to stop miscreants abusing the Helo header, not to just block them outright.

In actual fact there is no qpmstpd black list plugin.

Yes, it would be possible to write one, and as pointed out by Jean Philippe it should be done at the connect stage of the transaction to save wasted processing and overhead. However, that is time and effort that we are short of.

Tools like fail2ban, xtgeoip & geoip blocking will block at the firewall level so won't even get to bother qpsmtpd.

https://wiki.koozali.org/Fail2ban
https://wiki.koozali.org/GeoIP
https://wiki.koozali.org/Xt_geoip

If you have a particularly annoying door knocker you can block them outright with some examples here:

https://wiki.koozali.org/Firewall

You can open a new feature request for a blacklist plugin but not sure when we'll get to look at it.

Hope that helps and sorry for the confusion.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Email Whitelist-Blacklist Control
« Reply #23 on: November 11, 2021, 02:00:39 PM »
to be clear the test your are doing and what you are trying to obtain are irrelevant and might lead to inconsistent results .  See rfc https://datatracker.ietf.org/doc/html/rfc5321

emails with ip after the @ should have the ip enclosed in square bracket or could be interpreted as domains.

some systems are tolerating the absence of brackets and some software (horde last time i checked) are even not able to handle the square bracket.

this one more point to just filter ip BEFORE getting to the smtp deamon.