Koozali.org: home of the SME Server

Spam getting through spamassassin.

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #15 on: September 13, 2019, 07:47:00 AM »
As I read the uribl.com website (http://uribl.com/datafeed.shtml) if you want to continue using the current config (using DNS for lookups),

----cut-------

On your pi-hole system, run these commands to create /etc/dnsmasq.d/05-pihole.conf and restart dns:
Code: [Select]
nslookup -type=ns multi.uribl.com |grep '=' |sed 's/.*= //' |while read a; do host -i $a  |sed 's/.*address /server=\/multi.uribl.com\//'; done > /etc/dnsmasq.d/05-pihole.conf
pihole restartdns

----cut-----

In case of problems, remove the custom config file and restart dns again:
Code: [Select]
'rm' /etc/dnsmasq.d/05-pihole.conf
pihole restartdns


Wow! Many thanks for this - have tried it on the pihole system and it seems to work a treat. Many thanks for this.

I'd also posted on the pihole discourse forum here:

https://discourse.pi-hole.net/t/spamassassin-uribl-access-not-working-through-pihiole/23477/4

So I've posted a link to this thread.
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Spam getting through spamassassin.
« Reply #16 on: September 13, 2019, 02:02:38 PM »
Here is a script that I think could be scripted to run regularly to update multiple DNS BL services in pihole:
Code: [Select]
#!/bin/bash
# list of lists for conditional forwarding. Surround with quotes, separate with spaces
lists="zen.spamhaus.org bl.spamcop.net truncate.gbudb.net ix.dnsbl.manitu.net b.barracudacentral.org dbl.spamhaus.org rhsbl.sorbs.net multi.surbl.org black.uribl.com multi.uribl.com"

# path to custom config file to be updated
customcfg=/etc/dnsmasq.d/05-pihole.conf

# create customcfg file if it does not exist to avoid errors with 'sed -i'
if [ ! -f $customcfg ]
then
  touch $customcfg
fi
 

for list in $lists
do
  # delete the old entries for this list
  sed -i "/$list/d" $customcfg

  # get the name servers for this list, then get their IPs
  host -t ns $list |sed 's/.*name server //' | \
  while read ns
  do
    host -i $ns |\
    grep 'has address' |\
    sed "s/.*has\ address\ /server=\/$list\//"
  done >> $customcfg
done

# restart the dns service to activate the changes
pihole restartdns

Changes from the original command:
- using 'host -t ns' instead of 'nslookup -type=ns' since the pihole examples use host
- loops through multiple blocklists
- removes old entries for each blocklist, then recreates them, instead of overwriting 05-pihole.conf to avoid conflicting with other customizations

The script above works for all of the DNSBL services listed except multi.surbl.org.  multi.surbl.org blocks lookups from large dns servers -- so host -t ns multi.surbl.org fails from my pihole, or if I redirect the query to Google, but works from my SME server using tinydns/dnscache and direct lookups through the root servers.

To handle multi.surbl.org (until they change their name server naming scheme):
Code: [Select]
#!/bin/bash
customcfg=/etc/dnsmasq.d/05-pihole.conf

if [ -f $customcfg ]
then
  sed -i '/surbl.org/d' $customcfg
fi

for h in {a..n}
do
  host -i $h.surbl.org |\
  grep 'has address' |\
  sed "s/.*has\ address\ /server=\/$list\//"
done >> $customcfg

pihole restartdns

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #17 on: September 14, 2019, 08:25:05 AM »
Here is a script that I think could be scripted to run regularly to update multiple DNS BL services in pihole:

--cut---

So you think that the other blocklists also suffer from the same problem? I've not seem any sign of it, although I'm not discounting it.
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #18 on: September 14, 2019, 08:48:06 AM »
I've run the script on my pihole and it fails on:
bl.spamcop.net (not as expected?)
multi.surbl.org (as expected)



Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #19 on: September 14, 2019, 10:03:27 AM »
Shouldn't:

  sed "s/.*has\ address\ /server=\/$list\//"

be:

  sed "s/.*has\ address\ /server=\/$h.surbl.org\/$list\//"

In the "special" handling for surbl?
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Spam getting through spamassassin.
« Reply #20 on: September 14, 2019, 05:57:13 PM »
Shouldn't:
...

Worse than that - it should have been hard coded as 'multi.surbl.org'.

However, I think I have a new version of the script that works for multi.surbl.org without breaking it out.  I found that using "dig +trace ..." instead of "host -t ns ..." worked with my pihole configured to use OpenDNS for upstream.

I've posted the latest version to github here:
https://github.com/mmccarn/pihole/blob/master/bin/pihole-dnsbl.sh

and here is the code:
Code: [Select]
#!/bin/bash
# list of lists for conditional forwarding. Surround with quotes, separate with spaces
lists="zen.spamhaus.org"
lists="$lists bl.spamcop.net"

# truncate.gbudb.net does not have its own NS so we use gbudb.net
lists="$lists gbudb.net"
lists="$lists ix.dnsbl.manitu.net"
lists="$lists b.barracudacentral.org"
lists="$lists dbl.spamhaus.org"
lists="$lists rhsbl.sorbs.net"
lists="$lists multi.surbl.org"
lists="$lists black.uribl.com"
lists="$lists multi.uribl.com"


# path to custom config file to be updated
customcfg=/etc/dnsmasq.d/15-pihole.conf

# create customcfg file if it does not exist to avoid errors with 'sed -i'
if [ ! -f $customcfg ]
then
  touch $customcfg
fi
 

for list in $lists
do
  echo Processing $list
 
  # delete the old entries for this list
  sed -i "/$list/d" $customcfg

  # insert a divider in the config file
  printf "# $list\n" >> $customcfg

  # get the name servers for this list, then get their IPs
#  host -t ns $list |grep "name server " |sed 's/.*name server //' | \
  dig +trace $list |grep $list.*NS |sed 's/.*NS\s*//' | \
  while read ns
  do
    printf "  $list\t$ns\n"
    host -i $ns |\
    grep 'has address' |\
    sed "s/.*has\ address\ /server=\/$list\//" >> $customcfg
  done
done

# restart the dns service to activate the changes
pihole restartdns

Changes:
* I tried to make "lists" easier to read
* I switched from "host -t ns ..." to "dig +trace ..."
  ==> this change makes the separate script for multi.surbl.org unnecessary
* I am inserting a commented separator into the dnscache custom config file
* I have changed the custom config filename from 05-pihole.conf to 15-pihole.conf to avoid conflicting with other instructions I've seen online using 05-pihole.conf
* The program now lists each name server on the screen as it is processed

If you switch to this script you should remove the original custom conf at /etc/dnsmasq.d/05-pihole.conf

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Spam getting through spamassassin.
« Reply #21 on: September 15, 2019, 12:41:57 AM »
So you think that the other blocklists also suffer from the same problem? I've not seem any sign of it, although I'm not discounting it.

(I didn't answer this...)

Yes, I've seen the same problem with other DNSBL services - barracudacentral and at least one other (but I forget which)

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #22 on: September 16, 2019, 10:30:25 AM »
I'm getting this from that dig command:

[root@pihole ~]# dig +trace gbudb.net

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> +trace gbudb.net
;; global options: +cmd
;; connection timed out; no servers could be reached
[root@pihole ~]#

It "works" on my Fedora 30 desktop, but not on the pihole Container.

However:

dig @8.8.8.8 +trace gbudb.net

seems to work!
« Last Edit: September 16, 2019, 10:43:10 AM by brianr »
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Spam getting through spamassassin.
« Reply #23 on: September 16, 2019, 12:14:05 PM »
I got a few timeouts using 'dig +trace...', but then they would go away.  I had assumed I was breaking pihole while working on the script...

I've updated the script on github to add '@8.8.8.8':
https://github.com/mmccarn/pihole/commit/2650dfbd9a4de3382102d0621fd4b1cc0c4a026c

I notice from pihole.log that dnsmasq is sending the RBL lookups to every server listed in the config file - 44 servers in the case of multi.surbl.com.  I wonder if the config file should be limited to 1, 2, or 3 entries per blocklist...

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #24 on: September 16, 2019, 12:48:38 PM »
I notice from pihole.log that dnsmasq is sending the RBL lookups to every server listed in the config file - 44 servers in the case of multi.surbl.com.  I wonder if the config file should be limited to 1, 2, or 3 entries per blocklist...

Mine has 45. Do we need more than 1?
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Spam getting through spamassassin.
« Reply #25 on: September 16, 2019, 04:26:24 PM »
Been playing with this on mine.

However, when I run it via cron I get this error:

Code: [Select]
cat /etc/cron.d/dnsupdate

5 * * * *   root    PATH="$PATH:/usr/local/bin/" /root/scripts/pihole-dnsbl.sh

Quote
Restart pihole DNS  [✗] /usr/local/bin/pihole: line 121: service: command not found

Here are the lines but no idea why it fails :-(

Code: [Select]
114    svc="service ${resolver} ${svcOption}"
......
121    output=$( { ${svc}; } 2>&1 )
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #26 on: September 17, 2019, 10:19:16 AM »
Been playing with this on mine.

However, when I run it via cron I get this error:

Code: [Select]
cat /etc/cron.d/dnsupdate

5 * * * *   root    PATH="$PATH:/usr/local/bin/" /root/scripts/pihole-dnsbl.sh

Here are the lines but no idea why it fails :-(

Code: [Select]
114    svc="service ${resolver} ${svcOption}"
......
121    output=$( { ${svc}; } 2>&1 )

I dropped mine into /etc/cron.daily and it seems to have run ok last night:

Code: [Select]
root@pihole ~]# ls -l /etc/dnsmasq.d/
total 12
-rw-r--r-- 1 root root 1885 Sep 12 17:57 01-pihole.conf
-rw-r--r-- 1 root root 8310 Sep 17 04:13 15-pihole.conf
[root@pihole ~]#

and nothing in the logs that I can see.
« Last Edit: September 17, 2019, 10:24:21 AM by brianr »
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Spam getting through spamassassin.
« Reply #27 on: September 17, 2019, 10:44:21 AM »
I dropped mine into /etc/cron.daily and it seems to have run ok last night:

Ahhh OK - I had mine as a separate script and tried to run it from cron eg

Script is here

Code: [Select]
/root/scripts/pihole-dnsbl.sh
Cron like this

cat /etc/cron.d/dnsupdate

Code: [Select]
5 * * * *   root    PATH="$PATH:/usr/local/bin/" /root/scripts/pihole-dnsbl.sh
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Spam getting through spamassassin.
« Reply #28 on: September 17, 2019, 01:17:07 PM »
On my pi-hole server (running on raspbian 9 / stretch) the default path inside a cron job is /usr/bin:/bin

The default path in a user shall is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

"service" is in /usr/sbin:
Code: [Select]
which service
/usr/sbin/service

On my system both of these versions of '/etc/cron.d/dnsupdate' successfully regenerate /etc/dnsmasq.d/15-pihole.conf:

Quote from: /etc/cron.d/dnsupdate option 1 - global PATH
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
5 * * * *   root    /root/scripts/pihole-dnsbl.sh

Quote from: /etc/cron.d/dnsupdateoption 2 - local PATH[tt
5 * * * *   root PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" /root/scripts/pihole-dnsbl.sh

Offline brianr

  • *
  • 988
  • +2/-0
Re: Spam getting through spamassassin.
« Reply #29 on: September 17, 2019, 03:27:52 PM »
On my pi-hole server (running on raspbian 9 / stretch) the default path inside a cron job is /usr/bin:/bin

I'm not running on a Raspberry Pi at all. I think the container was built on Centos 7.
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........