Topic: SPF, DKIM, DMARC - problem of understanding if delivering email via ISP SMTP

[SchulzStefan]

Good evening,

as we have no static IP we deliver email via our Internet provider's SMTP server.

Running
# qpsmtpd-print-dns abc.de
brings:

Here are sample DNS entries you should add in your public DNS
The DKIM entry can be copied as is, but others will probably need to be adjusted
to your need. For example, you should either change the reporting email adress
for DMARC (or create the needed pseudonym)


default._domainkey IN TXT "v=DKIM1;p=M12345;t=y"
@ IN SPF "v=spf1 mx a -all"
@ IN TXT "v=spf1 mx a -all"
_dmarc IN TXT "v=DMARC1; p=none; adkim=s; aspf=r; rua=mailto:dmarc-feedback@abc.de; pct=100"

What is this information good for in a scenario like I described? How should I configure the TXT records including SPF, DKIM and DMARC in my domain management at my ISP? We're located in Germany, ISP is STRATO. Maybe someone knows or can help.

Regards,
stefan

[Michail Pappas]

I've encountered this issue on a friend's setup, after he, although a non-admin, decided abruptly to switch to another ISP, creating havoc. As an added issue the new ISP did not put the user on a static, but rather on a dynamic IP! So we used a the ISP's mailgate system for outbound email.

Disclaimer: not an expert here, someone with more knowledge feel free to correct/bash me.

You can use all technologies in your setup AFAIK. In a nutshell, you can work SPF with DKIM in parallel, leaving DMARC until everything is ok with them. Then, install DMARC, monitor results, finetune, monitor again.

More extensively:

1) SPF: Ask your ISP the IPs/DNS names of its outgoing mail servers. Include them in the SPF record. If IP addresses are provided, make sure that both IPv4 and IPv6 addresses are included. Who receives mail for your domain, SME or an ISP box?

An example SPF record might be:

@ IN SPF "v=spf1 ip4:a.b.c.d ip4:e.f.g.h/24 mx a ~all"

Note the ~all (soft fail) at the end. It means a soft fail will take place if someone does not match. You do want that, since an incorrect SPF setup might render outgoing email flagged as SPAM. Once you fine tune, you can switch the DNS record to a "-all" (hard fail).

In any case, do not send any DNS TXT records to your ISP yet. First, write here what your ISP has provided you with, with regard to my questions above. Write them down here, to help you build an SPF record.

2) DKIM: There's an excellent tutorial in the wiki for DKIM. Just pick (only) the DKIM record line from the output of qpsmtpd-print-dns and send it to your domain handler.

3) DMARC: Well, leave that for the end. Perhaps avoid using the line recommended in qpsmtpd-print-dns. Check perhaps a free service like dmarcian.org or dmarcanalyzer.com

[SchulzStefan]

@Michail,

thank you for answering.

Due to a few reflections and following a few discussions of SPF, DKIM and DMARC (besides SMEOptimizer) I decided to order a fix IP. Right now I think this will make a lot of things easier. I'll be back after the 26th of this month.

Regards,
stefan

[SchulzStefan]

@Michael,

as promised I'm back to this thread.

Yesterday we got our static IP from our DSL provider, which is in our case Deutsche Telekom. What I did so far is to set reverse DNS for the hostnames, www.xyz.de, mail.xyz.de and servername.xyz.de.

Secondly I set the A-record at our ISP (which is in our case STRATO) to the static IP for the main domain. I did not yet create any subdomain which I should probably do for mail.xyz.de and servername.xyz.de. I assume for the subdomains should also the A-record point to the static IP.

I didn't change the MX-record of the domain.

SPF, DKIM and DMARC is untouched. Also the DNS which is still a STRATO DNS.

Right now we send and receive emails still via ISP SMTP and POP3.

How do I have to proceed if I want the SME to be the one and only mailserver (sending and receiving, no more relay via ISP) with valid SPF, DKIM and DMARC policy? TXT Records including SPF and DKIM settings can be set in the general DNS settings at the ISP.

Last question: how can I avoid 550 5.1.0 Dynamic/Generic hostnames are blocked?
"(Connected to 213.46.255.45 but sender was rejected. Remote host said: 550 5.1.0 Dynamic/Generic hostnames are blocked. Please contact your Email Provider. Your IP was 87.xxx.xxx.xxx. Your hostname was p123456.dip0.t-ipconnect.de)"

Static IP is correct. The hostname seems to be the dial-in of the Deutsche Telekom. It was an active reverse DNS mapping which I deleted yesterday. The router has been restarted. I have no idea where/why this is still active. Where/how can I change/delete this?

I tried a test with the SME and a configuration I thought it might be work. Receiving of email worked fine, sending of email caused in this error and the mail has been rejected.

Thank's for any help.

Regards,
stefan

[kruhm]

Hi Stefan,

I understand what you want to do. This type of setup was a good setup 10 years ago. Possibly even 5 years ago.

But in 2017, it will be continuous problems. You are not going to be able to get around your ip being in a dynamic range unless you change service.

Your best bet is to use a smtp service like AWS SES or similar.

Regards from Jena

[Michail Pappas]

If your servers ip addresses are blacklisted in the PBL (https://www.spamhaus.org/pbl/), then no matter what you do, you will not be able to send directly...

[SchulzStefan]

You are not going to be able to get around your ip being in a dynamic range unless you change service.

Your best bet is to use a smtp service like AWS SES or similar.

Regards from Jena

Hi kruhm,

I don't get it. I ordered a static IP from the Telekom which is still dynamic? Wow. Don't know what to say. Are they cheating their customers?

You recommend a smtp service. Isn't it still a relay? Where is the difference to send over STRATO (what we're doing right now)?

Right now I'm more confused as before. What sense does it make to use SPF, DKIM and DMARC if there are huge efforts to take to make this work? In fact it seems to me, even with a (maybe so called) static IP, it'll not work for a (for me) normal setup. How can this be? Technically SME is able to work with it - but how to get this running in the real world?

I'd be very interested in how many SME admins have a mail setup using SME as a real (not only internal) mailserver. Receiving and sending email ONLY and DIRECT with the SME to the world. Anybody? Or is it more receiving and sending is over a relay? Would be nice to get more info.

Regards,
stefan

[ReetP]

I think you may find your IP is most likely in a 'block' that is considered to be 'Home Use/Residential'. I have that as well, despite being a supposedly 'Business IP'.

Most of the big boys do not want you running a local mail server, period. In part because that is where a lot of Spam comes from.

M$ has used that to leverage it's products. They offer 365 (346 1/2?) to ISPs and tell them that they will handle all the mail and take the cost and hassle off the ISP (and probably give them a backhander). You then find that your linuxy mail server can't use M$ 365 servers as a relay as it needs individual account authentication (see https://bugs.contribs.org/show_bug.cgi?id=9050)

Nothing much will change unless you either (most likely) pay a lot more for a different 'level of service' with your provider, or change provider entirely.

An alternative may be to get yourself a small cloudy VM say like Digital Ocean Vultr or somesuch and use that as your SMTP server/relay. I have a system like that myself for exactly this reason. It can be SME as you have virtual ETH ports. You can lock it right down, and just use it as a mail server. Even use an encrypted connection from your local host to the virtual one with ipec or openvpn.

That or you have to use a paid for SMTP service as previously suggested.

HTH

[kruhm]

ReetP is correct of course.

It's an issue with the World of today of cloud services and ransomsware. There's no such thing as Internet police. While there are official rules in the form of RFC's, it's simply not enough for today's current issues.

 SPF, DKIM and DMARC are not official rules but they are necessary in today's world of constant attacks and virtual machines.

Even From the enterprise level, most are moving to  some sort of hosted front ends to take the load off their servers for receiving or use a SMTP service for sending even if they use GA/ O365.

Products that I have experience with like Mimecast, and others found here are worth it but the bill is astronomical: https://community.spiceworks.com/cloud/anti-spam/reviews

Luckily the current situation only needs sending/smtp services. These are about $0.10/1000 emails. My bill last month was $3 usd.

Also, I have bare metal servers at datacenters that are in use and interface directly with the world. In an AWS cloud world, it becomes increasingly difficult to justify. The bill is way too high compared to cloud.

To sum up your options:
1-get a business class isp with static. (Not residential service)
2-host the server at a datacenter.
3-put the server on the cloud, AWS or others.
4-use a smtp service.
5-others???

Option 4 is going to be your most affordable. It may even save money from your current setup.

[ReetP]

ReetP is correct of course.

Reetp falls off his stool laughing ! I got something right for once

To sum up your options:
1-get a business class isp with static. (Not residential service)
2-host the server at a datacenter.
3-put the server on the cloud, AWS or others.
4-use a smtp service.
5-others???

Option 4 is going to be your most affordable. It may even save money from your current setup.

Yup, that is about the size of it.

I plumped for two solutions - one was to move my main email/crm server onto a cloud box (SME of course!). I did have other factors affecting this decision, but the email situation is what prompted me to get off my arse and move/change things. That did my work stuff.

I also have a small VM that I use JUST for VPN and SMTP send only for some other domains. Cost is about USD 10 per month.

No need to use my ISP for email now so it is all ISP independent. Worth the trouble IMHO.

B. Rgds
John

[SchulzStefan]

I think you may find your IP is most likely in a 'block' that is considered to be 'Home Use/Residential'. I have that as well, despite being a supposedly 'Business IP'.

We are a company, and Telekom says we are business customers. At least we pay for exactly to be this. So they are cheating their customers. Should I talk to my lawyer?

To sum up your options:
1-get a business class isp with static. (Not residential service)
2-host the server at a datacenter.
3-put the server on the cloud, AWS or others.
4-use a smtp service.
5-others???

Option 4 is going to be your most affordable. It may even save money from your current setup.

I did a blacklist check on https://mxtoolbox.com. The IP is *only* listed in Sender Score Reputation Network, no where else. Should I check more?

How about option 5: at https://senderscore.org/rtbl it is offered to remove the IP.

They say: "Removing your IP

If you have fixed problems on your network recently, or think this listing may be in error please use the form below to be removed from the list.  You should be aware that while this will remove you from the current list it does not indicate a permanent removal. New problems detected beyond this time will result in a new listing."

Doesn't this make sense?

Back to option 4 - does that mean you don't use SME as mailserver for sending emails? Where is the difference using a STRATO mailserver as relay? And in both cases, what is SPF, DKIM and DMARC good for?

In the case of STRATO I'll point the A-record to the static IP, I enter in the DNS-management for my domain SPF and DKIM as TXT-records. So why not using this? Assuming the IP is not blacklisted.

IMVHO SME was designed for small to medium sized enterprises. One can argue, that fileserver and internal mailserver (besides all other important features) is good enough. Sending and receiving email NOT using an ISP as relay, seems to be a battle. Therefore again the question, SPF, DKIM and DMARC what is this good for if only used internal, and how to use it if a mailserver of an ISP is used as relay. Or are most SME servers only be used to receive email? Sending goes over ISP?

Sorry, I still don't get it.

Regards,
stefan

[SchulzStefan]

I'm getting more and more confused.

The result of trying to remove the IP results in:

Your IP is not currently listed in the Return Path Reputation Network Blacklist.

http://whatismyipaddress.com/blacklist-check brings up no listing at all...
http://www.dnsbl.info/dnsbl-database-check.php brings up no listing at all...
http://mail-blacklist-checker.online-domain-tools.com/ no listing at all...
http://www.blacklistalert.org/ no listing at all...

Doing the test again on https://mxtoolbox.com

brings now up - nothing. No listing. How can that be? 20 minutes before it was listed in Sender Score Reputation Network.

The domain seems also nowhere blacklisted.

Don't know what to think about this all...

[SchulzStefan]

It seems, that some DNS servers need more than 24 hours to public all changes. I'd like to cool everything down and will report further.

But another issue occurs: is this still recommended?

https://forums.contribs.org/index.php/topic,50864.msg256865.html#msg256865

Got exactly the same message. Any problems known if a custom template is created in the way mmccarn is suggesting? There might be a difference: I think in my case it should be mail.xyz.de as the reverse DNS is named like that. How could that be done?

Instead of { "$SystemName.$DomainName" } would { "$HeloHost.$DomainName" } work?

Edit: it doesn't. What works is mail.$DomainName. I don't know if this would be smart. Is there a variable for "mail"?

Edit: not quite sure if the reverse DNS has to be set to mail.xyz.de or $SystemName.$DomainName. Right now it's mail.xyz.de.

Regards,
stefan

[kruhm]

I can see you are really determined to have it setup with your dedicated ip. It isn't impossible but it may take some extra time and effort. (This is option 1 for everyone following along).

It's going to be hard to diagnose without knowing specifics but I'll help as much as I can.

The server is designed to send and receive email all by itself.

The operational steps are rather simple:
-make sure your dns records are to your ip.
-make sure your spf records include your ip.
-send email out via your server (not through your isp smtp).

See what errors you come across. Diagnose and fix them.

There are, of course, options to change this default setup. Option 4 (Smtp service) is authorizing another company to handle the sending of your mail. There are reasons why you would want this.

And using your isp as a relay might be possible but a really bad option. So bad that it shouldn't be mentioned.

With spf, you are telling everyone where your email is coming from. This way if it comes from somewhere else, I know the email is invalid.

Your ip has to be clean and not on a blacklist. So if the person who owned it before you was naughty, you're cleaning up after them.

Your reverse dns (or ptr record) has to be clean as well. Above you mentioned the host name is t-ipconnect.de. This probably won't pass.

In any event, give it go and see how the results turn out.

[SchulzStefan]

As promised I'm back to share my experiences running SME mailserver with a static IP (which does not match anylonger the topic, maybe it should be changed).

The server location is in Germany, ISP is German Telekom and domain hoster is STRATO. If anyone intends to run on the same path I highly recommend to start the whole process on a weekend. This is because the DNS-servers needs time to sync everything properly.

First get a static IP. The IP I got from the Telekom was blacklistet first. I don't know why or how, but some time later it was not anymore on any blacklist. Which is still a miracle to me. This should be checked carefully, otherwise your email might not reach your receipients. Next step would be to point the static IP on your server. This has to be done in the Telekom settings. Go to the reverse DNS. It has to be your servername.foobar.com. The Telekom allows up to 30 reverse DNS entries. I made the mistake to enter also the mail.foobar.com as reverse DNS. That did not work. There can be only one reverse DNS and this has to be the servername.foobar.com.

Now go to your domain hoster which is in my case STRATO. If you don't have already subdomains you need to create at least mail.foobar.com, servername.foobar.com and if you have letsencrypt installed, you should also create the subdomain www.foobar.com.

In the domain settings of the domain you have to point the A-record and the AAAA-record to the static IP (IP4 and IP6) of your SME mailserver. The MX-record has to be your own mailserver and named to mail.foobar.com. You'll do this for your main domain and the two subdomains servername.foobar.com and mail.foobar.com.

In the main domain you add as TXT-records your spf and DKIM settings/values. For the DKIM is a prefix needed: default._domainkey. The spf needs no prefix, just the TXT between the quotes. Don't use the quotes if you grep from the command line qpsmtpd-print-dns. For STRATO it's not working with the quotes. Unfortunatley STRATO does not support DMARC in their DNS servers. You may enter the DMARC TXT-record anyway, it does not harm.

If all this is done, be patient. Better 48 than 24 hours.

If you used fetchmail in your email configuration of your SME server don't forget to change this now. Same for the SMTP settings. If you fetched the email from a catchall account you might want to set this account temporarily inactive. Now you're ready to test your SME mailserver with a static IP.

You may test your email with https://www.mail-tester.com/. In my case the result is a score of 8.9/10. There's still an issue with the DNS and DKIM (I'm told to do the test again in 12 hours, in fact the DKIM hasn't been changed in the last 48 hours, so I don't know why, maybe it's still taking time to sync all DNS servers.) There's shown the old and the new key, which is pretty much the same except of removing the "t=y" at the end of the key. This gives me a -1 of the overall score.

And of course, DMARC is missing simply because STRATO does not support it.

This works for me, maybe it can help others to save time.

Regards,
stefan