Zimbra Anti-Spam Best Practices 2019
Back in 2014 I wrote a good chunk of the Zimbra Anti-Spam Strategies community wiki. In the intervening five years, a lot has changed, but unfortunately email remains the number one attack vector for malware, phishing, whaling and other scams.
In this blog post we describe how we supplement the various open-source anti-spam capabilities that ship with Zimbra with two commercial block list providers for a low-cost, high-impact solution. For customers with additional compliance, email continuity, data loss prevention and archiving needs, we offer Mimecast, Gartner’s consistent top player in the Secure Email Gateway space.
Our two favorite commercial (i.e. paid) block list providers are invaluement and uribl. Our favorite free block list provider is Barracuda (but be sure to register your IP addresses before trying to use it.) Spamhaus’s terms of service now prohibit usage of their block lists for commercial purposes except via a paid subscription. We found in testing that Spamhaus caught very little that Barracuda didn’t. So we ourselves do not use Spamhaus, though many of our customers do and with good success.
But there is more to anti-spam than just blocking inbound junk email. A good anti-spam strategy extends to endeavoring to ensure that your email system and domain is not a source of spam, so we’ll introduce our best practices in that space as well in a future blog post.
Above all, we cannot overemphasize the need for periodic end-user training. Anti-spam will never be perfect, so users need to be trained when to pause, take a deep breath and think before they open or reply to an email. When a user receives an email from someone they know, but the email seems suspicious, DO NOT REPLY to the email! Instead, train your users to FORWARD the email to the (alleged) original sender. This is because the “From:” portion of an email can be easily forged, but the recipient will have the correct email address for the sender in their Address Book.
This “forged sender” technique is typically how those emails purporting to be from the CFO and ordering expediting payment to a vendor (really the scammer who sent the email in the first place) cause so much trouble; the poor accounting clerk thinks s/he is replying to the CFO, but is really replying to the scammer.
Lastly, in most of the recent email hacks with which I am familiar, two-factor authentication would have prevented the exploit from being deployed, so we are strongly encouraging our customers to deploy two-factor authentication — it’s really not as inconvenient to use as most people presume.
Anti-Spam Quick Level Set: Terminology, Approach and Techniques
Here are some key terms and concepts to make sure we are all on the same page:
Gray Mail – Email that technically isn’t spam, but that most users don’t want. You downloaded a whitepaper and had to give up your email address, so now you are receiving all kinds of offers via email that you technically authorized when you accepted the terms and conditions to be able to download that whitepaper.
Spam – For our purposes, “spam” is email that has been determined to be junk. In the United States, the CAN-SPAM Act has a more rigorous definition. In countries covered by the GDPR, “UCE” or Unsolicited Commercial Email is even more heavily restricted. If an email is unequivocally junk, Zimbra deletes it. If it is marginally junk, Zimbra can prepend the Subject line with a warning and place it in the user’s Junk folder. The threshold between deletion and sending on is adjustable.
Block List – A third-party service that Zimbra can use to identify IP addresses and domains that are known spam sources, either because they are just bad guys (in the eyes of the block list provider) or because they are nice guys that have been hacked. Some block lists are very aggressive in their listing policies; others less so. There are three types of block lists:
Left-hand side block lists for checking IP addresses (“RBLs”);
Right-hand side block lists for checking domains (“RHSBLs”), and;
URI block lists for checking links within an email body (“URIBLs”).
False Positive – A legitimate email incorrectly identified as spam. Block Lists are rated on their frequency of false positives. Typically, the higher the rate of false positives, the less spam a block list will allow. This trade off between efficacy and accuracy warrants a discussion with management. Customers that have high requirements for both low false positives and near-zero spam are typically prospects for additional commercial third-party services, like Mimecast.
Postfix – The open source MTA (Mail Transfer Agent) software included with Zimbra that accepts email from, and delivers email to, other mail servers and Zimbra mailboxes.
Amavisd – The open source software that is tightly integrated with Postfix, and which processes emails through SpamAssassin and ClamAV before allowing Postfix to deliver them.
SpamAssassin – A powerful open source software that ships with Zimbra and which performs hundreds of tests on an email. Each test result adds a positive (bad) or negative (good) score to an email. At the end of the exercise, SpamAssassin adds up all of the tests’ individual scores to get a combined score for the email. Scores above a certain threshold result in the email being classified as Spam. The thresholds at which SpamAssassin marks an email as Junk and delivers it to the end user, or deletes it outright are adjustable.
ClamAV – The open source anti-virus engine that ships with Zimbra and which is preconfigured to update itself.
Our anti-spam techniques comprise deploying sensible configuration adjustments to the the open source softwares that ship with Zimbra, supplemented with two paid third-party blacklists. Budget US$1.00 to US$4.00 per mailbox per year for third-party black lists to use within Zimbra, and you will be all set. If you need more enhanced email security services like data loss prevention, end-to-end encryption, user-managed quarantines, compliance assurance and email continuity so users can send and receive emails even when Zimbra is down, you are a candidate for a Mimecast subscription.
The rest of the article presumes you have already signed up for subscriptions from invaluement and uribl (or from alternative providers you deem better suited to your environment) and want to deploy their services in Zimbra.
Technique #1 – Outright Blocking With Postscreen
Zimbra’s Postfix now ships with Postscreen, a kind of inbound email firewall. Postscreen effectively blocks bad guys without putting any strain on the the Zimbra server. The Zimbra Postscreen wiki describes this tool, and how to configure it, in great detail. But the wiki recommends some very aggressive blacklists for the zimbraMtaPostscreenDnsblSites variable, and our experience has been that using the blacklists listed in that wiki cause a lot of false positives. As a result, we only use the free list provided by Barracuda Central (thank you Barracuda!) and one of our paid provider’s lists. To simplify, we ran as the Zimbra user:
zmprov mcf zimbraMtaPostscreenDnsblSites 'b.barracudacentral.org=127.0.0.2*7' zimbraMtaPostscreenDnsblSites 'sip-sip24.<your-account-code>.invaluement.com=127.0.0.2*6' zmprov mcf zimbraMtaPostscreenDnsblAction enforce zmprov mcf zimbraMtaPostscreenGreetAction enforce zmprov mcf zimbraMtaPostscreenNonSmtpCommandAction drop zmprov mcf zimbraMtaPostscreenPipeliningAction enforce
zimbra@zimbra:~$ zmprov gcf zimbraMtaPostscreenDnsblSites
zimbraMtaPostscreenDnsblSites: b.barracudacentral.org=127.0.0.2*7 zimbraMtaPostscreenDnsblSites: sip-sip24.<our_account_number>.invaluement.com=127.0.0.2*6
zimbra@zimbra:~$
Postscreen is really resource-efficient. It doesn’t require a lot of RAM, disk nor CPU cycles to implement. But this is where we recommend using only blacklists that have a very, very low known rate of “false positives”. And this screen is for the IP address of the sending server onlyThe Third Strategy – Front Door Blocking Via DNS Checks
Within a few minutes, Zimbra’s zmconfigd configuration daemon will pick up the above changes and reload Postfix appropriately.
Technique #2 – Outright Blocking With Postfix DNS Protocol Checks
Outside email servers trying to send you email should behave appropriately and be configured according to published and accepted standards. Most are not, and often for acceptable reasons. Zimbra’s Postfix allows you to check a number of these via settings in the Admin Console (Home > Configure > Global Settings > MTA), but all but two of these checks can generate frequent false positives. The two checks that don’t generate false positives, but which will block a lot of spam are:
zmprov mcf +zimbraMtaRestriction reject_non_fqdn_sender zmprov mcf +zimbraMtaRestriction reject_unknown_sender_domain
The “reject_unknown_sender_domain” according to the Postfix documentation, performs some specific checks to ensure the sender’s domain actually exists in public DNS. If the sender’s domain doesn’t exist, the email is not accepted. Similarly, the “reject_non_fqdn_sender” check will reject the email if the MAIL FROM isn’t a properly-formed email address. “johnny@hosting” would get rejected by the non-fqdn check and “johnny@hosting666.com” would get rejected by the unknown sender domain test because (at least as of this writing) the domain “hosting666.com” doesn’t exist.
Technique #3 – Outright Blocking of Bad Sending Servers
Now we start using blocklists within Postfix, and their are two kinds: “RBLs” or left-hand-side blocklists, and RHSBLs or right-hand-side blocklists. RBLs list IP addresses; RHSBLs list domains. Different blocklist providers have different philosophies about how IP addresses and domains get listed. Some are forthcoming and transparent; others less so. Some are clearly very aggressive in their listing methodologies and fully acknowledge they will cause some “false positives” periodically. Some have clear rules for delisting; others do not. So before you choose any blocklists to use, be sure you make sure the culture of the blacklist aligns with the culture of your company’s sensitivity to false positives. Here’s what we do (please be sure to substitute your own invalument account code first!):
zmprov mcf +zimbraMtaRestriction "reject_rbl_client b.barracudacentral.org" zmprov mcf +zimbraMtaRestriction "reject_rbl_client sip-sip24.<account_code>.invaluement.com" zmprov mcf +zimbraMtaRestriction "reject_rbl_client psbl.surriel.com" zmprov mcf +zimbraMtaRestriction "reject_rbl_client cbl.abuseat.org"
Included between the three free blocklists above, we use one paid blocklist from invaluement. We spent more than nine months testing a lot of RBL providers, and invaluement came to the top based on their ability to catch spam sources really early on (before many of the majors). They have a very low false positive rate and their pricing is very fair. Lastly, they also provide URIBL checking too (but we’ll get to that in a little bit…)
Those four RBLs together we have found do a fantastic job of blocking spam from known bad IP addresses.
Technique #4 – Outright Blocking of Bad Sending Domains
So far we’ve used Postfix’s Postscreen and Postfix itself, with some RBL add-ins, to block spam from known bad IP addresses. The next step is to use Postfix to block spam from known bad domains. For, that we use RHSBLs from one of our paid blocklist providers (again, be sure to use your account code after you subscribe):
zmprov mcf +zimbraMtaRestriction "reject_rhsbl_client <account_code>.df.uribl.com" zmprov mcf +zimbraMtaRestriction "reject_rhsbl_reverse_client <account_code>.df.uribl.com" zmprov mcf +zimbraMtaRestriction "reject_rhsbl_sender <account_code>.df.uribl.com"
Technique #5 – Outright Blocking of Certain Attachments
By default, Zimbra does not block what could be dangerous email attachments, like Visual Basic scripts, Windows Registry edits etc. So, we need to block those attachments and let the System Administrator and the recipient know we have done so. Zimbra by default also blocks encrypted attachments, but it is more common these days that many users email password-protected files to each other, so in our systems we allow this. To effect these changes we run:
zmprov mcf +zimbraMtaBlockedExtension asd zmprov mcf +zimbraMtaBlockedExtension bat zmprov mcf +zimbraMtaBlockedExtension cab zmprov mcf +zimbraMtaBlockedExtension chm zmprov mcf +zimbraMtaBlockedExtension cmd zmprov mcf +zimbraMtaBlockedExtension com zmprov mcf +zimbraMtaBlockedExtension dll zmprov mcf +zimbraMtaBlockedExtension do zmprov mcf +zimbraMtaBlockedExtension exe zmprov mcf +zimbraMtaBlockedExtension hlp zmprov mcf +zimbraMtaBlockedExtension hta zmprov mcf +zimbraMtaBlockedExtension js zmprov mcf +zimbraMtaBlockedExtension jse zmprov mcf +zimbraMtaBlockedExtension lnk zmprov mcf +zimbraMtaBlockedExtension ocx zmprov mcf +zimbraMtaBlockedExtension pif zmprov mcf +zimbraMtaBlockedExtension reg zmprov mcf +zimbraMtaBlockedExtension scr zmprov mcf +zimbraMtaBlockedExtension shb zmprov mcf +zimbraMtaBlockedExtension shm zmprov mcf +zimbraMtaBlockedExtension shs zmprov mcf +zimbraMtaBlockedExtension vbe zmprov mcf +zimbraMtaBlockedExtension vbs zmprov mcf +zimbraMtaBlockedExtension vbx zmprov mcf +zimbraMtaBlockedExtension vxd zmprov mcf +zimbraMtaBlockedExtension wsf zmprov mcf +zimbraMtaBlockedExtension wsh zmprov mcf +zimbraMtaBlockedExtension xl zmprov mcf +zimbraMtaBlockedExtensionWarnAdmin TRUE zmprov mcf +zimbraMtaBlockedExtensionWarnRecipient TRUE zmprov mcf zimbraVirusBlockEncryptedArchive FALSE
Technique #6 – Email Content Checking
So far Postfix/Postscreen have done all of the work via low-cost DNS lookups, and all of the checks have been binary: Is the sender on a blacklist? Yes? Drop the connection or don’t accept the email. For emails which have not yet been rejected, Postfix will now accept the email and pass it off to Amavisd for processing.
Zimbra ships Amavisd preconfigured to process emails through ClamAV and SpamAssassin. Amavisd can be configured to process emails through additional third-party services (configuring that is beyond the scope of this article). Once processed, the emails are (a) delivered to the recipient (either because the email is clean, or because the email has been marked as spammy and put in the user’s Junk folder), (b) placed in a quarantine or (c) just discarded if the email’s spam score is off the charts. We control the “discard/spam/not spam” thresholds and the Subject prepend for spammy emails like so:
zmprov mcf zimbraSpamKillPercent 75 zmprov mcf zimbraSpamTagPercent 20 zmprov mcf zimbraSpamSubjectTag "** CAUTION! SUSPICIOUS EMAIL **"
Zimbra’s log file “/var/log/zimbra.log” contains a lot of good information about which emails are accepted/rejected, but no detail on the components of an email’s spam score. To get that detail (so we can tune things later if needed, and explain to a disgruntled user why an email was marked as spam) we need to bump up Amavis’s logging level from the default 1 to 2, like so:
zmprov mcf zimbraAmavisLogLevel 2
Not all of Amavis’s configurations are adjustable via Zimbra Global Config variables. Zimbra ships Amavis’s config template file as “/opt/zimbra/conf/amavisd.conf.in”. Every time Zimbra (or just Amavis) is restarted, Zimbra’s zmconfigd configuration daemon reads this template file and rewrites the actual amavisd.conf file right before starting the amavisd service. We need to make a few minor changes…
First, SpamAssassin skips over emails with large bodies, on the premise that spammers won’t waste time sending big emails. Wrong. Spammers just pad emails with a bunch of invisible junk to avoid getting their emails’ contents checked. So we need to empower SpamAssassin to check those larger emails.
The second thing we need to do to amavisd.conf.in is whitelist Zimbra’s Daily Mail Report, else the lists of bad domains that got blocked will cause the Daily Mail Report itself to be marked as Spam.
To effect both of those changes, as the zimbra user edit “/opt/zimbra/conf/amavisd.conf.in” in two places:
# ORIGINAL LINE BELOW - CHANGES TO THIS FILE WILL NOT SURVIVE ZIMBRA UPGRADES # $sa_mail_body_size_limit = 512*1024; # don't waste time on SA if mail is larger $sa_mail_body_size_limit = 1024*1024; # don't waste time on SA if mail is larger
Lower down in the file, look for the following lines and insert a prescore for the virus notifications sender email you use (example in green):
'returns.groups.yahoo.com' => -3.0,
'clusternews@linuxnetworx.com' => -3.0,
lc('lvs-users-admin@LinuxVirtualServer.org') => -3.0,
lc('owner-textbreakingnews@CNNIMAIL12.CNN.COM') => -5.0,
# soft-blacklisting (positive score)
'sender@example.net' => 3.0,
'.example.net' => 1.0,
Now that SpamAssassin is going to be asked to process a lot more, and bigger emails, Amavis needs a RAM disk for SpamAssassin to do that work. RAM disk provisioning varies from distribution to distribution, but on Ubuntu 16.04 with Zimbra 8.8.x and where the zimbra user’s UID and GID are both 999, you can add the following line to /etc/fstab to create a 1GB RAM disk:
tmpfs /opt/zimbra/data/amavisd/tmp tmpfs defaults,noexec,nodev,nosuid,size=1024m,mode=750,uid=999,gid=999 0 0
For most Zimbra MTAs, a 1GB RAM disk will be large enough. But if you have more than the standard amavis processes running, or you allow large file size attachments, it’s possible you may need a larger RAM disk. Ralf Hildebrandt, the author of the seminal Book of Postfix, has a blog that contains the RAM disk sizing formula you can use.
Note that Amavis cannot be running if you want to implement this without a reboot. To do so on a live Zimbra system you would do:
su - zimbra zmamavisdctl stop ls /opt/zimbra/data/amavisd/tmp exit mount /opt/zimbra/data/amavisd/tmp df -h su - zimbra zmamavisdctl start postfix flush ls /opt/zimbra/data/amavisd/tmp
The SpamAssassin project periodically updates the hundreds of rules SpamAssassin uses, but Zimbra by default does not update SpamAssassin’s rules, so we need to tell Zimbra to update SpamAssassin periodically and restart SpamAssassin after an update:
zmlocalconfig -e antispam_enable_rule_updates=true zmlocalconfig -e antispam_enable_restarts=true
ClamAV takes care of updating itself by default; Zimbra runs ClamAV’s companion “freshclam” service periodically to update ClamAV’s virus definitions. Unfortunately, the default update frequency is too infrequent for us, so to update the virus definitions every two hours we run:
zmprov mcf zimbraVirusDefinitionsUpdateFrequency 2h
Now that Amavis, SpamAssasin and ClamAV are ready to zoom, we need to tweak SpamAssassin to use our paid blacklist providers, and to prevent some common false positives.
Both invaluement and uribl.com provide customers with SpamAssassin configuration files that should be placed in /opt/zimbra/data/spamassassin/localrules. These files cause SpamAssasin to perform the additional/changed email content scanning tests supported by the providers. In addition, non-Zimbra files in this directory are left intact by Zimbra’s installer, so Zimbra Patches and indeed Zimbra version upgrades will not “break” your SpamAssassin customizations.
We do need to do one more thing to customize SpamAssassin… If you have not already done so, as the zimbra user run:
nano ~/data/spamassassin/localrules/sauser.cf
Add the following lines to the above sauser.cf file:
score DOS_OUTLOOK_TO_MX 0 score TO_EQ_FM_DIRECT_MX 0 score RCVD_IN_PBL 0.1 score RDNS_NONE 0.1 score FREEMAIL_FORGED_REPLYTO 4.0 score RCVD_IN_RP_RNBL 4.0 score FROM_FMBLA_NEWDOM 2.5 score FROM_NEWDOM_BTC 3.0 use_bayes 0 internal_networks 10.7.57.17/32 internal_networks 10.7.57.129/32 internal_networks 10.7.57.130/32 internal_networks 10.7.57.131/32 internal_networks 10.7.57.29/32
The first three lines containing “_MX” and “_PBL” SpamAssassin tests have default scores that are really high. These tests will fire when a user with Zimbra Connector for Outlook sends emails, so we need to adjust the scores of those tests down to zero. Otherwise, emails sent from Outlook using the Outlook Connector will often be flagged as spam. The PBL test however is used by other tests, so (see below) we set this to a small value.
The next test, for Reverse DNS, also comes with a default high score, and a quick Google search will yield a lot postings complaining about the high false positive rate from this test. We can’t set this RDNS_NONE score to zero, because several other useful SpamAssassin tests depend on RDNS_NONE. But, we can set the score for this test to be really small, so it won’t result in any false positives.
The forged freemail test we give a higher score to than the default SpamAssassin score, because we see that spammers will send junk from a “real” email server, but forge the reply-to to point to a gmail, hotmail etc. mailbox. This is so that the spammer can receive replies to an email account that was not the source of the spam.
The RCVD_IN_RP_RNBL test queries a blacklist maintained by ReturnPath, who’s business centers around senders’ mostly positive reputations. You have to work very hard to get on their blacklist, and as such it has a very low false positive rate, so we boost the default score that SpamAssassin otherwise provides. More info at https://senderscore.org/.
The last two “NEWDOM” tests add points for emails being sent from brand new domains. Spammers often register new domains and send out spam until the right-hand-side blacklists start blacklisting them. These two tests cover that gap, but may cause false positives for legitimate email being sent from domains which have been registered in the past few days. But, considering it usually takes a day or two for legitimate users to set up their email after they register their domain, we consider the risk of false positives from these tests to be extremely low.
SpamAssassin also comes with a Bayes filtering engine. Bayes works great for some but not for others. In my case, I found a lot of spam-like emails would NOT get flagged as spam because the Bayes “ham” tests would add a few negative points to the email’s overall score, keeping the problematic email from being flagged as spam when it really should be. This “use_bayes 0” line turns off Bayes filtering. Another reason I try to avoid using Bayes is because doing so means the Bayes database needs to be backed up as part of a good Disaster Recovery plan. Neither the Zimbra “Classic” Network Edition backup engine, nor the ZeXtras provided Backup/Backup NG engine, back up the Bayes database themselves. Your Zimbra System Administrator would need to do that manually using the sa-learn command.
The last few lines are the IP addresses of the servers in a multi-server Zimbra hosting farm, and not just the MTA servers. The documentation on this doesn’t IMHO quite match up with reality, and the “internal_networks” parameter isn’t a white list, but in my experience you’ll have less problems if you list all of your Zimbra servers here.
Recap
So what have we accomplished? We:
Configured Postfix’s Postscreen and Postfix itself to outright block a huge percentage of junk email in a way that doesn’t tax the Zimbra server, because we do so using nothing more than a raft of DNS lookups.
Blocked known dangerous attachments and configured Zimbra to let the recipient and System Administrator know that we have done so.
Adjusted the spam scoring thresholds to values that will block more spam without increasing false positives.
Enabled Amavis to give us more actionable information by increasing its log level.
Improved performance by deploying a RAM disk.
Improved accuracy by:
Enabling SpamAssassin to update itself.
Enabling ClamAV to update its virus definitions more frequently.
Customizing SpamAssassin’s scores to eliminate false positives specifically for Zimbra Connector for Outlook users as well as for all users in general.
Customizing SpamAssassin’s scores to improve spam detection overall.
Spent somewhere between $1 and $4 per mailbox per year to get the benefit of two very good commercial blacklist providers.
Made our users very happy and reduced frustration for the email administrators!
Zimbra also allows for and supports additional customizations to block spam that the above techniques on their own may not catch, but which may be specific to your environment. One example of this is nested email addresses intended to spoof legitimate email addresses. We wrote about Zimbra’s solution in a blog post late last year. Hopefully this solution makes it into the shipping product in the near term.
Hope that helps,
L. Mark Stone
Mission Critical Email
21 March 2019
Updated 19 August 2019
Updated 25 September 2019
P.S. Thanks to Lorenzo Milesi at YetOpen S.r.l. and Sebastián Greco for their helpful suggestions.
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone personally. The contents of this site are not intended as advice for any purpose and are subject to change without notice. Mission Critical Email makes no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.
Requires Login