Table of Contents
In this article, we will focus on an extremely common problem among system administrators and web developers: the fact that the e-mail messages coming from our domain are being marked as "spam" by the receiving mail servers. An issue that might affect not only the e-mail messages sent by our users but also the so-called "automated e-mails" sent from our systems (software, servers, services, etc) as a result of certain interactions: registration confirmation, shipping info for an online purchase, subscription to a newsletter, and so on.
After a brief introduction to explain how the e-mail messaging system works and what could likely cause this behavior, we will provide a complete guide on what you can do to solve (or minimize) this problem.
How E-mail messages work?
Before delving into the issues related to spam email classification mechanisms, it is advisable to take a step back to understand how the mechanism underlying the sending of our emails works.
Typically, if we own a domain and we want to send e-mail messages ending with that domain (such as something@ourdomain.com), we need to provide ourselves with a MTA (Mail-Transfer Agent) capable of accepting incoming e-mail messages through the SMTP protocol; this can be done by either using external (free or paid) servers or implementing a proprietary MTA.
The first scenario is usually the simplest one, as long as the chosen service offers an acceptable level of security, privacy, performance, and reliability - and we can sustain its costs; if that's not the case, we would need to fall back to a proprietary implementation using one of the many software available for Windows and/or Linux: some popular alternatives include SendMail/ProofPoint, Postfix, Exim, QMail , Mutt, Alpine, sSMTP, and so on. Configuring these tools can be a simple or complex matter, depending on the features we need, the number of e-mail accounts (addresses) we need to handle, the results given by the SMTP testing platform adopted, and the security level we want to achieve. If you chose to go with that route, be sure to check out the latest guides available on the internet or seek the aid of a professional.
Regardless of the solution chosen, the end goal is to provide yourself with an e-mail server that can be used to send and receive e-mail messages from dedicated clients (such as Mozilla Thunderbird or MS Outlook) and/or software applications created with ASP.NET, PHP, Java, Phyton, NodeJS and/or any other development frameworks (using programmatic techniques such as the one described in this dot net email post).
This is the starting point of our post: the presence of an MTA used by our users and/or software to send e-mail messages that, for some unknown reasons, sometimes end up in the Spam folder of the recipient(s). The next sections of this article are devoted to understanding the mechanisms that typically cause this behavior and how to deal with them.
Spam or not Spam?
In most cases, email messages are classified as spam by mail servers (and/or clients) following an automatic classification process, usually carried out by dedicated anti-spam software or module, which assigns a score to the message in question (Spam Score or Spam Rating) depending on a series of parameters and factors, including:
- Reliability of the sender (source e-mail, IP address, etc)
- Reliability of the sender's domain (i.e. the part after the @ character of the e-mail address)
- Words or phrases contained in the subject and/or in the message body
- Custom rules being set on the recipient's server and/or client (spam list, black list, etc)
If the "spam score" assigned to the message as a result of the analysis of these factors exceeds a certain threshold, the message is considered as "likely spam" and is therefore classified as such, moving it to a specific folder (Spam or Junk Mail) or marking it in various ways according to the rules set on the server and/or on the mail client.
Based on the above, we can therefore understand how, if the e-mails sent by our users (and/or services) end up in the recipient's "spam" box, the reason is most likely due to one of the above factors.
Measure the Spam Score with Mail Tester
To check the Spam Score of our messages, you can use Mail Tester, a free online tool that allows you to send an e-mail to a virtual address to subject it to a series of checks relating to the sender, the mail domain, the content of the message and a number of other factors: the results of the checks will be displayed in the form of a report, allowing the administrator to understand where the problem (or problems) is and how to fix.
In the next sections, we will analyze the two categories of problems that, statistically, most often determine a bad Spam Score: the problems related to the sender and those related to the e-mail domain, with the aim of understanding how they can be responsible for our bad "spam score", assuming that there are no problems in the content of the e-mail message and/or that the recipient has not blacklisted us.
The first thing to check is that there are no problems related to the e-mail address and/or IP address used to send the message that is marked as spam. More precisely, we should ensure that the e-mail address is valid, as well as not present in one of the many blacklists used by anti-spam software to determine the spam score of messages received: the same goes for the sender's IP address.
To verify the presence of these "blocks" we can use some specific software, such as SenderScore. It goes without saying that, if our e-mail address and/or IP address are compromised, it might be necessary to send a removal request to the service that takes care of maintaining the blacklist.
Unlike sender issues, which affect a single email address (e.g. mary@email.com), domain issues affect all email addresses related to that domain (i.e. all email addresses that end with @email.com).
In most cases, the problems related to the email domain used by the sender play a decisive role in the classification of messages as spam: it is in fact the simplest and most effective method to identify and block a series of abuses that can be carried out using the SMTP protocol (Simple Mail Transfer Protocol), the mechanism underlying the sending of e-mail messages, and which today constitute a good 80% of e-mails commonly classified as spam.
This type of classification is based on the coexistence of the following controls:
- Presence of the domain within the blacklists used by anti-spam software, verifiable in real-time using free tools such as E-Mail blacklist check.
- Violation of the SPF limitations set for the domain, or total absence of SPF records (which is interpreted by most anti-spam systems as neglect by the domain administrator, and therefore sanctioned at the level of spam score).
- Violation of the DKIM settings set for the domain, or total absence of DKIM records (also often interpreted as carelessness and therefore sanctioned at the level of spam score).
- Absence of the DMARC record, which documents the SPF and DKIM settings present for validating emails sent from the domain (if any). Also, in this case, the absence of the record is often interpreted as neglect by anti-spam systems and therefore sanctioned at the level of spam score.
Have you never heard of SPF, DKIM, and DMARC? Don't worry: in the following sections, we will explain in more detail the meaning of each of these acronyms and how to configure the related records on your web domain to improve the spam score of all emails sent.
SPF record
SPF stands for Sender Policy Framework, an email validation system designed to prevent spam due to email spoofing attempts, i.e. senders who pretend to be what they are not by sending messages from email addresses related to domains that they are not under their control; Specifically, these are attempts that exploit a known vulnerability in the SMTP protocol whereby it is possible to set an arbitrary sender, provided that the MTA used to send the message does not block the sending.
Since it is not possible to "force" the sending MTA to check the identity of the sender, all that remains is to act on the destination MTA: this is the purpose of the SPF record, through which the domain administrator can define the hosts authorized to send messages from that domain, allowing those who receive them to check their validity. This simple mechanism, if correctly implemented by the destination MTA, allows you to "cut out" all MTAs not explicitly authorized by the domain owner, classifying e-mail messages sent through those servers as spam.
In practical terms, the SPF record is nothing more than a TXT record that can be configured on your domain following some pre-determined conventions: within the TXT record, it will be possible to specify one or more hosts (i.e. the IP addresses or hostnames of the MTAs authorized to send e-mails) and set the rules that the destination MTA will have to follow to handle any e-mails from unauthorized domains.
To check for the presence of a suitable SPF record on your domain, as well as configure it correctly if it is not present, we recommend using this handy SPF configurator provided free of charge by MXToolbox, which allows you to scan your domain to identify any existing SPF records and also to set the necessary changes to change authorized hosts (and/or include new ones).
DKIM record
DKIM stands for Domain Keys Identified Mail (DKIM), an email authentication method that helps prevent spammers from impersonating a legitimate domain. Unlike SPF, which is a simple check on the host of the mail server responsible for sending the message, DKIM uses a cryptographic mechanism based on a private key (used by the sender to sign the message) and a public key (which can be used to verify the signature). It is therefore a system that, just like SPF, allows you to verify the identity of the sender, but based on a different criterion, that is, by checking the ointment key used instead of the source host.
In practical terms, the DKIM record is nothing more than a TXT record that contains the public key with which to verify the private key.
To verify the presence of a suitable DKIM record on your domain we recommend using the DKIM record lookup provided free of charge by MXToolbox, which also allows you to scan your domain to identify any existing DKIM records.
DMARC record
DMARC stands for Domain-based Message Authentication, Reporting & Conformance, and is the email message validation system to which SPF and DKIM belong. Developed starting from 2010 with the aim of combating email spoofing, it was formalized in 2015 by defining the shared criteria for the standardized use of SPF and DKIM by mail servers.
Basically, the DMARC check verifies that:
- The domain contained in the header's From field coincides with the DKIM signature domain (or a subdomain thereof);
- The domain contained in the header's From field coincides with the domain contained in the Mail From field and is one of the domains authorized through SPF.
An email passes the DMARC check when at least one of the above two checks is successful. Obviously, it is allowed to choose the behavior that the recipient should apply in case of check failure: specifically, it is possible to adopt a neutral behavior (neutral), consider the email as suspicious (quarantine), or reject it (reject). Finally, the recipient system can optionally send a report containing the source server, the destination domain, the result of the DKIM and SPF checks, and related alignments.
To generate your own DMARC record we recommend using the free DMARC generator provided by UnlockTheInbox.
Conclusions
We have come to the end of our overview of the Spam Score and the main systems used today to classify e-mail messages as spam: we hope that the information and tools we have provided will be useful for you to better configure your web domain in a way to guarantee a good score for your e-mails.
