Post Office Protocol and Internet Mail Access Protocol (POP and IMAP)
In Chapter 20, we covered Simple Mail Transfer Protocol (SMTP), the underlying e-mail transport mechanism or protocol by which e-mail is sent from e-mail clients to the server and from one mail server to another mail server. We also mentioned how mail delivery agents (MDAs) facilitate the processing of incoming mail (such as sorting or filtering mail according to sender, subject line, length of message, keywords, and so on). All of this processing is usually done after the mail has been transported to the final destination mail server. For the MDA functionality, a program like Procmail can be used. Procmail can make copies of user e-mails available to users in the mbox format. The mbox format is a simple text format that can be read by a number of console mail user agents (MUAs) such as pine, elm, mailx, and mutt, as well as some GUI-based mail clients such as Thunderbird.
To make the mbox format usable, the e-mail client (MUA) needs to have direct access (at the file system level) to the mbox file itself. This works well enough in tightly administered environments where the administrator of the mail server is also the administrator of the client hosts; however, this system of mail folder administration might not scale well in certain scenarios. The following sample scenarios might prove to be a bit thorny:
• Users are unable to stay reasonably connected to a fast/secure network for file system access to their mbox file (for example, roaming laptops).
• Users need local copies of e-mail for offline viewing.
• Security requirements dictate that users not have direct access to the mail store; for example, Network File System (NFS) shared mail spool directories are considered unacceptable.
• MUA does not support the mbox format (typical of MS Windows-based e-mail clients).
To handle these thorny cases and others where Procmail and other traditional MDAs will simply not suffice, another class of protocols was created. We’ll collectively describe this class of protocols as mail access protocols. This chapter covers two popular mail access protocols: Post Office Protocol (POP) and Internet Message Access Protocol (IMAP).
POP was created to allow for network-based access to mail stores. Many early Windows-based mail clients used POP for access to Internet e-mail, because it allowed users to access UNIX-based mail servers (the dominant type of mail server on the Internet until the rise of Microsoft Exchange in the late 1990s).
The idea behind POP is simple: A central mail server remains online at all times and can receive and store mail for all of its users. Mail that is received is queued on the server until a user connects via POP and downloads the queued mail. The mail on the server itself can be stored in any format (such as mbox) so long as it adheres to the POP protocol.
When a user wants to send an e-mail, the e-mail client relays it through the central mail server via SMTP. This allows the client the freedom to disconnect from the network after passing on its e-mail message to the server. The task/responsibility of forwarding the message, taking care of retransmissions, handling delays, and so on, is then left to the well-connected mail server. Figure 21-1 shows this relationship.
Figure 21-1 Sending and receiving mail with SMTP and POP
Certain aspects of the POP protocol are too limiting. Features such as being able to keep a master copy of a user’s e-mail on the server with only a cached copy on the client were missing. This led to the development of IMAP.
The earliest RFC (request for comments) documenting the inner workings of IMAPv2 is RFC 1064, dated 1988. After IMAPv2 came IMAP version 4 (IMAPv4) in 1994. Most e-mail clients are compatible with IMAPv4. Some design deficiencies inherent in IMAPv4 led to another update in the protocol specifications, and, thus, IMAPv4 is currently at its first revision—IMAP4rev1 (RFC 3501).
The evolution of IMAP can best be understood by thinking of mail access as working in one of three distinct modes: online, offline, and disconnected. The online mode is akin to having direct file system access to the mail store (for example, having read access to the /var/mail file system). The offline mode is how POP works, where the client is assumed to be disconnected from the network, except when explicitly pulling down its e-mail. In offline mode, the server normally does not retain a copy of the mail.
Disconnected mode works by allowing users to retain cached copies of their mail stores. When the client is connected, any incoming/outgoing e-mail is immediately recognized and synchronized; however, when the client is disconnected, changes made on the client are kept until reconnection, when synchronization occurs. Because the client retains only a cached copy, a user can move to a completely different client and re-synchronize his or her e-mail.
By using IMAP, your mail server will support all three modes of access. After all is said and done, deploying and supporting both POP and IMAP is usually a good idea. It allows users the freedom to choose whatever mail client, protocol, and workflow that best suits them.
There are several Free and Open Source Software (FOSS) mail servers that implement POP and IMAP. Some of them are Dovecot, University of Washington IMAP server (UW IMAP), Cyrus IMAP server, and Courier IMAP server. This chapter covers the installation and configuration of the popular Dovecot server software. This particular mail server has been available for many years. The installation process is also easy.
POP3 and IMAP Protocol Basics
Like the other services discussed so far, POP3 and IMAP each need a server process to handle requests. The POP3, POP3S, IMAP, and IMAPS server processes listen on ports 110, 995, 143, and 993, respectively.
Each request to and response from the server is in clear-text ASCII, which means it’s easy for us to test the functionality of the server using Telnet. This is especially useful for quickly debugging mail server connectivity/availability issues. Like with an SMTP server, you can interact with a POP3 or IMAP server using a short list of commands. Although there are many POP commands, a couple worth mentioning are
And a few noteworthy IMAP commands are
Later in this chapter, we’ll use some of these commands and walk through the connection and login process on a POP3 server and an IMAP server. This will allow us to verify that the server does in fact work.
Dovecot (IMAP and POP3 Server)
The Dovecot POP3 and IMAP server software is well regarded and is used in many production sites around the world. It is a well-tested implementation and readily available on most of the mainstream Linux distros.
Dovecot provides its IMAP and POP3 functions through the following main processes. Collectively, these processes provide various services that make up the Dovecot ecosystem.
• Master process As its name implies, the master process is the primary/overseer process. It is responsible for starting and keeping all the other processes running as needed. It reads the settings/options in the configuration files and exports the values to the other processes. The master process is responsible for collecting and managing all logging information that Dovecot generates. The master process runs under the
• Login processes The login processes listen for connection requests for the POP3 and IMAP protocols and implement the minimum handshaking protocol requirement before a user logs in successfully. The login processes run under the imap-login and pop3-login executables.
• Authentication (auth) process Once the login processes complete the underlying POP3 or IMAP protocol handshaking and setup, control is passed on to the appropriate authentication process. The auth process is responsible for performing the actual user authentication (Simple Authentication and Security Layer [SASL] functions) to verify that the user is who she says she is. The authentication processes run under a similarly named auth executable.
• Mail processes (IMAP, POP3) After authentication is successfully completed, the desired mail process kicks in and provides the user access to her mailboxes. The mail process is the actual workhorse that implements the POP3 and IMAP protocol details. The IMAP process runs under an executable called imap, and the POP3 process runs under an executable named pop3.
Most Linux distributions have prepackaged binaries for Dovecot in the distros’ repositories. For example, Dovecot can be installed in Fedora/CentOS/RHEL by using
dnf like so:
On Debian-like systems, such as Ubuntu, Dovecot’s IMAP and POP3 functionality is provided in two separate packages: dovecot-imapd and dovecot-pop3d, respectively. They can be installed by using Advanced Packaging Tool (APT) like so:
During the package installation on Debian-based distros, you will be prompted to create self-signed certificates for using IMAP and POP3 over SSL/TLS. Select Yes when prompted.
You will also be prompted for the hostname to use for the commonName field of the self-signed certificate. Input the correct hostname for your system and select Ok to continue.
Dovecot Configuration Files and Options
Dovecot provides you, the system administrator, with a wide range of configuration options to help you meet the mailing needs of your end users. The software is helpfully modularized, and you can easily pick and choose which modules you want to customize.
The primary configuration file for Dovecot is dovecot.conf, which most mainstream Linux distros store under /etc/dovecot/. Using an
include directive (
!include conf.d/*.conf) in dovecot.conf, some Linux distros go further and organize (split) various files that control different aspects of Dovecot under the /etc/dovecot/conf.d/ subdirectory.
Table 21-1 shows the location and descriptions of some Dovecot configuration files.
Table 21-1 Dovecot Configuration Files
The configuration files accept a rich set of options that can be used to control and tune various aspects of Dovecot as well as turn features on and off. Table 21-2 describes only a small subset of the more oft-used configuration options.
Table 21-2 Dovecot Configuration Options
After installation (from source or binary), the next step is to configure or customize your Dovecot instance to suit your environment. The software ships with many sane default settings that can be used as is out of the box with very little customization from you.
To get Dovecot up and running quickly, you may have to tweak some configuration parameters at a minimum. The parameters that we will be changing are shown in Table 21-3, along with the desired target values. Table 21-2 also describes the parameters.
Table 21-3 Target Dovecot Configuration Settings
doveconf utility to make sure that your server supports LMTP, POP3, and IMAP:
If any of the protocols are missing, open the main /etc/dovecot/dovecot.conf configuration file with any text editor, look for the
protocols setting, and update it to look like the line here:
You can also use the
sed utility to quickly find and replace the option you want in the configuration file, like so (all on one line!):
Configure the listen Parameter
Edit the /etc/dovecot/dovecot.conf file if necessary and make sure that the
listen entry exists and looks like the one here:
You can use the
sed utility to quickly edit the file in place by running the following:
Configure System Users and Password Databases (passdb and userdb)
doveconf to check that the PAM database is one of the drivers configured for authenticating system users. In the same command, also check for the driver being used for the user database:
driver entries for the
userdb sections of the output should look similar to the following:
driver entries in your output are different, open the /etc/dovecot/conf.d/auth-system.conf.ext configuration file and ensure that the
driver values are set to the corresponding values in Table 21-3.
Configure Mail Location
doveconf to check the current value of the
By default, the
mail_location parameter is unset. Use any text editor to edit and set the parameter so that the entry in /etc/dovecot/conf.d/10-mail.conf looks like the one here:
If for some reason you like pain and you derive pleasure from constructing and debugging regular expressions to make edits that are probably easier done by using a text editor, you can use the
sed utility with some convoluted-looking options (all on one line!) to “quickly” edit the file in place and make the changes that you want by running the following:
Configure Mail Access Group
doveconf to check the current value of the
By default, the
mail_access_groups parameter is unset. Use any text editor to edit and set the parameter so that the entry in /etc/dovecot/conf.d/10-mail.conf looks like the one here:
You can use the
sed utility to quickly edit the file in place by running the following:
Configure Authentication Mechanisms
Use doveconf to check the current value of the
Edit the /etc/dovecot/conf.d/10-auth.conf file if necessary and make sure that the
auth_mechanisms entry exists and looks like the one here:
Configure Dovecot SSL Parameters
doveconf to check the current values of the
Edit the /etc/dovecot/conf.d/10-ssl.conf file if necessary and make sure that the following parameters are set to the values here:
TIP Dovecot IMAP and POP3 server software installed via the package management system on most distros ships with generic SSL certificates and key files that are used in the 10-ssl.conf configuration file.
Dovecot ships with a simple script named mkcert.sh that can be used to generate custom certificates and keys that can be used with your Dovecot instance. To use the script, first customize the /etc/pki/dovecot/dovecot-openssl.cnf file with your custom settings and then execute the script. To run the script on a Red Hat–like distro such as Fedora, type the following:
CAUTION While trying to connect to your Dovecot server, users will receive a warning that the certificate is not properly signed if you create and use self-signed certificates. If you do not want this warning to appear, you can obtain a certificate (for free) from a certificate authority (CA) such as the Let’s Encrypt Project (https://letsencrypt.org/) or purchase one from Comodo, Symantec/Thawte, Symantec/VeriSign, and so on. Depending on your specific environment, this might or might not be a requirement. However, if all you need is an encrypted tunnel through which passwords can be sent, a self-signed certificate works fine.
After configuration, the next step is to learn how to control the Dovecot services. This includes how to start, restart, stop, and enable Dovecot. The following instructions apply to a Dovecot instance installed via the distro’s package management system. To control or manage the Dovecot instance compiled and installed from source, you will have to tweak the steps a bit and specify the correct paths to the commands/binaries.
On Linux distros such as modern versions of Fedora, CentOS, Ubuntu, and RHEL that use systemd as the service manager, check the status of the Dovecot service by running the following:
To stop Dovecot (assuming it’s currently running), type this:
To start Dovecot, type this:
To disable Dovecot from automatic startup and stopping it (if running), type this:
To configure the Dovecot IMAP and POP3 services to automatically start up during system boot and simultaneously start the services, type the following:
TIP Dovecot software suite comes with a built-in administration tool named doveadm. This tool is distribution agnostic, so you can expect it to work in the same way regardless of how your Dovecot instance was installed. doveadm is a powerful tool and can be used to control and manage many aspects of Dovecot, such as reloading, stopping, logging, testing, and so on.
Checking Basic POP3 Functionality
If everything has worked correctly, you should now have a running IMAP server and POP3 server. The next logical step is to test the services for actual functionality.
We begin by using Telnet to connect to the POP3 server (localhost in this example). From a command prompt, type the following:
The server is now waiting for you to give it a command. (Don’t worry that you don’t see a prompt.) Start by submitting your login name as follows:
yourlogin is, of course, your login ID. The server might respond with something like this:
Now tell the server your password using the
yourpassword is your password. The server might respond with this:
You’re now logged in and can issue commands (such as
RETR) to read and manage your mail. Since you are simply validating that the server is working, you can log out now. Simply type
QUIT, and the server will close the connection:
Checking Basic IMAP Functionality
We will use Telnet to connect to the IMAP server (localhost in this example) and test it for basic IMAP functionality. From the command prompt, type the following:
The IMAP server will respond with something similar to this:
The server is now ready for you to enter commands. Note that like the POP3 server, the IMAP server will not issue a prompt.
The format for IMAP commands is shown here:
<tag> represents any unique (user-generated) value used to identify (tag) the command. Example tags are
3, and so on. Commands can be executed asynchronously, meaning that it is possible for you to enter one command and, while waiting for the response, enter another command. Because each command is tagged, the output will clearly reflect what output corresponds to what request.
To log into the IMAP server, simply enter the
login command, like so:
<username> is the username you want to test and
<password> is the user’s password. If the authentication is a success, the server will respond with something like this:
That is enough to tell you two things:
• The username and password are valid.
• The mail server was able to locate and access the user’s mailbox.
With the server validated, you can choose from and issue a multitude of IMAP commands to manage your mailbox or logoff by simply typing the
The server will reply with something similar to this:
Other Issues with Mail Services
Thus far, we’ve covered enough material to get you started with a working mail server, but there is still a lot of room for improvements. In this section, we walk through some of the issues you might encounter and some common techniques to address them.
Best security practices should be a big objective in any mail server (such as POP3 and IMAP) implementation. Some mail server implementations do not ship with secure options enabled out of the box (possibly to help make initial configuration easier). Some implementations may offer varying levels of support for encryption, password-hashing schemes, user/password databases, and so on. There is also the issue of ensuring that the majority of e-mail clients that will use the mail server are properly supported. Regardless of the e-mail server software stack you settle on, you should ensure that, at a minimum, you enable encryption of the entire protocol stream whenever possible.
Fortunately for us, the Dovecot IMAP and POP3 server implementation that we installed earlier was written from the ground up with security in mind, and it also ships with sane and secure default configuration options. This is one of the reasons why we did not need to do too much in the way of configuring SSL support for our Dovecot instance, thus keeping things simple! We made sure that the SSL support is enabled and we accepted the default certificates and keys. Besides keeping things simple, our approach hopefully made for a nice confidence booster for you to be able to get something working quickly—before we start tinkering too much and adding other layers of complexity.
The Telnet protocol that we used for testing POP3 and IMAP functionality earlier is not a secure protocol by default. Rather, by default everything done over Telnet is transmitted in plain text. So we were connecting and testing the POP3 and IMAP server over an unencrypted channel. You may be wondering, then, why we said that Dovecot is secure out of the box. You may also be wondering what’s the point of enabling SSL when the system allows us to successfully connect insecurely.
Well, we are glad you caught us and called us out before we continue venturing down this possible path of lies and vignette of deceit. Kindly allow us to explain:
• We assumed that all our testing with Telnet was being done from and to the same computer (localhost) running the Dovecot server software. The Telnet testing would have failed if we had tried to do it from a different system.
• By default in Dovecot, plain-text (nonsecure) authentication is always permitted for connections originating from the localhost. This means you can connect to Dovecot without using SSL, or even configuring it, whenever you connect from localhost.
Now that we have (hopefully) earned your trust again, we’ll go ahead and make sure our POP3 and IMAP server is truly secure by testing it from a different computer. We will use the Swiss Army program of all things related to SSL—the OpenSSL program suite—to do our testing in the following section.
TIP The operating system’s logging subsystem is an indispensable tool when troubleshooting mail server issues, so make sure you keep an eye on the log files when you are troubleshooting issues such as the dovecot service failing to start or restart properly (for example, via “
journalctl -xe -u dovecot”).
Testing POP3 and IMAP Connectivity over SSL/TLS
While still logged onto the Dovecot server, first make sure that the TCP ports for POP3 and IMAP are open on the firewall for external connections.
On a Red Hat–like distro such as our Fedora server, we can use the
firewall-cmd command to do this. We’ll permanently add a rule to the current firewall ruleset to achieve this by running the following command:
Next, reload the firewall rules so that the new rule is immediately active:
Now let’s hop over to a different box and remotely test our Dovecot POP3 and IMAP services. Let’s assume that our remote Dovecot server’s IP address is 192.168.56.101.
1. Make sure our previous Telnet test will fail if we try to do it from a remote system:
2. The initial connection to the POP3 port, 110, succeeded. Let’s issue the first POP3 command to begin the SASL process. At the Dovecot prompt, start by submitting your login name as follows:
Here, remoteloginname is the username of a user on the remote Dovecot server with a POP3 mailbox.
We are immediately stopped from going any further by the response we get from the remote Dovecot server:
So, we’ve further supported our claim that nonsecure connections are not supported by default on a Dovecot server. Issue the
QUIT command to end the current POP3 session.
3. Now let’s try to connect to the remote server using
openssl to connect to the same remote POP3 server using
If the server is listening securely on port 110, you should be greeted with a bunch of SSL transaction-related messages and a prompt similar to this one:
4. As before, the initial connection to the POP3 port 110 succeeded. Let’s issue the first POP3 command to begin the SASL process. At the Dovecot prompt, submit a login name as follows:
The first sign of progress is that the POP3 server allowed us to issue a
USER command and submit the username over the
Issue the POP3
PASS command to submit the password:
Here, password_for_remoteloginname is the password associated with the remoteloginname username.
+OK Logged in.
The “+OK Logged in” output from the server shows that we logged in successfully! You can now continue to issue other POP3-related commands to interact with the remote mailbox. Issue
QUIT to exit.
5. You can similarly test the IMAP service on the remote Dovecot server using
openssl to connect to the remote IMAP server listening on port 143 using
6. You should be able to continue issuing supported IMAP protocol commands (such as
LOGOUT, and so on) to authenticate yourself and interact with the remote server.
TIP Remember that there isn’t too much point in implementing security if nobody is using it, so make sure that your mail clients use SSL when connecting to the IMAP or POP3 server. In most of the popular e-mail client programs, such as Thunderbird, Evolution, Outlook, and so on, the option to enable SSL may be as simple as a check box in the Email Account configuration options.
In managing a mail server, you will quickly find that e-mail qualifies as one of the most visible resources on your network. When the mail server goes down, everyone will know, they will know quickly, and, worst of all, they may even alert you, the administrator, before you even realize that something is amiss! Thus, it is important that you carefully consider how you will be able to provide 24/7 availability for e-mail services.
A simple issue that can threaten mail servers is “fat fingering” a configuration—in other words, making an error when performing basic administration tasks. There is no solution to this problem other than being careful! When you’re dealing with any kind of production server, it is prudent to perform each step carefully and make sure you type what you meant to type. When at all possible, work as a normal user rather than root and use sudo for specific commands that need root permissions.
The second big issue with managing mail servers is hardware availability. Unfortunately, this is best addressed with money. The more the better! Make an investment up front in a good server chassis. Adequate cooling and as much redundancy as you can afford is a good way to make sure the server doesn’t take a fall over something silly like a CPU fan going out. Employing dual power supplies is another way to help keep mechanical things from failing on you. Uninterruptible power supplies (UPSs) for your servers are almost always a must. Make sure that the server disks are configured in some kind of RAID fashion. This is all to help mitigate the risk of hardware failure.
Finally, consider expansion and growth early in your design. Your users will inevitably consume all of your available disk space. The last thing you will want is to start bouncing mail because the mail server has run out of disk space! To address this issue, consider using disk volumes that can be expanded on the fly and RAID systems that allow new disks to be added quickly. This will allow you to add disks to the volume with minimal downtime and without having to move to a completely new server.
Although we’ve mentioned this earlier in the chapter, watching the /var/log/messages, /var/log/syslog, /var/log/maillog, and /var/log/mail.log files is a prudent way to manage and track the activities on your mail server. The Dovecot software provides a rich array of logging options and log messages to help you understand what is happening with your server and troubleshoot any peculiar behavior. In short, when in doubt, take a moment to look through the log files. You’ll probably find a solution or pointer to your problem there.
This chapter covered some theory behind the IMAP and POP3 protocols, ran through the complete installation for the Dovecot software (from source and from prepackaged binaries), and discussed how to test connectivity to each service manually. With this chapter, you have enough information to set up and run a simple POP3 and IMAP server instance.
The chapter also covered enabling secure access to your mail server assets via SSL/TLS. This is an easy way to prevent clear-text passwords (embedded in IMAP or POP3 traffic) from making their way into hands that should not have them. We ended by touching on some basic human- and hardware-related concerns, necessities, and precautions in regard to ensuring that your mail server is available 24/7.
If you find yourself needing to build out a larger mail system, take the time to read/learn more about the mail server software of your choice (such as Dovecot, Cyrus, UW IMAP, or Courier). If you find that your environment requires more groupware functionality (such as provided with Microsoft Exchange Server), you might want to check out other software, such as Scalix, Open-Xchange, Zimbra, Horde Groupware, and EGroupware. They all provide significant extended capabilities at the expense of additional complexity in setup and configuration.
As with any server software that is visible to the outside world, you will want to keep up to date with the latest releases. Thankfully, the Dovecot package has shown sufficient stability and security so as to minimize the need for frequent updates, but a watchful eye is still nice. Finally, consider perusing the latest IMAP and POP RFCs to understand more about the protocols. The more familiar you are with the underlying protocols, the easier you’ll find troubleshooting to be.