Configuring Sendmail's STARTTLS (SSL) and Relaying


This page documents how to compile and configure Sendmail to support STARTTLS. STARTTLS is the SMTP command to "Start Transport Layer Security"; or, in other words, to turn on SSL. Using SSL with SMTP isn't terribly useful for protecting the contents of messages. Email generally goes through multiple hops between the sender and the recipient and the sender has no way to ensure they every one of those hops will use SSL. However, it it very useful when authenticating senders. You can either use SSL to protect a plain text login (SMTP AUTH), or use SSL certificates to do authentication directly. SMTP authentication is useful for allowing remote users to relay mail through the external company mail server. Claus Aßmann has some documentation, but I found it a bit hard to follow, so hopefully this is a little clearer.

Compiling Sendmail with SSL Support

Most recent Linux distributions come with Sendmail compiled with SSL support already. Solaris unfortunately does not. I don't know about other operating systems. Double check your vendor's Sendmail before compiling your own.

First, you need to have OpenSSL installed. Once you have that installed, you need to create/modify your site.config.m4 so that Sendmail knows to compile in support for SSL. site.config.m4 resides in devtools/Site/ within the Sendmail source tree. Add the following lines (modifying the paths as appropriate). If your OpenSSL install is someplace searched by your compiler by default, then the confINCDIRS and confLIBDIRS lines are not necessary.

dnl Stuff for TLS
APPENDDEF(`confINCDIRS', `-I/usr/local/include')
APPENDDEF(`confLIBDIRS', `-L/usr/local/lib')
APPENDDEF(`conf_sendmail_LIBS', `-lssl -lcrypto')

Then a standard ./Build and ./Build install should get you a Sendmail binary with STARTTLS support.

Server Configuration for STARTTLS

You need to add a few lines to your m4 config file and regenerate your (See my page about Sendmail in the DMZ for a sample m4 file for a DMZ mail server.) The additional lines should look something like this:

define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/cacert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/cert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/key.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/cert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/key.pem')dnl

This defines the home of your server certificate and key to be MAIL_SETTINGS_DIR/certs, or probably /etc/mail/certs in most cases. Within that directory should be three files (instructions for generating them are in the next section):

Certificate Authority (CA) certificate
x.509 certificate, signed by CA
x.509 private key

Server Certificate Generation

Although you could go purchase certificates from a public CA like Verisign, for our purposes a local CA is sufficient. To setup your CA you can use my JavaCA tool or follow Greg Shapiro's instructions in the "To make certificate authority" section. The resulting cacert.pem should be copied to /etc/mail/certs.

Next you need to create a key and certificate for the server. Again follow the instructions on the above page in the "To make a new certificate" and "To sign new certificate with certificate authority" sections. The first openssl command will prompt you for a bunch of information. Generally it is obvious what to enter, but make sure to enter the hostname that users will be accessing in the Common Name field (openssl suggests your name when it prompts for the CN). Otherwise most MUAs will warn the user that the server hostname doesn't match the name in the cert. Edit newreq.pem and remove the unsigned certificate (leaving the private key). Copy the resulting newreq.pem to /etc/mail/certs/key.pem and copy newcert.pem to /etc/mail/certs/cert.pem. Set the permissions on key.pem to 400 and the owner to whoever Sendmail runs as (confRUN_AS_USER in the m4, RusAsUser in the .cf; defaults to root but possibly something like mailnull on a DMZ mail server).

Chris Dagdigian sent me a note with a good bit of info, newer versions of Mozilla and Netscape won't accept the server certificate if it is self-signed. I.e. if the CN of the CA certificate and CN of the server certificate are the same. So make sure you use something like "Joe's Certificate Authority" for the CN of the CA certificate and use the server's hostname as the CN of the server cert.

You should now be able to restart Sendmail. Check syslog for any errors. Then telnet to port 25, issue an ehlo and make sure that STARTTLS is listed in the supported features.

Allowing Relaying Based on Certificate Authentication

One of the major reasons for authenticating SMTP senders is to allow your remote users to relay mail through your mail server. You could proceed to configure SMTP AUTH at this point and have your users authenticate with a regular username and password. However, there is another option. You can issue an SSL certificate to each user and they can use the SSL cert to authenticate. In my case, I allow users to relay if their certificate has been signed by my CA. To do this, you need to get the Distinguished Name (DN) from your CA certificate. Run the following command:

openssl x509 -in cacert.pem -noout -text

and look for the Subject line (or run it on your server cert and look for the Issuer line). It should look something like this:

Subject: C=US, ST=Washington, L=Seattle, O=Foo Enterprises, OU=Security Division, CN=Jason Heiss

That is the DN for your CA certificate. That needs to go in your access map (you are using an access map to control relaying, right?), but Sendmail expects a slightly different format. The corresponding entry in the access map to allow relaying by all clients with certs signed with this CA would be:

CERTISSUER:/C=US/ST=Washington/L=Seattle/O=Foo+20Enterprises/OU=Security+20Division/CN=Jason+20Heiss        RELAY

In other words, all of the elements have changed from being seperated by a comma and a space to being seperated by a forward slash. In addition, a leading slash has been added and the text has been passed though a "CGI-like" filter where certain characters have been replaced by their hex values with a leading plus sign (spaces have been replaced by +20, plus signs by +2B, etc.) Claus lists the complete set of characters to be converted to their hex values as non-printable characters plus {<>()"+ } (the curly brackets denote the contents of the set and are not themselves included). I did the conversion by hand, but you could whip up a Perl script if you have a lot to convert.

You can get more granular and allow or deny relaying based on the DN of the cert itself (instead of the CA). To do this, replace CERTISSUER with CERTSUBJECT in the above line and specify DNs for individual certs. There is support for some amount of regular expressions in the DN, see Claus' page for info on that.

Client Certificate Generation

Now you need to give certificates to your clients to complete this puzzle. If your client is a UNIX box running Sendmail (i.e. a Linux laptop), then you just configure it the same way as the server (with regards to STARTTLS, other configuration will obviously be different), generate a cert and sign it with the same CA as the server and everything will be fine. Sendmail will use the CLIENT_CERT and CLIENT_KEY configured above in the m4 file when connecting to other SMTP servers. This will work fine for most UNIX MUAs, since most of them just hand outbound mail off to the local Sendmail.

If your clients' MUAs don't hand mail off to the local Sendmail, or are not running some form of UNIX, then you need to configure those MUAs to use your Sendmail server for outbound mail and generate a certificate for the user to install into the MUA. Netscape Messanger 4.x is the MUA that I'm familiar with, so I'll describe that here. First off you need to generate a key and cert (signed by your CA) just like you would for a Sendmail server. But then you need to concatenate the CA cert, the signed user cert, and the private key (in that order) into one file. Then you need to package that bundle up in PKCS#12 format, which is the format that Netscape expects certs to come in. To do that, you run the following command on the concatenated file (input.pem in this example):

openssl pkcs12 -export -in input.pem -out joeusercert.p12 -name "Joe User's Personal Certificate"

Then send the resulting joeusercert.p12 off to Joe User. He then selects Communicator->Tools->Security Info in the browser. Then he selects Yours under the Certificates section on the left side of the Security Info dialog. Then he clicks the Import a Certificate button and selects joeusercert.p12. Hit OK to close the Security Info dialog box, then select Edit->Preferences, choose Mail Servers under the Mail & Newsgroups section on the left side of the Preferences dialog. Click If Possible under the Use Secure Socket Layer (SSL) or TLS for outgoing messages.

Now when the user tries to send outgoing mail, Netscape will realize that the SMTP server understands TLS and will prompt the user for the appropriate cert to present to the server. The cert has been signed by your CA, so Sendmail will let the user relay mail.

jheiss at
$Id: tlsandrelay.shtml,v 1.9 2009/05/26 17:05:09 jheiss Exp $