10 ways to secure your Windows-based SMTP relays

By Thomas W. Shinder, MCSE, MVP

Organizations hosting their own mail services must accept incoming connections from anonymous SMTP servers on the Internet. In addition, outbound mail must allow connections to SMTP servers of unknown security configuration. You can mitigate some of the security risks inherent in connecting to and from anonymous SMTP servers by using inbound and outbound SMTP relays. These 10 tips will help you bolster the security provided by your company’s inbound and outbound SMTP relays.

Use an SMTP relay

An SMTP relay is a computer that accepts SMTP messages from an SMTP client computer. The SMTP client can be an end-user e-mail application or another SMTP server. An SMTP relay can be used for incoming and outbound e-mail. The security benefit you get from using an SMTP relay is that it prevents connections from untrusted hosts to and from your mailbox servers on the corporate network. The inbound SMTP relay accepts incoming connections from Internet SMTP servers sending mail to your SMTP domains, and an outbound SMTP relay accepts mail from your internal mail servers and forwards it to external SMTP servers on the Internet. In both cases, your secure mail servers on the corporate network never need to communicate with untrusted hosts; the SMTP relay takes care of that dangerous business.

Disable anonymous and authenticated relay at the server level

The best way to get your SMTP server blacklisted is to allow users to use it to relay messages to any e-mail domain they like. This is often called an "open relay." The open relay accepts incoming SMTP messages and sends them to any e-mail domain. Spammers use open relays to send spam while covering their tracks. You should configure your inbound SMTP relay to deny relay for all domains except for your own e-mail domain. You can do this in the IIS 6 SMTP service by going into Properties for the SMTP service and clicking the Access tab. On the Access tab, click the Relay button. In the Relay Restrictions dialog box, confirm that the All Except The List Below option is enabled and that there are no IP addresses in the Computers list. Deselect the Allow All Computers Which Successfully Authenticate To Relay, Regardless Of The List Above check box. This prevents spammers who try to brute-force user credentials from relaying through the inbound SMTP relay server.

Place the inbound SMTP relay in an anonymous access DMZ

There are two general types of DMZs: anonymous access and authenticated access. The anonymous access DMZ allows unauthenticated connections through the firewall and into servers on that DMZ segment. An authenticated access DMZ requires that the user authenticate at the firewall, and only after successfully authenticating does the firewall allow the connection to the server located on the authenticated access DMZ. External SMTP servers can’t authenticate to your inbound SMTP relay, so place it on an anonymous access DMZ to segregate it from domain member computers, which are typically placed on authenticated access DMZs.

Use different SMTP relays for inbound and outbound mail

Inbound and outbound SMTP relays serve different roles and require significantly different configuration for a secure environment. Inbound SMTP relays allow connections only from unauthenticated SMTP servers on the Internet and allow relay only to your own SMTP domains. Outbound SMTP relays accept connections only from SMTP servers on your corporate network and should be configured to require authentication from the SMTP server before accepting mail for outbound relay. Although it’s possible to configure the same machine to perform both roles, a more secure and easily managed configuration is to place these relays on different physical machines and place them on different DMZ segments.

Use an Application Layer inspection firewall in front of the inbound SMTP relay

Although the inbound SMTP relay is designed to accept incoming connections from Internet SMTP servers, there’s no way you can guarantee that incoming SMTP connections are from legitimate servers. Attackers can use the anonymous inbound connection to attack your inbound SMTP relay. One way you can help protect your inbound SMTP relay from attack is to use an Application Layer inspection firewall in front of the SMTP relay. Application Layer inspection firewalls can perform both stateful packet

and Application Layer inspection. One example of such a firewall is the ISA firewall (ISA Server 2004). It will allow only legitimate SMTP verbs and prevent buffer overflows for them.

Enable logging for the SMTP service

The SMTP service should be able to log all connections made to the SMTP service. The IIS SMTP service can perform comprehensive logging for all incoming connections to the service. SMTP service logs are valuable in troubleshooting connectivity issues and can also be used for forensics. The IIS SMTP service does not turn on logging by default, so you will need to enable it. Just open the Properties dialog box for the SMTP service and on the General tab, select the Enable Logging check box. Then, select the type of logging format you want to use and click the Properties button to enable the fields you want to include in the logs.

Perform DNS lookups on incoming messages

Reverse DNS lookups enable the SMTP server to validate the name of the SMTP server establishing the connection to your SMTP relay. As part of the connection process, the external SMTP server provides its name to your inbound SMTP relay. This name should be a resolvable Internet host name. If the reverse DNS lookup is successful, the RECEIVED header will remain intact. If the verification is unsuccessful, "unverified" appears after the IP address in the RECEIVED header of the message. If the DNS lookup fails, "RDNS failed" will appear in the RECEIVED header of the message. Because this feature verifies addresses for all incoming messages, its use could affect Microsoft SMTP Service performance. Note that this does not prevent these messages from passing through the SMTP relay, but it does provide information about potentially illegitimate messages.

Require authentication for outbound SMTP relays

Outbound SMTP relays are used only by SMTP servers on the corporate network. All mail that users send outbound to the Internet should be routed through the outbound SMTP relay. For example, Microsoft Exchange Server can be configured to use a smart host instead of doing DNS resolution for routing outbound mail. You configure the Exchange Server’s SMTP service to use the outbound SMTP relay as its smart host. You want to prevent any other machine from using the outbound SMTP server for outbound relay. You can do this by requiring authentication at the relay. Create an account that your corporate mail servers can use to authenticate to the relay and then configure your corporate Exchange Server to use those credentials to authenticate.

Use SSL and authentication for end-user relays

You may have users who need an SMTP server to which they can send outbound mail when they’re at remote locations. This is the case when they’re using conventional e-mail client applications, such as Outlook Express. These users will usually not have access to an SMTP server when at a remote location. The SMTP server must be able to relay messages to any SMTP domain, including your own. If you need to configure an SMTP relay for external users, you should configure the server to require authentication and also require SSL. You can then install an SSL certificate on the SMTP server and configure it to require SSL. Incoming connections that can’t authenticate and establish an SSL connection to the "end-user" SMTP will fail.

Allow only triggered delivery from inbound SMTP relays

Inbound SMTP relays accept incoming connections from anonymous SMTP servers on the Internet. Because this Internet-facing host allows anonymous connections, there is an increased risk that the SMTP server will be compromised. Since this SMTP server is allowed inbound access to TCP port 25 from the anonymous access DMZ to the mail server on the corporate network, someone could potentially attack the internal SMTP server over this port by initiating a connection from the anonymous access inbound SMTP relay to the corporate mail server. One way to mitigate this threat is to allow the inbound SMTP relay to forward SMTP messages only when the corporate SMTP server on the internal network authenticates and requests the mail to be sent. The SMTP server issues an authenticated TURN (ATRN) command. The inbound SMTP relay then sends messages to the ATRN domain. You can create a remote domain on the IIS inbound SMTP relay for an address that periodically reviews messages and enable ATRN for the domain.

 

Published: December 13, 2005

Version history

Version: 1.0