Email forwarding problem

The email forwarding problem refers to the authentication failures that occur in some popular IP-based email authentication methods when a forwarder is involved. In this discussion, we will assume the reader understands the basics of Email authentication and the roles of different agents in the Email system, as illustrated in this diagram:

|--- Sender's Network ---|           |--------- Recipient's Network --------|
                                /
Author ==> MSA/Transmitter --> / --> Receiver/Forwarder ~~> MDA ==> Recipient
                    /         /        /
                   /       Border     /
                  /                  /
                  ------ DNS -------  

SPF, and its offshoot Sender ID, seek to authenticate the domain name of an originator, usually the MSA, with an IP address from the last agent in a chain of trust, usually the Transmitter. This works well because there is a close relationship between the MSA and the Transmitter. Often they are the same agent. When a Forwarder is involved, however, the source IP address on the "last hop" to the MDA is no longer related to the originator's domain name. An SPF check by the MDA will likely fail.

The first solution to this problem was proposed by SPF, a campaign to get Forwarders to re-write the Return Address on forwarded messages, so that it would have the Forwarder's domain name. (See step 5 in the SPF "How it works" example.) This failed because Forwarders were not willing to install special software and take on responsibility for bounced messages. Also, it required changing a long-standing interpretation of the Return Address as being that of the original sender.

A simpler solution was proposed by Sender ID. The domain of the Forwarder is provided as part of the command that sends the Return Address. (See step XX in the Sender ID "How it works" example.) This required no change in the Return Address or its interpretation, and would not have changed the handling of bounced messages. The solution was never widely deployed, however, probably because Sender ID itself was never widely adopted.

The current solution is to emphasize proper setup within the recipient's network. SPF checks should be done only at the Border. When a recipient sets up forwarding from his old Receiver/Forwarder to his new MDA, he should make sure that the Forwarder is whitelisted by the MDA. Forwarders and MDAs should understand their roles in the setup, and should make sure that recipients (non-expert users) don't make common mistakes. MDAs should understand that forwarding is a common need, and make it easy for recipients to whitelist their forwarders.

Unfortunately, the confusion over forwarding has persisted so long that it has led to the second big problem with SPF/Sender_ID, the "chicken-and-egg" standoff. Senders are unwilling to publish robust SPF records ending in "-all" due to the fear that their mail will be rejected in mis-configured forwarding setups. The common practice among nearly all domains publishing SPF records is to end them in ?all. This says to treat all other IP addresses not as Fail, but Neutral. This takes away the major motivation for Receivers to do SPF checks. Without large numbers of Receivers taking these records seriously, senders find that they can be sloppy with their SPF records, or not publish them at all, and it doesn't matter.

Current research on the forwarding problem includes developing protocols for Forwarders and MDAs to communicate with each other when a recipient requests forwarding. A widely-accepted protocol would allow the agents involved to set up a forwarding arrangement with just a simple request from the recipient.