The FST API

Input Sources

There are many possible ways for a mail system to use the Forensic Sender Test™ and depending on the exact implementation, two basic ways to find the alleged sender address(es) to test against the delivering IP address.

For SMTP Firewalls

When used in a protocol-level firewall, the inputs are simply the connected peer's IP address and the email address portion of the SMTP MAIL FROM command. It is recommended that the mail system programmer be fully familiar with BOTH the official requirements of RFC 5321 Section 3.3 "Mail Transactions" AND the common practice of submitting a MAIL FROM address containing only an email address without even the text identifiers "<" and ">".

NOTE: There is much discussion on the required use of the "null sender" seen as "<>" in the MAIL FROM command which theoretically exists to keep a bounce of a bounce from being transmitted. We recommend that you neither generate external bounce messages nor accept the null sender in your MAIL FROM command. Our reasoning for this is that modern mail systems are capable of refusing mail after the RCPT TO command in the SMTP transaction making bounce messages unnecessary. Old systems that use bounce messages instead of instant verification (such as Exchange 2000) are used by spammers to relay spam. That means that your system is either part of the problem or absorbing the destructive power of the problem and either way, the null sender exists only as a problem. (Submitting a null sender to the FST is a waste of a transaction because there is nothing to test.)

For Post-SMTP Processing

After a message has been received, a post processor must be able to reliably recognize several pieces of information in the email headers to be certain that all possible senders have been checked for authenticity. Some organized criminals, for instance, will use a valid MAIL FROM in the SMTP transaction but a bank's actual customer service email address in the From field in the mail header. RFC 5322 actually describes many more sources of sender addresses. Different mail clients interpret these header fields in their own way, but the primary sources of input are:

1. The IP address of the host that delivered the email to the mail exchanger: Some SMTP server programs inject this IP address as an easily identifiable header field and others leave it in the more ubiquitous "Received" header stamp stacked at the top of the message. Since this is a pivotal piece of information, you'll want to be certain that your post-processor gets the correct IP address and identifies things like internal mail relays.
   
   
2. If you didn't pre-screen the message with a call to the FST to check the MAIL FROM during the SMTP transaction, you'll need to find it now: This address SHOULD, in theory, be in the Return-Path field of the header, but may be stamped as a header field or embedded in the Received stack.
   
   
3. The From header field: This field can actually contain a list of addresses separated by commas and optionally surrounded by <>. Check RFC 5322 for complete details. Each address presented in From should be sent to the FST because of the identity theft possibilities. A spam scoring system might incorporate each FST result as a separate score component.
   
   
4. The Sender header field: If the From field contains more than one address, the Sender field must contain exactly one address. This address is more important than any address in the From field for determining the authenticity of the overall message, but of equal importance in making the identity theft determination. Financial institutions do NOT send email from a mix of addresses and especially do not include domains outside of their own.
   
   
5. The Reply-To header fields: These fields are primarily important when making identity theft determinations. In person-to-person email, redirected replies often facilitate the use of aggregating mailboxes and may or may not even be a part of a spam score. But in email from identity theft targets, security and privacy policies (as well as a some regulations) make it impossible or at least grossly irresponsible to see a Reply-To address cause an FST failure in an authentic email.
   
   
6. Other fields: RFC 5322 makes mention of various other header fields which may contain possible sender addresses. The use of those fields is up to the post-processor programmer and the specific needs and trends of the mail system. Currently, neither identity thieves nor mail clients are RFC-aware enough to make the testing of other fields valuable, but that could always change since the fields exist in a formal definition.

The HTTP Transport

The FST interface is reached via standard HTTP request with its inputs in the URL and output presented in a plain text HTTP response. Since a mail system might employ a direct socket link to accomplish the request, the following sample request packet may be helpful:

GET /FST2.dll?Key=xyz&IP=1.2.3.4&Senders=joe@test.com%20jane@test.com&Also=whois HTTP/1.1
Accept: */*
User-Agent: Custom Socket Interface
Host: service.fst.ingenisys.com
Connection: Keep-Alive

And a sample result packet:

HTTP/1.1 200 OK
Date: Wed, 01 Oct 2008 21:28:53 GMT
Server: Microsoft-IIS/6.0
Content-Type: text/html
Content-Length: 62

IPF=111
MXF=110 110
SUM=110
WHOIS=Some Company Name, Inc.

The specific parameters of the URL and named result codes returned are discussed in detail on their own pages. General guidelines for the transport are as follows:

1. The request MUST be URL encoded. Some HTTP components do this for you, but if you plan to use a direct socket connection, it's up to the programmer. URL encoding algorithms are available for virtually every programming language with a quick Google search.
   
   
2. The value of the Key parameter may change from time to time for security purposes. Your system should store and read this value in a manner that can be updated without recompilation of executable code or interruption of mail processing.
   
   
3. Delimit lists in parameters using a space character (%20 encoded). (This is used in the "Senders" and "Also" parameters.)
   
   
4. The response is given in a series of name/value pairs with everything to the left of the first "=" on a line being the name and everything after being the value. Values MAY contain "=". Most modern programming languages have built-in facilities for accessing name/value lists.
   
   
5. Each request on a connection should be handled as asynchronous but if your software is able to do so, continued processing of an email while waiting for an FST response is up to your own preferences and possibilities.
   
   
6. The same connection may and should be kept alive for streaming requests. This cuts down on the overhead of building and destroying the TCP connection (which is necessary to ensure the integrity of the transaction). You will be allocated a maximum number of concurrent connections based on your volume. This number will more or less define the maximum number of processing threads (or forks) described below.
   
   
7. The total time to process an FST request may be as high as a few seconds so your post processor should use as many concurrent threads as you deem appropriate for your volume. If you are able to use the same threads to process continuous emails, connection pooling becomes inherent in your design. If you plan to spawn (or fork) a terminating thread for each email received into the system, your threads should communicate with a central connection pooler.
   
   
8. Your system must use a timeout of 30 seconds on each request. (This is many times the actual expected wait time and is elongated to minimize unnecessary fail-over invocations.) A timeout should invoke the fail-over as described in the next section.

You will be assigned to a dedicated development support agent during your integration process. If any of the above is unclear, your DS agent will clarify it to your satisfaction. Our goal is to ensure your success and we will go to great lengths to achieve that goal.

Failure Safety and Load Balancing

The FST is delivered via a distributed network of servers. The method of selecting and connecting to a server is specifically designed to achieve:

1. Fail-over redundancy
2. Dynamic resource allocation
3. Load balancing
4. Maintenance and upgrade transparency

When opening an HTTP connection to the FST service, DNS should be queried for the A record of "service.fst.ingenisys.com". Multiple IP addresses will be returned. The connection should be attempted randomly on each of the available IP addresses until a connection is established. If a connection fails, that IP should be removed from the candidate list and another random selection made. If all IP addresses are eliminated without a successful connection, DNS should be re-queried and the process started over.

In the event that an FST query is issued and no response received within the 30 second timeout window described in the section above, the connection should be dropped and a new DNS query and HTTP connection constructed.