The outgoing mail transport agent (“mail server”) computes a hash of the body of the mail message and adds a special, cryptographically signed DKIM-Signature header to the mail message that contains this hash as well as some other information about the message. The public part of the key, used by receiving mail transport agents to verify the signature on incoming messages, is stored as a TXT record in the DNS for the signing domain.

The result of all of this is that the recipient can now know that the body and certain headers of the e-mail message were not tampered with or changed by a third party while the message was in transit. If the signature contained within the DKIM-Signature header doesn’t verify, then it’s possible that the message is a phishing message, spam, or some other false representation of members of the signing domain. (IMHO, however, it is more likely to just indicate a broken DKIM implementation, as there are several obnoxious corner cases that we will see.)

Where is the Exchange support?

No currently released version of Microsoft Exchange supports DKIM natively, not even 2010. This is purportedly because Microsoft threw its weight behind another standard called SPF, or Sender Protection Framework. SPF is much simpler to implement because it does not make a statement about the integrity of the message contents; that is, there is no signing step that requires processing of each outbound message and stamping it with a special header. Instead, a TXT record in the DNS for the sending domain lists the IP addresses from which e-mail messages from that domain are allowed to originate.

For example, let’s say that a store has a mail transport agent at mail.example.com and a Web site at www.example.com. An SPF record for example.com might make a statement like the following:

“E-mail messages purporting to be from @example.com should only originate from either mail.example.com or www.example.com. All other originating IP addresses should be treated with suspicion.”

You can see why this is easier to implement; you just add an entry to DNS, and you don’t have to configure the outbound mail transport agent to do anything. The only reason a mail transport agent would need to change is for it to gain the ability to inspect incoming messages and validate the MAIL FROM part of the message headers against the corresponding SPF TXT records in DNS. And that is what Exchange does today.

While SPF does not validate the integrity of the message–a hacked, legitimate company mail server that spews out spam unwittingly will still pass an SPF–it is a simple, convenient way to mark phishing and spoofing attempts originating from other locations, like a random house in Wisconsin that is unwittingly part of a botnet, with suspicion.

But, then again, a hacked mail transport agent using DKIM that spews out spam unwittingly will still pass a DKIM verification. Spam can be signed just like regular mail. So the advantage of DKIM over SPF is that DKIM provides the ability to prove that the contents of the message were not modified in transit, and if an organization so chose, they could sign different messages with different keys or decide to not sign certain kinds of messages at all, allowing the recipient to interpret these different kinds of mail in different ways.

In reality, most e-mail doesn’t need to be that secure, so the utility-to-difficulty ratio may be a little out-of-whack here, especially when other standards for signing messages already exist.

So DKIM is more complicated and not supported by Microsoft. Why implement DKIM on Exchange?

Because fuck Yahoo!, that’s why.

Actually, let’s step back a minute.

Yahoo! invented something called Domain Keys, which predates DKIM and which DKIM is largely based on, in an attempt to combat spam in the ways described previously. The aim was for Yahoo! mail servers to be able to easily identify entire classes of suspect e-mail and send them to people’s spam folders (which, in Yahoo! weasel words, is referred to as the “bulk mail folder”). Then they integrated it into the Yahoo! mail system.

If your business sends mail of any kind of volume to Yahoo! mail servers, such as an e-mail newsletter that users subscribe to, you’ll quickly see a message along the lines of the following:

421 4.16.55 [TS01] Messages from x.x.x.x temporarily deferred due to excessive user complaints

And you might also notice your e-mail messages going straight to the spam folder on Yahoo! mail accounts by default, with a X-Yahoo-Filtered-Bulk: x.x.x.x header appearing in the message headers. If this has happened, it is because Yahoo! has decided that you are naughty and placed the IP address of your mail transport agent on an internal blacklist. If a Yahoo! user has previously marked e-mail from your domain as “Not Spam,” then that user will continue to receive your e-mail, but any other Yahoo! users will find your e-mail going to the spam folder by default until they either click the “Not Spam” button on one of them or add your from address to their contact list.

“But we’re a legitimate small business,” you say. “Only about 5,000 of the e-mails on our mailing list are actual Yahoo! addresses, the newsletter list is opt-in, and there are ‘Unsubscribe’ links in every e-mail. It’s not like we’re spamming users out of the blue,” you continue. “These are users who have asked for this service and can cancel at anytime. So why has Yahoo! blacklisted us and treat us like spam?”

The answer, unfortunately, is that Yahoo! is incompetent and can’t build a mail service that can handle spam like Gmail can handle spam. Instead, their e-mail system depends on the following metrics:

• If a large burst of e-mail comes from the same IP address within a certain time period, temporarily consider it as a spam and temporarily deny the requests. The mail will still arrive, but the “fat pipe” will be squashed down so that overall delivery takes longer to complete. The idea is that if the mail is originating from a spammer, they are unlikely to try again.
• If a “significant” number of Yahoo! users mark mail messages originating from the same IP address as spam by clicking on the “Spam” button in their account, add that IP address to the X-Yahoo-Filtered-Bulk blacklist.

In other words, if enough people mark your e-mail as spam, your IP address is fucked, and you’ll soon be fielding complaints from customers who insist that they aren’t receiving your Web sites transactional e-mails, such as a “forgot my password” or “shipping confirmation” e-mail, because Yahoo! has now decided to deliver all e-mails that originate from your IP address to the spam folder, which many users do not think to check or cannot find. So they blame you.

If you think that Yahoo! has blacklisted you in error, or if you have improved your e-mail sending practices and wish to have them consider you for removal, you can fill out this form with incorrect JavaScript required field validation to request that the Yahoo! postmaster take you off the naughty list. About 27 hours later, a Yahoo! representative with a sub-room temperature IQ will e-mail you back a long, canned reply that generally amounts to “No.”

But in that “no” message, they offer the possibility of signing up to the Complaint Feedback Loop. When you participate in this program, if a Yahoo! user presses the “Spam” button on one of your e-mails, then Yahoo! will e-mail you the e-mail address of that user along with a copy of the e-mail that you sent. You can then unsubscribe that user from your mailing list yourself (since they neglected to use the unsubscribe link in your e-mail and will continue to damage the reputation of your IP address if you continue to send mail to them), or you can see which kinds of e-mails you are sending are creating the most problems.

But participation in their feedback loop requires that you use DKIM to sign all of your outbound mail messages. The reasons for this are not clear, but I assume it has to do with ensuring that you are only notified about the problematic e-mails that legitimately came from your organization, and not ones generated by a spammer. Or, it could simply be a way to drive adoption of Yahoo!’s DKIM standard.

At any rate, Yahoo! mail is the most widely used e-mail service on the planet, so when the idiots in the higher echelons of Yahoo! say “Jump!” then we must ask “How high?” When they say “implement DKIM”, we implement DKIM.

So what are options for implementing DKIM on Exchange?

One option is to simply not do it in Exchange and set up a relaying mail server that has DKIM support, like hMailServer or Postfix with dkim-milter. But if you’re at a small business, the idea of maintaining yet another server for what is conceptually a simple task is not a pleasant thought.

Another option is to use a dedicated device like a Barracuda or an IronPort. This device would sit in front of Exchange and rewrite the mail headers in transit, adding the DKIM-Signature header as it flies out of your office. But these devices are not cheap and are out of reach of many small businesses. And the thought of acquiring a specialized device for doing something any mail transport agent should be able to do natively is not a pleasant thought.

You can buy an off-the-shelf plug-in for Exchange like this one from a company in Hong Kong. The reviews on the Internet do generally seem to indicate that it does work. But do we really want to spend $300 – $800 on a component from a company without a reputation and give that component read access to all of our organization’s outbound mail messages? Trust is certainly a driving factor here.

A fourth option, and the option I pursued since I considered giving up on DKIM if Yahoo! didn’t change its ways after our implementation, is to take advantage of Exchange 2007′s new Transport Agents functionality. This allows you to write custom, managed code that runs on that .NET Framework that integrates with the Exchange message processing pipeline. In our case, we could write a custom transport agent that appends a DKIM-Signature header to outgoing MIME messages.

Setting up the project

In Visual Studio, we just need to use a plain old “C# Class Library” project. The project must target version 2.0 of the CLR (that’s .NET Framework 2.0, .NET Framework 3.0, or .NET Framework 3.5, thanks to the dipshits in Microsoft’s marketing department) and not the .NET Framework 1.x or, maddeningly, the .NET Framework 4.x. This is because the transport agent process provided by Exchange Server 2007 that will load our transport agent is running on CLR 2.0, and a process that’s been loaded with an earlier version of the CLR can’t load a later version.

We also need to reference two assemblies provided by the Exchange Server, Microsoft.Exchange.Data.Common.dll and Microsoft.Exchange.Data.Transport.dll. You would think that you would be able to find these in, say, the Exchange Server SDK that’s available from Microsoft Downloads. But you’d be wrong. Microsoft keeps diddling with the version of these assemblies with various update rollups for Exchange, and the only way to get a copy is to pull them off an actual Exchange Server 2007 installation. Mine were located in C:\Program Files\Microsoft\Exchange Server\Public; I just copied them to my local computer and referenced them in my new class library assembly for local development.

To start, we just need to create two classes. The first is our actual agent, which derives from the RoutingAgent class defined in the DLLs we just copied:

public sealed class DkimSigningRoutingAgent : RoutingAgent
{
	public DkimSigningRoutingAgent()
	{
		// What "Categorized" means in this sense,
		// only the Exchange team knows
		this.OnCategorizedMessage += this.WhenMessageCategorized;
	}
 
	private void WhenMessageCategorized(
		CategorizedMessageEventSource source,
		QueuedMessageEventArgs e)
	{
		// This is where the magic will happen
	}
}

The second is a factory class that the Exchange server’s little plug-in agent looks for; it’ll instance new copies of our agent:

public sealed class DkimSigningRoutingAgentFactory : RoutingAgentFactory
{
	public override RoutingAgent CreateAgent(SmtpServer server)
	{
		return new DkimSigningRoutingAgent();
	}
}

You’re probably thinking, “Wow! The documentation provided by the Exchange team sure blows donkey chunks, how did you ever figure out that that was what you needed to do?” The answer is by spelunking through the Exchange Server SDK examples and by religiously following this guy’s blog.

Now we’re ready to start reading mail messages and “canonicalizing” them into the format that the DKIM spec expects.

Exchange and “Internet Mail”

If there is one thing that is annoying about Exchange, it is that 20-some years after the fact it seems skeptical that this whole “Internet Mail” thing is going to catch on. Getting Exchange to give you the raw message in MIME format is not as simple as one might think.

You see, to be able to hash the body of the message and sign certain headers in the message, we need to know exactly how Exchange is going to format it when it is sent. Even a single space or newline that is added after we compute the hash can throw the whole thing off.

In our routing agent’s OnCategorizedMessage event listener, the event arguments give us access to an instance of the MailItem class. It has a boatload of properties for accessing the body and headers of the message programmatically. Unfortunately, we can’t use these properties because they represent the semantic values, not the raw ones. Instead, we’ll need to use the GetMimeReadStream() and GetMimeWriteStream() methods to read the raw mail message and write out the modified version, respectively.

Implementing the routing agent

Let’s start by completing the Routing Agent implementation. We’ll keep it simple by moving all of the hard signing stuff into an IDkimSigner interface, which we’ll worry about implementing later:

public sealed class DkimSigningRoutingAgent : RoutingAgent
{
    private static ILog log = LogManager.GetLogger(
        MethodBase.GetCurrentMethod().DeclaringType);
 
    private IDkimSigner dkimSigner;
 
    public DkimSigningRoutingAgent(IDkimSigner dkimSigner)
    {
        if (dkimSigner == null)
        {
            throw new ArgumentNullException("dkimSigner");
        }
 
        this.dkimSigner = dkimSigner;
 
        this.OnCategorizedMessage += this.WhenMessageCategorized;
    }
 
    private void WhenMessageCategorized(
        CategorizedMessageEventSource source,
        QueuedMessageEventArgs e)
    {
        try
        {
            this.SignMailItem(e.MailItem);
        }
        catch (Exception ex)
        {
            log.Error(
                Resources.DkimSigningRoutingAgent_SignFailed,
                ex);
        }
    }
 
    private void SignMailItem(MailItem mailItem)
    {
        if (!mailItem.Message.IsSystemMessage &&
            mailItem.Message.TnefPart == null)
        {
            using (var inputStream = mailItem.GetMimeReadStream())
            {
                if (this.dkimSigner.CanSign(inputStream))
                {
                    using (var outputStream = mailItem.GetMimeWriteStream())
                    {
                        this.dkimSigner.Sign(inputStream, outputStream);
                    }
                }
            }
        }
    }
}

The only real quirk is the if statement in the SignMailItem() function, which I mostly discovered through trial and error. If the mail item is a “system message” (whatever that means) then all of the mailItem‘s methods will be read only (throwing exceptions if we try to mutate), so we shouldn’t even bother. And if the mail item has a TNEF part, then in it’s in a bizarro proprietary Microsoft format, and the DKIM spec just isn’t going to work with that. Finally, if something blows up, we catch the exception and log it–better to send a message without a signature than not send it all.

Defining an interface for DKIM signing

So the next step is to make up that IDkimSigner implementation and make it do the dirty work. You can see that I’ve made it simple in that we only need to write two methods:

public interface IDkimSigner : IDisposable
{
    bool CanSign(Stream inputStream);
 
    void Sign(Stream inputStream, Stream outputStream);
}

A method for sanity checking

The first method will scan our mail item’s content stream and do a sanity check and ensure that we can actually sign the message. For example, if our IDkimSigner implementation is configured to sign messages originating from warehouse1.example.com and we pass CanSign() a message from warehouse2.example.com, then we can return false to indicate that we just don’t know what to do with the message. Let’s implement that method.

private string domain;
 
public bool CanSign(Stream inputStream)
{
    bool canSign;
    string line;
    StreamReader reader;
 
    if (this.disposed)
    {
        throw new ObjectDisposedException("DomainKeysSigner");
    }
 
    if (inputStream == null)
    {
        throw new ArgumentNullException("inputStream");
    }
 
    canSign = false;
    reader = new StreamReader(inputStream);
 
    inputStream.Seek(0, SeekOrigin.Begin);
 
    line = reader.ReadLine();
    while (line != null)
    {
        string header;
        string[] headerParts;
 
        // We've reached the end of the headers (headers are
        // separated from the body by a blank line).
        if (line.Length == 0)
        {
            break;
        }
 
        // Read a line. Because a header can be continued onto
        // subsequent lines, we have to keep reading lines until we
        // run into the end-of-headers marker (an empty line) or another
        // line that doesn't begin with a whitespace character.
        header = line + "\r\n";
        line = reader.ReadLine();
        while (!string.IsNullOrEmpty(line) &&
            (line.StartsWith("\t", StringComparison.Ordinal) ||
            line.StartsWith(" ", StringComparison.Ordinal)))
        {
            header += line + "\r\n";
            line = reader.ReadLine();
        }
 
        // Extract the name of the header. Then store the full header
        // in the dictionary. We do this because DKIM mandates that we
        // only sign the LAST instance of any header that occurs.
        headerParts = header.Split(new char[] { ':' }, 2);
        if (headerParts.Length == 2)
        {
            string headerName;
 
            headerName = headerParts[0];
 
            if (headerName.Equals("From", StringComparison.OrdinalIgnoreCase))
            {
                // We don't break here because we want to read the bottom-most
                // instance of the From: header (there should be only one, but
                // if there are multiple, it's the last one that matters).
                canSign = header
                    .ToUpperInvariant()
                    .Contains("@" + this.domain.ToUpperInvariant());
            }
        }
    }
 
    inputStream.Seek(0, SeekOrigin.Begin);
 
    return canSign;
}

Barf. But we have to do this style of ghetto parsing because, after all, we’re dealing with the raw e-mail message format. All we’re doing is scanning through the headers until we reach the last From: header, and then we make sure that the From: e-mail address belongs to the domain that our instance knows how to sign. Then we seek back to the beginning of the stream to be polite.

A method for signing

The second method that we have to implement is the one that actually does all of the dirty work. And in DKIM signing, we can break it down into five steps:

1. Compute a hash of the body of the message.
2. Create an unsigned version of the DKIM-Signature header that contains that body hash value and some other information, but has the signature component set to an empty string.
3. “Canonicalize” the headers that we are going to sign. By “canonicalize”, we mean “standardize capitalization, whitespace, and newlines into a format required by the spec, since other mail transport agents who get their grubby paws on this message might reformat the headers”.
4. Slap our unsigned version of the DKIM-Signature header to the end of our “canonicalized” headers, sign that data, and slap the resulting signature to the end of the DKIM-Signature header.
5. Write this signed DKIM-Signature into the headers of the mail message, and send it on its merry way.

Divide and conquer!

Implementing the Sign() method

Our implementation for the Sign() method will tackle each step in turn:

public void Sign(Stream inputStream, Stream outputStream)
{
    if (this.disposed)
    {
        throw new ObjectDisposedException("DomainKeysSigner");
    }
 
    if (inputStream == null)
    {
        throw new ArgumentNullException("inputStream");
    }
 
    if (outputStream == null)
    {
        throw new ArgumentNullException("outputStream");
    }
 
    var bodyHash = this.GetBodyHash(inputStream);
    var unsignedDkimHeader = this.GetUnsignedDkimHeader(bodyHash);
    var canonicalizedHeaders = this.GetCanonicalizedHeaders(inputStream);
    var signedDkimHeader = this.GetSignedDkimHeader(unsignedDkimHeader, canonicalizedHeaders);
 
    WriteSignedMimeMessage(inputStream, outputStream, signedDkimHeader);
}

Computing the body hash

The first step, computing the hash of the body, is actually pretty easy. There is only one quirk in that DKIM spec says that if the body ends with multiple empty lines, then the body should be normalized to just one terminating newline for the purposes of computing the hash. The code is not exciting, and you can download it at the end of this article.

Creating the “unsigned” header

The next step is to create the “unsigned” DKIM-Signature header. This is where the DKIM spec is just weird. The DKIM-Signature header contains a lot of information in it, such as the selector, domain, and the hashing algorithm (SHA1 or SHA256) being used. Since that information is vital to ensuring the integrity of the signature, it’s important that that information be a part of the DKIM signature.

If I were designing this, I would append two headers to e-mail messages: a DKIM-Information header that contained all of the above information and is part of the data that is signed and a DKIM-Signature header that contains just the signature data. But the DKIM spec makes use only the one DKIM-Signature header, and for the purposes of signing, we treat the “signature part” of the header (b=) as an empty string:

private string GetUnsignedDkimHeader(string bodyHash)
{
    return string.Format(
        CultureInfo.InvariantCulture,
        "DKIM-Signature: v=1; a={0}; s={1}; d={2}; c=simple/simple; q=dns/txt; h={3}; bh={4}; b=;",
        this.hashAlgorithmDkimCode,
        this.selector,
        this.domain,
        string.Join(" : ", this.eligibleHeaders.OrderBy(x => x, StringComparer.Ordinal).ToArray()),
        bodyHash);
}

You can see here that I’ve got some instance variables that were set in our IDkimSigner implementation’s constructor, such as the hash algorithm to use, the selector, domain, headers to include in the signature, and so on. We also insert our recently-computed hash of the body here.

You can also see that I’m using “simple” body canonicalization and “simple” header canonicalization. The DKIM spec gives us a few options in determining how the message is represented for signing and verification purposes. For the “simple” body canonicalization, it means “exactly as written, except for the weird rule about multiple newlines at the end of the body”. For the “simple” header canonicalization, it means “exactly as written, whitespace, newlines, and everything”.

There is a “relaxed” canonicalization method, but it’s more work, since you have to munge the headers and body into a very particular format, and I didn’t feel like writing a MIME parser.

Extracting “canonicalized” headers

The third step is to get a list of the canonicalized headers. In the constructor, I accept a list of headers to sign: From, To, Message-ID, and so on. (From is always required to be signed.) Then I use parsing code similar to that used in the CanSign() method and build a list of of the raw headers. The only real gotcha to watch out for is that headers can be wrapped onto more than one line, and since we’re using the “simple” canonicalization algorithm, we’ll need to preserve those whitespaces and newlines exactly as we extract them from the stream. Then I sort the headers alphabetically, since that’s how I specified them in the GetUnsignedDkimHeader() method specified above.

Signing the message

The logic behind signing the message is not that difficult. We smash all of the canonicalized headers together, add our unsigned DKIM-Signature header to the end, and compute our signature on this. Then we append the signature to the b= element, previously empty, of our DKIM-Signature header:

private string GetSignedDkimHeader(
    string unsignedDkimHeader, 
    IEnumerable<string> canonicalizedHeaders)
{
    byte[] signatureBytes;
    string signatureText;
    StringBuilder signedDkimHeader;
 
    using (var stream = new MemoryStream())
    {
        using (var writer = new StreamWriter(stream))
        {
            foreach (var canonicalizedHeader in canonicalizedHeaders)
            {
                writer.Write(canonicalizedHeader);
            }
 
            writer.Write(unsignedDkimHeader);
            writer.Flush();
 
            stream.Seek(0, SeekOrigin.Begin);
 
            signatureBytes = this.cryptoProvider.SignData(stream, this.hashAlgorithmCryptoCode);
        }
    }
 
    signatureText = Convert.ToBase64String(signatureBytes);
    signedDkimHeader = new StringBuilder(unsignedDkimHeader.Substring(0, unsignedDkimHeader.Length - 1));
 
    signedDkimHeader.Append(signatureText);
    signedDkimHeader.Append(";\r\n");
 
    return signedDkimHeader.ToString();
}

The only gotcha here, which I lost a few hours to, is a weird quirk of the .NET Framework 3.5 implementation of the SignData() function of the RSACryptoServiceProvider class. One of the overloads of the SignData() function accepts an instance of a HashAlgorithm to specify the kind of hash to use. The SHA-256 implementation was added in .NET 3.5 SP1, but it was done in such a way that an internal switch statement used internally by the .NET crypto classes wasn’t updated until .NET 4.0 to recognize the new SHA256CryptoServiceProvider type. Some guy blogs about why this is, but what it essentially means is that if you pass a SHA256CryptoServiceProvider instance to the SignData() method on .NET 2.0/3.0/3.5/3.5SP1, you get an exception, and on .NET 4.0 you don’t. Since Exchange 2007 uses .NET 3.5 SP1, we have to use the recommended workaround of using the overload that accepts a string representation of the hash algorithm.

Writing out the message

The last step is to write out the message with our newly created DKIM-Signature header. This really is a simple as taking the output stream, writing the DKIM-Signature header, and then dumping in the entire contents of the input stream.

Getting a key to sign messages with

Let us take a brief interlude from our DKIM circles of hell and obtain a key with which we will actually sign the DKIM-Signature header we’ve worked so hard to create.

We need to generate an RSA public/private key pair: a public key to store in DNS in the format required by the DKIM spec, and a private key to actually sign the messages with. The nice folks over at Port25 have a DKIM wizard that does exactly that.

It’s smashingly simple–just enter your domain name (say, “example.com“), a “selector” (say, “key1“), and select a key size (bigger is better, right?). The “selector” is a part of the DKIM spec that allows a single signing domain to use multiple keys. For example, you could use a key with selector name “newsletters” to sign all of the crap newsletter e-mails that you send out, and another key with selector name “tx” to sign all of the transactional e-mails that you send out.

It then spits out the syntax of the TXT records that you need to add to DNS for that selector:

key1._domainkey.example.com IN  TXT     "k=rsa\; p={BIG HONKING PUBLIC KEY HERE}"
_domainkey.example.com      IN  TXT     "t=y; o=~;"

The first record is where the public part of the key is stored. Whenever a mail transport agent sees one of our DKIM-Signature headers with a selector of key1, it’ll know to go hunting in DNS for a TXT record named key1._domainkey.example.com and pull the public key for verification from there.

The second record is part of the older DomainKeys specification and it is not strictly necessary. As written here, it means that we’re in testing mode (“t=y”)–that is, don’t freak out if you see a bad signature because we’re still dicking around with the setup of our implementation–and that not all messages originating from this domain will be signed (“o=~”)–maybe we won’t bother signing our newsletter e-mails, for example.

We’ll also have the private key specified in a format similar to the one below:

-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBANXBbZybdmjKDTONFVqAWXmGzR6GSZX5LV3OF//1jRz7dzGWTCKK
jembqBxqhr0Y2ua2l4D4EZi6FwDmdqgLS6MCAwEAAQJAD4qhypovEM1oClB+tfbR
Cpn3ffmrjgDxAHoEmrKi0PGBn8fumW22bad2tmrAjWWTVmeXJvQyEy1awq0M2PMR
0QIhAPEnqivb5dKZbTeKhiF4c6IUHfwEq8wNf2LWZvdH3ROrAiEA4un604mDss4Q
qAVEx686pUttfWyJrYkcZ/tx7kOoL+kCICEysqyDAypw0KY6vahR6qk/V7lf8z6O
BSFYHqigDgEtAiEAsK9r5UcQSyv1AD+J/MpOqeJ/kMfwtDUs7zJ01gfMb/ECIQDg
8d/XVJDi4Cqbt4wfcHZxADAgqyK8Z5M69fBecnExVg==
-----END RSA PRIVATE KEY-----

One thing I have glossed over in the code discussion until now was how that this.cryptoProvider instance that actually computes the signature got created.

We’ll need to read this key and load it into the cryptography classes used by the .NET Framework and by Windows to actually sign mail messages and get that this.cryptoProvider instance. Surely there is a simple API for this, yes?

Instancing a CryptoProvider

One problem is that the documentation in MSDN for the CryptoAPI is bad. I say “bad” because it certainly seems like .NET and Windows don’t expose native support for processing a PEM-encoded key, and if it does, well, I couldn’t find the documentation for it. Instead, the RSACryptoServiceProvider prefers to store its keys in an XML format that nothing else in the world seems to use.

This means that our implementation is so close to being finished that we can almost taste it, but now we have to complete a side quest to actually read our damn key and get an instance of the RSACryptoServiceProvider. Or, we could generate a certificate ourselves and store it in the Certificates MMC snap-in, but why should we have to do that? I’d rather just plop the damn key in the application configuration file like the rest of the goddamned world does it, “secure container” my ass.

We can thank the moon and the stars that some guy has written a PEM reader for us. How does it work? I have no idea, but I tested it on several keys and it seemed to work fine, which is good enough for me. I tossed this code into a static CryptHelper class, and now getting an instance of the RSACryptoServiceProvider is as simple as

this.cryptoProvider = CryptHelper
     .GetProviderFromPemEncodedRsaPrivateKey(encodedKey);

Loading the routing agent into Exchange Server 2007

I took all of this code and then added boring administrative stuff like logging and moving some hardcoded values (such as the PEM-encoded key, the selector, and the domain) into the usual .NET App.config file mechanism.

Installing the agent on the Exchange Server is surprisingly simple. After compiling the project and futzing with the configuration file, we just copy the DLLs and configuration file to a folder on the Exchange Server, say, C:\Program Files\Skiviez\Wolverine\DKIM Signer for Exchange\.

Then we launch the Exchange Management Shell (remember to right-click it and “Run as Administrator”) and execute a command to tell Exchange to actually register our agent:

Install-TransportAgent 
     -Name "DKIM Signer for Exchange" 
     -TransportAgentFactory "Skiviez.Wolverine.Exchange.DkimSigner.DkimSigningRoutingAgentFactory" 
     -AssemblyPath "C:\Program Files\Skiviez\Wolverine\DKIM Signer for Exchange\Skiviez.Wolverine.Exchange.DkimSigner.dll"

followed by

Enable-TransportAgent -Name "DKIM Signer for Exchange"

Interestingly, there will be a note telling you to close the Powershell window. It is not kidding. For some reason, the Install-TransportAgent cmdlet will keep a file handle open on our DLL, preventing Exchange from actually loading it until we close the Powershell window.

To make it actually work, we need to restart the Microsoft Exchange Transport service. I’ve found that restarting the Microsoft Exchange Mail Submission right after that is a good idea; otherwise, there can be a short delay of about 15 minutes before people’s Outlooks attempt to send outbound mail again.

Testing the implementation

To make sure things are actually working, Port25 comes to the rescue with their verification tool. You just send an e-mail to check-auth@verifier.port25.com and within a few minutes, they’ll send you an e-mail back with a boatload of debugging information. If it’s all good, you’ll see a result like the following:

Summary of Results
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

(What you’re looking for is the “pass” next to the DKIM check. The DomainKeys part being neutral is OK, since DomainKeys is the older standard and we’re choosing not to implement it.)

Conclusions and Delusions

I’ve been using this code for a few weeks now and it seems to work fine–the messages that I’ve sent through the server to Port25 and my Yahoo! test account all end up showing the DKIM passing. The usual “it works on my machine!” disclaimers apply, however, as I’m sure there are myriad configuration differences in Exchange that could this not to work. Bug fixes are welcome, but don’t come crying if it sends all of your e-mail to that big junk folder in the sky.

And thanks to some blowhards at Yahoo!, the world now has a public domain implementation of DKIM signing for Exchange to play with.

And in case you’re curious–after doing all this work to set up DKIM and participate in the Complaint Feedback Loop at their suggestion–their answer is still “no,” without elaboration. When Yahoo! finally goes under, I won’t be one shedding a nostalgic tear.

There are some unit tests, but they do have our private key in them, and I couldn’t be bothered to siphon those out. The code below is just the bits that do the actual signing.

Download the code used in this article.

For a long time, Skiviez dealt with this problem in a way that I would say the vast majority of US-based e-commerce based retailers dealt with it: they added a country dropdown, turned the State field into a textbox, and made the State field no longer required. And if you are not obsessive compulsive like I am, then this works reasonably well enough: international customers have to shoehorn their address into the format presented, but the data can still get in there.

However, after 10 years in operation, this approach was beginning to show some deficiencies:

Magnetic attraction. There is, for lack of a better term, a magnetic attraction between users and text boxes. Even if the field is not required, users have a tendency to enter “N/A”, “.”, “?”, “none”, or some other placeholder value in fields that do not apply to them.

Address quality problems. With the above method, the bumpers are off–a user could type “cheese” into the postal code field, and we’d just have to accept it on their word that “cheese” is a valid postal code for the United Kingdom because the site wasn’t doing any sort of validation except for United States-based addresses. Shipping a parcel to another country only to have it returned or destroyed as a result of an invalid or insufficient address is an expensive issue.

Integration issues. To actually ship out international packages, we have to do deal with third-party carrier APIs (such as Endicia, FedEx, and UPS) to purchase postage and print shipping labels. Some of these APIs do perform more advanced validation on international addresses. This could cause problems because a poorly formatted address would make it nearly all of the way through our fulfillment process–order acceptance, picking, processing, and packaging–only to blow up with an error at the shipping desk. It’d be nice if we could catch problems earlier to avoid disrupting the shipping flow and eliminate the potential for misrouting the package.

A smarter system

With the October 2010 release of the Skiviez Web site (which was a pretty huge update), the Web site is now much, much smarter in how it asks for address information. For example, here’s the default address form for the United States, which is about what you’d expect:

But if you choose, say, Ireland in the countries drop down, the page refreshes and displays a new form:

The state drop down is gone (since Ireland doesn’t have states), the ZIP/postal code text box is gone (since Ireland does not use postal codes), and a new County text box is available.

Once an address is entered, it is formatted in the way a person native to that country would expect. For example, in a German address, the postal code comes before the city:

Taking it even further, if you select, say, Finland from the drop down, you get a form back that looks like this:

The postal code validates against the postal code format used in Finland, and that field appears before the Municipality field, since the convention throughout most of the EU is to write “{Postal Code} {City}” instead of the American “{City} {State} {Postal Code}” way.

Clearly, we’ve made a lot of improvements to address handling at Skiviez, and this simple input format actually has a lot of complexity behind it. So how does it all work?

Tweaking the existing database schema

At Skiviez, we were clearly working from a legacy database schema. The customer addresses table has been sitting there for 10 years, and we’re not about to change it any time soon. And while there is some internal cruft, the fields in the database more or less looked like the following:

This is a normal, relational schema. So how do we enhance this table so that it can store addresses from the dozens of countries that we ship to? That sounds like a tall order: surely the addresses of the world are all wildly different from one another, and we’ll have to go entity-attribute-value, throwing away the relational bit of our relational data store, to be able to accurately store this diverse information, right?

Well, it turns out that you can take these existing fields, relax away any NOT NULL constraints, and rename them to field names that are operating at a higher level of abstraction. (We didn’t actually rename the fields in the database, as this would have broken old code, but the new field names do exist in the object model, described later.)

In other words, you can shoehorn most of the addresses in the world (most importantly, the ones that Skiviez ships to), by just thinking of the existing fields in these terms:

Nothing has really changed. We’re just making the concession that it’s OK that the meaning of the data within these fields will change slightly depending on the country we’re talking about. In the United States, the “AdministrativeArea” is the state, territory, or military post office. In Ireland, the “AdministrativeArea” is the county. And in Finland, the “AdministrativeArea” is NULL because it simply isn’t used at all.

Creating a sane object model

There is something wrong with the database schema, though, and for the sake of preserving legacy code and data, I worked around it in the new object model that is persisted to this schema. The problem is as follows:

A mailing address is not the same as a contact.

Frequently while coding over the years, I found myself wanting to deal with a physical address when I didn’t have any related contact information. That is, I had a group of data for 123 Main St; Richmond, VA 23221; United States, but I didn’t know who lived there or what the phone number was. The only object I had in my model was an Address object that was a 1-1 mapping with the old database schema, so this meant having Address instances with a null name and a null phone number.

A more natural model is to split the fields into three objects and create a fourth object that is composed of all of them:

So instead of one big honking Address object, we have

• an Address object that just contains physical address information;
• a Country object (not shown in the above diagram) that contains information necessary to format and validate an address (more on this later);
• a PersonalName object that binds all of the name fields together;
• a Contact object that groups together a PersonalName, an optional company name string, and a PhoneNumberBase object; and
• a MailingAddress object that pairs a Contact with an Address.

By splitting the old Address object into these individual components, I can now freely re-use these components elsewhere in the application. For example, the PersonalName object also gets used in the Customer object. The return form generator expects a MailingAddress object, but a shipping rate calculator can deal with an Address object directly, since the latter has no need for the Contact information. In other words, the object model went from coarse-grained to fine-grained.

But perhaps the most important aspect of this model is that each of the four above classes is immutable.

Why are immutable addressing objects better?

When an object is immutable, it can’t be altered after it is first instanced. In other words, everything about the object is permanently set at construction time; its instance methods won’t mutate the object, and its properties don’t have public setters. In domain driven design, this is often referred to as a value type, but I don’t like that terminology as it can cause some confusion with value-vs-reference types in the .NET Framework.

This means that if I have an instance of an Address and want to edit the PostalCode property, then I will need to create a brand new instance of an Address, copy most of the properties over, and set my new postal code at construction time.

There are a few advantages to this approach:

• The instance can validate itself once at construction time. This can be a big win if validation is particularly complex and dependent on the values of other properties (which is exactly what happens in an address, since the country determines whether or not a field is required and what format it needs to be in). If anything else is referencing that Address instance, then it can assume that that instance won’t get put into an invalid state. For example, a SalesOrder has a ShippingAddress property that takes a MailingAddress instance. The SalesOrder doesn’t have to worry about someone coming along and saying ShippingAddress.PostalCode = null at some point, bypassing any validation checks. The whole ShippingAddress instance would have to be replaced, which means building a new Address instance, which means running the validation logic built into the address builder.
• The hash code of the instance is consistent throughout the object’s lifetime. This is useful for modeling an address property as a component in NHibernate, which I won’t get into here.

But a big disadvantage to this approach is that it can be tedious to deal with in code. At face level, this means that our Address object would need a gigantic constructor or support an IFreezable-type of implementation. A cleaner way is to use the builder pattern to allow us to set the fields in any order and validate the Address in one shot:

var country = CountryMother.UnitedStates();
 
return new Address.Builder()
    .WithPrimaryStreetLine("3005 SOME ST")
    .WithMunicipality("RICHMOND")
    .WithAdministrativeArea("VIRGINIA")
    .WithPostalCode("23221")
    .WithCountry(country)
    .Build(); // exception thrown here if validation errors

If we want to get a list of detailed validation problems instead of throwing an exception, the builder provides a mechanism for that:

var country = CountryMother.UnitedStates();
var builder = new Address.Builder()
    .WithPrimaryStreetLine("3005 SOME ST")
    .WithMunicipality("RICHMOND")
    .WithAdministrativeArea("VIRGINIA")
    .WithPostalCode("23221")
    .WithCountry(country);
var results = builder.Validate();
 
if (results.Any(x => !x.Success))
{
    foreach (var error in results.Where(x => !x.Success))
    {
        Console.WriteLine(
            "Field {0}, Error {1}", 
            error.Field, 
            error.Message);
    }
}
else
{
    var instance = builder.Build();
}

This allows the user interface to be able to get to detailed error messages pretty easily. But how does that call to Validate() work? How does the builder really know what format an address should be in?

Validation

We need to define a way for a Country to know how its mailing addresses need to be formatted. Since we have all countries of the world in our database, but we don’t ship to all countries in the world, I created a nullable property on the Country object called Scheme. The Scheme is an instance of the AddressScheme class and it’s persisted to the database along with the country.

If we try to build an address without a country specified or with a country that doesn’t have an AddressScheme, then we throw an exception saying that we just can’t do that. Otherwise, our Validate() method will run through a list of the AddressFieldRules that are contained within the AddressScheme instance, calling the Validate(string) method on the rule for a particular field.

To be able to easily persist these rules in the database, I created an enum that parallels the generic properties of the Address class that we defined earlier:

public enum AddressField : int
{
    Unknown = 0,
    AdministrativeArea = 1,
    Country = 2,
    Municipality = 3,
    PostalCode = 4,
    PrimaryStreetLine = 5,
    SecondaryStreetLine = 6,
    SubAdministrativeArea = 7
}

Internally, the Address object maintains its state via a Hashtable of fields, where the key is the AddressField enumerated value and the value is an object, usually string but sometimes a Country. The properties of the Address object are really just strongly-typed convenience mappings onto that Hashtable as in the following example:

public virtual string AdministrativeArea
{
	get
	{
		return this[AddressField.AdministrativeArea] as string;
	}
 
	private set
	{
		this[AddressField.AdministrativeArea] = value;
	}
}

The implementation of the AddressFieldRule is fairly straightforward. I provide the capability to validate against a regular expression (such as might happen with a postal code field), to simply consider the field required (such as might happen with a city field), and to ensure that the value is a member of a list of a predefined list of values (such as might happen with a province field). The Validate(string) method figures out which rules to apply based on the properties that are set and returns an AddressFieldRuleValidationResult that describes what went right and what went wrong.

/// <summary>
/// Describes a requirement for an address field for a particular address.
/// </summary>
public class AddressFieldRule
{
    /// <summary>
    /// The list of allowed values; this may be null if no particular set of
    /// values is required.
    /// </summary>
    private IEnumerable<AddressFieldAllowedValue> allowedValues;
 
    /// <summary>
    /// The description of the field and its format.
    /// </summary>
    private string description;
 
    /// <summary>
    /// The address field.
    /// </summary>
    private AddressField field;
 
    /// <summary>
    /// Whether or not the field is required.
    /// </summary>
    private bool isRequired;
 
    /// <summary>
    /// The localized name.
    /// </summary>
    private string name;
 
    /// <summary>
    /// An optional validation regular expression template.
    /// </summary>
    private string validationRegex;
 
    /// <summary>
    /// Initializes a new instance of the <see cref="AddressFieldRule"/> class.
    /// </summary>
    /// <param name="field">The field.</param>
    /// <param name="isRequired">if set to <c>true</c> [is required].</param>
    /// <param name="name">English, localized name for the field. Required.</param>
    /// <param name="description">The english description.</param>
    /// <param name="validationRegex">The validation regex. Optional.</param>
    /// <param name="allowedValues">The allowed values. Optional; if null, then
    /// no particular set of values is required.</param>
    /// <exception cref="T:System.ArgumentNullException">if the englishName is null
    /// or empty</exception>
    /// <exception cref="T:System.ArgumentOutOfRangeException">if the address field
    /// is not in a valid range</exception>
    public AddressFieldRule(
        AddressField field,
        bool isRequired,
        string name,
        string description,
        string validationRegex,
        IEnumerable<AddressFieldAllowedValue> allowedValues)
    {
        if (string.IsNullOrEmpty(name))
        {
            throw new ArgumentNullException("name");
        }
 
        if (!Enum.IsDefined(typeof(AddressField), field))
        {
            throw new ArgumentOutOfRangeException("field");
        }
 
        if (!string.IsNullOrEmpty(validationRegex) &&
            allowedValues != null &&
            allowedValues.Count() > 0)
        {
            throw new ArgumentException(
                "The ValidationRegex and the AllowedValues options are mutually exclusive.");
        }
 
        this.field = field;
        this.description = description;
        this.isRequired = isRequired;
        this.name = name;
        this.allowedValues = allowedValues ?? new List<AddressFieldAllowedValue>();
        this.validationRegex = validationRegex;
    }
 
    /// <summary>
    /// Initializes a new instance of the <see cref="AddressFieldRule"/> class.
    /// </summary>
    protected AddressFieldRule()
    {
    }
 
    /// <summary>
    /// Gets values that, if not empty, then the address field value 
    /// should be one of the membersin the First property of each of these 
    /// Pairs. The Second property, if
    /// set, is an alternative name that could be displayed in a dropdown to
    /// the user.
    /// </summary>
    public virtual IEnumerable<AddressFieldAllowedValue> AllowedValues
    {
        get
        {
            return this.allowedValues;
        }
 
        private set
        {
            this.allowedValues = value;
        }
    }
 
    /// <summary>
    /// Gets a human-readable string that describes the required format for the
    /// field. It may give an example, such as "JYX 938".
    /// </summary>
    public virtual string Description
    {
        get { return this.description; }
        private set { this.description = value; }
    }
 
    /// <summary>
    /// Gets the field that this metadata describes.
    /// </summary>
    public virtual AddressField Field
    {
        get { return this.field; }
        private set { this.field = value; }
    }
 
    /// <summary>
    /// Gets a value indicating whether or not the field must be 
    /// included in the address.
    /// </summary>
    public virtual bool IsRequired
    {
        get
        {
            return this.isRequired;
        }
 
        private set
        {
            this.isRequired = value;
        }
    }
 
    /// <summary>
    /// Gets an English, localized name for the field. For example, the
    /// StateOrProvince field has a name of "Province or Territory" for
    /// Canadian addresses.
    /// </summary>
    public virtual string Name
    {
        get { return this.name; }
        private set { this.name = value; }
    }
 
    /// <summary>
    /// Gets an optional validation regular expression template, which may
    /// be null.
    /// </summary>
    public virtual string ValidationRegex
    {
        get
        {
            return this.validationRegex;
        }
 
        private set
        {
            this.validationRegex = value;
        }
    }
 
    /// <summary>
    /// Gets or sets the data store identifier for the rule.
    /// </summary>
    private int? Identity
    {
        get;
        set;
    }
 
    /// <summary>
    /// Validates the value.
    /// </summary>
    /// <param name="value">The value to validate.</param>
    /// <returns>The validation result.</returns>
    public virtual AddressFieldRuleValidationResult Validate(string value)
    {
        if ((value == null || value.Trim().Length == 0) && this.IsRequired)
        {
            var message = string.Format(
                CultureInfo.CurrentCulture,
                "The field '{0}' is required, but the value was empty or blank.",
                this.Name);
 
            return new AddressFieldRuleValidationResult()
            {
                Field = this.Field,
                Original = value,
                Message = message
            };
        }
 
        if (value != null)
        {
            var input = value.Trim();
 
            if (input.Length > 0)
            {
                if (!string.IsNullOrEmpty(this.ValidationRegex) &&
                    !Regex.IsMatch(input, this.ValidationRegex))
                {
                    var message = string.Format(
                        CultureInfo.CurrentCulture,
                        "The field '{0}' has an unexpected or incorrectly formatted value.",
                        this.Name);
 
                    return new AddressFieldRuleValidationResult()
                    {
                        Field = this.Field,
                        Original = value,
                        Message = message
                    };
                }
 
                if (this.AllowedValues.Count() > 0)
                {
                    string preferred;
 
                    // If an alternative value is detected, select the preferred.
                    var alternativeAllowedValue =
                        (from av in this.AllowedValues
                            where !string.IsNullOrEmpty(av.Alternative) &&
                            av.Alternative.Equals(input, StringComparison.OrdinalIgnoreCase)
                            select av).SingleOrDefault();
                    if (alternativeAllowedValue != null)
                    {
                        // We matched an alternative value, so let's use the
                        // corresponding primary value
                        preferred = alternativeAllowedValue.Preferred;
 
                        return new AddressFieldRuleValidationResult()
                        {
                            Field = this.Field,
                            Original = value,
                            Preferred = preferred,
                            Success = true
                        };
                    }
                    else
                    {
                        // Search on the primary value, then.
                        var primaryAllowedValue =
                            (from av in this.AllowedValues
                                where av.Preferred.Equals(input, StringComparison.OrdinalIgnoreCase)
                                select av).SingleOrDefault();
                        if (primaryAllowedValue != null)
                        {
                            // We matched the primary value, let's re-assign
                            // it to get the correct casing
                            preferred = primaryAllowedValue.Preferred;
 
                            return new AddressFieldRuleValidationResult()
                            {
                                Field = this.Field,
                                Original = value,
                                Preferred = preferred,
                                Success = true
                            };
                        }
                        else
                        {
                            var message = string.Format(
                                CultureInfo.CurrentCulture,
                                "The field '{0}' was not in the list of expected values.",
                                this.Name);
 
                            return new AddressFieldRuleValidationResult()
                            {
                                Field = this.Field,
                                Original = value,
                                Message = message
                            };
                        }
                    }
                }
            }
        }
 
        return new AddressFieldRuleValidationResult()
        {
            Field = this.Field,
            Original = value,
            Preferred = value,
            Success = true
        };
    }
}

(In the case of a state or province field, where we ensure that the value belongs to a predefined list of AddressFieldAllowedValues, we also let the rule tell the consumer which value is “preferred.” For example, a user might enter “Virginia”, but the system will spit back “VA” as the preferred formatting for that value.)

As a result, that Validate() method on the builder doesn’t have to do much heavy lifting:

public IEnumerable<AddressFieldRuleValidationResult> Validate()
{
    var failures = new List<AddressFieldRuleValidationResult>();
 
    if (this.address.Country == null)
    {
        var result = new AddressFieldRuleValidationResult()
        {
            Field = AddressField.Country,
            Message = "The country is required."
        };
 
        failures.Add(result);
    }
    else if (this.address.Country.AddressScheme == null)
    {
        var result = new AddressFieldRuleValidationResult()
        {
            Field = AddressField.Country,
            Message = "The country must have an address scheme."
        };
 
        failures.Add(result);
    }
    else
    {
        var scheme = this.address.Country.AddressScheme;
 
        foreach (AddressField field in new ArrayList(this.address.fields.Keys))
        {
            if (field != AddressField.Country &&
                field != AddressField.Unknown)
            {
                // If the field is not Country or Unknown, let's
                // see if the scheme has a rule defined for it.
                if (scheme.HasRuleFor(field))
                {
                    // Since a rule is defined, let's validate the field.
                    var result = scheme.ValidateField(field, this.address[field] as string);
                    if (result.Success)
                    {
                        this.address[field] = result.Preferred;
                    }
                    else
                    {
                        failures.Add(result);
                    }
                }
                else
                {
                    // No rule is defined, so strip the data out of the
                    // address.
                    this.address[field] = null;
                }
            }
        }
    }
 
    return failures;
}

Where do all of these address rules come from? There is nothing more valuable than Frank’s Compulsive Guide to Postal Addresses.

Formatting

Another important property on the AddressScheme is the Formatter. This is an object that knows how to format an address for a particular country. The ToString() implementation on the Address can then just call this.Country.Formatter.Format("M", this) to get a string back that contains the correctly formatted address, where “M” is our IFormattable code for an address format that includes newlines.

My AddressFormatter object works by taking a template string that is loaded from the database. The template format is pretty simple: it’s a combination of the AddressField enumerated values (delimited by braces) and the semicolon (which indicates separate lines of the address). For example, the template for a German address is specified as:

{PrimaryStreetLine};{SecondaryStreetLine};DE-{PostalCode}  {Municipality};{Country}

Given an “S” format code (for “single-line output”), we’ll substitute the values and leave the semicolons. Given an “M” format code (for “mailing output”), we’ll substitute the values and swap the semicolons with newlines. We also remove any contiguous separators that might happen, such as when the SecondaryStreetLine is blank, to avoid having a blank line in the address.

So given an address that was built as

var country = CountryMother.Germany();
 
return new Address.Builder()
    .WithPrimaryStreetLine("42 Kaiserstrasse")
    .WithMunicipality("Berlin")
    .WithPostalCode("12345")
    .WithCountry(country)
    .Build();

our formatter, given a format code of “M”, will spit out

42 Kaiserstrasse
DE-12345 Berlin
Germany

Since the address object is immutable, we know it’s valid, so the formatter doesn’t have to worry about handling a potentially invalid address object.

The user interface of the Web site also uses the template to figure what input fields to display and what order to display them in.

Determining address kind

When shipping, it’s often important to know the “kind” of an address, as in the following enumeration demonstrates:

public enum AddressKind : int
{
    Unknown = 0,
    Commercial = 1,
    PostOfficeBox = 2,
    Residential = 3,
    Military = 4
}

Why is this important? For example, you can only use USPS to ship to a PO box or a military (APO/FPO/DPO) address, and you can’t use FedEx Home Service on a commercial address.

One mistake that I made over the years was trying to store the AddressKind right in the Address object. This was a mistake for a few reasons:

• I don’t always care about the address kind, such as for an international address, so having it appear on every Address instance is wasteful.
• Computing the address kind can be expensive. While figuring out a PO box or a military address just involves scanning the address fields, determining commercial or residential status means calling a Web service supplied by FedEx, which can take seconds to complete. By having the Kind property on the address, I’m left with a bad situation: I can either keep the immutability of my address object by determining the address kind for every address when I instantiate them, or I can break immutability, which negates the advantage of not having to worry about change tracking on these objects.

As a result, our new Web site simply doesn’t store the address kind at all. The kind, if needed, is determined on the fly by calling the DetermineAddressKind() method of an IAddressKindDeterminer instance.

Being flexible: Legacy data and third-party integration points

Finally, there’s one last point to be made. This new address validation and formatting is great and all, but there are two potential problems:

• There is 10 years of less-than-satisfactory address data already in the database, and it could be in an “invalid” format according to these new guidelines. Obviously, we want to maintain backward compatibility here, as we don’t want to force our customers to re-type their address data, and we don’t want to blow up when simply trying to load the address from the database.
• Addresses sometimes get force-fed to the system via third-party integrations such as PayPal Express Checkout or Google Checkout. I can’t really reject addresses coming from these sources, so I have to be able to “take it like it is,” even if it means it doesn’t quite adhere to my preferred format.

As a result, the new address system has to be able to grandfather in old addresses and graciously accept other addresses from integration. For accepting addresses from integration, I simply added an overload to the address builder’s Build() method that does not perform any validation–and trust myself only to use that overload in the proper scenarios.

For handling legacy address data, I simply let NHibernate set the address fields directly when loading objects from the database. That is, since the address isn’t constructed via the address builder but is instead built up via reflection, no validation is run on the legacy address and so the legacy address will still load.

One advantage of this model comes from the Address object’s immutability. Since the address can’t be changed, if the user wishes to edit an old address, they’ll be forced to bring the whole address up to snuff since a new Address instance will need to be created.

Conclusions and delusions

At the end of the day, we’re left with a robust implementation of address storage and address validation:

International address validation, input, and handling has never been better at Skiviez, and for the first time in years, I feel like it’s finally under control–thanks to a clean object design, immutability, and a few days of thinking really hard about the problem. I hope this post gives someone some insight on their own project out there. Good luck!

At Skiviez, we use two HandHeld Honeywell Dolphin D7600 Mobile Computers to pick orders. They’re devices with a barcode scanner, a touch screen, and WiFi capability. They run a Platform Builder variant of Windows CE 5.0. And in Windows XP, they used ActiveSync to connect to Windows and provide the basic service of being able to access the file system of the device (the other services provided by ActiveSync, like syncing mail, contacts, and calendars, don’t really make sense for an industrial device like this).

ActiveSync is a pretty wonky and ugly looking program, but it worked.

In Windows Vista and Windows 7, ActiveSync has been replaced–though the tradition of fragility and wonkiness continued–by an abomination called the Windows Mobile Device Center. (Indeed, it is telling that the new Windows Phone 7 drops ActiveSync/Windows Mobile Device Center completely and instead uses its own synchronization mechanism through the Zune software.) Which, like its predecessor, still worked for providing the basic service of accessing the file system and allowing Visual Studio to connect a debugger.

Except, one day, it stopped working. And I had no idea why.

Suddenly, plunking the device into its cradle would have the following behavior:

• The device would authenticate and think it is connected.
• The PC’s USB subsystem would recognize the device and think it is connected.
• Windows Mobile Device Center would permanently hang at the splash screen:

Which is just awesome.

If I re-launched Windows Mobile Device Center via the Start menu, the program would open, but it would insist that the device is “Not Connected.”

So, I began to troubleshoot.

• I looked in the Event Log. (My first mistake. I’m renaming the Event Log the “Red Herrings Viewer”.)
• I tried enabling verbose logging for Windows Mobile Device Center (which reports nothing useful).
• I tried uninstalling the the device and letting Windows reinstall its drivers.
• I tried uninstalling Windows Mobile Device Center and reinstalling it.
• I tried soft resetting the device.
• I tried hard resetting the device.
• I made sure that the “Windows Mobile *” firewall rules existed in Windows Firewall.
• I tried a different USB port.
• I tried a different USB cable.
• I tried a different D7600 device.
• I tried a different D7600 cradle.
• I tried merged the registry settings from another Windows 7 machine that the device would successfully connect to.
• I tried switching to the RNDIS connectivity model.
• I tried granting additional permissions on the “Windows CE Services” registry keys.
• I tried diddling with the various “Windows CE Services” registry keys.

You might think that anyone of these things would contain the solution. But you’d wrong.

The problem was that I had FileZilla FTP Server installed on my machine, configured to allow FTPS connections. (We use an FTP server to manage images and files on the Skiviez Web site, and I had a local copy on my machine from when I was testing the configuration.)

Now, some people might ask “Why the hell would an FTP server break Windows Mobile device connectivity?” Apparently, Windows Mobile Device Center uses port 990 to orchestrate the connection.

Port 990 just so happens to be the standard control port for FTPS connections. If anything else is consuming port 990, then the Windows Mobile Device Center either hang, reports that the device is not connected, or stupidly tries to keep connecting to it. (A message like “whoops port 990 is in use or does not seem to be a mobile device” would go a long way.)

So make sure nothing is using port 990; then go pour yourself a g&t.

One of these new features is a fast, faceted product catalog. Faceting is the technical name for something you’ve seen in many e-commerce stores like NewEgg, Zappo’s, and Lowe’s: a list of links that quickly allow you to drill down to the content that you’d like to see.

Trying to do this kind of thing efficiently in a traditional relational database is not going to happen unless you want to start adding and removing indexes on the fly or go full on Entity-Attribute-Value, which somewhat defeats the purpose of using a relational database. You lose efficient indexing, you lose type information, you lose, well, the relations.

Instead, Skiviez now uses Solr–an open source search platform that is based on Lucene, a text search engine library–for managing catalog listing pages and search pages.

Solr from 10,000 feet

Solr is not relational. It doesn’t have tables. The schema is flat; all “documents” in the Solr store share the same schema. From a relational perspective, you could think of it as one table with many columns, except that not every row in the table uses all of the columns. (This isn’t how it internally works at all, but it’s a useful metaphor.)

Some fields are “stored,” which means you can get back the data that you put in. Some fields are “indexed,” which means that you can query on them directly and use them for faceting. And some fields are “multivalued”, which means that they can hold more than one value at a time. For example, our <sizeInStock> field can hold S, M, and L values all at once for a particular product, and just L for another.

Why would you ever want a value to not be “stored”? Why add data that you can’t get back? Well, in Solr, storing and indexing are distinct concepts, so you can a value that is not stored, just indexed. A good example from the Skiviez Solr schema would be the salesVolume field, which indicates how many units of a particular item were sold in the past week:

		<field
			name="salesVolume"
			type="sint"
			indexed="true"
			stored="false" />

This is just an integer, a number of units sold. In my UI, I’m not going to ever actually display that number–that would reveal a little too much information about our business. But I still want our users to be able to sort catalog listings by “Top Selling” products. By creating an index, we enable sorting; by disallowing storage, we save some space.

The Skiviez Solr schema defines a “type” field, some fields that are shared among all document types, and a few type-specific fields. This works well because the only intended use of Solr is catalog listings and searching; if I had to store more disparate kinds of data, I might create several Solr schemas.

<!--
	This field is used to indicate the type of data that is stored within
	the document. Since the Skiviez Solr instance represents a variety of
	different types, this field helps to indicate what kind of data you are 
	looking at.
 
	Types that are currently understood the Web site include:
 
		BLOGARTICLE
		BRAND
		LINE
		PRODUCT
		PRODUCTSUMMARY
                PROMOTION
		STYLE
		WEBPAGE
-->
<field
	name="type"
	type="string"
	indexed="true"
	stored="true"
	required="true" />

Why is this useful? Well, the search portion of the Skiviez Web site can use the <type> field and the fields that are used by all of the different document types to quickly search a wide variety of sources:

In this case, we’re looking at our <type>, <urlKey>, <name>, and <image> fields. The Web site can use this to quickly sort information and build links to content.

The PRODUCT type uses a wealth of product-specific fields, however, and when we query to display catalog pages, we tell Solr to only return documents of type PRODUCT. I could have created a second Solr schema for this purpose, but why bother?

Getting data into Solr

The traditional SQL database is still the “authoritative” data store, and the Solr indexes are read-only snapshots of that data. This means that the data coming from Solr is always slightly stale, so I had to ask myself:

• How stale is too stale?
• When do I value speed or querying over staleness?

Part of the new Skiviez Web site is a Windows service that I call the “Worker.” It uses Quartz.NET to execute C# IJob implementations periodically. You can think of them as traditional scheduled tasks in Windows; the only difference is that I am explicitly managing them in code, using the same object-oriented model of our domain in those jobs, and, as long as the service is installed, I don’t need to worry about configuring scheduled tasks.

Every three hours, one of those IJobs that gets executed is the RefreshSolrIndexesJob, and all that job does is ping an HttpWebRequest over to http://solr.example.com/dataimport?command=full-import, where solr.example.com is placed with the FQDN of our internal Solr server. This is because I use Solr’s built-in DataImportHandler to actually suck in the data from the SQL database; the job just has to “touch” that URL periodically to make the sync work. Because the DataImportHandler commits the changes periodically, this is all effectively running in the background, transparent to the users of the Web site. And because the Skiviez product catalog is reasonably small (a few thousand items), we can blow away the whole Solr index and re-build it in fewer than two minutes.

There’s also a function in our backend application that allows employees to trigger the index rebuilding immediately; this can happen when new product arrives in the warehouse and we want to get it up on the Web site right away.

The DataImportHandler is built into Solr, and configuring it is a little confusing because it uses some strange terminology. It just takes an XML configuration file, and whenever you ping its request handler, it performs synchronization tasks based on what has been specified in its configuration.

      <entity
        name="brands"
        dataSource="undiesDatabase"
        transformer="TemplateTransformer"
        query="
            SELECT
                b.ID AS BrandId,
                b.[Name] AS BrandName,
                b.Description AS BrandDescription,
                b.UrlKey AS BrandUrlKey,
                CASE b.Active
                  WHEN 'Y' THEN 'True'
                  ELSE 'False'
                END AS BrandIsActive
            FROM na.brands AS b
            ORDER BY b.[Name];">
        <field column="id" template="BRAND-${brands.BrandId}" />
        <field column="type" template="BRAND" />
        <field name="identity" column="BrandId" />
        <field name="name" column="BrandName" />
        <field name="description" column="BrandDescription" />
        <field name="isActive" column="BrandIsActive" />
        <field name="urlKey" column="BrandUrlKey" />
      </entity>

I say the terminology is confusing because in <field> elements, the @column attribute is the name of the Solr field and the name of the column in the SQL result set. But if the @name attribute is specified, then the @column attribute is the name of the column in the result set and the @name attribute is the name of the Solr field. It’s confusing because some <field> elements don’t pull directly from the result set, instead relying on a “transformer.” You would expect to just specify a @name for those fields, but specifying just @column is correct. In the above example, the “Template Transformer” is used on the id field to format Solr identities in the format of BRAND-127, where 127 is the SQL database’s primary key for the brand.

(And yes, attributes can have newlines in them. I’m not sure how I went through years of programming without realizing this, but once I realized that I could do it, it made reading those SQL queries much easier! This example is by far the shortest query for pulling Solr data from the SQL database; the one for products is over 200 lines long.)

Querying data: how stale is too stale?

This does mean that information in the Skiviez product catalog can be up to three hours stale. A user might click a link for “Medium In Stock (3)” on the catalog page (since this kind of faceted data is generated by querying Solr) but then see on the product detail page that no mediums are in stock (since on this page, the quantity information is one of the few things not cached and queried directly against the database). This is annoying, but generally rare in our particular scenario (we are a reasonably small business and not that high traffic), and it will be fixed up in 3 hours anyway when we rebuild the whole index again from scratch, so I have accepted this as a reasonable trade-off.

(If I really wanted to solve this problem, there are a few approaches that I could take. I could use domain events to fire off partial updates to the Solr index whenever a Save/Update operation occurred on an Item or ItemGroup in the Skiviez domain model, or I could insert a record into a table named, say, dbo.IdentitiesOfStuffThatNeedsUpdatingInSolr, and have an IJob that reads that list and executes partial updates every minute. And even if I did these things, I’d still probably want to do a periodic “blow it all away and refresh” in case one of those partial updates failed in the background.)

As for querying this data from Solr, there are a few approaches that I could have taken. One is to hide the fact that Solr exists entirely via the methods of a repository-like class: the Get*() methods would access Solr, and the Create/Update/Delete methods would access the database. I didn’t like this approach because my Solr schema is already shamelessly tailored to the UI that will be accessing that data, as it should be–I’ve already made the decision to use Solr to provide easy faceting, sorting, and fast display of information, so I might as well use it to its fullest extent. This means making it explicit in code as to when I mean to access Solr and when I mean to access the up-to-date, non-cached database object.

In my case, I ended up using NHibernate to do the CRUD access (e.g., loading an ItemGroup, futzing with its pricing rules, and then saving it back), forgoing the repository pattern because I don’t typically see its value when NHibernate and its mappings are already abstracting the database. (Sometimes abstracting out NHibernate is useful, however, as we’ll see below.)

When querying for data, I know pretty well if I’m using it for catalog-oriented purposes (in which I care about speed and querying features) or for displaying in a table on a back-end administrative application (I care about currency). For querying on the Web site, I have an interface called IListingQuery. It has a Search() method that accepts a ListingRequest where I define some parameters–selected facets, search terms, page number, number of items per page, etc.–and gives back a ListingResponse–remaining facets, number of results, the results on this page, etc. This interface is pretty boring stuff.

Where it gets interesting is that the implementation of IListingQuery that gets dependency injected into my Web site’s ProductsController is using a list of IListingQueryStrategys underneath. The default strategy, the SolrListingQueryStrategy, hits Solr directly via a plain old-fashioned HttpWebRequest and parses the XML in the HttpWebResponse (which is much easier to use, in my humble opinion, than some of the Solr client libraries).

If the Solr-based strategy throws an exception or vomits for some reason, then the DatabaseListingQueryStrategy runs next and hits the database directly–although it ignores some parameters of the ListingRequest, like faceting or advanced text searching, since that is inefficient to do there and is the whole reason I am using Solr in the first place. The idea is that usually Solr is answering my search requests quickly in their full-featured glory, but if something blows up and Solr goes down, then the catalog pages of the site can still function in “reduced-functionality mode” by hitting the database with a limited feature set directly. (As implemented on the Skiviez site today, you would realize that this happened if the filters list on the left-hand side becomes empty and search parameters are ignored.)

(The explosion of classes that you see in the screenshot above is mostly due to testability. For example, Solr works by sending long, complicated GET requests to it. So I’ve pulled out of the URI-building functionality into its own class; it serves a single purpose and can be tested as such. The actual SolrListingQueryStrategy class implementation is very short, with most of the work delegated to other classes within that folder.)

Conclusions and delusions

What is important is that I have made explicit in code that this is a search–by using IListingQuery instead of NHibernate–so the database-based strategy can take some liberties in ignoring some of the search parameters without worrying about affecting some of its down-level callers too severely. The decision to perform a query against a possibly-stale data store versus the authoritative data store has been made explicit–if I want fast, possibly stale data with advanced search features, I use IListingQuery. If I want slow, up-to-date data with insert/update/delete capability, I use NHibernate’s named queries. And if I make a change in the SQL database, I know that the out-of-process Worker service will update Solr eventually, making things eventually consistent.

The end result? Fast catalog pages for our customers that gracefully fall back to the old behavior when something doesn’t work.

First, it was FedEx

It has been 45 days since I reported to FedEx that the new SmartPost integration offered by their Web services simply does not work when an EPL2 label is requested:

I Hate Software, Part 1

The problem is that the USPS Delivery Confirmation barcode does not print when an EPL2 label is requested. That’s because in the EPL2 document that FedEx sends back, quotation marks in the barcode command are not properly escaped:

A90,473,0,3,1,2,N,"ZIP - USPS DELIVERY CONFIRMATION e-VS"
B50,535,0,1,3,4,175,N,""F1"42023221"F1"0000000000000000000000"

That should read "\"F1\"420..." for it to print correctly. (The ZPLII version of the label works fine.)

I understand that larger corporations have fixed software release cycles and different bug triage mechanisms, but having to through three levels of support and waiting 45+ days to simply say “this simply doesn’t work please add a backslash” is somewhat frustrating.

Then, there was UPS

Similarly, I’ve discovered a recently-introduced error with the UPS Web services. If you request a 4″ x 6″ EPL shipping label, then the UPS Web service will happily ignore request and send back a 4″ x 8″ label instead. That’s because they’re sending back the wrong width and height settings in the label response:

q795
Q1600,24

That would be 795 dots / 203 dpi == 3.91″ wide (OK) and 1600 dots / 203 dpi == 7.88″ high (hmm, not what I asked for). (The ZPLII version of the label works fine.)

Finally, SmartFTP sent me over the edge

Since updating to the latest version of SmartFTP, I found myself frequently being unable to connect to the Skiviez private FTP server that is used to manage software updates, e-mails, and product catalog images. It would fail about 80% of the time with

[13:51:38] 234 Using authentication type TLS
[13:51:38] SSL: Error (Error=0x80090308).

I’m sure that this means something. And even if I knew what it was, sometimes it would just work without me changing anything (about 10 percent of the time). Further still, I hadn’t touched the FTP server for some time.

That’s when I noticed in the SmartFTP change log:

FTP: Completely rewrote SSL layer

Sigh. Downgraded to the version prior to that changelog entry, and it works fine.

Conclusions and delusions

I have certainly done my part to contribute buggy, crappy software to the world. I continue to spew out more buggy, crappy software with each passing day. But it is extra depressing and disheartening to know that I, some idiot working at a small company, can run across such simply-does-not-work bugs (and, in FedEx’s case, never-actually-worked-ever bugs) in software produced by large corporations and used by presumably hundreds to thousands of people around the world.

It would be nice if I could run an automated test that would “exercise” my NHibernate mapping file against an actual database to see if it all worked. It’s not really a unit test–I’m not testing NHibernate itself, I’m testing my configuration of NHIbernate–but it’s still appropriate as an integration test.

Ayende Rahien has a spiffy implementation of an in-memory database test that uses NHibernate’s SchemaExport to build up an empty, in-memory SQLite database at the start of each NUnit test. His implementation serves as the inspiration for my example.

In my case, I’m working on a little moonlighting project that uses some database-specific features of SQL Server Express 2008, in particular the GEOGRAPHY type and SPATIAL indexes. SQLite wouldn’t help me here–unless I just didn’t want to use it for those particular tests–but I figure that if I’m bothering to write an integration test, then it should actually test the integration against the database that I’m actually using. Additionally, unlike Ayende’s example, I’m using MSTest instead of nUnit because it’s built into Visual Studio and I am a masochist.

Understanding SQL Server Express User Instances

SQL Server Express has the concept of a “user instance” that allows a non-administrator user to attach and detach an *.mdf file to the local SQL Server Express instance. I thought this was what I needed. But there are some things to understand about user instances that confused me and probably confuse other people as well:

• They’re not the same thing as an in-memory database or even a “file-based” database like SQLite. It’s a regular SQL Server database file.
• The file has to exist. Unlike SQLite, if you tell SQL Server Express to attach to an *.mdf that isn’t there, you’ll get an error instead of having the database created for you just in time.
• To access the database, it has to be attached to a SQL Server instance. This means downloading SQL Server Express and installing the service on the machine. It’s not as if your application can just directly access the *.mdf file through some special DLL or anything like that. It really is still client/server.
• Permissions can still be an issue. The SQL Server instance will need to be able to read the *.mdf file, so the NTFS permissions need to be set appropriately; in most cases, this means that NETWORK SERVICE will need to be able to read and write to it.

After spending a few hours learning such things the hard way, I realized that user instances were not necessary for me (VS already runs as administrator, can’t debug in IIS without it). Instead I’ll just use SQL Server Management Objects (SMO) to programmatically set up the database at the start of each test. (More on this later.)

Understanding MSTest Execution Order

On my first attempt, I put my database and NHibernate set-up and tear-down code in methods decorated with the MSTest [TestInitialize] and [TestCleanup] attributes. And this does indeed work well when a single [TestMethod] is executed in isolation. But if I executed more than one test in a single test run, all hell would break loose because the [TestInitialize] for a test that is scheduled to execute might get executed before the [TestCleanup] from a test that has already run.

This seemed like astonishing behavior to me and seems to be just a quirk of Microsoft’s design of MSTest. In nUnit and pretty much any other xUnit framework I’ve tried, the initialize/test/cleanup triumvirate is guaranteed to execute “atomically” before any part of any other test triumvirate is executed. In other words, if you’re running a test run with two tests A and B, then you might see this behavior in the two testing frameworks:

• nUnit: Initialize A > Test A > Cleanup A > Initialize B > Test B > Cleanup B
• MSTest: Initialize A > Test A > Initialize B > Cleanup A > Test B > Cleanup B

That’s because MSTest executes tests in multiple threads, and while for a particular test method you’re guaranteed that its [TestInitialize] and [TestCleanup] methods will be called before and after the test itself executes, respectively, there is no guarantee about its relationship with the other simultaneously executing test methods.

This causes a problem when the test method is assuming exclusive access to a shared system resource, like a test SQL Server Express database that has just been set up for that test method’s exclusive use. My brute force workaround is to use Monitor.Enter() and Monitor.Exit() in the [TestInitialize] and [TestCleanup] methods to force MSTest to execute them in the correct order:

private static readonly object lockObject = new object();
 
[TestInitialize]
public void TestInitialize()
{
	Monitor.Enter(lockObject);
 
	try
	{
		// TODO: Delete any pre-existing SQL Server Express Instance
		// TODO: Set up the SQL Server Express Instance
		// TODO: Set up NHibernate
	}
	catch
	{
		// If something went horribly wrong, release the lock
		// or else MSTest will never finish the test run!
		Monitor.Exit(lockObject);
 
		throw;
	}
}
 
[TestCleanup]
public void TestCleanup()
{
	// TODO: Delete the SQL Server Express instance
	Monitor.Exit(lockObject);
}

Using SMO to Bind It All Together

All that was left was to fill in those TODOs.

Deleting an Existing Database

First up is deleting an existing instance, which is pretty trivial with SMO (just add a reference to Microsoft.SqlServer.Smo.dll, which is available after you installing SQL Server Express, and you’re set):

private static readonly string serverInstance = @"(local)\SQLEXPRESS";
 
private static readonly string databaseName = "EquineTest";
 
private string dataFilePath;
 
private string logFilePath;
 
private void DeleteDatabaseIfExists()
{
	var server = new Server(serverInstance);
 
	if (server.Databases.Contains(databaseName))
	{
		// Something might be caching an open connection, so tell everyone
		// to screw off by forcing their connections shut. If we don't do this,
		// the DetachDatabase() call could faile.
		var sql = string.Format(
			CultureInfo.InvariantCulture,
			"ALTER DATABASE {0} SET SINGLE_USER WITH ROLLBACK IMMEDIATE",
			databaseName);
		server.Databases[databaseName].ExecuteNonQuery(sql);
 
		server.DetachDatabase(databaseName, true);
 
		File.Delete(this.dataFilePath);
		File.Delete(this.logFilePath);
	}
}

Why check for an delete an existing database at the start of a test? In case a previous test failed. Shockingly, MSTest won’t call [TestCleanup] if a test method explodes with an exception. I know, I’m in total agreement with you, what were they thinking?

Figuring Out Where to Put the Files

Simple enough, but you’ll notice that my dataFilePath and logFilePath variables, which should be pointing to my *.mdf and *.ldf database files, respectively, aren’t initialized in my above example. Prior to calling this function, I make sure that my InitializePaths() method is called:

private string GetExecutingDirectory()
{
	var path = Assembly.GetExecutingAssembly().GetName().CodeBase;
	return Path.GetDirectoryName(new Uri(path).LocalPath);
}
 
private void InitializePaths()
{
	var directoryPath = this.GetExecutingDirectory();
 
	this.dataFilePath = Path.Combine(directoryPath, databaseName + ".mdf");
	this.logFilePath = Path.Combine(directoryPath, databaseName + "_log.ldf");
}

This is convoluted way of determining the directory that MSTest created for the current test run. (MSTest sets the working directory to some nonsense in Program Files, so no help there. And [DeploymentItem()] is an absolute nightmare!)

Creating the Test Database

The code for creating the database via SMO is similarly straightforward:

private void CreateDatabase()
{
	var server = new Server(serverInstance);
	var database = new Database(server, databaseName);
 
	var fileGroup = new FileGroup(database, "PRIMARY");
	database.FileGroups.Add(fileGroup);
 
	var dataFile = new DataFile(fileGroup, databaseName, this.dataFilePath);
	dataFile.Growth = 10;
	dataFile.GrowthType = FileGrowthType.Percent;
	fileGroup.Files.Add(dataFile);
 
	var logFile = new LogFile(database, databaseName + "_log", this.logFilePath);
	logFile.Growth = 10;
	logFile.GrowthType = FileGrowthType.Percent;
	database.LogFiles.Add(logFile);
 
	database.Create();
}

The only quirk is that I explicitly set the Growth and GrowthType because on some machines it was defaulting to None causing the integration tests to fail when lots of data is inserted under some circumstances. It seems to be a per-server default setting, but I haven’t bothered to figure out why the default behavior is different among the multiple machines that I own; setting it here guarantees the tests will work consistently.

Initializing the Database and Configuring NHibernate

The last piece of the puzzle is to configure NHibernate and set up the freshly-minted database’s schema. I use the NHibernate’s SchemaExport tool to create the schema for me based on my mapping files. The only caveat is that when using SQL SCHEMAs, the SchemaExport tool won’t create those for you, so they have to be handled explicitly:

[TestInitialize]
public void TestInitialize()
{
	Monitor.Enter(lockObject);
 
	try
	{
		this.InitializePaths();
		this.DeleteDatabaseIfExists();
		this.CreateDatabase();
 
		// The configuration is a static variable, I don't care if the
		// configuration is shared among test methods
		if (configuration == null)
		{
			// Disabling connection pooling is important
			var connectionString = string.Format(
				CultureInfo.InvariantCulture,
				@"Data Source={0}; Integrated Security=true; Initial Catalog={1}; Connection Timeout=120; Pooling=false;",
				serverInstance,
				databaseName);
 
			configuration = new Configuration()
				.SetProperty(Environment.ReleaseConnections, "on_close")
				.SetProperty(Environment.Dialect, typeof(MsSql2008Dialect).AssemblyQualifiedName)
				.SetProperty(Environment.ConnectionDriver, typeof(SqlClientDriver).AssemblyQualifiedName)
				.SetProperty(Environment.ConnectionString, connectionString)
				.SetProperty(Environment.ProxyFactoryFactoryClass, typeof(ProxyFactoryFactory).AssemblyQualifiedName)
				.AddAssembly("EquineBusinessBureau.Model");
 
			sessionFactory = configuration.BuildSessionFactory();
		}
 
		// We definitely need a new session for each test method, though		
		this.session = sessionFactory.OpenSession();
 
		// SchemaExport won't create the database schemas for us
		foreach (string schema in schemas)
		{
			var sql = string.Format(CultureInfo.InvariantCulture, "CREATE SCHEMA {0}", schema);
			this.session.CreateSQLQuery(sql).ExecuteUpdate();
		}
 
		new SchemaExport(configuration).Execute(
			true,
			true,
			false,
			session.Connection,
			Console.Out);
	}
	catch
	{
 
		Monitor.Exit(lockObject);
 
		throw;
	}
}

Conclusions and Delusions

Wrap all of this up in an abstract class called IntegrationTestBase, and you’ve got an easy way to write integration tests in a style that’s similar to what Ayende originally posted:

[TestClass]
public class RoleTest : InMemoryDatabaseTestBase
{
    [TestMethod]
    public void CanSaveAndLoadRole()
    {
        object identity;
 
        using (var tx = this.session.BeginTransaction())
        {
            identity = session.Save(new Role(EquineRoles.Administrator, "Administrator"));
 
            tx.Commit();
        }
 
        session.Clear();
 
        using (var tx = this.session.BeginTransaction())
        {
            var role = session.Get<Role>(identity);
 
            Assert.AreEqual<string>(EquineRoles.Administrator, role.RoleEnum);
            Assert.AreEqual<string>("Administrator", role.RoleName);
 
            tx.Commit();
        }
    }
}

The first test in a test run takes a while to run, on the order of 6 – 8 seconds. Painful, but this is also an integration test, so it’s not like I’m running them every five minutes, and it’s faster than spinning up the Web site and seeing if it crashed. Subsequent tests, since the NHibernate configuration is already initialized in the static variable, execute much more quickly, about one second per test.

Hope this saves someone a few hours of putting the puzzle pieces together.

Download the IntegrationTestBase example source file.

In other words, I was expecting the BinaryWriter‘s Write(string) implementation to look something like this pseudocode:

public void Write(string text)
{
     this.buffer.Append(this.encoding.GetBytes(text));
}

After all, when you call Write() with an int, you get four bytes written. If I call Write() with the string “TEST” and the BinaryWriter is using the ASCII encoding, I’d expect four bytes to be written. But instead it writes five.

That’s because the BinaryWriter writes the length of the string in one byte and then calls GetBytes() to output the string data.

Now that I think about it, this makes perfect sense: strings in the .NET framework are typically not thought of as being null-terminated, they’ve got a length, and in order for the BinaryReader‘s Read(string) method to work, it’ll need to be able to know the length of the string to determine how many bytes to read.

In my case, I was writing data to an Epson TM-T88III receipt printer, and given the structure of the commands that the printer expects, it doesn’t need or want the length of the string in this way. Because I didn’t read the MSDN documentation closely, I was left scratching my head as to why weird characters were showing up or characters were being omitted in my output.

The workaround is to just call GetBytes() myself and pass the byte buffer to the BinaryWriter‘s Write(byte[]) overload:

// "bw" is an instance of a BinaryWriter
// "barcode" is a string
 
bw.Write(AsciiControlChars.GroupSeparator);
bw.Write('k');
bw.Write((byte)4);
bw.Write('*');
bw.Write(Encoding.ASCII.GetBytes(barcode)); // NOT bw.Write(barcode);
bw.Write('*');
bw.Write(AsciiControlChars.Null);

Hope this saves someone from a forehead-slapping moment.

TM-T88III Receipt Printer

You’ve probably seen the TM-T88III before, though the current model is the TM-T88IV–they’re ubiquitous and especially common in restaurants. We chose it because the receipt prints quickly (which reduces the probability that a stack of order receipts gets missorted such that an employee puts the wrong receipt in the wrong parcel), is thermal so there is no ink to replace (which reduces the cost of consumables), uses standard thermal receipt paper (which is cheap, and we can always run out to Staples if we run out), and, perhaps most importantly, we found one in a box in the back of the warehouse.

At the top of the retail receipt, I print out the Skiviez logo. This is stored in a non-volatile area of memory in the printer (called NV RAM in EPSON parlance), and I dole out a command that essentially says “print out the image stored in slot #1,” having previously uploaded the bitmap via the printer driver’s flash utility. This worked great for as long as we only shipped Skiviez orders and only had one TM-T88III printer. But with the advent of WFS, however, Skiviez now ships orders (we call them “fulfillment requests”) for multiple e-commerce stores, each requiring a different logo to appear at the top of the receipt. So I now had two options:

• I could manually upload the logos for all of the merchants that we ship for into all of the receipt printers that we use, making sure to upload each merchant image into the same NV memory slot for each printer.
• I could store the merchant’s logo in our database, figure out how to use the “select bit image mode” receipt printer command, and just send the image data just-in-time whenever I generated a receipt.

The second option has obvious maintenance benefits, so that’s the path I started hacking my way down.

A brief primer on ESC/POS and data

Like the Zebra LP2844 and its brethren, the TM-T88III comes with a Windows printer driver that allows Windows and applications to see it as any other GDI-based printer. You can print a Word document to the TM-T88III and it will print, albeit slowly, and it’ll look like crap. That’s because, as with any other GDI printer, the Word document gets rendered as a bitmap and the entire big-ass bitmap gets sent to the poor little receipt printer which then prints it 24 dots at a time. A better method is to use the printer for its intended purpose and use the native printer language to generate the receipt. This allows us to select printer fonts that are specifically designed for the printer’s resolution and speed characteristics and use other advanced features of the printer such as the paper cutter.

That printer language is called ESC/POS, which is entirely unpronounceable and a stupid name. The “POS” stands for “Point of Sale” since this printer belongs to a class of devices commonly known as point of sale devices–cash drawers, VFDs, barcode scanners, and the like. The “ESC” stands for “escape” because the printer treats any data that is sent to it as text–passed straight through–unless it is escaped with a special character to indicate that a command instruction follows in the input stream. The most common escape character used is the ASCII ESC character, and so we have “ESC/POS” as our language name.

If you’ve just picked up one of these printers on eBay, you may find that getting documentation on this language is hard to come by because you need to get a copy of the “ESC/POS Application Programming Guide” from your “EPSON Authorized Dealer,” whoever the hell that is. (Again, we found ours in a box in the back of the warehouse, so my “authorized dealer” is long gone.) Still, some Googling will give you more or less complete and current copies of these guides, which I’ve mirrored here. Go ahead and download them:

• ESC POS Command Guide
• ESC POS Programming Guide

(I particularly like how the programming guide has “proprietary” and “confidential” stamped all over it and then proceeds to describe how great EPSON is for “taking initiative” for “expandability” and “universal applicability” on page 7 of the same document. You can’t make this crap up–it’s full of oxymorons.)

Having just come from writing in the EPL2 printer language for our Zebra LP2844 and Zebra ZP 500 Plus thermal label printers, I found the ESC/POS language to be confusing. My confusion mostly stemmed partly from the way the documentation was written and partly because I was used to the way EPL2 worked. Let’s take a look at the documentation for the bit image command that I’m trying to use:

If this were EPL2, I would actually send the document as a string to the printer, so if a command needed the integer 33 as a parameter, I would send the string “33″ (two bytes of data). In ESC/POS, each parameter is a single byte, so if I want to send 33 as a parameter for m in the bit image command above, then I need to send 33 in a single byte: that is, 0b100001, which is 2⁵ + 2⁰ = 32 + 1 = 33.

The reason that this is super confusing is that other parameters in the documentation are specified to be ASCII characters, such as the ‘*’ parameter in the bit image command above. This is because low-level programmers, such as those who designed the ESC/POS language, tend to blur the lines between data types: it’s all bytes at the end of the day. As a result, if you’re using C#, you might be tempted to use a StringBuilder to build up your document to send to the printer. Don’t do it! You’ll inevitably get confused by its overloads. Let’s take that m = 33 parameter as an example:

var sb = new StringBuilder();
 
// ASCII escape
sb.Append((char)27); 
 
// ASCII '*' for the bit image command
sb.Append('*'); 
 
// oops! this appends the string "33" in two bytes, doesn't work
sb.Append(33); 
 
// this is what I want, but semantically weird
sb.Append((char)33);

A much better and less confusing in the long run solution is to eschew any notion that we are dealing with text. Instead, let’s use the semantics that we are sending bytes of raw binary data directly to the printer:

using (var ms = new MemoryStream())
using (var bw = new BinaryWriter(bw))
{
     bw.Write(AsciiControlChars.Escape);
     bw.Write('*');         // bit-image mode
     bw.Write((byte)33);    // 24-dot double-density
}

But remember that cast to byte! Otherwise, you’ll get a C# int, which is four bytes, written to the stream, when we only wanted one. Yes, that means that any parameter that an ESC/POS command takes has a maximum value of 255.

You could get by on the StringBuilder method for a while and not have it burn you until you try the bit image command because all the characters that you’re appending happen to be less than 128–greater than that and you start getting into bytes that don’t map to a Unicode character for quirky historical reasons, such as the range 128 to 159, and you’ll be pulling your hair out as to why some of your data is getting “lost.” Just use the BinaryWriter method. You can thank me later.

Why is the bit image command designed this way?

The m parameter in the bit image command as defined in the previous section has 4 values, and each value changes the way we construct the image data bytes and the way the printer interprets them. The first three values are of dubious value, so let’s throw them out for now. We’ll focus on m = 33, which means “24-dot double density mode.” So keep this in mind and set this aside.

When we think of drawing a bitmap in high level software, we usually think of drawing it pixel by pixel, left to right, top to bottom. So the order that the dots get printed as indicated by the programming guide seems strange:

That is, the dots are rendered from top to bottom, then left to right.

Why in the blue hell do I need to rearrange my nice and neat array of bitmap data into this Connect Four scheme of dots? It all comes down to the size of the print head.

For our mental model, the thermal print head in the receipt printer is physically 1 dot wide by 24 dots high where there are 203 dots per inch (square pixels). It moves left to right across the receipt paper, burning up to 24 dots in the vertical (y) direction for each dot in the horizontal (x) direction.

Since we’re talking about a thermal printer here, there’s no concept of gray scale tones–we’ve either burned a dot into the paper (black) or we haven’t. So if we’re sending image data to the printer in bytes, it would make sense to say that a bit set to 1 means “burn a dot” and a bit set to “0″ means do nothing.

But a byte is only 8 bits wide, and we have 24 dots to burn in each “slice” of the image. 24 just happens to be divisible by 8, so we can send 3 bytes of data for each slice to represent our 24 dots. (If the bitmap I want to draw is taller than 24 dots/pixels, then I need to send multiple bit image commands, effectively doing multiple stripes of the image that are 24 dots high; more on this later.)

So, now, the diagram in the programming guide makes sense: we’re not burning 1 dot at time, we’re burning a vertical stripe that is 1 dot wide and 24 dots high at a time, and then moving to the right. So our printer reads our first 3 bytes of image data, burns the dots specified by those 24 bits, then reads the next 3 bytes, and so on.

Aside: Converting the bitmap to monochrome

Right. So a merchant has given me a *.bmp file, and I need to convert that into array of bits that I can send to the receipt printer. It’s a good bet that the bitmap that the merchant sent me for the logo is not monochrome (that is, every pixel is either 100% black or 100% white).

So what we can do is look at each pixel in the bitmap, determine its luma, and if that’s below a certain threshold, count that pixel as black. How did I write the code to do this? I looked the formula up via the intertubes and modified it to fit my needs:

private static BitmapData GetBitmapData(string bmpFileName)
{
    using (var bitmap = (Bitmap)Bitmap.FromFile(bmpFileName))
    {
        var threshold = 127;
        var index = 0;
        var dimensions = bitmap.Width * bitmap.Height;
        var dots = new BitArray(dimensions);
 
        for (var y = 0; y < bitmap.Height; y++)
        {
            for (var x = 0; x < bitmap.Width; x++)
            {
                var color = bitmap.GetPixel(x, y);
                var luminance = (int)(color.R * 0.3 + color.G * 0.59 + color.B * 0.11);
                dots[index] = (luminance < threshold);
                index++;
            }
        }
 
        return new BitmapData()
            {
                Dots = dots,
                Height = bitmap.Height,
                Width = bitmap.Width
            };
    }
}

BitmapData is just a little struct that contains the three properties that you see here. After all, once I have my BitArray with 1 indicating black dots and 0 indicating white dots, I don’t need the original Bitmap instance anymore.

Just to clarify, this means that we’re holding onto a BitArray of monochrome data in the Dots property that conceptually looks something like this:

Marshalling the monochrome data

Now for the trickiest part of all. We need to take our BitArray of monochrome data, divide it up into 8-dot chunks, represent those 8-dot chunks as bytes, and send them to the printer in the order required by the bit image command, discussed above. That is, I’ll need to send the data as bytes in this order:

So the printer will draw them one vertical stripe of three bytes each at a time. (If you don’t get it, print out that image, cut up the bytes, and then arrange them in the order shown in the diagram from the EPSON documentation. It’ll instantly make sense once you see it.) The X’s in the diagram correspond to bits that aren’t in our original image that we have to send anyway as padding. Remember, since we selected the 24-dot double density mode, the printer is going to draw a 1 x 24 dot slice as it moves the print head. My example bitmap is only 4 pixels tall, so I have to send 20 zero bits as padding.

And thus we would end up with our pretty image.

Unfortunately, there is some math involved in translating the bits in our BitArray into these bytes that we need to send. Assuming bw is our reference to a BinaryWriter, here’s the code that does just that:

// So we have our bitmap data sitting in a bit array called "dots."
// This is one long array of 1s (black) and 0s (white) pixels arranged
// as if we had scanned the bitmap from top to bottom, left to right.
// The printer wants to see these arranged in bytes stacked three high.
// So, essentially, we need to read 24 bits for x = 0, generate those
// bytes, and send them to the printer, then keep increasing x. If our
// image is more than 24 dots high, we have to send a second bit image
// command to draw the next slice of 24 dots in the image.
 
// Set the line spacing to 24 dots, the height of each "stripe" of the
// image that we're drawing. If we don't do this, and we need to
// draw the bitmap in multiple passes, then we'll end up with some
// whitespace between slices of the image since the default line
// height--how much the printer moves on a newline--is 30 dots.
bw.Write(AsciiControlChars.Escape);
bw.Write('3'); // '3' just means 'change line height command'
bw.Write((byte)24);
 
// OK. So, starting from x = 0, read 24 bits down and send that data
// to the printer. The offset variable keeps track of our global 'y'
// position in the image. For example, if we were drawing a bitmap
// that is 48 pixels high, then this while loop will execute twice,
// once for each pass of 24 dots. On the first pass, the offset is
// 0, and on the second pass, the offset is 24. We keep making
// these 24-dot stripes until we've run past the height of the
// bitmap.
int offset = 0;
 
while (offset < data.Height)
{
    // The third and fourth parameters to the bit image command are
    // 'nL' and 'nH'. The 'L' and the 'H' refer to 'low' and 'high', respectively.
    // All 'n' really is is the width of the image that we're about to draw.
    // Since the width can be greater than 255 dots, the parameter has to
    // be split across two bytes, which is why the documentation says the
    // width is 'nL' + ('nH' * 256).
    bw.Write(AsciiControlChars.Escape);
    bw.Write('*');         // bit-image mode
    bw.Write((byte)33);    // 24-dot double-density
    bw.Write(width[0]);  // width low byte
    bw.Write(width[1]);  // width high byte
 
    for (int x = 0; x < data.Width; ++x)
    {
        // Remember, 24 dots = 24 bits = 3 bytes. 
        // The 'k' variable keeps track of which of those
        // three bytes that we're currently scribbling into.
        for (int k = 0; k < 3; ++k)
        {
            byte slice = 0;
 
            // A byte is 8 bits. The 'b' variable keeps track
            // of which bit in the byte we're recording.                 
            for (int b = 0; b < 8; ++b)
            {
                // Calculate the y position that we're currently
                // trying to draw. We take our offset, divide it
                // by 8 so we're talking about the y offset in
                // terms of bytes, add our current 'k' byte
                // offset to that, multiple by 8 to get it in terms
                // of bits again, and add our bit offset to it.
                int y = (((offset / 8) + k) * 8) + b;
 
                // Calculate the location of the pixel we want in the bit array.
                // It'll be at (y * width) + x.
                int i = (y * data.Width) + x;
 
                // If the image (or this stripe of the image)
                // is shorter than 24 dots, pad with zero.
                bool v = false;
                if (i < dots.Length)
                {
                    v = dots[i];
                }
 
                // Finally, store our bit in the byte that we're currently
                // scribbling to. Our current 'b' is actually the exact
                // opposite of where we want it to be in the byte, so
                // subtract it from 7, shift our bit into place in a temp
                // byte, and OR it with the target byte to get it into there.
                slice |= (byte)((v ? 1 : 0) << (7 - b));
            }
 
            // Phew! Write the damn byte to the buffer
            bw.Write(slice);
        }
    }
 
    // We're done with this 24-dot high pass. Render a newline
    // to bump the print head down to the next line
    // and keep on trucking.
    offset += 24;
    bw.Write(AsciiControlChars.Newline);
}
 
// Restore the line spacing to the default of 30 dots.
bw.Write(AsciiControlChars.Escape);
bw.Write('3');
bw.Write((byte)30);

This code looks confusing, and is, but I’ve tried to document it with explanatory comments in line.

Sending the document to the printer

Finally, there is the task of sending this array of bytes directly to the printer. I’ve talked about this before and the code is uninteresting, requiring some P/Invokes into native APIs. Essentially, you pass in the name of the printer as it appears in the Control Panel, the array of bytes from our BinaryWriter, and you’re all set:

private static void Print(string printerName, byte[] document)
{
    NativeMethods.DOC_INFO_1 documentInfo;
    IntPtr printerHandle;
 
    documentInfo = new NativeMethods.DOC_INFO_1();
    documentInfo.pDataType = "RAW";
    documentInfo.pDocName = "Bit Image Test";
 
    printerHandle = new IntPtr(0);
 
    if (NativeMethods.OpenPrinter(printerName.Normalize(), out printerHandle, IntPtr.Zero))
    {
        if (NativeMethods.StartDocPrinter(printerHandle, 1, documentInfo))
        {
            int bytesWritten;
            byte[] managedData;
            IntPtr unmanagedData;
 
            managedData = document;
            unmanagedData = Marshal.AllocCoTaskMem(managedData.Length);
            Marshal.Copy(managedData, 0, unmanagedData, managedData.Length);
 
            if (NativeMethods.StartPagePrinter(printerHandle))
            {
                NativeMethods.WritePrinter(
                    printerHandle,
                    unmanagedData,
                    managedData.Length,
                    out bytesWritten);
                NativeMethods.EndPagePrinter(printerHandle);
            }
            else
            {
                throw new Win32Exception();
            }
 
            Marshal.FreeCoTaskMem(unmanagedData);
 
            NativeMethods.EndDocPrinter(printerHandle);
        }
        else
        {
            throw new Win32Exception();
        }
 
        NativeMethods.ClosePrinter(printerHandle);
    }
    else
    {
        throw new Win32Exception();
    }
}

The P/Invoke declarations can be found in the sample code at the end of the article, and the documentation is available on MSDN. But, again, there’s nothing sexy going on here.

Conclusions and delusions

And that’s a wrap. I always find it interesting when I encounter a problem at work–which is a small business by any definition of the phrase–and the Google indicates that no one has ever encountered or documented this problem before. And it’s these types of integration problems that I find fascinating; after years of playing in castles in the sky, dealing with databases and exceptions and HTML forms, it’s nice to know that I can dust off the corners of my brain and my college degree and deal with bits and bytes when I need to. Writing software that makes hardware do stuff is always fun, too.

And after hours of watching the receipt printer spew out lines and lines of gibberish, there’s nothing like the feeling of seeing the bitmap print out and knowing that, finally, things have simply started to work.

Good luck!

Download the code used in this article.

Yes, it epitomizes Battleship Grey. You love it.

But, behind the scenes, it’s not quite as simple as all of that. It’s aggregating order data from heterogeneous data sources–some in our legacy database, some in our new fulfillment system, some in a custom integration with a third party. It has to figure out which postage account to pay for postage with or which FedEx account number to use to ship a package. For orders that aren’t coming from our new fulfillment system, it has to “cleanse” the address against the current USPS address database. It has to figure out the cheapest way to ship a package, compute customs values correctly, generate certificates of origin and commercial invoices for international shipments, and determine what box types an order is allowed to be packed in. And it has to write shipment information back to one of those three disparate data sources.

What this means is that I’m frequently making adjustments and bug fixes to the application. And managing the deployment and installation of those bug fixes had been, up until now, a pain.

A brief interlude on ClickOnce

The other internal application that we use (“Undies Client”) for our long-time running e-commerce store is deployed via ClickOnce, which is essentially the Java Web Start of the .NET world.

While ClickOnce is a neat technology and has its applications, to be sure, I probably wouldn’t use it again on Undies Client if I were starting that application over today, just as I decided not to use it for the Fulfillment Manager (which is an effort to divorce the processing and shipping features from Undies Client and make them simpler and applicable to multiple e-commerce stores).

First, there’s user confusion. If I deploy a Windows Installer MSI via Group Policy, then the application is magically there on all computers in the office. But for Undies Client, you have to go to a special Web page and click on a link. With ClickOnce, the installation happens per user, so employees can get confused if they go to another computer one day, log into their account, and see that the app isn’t there (“but Jennifer runs it on this computer so I thought it was already installed”).

Second, there’s deployment headaches. Like Java Web Start, you get a retarded warning if the deployment manifest wasn’t signed with an expensive code signing certificate. To mitigate that, you either buy one or start diddling with the self-signing certificate capability within the context of your own Active Directory domain. Not a show-stopper, and it makes sense, I guess, but it’s One More Thing that you have to deal with.

Third, when that certificate expires and needs to be renewed, you’re in for a world of hurt, because essentially all users will need to uninstall and re-visit the Web site download link and reinstall. Otherwise, the application simply stops seeing the newer updated versions and doesn’t update itself.

Fourth, the distribution of your app now has a dependency on an IIS installation somewhere, so that’s something else to maintain–both the configuration of that virtual directory in IIS as well as the shared drive to which Visual Studio dumps its files when clicking the “Publish” button.

Fifth, the installation can’t do much. Until recently, you couldn’t even create an icon on the desktop as part of the installation process. Nor can you do anything that would require elevated permissions for actions that you might typically do when running an installer, such as registering a COM DLL, or installing some third-party dependency. So the Web page at which you download the app usually contains things like “ooh be sure to install this that and the other first,” defeating the deployment simplicity of ClickOnce. And if you need to update one of those third party dependencies and your app because dependent on one of those updates, you have no way to update that dependency with ClickOnce, unless you take it upon yourself to have your application manage the upgrade during its next run. That’s just more work than you shouldn’t have to do.

After writing all of this, it may seem like I am saying that ClickOnce is a half-baked load of crap; it’s not half-baked. I’m saying it’s a fully-baked, complete load of crap. (Kidding. ClickOnce has its applications for applications that can be completely self-contained, but if at any point you become dependent on anything COM, it’s time to move on to real deployment technology.)

Using WiX to create a Windows Installer MSI file

I’ve blogged about WiX before. It’s a great open source tool put together by some guys who decided to write a reasonable mechanism for generating Windows installers because, for some reason, the Windows installer team has seemed to think that editing database tables in a cheeseball editor called Orca was sufficient. This would be like saying that our warehouse workers could ship orders by updating data in a Microsoft Excel spreadsheet.

You could also pay lots of money for InstallShield or something similar, which would create MSI installers for you, but installation is a convenience for me–as a small business whose primary focus is not end-user software, paying for that doesn’t make much sense. There’s also NSIS, but, oh–I just threw up all over myself. We’ll save NSIS for another post.

Additionally, the WiX guys have realized that installers usually need to do useful things, like install certificates, set up Web sites in IIS, and run database scripts, whereas the Windows Installer team seems to have been trying to make writing custom actions harder, not easier, with their subsequent releases, because that’s where most of the crashes and problems in setup packages happen. With WiX, we now have a suite of well-tested custom actions that lots of people are using; this should have been the Installer team’s original response instead of depending on the community and third parties to fill in this gap for them, but it is what it is.

The point is that WiX enables a whole class of small business developers like me to build first-class deployment methods into their applications. With a 300-odd line XML file, the Fulfillment Manager now builds to an MSI file. And since WiX integrates with Visual Studio, I can generate that XML file as part of my build process.

Indeed, I’ve set up TeamCity and use this as a continuous integration server. Whenever I commit a change to Subversion, TeamCity picks up the change, compiles the solution, runs the tests, and if they pass, copies the newly generated MSI file to a network share for potential deployment. It’s pretty sweet.

The missing piece of the puzzle, then, is actually getting this freshly baked MSI file onto all of the client machines in the office.

Deploying via the Group Policy Software Installation Extension

My first instinct was to use the Group Policy Software Installation Extension. This is the thingie where when you open up a group policy in the Group Policy Management Editor, you can drill down to the Software Installation thing under Computer Configuration, specify an MSI file for deployment, and (presto!) any computers linked to that GPO will install the MSI on next boot.

This worked swimmingly well for the first release of Fulfillment Manager.

We pause for another brief interlude on update strategies

Let me explain the design of the installation for a moment. I’ve written my WiX file such that each new MSI file that it generates is a “major upgrade” in the Windows Installer parlance–it has a different product code, a different package code, and a different version number (since my version numbers are a combination of an incrementing build number and the Subversion revision number). But they all have the same upgrade code and I schedule RemoveExistingProducts during the install.

This means that if you have an older version of the Fulfillment Manager on your machine and double-click the MSI file for a newer, updated version, you don’t have to do anything–the existing version is completely uninstalled and the new version is installed on top of it.

The Windows Installer has support for “minor upgrades” and “small updates”, but I can never keep the damn things straight. Can I add a new component? Can I change a file? Can I reorganize a feature? Do I really want to be thinking about this every time I press Build in Visual Studio? My application is small, so I think it’s far easier and more reassuring to just blow away the whole thing and install again during an update, starting with a clean slate each time. In fact, I think this is a reasonable approach for many reasonably-sized applications (Paint.NET, for example, does this) and only becomes a problem when you start getting really large (such as Visual Studio or Microsoft Word).

Getting the update out there

OK. So all I really need to do is get all of the client computers to run msiexec /i FulfillmentManager.msi on the MSI and I’ll be good to go.

You might think that I could just overwrite the old MSI file on the network share with the new one, and the computers would notice the change at the next boot. But you would be, as I was, a fool–machines that already had the install would not notice the change and machines that did not have the install would freak out because they could not find the correct MSI file. After positing my question on ServerFault, I discovered the way the Group Policy Software Installation Extension works is by creating an advertisement script (*.aas file) and referencing that script via an object sitting in the Active Directory. That LDAP entry and the script file both do annoying things like reference a specific package code and product code, both of which I change with every new build of my software. So this method is out for the count.

Similarly, lugging out the Group Policy Editor and trudging down to the package entry and clicking “Redeploy application…” won’t work for the reasons described above–except that it’ll break the machines that already have the software installed, too.

What works is lugging out the Group Policy Editor, trudging down to the package entry, clicking “Remove” and “Immediately remove”, and then adding the package right back. This creates an updated *.aas file and correspondingly updated LDAP entries, and the old LDAP entry is flagged with a “remove me now please” flag. This works, but there are three things that I don’t like about it:

• It’s a manual process, so I have to remember to do it every time I create a new deployable build.
• References to all of the old versions hang around by necessity, since it’s recording the fact that “hey, if I see this particular product code then I need to uninstall it.” Indeed, upon inspecting the SYSVOL share, I saw that there about 45 of such files sitting in there since development of this app started in early July.
• There is no way to perform this process programmatically. (Well, there is, it’s documented as part of the EU anti-trust settlement, but let’s get real now: if it has LDAP in the spec then I’m not touching it with a ten-foot pole. Plus, this would be work that is totally tangential to my problem, which is automating a 1-minute task that ignores the crap out of me. At several days’ worth of work, it’d take me a long time to climb out of that time deficit to realize any savings.)

Nirvana: Automatic updating

The Software Installation extension for Group Policy can do a lot of things that I don’t need, like using patches or transforms, or shifting installed software by just moving a computer to a different OU, when at the end of the day all I really wanted was something that looked like this:

Must this be so difficult?

Since my installer will uninstall any previous versions, all I need to do is run the installation package via msiexec. This is easy enough to do via batch script that I’ve configured to run at startup via group policy, since those batch scripts run as SYSTEM and will have the necessary privileges to complete successfully.

You would think that in addition to /i and /x (and the idiotic /vomus), msiexec would have a somewhat useful parameter called, oh, I don’t know, /install-it-only-if-the-damn-thing-isn't-already-installed, but that would be a useful feature, so of course the Windows Installer team didn’t actually implement it.

Now, granted, there is actually no harm in running my installer when my app is already installed. It just would check that all the components are indeed installed and exit. But this still leaves a bad taste in my mouth. Not all MSI’s that I want to use in this way might behave like this. And, if I have any custom actions, that means that they’ll also get run on every boot, which seems like a waste.

While I could spelunk through the registry to try and see if I can find my current product code in the list, I’d rather not do that, because (1) that might not work consistently on all versions of Windows and (2) really, I’m just not that comfortable with batch scripts to begin with.

What ended up doing is writing a little command-line program called msicheck. The source file is boring and decently commented. Because all of the Windows Installer APIs are in C, my utility is in C. I briefly thought about writing it in C# with some interop, because I hate C just that much, but that did seem awfully overkill for such a simple application.

Ah, C in Windows. I had long forgotten the days of Hungarian variable names that sound like Yosemite Sam cussing (“lpszPackageVer“) and the land of if ("swizzle" == "swizzle") returning false. Heck, I had forgotten the days of not evening having false! I digress; the code is probably awful, but seems to work well enough for my purposes.

> msicheck.exe /?
 
Determines if a given Windows Installer package is installed.
 
msicheck package
 
Exit Codes:
        0       Exact version installed
        1       General failure
        2       Path to MSI package not found or not accessible
        3       Error determining package status
        4       Newer version installed
        5       No version installed
        6       Older version installed

All msicheck does is take the full path to an MSI file as an argument and returns an exit code that indicates the status of that particular package by looking at the product code and the version. (That is, if the product code is not installed, it returns 5; if the same product code is found, it compares the version and returns 0, 4, or 6.) I call it in my batch script (which, I don’t do this for a living, it’s probably retarded) like so:

@ECHO OFF
 
REM ---------------------------------------------------------------------
REM VARIABLES
REM ---------------------------------------------------------------------
 
SET path=\\SKIVIEZSBS2008\Group Policy Installations\Fulfillment Manager\
SET package=%path%Skiviez.FulfillmentManager.Installer.WinForms.msi
SET msicheck=%path%msicheck.exe
 
ECHO Checking installed version of Fulfillment Manager...
 
"%msicheck%" "%package%"
 
IF ERRORLEVEL 0 IF NOT ERRORLEVEL 1 (
	echo Fulfillment Manager is up to date.
) ELSE (
	IF ERRORLEVEL 5 (
		echo Installing latest version...
		%SYSTEMROOT%\system32\msiexec /qn /i "%package%"
		IF ERRORLEVEL 0 (
			echo Fulfillment Manager successfully updated.
		) ELSE (
			echo Errors may have occurred during installation.
		)
	) ELSE (
		echo Newer version installed or error occurred.
	)
)

Keeping in mind that IF ERRORLEVEL 5 evaluates to true for error levels of 5 or greater, this means that I call msiexec only if the product isn’t installed or if an older version is installed.

Conclusions and Delusions

So, by using a batch script and my msicheck utility, I have things the way I want them. I commit a change; TeamCity builds it, runs the tests, and copies the resulting MSI to the network share that is referenced in my batch script; and the batch script runs at the next boot, uses msicheck to note that that package is not installed, and so runs msiexec to install the update (which removes the old version as part of the install process).

It’s certainly not applicable to every software deployment scenario, much like ClickOnce, but hopefully it’ll help someone out there who wanted to do something similar.

Download MsiCheck, which requires Windows Installer 4.5 to run. Complete with the “works on my machine!” guarantee of absolutely no warranties.

Good luck!