To stop fraud, we need to stop allowing purchases to be made solely with the information that appears on the card because this is not secret information; it is basic information about an account. This is not something that merchants can fix alone. Merchants already have to foot the bill for chargebacks, and PCI standards are really just an attempt to make merchants responsible for maintaining the security of what is, at its core, a payment system that broke in the 1990s. CVN didn’t work–10 years later, we still have to explain to consumers where the number is because the card associations have not spent enough effort educating the public and, quite frankly, they don’t care. To use a credit card, anywhere, you should have to swipe it (PCI is still relevant in this scenario). Period.

Two-factor authentication, in the traditional sense, is not a solution. 3-D Secure is confusing for both merchants and consumers, expensive, and little used: for the program to work, it needs to be required. A better solution would be programs like Discover’s Secure Online Account Number (where you can generate a temporary account number), but they fucked that up by making the temporary account numbers expire on the same day as the real account number (to be effective, they should expire after a certain number of transactions or be able to be locked down to a specific merchant). And for such a program to work, this also needs to be required, too.

It’s time for consumers and banks to grow up and deal with credit card fraud because the problems with it today are largely ones that merchants cannot and no amount of obfuscating cardholder data can fix. Unfortunately, at the end of the day, it’s the merchant who usually ends up paying for the fraudulent charges when the dime finally drops, so we’ll continue to see little help from the card associations and the issuing banks: no one has the balls to make sweeping consumer- and bank-initiated security changes to the industry and require those changes for future transactions, unless they’re pointing their fingers at the merchants.

I guess we’ll just have to keep bending over and taking it–the responsibility, the consequences of fraud, and all of the blame when the system doesn’t work.

The outgoing mail transport agent (“mail server”) computes a hash of the body of the mail message and adds a special, cryptographically signed DKIM-Signature header to the mail message that contains this hash as well as some other information about the message. The public part of the key, used by receiving mail transport agents to verify the signature on incoming messages, is stored as a TXT record in the DNS for the signing domain.

The result of all of this is that the recipient can now know that the body and certain headers of the e-mail message were not tampered with or changed by a third party while the message was in transit. If the signature contained within the DKIM-Signature header doesn’t verify, then it’s possible that the message is a phishing message, spam, or some other false representation of members of the signing domain. (IMHO, however, it is more likely to just indicate a broken DKIM implementation, as there are several obnoxious corner cases that we will see.)

Where is the Exchange support?

No currently released version of Microsoft Exchange supports DKIM natively, not even 2010. This is purportedly because Microsoft threw its weight behind another standard called SPF, or Sender Protection Framework. SPF is much simpler to implement because it does not make a statement about the integrity of the message contents; that is, there is no signing step that requires processing of each outbound message and stamping it with a special header. Instead, a TXT record in the DNS for the sending domain lists the IP addresses from which e-mail messages from that domain are allowed to originate.

For example, let’s say that a store has a mail transport agent at mail.example.com and a Web site at www.example.com. An SPF record for example.com might make a statement like the following:

“E-mail messages purporting to be from @example.com should only originate from either mail.example.com or www.example.com. All other originating IP addresses should be treated with suspicion.”

You can see why this is easier to implement; you just add an entry to DNS, and you don’t have to configure the outbound mail transport agent to do anything. The only reason a mail transport agent would need to change is for it to gain the ability to inspect incoming messages and validate the MAIL FROM part of the message headers against the corresponding SPF TXT records in DNS. And that is what Exchange does today.

While SPF does not validate the integrity of the message–a hacked, legitimate company mail server that spews out spam unwittingly will still pass an SPF–it is a simple, convenient way to mark phishing and spoofing attempts originating from other locations, like a random house in Wisconsin that is unwittingly part of a botnet, with suspicion.

But, then again, a hacked mail transport agent using DKIM that spews out spam unwittingly will still pass a DKIM verification. Spam can be signed just like regular mail. So the advantage of DKIM over SPF is that DKIM provides the ability to prove that the contents of the message were not modified in transit, and if an organization so chose, they could sign different messages with different keys or decide to not sign certain kinds of messages at all, allowing the recipient to interpret these different kinds of mail in different ways.

In reality, most e-mail doesn’t need to be that secure, so the utility-to-difficulty ratio may be a little out-of-whack here, especially when other standards for signing messages already exist.

So DKIM is more complicated and not supported by Microsoft. Why implement DKIM on Exchange?

Because fuck Yahoo!, that’s why.

Actually, let’s step back a minute.

Yahoo! invented something called Domain Keys, which predates DKIM and which DKIM is largely based on, in an attempt to combat spam in the ways described previously. The aim was for Yahoo! mail servers to be able to easily identify entire classes of suspect e-mail and send them to people’s spam folders (which, in Yahoo! weasel words, is referred to as the “bulk mail folder”). Then they integrated it into the Yahoo! mail system.

If your business sends mail of any kind of volume to Yahoo! mail servers, such as an e-mail newsletter that users subscribe to, you’ll quickly see a message along the lines of the following:

421 4.16.55 [TS01] Messages from x.x.x.x temporarily deferred due to excessive user complaints

And you might also notice your e-mail messages going straight to the spam folder on Yahoo! mail accounts by default, with a X-Yahoo-Filtered-Bulk: x.x.x.x header appearing in the message headers. If this has happened, it is because Yahoo! has decided that you are naughty and placed the IP address of your mail transport agent on an internal blacklist. If a Yahoo! user has previously marked e-mail from your domain as “Not Spam,” then that user will continue to receive your e-mail, but any other Yahoo! users will find your e-mail going to the spam folder by default until they either click the “Not Spam” button on one of them or add your from address to their contact list.

“But we’re a legitimate small business,” you say. “Only about 5,000 of the e-mails on our mailing list are actual Yahoo! addresses, the newsletter list is opt-in, and there are ‘Unsubscribe’ links in every e-mail. It’s not like we’re spamming users out of the blue,” you continue. “These are users who have asked for this service and can cancel at anytime. So why has Yahoo! blacklisted us and treat us like spam?”

The answer, unfortunately, is that Yahoo! is incompetent and can’t build a mail service that can handle spam like Gmail can handle spam. Instead, their e-mail system depends on the following metrics:

• If a large burst of e-mail comes from the same IP address within a certain time period, temporarily consider it as a spam and temporarily deny the requests. The mail will still arrive, but the “fat pipe” will be squashed down so that overall delivery takes longer to complete. The idea is that if the mail is originating from a spammer, they are unlikely to try again.
• If a “significant” number of Yahoo! users mark mail messages originating from the same IP address as spam by clicking on the “Spam” button in their account, add that IP address to the X-Yahoo-Filtered-Bulk blacklist.

In other words, if enough people mark your e-mail as spam, your IP address is fucked, and you’ll soon be fielding complaints from customers who insist that they aren’t receiving your Web sites transactional e-mails, such as a “forgot my password” or “shipping confirmation” e-mail, because Yahoo! has now decided to deliver all e-mails that originate from your IP address to the spam folder, which many users do not think to check or cannot find. So they blame you.

If you think that Yahoo! has blacklisted you in error, or if you have improved your e-mail sending practices and wish to have them consider you for removal, you can fill out this form with incorrect JavaScript required field validation to request that the Yahoo! postmaster take you off the naughty list. About 27 hours later, a Yahoo! representative with a sub-room temperature IQ will e-mail you back a long, canned reply that generally amounts to “No.”

But in that “no” message, they offer the possibility of signing up to the Complaint Feedback Loop. When you participate in this program, if a Yahoo! user presses the “Spam” button on one of your e-mails, then Yahoo! will e-mail you the e-mail address of that user along with a copy of the e-mail that you sent. You can then unsubscribe that user from your mailing list yourself (since they neglected to use the unsubscribe link in your e-mail and will continue to damage the reputation of your IP address if you continue to send mail to them), or you can see which kinds of e-mails you are sending are creating the most problems.

But participation in their feedback loop requires that you use DKIM to sign all of your outbound mail messages. The reasons for this are not clear, but I assume it has to do with ensuring that you are only notified about the problematic e-mails that legitimately came from your organization, and not ones generated by a spammer. Or, it could simply be a way to drive adoption of Yahoo!’s DKIM standard.

At any rate, Yahoo! mail is the most widely used e-mail service on the planet, so when the idiots in the higher echelons of Yahoo! say “Jump!” then we must ask “How high?” When they say “implement DKIM”, we implement DKIM.

So what are options for implementing DKIM on Exchange?

One option is to simply not do it in Exchange and set up a relaying mail server that has DKIM support, like hMailServer or Postfix with dkim-milter. But if you’re at a small business, the idea of maintaining yet another server for what is conceptually a simple task is not a pleasant thought.

Another option is to use a dedicated device like a Barracuda or an IronPort. This device would sit in front of Exchange and rewrite the mail headers in transit, adding the DKIM-Signature header as it flies out of your office. But these devices are not cheap and are out of reach of many small businesses. And the thought of acquiring a specialized device for doing something any mail transport agent should be able to do natively is not a pleasant thought.

You can buy an off-the-shelf plug-in for Exchange like this one from a company in Hong Kong. The reviews on the Internet do generally seem to indicate that it does work. But do we really want to spend $300 – $800 on a component from a company without a reputation and give that component read access to all of our organization’s outbound mail messages? Trust is certainly a driving factor here.

A fourth option, and the option I pursued since I considered giving up on DKIM if Yahoo! didn’t change its ways after our implementation, is to take advantage of Exchange 2007′s new Transport Agents functionality. This allows you to write custom, managed code that runs on that .NET Framework that integrates with the Exchange message processing pipeline. In our case, we could write a custom transport agent that appends a DKIM-Signature header to outgoing MIME messages.

Setting up the project

In Visual Studio, we just need to use a plain old “C# Class Library” project. The project must target version 2.0 of the CLR (that’s .NET Framework 2.0, .NET Framework 3.0, or .NET Framework 3.5, thanks to the dipshits in Microsoft’s marketing department) and not the .NET Framework 1.x or, maddeningly, the .NET Framework 4.x. This is because the transport agent process provided by Exchange Server 2007 that will load our transport agent is running on CLR 2.0, and a process that’s been loaded with an earlier version of the CLR can’t load a later version.

We also need to reference two assemblies provided by the Exchange Server, Microsoft.Exchange.Data.Common.dll and Microsoft.Exchange.Data.Transport.dll. You would think that you would be able to find these in, say, the Exchange Server SDK that’s available from Microsoft Downloads. But you’d be wrong. Microsoft keeps diddling with the version of these assemblies with various update rollups for Exchange, and the only way to get a copy is to pull them off an actual Exchange Server 2007 installation. Mine were located in C:\Program Files\Microsoft\Exchange Server\Public; I just copied them to my local computer and referenced them in my new class library assembly for local development.

To start, we just need to create two classes. The first is our actual agent, which derives from the RoutingAgent class defined in the DLLs we just copied:

public sealed class DkimSigningRoutingAgent : RoutingAgent
{
	public DkimSigningRoutingAgent()
	{
		// What "Categorized" means in this sense,
		// only the Exchange team knows
		this.OnCategorizedMessage += this.WhenMessageCategorized;
	}
 
	private void WhenMessageCategorized(
		CategorizedMessageEventSource source,
		QueuedMessageEventArgs e)
	{
		// This is where the magic will happen
	}
}

The second is a factory class that the Exchange server’s little plug-in agent looks for; it’ll instance new copies of our agent:

public sealed class DkimSigningRoutingAgentFactory : RoutingAgentFactory
{
	public override RoutingAgent CreateAgent(SmtpServer server)
	{
		return new DkimSigningRoutingAgent();
	}
}

You’re probably thinking, “Wow! The documentation provided by the Exchange team sure blows donkey chunks, how did you ever figure out that that was what you needed to do?” The answer is by spelunking through the Exchange Server SDK examples and by religiously following this guy’s blog.

Now we’re ready to start reading mail messages and “canonicalizing” them into the format that the DKIM spec expects.

Exchange and “Internet Mail”

If there is one thing that is annoying about Exchange, it is that 20-some years after the fact it seems skeptical that this whole “Internet Mail” thing is going to catch on. Getting Exchange to give you the raw message in MIME format is not as simple as one might think.

You see, to be able to hash the body of the message and sign certain headers in the message, we need to know exactly how Exchange is going to format it when it is sent. Even a single space or newline that is added after we compute the hash can throw the whole thing off.

In our routing agent’s OnCategorizedMessage event listener, the event arguments give us access to an instance of the MailItem class. It has a boatload of properties for accessing the body and headers of the message programmatically. Unfortunately, we can’t use these properties because they represent the semantic values, not the raw ones. Instead, we’ll need to use the GetMimeReadStream() and GetMimeWriteStream() methods to read the raw mail message and write out the modified version, respectively.

Implementing the routing agent

Let’s start by completing the Routing Agent implementation. We’ll keep it simple by moving all of the hard signing stuff into an IDkimSigner interface, which we’ll worry about implementing later:

public sealed class DkimSigningRoutingAgent : RoutingAgent
{
    private static ILog log = LogManager.GetLogger(
        MethodBase.GetCurrentMethod().DeclaringType);
 
    private IDkimSigner dkimSigner;
 
    public DkimSigningRoutingAgent(IDkimSigner dkimSigner)
    {
        if (dkimSigner == null)
        {
            throw new ArgumentNullException("dkimSigner");
        }
 
        this.dkimSigner = dkimSigner;
 
        this.OnCategorizedMessage += this.WhenMessageCategorized;
    }
 
    private void WhenMessageCategorized(
        CategorizedMessageEventSource source,
        QueuedMessageEventArgs e)
    {
        try
        {
            this.SignMailItem(e.MailItem);
        }
        catch (Exception ex)
        {
            log.Error(
                Resources.DkimSigningRoutingAgent_SignFailed,
                ex);
        }
    }
 
    private void SignMailItem(MailItem mailItem)
    {
        if (!mailItem.Message.IsSystemMessage &&
            mailItem.Message.TnefPart == null)
        {
            using (var inputStream = mailItem.GetMimeReadStream())
            {
                if (this.dkimSigner.CanSign(inputStream))
                {
                    using (var outputStream = mailItem.GetMimeWriteStream())
                    {
                        this.dkimSigner.Sign(inputStream, outputStream);
                    }
                }
            }
        }
    }
}

The only real quirk is the if statement in the SignMailItem() function, which I mostly discovered through trial and error. If the mail item is a “system message” (whatever that means) then all of the mailItem‘s methods will be read only (throwing exceptions if we try to mutate), so we shouldn’t even bother. And if the mail item has a TNEF part, then in it’s in a bizarro proprietary Microsoft format, and the DKIM spec just isn’t going to work with that. Finally, if something blows up, we catch the exception and log it–better to send a message without a signature than not send it all.

Defining an interface for DKIM signing

So the next step is to make up that IDkimSigner implementation and make it do the dirty work. You can see that I’ve made it simple in that we only need to write two methods:

public interface IDkimSigner : IDisposable
{
    bool CanSign(Stream inputStream);
 
    void Sign(Stream inputStream, Stream outputStream);
}

A method for sanity checking

The first method will scan our mail item’s content stream and do a sanity check and ensure that we can actually sign the message. For example, if our IDkimSigner implementation is configured to sign messages originating from warehouse1.example.com and we pass CanSign() a message from warehouse2.example.com, then we can return false to indicate that we just don’t know what to do with the message. Let’s implement that method.

private string domain;
 
public bool CanSign(Stream inputStream)
{
    bool canSign;
    string line;
    StreamReader reader;
 
    if (this.disposed)
    {
        throw new ObjectDisposedException("DomainKeysSigner");
    }
 
    if (inputStream == null)
    {
        throw new ArgumentNullException("inputStream");
    }
 
    canSign = false;
    reader = new StreamReader(inputStream);
 
    inputStream.Seek(0, SeekOrigin.Begin);
 
    line = reader.ReadLine();
    while (line != null)
    {
        string header;
        string[] headerParts;
 
        // We've reached the end of the headers (headers are
        // separated from the body by a blank line).
        if (line.Length == 0)
        {
            break;
        }
 
        // Read a line. Because a header can be continued onto
        // subsequent lines, we have to keep reading lines until we
        // run into the end-of-headers marker (an empty line) or another
        // line that doesn't begin with a whitespace character.
        header = line + "\r\n";
        line = reader.ReadLine();
        while (!string.IsNullOrEmpty(line) &&
            (line.StartsWith("\t", StringComparison.Ordinal) ||
            line.StartsWith(" ", StringComparison.Ordinal)))
        {
            header += line + "\r\n";
            line = reader.ReadLine();
        }
 
        // Extract the name of the header. Then store the full header
        // in the dictionary. We do this because DKIM mandates that we
        // only sign the LAST instance of any header that occurs.
        headerParts = header.Split(new char[] { ':' }, 2);
        if (headerParts.Length == 2)
        {
            string headerName;
 
            headerName = headerParts[0];
 
            if (headerName.Equals("From", StringComparison.OrdinalIgnoreCase))
            {
                // We don't break here because we want to read the bottom-most
                // instance of the From: header (there should be only one, but
                // if there are multiple, it's the last one that matters).
                canSign = header
                    .ToUpperInvariant()
                    .Contains("@" + this.domain.ToUpperInvariant());
            }
        }
    }
 
    inputStream.Seek(0, SeekOrigin.Begin);
 
    return canSign;
}

Barf. But we have to do this style of ghetto parsing because, after all, we’re dealing with the raw e-mail message format. All we’re doing is scanning through the headers until we reach the last From: header, and then we make sure that the From: e-mail address belongs to the domain that our instance knows how to sign. Then we seek back to the beginning of the stream to be polite.

A method for signing

The second method that we have to implement is the one that actually does all of the dirty work. And in DKIM signing, we can break it down into five steps:

1. Compute a hash of the body of the message.
2. Create an unsigned version of the DKIM-Signature header that contains that body hash value and some other information, but has the signature component set to an empty string.
3. “Canonicalize” the headers that we are going to sign. By “canonicalize”, we mean “standardize capitalization, whitespace, and newlines into a format required by the spec, since other mail transport agents who get their grubby paws on this message might reformat the headers”.
4. Slap our unsigned version of the DKIM-Signature header to the end of our “canonicalized” headers, sign that data, and slap the resulting signature to the end of the DKIM-Signature header.
5. Write this signed DKIM-Signature into the headers of the mail message, and send it on its merry way.

Divide and conquer!

Implementing the Sign() method

Our implementation for the Sign() method will tackle each step in turn:

public void Sign(Stream inputStream, Stream outputStream)
{
    if (this.disposed)
    {
        throw new ObjectDisposedException("DomainKeysSigner");
    }
 
    if (inputStream == null)
    {
        throw new ArgumentNullException("inputStream");
    }
 
    if (outputStream == null)
    {
        throw new ArgumentNullException("outputStream");
    }
 
    var bodyHash = this.GetBodyHash(inputStream);
    var unsignedDkimHeader = this.GetUnsignedDkimHeader(bodyHash);
    var canonicalizedHeaders = this.GetCanonicalizedHeaders(inputStream);
    var signedDkimHeader = this.GetSignedDkimHeader(unsignedDkimHeader, canonicalizedHeaders);
 
    WriteSignedMimeMessage(inputStream, outputStream, signedDkimHeader);
}

Computing the body hash

The first step, computing the hash of the body, is actually pretty easy. There is only one quirk in that DKIM spec says that if the body ends with multiple empty lines, then the body should be normalized to just one terminating newline for the purposes of computing the hash. The code is not exciting, and you can download it at the end of this article.

Creating the “unsigned” header

The next step is to create the “unsigned” DKIM-Signature header. This is where the DKIM spec is just weird. The DKIM-Signature header contains a lot of information in it, such as the selector, domain, and the hashing algorithm (SHA1 or SHA256) being used. Since that information is vital to ensuring the integrity of the signature, it’s important that that information be a part of the DKIM signature.

If I were designing this, I would append two headers to e-mail messages: a DKIM-Information header that contained all of the above information and is part of the data that is signed and a DKIM-Signature header that contains just the signature data. But the DKIM spec makes use only the one DKIM-Signature header, and for the purposes of signing, we treat the “signature part” of the header (b=) as an empty string:

private string GetUnsignedDkimHeader(string bodyHash)
{
    return string.Format(
        CultureInfo.InvariantCulture,
        "DKIM-Signature: v=1; a={0}; s={1}; d={2}; c=simple/simple; q=dns/txt; h={3}; bh={4}; b=;",
        this.hashAlgorithmDkimCode,
        this.selector,
        this.domain,
        string.Join(" : ", this.eligibleHeaders.OrderBy(x => x, StringComparer.Ordinal).ToArray()),
        bodyHash);
}

You can see here that I’ve got some instance variables that were set in our IDkimSigner implementation’s constructor, such as the hash algorithm to use, the selector, domain, headers to include in the signature, and so on. We also insert our recently-computed hash of the body here.

You can also see that I’m using “simple” body canonicalization and “simple” header canonicalization. The DKIM spec gives us a few options in determining how the message is represented for signing and verification purposes. For the “simple” body canonicalization, it means “exactly as written, except for the weird rule about multiple newlines at the end of the body”. For the “simple” header canonicalization, it means “exactly as written, whitespace, newlines, and everything”.

There is a “relaxed” canonicalization method, but it’s more work, since you have to munge the headers and body into a very particular format, and I didn’t feel like writing a MIME parser.

Extracting “canonicalized” headers

The third step is to get a list of the canonicalized headers. In the constructor, I accept a list of headers to sign: From, To, Message-ID, and so on. (From is always required to be signed.) Then I use parsing code similar to that used in the CanSign() method and build a list of of the raw headers. The only real gotcha to watch out for is that headers can be wrapped onto more than one line, and since we’re using the “simple” canonicalization algorithm, we’ll need to preserve those whitespaces and newlines exactly as we extract them from the stream. Then I sort the headers alphabetically, since that’s how I specified them in the GetUnsignedDkimHeader() method specified above.

Signing the message

The logic behind signing the message is not that difficult. We smash all of the canonicalized headers together, add our unsigned DKIM-Signature header to the end, and compute our signature on this. Then we append the signature to the b= element, previously empty, of our DKIM-Signature header:

private string GetSignedDkimHeader(
    string unsignedDkimHeader, 
    IEnumerable<string> canonicalizedHeaders)
{
    byte[] signatureBytes;
    string signatureText;
    StringBuilder signedDkimHeader;
 
    using (var stream = new MemoryStream())
    {
        using (var writer = new StreamWriter(stream))
        {
            foreach (var canonicalizedHeader in canonicalizedHeaders)
            {
                writer.Write(canonicalizedHeader);
            }
 
            writer.Write(unsignedDkimHeader);
            writer.Flush();
 
            stream.Seek(0, SeekOrigin.Begin);
 
            signatureBytes = this.cryptoProvider.SignData(stream, this.hashAlgorithmCryptoCode);
        }
    }
 
    signatureText = Convert.ToBase64String(signatureBytes);
    signedDkimHeader = new StringBuilder(unsignedDkimHeader.Substring(0, unsignedDkimHeader.Length - 1));
 
    signedDkimHeader.Append(signatureText);
    signedDkimHeader.Append(";\r\n");
 
    return signedDkimHeader.ToString();
}

The only gotcha here, which I lost a few hours to, is a weird quirk of the .NET Framework 3.5 implementation of the SignData() function of the RSACryptoServiceProvider class. One of the overloads of the SignData() function accepts an instance of a HashAlgorithm to specify the kind of hash to use. The SHA-256 implementation was added in .NET 3.5 SP1, but it was done in such a way that an internal switch statement used internally by the .NET crypto classes wasn’t updated until .NET 4.0 to recognize the new SHA256CryptoServiceProvider type. Some guy blogs about why this is, but what it essentially means is that if you pass a SHA256CryptoServiceProvider instance to the SignData() method on .NET 2.0/3.0/3.5/3.5SP1, you get an exception, and on .NET 4.0 you don’t. Since Exchange 2007 uses .NET 3.5 SP1, we have to use the recommended workaround of using the overload that accepts a string representation of the hash algorithm.

Writing out the message

The last step is to write out the message with our newly created DKIM-Signature header. This really is a simple as taking the output stream, writing the DKIM-Signature header, and then dumping in the entire contents of the input stream.

Getting a key to sign messages with

Let us take a brief interlude from our DKIM circles of hell and obtain a key with which we will actually sign the DKIM-Signature header we’ve worked so hard to create.

We need to generate an RSA public/private key pair: a public key to store in DNS in the format required by the DKIM spec, and a private key to actually sign the messages with. The nice folks over at Port25 have a DKIM wizard that does exactly that.

It’s smashingly simple–just enter your domain name (say, “example.com“), a “selector” (say, “key1“), and select a key size (bigger is better, right?). The “selector” is a part of the DKIM spec that allows a single signing domain to use multiple keys. For example, you could use a key with selector name “newsletters” to sign all of the crap newsletter e-mails that you send out, and another key with selector name “tx” to sign all of the transactional e-mails that you send out.

It then spits out the syntax of the TXT records that you need to add to DNS for that selector:

key1._domainkey.example.com IN  TXT     "k=rsa\; p={BIG HONKING PUBLIC KEY HERE}"
_domainkey.example.com      IN  TXT     "t=y; o=~;"

The first record is where the public part of the key is stored. Whenever a mail transport agent sees one of our DKIM-Signature headers with a selector of key1, it’ll know to go hunting in DNS for a TXT record named key1._domainkey.example.com and pull the public key for verification from there.

The second record is part of the older DomainKeys specification and it is not strictly necessary. As written here, it means that we’re in testing mode (“t=y”)–that is, don’t freak out if you see a bad signature because we’re still dicking around with the setup of our implementation–and that not all messages originating from this domain will be signed (“o=~”)–maybe we won’t bother signing our newsletter e-mails, for example.

We’ll also have the private key specified in a format similar to the one below:

-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBANXBbZybdmjKDTONFVqAWXmGzR6GSZX5LV3OF//1jRz7dzGWTCKK
jembqBxqhr0Y2ua2l4D4EZi6FwDmdqgLS6MCAwEAAQJAD4qhypovEM1oClB+tfbR
Cpn3ffmrjgDxAHoEmrKi0PGBn8fumW22bad2tmrAjWWTVmeXJvQyEy1awq0M2PMR
0QIhAPEnqivb5dKZbTeKhiF4c6IUHfwEq8wNf2LWZvdH3ROrAiEA4un604mDss4Q
qAVEx686pUttfWyJrYkcZ/tx7kOoL+kCICEysqyDAypw0KY6vahR6qk/V7lf8z6O
BSFYHqigDgEtAiEAsK9r5UcQSyv1AD+J/MpOqeJ/kMfwtDUs7zJ01gfMb/ECIQDg
8d/XVJDi4Cqbt4wfcHZxADAgqyK8Z5M69fBecnExVg==
-----END RSA PRIVATE KEY-----

One thing I have glossed over in the code discussion until now was how that this.cryptoProvider instance that actually computes the signature got created.

We’ll need to read this key and load it into the cryptography classes used by the .NET Framework and by Windows to actually sign mail messages and get that this.cryptoProvider instance. Surely there is a simple API for this, yes?

Instancing a CryptoProvider

One problem is that the documentation in MSDN for the CryptoAPI is bad. I say “bad” because it certainly seems like .NET and Windows don’t expose native support for processing a PEM-encoded key, and if it does, well, I couldn’t find the documentation for it. Instead, the RSACryptoServiceProvider prefers to store its keys in an XML format that nothing else in the world seems to use.

This means that our implementation is so close to being finished that we can almost taste it, but now we have to complete a side quest to actually read our damn key and get an instance of the RSACryptoServiceProvider. Or, we could generate a certificate ourselves and store it in the Certificates MMC snap-in, but why should we have to do that? I’d rather just plop the damn key in the application configuration file like the rest of the goddamned world does it, “secure container” my ass.

We can thank the moon and the stars that some guy has written a PEM reader for us. How does it work? I have no idea, but I tested it on several keys and it seemed to work fine, which is good enough for me. I tossed this code into a static CryptHelper class, and now getting an instance of the RSACryptoServiceProvider is as simple as

this.cryptoProvider = CryptHelper
     .GetProviderFromPemEncodedRsaPrivateKey(encodedKey);

Loading the routing agent into Exchange Server 2007

I took all of this code and then added boring administrative stuff like logging and moving some hardcoded values (such as the PEM-encoded key, the selector, and the domain) into the usual .NET App.config file mechanism.

Installing the agent on the Exchange Server is surprisingly simple. After compiling the project and futzing with the configuration file, we just copy the DLLs and configuration file to a folder on the Exchange Server, say, C:\Program Files\Skiviez\Wolverine\DKIM Signer for Exchange\.

Then we launch the Exchange Management Shell (remember to right-click it and “Run as Administrator”) and execute a command to tell Exchange to actually register our agent:

Install-TransportAgent 
     -Name "DKIM Signer for Exchange" 
     -TransportAgentFactory "Skiviez.Wolverine.Exchange.DkimSigner.DkimSigningRoutingAgentFactory" 
     -AssemblyPath "C:\Program Files\Skiviez\Wolverine\DKIM Signer for Exchange\Skiviez.Wolverine.Exchange.DkimSigner.dll"

followed by

Enable-TransportAgent -Name "DKIM Signer for Exchange"

Interestingly, there will be a note telling you to close the Powershell window. It is not kidding. For some reason, the Install-TransportAgent cmdlet will keep a file handle open on our DLL, preventing Exchange from actually loading it until we close the Powershell window.

To make it actually work, we need to restart the Microsoft Exchange Transport service. I’ve found that restarting the Microsoft Exchange Mail Submission right after that is a good idea; otherwise, there can be a short delay of about 15 minutes before people’s Outlooks attempt to send outbound mail again.

Testing the implementation

To make sure things are actually working, Port25 comes to the rescue with their verification tool. You just send an e-mail to check-auth@verifier.port25.com and within a few minutes, they’ll send you an e-mail back with a boatload of debugging information. If it’s all good, you’ll see a result like the following:

Summary of Results
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

(What you’re looking for is the “pass” next to the DKIM check. The DomainKeys part being neutral is OK, since DomainKeys is the older standard and we’re choosing not to implement it.)

Conclusions and Delusions

I’ve been using this code for a few weeks now and it seems to work fine–the messages that I’ve sent through the server to Port25 and my Yahoo! test account all end up showing the DKIM passing. The usual “it works on my machine!” disclaimers apply, however, as I’m sure there are myriad configuration differences in Exchange that could this not to work. Bug fixes are welcome, but don’t come crying if it sends all of your e-mail to that big junk folder in the sky.

And thanks to some blowhards at Yahoo!, the world now has a public domain implementation of DKIM signing for Exchange to play with.

And in case you’re curious–after doing all this work to set up DKIM and participate in the Complaint Feedback Loop at their suggestion–their answer is still “no,” without elaboration. When Yahoo! finally goes under, I won’t be one shedding a nostalgic tear.

There are some unit tests, but they do have our private key in them, and I couldn’t be bothered to siphon those out. The code below is just the bits that do the actual signing.

Download the code used in this article.

First, it was FedEx

It has been 45 days since I reported to FedEx that the new SmartPost integration offered by their Web services simply does not work when an EPL2 label is requested:

I Hate Software, Part 1

The problem is that the USPS Delivery Confirmation barcode does not print when an EPL2 label is requested. That’s because in the EPL2 document that FedEx sends back, quotation marks in the barcode command are not properly escaped:

A90,473,0,3,1,2,N,"ZIP - USPS DELIVERY CONFIRMATION e-VS"
B50,535,0,1,3,4,175,N,""F1"42023221"F1"0000000000000000000000"

That should read "\"F1\"420..." for it to print correctly. (The ZPLII version of the label works fine.)

I understand that larger corporations have fixed software release cycles and different bug triage mechanisms, but having to through three levels of support and waiting 45+ days to simply say “this simply doesn’t work please add a backslash” is somewhat frustrating.

Then, there was UPS

Similarly, I’ve discovered a recently-introduced error with the UPS Web services. If you request a 4″ x 6″ EPL shipping label, then the UPS Web service will happily ignore request and send back a 4″ x 8″ label instead. That’s because they’re sending back the wrong width and height settings in the label response:

q795
Q1600,24

That would be 795 dots / 203 dpi == 3.91″ wide (OK) and 1600 dots / 203 dpi == 7.88″ high (hmm, not what I asked for). (The ZPLII version of the label works fine.)

Finally, SmartFTP sent me over the edge

Since updating to the latest version of SmartFTP, I found myself frequently being unable to connect to the Skiviez private FTP server that is used to manage software updates, e-mails, and product catalog images. It would fail about 80% of the time with

[13:51:38] 234 Using authentication type TLS
[13:51:38] SSL: Error (Error=0x80090308).

I’m sure that this means something. And even if I knew what it was, sometimes it would just work without me changing anything (about 10 percent of the time). Further still, I hadn’t touched the FTP server for some time.

That’s when I noticed in the SmartFTP change log:

FTP: Completely rewrote SSL layer

Sigh. Downgraded to the version prior to that changelog entry, and it works fine.

Conclusions and delusions

I have certainly done my part to contribute buggy, crappy software to the world. I continue to spew out more buggy, crappy software with each passing day. But it is extra depressing and disheartening to know that I, some idiot working at a small company, can run across such simply-does-not-work bugs (and, in FedEx’s case, never-actually-worked-ever bugs) in software produced by large corporations and used by presumably hundreds to thousands of people around the world.

We happen to like our system

So we’ve branched out and offered fulfillment services to other merchants, as I’ve mentioned before. Essentially, other e-commerce stores or merchants can store their items in our warehouse and then transmit orders as “fulfillment requests” to us. We either ship the request or mark it as invalid with a little message (“item has insufficient stock, delivery point validation failed, tax identification number required, etc.”). If they want to re-attempt, then they re-submit an entirely new fulfillment request.

This simple model has worked surprisingly well: the system doesn’t track “orders” (since each merchant handles backordering and cancellations differently), and while it does maintain an inventory log, it doesn’t track the cost of goods in inventory or anything like that, since that is not necessary for us to do our job. (Instead, it tracks a whole boatload of other information that people don’t normally think about, like HS codes or unit weight.)

Some of our merchants need some hand-holding

We have an API so merchants can integrate with our system. And so I’ve written a few plug-ins for Magento and osCommerce that auto-transmit their orders as fulfillment requests, sync outbound shipments back, and deduct inventory from their systems (since we are the authoritative inventory count). This works great for their retail businesses.

We have a merchant that we’ve been doing retail business with for some time who now wants to do wholesale stuff with us. He wants a sales order/invoicing/inventory management solution, and it needs to be able to track multiple inventories across multiple warehouses (our integration, if any, would only be adjusting inventory for a particular warehouse). He wants to enter sales orders remotely, press a button that shows him how much is ordered so he knows how much to make, have that manufactured and sent to us, click another button to transmit the sales order as a fulfillment request to us once it’s in stock, and then have us sync back with the shipment info, creating an invoice.

“Shouldn’t they have had these features prior to switching to you guys?” you ask. Well, yes. In this case, all of these features were provided in an all-in-one system provided by their previous fulfillment warehouse. They have since learned their lesson about keeping all of their data in the hands of a third party because when that relationship went south, so did their access to their own data.

We would rather not add these features to our system. Since all merchants have different ways of handling backorders, pre-orders, cancellations, cost of goods sold (FIFO, LIFO, average, priority), we’ve been maintaining the position that–unlike our competitors–our system is essentially feature complete, since it’s ours and does what we need it to do to ship things out. The features that I’ve mentioned should be things that the merchants are keeping track of themselves–since that’s their business–and integrate with our system via the programming API. While an argument could be made that our system would be abso-freaking-fantastic for merchants who need an all-in-one software and data solution (yes, it certainly would), the reality is that our competitors have an outsourced team of software developers, and we are a small business working in an area that is tangential to our core business as a result of the “new economy” that has a software development team of just one person (me) and can’t even begin to dream of hiring any more until we start seeing some serious cash flow.

In any case, to land this deal, we need to find a system for this merchant, and fast, because there are some important trade shows coming up.

Welcome to marketing Hell

Now for the rant, because you would think that these requirements are not exotic:

• Let salespeople enter sales orders remotely.
• Keep track of inventory in multiple locations, and track the cost of inventory.
• Provide integration hooks so the user can send orders to the warehouse and so the warehouse can send shipment data back.
• Keep track of sales-order-to-invoice conversions and payments.
• Provide reporting features.

QuickBooks 2010: Same as last year! Now with more shininess!

You might take a look at integrating with QuickBooks, but you’d realize that once you’ve penetrated the marketing speak that the software in 2010 is essentially no different in terms of fundamental feature set as it was in 2006, and that it doesn’t support inventory in multiple locations and doesn’t scale well. In fact, QuickBooks performance once you start approaching 10,000 SKUs is so bad that they sell an “enterprise” version that essentially–aside from some fine-grained access permissions–has no added features other than the feature of not crashing when dealing with large lists of information.

We could pay another couple of thousand dollars for Fishbowl inventory, which would add multiple location support to Quickbooks, but then we’d have created a Rube Goldberg machine straight out of the gate, with me synchronizing with Fishbowl which is then synchronizing with Quickbooks. I’m sure nothing would go wrong there. That would be insane; we might as well just stick a few fax machines into the sync process and call it an insurance company.

A gap in the market

The reality is that there is a huge gap in the marketplace between merchants who are moving $200k or less per year–just use commercial off-the-shelf (COTS) QuickBooks, you can do most things manually and use your e-commerce system’s native order management functions–and merchants who are moving $5m or more–just use SAP or some other enterprisey software. If you’re in between, like we are and like the merchant that I’m researching for is, the options available to you are not pretty.

I’m not sure why this is. All I can think of is that perhaps companies historically did not spend much time in this space–they either stayed small or had venture capital to acquire the big boy systems and grow quickly. People aren’t exactly lending money anymore, so I suspect that this is a segment that is only going to grow.

If you try to look for COTS software in this segment, you’ll never find the feature matrix that you need:

• inFlow Inventory is pretty, but offers no integration features, as if an entire business could be run out of one app, and doesn’t offer Web access.
• WorkingPoint is Web-based but doesn’t offer inventory tracking in multiple locations or an integration API.
• QuickBooks has a Web-based extension that lets QuickBooks understand multiple inventories but costs thousands of dollars, assumes that the company owns its own warehouse (that is, needing picking/packing/shipping capabilities), and still does the same style of synchronization as Fishbowl does. You’d think Intuit would just add the @#$#@ feature to QuickBooks itself!

No COTS to sleep in

The market seems to have determined that people in this segment have outgrown COTS software and need some consulting help. So any Web sites that advertise products will be full of pages and pages of impenetrable marketing bullshit that use obnoxious acronyms like ERP, CRM, MRP, and WMS, promise the moon, and coyly make no reference to pricing or contract requirements so you can’t even tell if you’re dealing with the right league of product, when the reality is at the end of the day you could look at two or three screenshots and the SDK’s API and immediately tell if the product fit your needs.

Instead, I notice a disturbing trend of “pretty Web site, crap product,” such as Sage’s Simply Accounting, which certainly appears to have an impressive array of features but in reality doesn’t even know the difference between a sales order and an invoice. You can try going to Microsoft’s Dynamics site, but good luck figuring out what the difference between Dynamics AX, Dynamics CRM, Dynamics NAV, and Dynamics GP are: you’ll be told to contact your “Microsoft Dynamics solutions representative” for help. At that point, you’re thinking “Microsoft solutions representative? Who said I committed to Microsoft?! I’m just trying to figure out what in the blue hell your product even does.”

If you do find a vendor that maybe sorta-kinda-hard-to-tell meets your solutions, then you can expect days of scheduling WebEx teleconferences and meetings and run-around with your “account rep” so that they can determine how much you’re worth and willing to pay so that they can charge you a completely different amount than what they charged Bob next door for the same services and bits. Trying to extract “$X/user” and “the login starts working on MM/DD/YYYY” and “the developer get a demo account so if you can know if this is feasible” answers from these people seems to require a hammer in one hand and their genitals in the other. We both know that to add a new account, they’re pressing a button that says “they really bought into that ‘enterprise’ crap” and poof! a new account is created. Let’s quit pretending that the world’s carbon footprint has increased ten-fold by us merely asking to be on the platform.

Trying to extract technical capabilities from these salespeople is nigh-impossible either. I think part of the problem is that they seem to actually believe that the features that they are promising really exist, when in reality I just need them to show me what the data dictionary looks like and how the session needs to be handled and then I can tell for myself whether or not my scenario is actually supported. Instead? I’m waiting on a “discovery session” teleconference with an “engineer” tomorrow.

Conclusions and Delusions

It has to be easier than this. No wonder there are so many not-invented-here software solutions in the world today–custom crap that barely works at home may yet indeed be better than generic crap that you have to waste hundreds of dollars on in productivity time and research before you even get it in your hands and realize that it is also crap, just with a maintenance contract.

If it takes a consultant to help people decide what software to buy, or which of your products is right for them, or whether or not your product even applies to their problem domain, then your marketing simply does not work.

The reason? Oh, it just does that sort of thing every 25 days or so.

Behind that magic <ENTER> press, though, a lot of magic is happening behind the scenes:

• I call our payment processing service to cleanse the address into a format suitable for mailing.
• I call the FedEx Web service to determine if the address is residential or commercial.
• If standard shipping, I call the USPS XML API, FedEx SOAP API, and UPS XML APIs simultaneously to find the cheapest rate and best transit time.
• I call the FedEx or UPS API directly to print a shipping label immediately OR invoke a Stamps.com COM object to pop up an extra dialog, which the user then presses enter to spit out the label interminably slowly.

It’s the last part that has been the bane of my existence. USPS does not offer any online mechanism to print postage-paid shipping labels in an automated fashion. You get to choose between Endicia and Stamps.com. While Endicia has an XML API that functions nearly identically to those offered by UPS and FedEx, it’s ridiculously expensive, and Stamps.com has no such API. Instead, I have to interact with a COM object that is popping up dialogs and forcing the warehouse worker to worry about buying postage when it runs low.

To add insult to injury, Stamps.com doesn’t understand EPL2 and instead sends its labels as gigantic GDI documents to the label printer, giving us print times of nearly 15 seconds for a 4″ x 6″ label when going over parallel. Compare to less than 2 seconds for a label sent as an EPL2 document. But Stamps.com is cheap.

And there are no alternatives.

Sure, FedEx has SmartPost and FIMS and UPS has Mail Innovations. These are services where we would put our non-postage paid parcels into a trash bag that FedEx or UPS would pick up, sort, route through their system to a post office near the parcel’s final destination, and affix postage there and let it enter the USPS system by mailing it for us at the “last mile post office.” But Mail Innovations is worthless (a statistically significant number of our packages shipped through Mail Innovations to Canada were tampered with), FIMS is mind-boggingly slow (up to 60 days to send a package to Finland, and FedEx reps seem to have no idea that it is actually taking that long for their service to actually work), and SmartPost is expensive unless you’re talking enterprise-level volumes, which I am not. Plus, if a package disappears, there’s plenty of finger pointing, trust me.

And sure, there’s Click-N-Ship at the USPS’s Web site. But Click-N-Ship provides zero integration capabilities and as such is useless when you’re talking more than, say, 3 packages per day. It prints lots of instructions that you just end up throwing away, uses PDF for label output (so you can’t really automate printing unless you want to rig up a harebrained routing-through-Ghostscript scheme), and is really intended for home users.

There’s permit imprint, which would work if we were mailing 200 identically weighted parcel post packages, but that is never going to happen. Some widgets are bigger than others.

And sure, there’s that moronic new commercial where they show the USPS employees touting the simplicity of the new priority mail flat-rate boxes for businesses. You still have to stamp them–that’s half my problem. And using a flat rate box means that you’re throwing money away a good 50% of the time. We’re not stupid. It does not cost $9.85 to send a 12″ x 9″ x 6″ box from Virginia to North Carolina. And where’s my delivery confirmation barcode? I’ll have to affix that too and record the number–how is this saving me any time? I want to throw my remote at the TV every time this commercial comes on.

So, I don’t understand why the post office doesn’t offer a postage Web service like UPS’s or FedEx’s XML shipping services, unless it’s just too much competition with their own licensees. The Visa business model does not make sense for everyone.

I don’t understand why the post office doesn’t sell laser printer versions of its CN-22 and CP-72 customs forms. We had to have our own specially made, and use the USPS’s API to generate the actual form. (More on this later.)

I don’t understand why the USPS has an address verification API available through their Web tools but doesn’t let anybody use it. (By anybody, I mean that only non-profits can use it. Not even local governments can use it.) Shouldn’t it be an obvious service to help customers verify that they’re shipping to a valid address? Well, perhaps I do understand, because there’s lots of money to be made selling CASS certification to vendors who then can charge ridiculously high fees for what is, at its core, just a giant database table that gets updated once a month. By giant, I mean it fits on a CD-ROM. CASS is beyond stupid.

I don’t understand why the USPS doesn’t get its act together and offer business accounts for its Priority mail and higher services. With the USPS, indicia is like currency: a misprinted label means you lose money, whereas with FedEx and UPS, a misprinted label means you throw it in the trash. But far, far more importantly, this means that businesses can’t leverage the “float” when using USPS: the shipping cost is incurred immediately instead of on net 30 terms.

I don’t understand why the USPS doesn’t allow scheduling a daily pickup for businesses. Sure, there’s the insane carrier request API, which we use, and have to remember to update every few months until we max out the pickup requests again. But it’s particularly insane that it requests the number of packages that will be picked up and that mail carriers treat this as some sort of contractual obligation. I am setting a date in the future–I can give you an estimated average, but please don’t complain when I have 90 packages instead of 50. I can’t see the future to know how many orders we will have to process that morning.

I don’t understand why the ICCC is staffed by weasels. When an API is not on your test server but you require us to test against the requested API on the test server before getting access to the production server, then something is really wrong, and someone at the USPS needs to be fired. Right now.

If anyone at the post office is listening, one of the greatest barriers to using the post office directly is a lack of integration points. Quit moaning about the number of parcels dropping and make it easier to actually ship packages, and maybe we’ll be more apt to use it.

Update 11/11/2009: See my comment below for more detailed information since this post was written.

I can't log in!

What has happened here is that my OpenID provider, Verisign Labs, is down for some reason while StackOverflow, the site that I’m trying to log on to, is still up.

A single point of failure

While this isn’t a big deal for a site that I use recreationally, it is kind of a big deal in principle: all sites that use OpenID are inaccessible to me, since I only have this single OpenID provider. It is a single point of failure.

Sure, I could have multiple OpenID accounts, such as a “backup” OpenID account at another provider, but this defeats the purpose of OpenID, of having a central point of authority regarding my online identity. If I have to create a second OpenID provider, I might as well just make a native username and password at the site that I’m trying to log into. I could also be my own OpenID provider, but I shouldn’t have to do that, and the average OpenID consumer is not going to know how to do that, anyway. It’s faster just to create a “native account” at the Web site that I’m trying to access and be done with it.

An unnecessary third-party dependency

For any e-commerce business, it’s usually unwise to take on any third-party dependency. Speaking from experience, that third-party will go down for some reason at some point and you will be the one fielding angry customer support calls for something that isn’t your problem. Even worse, if something as critical as authentication through OpenID goes down, then you’re losing sales, and that means that you’re not doing your job.

If you’re going to add OpenID to an e-commerce Web site or a “mission-critical” Web site, then you should implement it as a red-headed step-child, “convenience only” implementation; this way, your users can fall back onto their “native credentials” (those stored in your database) when their preferred OpenID provider explodes. In this way, OpenID is a shortcut, not a single authoritative source, much how like customers use PayPal’s Express Checkout to save time when entering payment and addressing information during checkout. If PayPal’s Web Service API goes down (and believe you me, it’s gone down several times this year alone), customers can still begrudgingly use the “native checkout” to complete their purchase. Some lost conversions due to the inconvenience–and for incorrectly faulting the integrity of your Web site for the third party’s failure–but at least the sale is still actually possible. If OpenID is your only authentication mechanism, and a user’s OpenID provider goes down, then that user is simply screwed. The onus should not be on the consumer to provide a backup authentication mechanism to your Web site.

It’s just complicated

OpenID is still just too complicated. I’m not the smartest software developer in the world by any means, but if I can barely wrap my mind around it enough just to get my first account, then I am 100% certain that our customers would have problems with it:

• Why do I have to go to some third-party provider to sign in? That doesn’t make any sense.
• Why should I trust that third-party provider? I’m not even sure that I trust you!
• I still don’t know which third-party provider to use! Oh well, I guess I’ll use Verisign, at least I’ve heard of them.
• What happens to my credentials when that third-party provider goes under?
• Can I call your business to reset my password when I forget it? I will anyway!

Even with our “native registration” of e-mail addresses and passwords, we had to use an Amazon.com style login form because users kept filling out the “new customer” form even when they already had an account:

Login screen. Amazon’s sign-in screen remains a model to be emulated, minimizing the common problem of new customers who try to log in without having registered. Amazon presents two questions in linear order: (1) “What is your email address?” and (2) “Do you have an Amazon.com password?” For the second question, users can select one of two radio buttons: “No, I am a new customer,” or “Yes, I have a password.” Many other sites present the new- and established-user sections side-by-side, and thereby divert new users to the established-user section through the magnetic attraction of type-in fields. — Jakob Niesen, useit.com

When our users have trouble navigating two fields and a radio button, you can imagine the hilarity that ensues when they are presented with multiple authentication mechanisms. Especially one that doesn’t ask for a password.

A leaky abstraction

While the idea of consolidating your online identity to a single source sounds like a good idea, it simply does not work in practice. Not even the United States government has a consistent picture of my identity: my identity is stored with the IRS, the Social Security Administration, the United States Postal service, the Virginia Department of Taxation, the Virginia Department of Motor Vehicles, and the City of Richmond’s Voter Registrar. These are all independent identity stores that, in general, agree that I am who I say that I am, but each identity is stored and updated independently of one another. Consider OpenID the RealID act of the Internet: what happens when your centralized identity gets screwed up? It’s an existentialist crisis and you just have to deal with it: identity is what other people say you are; it’s not what you say you are. Better to have multiple voices confirming your identity than just one.

Conclusions and Delusions

If you’re implementing a social-oriented Web site that is designed to integrate with Facebook or some of the Web-2.0-savvy consumers, then OpenID might make sense. Those nerds may have heard of OpenID and know how to use it, foibles and all. Until the dust settles on OpenID, though, I wouldn’t add it to a commercial site even if you paid for it: nobody has asked for it; it will cause problems.

My general recommendation would be to have the usual in house username and password mechanism that can be supplemented by OpenID. But realize that each alternative identification mechanism that you add runs the risk of customer confusion, bugs, security holes, and increased customer support. When in doubt, leave it out.

My boss at work used Adobe InDesign CS4 to create a new, 20-page printed catalog for our customers. He discovered that InDesign has this nifty “Export to SWF” feature whereby, insto-presto, the same document can be converted into a snazzy Flash version, complete with super fancy page-turning effects. Neat!

Unfortunately, the file size of the generated Flash file is enormous–at 7.4 megabytes, it definitely wasn’t something that we could realistically put on our Web site.

Seems easy enough to fix! Let’s look at the options dialog that InDesign pops up during the export process:

Words, words, words...

Awesome! “JPEG Quality” sounds like just what we need, so let’s set it to “Minimum” and export the Flash–oh, wait, it’s the same size: 7.4 MB. Go back and set it to “Maximum”–7.4 MB. Set it to “Low”–7.4 MB.

Um.

A few wasted minutes of Googling gives us this gem on Adobe’s Forums from the purported Product Manager for InDesign:

First and foremost, unfortunately there is a bug in the JPEG Quality setting of the InDesign CS4 SWF Export dialog. The bug is that no matter what value you choose from the pop-up menu (Minimum, Low, Medium, High, Maximum), they will all end up as Medium quality. The bad news is that this bug was not discovered (internally) or reported (by pre-release and CS4 customers) before we shipped CS4 (6.0.0), or the 6.0.1 dot release. The good news is that it will be fixed in the next dot release (6.0.2). I can’t provide a specific date for the next dot release yet.

In other words, the feature Simply Does Not Work. This is the type of programming mistake that I make in our backend applications at a small business, sure: but it is not the type of mistake that you would expect from a multi-million dollar software development powerhouse who has been doing this for decades and charges over $600 for the product. Ridiculous.

Try exporting to a source file

Great. So the fix is not available, and I have to get this stupid Flash file up before I can go home. Maybe I can use the other InDesign option to export to XFL. Then we can open up this file in Adobe Flash proper and sort out the images there.

That works great, but when exporting to XFL, InDesign doesn’t export any of the hyperlinks or page animations, which is kind of half the point of the exporting feature. The Help manual even documents this as if this were useful behavior. Obviously InDesign had to generate some source file to generate the enormous SWF; why can’t I have that file, and not this half-baked XFL thing?

Now I’m getting irritated

Okay. Now what? Perhaps I can decompile the enormous SWF file so that I can access the image resources and lower the quality of the JPEGs. If you Google this long enough, you will find that generally all third-party Flash products (especially those relating to decompilation) fall into three categories:

• outdated open source software that probably never worked;
• poorly written shareware; and
• Trojan horses.

I finally downloaded a demo of Eltima’s Flash Decompiler Trillix. (What the hell does “Trillix” mean?) This application looks pretty, and it does let me see the image resources on the SWF and adjust their quality. But saving is disabled in the demo.

After roughly calculating out my hourly pay, I’d by that point determined that I had wasted much more of the company’s money than the $67 licensing fee, so I broke down and paid three tanks of gas for a magic license key number. I finally degraded the quality of all of the images and got the file size from 7.4 MB to 1.0 MB, which is good enough for us. Hopefully by the time the next catalog rolls around, Adobe will have released the 6.0.2 update.

The pièce de résistance

My boss asked me if I could use the new tool to just convert the SWF to a plain old FLA source file, the idea being that in the future we’d use this file as a template and bypass InDesign entirely.

Yep, you guessed it–the Eltiva software crashes with a bad pointer reference whenever it tries to decompile the scripts in the InDesign-generated Flash file. Why? Who knows–I don’t.

Conclusions and Delusions

This rant has some purpose: hopefully, it will help prevent some future poor sap from spending nearly half a day chasing his tale. The lessons learned: avoid the Export to SWF feature in Adobe InDesign CS4, and, if you must use it, use a decompiler to manually fix up the image resources.

This is why we drink.

Let’s say that you’ve gone the extra mile and you’ve implemented your own custom trace listener. Even better, let’s say that you’ve created a trace listener that extends from TextWriterTraceListener. After all, you’re probably logging to a text file, but perhaps you wanted to change the format around a little bit.

Recall that the way for us to specify the location of our trace logging file via configuration is to use the initializeData attribute as in the following example:

<system.diagnostics>
  <sharedListeners>
    <add
      name="Listener:ApplicationText"
      type="Skiviez.Hedgehog.Model.AlignedTextWriterTraceListener, Skiviez.Hedgehog.Model"
      initializeData="Media\Logs\Hedgehog.txt"
      traceOutputOptions="ThreadId, DateTime, ProcessId" />
  </sharedListeners>
</system.diagnostics>

Right. Looks pretty sane: I’ve got my custom trace listener type (we’ll assume that AlignedTextWriterTraceListener simply extends the built-in TextWriterTraceListener), I’m telling it to log to Media\Logs\Hedgehog.txt (relative to my working directory, I would presume), and I’m passing in some output options and giving it a name.

Nothing we have done would here would leave us to believe that we have broken the way TextWriterTraceListener works. We have, but it won’t be apparent until we try to run this in a ASP.NET Web site.

What have you got against Web sites?

So when we run it an ASP.NET Web site, we will note that while we receive no error, we also see no log file sitting in the Media\Logs directory as we specified. I can understand how the tracing code probably swallows exceptions–so as to not take down your entire Web site or application because some hoo hah misconfigured a tracing statement in the configuration file–so let’s assume that we’ve screwed something up. Check the permissions on the Media\Logs directory? Check. Check the working directory? Hmm.

The working directory of ASP.NET applications is usually strange, somewhere in the %SYSTEMDIR% area–that’s because your code is usually running from some temporary location where your ASP.NET Web site was compiled just-in-time to serve the first request. So our trace listener is trying to be relative to a highly privileged directory–not relative to our Web application’s root directory–and obviously the log file can’t be created there, failing silently.

Okay, that makes sense. But that seems astonishing somehow.

You’re not going crazy

The reason why this seems astonishing is that if you replace your configuration to use the standard TextWriterTraceListener instead of your custom type that extends from it, then the log file will magically appear in the expected location, relative to your Web application’s root directory, and not relative to the system directory.

<system.diagnostics>
  <sharedListeners>
    <add
      name="Listener:ApplicationText"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="Media\Logs\Hedgehog.txt"
      traceOutputOptions="ThreadId, DateTime, ProcessId" />
  </sharedListeners>
</system.diagnostics>

Okay. So you double-check your custom type to make sure you’re really not doing anything strange with configuration. Which of course you aren’t–you’re just forwarding constructors to the base TextWriterTraceListener‘s constructors. You’re not supposed to care how they actually work.

So why is the path that the initializeData attribute seems to be relative to changing wildly between the two types?

Enter the Reflector

So after spelunking through Reflector for a little while, we stumble across this little gem in the TraceUtil‘s class GetRuntimeObject() method:

internal static object GetRuntimeObject(
	string className, 
	Type baseType, 
	string initializeData)
{
	// ... snip ...
 
	if (string.IsNullOrEmpty(initializeData))
	{
		if (IsOwnedTextWriterTL(c))
		{
			throw new ConfigurationErrorsException(
				SR.GetString(
					"TextWriterTL_DefaultConstructor_NotSupported"));
		}
		ConstructorInfo constructor = c.GetConstructor(new Type[0]);
		if (constructor == null)
		{
			throw new ConfigurationErrorsException(
				SR.GetString(
					"Could_not_get_constructor", 
					new object[] { className }));
		}
		obj2 = constructor.Invoke(new object[0]);
	}
	else
	{
		ConstructorInfo info2 = 
			c.GetConstructor(new Type[] { typeof(string) });
		if (info2 != null)
		{
			if ((IsOwnedTextWriterTL(c) && 
				(initializeData[0] != Path.DirectorySeparatorChar)) && 
				((initializeData[0] != Path.AltDirectorySeparatorChar) && 
				!Path.IsPathRooted(initializeData)))
			{
				string configFilePath = DiagnosticsConfiguration.ConfigFilePath;
				if (!string.IsNullOrEmpty(configFilePath))
				{
					string directoryName = Path.GetDirectoryName(configFilePath);
					if (directoryName != null)
					{
						initializeData = Path.Combine(
							directoryName, 
							initializeData);
					}
				}
			}
			obj2 = info2.Invoke(new object[] { initializeData });
		}
		// ... snip ...
	}
		// ... snip ...
}
 
internal static bool IsOwnedTextWriterTL(Type type)
{
	if ((typeof(XmlWriterTraceListener) != type) && 
		(typeof(DelimitedListTraceListener) != type))
	{
		return (typeof(TextWriterTraceListener) == type);
	}
	return true;
}

Now, I know that I for one have written some strange and hackish code in my time, but whoever wrote this function was obviously wearing his silly pants that day. Here’s what the function does:

If the following are ALL true:

• The trace output listener type is one of the three concrete types explicitly provided by Microsoft (and not an inheritor of that type)
• The initializeData attribute is provided
• The initializeData path seems to be relative

Then the function mangles the initializeData attribute by combining it with the path of the configuration file that contained the attribute. It then instances the trace listener by providing this faked-just-in-time full path as the fileName argument.

So if I had configured a TextWriterTraceListener in my configuration file, and my configuration file were in C:\Foo, and I specified Media\Logs\Hedgehog.txt as the initializeData parameter, then the TraceUtils class would give the TextWriterTraceListener instance the value of C:\Foo\Media\Logs\Hedgehog.txt as the fileName argument.

But if I had configured anything but TextWriterTraceListener, XmlWriterTraceListener, or DelimitedListTraceListener as my output listener in my configuration file, and my configuration file were in C:\Foo, and I specified Media\Logs\Hedgehog.txt as the initializeData parameter, then the TraceUtils would kindly tell me to screw myself, pass the TextWriterTraceListener instance the value of Media\Logs\Hedgehog.txt as the fileName argument, which the instance would then match with the current working directory, which is something insane like C:\Windows\System32\Temporary ASP.NET Files if we’re talking about an ASP.NET Web site, and so I would end up with C:\Windows\System32\Temporary ASP.NET Files\Media\Logs\Hedgehog.txt, which is about as useful as sticking my thumb up my butt.

So just by extending a class, we have completely broken the way one of its configuration attributes works. That’s pretty astonishing when it comes to object-oriented design.

An ugly work-around

So I wasn’t about to specify a full path in initializeData because I don’t enjoy munging a configuration value on every single machine that I happen to build and deploy my solution to. As this is written in the .NET Framework’s internal TraceUtils class, I simply cannot avoid this behavior when extending a Microsoft-provided type.

So if I want to be able to specify a relative path in initializeData with a type that extends from TextWriterTraceListener and have it mimic the TextWriterTraceListener‘s initializeData relative-path-resolution behavior, my only recourse is to copy TextWriterTraceListener out of Reflector and make it my own type with my own relative-path munging semantics.

I change the constructors of my new type, which I’ve christened TextWriterTraceListener in my own namespace, as follows:

/// <summary>
/// Initializes a new instance of the TextWriterTraceListener class, 
/// using the file as the recipient of the debugging and tracing output. 
/// </summary>
/// <param name="fileName">The name of the file the 
/// TextWriterTraceListener writes to. </param>
public TextWriterTraceListener(string fileName)
{
	this.fileName = MungeFileName(fileName);
}
 
/// <summary>
/// Initializes a new instance of the TextWriterTraceListener class 
/// with the specified name, using the file as the recipient of the 
/// debugging and tracing output. 
/// </summary>
/// <param name="fileName">the fileName to output to</param>
/// <param name="name">the name of the trace listener</param>
public TextWriterTraceListener(string fileName, string name) : base(name)
{
	this.fileName = MungeFileName(fileName);
}

and the gloriously-named MungeFileName() method is

/// <summary>
/// Munges the file name such that if it is a relative path, we go
/// relative from the configuration file and not from the current working
/// directory. This makes things work as expected on ASP.NET sites and
/// makes other applications similarly work with non-astonishing
/// behavior.
/// </summary>
/// <param name="fileName">the file name to munge</param>
/// <returns>the munged filename</returns>
private static string MungeFileName(string fileName)
{
	string configPath;
	string mungedFileName;
 
	mungedFileName = fileName;
 
	if (fileName[0] != Path.DirectorySeparatorChar &&
		fileName[0] != Path.AltDirectorySeparatorChar &&
		!Path.IsPathRooted(fileName))
	{
		ConfigurationSection configSection;
 
		configSection = (ConfigurationSection)ConfigurationManager
			.GetSection("system.diagnostics");
		if (configSection != null)
		{
			configPath = configSection.ElementInformation.Source;
 
			if (!string.IsNullOrEmpty(configPath))
			{
				string directoryName;
 
				directoryName = Path.GetDirectoryName(configPath);
 
				if (directoryName != null)
				{
					mungedFileName = Path.Combine(
						directoryName, 
						fileName);
				}
			}
		}
	}
 
	return mungedFileName;
}

What I’m doing here is saying, “Is the path relative? Then figure out where my system.diagnostics configuration came from, and make myself relative to THAT directory.” This way, I don’t have to have different code paths for this class depending on whether it’s running in a traditional Windows application or an ASP.NET Web site–the initializeData is always relative to the configuration file and not the current working directory, which is what I want 99% of the time, and it’s probably what most people want, too.

Conclusions and Delusions

I think the primary lesson to take away from this is to avoid doing magic configuration and baby-sitting for a class’s constructors. I have a feeling that TextWriterTraceListener was implemented by somebody at Microsoft and sometime much later in the development process they realized that specifying a relative initializeData was difficult for ASP.NET applications. Being unable to alter TextWriterTraceListener without breaking compatibility for non-ASP.NET applications that may have already been built, they devised an unusual TraceUtils class to munge the parameters read from configuration before they were passed to a new instance’s constructor.

The result? It worked well if you stuck to the built-in classes, but sure was damn surprising if you created a custom type and eventually used it on an ASP.NET project.

Hope this helps someone out there. You’re not insane, after all.

Here’s the implementation for your convenience.

Currently, users are bewildered by a dizzying array of Not-Invented-Here update mechanisms. When I start Paint.NET (a program that I love and use often), it’ll irritatingly ask me to update as soon as I’ve started it. Adobe Acrobat will similarly pop up an insanely complex dialog at start up. Java has a process that sits there sucking up memory and displays a balloon in the system tray. Flash will display an owner-drawn dialog (that looks just like spyware, by the way) when I first log into my Windows session. And Apple will pop up its own updater dialog seemingly at random.

Then there are the processes that sit there in memory, sitting around twiddling their thumbs 99% of the time:

• GoogleUpdate.exe — Google Chrome’s updater
• jusched.exe — Java’s updater
• SoftwareUpdate.exe — Apple’s hog takes up 13 MB

This isn’t the individual developer’s fault. When it comes to pushing updates, you can either use a technology like ClickOnce (which is useful but in a very specific environment, namely deploying internal business applications), or you can roll your own solution.

This is stupid.

Windows and Microsoft Update finally got an integrated user interface in Windows Vista. It’s unremarkable, which is a good thing, because that means that I must not hate it. Imagine how wonderful it would be–both for the user experience and memory usage–if all applications could integrate with this same user interface. An MSI installer could register a URL that Windows could check for updates periodically. If updates are found, it could use the same BITS mechanism to unobtrusively download it in the background. And to verify integrity, it would verify that the update has been signed by the same key that was used to sign the original installation MSI, or it could be based on some sort of other trust mechanism. This way, all of the user’s updates would be accessible through one interface.

To make sure companies use it, make it a requirement for the Windows Logo program.

Why hasn’t this happened yet?