For a long time, Skiviez dealt with this problem in a way that I would say the vast majority of US-based e-commerce based retailers dealt with it: they added a country dropdown, turned the State field into a textbox, and made the State field no longer required. And if you are not obsessive compulsive like I am, then this works reasonably well enough: international customers have to shoehorn their address into the format presented, but the data can still get in there.

However, after 10 years in operation, this approach was beginning to show some deficiencies:

Magnetic attraction. There is, for lack of a better term, a magnetic attraction between users and text boxes. Even if the field is not required, users have a tendency to enter “N/A”, “.”, “?”, “none”, or some other placeholder value in fields that do not apply to them.

Address quality problems. With the above method, the bumpers are off–a user could type “cheese” into the postal code field, and we’d just have to accept it on their word that “cheese” is a valid postal code for the United Kingdom because the site wasn’t doing any sort of validation except for United States-based addresses. Shipping a parcel to another country only to have it returned or destroyed as a result of an invalid or insufficient address is an expensive issue.

Integration issues. To actually ship out international packages, we have to do deal with third-party carrier APIs (such as Endicia, FedEx, and UPS) to purchase postage and print shipping labels. Some of these APIs do perform more advanced validation on international addresses. This could cause problems because a poorly formatted address would make it nearly all of the way through our fulfillment process–order acceptance, picking, processing, and packaging–only to blow up with an error at the shipping desk. It’d be nice if we could catch problems earlier to avoid disrupting the shipping flow and eliminate the potential for misrouting the package.

A smarter system

With the October 2010 release of the Skiviez Web site (which was a pretty huge update), the Web site is now much, much smarter in how it asks for address information. For example, here’s the default address form for the United States, which is about what you’d expect:

But if you choose, say, Ireland in the countries drop down, the page refreshes and displays a new form:

The state drop down is gone (since Ireland doesn’t have states), the ZIP/postal code text box is gone (since Ireland does not use postal codes), and a new County text box is available.

Once an address is entered, it is formatted in the way a person native to that country would expect. For example, in a German address, the postal code comes before the city:

Taking it even further, if you select, say, Finland from the drop down, you get a form back that looks like this:

The postal code validates against the postal code format used in Finland, and that field appears before the Municipality field, since the convention throughout most of the EU is to write “{Postal Code} {City}” instead of the American “{City} {State} {Postal Code}” way.

Clearly, we’ve made a lot of improvements to address handling at Skiviez, and this simple input format actually has a lot of complexity behind it. So how does it all work?

Tweaking the existing database schema

At Skiviez, we were clearly working from a legacy database schema. The customer addresses table has been sitting there for 10 years, and we’re not about to change it any time soon. And while there is some internal cruft, the fields in the database more or less looked like the following:

This is a normal, relational schema. So how do we enhance this table so that it can store addresses from the dozens of countries that we ship to? That sounds like a tall order: surely the addresses of the world are all wildly different from one another, and we’ll have to go entity-attribute-value, throwing away the relational bit of our relational data store, to be able to accurately store this diverse information, right?

Well, it turns out that you can take these existing fields, relax away any NOT NULL constraints, and rename them to field names that are operating at a higher level of abstraction. (We didn’t actually rename the fields in the database, as this would have broken old code, but the new field names do exist in the object model, described later.)

In other words, you can shoehorn most of the addresses in the world (most importantly, the ones that Skiviez ships to), by just thinking of the existing fields in these terms:

Nothing has really changed. We’re just making the concession that it’s OK that the meaning of the data within these fields will change slightly depending on the country we’re talking about. In the United States, the “AdministrativeArea” is the state, territory, or military post office. In Ireland, the “AdministrativeArea” is the county. And in Finland, the “AdministrativeArea” is NULL because it simply isn’t used at all.

Creating a sane object model

There is something wrong with the database schema, though, and for the sake of preserving legacy code and data, I worked around it in the new object model that is persisted to this schema. The problem is as follows:

A mailing address is not the same as a contact.

Frequently while coding over the years, I found myself wanting to deal with a physical address when I didn’t have any related contact information. That is, I had a group of data for 123 Main St; Richmond, VA 23221; United States, but I didn’t know who lived there or what the phone number was. The only object I had in my model was an Address object that was a 1-1 mapping with the old database schema, so this meant having Address instances with a null name and a null phone number.

A more natural model is to split the fields into three objects and create a fourth object that is composed of all of them:

So instead of one big honking Address object, we have

• an Address object that just contains physical address information;
• a Country object (not shown in the above diagram) that contains information necessary to format and validate an address (more on this later);
• a PersonalName object that binds all of the name fields together;
• a Contact object that groups together a PersonalName, an optional company name string, and a PhoneNumberBase object; and
• a MailingAddress object that pairs a Contact with an Address.

By splitting the old Address object into these individual components, I can now freely re-use these components elsewhere in the application. For example, the PersonalName object also gets used in the Customer object. The return form generator expects a MailingAddress object, but a shipping rate calculator can deal with an Address object directly, since the latter has no need for the Contact information. In other words, the object model went from coarse-grained to fine-grained.

But perhaps the most important aspect of this model is that each of the four above classes is immutable.

Why are immutable addressing objects better?

When an object is immutable, it can’t be altered after it is first instanced. In other words, everything about the object is permanently set at construction time; its instance methods won’t mutate the object, and its properties don’t have public setters. In domain driven design, this is often referred to as a value type, but I don’t like that terminology as it can cause some confusion with value-vs-reference types in the .NET Framework.

This means that if I have an instance of an Address and want to edit the PostalCode property, then I will need to create a brand new instance of an Address, copy most of the properties over, and set my new postal code at construction time.

There are a few advantages to this approach:

• The instance can validate itself once at construction time. This can be a big win if validation is particularly complex and dependent on the values of other properties (which is exactly what happens in an address, since the country determines whether or not a field is required and what format it needs to be in). If anything else is referencing that Address instance, then it can assume that that instance won’t get put into an invalid state. For example, a SalesOrder has a ShippingAddress property that takes a MailingAddress instance. The SalesOrder doesn’t have to worry about someone coming along and saying ShippingAddress.PostalCode = null at some point, bypassing any validation checks. The whole ShippingAddress instance would have to be replaced, which means building a new Address instance, which means running the validation logic built into the address builder.
• The hash code of the instance is consistent throughout the object’s lifetime. This is useful for modeling an address property as a component in NHibernate, which I won’t get into here.

But a big disadvantage to this approach is that it can be tedious to deal with in code. At face level, this means that our Address object would need a gigantic constructor or support an IFreezable-type of implementation. A cleaner way is to use the builder pattern to allow us to set the fields in any order and validate the Address in one shot:

var country = CountryMother.UnitedStates();
 
return new Address.Builder()
    .WithPrimaryStreetLine("3005 SOME ST")
    .WithMunicipality("RICHMOND")
    .WithAdministrativeArea("VIRGINIA")
    .WithPostalCode("23221")
    .WithCountry(country)
    .Build(); // exception thrown here if validation errors

If we want to get a list of detailed validation problems instead of throwing an exception, the builder provides a mechanism for that:

var country = CountryMother.UnitedStates();
var builder = new Address.Builder()
    .WithPrimaryStreetLine("3005 SOME ST")
    .WithMunicipality("RICHMOND")
    .WithAdministrativeArea("VIRGINIA")
    .WithPostalCode("23221")
    .WithCountry(country);
var results = builder.Validate();
 
if (results.Any(x => !x.Success))
{
    foreach (var error in results.Where(x => !x.Success))
    {
        Console.WriteLine(
            "Field {0}, Error {1}", 
            error.Field, 
            error.Message);
    }
}
else
{
    var instance = builder.Build();
}

This allows the user interface to be able to get to detailed error messages pretty easily. But how does that call to Validate() work? How does the builder really know what format an address should be in?

Validation

We need to define a way for a Country to know how its mailing addresses need to be formatted. Since we have all countries of the world in our database, but we don’t ship to all countries in the world, I created a nullable property on the Country object called Scheme. The Scheme is an instance of the AddressScheme class and it’s persisted to the database along with the country.

If we try to build an address without a country specified or with a country that doesn’t have an AddressScheme, then we throw an exception saying that we just can’t do that. Otherwise, our Validate() method will run through a list of the AddressFieldRules that are contained within the AddressScheme instance, calling the Validate(string) method on the rule for a particular field.

To be able to easily persist these rules in the database, I created an enum that parallels the generic properties of the Address class that we defined earlier:

public enum AddressField : int
{
    Unknown = 0,
    AdministrativeArea = 1,
    Country = 2,
    Municipality = 3,
    PostalCode = 4,
    PrimaryStreetLine = 5,
    SecondaryStreetLine = 6,
    SubAdministrativeArea = 7
}

Internally, the Address object maintains its state via a Hashtable of fields, where the key is the AddressField enumerated value and the value is an object, usually string but sometimes a Country. The properties of the Address object are really just strongly-typed convenience mappings onto that Hashtable as in the following example:

public virtual string AdministrativeArea
{
	get
	{
		return this[AddressField.AdministrativeArea] as string;
	}
 
	private set
	{
		this[AddressField.AdministrativeArea] = value;
	}
}

The implementation of the AddressFieldRule is fairly straightforward. I provide the capability to validate against a regular expression (such as might happen with a postal code field), to simply consider the field required (such as might happen with a city field), and to ensure that the value is a member of a list of a predefined list of values (such as might happen with a province field). The Validate(string) method figures out which rules to apply based on the properties that are set and returns an AddressFieldRuleValidationResult that describes what went right and what went wrong.

/// <summary>
/// Describes a requirement for an address field for a particular address.
/// </summary>
public class AddressFieldRule
{
    /// <summary>
    /// The list of allowed values; this may be null if no particular set of
    /// values is required.
    /// </summary>
    private IEnumerable<AddressFieldAllowedValue> allowedValues;
 
    /// <summary>
    /// The description of the field and its format.
    /// </summary>
    private string description;
 
    /// <summary>
    /// The address field.
    /// </summary>
    private AddressField field;
 
    /// <summary>
    /// Whether or not the field is required.
    /// </summary>
    private bool isRequired;
 
    /// <summary>
    /// The localized name.
    /// </summary>
    private string name;
 
    /// <summary>
    /// An optional validation regular expression template.
    /// </summary>
    private string validationRegex;
 
    /// <summary>
    /// Initializes a new instance of the <see cref="AddressFieldRule"/> class.
    /// </summary>
    /// <param name="field">The field.</param>
    /// <param name="isRequired">if set to <c>true</c> [is required].</param>
    /// <param name="name">English, localized name for the field. Required.</param>
    /// <param name="description">The english description.</param>
    /// <param name="validationRegex">The validation regex. Optional.</param>
    /// <param name="allowedValues">The allowed values. Optional; if null, then
    /// no particular set of values is required.</param>
    /// <exception cref="T:System.ArgumentNullException">if the englishName is null
    /// or empty</exception>
    /// <exception cref="T:System.ArgumentOutOfRangeException">if the address field
    /// is not in a valid range</exception>
    public AddressFieldRule(
        AddressField field,
        bool isRequired,
        string name,
        string description,
        string validationRegex,
        IEnumerable<AddressFieldAllowedValue> allowedValues)
    {
        if (string.IsNullOrEmpty(name))
        {
            throw new ArgumentNullException("name");
        }
 
        if (!Enum.IsDefined(typeof(AddressField), field))
        {
            throw new ArgumentOutOfRangeException("field");
        }
 
        if (!string.IsNullOrEmpty(validationRegex) &&
            allowedValues != null &&
            allowedValues.Count() > 0)
        {
            throw new ArgumentException(
                "The ValidationRegex and the AllowedValues options are mutually exclusive.");
        }
 
        this.field = field;
        this.description = description;
        this.isRequired = isRequired;
        this.name = name;
        this.allowedValues = allowedValues ?? new List<AddressFieldAllowedValue>();
        this.validationRegex = validationRegex;
    }
 
    /// <summary>
    /// Initializes a new instance of the <see cref="AddressFieldRule"/> class.
    /// </summary>
    protected AddressFieldRule()
    {
    }
 
    /// <summary>
    /// Gets values that, if not empty, then the address field value 
    /// should be one of the membersin the First property of each of these 
    /// Pairs. The Second property, if
    /// set, is an alternative name that could be displayed in a dropdown to
    /// the user.
    /// </summary>
    public virtual IEnumerable<AddressFieldAllowedValue> AllowedValues
    {
        get
        {
            return this.allowedValues;
        }
 
        private set
        {
            this.allowedValues = value;
        }
    }
 
    /// <summary>
    /// Gets a human-readable string that describes the required format for the
    /// field. It may give an example, such as "JYX 938".
    /// </summary>
    public virtual string Description
    {
        get { return this.description; }
        private set { this.description = value; }
    }
 
    /// <summary>
    /// Gets the field that this metadata describes.
    /// </summary>
    public virtual AddressField Field
    {
        get { return this.field; }
        private set { this.field = value; }
    }
 
    /// <summary>
    /// Gets a value indicating whether or not the field must be 
    /// included in the address.
    /// </summary>
    public virtual bool IsRequired
    {
        get
        {
            return this.isRequired;
        }
 
        private set
        {
            this.isRequired = value;
        }
    }
 
    /// <summary>
    /// Gets an English, localized name for the field. For example, the
    /// StateOrProvince field has a name of "Province or Territory" for
    /// Canadian addresses.
    /// </summary>
    public virtual string Name
    {
        get { return this.name; }
        private set { this.name = value; }
    }
 
    /// <summary>
    /// Gets an optional validation regular expression template, which may
    /// be null.
    /// </summary>
    public virtual string ValidationRegex
    {
        get
        {
            return this.validationRegex;
        }
 
        private set
        {
            this.validationRegex = value;
        }
    }
 
    /// <summary>
    /// Gets or sets the data store identifier for the rule.
    /// </summary>
    private int? Identity
    {
        get;
        set;
    }
 
    /// <summary>
    /// Validates the value.
    /// </summary>
    /// <param name="value">The value to validate.</param>
    /// <returns>The validation result.</returns>
    public virtual AddressFieldRuleValidationResult Validate(string value)
    {
        if ((value == null || value.Trim().Length == 0) && this.IsRequired)
        {
            var message = string.Format(
                CultureInfo.CurrentCulture,
                "The field '{0}' is required, but the value was empty or blank.",
                this.Name);
 
            return new AddressFieldRuleValidationResult()
            {
                Field = this.Field,
                Original = value,
                Message = message
            };
        }
 
        if (value != null)
        {
            var input = value.Trim();
 
            if (input.Length > 0)
            {
                if (!string.IsNullOrEmpty(this.ValidationRegex) &&
                    !Regex.IsMatch(input, this.ValidationRegex))
                {
                    var message = string.Format(
                        CultureInfo.CurrentCulture,
                        "The field '{0}' has an unexpected or incorrectly formatted value.",
                        this.Name);
 
                    return new AddressFieldRuleValidationResult()
                    {
                        Field = this.Field,
                        Original = value,
                        Message = message
                    };
                }
 
                if (this.AllowedValues.Count() > 0)
                {
                    string preferred;
 
                    // If an alternative value is detected, select the preferred.
                    var alternativeAllowedValue =
                        (from av in this.AllowedValues
                            where !string.IsNullOrEmpty(av.Alternative) &&
                            av.Alternative.Equals(input, StringComparison.OrdinalIgnoreCase)
                            select av).SingleOrDefault();
                    if (alternativeAllowedValue != null)
                    {
                        // We matched an alternative value, so let's use the
                        // corresponding primary value
                        preferred = alternativeAllowedValue.Preferred;
 
                        return new AddressFieldRuleValidationResult()
                        {
                            Field = this.Field,
                            Original = value,
                            Preferred = preferred,
                            Success = true
                        };
                    }
                    else
                    {
                        // Search on the primary value, then.
                        var primaryAllowedValue =
                            (from av in this.AllowedValues
                                where av.Preferred.Equals(input, StringComparison.OrdinalIgnoreCase)
                                select av).SingleOrDefault();
                        if (primaryAllowedValue != null)
                        {
                            // We matched the primary value, let's re-assign
                            // it to get the correct casing
                            preferred = primaryAllowedValue.Preferred;
 
                            return new AddressFieldRuleValidationResult()
                            {
                                Field = this.Field,
                                Original = value,
                                Preferred = preferred,
                                Success = true
                            };
                        }
                        else
                        {
                            var message = string.Format(
                                CultureInfo.CurrentCulture,
                                "The field '{0}' was not in the list of expected values.",
                                this.Name);
 
                            return new AddressFieldRuleValidationResult()
                            {
                                Field = this.Field,
                                Original = value,
                                Message = message
                            };
                        }
                    }
                }
            }
        }
 
        return new AddressFieldRuleValidationResult()
        {
            Field = this.Field,
            Original = value,
            Preferred = value,
            Success = true
        };
    }
}

(In the case of a state or province field, where we ensure that the value belongs to a predefined list of AddressFieldAllowedValues, we also let the rule tell the consumer which value is “preferred.” For example, a user might enter “Virginia”, but the system will spit back “VA” as the preferred formatting for that value.)

As a result, that Validate() method on the builder doesn’t have to do much heavy lifting:

public IEnumerable<AddressFieldRuleValidationResult> Validate()
{
    var failures = new List<AddressFieldRuleValidationResult>();
 
    if (this.address.Country == null)
    {
        var result = new AddressFieldRuleValidationResult()
        {
            Field = AddressField.Country,
            Message = "The country is required."
        };
 
        failures.Add(result);
    }
    else if (this.address.Country.AddressScheme == null)
    {
        var result = new AddressFieldRuleValidationResult()
        {
            Field = AddressField.Country,
            Message = "The country must have an address scheme."
        };
 
        failures.Add(result);
    }
    else
    {
        var scheme = this.address.Country.AddressScheme;
 
        foreach (AddressField field in new ArrayList(this.address.fields.Keys))
        {
            if (field != AddressField.Country &&
                field != AddressField.Unknown)
            {
                // If the field is not Country or Unknown, let's
                // see if the scheme has a rule defined for it.
                if (scheme.HasRuleFor(field))
                {
                    // Since a rule is defined, let's validate the field.
                    var result = scheme.ValidateField(field, this.address[field] as string);
                    if (result.Success)
                    {
                        this.address[field] = result.Preferred;
                    }
                    else
                    {
                        failures.Add(result);
                    }
                }
                else
                {
                    // No rule is defined, so strip the data out of the
                    // address.
                    this.address[field] = null;
                }
            }
        }
    }
 
    return failures;
}

Where do all of these address rules come from? There is nothing more valuable than Frank’s Compulsive Guide to Postal Addresses.

Formatting

Another important property on the AddressScheme is the Formatter. This is an object that knows how to format an address for a particular country. The ToString() implementation on the Address can then just call this.Country.Formatter.Format("M", this) to get a string back that contains the correctly formatted address, where “M” is our IFormattable code for an address format that includes newlines.

My AddressFormatter object works by taking a template string that is loaded from the database. The template format is pretty simple: it’s a combination of the AddressField enumerated values (delimited by braces) and the semicolon (which indicates separate lines of the address). For example, the template for a German address is specified as:

{PrimaryStreetLine};{SecondaryStreetLine};DE-{PostalCode}  {Municipality};{Country}

Given an “S” format code (for “single-line output”), we’ll substitute the values and leave the semicolons. Given an “M” format code (for “mailing output”), we’ll substitute the values and swap the semicolons with newlines. We also remove any contiguous separators that might happen, such as when the SecondaryStreetLine is blank, to avoid having a blank line in the address.

So given an address that was built as

var country = CountryMother.Germany();
 
return new Address.Builder()
    .WithPrimaryStreetLine("42 Kaiserstrasse")
    .WithMunicipality("Berlin")
    .WithPostalCode("12345")
    .WithCountry(country)
    .Build();

our formatter, given a format code of “M”, will spit out

42 Kaiserstrasse
DE-12345 Berlin
Germany

Since the address object is immutable, we know it’s valid, so the formatter doesn’t have to worry about handling a potentially invalid address object.

The user interface of the Web site also uses the template to figure what input fields to display and what order to display them in.

Determining address kind

When shipping, it’s often important to know the “kind” of an address, as in the following enumeration demonstrates:

public enum AddressKind : int
{
    Unknown = 0,
    Commercial = 1,
    PostOfficeBox = 2,
    Residential = 3,
    Military = 4
}

Why is this important? For example, you can only use USPS to ship to a PO box or a military (APO/FPO/DPO) address, and you can’t use FedEx Home Service on a commercial address.

One mistake that I made over the years was trying to store the AddressKind right in the Address object. This was a mistake for a few reasons:

• I don’t always care about the address kind, such as for an international address, so having it appear on every Address instance is wasteful.
• Computing the address kind can be expensive. While figuring out a PO box or a military address just involves scanning the address fields, determining commercial or residential status means calling a Web service supplied by FedEx, which can take seconds to complete. By having the Kind property on the address, I’m left with a bad situation: I can either keep the immutability of my address object by determining the address kind for every address when I instantiate them, or I can break immutability, which negates the advantage of not having to worry about change tracking on these objects.

As a result, our new Web site simply doesn’t store the address kind at all. The kind, if needed, is determined on the fly by calling the DetermineAddressKind() method of an IAddressKindDeterminer instance.

Being flexible: Legacy data and third-party integration points

Finally, there’s one last point to be made. This new address validation and formatting is great and all, but there are two potential problems:

• There is 10 years of less-than-satisfactory address data already in the database, and it could be in an “invalid” format according to these new guidelines. Obviously, we want to maintain backward compatibility here, as we don’t want to force our customers to re-type their address data, and we don’t want to blow up when simply trying to load the address from the database.
• Addresses sometimes get force-fed to the system via third-party integrations such as PayPal Express Checkout or Google Checkout. I can’t really reject addresses coming from these sources, so I have to be able to “take it like it is,” even if it means it doesn’t quite adhere to my preferred format.

As a result, the new address system has to be able to grandfather in old addresses and graciously accept other addresses from integration. For accepting addresses from integration, I simply added an overload to the address builder’s Build() method that does not perform any validation–and trust myself only to use that overload in the proper scenarios.

For handling legacy address data, I simply let NHibernate set the address fields directly when loading objects from the database. That is, since the address isn’t constructed via the address builder but is instead built up via reflection, no validation is run on the legacy address and so the legacy address will still load.

One advantage of this model comes from the Address object’s immutability. Since the address can’t be changed, if the user wishes to edit an old address, they’ll be forced to bring the whole address up to snuff since a new Address instance will need to be created.

Conclusions and delusions

At the end of the day, we’re left with a robust implementation of address storage and address validation:

International address validation, input, and handling has never been better at Skiviez, and for the first time in years, I feel like it’s finally under control–thanks to a clean object design, immutability, and a few days of thinking really hard about the problem. I hope this post gives someone some insight on their own project out there. Good luck!

One of these new features is a fast, faceted product catalog. Faceting is the technical name for something you’ve seen in many e-commerce stores like NewEgg, Zappo’s, and Lowe’s: a list of links that quickly allow you to drill down to the content that you’d like to see.

Trying to do this kind of thing efficiently in a traditional relational database is not going to happen unless you want to start adding and removing indexes on the fly or go full on Entity-Attribute-Value, which somewhat defeats the purpose of using a relational database. You lose efficient indexing, you lose type information, you lose, well, the relations.

Instead, Skiviez now uses Solr–an open source search platform that is based on Lucene, a text search engine library–for managing catalog listing pages and search pages.

Solr from 10,000 feet

Solr is not relational. It doesn’t have tables. The schema is flat; all “documents” in the Solr store share the same schema. From a relational perspective, you could think of it as one table with many columns, except that not every row in the table uses all of the columns. (This isn’t how it internally works at all, but it’s a useful metaphor.)

Some fields are “stored,” which means you can get back the data that you put in. Some fields are “indexed,” which means that you can query on them directly and use them for faceting. And some fields are “multivalued”, which means that they can hold more than one value at a time. For example, our <sizeInStock> field can hold S, M, and L values all at once for a particular product, and just L for another.

Why would you ever want a value to not be “stored”? Why add data that you can’t get back? Well, in Solr, storing and indexing are distinct concepts, so you can a value that is not stored, just indexed. A good example from the Skiviez Solr schema would be the salesVolume field, which indicates how many units of a particular item were sold in the past week:

		<field
			name="salesVolume"
			type="sint"
			indexed="true"
			stored="false" />

This is just an integer, a number of units sold. In my UI, I’m not going to ever actually display that number–that would reveal a little too much information about our business. But I still want our users to be able to sort catalog listings by “Top Selling” products. By creating an index, we enable sorting; by disallowing storage, we save some space.

The Skiviez Solr schema defines a “type” field, some fields that are shared among all document types, and a few type-specific fields. This works well because the only intended use of Solr is catalog listings and searching; if I had to store more disparate kinds of data, I might create several Solr schemas.

<!--
	This field is used to indicate the type of data that is stored within
	the document. Since the Skiviez Solr instance represents a variety of
	different types, this field helps to indicate what kind of data you are 
	looking at.
 
	Types that are currently understood the Web site include:
 
		BLOGARTICLE
		BRAND
		LINE
		PRODUCT
		PRODUCTSUMMARY
                PROMOTION
		STYLE
		WEBPAGE
-->
<field
	name="type"
	type="string"
	indexed="true"
	stored="true"
	required="true" />

Why is this useful? Well, the search portion of the Skiviez Web site can use the <type> field and the fields that are used by all of the different document types to quickly search a wide variety of sources:

In this case, we’re looking at our <type>, <urlKey>, <name>, and <image> fields. The Web site can use this to quickly sort information and build links to content.

The PRODUCT type uses a wealth of product-specific fields, however, and when we query to display catalog pages, we tell Solr to only return documents of type PRODUCT. I could have created a second Solr schema for this purpose, but why bother?

Getting data into Solr

The traditional SQL database is still the “authoritative” data store, and the Solr indexes are read-only snapshots of that data. This means that the data coming from Solr is always slightly stale, so I had to ask myself:

• How stale is too stale?
• When do I value speed or querying over staleness?

Part of the new Skiviez Web site is a Windows service that I call the “Worker.” It uses Quartz.NET to execute C# IJob implementations periodically. You can think of them as traditional scheduled tasks in Windows; the only difference is that I am explicitly managing them in code, using the same object-oriented model of our domain in those jobs, and, as long as the service is installed, I don’t need to worry about configuring scheduled tasks.

Every three hours, one of those IJobs that gets executed is the RefreshSolrIndexesJob, and all that job does is ping an HttpWebRequest over to http://solr.example.com/dataimport?command=full-import, where solr.example.com is placed with the FQDN of our internal Solr server. This is because I use Solr’s built-in DataImportHandler to actually suck in the data from the SQL database; the job just has to “touch” that URL periodically to make the sync work. Because the DataImportHandler commits the changes periodically, this is all effectively running in the background, transparent to the users of the Web site. And because the Skiviez product catalog is reasonably small (a few thousand items), we can blow away the whole Solr index and re-build it in fewer than two minutes.

There’s also a function in our backend application that allows employees to trigger the index rebuilding immediately; this can happen when new product arrives in the warehouse and we want to get it up on the Web site right away.

The DataImportHandler is built into Solr, and configuring it is a little confusing because it uses some strange terminology. It just takes an XML configuration file, and whenever you ping its request handler, it performs synchronization tasks based on what has been specified in its configuration.

      <entity
        name="brands"
        dataSource="undiesDatabase"
        transformer="TemplateTransformer"
        query="
            SELECT
                b.ID AS BrandId,
                b.[Name] AS BrandName,
                b.Description AS BrandDescription,
                b.UrlKey AS BrandUrlKey,
                CASE b.Active
                  WHEN 'Y' THEN 'True'
                  ELSE 'False'
                END AS BrandIsActive
            FROM na.brands AS b
            ORDER BY b.[Name];">
        <field column="id" template="BRAND-${brands.BrandId}" />
        <field column="type" template="BRAND" />
        <field name="identity" column="BrandId" />
        <field name="name" column="BrandName" />
        <field name="description" column="BrandDescription" />
        <field name="isActive" column="BrandIsActive" />
        <field name="urlKey" column="BrandUrlKey" />
      </entity>

I say the terminology is confusing because in <field> elements, the @column attribute is the name of the Solr field and the name of the column in the SQL result set. But if the @name attribute is specified, then the @column attribute is the name of the column in the result set and the @name attribute is the name of the Solr field. It’s confusing because some <field> elements don’t pull directly from the result set, instead relying on a “transformer.” You would expect to just specify a @name for those fields, but specifying just @column is correct. In the above example, the “Template Transformer” is used on the id field to format Solr identities in the format of BRAND-127, where 127 is the SQL database’s primary key for the brand.

(And yes, attributes can have newlines in them. I’m not sure how I went through years of programming without realizing this, but once I realized that I could do it, it made reading those SQL queries much easier! This example is by far the shortest query for pulling Solr data from the SQL database; the one for products is over 200 lines long.)

Querying data: how stale is too stale?

This does mean that information in the Skiviez product catalog can be up to three hours stale. A user might click a link for “Medium In Stock (3)” on the catalog page (since this kind of faceted data is generated by querying Solr) but then see on the product detail page that no mediums are in stock (since on this page, the quantity information is one of the few things not cached and queried directly against the database). This is annoying, but generally rare in our particular scenario (we are a reasonably small business and not that high traffic), and it will be fixed up in 3 hours anyway when we rebuild the whole index again from scratch, so I have accepted this as a reasonable trade-off.

(If I really wanted to solve this problem, there are a few approaches that I could take. I could use domain events to fire off partial updates to the Solr index whenever a Save/Update operation occurred on an Item or ItemGroup in the Skiviez domain model, or I could insert a record into a table named, say, dbo.IdentitiesOfStuffThatNeedsUpdatingInSolr, and have an IJob that reads that list and executes partial updates every minute. And even if I did these things, I’d still probably want to do a periodic “blow it all away and refresh” in case one of those partial updates failed in the background.)

As for querying this data from Solr, there are a few approaches that I could have taken. One is to hide the fact that Solr exists entirely via the methods of a repository-like class: the Get*() methods would access Solr, and the Create/Update/Delete methods would access the database. I didn’t like this approach because my Solr schema is already shamelessly tailored to the UI that will be accessing that data, as it should be–I’ve already made the decision to use Solr to provide easy faceting, sorting, and fast display of information, so I might as well use it to its fullest extent. This means making it explicit in code as to when I mean to access Solr and when I mean to access the up-to-date, non-cached database object.

In my case, I ended up using NHibernate to do the CRUD access (e.g., loading an ItemGroup, futzing with its pricing rules, and then saving it back), forgoing the repository pattern because I don’t typically see its value when NHibernate and its mappings are already abstracting the database. (Sometimes abstracting out NHibernate is useful, however, as we’ll see below.)

When querying for data, I know pretty well if I’m using it for catalog-oriented purposes (in which I care about speed and querying features) or for displaying in a table on a back-end administrative application (I care about currency). For querying on the Web site, I have an interface called IListingQuery. It has a Search() method that accepts a ListingRequest where I define some parameters–selected facets, search terms, page number, number of items per page, etc.–and gives back a ListingResponse–remaining facets, number of results, the results on this page, etc. This interface is pretty boring stuff.

Where it gets interesting is that the implementation of IListingQuery that gets dependency injected into my Web site’s ProductsController is using a list of IListingQueryStrategys underneath. The default strategy, the SolrListingQueryStrategy, hits Solr directly via a plain old-fashioned HttpWebRequest and parses the XML in the HttpWebResponse (which is much easier to use, in my humble opinion, than some of the Solr client libraries).

If the Solr-based strategy throws an exception or vomits for some reason, then the DatabaseListingQueryStrategy runs next and hits the database directly–although it ignores some parameters of the ListingRequest, like faceting or advanced text searching, since that is inefficient to do there and is the whole reason I am using Solr in the first place. The idea is that usually Solr is answering my search requests quickly in their full-featured glory, but if something blows up and Solr goes down, then the catalog pages of the site can still function in “reduced-functionality mode” by hitting the database with a limited feature set directly. (As implemented on the Skiviez site today, you would realize that this happened if the filters list on the left-hand side becomes empty and search parameters are ignored.)

(The explosion of classes that you see in the screenshot above is mostly due to testability. For example, Solr works by sending long, complicated GET requests to it. So I’ve pulled out of the URI-building functionality into its own class; it serves a single purpose and can be tested as such. The actual SolrListingQueryStrategy class implementation is very short, with most of the work delegated to other classes within that folder.)

Conclusions and delusions

What is important is that I have made explicit in code that this is a search–by using IListingQuery instead of NHibernate–so the database-based strategy can take some liberties in ignoring some of the search parameters without worrying about affecting some of its down-level callers too severely. The decision to perform a query against a possibly-stale data store versus the authoritative data store has been made explicit–if I want fast, possibly stale data with advanced search features, I use IListingQuery. If I want slow, up-to-date data with insert/update/delete capability, I use NHibernate’s named queries. And if I make a change in the SQL database, I know that the out-of-process Worker service will update Solr eventually, making things eventually consistent.

The end result? Fast catalog pages for our customers that gracefully fall back to the old behavior when something doesn’t work.

AVS Information In Magento

We’ve recently set up a new e-commerce store at Men’s Underwear Discounters that represents a division of the company that I work for, Skiviez. We decided to save time for this discount store by using Magento, the current darling child in the open source e-commerce platform market segment.

A brief interlude on consulting, documentation, and open source

Now, to be clear, it doesn’t take much to exceed expectations in the “open source e-commerce platform” market because prior to Magento, your options were pretty much limited to osCommerce, which is quite possibly the most reckless, idiotic excuse of an e-commerce platform that I have seen (and I don’t like to insult code), and its slightly-less-retarded sibling Zen Cart. (I will save the osCommerce rants for another post.)

Magento seems to be a dream come true: thousands of man-hours of development into an e-commerce platform for free. So what’s the catch? While Varien (“the Magento company”) has done a great service to the world and the open source community by creating Magento, they’re still in it to make a buck–and that’s not a bad thing–but you might not realize how they make those bucks until you’ve started playing around with your own Magento store. They make money on support, consulting, and customization because Magento is huge and poorly documented.

Actually, “poorly documented” would imply that there was at least some documentation, and while there truly is some documentation available in the source code and on Varien’s Magento Web site, in reality, that amount of documentation is a statistical rounding error. And when you actually view your Magento installation in an FTP program, you will be absolutely astounded at the number of directories and files. You can try inspecting the source files to try to put it all together, but a lot of it is “magic”: a framework erected at runtime by XML configuration files and PHP’s idiotic __call mechanism for magic functions that appear at runtime out of thin air. You can see references to a function called getOrder(), for example, grep the entire installation for “getOrder()”, and not get any results because the function doesn’t actually exist at compile time. Personally, I’ve always considered magic methods to be an extremely unwise idea, a “too clever” approach to programming. But for people new to PHP looking to do a little customization to their Magento store, it makes the codebase all the more impenetrable. That’s more consulting dollars.

You could go to the Magento forums, but you won’t find too many helpful answers because Magento obviously doesn’t want to give help for free, since giving help is central to their business model. So if you need help, your best bet is to generally (a) pay for it or (b) hope that you stumble across somebody’s blog who took the time to post a solution to your exact problem.

Why do I need to do this, anyway?

Magento has a lot of stuff built in. But we need to remember that Magento was built by software developers, and software developers generally don’t understand the problem domain of running an e-commerce business very well. Sure, every software developer has built the pet shop–products, categories, inventory, orders, how hard could it be?–but they stop or fail to understand that there’s a whole class of operations that need to happen after the order has been placed.

One of those operations is fraud analysis. The Address Verification System (AVS) and the Card Verification Number (CVN) system for credit cards enable merchants to help determine if a credit card has been fraudulently used on an order. AVS will tell the merchant whether or not the billing address specified by the customer matches or partially matches the billing address on the credit card’s billing statement. And the CVN number represents the number that (supposedly) exists in only two places in the world: on the physical credit card and in the issuing bank’s files. If a CVN doesn’t match and the AVS doesn’t match, then there’s a good bet that there is something wrong with the order.

Because software developers are nerds, this information doesn’t currently display by default in Magento’s administration section. So let’s add it.

Can’t I just edit the template?

I’m talking about adding this AVS display functionality in the context of the CyberSource extension for Magento. After spelunking through the myriad files and directories, I discovered that the Varien-provided CyberSource extension is saving the AVS (setCcAvsStatus()) and CVN (setCcCidStatus()) information when it authorizes the card, but it doesn’t display in the information block in the administration section.

While you could just modify app/design/adminhtml/default/default/template/cybersource/info.phtml directly (yes, that is the actual path to the file that displays the little “Payment” block), your changes would be lost and overwritten in the future if you ever upgraded the CyberSource extension via Magento Connect. That’s obviously less than ideal.

A better way would be to use Magento’s “rewrite” mechanism. We’ll create our own Magento module and instruct Magento to replace the CyberSource extension’s information block with our own.

Creating our own module

First, we’ll create a new empty directory to hold our files. Let’s call it avsdisplay. Then we’ll create code, etc, and design subdirectories. These directories mirror the subdirectories in the app folder of your Magento installation.

In the etc directory, we need to create an XML configuration file. Magento scans this directory at runtime for XML files and reads them to know which modules to load. I’ve named my configuration file Mud_Avsdisplay.xml, and the naming is very important. (Magento uses a mixture of convention and configuration.) The part of the filename before the underscore is your “namespace,” since PHP doesn’t support namespaces. And the part of the filename after the underscore is the name of your module. Mine is “Avsdisplay.” Don’t try to use something like “AvsDisplay,” however, because that would break Magento’s convention as we’ll see in a moment.

Inside this file, we tell Magento to load our module in the local pool and to actually run it (by setting “active” to “true”). (The other pools are core, for those modules provided by Varien, and community, for those modules that you can download via Magento Connect.)

<?xml version="1.0"?>
<config>
	<modules>
		<Mud_Avsdisplay>
			<active>true</active>
			<codePool>local</codePool>
		</Mud_Avsdisplay>
	</modules>
</config>

Now it’s time to create the meat of the module. In the code folder, create a local directory. Within that, create a directory named after your chosen namespace (Mud in my example) and within that yet another subdirectory named after your module (Avsdisplay in my example). (This is why naming was important. If I had called it “Mud_AvsDisplay,” with a capital ‘D’, then Magento would have looked in app/code/local/Mud/Avs/Display for my module instead of app/code/local/Mud/Avsdisplay.)

Within your module directory, you’ll need to create three more subdirectories: Block, etc, and Helper. When it’s all said and done, you’ll end up with a hierarchy that looks like this (ignore the “design” directory, we’ll get to that later):

Directory Structure

Creating the block

Next, we need to create the “block” that we’ll be replacing. After searching through the codebase, I figured out that I wanted to replace the Mage_Cybersource_Block_Info block (for reference, by Magento’s convention, that file lives in app/core/Mage/Cybersource/Block/Info.php — see the convention?). But I really only want to add stuff to it, not replace it outright, so I’ll create a new file in my Block folder called Info.php that extends that class. Something like:

<?php
 
class Mud_AvsDisplay_Block_Info extends Mage_Cybersource_Block_Info
{
	// Yes, these really have to be protected, not private, due to more Magento
	// "magic"
	protected $avsDescriptions = array(
		// Snipped for brevity; download file at end of blog post
	);
 
	protected $ccCidDescriptions = array(
		// Snipped for brevity; download file at end of blog post
	);
 
	protected function _construct()
	{
		parent::_construct();
		$this->setTemplate('avsdisplay/info.phtml');
	}
 
	public function getAvsStatusDescription()
	{
		$info = $this->getInfo();
		$avsCode = $info->getCcAvsStatus();
		$avsDescription = 'Unrecognized response code.';
 
		if (array_key_exists($avsCode, $this->avsDescriptions))
		{
			$avsDescription = $this->avsDescriptions[$avsCode];
		}
 
		return $avsDescription;
	}
 
	public function getCcCidDescription()
	{
		$info = $this->getInfo();
		$ccCidCode = $info->getCcCidStatus();
		$ccCidDescription = 'Unrecognized response code.';
 
		if (array_key_exists($ccCidCode, $this->ccCidDescriptions))
		{
			$ccCidDescription = $this->ccCidDescriptions[$ccCidCode];
		}
 
		return $ccCidDescription;
	}
}

Okay, this makes sense. I’m doing everything the old CyberSource info block did, except I added two new functions that give me human-friendly descriptions of the AVS and CVN codes, and I’m setting the template to something called avsdisplay/info.phtml instead of cybersouce/info.phtml. How this comes into play will make sense in a minute.

Creating the helper

In my Helper directory, I need to add a Data.php file that looks like this:

<?php
 
class Mud_Avsdisplay_Helper_Data extends Mage_Core_Helper_Abstract
{
}

That’s right, it does nothing except inherit from a Magento-provided abstract class. What is a helper? I have no idea, but you have to do it–Magento looks for this file.

Telling Magento to do the swap

In my etc directory, I need to add a config.xml file that will tell Magento some information about my module and, most importantly, that I want to be swapped with the CyberSource extension’s info block:

<?xml version="1.0"?>
<config>
	<modules>
		<Mud_Avsdisplay>
			<version>0.0.1</version>
		</Mud_Avsdisplay>
	</modules>
	<global>
		<helpers>
			<avsdisplay>
				<class>Mud_Avsdisplay_Helper_Data</class>
			</avsdisplay>
		</helpers>
		<blocks>
			<cybersource>
				<rewrite>
					<info>Mud_Avsdisplay_Block_Info</info>
				</rewrite>
			</cybersource>
		</blocks>
	</global>
</config>

Under <modules>, I’m just telling Magento the version number of my module. Under <global>, I say that I want to “rewrite” the CyberSource info block with my own class named Mud_Avsdisplay_Block_Info, which, by convention, Magento will look for in [module-root]/Mud/Avsdisplay/Block/Info.php. That is the file that I just created. Great.

Finally, writing the template that displays the information

But we haven’t actually changed any HTML at this point! Remember in my Info.php file where I set the template to something called info.phtml? Well, I need to create that file. At the top of my module folder–the one with the code and etc subdirectories–I’ll need to create some more subdirectories. A lot of them. In fact, I’ll need to create a bunch of empty directories that look like this:

In the bottom-most subdirectory, I’ll create the info.phtml file. This is a PHP file, except that when I use the $this pseudo-variable in this file, $this will be pointing to an instance of my Mud_Avsdisplay_Block_Info class. So I’ll be able to call my getAvsStatusDescription() method here among other things:

<?php
	if ($info = $this->getInfo())
	{
		?>
			<strong><?php echo $this->getMethod()->getTitle(); ?></strong><br />
			<strong>Type:</strong> <?php echo $this->__($this->getCcTypeName()); ?><br />
			<strong>Number:</strong> xxxx-<?php echo $this->__($info->getCcLast4()); ?><br />
			<strong>Expiry:</strong> <?php echo $this->__('%s/%s', $this->getCcExpMonth(), $info->getCcExpYear()); ?><br />
			<strong>AVS:</strong> (<?php echo $this->__($info->getCcAvsStatus()); ?>) <?php echo $this->__($this->getAvsStatusDescription()); ?><br />
			<strong>CCID:</strong> (<?php echo $this->__($info->getCcCidStatus()); ?>) <?php echo $this->__($this->getCcCidDescription()); ?><br />
		<?php
	}
?>

Conclusions and delusions

And that’s it. Upload this entire directory structure to your Magento installation’s app directory, merging the contents as necessary, and you should be able to see AVS information for your CyberSource orders in the administration section.

How did I figure out how to learn all of this? It was mostly by lots of Googling, picking apart existing modules to see how they were organized, and piecing together lots of blog posts. I hope this helps someone out there. Good luck!

Download the code used in this article.

I can't log in!

What has happened here is that my OpenID provider, Verisign Labs, is down for some reason while StackOverflow, the site that I’m trying to log on to, is still up.

A single point of failure

While this isn’t a big deal for a site that I use recreationally, it is kind of a big deal in principle: all sites that use OpenID are inaccessible to me, since I only have this single OpenID provider. It is a single point of failure.

Sure, I could have multiple OpenID accounts, such as a “backup” OpenID account at another provider, but this defeats the purpose of OpenID, of having a central point of authority regarding my online identity. If I have to create a second OpenID provider, I might as well just make a native username and password at the site that I’m trying to log into. I could also be my own OpenID provider, but I shouldn’t have to do that, and the average OpenID consumer is not going to know how to do that, anyway. It’s faster just to create a “native account” at the Web site that I’m trying to access and be done with it.

An unnecessary third-party dependency

For any e-commerce business, it’s usually unwise to take on any third-party dependency. Speaking from experience, that third-party will go down for some reason at some point and you will be the one fielding angry customer support calls for something that isn’t your problem. Even worse, if something as critical as authentication through OpenID goes down, then you’re losing sales, and that means that you’re not doing your job.

If you’re going to add OpenID to an e-commerce Web site or a “mission-critical” Web site, then you should implement it as a red-headed step-child, “convenience only” implementation; this way, your users can fall back onto their “native credentials” (those stored in your database) when their preferred OpenID provider explodes. In this way, OpenID is a shortcut, not a single authoritative source, much how like customers use PayPal’s Express Checkout to save time when entering payment and addressing information during checkout. If PayPal’s Web Service API goes down (and believe you me, it’s gone down several times this year alone), customers can still begrudgingly use the “native checkout” to complete their purchase. Some lost conversions due to the inconvenience–and for incorrectly faulting the integrity of your Web site for the third party’s failure–but at least the sale is still actually possible. If OpenID is your only authentication mechanism, and a user’s OpenID provider goes down, then that user is simply screwed. The onus should not be on the consumer to provide a backup authentication mechanism to your Web site.

It’s just complicated

OpenID is still just too complicated. I’m not the smartest software developer in the world by any means, but if I can barely wrap my mind around it enough just to get my first account, then I am 100% certain that our customers would have problems with it:

• Why do I have to go to some third-party provider to sign in? That doesn’t make any sense.
• Why should I trust that third-party provider? I’m not even sure that I trust you!
• I still don’t know which third-party provider to use! Oh well, I guess I’ll use Verisign, at least I’ve heard of them.
• What happens to my credentials when that third-party provider goes under?
• Can I call your business to reset my password when I forget it? I will anyway!

Even with our “native registration” of e-mail addresses and passwords, we had to use an Amazon.com style login form because users kept filling out the “new customer” form even when they already had an account:

Login screen. Amazon’s sign-in screen remains a model to be emulated, minimizing the common problem of new customers who try to log in without having registered. Amazon presents two questions in linear order: (1) “What is your email address?” and (2) “Do you have an Amazon.com password?” For the second question, users can select one of two radio buttons: “No, I am a new customer,” or “Yes, I have a password.” Many other sites present the new- and established-user sections side-by-side, and thereby divert new users to the established-user section through the magnetic attraction of type-in fields. — Jakob Niesen, useit.com

When our users have trouble navigating two fields and a radio button, you can imagine the hilarity that ensues when they are presented with multiple authentication mechanisms. Especially one that doesn’t ask for a password.

A leaky abstraction

While the idea of consolidating your online identity to a single source sounds like a good idea, it simply does not work in practice. Not even the United States government has a consistent picture of my identity: my identity is stored with the IRS, the Social Security Administration, the United States Postal service, the Virginia Department of Taxation, the Virginia Department of Motor Vehicles, and the City of Richmond’s Voter Registrar. These are all independent identity stores that, in general, agree that I am who I say that I am, but each identity is stored and updated independently of one another. Consider OpenID the RealID act of the Internet: what happens when your centralized identity gets screwed up? It’s an existentialist crisis and you just have to deal with it: identity is what other people say you are; it’s not what you say you are. Better to have multiple voices confirming your identity than just one.

Conclusions and Delusions

If you’re implementing a social-oriented Web site that is designed to integrate with Facebook or some of the Web-2.0-savvy consumers, then OpenID might make sense. Those nerds may have heard of OpenID and know how to use it, foibles and all. Until the dust settles on OpenID, though, I wouldn’t add it to a commercial site even if you paid for it: nobody has asked for it; it will cause problems.

My general recommendation would be to have the usual in house username and password mechanism that can be supplemented by OpenID. But realize that each alternative identification mechanism that you add runs the risk of customer confusion, bugs, security holes, and increased customer support. When in doubt, leave it out.

[npiaseck ~]$ telnet www.example.com 80
Trying 10.1.10.38...
Connected to www.example.com (10.1.10.38).
Escape character is '^]'.
GET /Media HTTP/1.0
 
HTTP/1.1 301 Moved Permanently
Content-Length: 152
Content-Type: text/html
Location: http://10.1.10.38/Media/
Server: Microsoft-IIS/6.0
Date: Fri, 05 Dec 2008 13:50:00 GMT
Connection: close
 
<head><title>Document Moved</title></head>
<body><h1>Object Moved</h1>
This document may be found <a HREF="http://10.1.10.38/Media/">here</a>
</body>
 
Connection closed by foreign host.

I’ve made up an internal IP address, but the point is that IIS is inserting the internal IP address in the body of the document as well as the Location header. If you make the same request with the Host header supplied, such as in Host: www.example.com, then IIS would use www.example.com instead of supplying the IP address.

The recommended Microsoft solution is detailed in KB834141. Namely, you edit the SetHostName metabase entry for the Web site to www.example.com, and now IIS will use that instead of the IP address in the above redirection scenario. Easy as pie, right?

Well, that change broke some code. For reasons that are not clear to me, here’s the mechanism by which IIS 6 determines what to use in these redirect requests:

1. Is the UseHostName property set to true? If so, use the machine’s host name.
2. Is the SetHostName property set to some value? If so, use that value.
3. Is a Host header supplied in the HTTP request? If so, use that value.
4. If all of the above fail, just use the IP address, which is probably internal.

For search engine optimization reasons, all of the requests on our Web site redirect all requests that come in without the www subdomain to ones with the www subdomain. This prevents Google from dinging us for having duplicated content. The code was implemented as an ASP.NET HTTP module that ran before every request and inspected the incoming URL by looking at the HttpContext.Current.Request.Url.OriginalString property. If the domain didn’t match, then it issued a permanent redirect.

The problem with the above precedence order is that IIS will substitute the value of HTTP_HOST and other server variables passed into ASP.NET with the SetHostName value, not the Host header if so supplied. The net effect is that our redirection method suddenly stopped working because the code was always seeing the SetHostName value, www.example.com, regardless of the URL that was actually used to make the request. To make matters worse, if we were to set the UseHostName property, the code would always see a mismatch and enter a redirect loop. Neither are desirable scenarios.

The solution is to change the code to inspect the Host header directly, if present, instead of looking at the URL as passed into the server variables array. It ends up looking something like this:

private void OnContextBeginRequest(object sender, EventArgs e)
{
	HttpContext context;
	string originalRequestHost;
	int portIndex;
	HttpRequest request;
	string redirectUri;
 
	context = HttpContext.Current;
	request = context.Request;
 
	originalRequestHost = request.Headers["Host"];
 
	// Some clients will pass in the port number in the Host header if
	// it's not going over port 80
	portIndex = originalRequestHost.LastIndexOf(':');
	if (portIndex != -1)
	{
		originalRequestHost = 
			originalRequestHost.Substring(0, portIndex);
	}
 
	if (originalRequestHost != null && originalRequestHost != mPreferredSubdomain)
	{
		redirectUri = string.Format(
			"http://{0}{1}{2}",
			mPreferredSubdomain, 
			request.Url.Port != 80 ? ":" + request.Url.Port.ToString() : string.Empty,
			request.RawUrl);
		cLog.InfoFormat("Redirecting from to {0} (domain: {1}, preferred: {2}).",
			redirectUri, originalRequestHost, mPreferredSubdomain);
		context.Response.Redirect(redirectUri, true);
	}
}

The code looks at the host header. If it’s set, and doesn’t match, then it redirects. (It does a little bit of finagling to strip off the port number if a non-standard port is used in the request.)

With this code in place, we can now use the SetHostName property and still have our URL redirection work properly for HTTP/1.1 requests. We’re out of luck when it comes to HTTP/1.0 requests, but this is something that we can live with.