Simply Does Not Work

If you’re like me and you’re using the .NET Framework’s BinaryWriter class to write bytes to a stream, you might call the overload that takes a string as its parameter and expect to get the sequence of characters that is that string written to the stream as a series of bytes according to the encoding set in the BinaryWriter instance.

In other words, I was expecting the BinaryWriter‘s Write(string) implementation to look something like this pseudocode:

public void Write(string text)
{
     this.buffer.Append(this.encoding.GetBytes(text));
}

After all, when you call Write() with an int, you get four bytes written. If I call Write() with the string “TEST” and the BinaryWriter is using the ASCII encoding, I’d expect four bytes to be written. But instead it writes five.

That’s because the BinaryWriter writes the length of the string in one byte and then calls GetBytes() to output the string data.

Now that I think about it, this makes perfect sense: strings in the .NET framework are typically not thought of as being null-terminated, they’ve got a length, and in order for the BinaryReader‘s Read(string) method to work, it’ll need to be able to know the length of the string to determine how many bytes to read.

In my case, I was writing data to an Epson TM-T88III receipt printer, and given the structure of the commands that the printer expects, it doesn’t need or want the length of the string in this way. Because I didn’t read the MSDN documentation closely, I was left scratching my head as to why weird characters were showing up or characters were being omitted in my output.

The workaround is to just call GetBytes() myself and pass the byte buffer to the BinaryWriter‘s Write(byte[]) overload:

// "bw" is an instance of a BinaryWriter
// "barcode" is a string
 
bw.Write(AsciiControlChars.GroupSeparator);
bw.Write('k');
bw.Write((byte)4);
bw.Write('*');
bw.Write(Encoding.ASCII.GetBytes(barcode)); // NOT bw.Write(barcode);
bw.Write('*');
bw.Write(AsciiControlChars.Null);

Hope this saves someone from a forehead-slapping moment.

Puerto Rico is not a country.

At Skiviez/WFS, our “low-volume fulfillment requests” (which usually simply means “retail orders”) get a simple receipt printed out via an Epson TM-T88III receipt printer.

TM-T88III Receipt Printer

You’ve probably seen the TM-T88III before, though the current model is the TM-T88IV–they’re ubiquitous and especially common in restaurants. We chose it because the receipt prints quickly (which reduces the probability that a stack of order receipts gets missorted such that an employee puts the wrong receipt in the wrong parcel), is thermal so there is no ink to replace (which reduces the cost of consumables), uses standard thermal receipt paper (which is cheap, and we can always run out to Staples if we run out), and, perhaps most importantly, we found one in a box in the back of the warehouse.

At the top of the retail receipt, I print out the Skiviez logo. This is stored in a non-volatile area of memory in the printer (called NV RAM in EPSON parlance), and I dole out a command that essentially says “print out the image stored in slot #1,” having previously uploaded the bitmap via the printer driver’s flash utility. This worked great for as long as we only shipped Skiviez orders and only had one TM-T88III printer. But with the advent of WFS, however, Skiviez now ships orders (we call them “fulfillment requests”) for multiple e-commerce stores, each requiring a different logo to appear at the top of the receipt. So I now had two options:

• I could manually upload the logos for all of the merchants that we ship for into all of the receipt printers that we use, making sure to upload each merchant image into the same NV memory slot for each printer.
• I could store the merchant’s logo in our database, figure out how to use the “select bit image mode” receipt printer command, and just send the image data just-in-time whenever I generated a receipt.

The second option has obvious maintenance benefits, so that’s the path I started hacking my way down.

A brief primer on ESC/POS and data

Like the Zebra LP2844 and its brethren, the TM-T88III comes with a Windows printer driver that allows Windows and applications to see it as any other GDI-based printer. You can print a Word document to the TM-T88III and it will print, albeit slowly, and it’ll look like crap. That’s because, as with any other GDI printer, the Word document gets rendered as a bitmap and the entire big-ass bitmap gets sent to the poor little receipt printer which then prints it 24 dots at a time. A better method is to use the printer for its intended purpose and use the native printer language to generate the receipt. This allows us to select printer fonts that are specifically designed for the printer’s resolution and speed characteristics and use other advanced features of the printer such as the paper cutter.

That printer language is called ESC/POS, which is entirely unpronounceable and a stupid name. The “POS” stands for “Point of Sale” since this printer belongs to a class of devices commonly known as point of sale devices–cash drawers, VFDs, barcode scanners, and the like. The “ESC” stands for “escape” because the printer treats any data that is sent to it as text–passed straight through–unless it is escaped with a special character to indicate that a command instruction follows in the input stream. The most common escape character used is the ASCII ESC character, and so we have “ESC/POS” as our language name.

If you’ve just picked up one of these printers on eBay, you may find that getting documentation on this language is hard to come by because you need to get a copy of the “ESC/POS Application Programming Guide” from your “EPSON Authorized Dealer,” whoever the hell that is. (Again, we found ours in a box in the back of the warehouse, so my “authorized dealer” is long gone.) Still, some Googling will give you more or less complete and current copies of these guides, which I’ve mirrored here. Go ahead and download them:

• ESC POS Command Guide
• ESC POS Programming Guide

(I particularly like how the programming guide has “proprietary” and “confidential” stamped all over it and then proceeds to describe how great EPSON is for “taking initiative” for “expandability” and “universal applicability” on page 7 of the same document. You can’t make this crap up–it’s full of oxymorons.)

Having just come from writing in the EPL2 printer language for our Zebra LP2844 and Zebra ZP 500 Plus thermal label printers, I found the ESC/POS language to be confusing. My confusion mostly stemmed partly from the way the documentation was written and partly because I was used to the way EPL2 worked. Let’s take a look at the documentation for the bit image command that I’m trying to use:

If this were EPL2, I would actually send the document as a string to the printer, so if a command needed the integer 33 as a parameter, I would send the string “33″ (two bytes of data). In ESC/POS, each parameter is a single byte, so if I want to send 33 as a parameter for m in the bit image command above, then I need to send 33 in a single byte: that is, 0b100001, which is 2⁵ + 2⁰ = 32 + 1 = 33.

The reason that this is super confusing is that other parameters in the documentation are specified to be ASCII characters, such as the ‘*’ parameter in the bit image command above. This is because low-level programmers, such as those who designed the ESC/POS language, tend to blur the lines between data types: it’s all bytes at the end of the day. As a result, if you’re using C#, you might be tempted to use a StringBuilder to build up your document to send to the printer. Don’t do it! You’ll inevitably get confused by its overloads. Let’s take that m = 33 parameter as an example:

var sb = new StringBuilder();
 
// ASCII escape
sb.Append((char)27); 
 
// ASCII '*' for the bit image command
sb.Append('*'); 
 
// oops! this appends the string "33" in two bytes, doesn't work
sb.Append(33); 
 
// this is what I want, but semantically weird
sb.Append((char)33);

A much better and less confusing in the long run solution is to eschew any notion that we are dealing with text. Instead, let’s use the semantics that we are sending bytes of raw binary data directly to the printer:

using (var ms = new MemoryStream())
using (var bw = new BinaryWriter(bw))
{
     bw.Write(AsciiControlChars.Escape);
     bw.Write('*');         // bit-image mode
     bw.Write((byte)33);    // 24-dot double-density
}

But remember that cast to byte! Otherwise, you’ll get a C# int, which is four bytes, written to the stream, when we only wanted one. Yes, that means that any parameter that an ESC/POS command takes has a maximum value of 255.

You could get by on the StringBuilder method for a while and not have it burn you until you try the bit image command because all the characters that you’re appending happen to be less than 128–greater than that and you start getting into bytes that don’t map to a Unicode character for quirky historical reasons, such as the range 128 to 159, and you’ll be pulling your hair out as to why some of your data is getting “lost.” Just use the BinaryWriter method. You can thank me later.

Why is the bit image command designed this way?

The m parameter in the bit image command as defined in the previous section has 4 values, and each value changes the way we construct the image data bytes and the way the printer interprets them. The first three values are of dubious value, so let’s throw them out for now. We’ll focus on m = 33, which means “24-dot double density mode.” So keep this in mind and set this aside.

When we think of drawing a bitmap in high level software, we usually think of drawing it pixel by pixel, left to right, top to bottom. So the order that the dots get printed as indicated by the programming guide seems strange:

That is, the dots are rendered from top to bottom, then left to right.

Why in the blue hell do I need to rearrange my nice and neat array of bitmap data into this Connect Four scheme of dots? It all comes down to the size of the print head.

For our mental model, the thermal print head in the receipt printer is physically 1 dot wide by 24 dots high where there are 203 dots per inch (square pixels). It moves left to right across the receipt paper, burning up to 24 dots in the vertical (y) direction for each dot in the horizontal (x) direction.

Since we’re talking about a thermal printer here, there’s no concept of gray scale tones–we’ve either burned a dot into the paper (black) or we haven’t. So if we’re sending image data to the printer in bytes, it would make sense to say that a bit set to 1 means “burn a dot” and a bit set to “0″ means do nothing.

But a byte is only 8 bits wide, and we have 24 dots to burn in each “slice” of the image. 24 just happens to be divisible by 8, so we can send 3 bytes of data for each slice to represent our 24 dots. (If the bitmap I want to draw is taller than 24 dots/pixels, then I need to send multiple bit image commands, effectively doing multiple stripes of the image that are 24 dots high; more on this later.)

So, now, the diagram in the programming guide makes sense: we’re not burning 1 dot at time, we’re burning a vertical stripe that is 1 dot wide and 24 dots high at a time, and then moving to the right. So our printer reads our first 3 bytes of image data, burns the dots specified by those 24 bits, then reads the next 3 bytes, and so on.

Aside: Converting the bitmap to monochrome

Right. So a merchant has given me a *.bmp file, and I need to convert that into array of bits that I can send to the receipt printer. It’s a good bet that the bitmap that the merchant sent me for the logo is not monochrome (that is, every pixel is either 100% black or 100% white).

So what we can do is look at each pixel in the bitmap, determine its luma, and if that’s below a certain threshold, count that pixel as black. How did I write the code to do this? I looked the formula up via the intertubes and modified it to fit my needs:

private static BitmapData GetBitmapData(string bmpFileName)
{
    using (var bitmap = (Bitmap)Bitmap.FromFile(bmpFileName))
    {
        var threshold = 127;
        var index = 0;
        var dimensions = bitmap.Width * bitmap.Height;
        var dots = new BitArray(dimensions);
 
        for (var y = 0; y < bitmap.Height; y++)
        {
            for (var x = 0; x < bitmap.Width; x++)
            {
                var color = bitmap.GetPixel(x, y);
                var luminance = (int)(color.R * 0.3 + color.G * 0.59 + color.B * 0.11);
                dots[index] = (luminance < threshold);
                index++;
            }
        }
 
        return new BitmapData()
            {
                Dots = dots,
                Height = bitmap.Height,
                Width = bitmap.Width
            };
    }
}

BitmapData is just a little struct that contains the three properties that you see here. After all, once I have my BitArray with 1 indicating black dots and 0 indicating white dots, I don’t need the original Bitmap instance anymore.

Just to clarify, this means that we’re holding onto a BitArray of monochrome data in the Dots property that conceptually looks something like this:

Marshalling the monochrome data

Now for the trickiest part of all. We need to take our BitArray of monochrome data, divide it up into 8-dot chunks, represent those 8-dot chunks as bytes, and send them to the printer in the order required by the bit image command, discussed above. That is, I’ll need to send the data as bytes in this order:

So the printer will draw them one vertical stripe of three bytes each at a time. (If you don’t get it, print out that image, cut up the bytes, and then arrange them in the order shown in the diagram from the EPSON documentation. It’ll instantly make sense once you see it.) The X’s in the diagram correspond to bits that aren’t in our original image that we have to send anyway as padding. Remember, since we selected the 24-dot double density mode, the printer is going to draw a 1 x 24 dot slice as it moves the print head. My example bitmap is only 4 pixels tall, so I have to send 20 zero bits as padding.

And thus we would end up with our pretty image.

Unfortunately, there is some math involved in translating the bits in our BitArray into these bytes that we need to send. Assuming bw is our reference to a BinaryWriter, here’s the code that does just that:

// So we have our bitmap data sitting in a bit array called "dots."
// This is one long array of 1s (black) and 0s (white) pixels arranged
// as if we had scanned the bitmap from top to bottom, left to right.
// The printer wants to see these arranged in bytes stacked three high.
// So, essentially, we need to read 24 bits for x = 0, generate those
// bytes, and send them to the printer, then keep increasing x. If our
// image is more than 24 dots high, we have to send a second bit image
// command to draw the next slice of 24 dots in the image.
 
// Set the line spacing to 24 dots, the height of each "stripe" of the
// image that we're drawing. If we don't do this, and we need to
// draw the bitmap in multiple passes, then we'll end up with some
// whitespace between slices of the image since the default line
// height--how much the printer moves on a newline--is 30 dots.
bw.Write(AsciiControlChars.Escape);
bw.Write('3'); // '3' just means 'change line height command'
bw.Write((byte)24);
 
// OK. So, starting from x = 0, read 24 bits down and send that data
// to the printer. The offset variable keeps track of our global 'y'
// position in the image. For example, if we were drawing a bitmap
// that is 48 pixels high, then this while loop will execute twice,
// once for each pass of 24 dots. On the first pass, the offset is
// 0, and on the second pass, the offset is 24. We keep making
// these 24-dot stripes until we've run past the height of the
// bitmap.
int offset = 0;
 
while (offset < data.Height)
{
    // The third and fourth parameters to the bit image command are
    // 'nL' and 'nH'. The 'L' and the 'H' refer to 'low' and 'high', respectively.
    // All 'n' really is is the width of the image that we're about to draw.
    // Since the width can be greater than 255 dots, the parameter has to
    // be split across two bytes, which is why the documentation says the
    // width is 'nL' + ('nH' * 256).
    bw.Write(AsciiControlChars.Escape);
    bw.Write('*');         // bit-image mode
    bw.Write((byte)33);    // 24-dot double-density
    bw.Write(width[0]);  // width low byte
    bw.Write(width[1]);  // width high byte
 
    for (int x = 0; x < data.Width; ++x)
    {
        // Remember, 24 dots = 24 bits = 3 bytes. 
        // The 'k' variable keeps track of which of those
        // three bytes that we're currently scribbling into.
        for (int k = 0; k < 3; ++k)
        {
            byte slice = 0;
 
            // A byte is 8 bits. The 'b' variable keeps track
            // of which bit in the byte we're recording.                 
            for (int b = 0; b < 8; ++b)
            {
                // Calculate the y position that we're currently
                // trying to draw. We take our offset, divide it
                // by 8 so we're talking about the y offset in
                // terms of bytes, add our current 'k' byte
                // offset to that, multiple by 8 to get it in terms
                // of bits again, and add our bit offset to it.
                int y = (((offset / 8) + k) * 8) + b;
 
                // Calculate the location of the pixel we want in the bit array.
                // It'll be at (y * width) + x.
                int i = (y * data.Width) + x;
 
                // If the image (or this stripe of the image)
                // is shorter than 24 dots, pad with zero.
                bool v = false;
                if (i < dots.Length)
                {
                    v = dots[i];
                }
 
                // Finally, store our bit in the byte that we're currently
                // scribbling to. Our current 'b' is actually the exact
                // opposite of where we want it to be in the byte, so
                // subtract it from 7, shift our bit into place in a temp
                // byte, and OR it with the target byte to get it into there.
                slice |= (byte)((v ? 1 : 0) << (7 - b));
            }
 
            // Phew! Write the damn byte to the buffer
            bw.Write(slice);
        }
    }
 
    // We're done with this 24-dot high pass. Render a newline
    // to bump the print head down to the next line
    // and keep on trucking.
    offset += 24;
    bw.Write(AsciiControlChars.Newline);
}
 
// Restore the line spacing to the default of 30 dots.
bw.Write(AsciiControlChars.Escape);
bw.Write('3');
bw.Write((byte)30);

This code looks confusing, and is, but I’ve tried to document it with explanatory comments in line.

Sending the document to the printer

Finally, there is the task of sending this array of bytes directly to the printer. I’ve talked about this before and the code is uninteresting, requiring some P/Invokes into native APIs. Essentially, you pass in the name of the printer as it appears in the Control Panel, the array of bytes from our BinaryWriter, and you’re all set:

private static void Print(string printerName, byte[] document)
{
    NativeMethods.DOC_INFO_1 documentInfo;
    IntPtr printerHandle;
 
    documentInfo = new NativeMethods.DOC_INFO_1();
    documentInfo.pDataType = "RAW";
    documentInfo.pDocName = "Bit Image Test";
 
    printerHandle = new IntPtr(0);
 
    if (NativeMethods.OpenPrinter(printerName.Normalize(), out printerHandle, IntPtr.Zero))
    {
        if (NativeMethods.StartDocPrinter(printerHandle, 1, documentInfo))
        {
            int bytesWritten;
            byte[] managedData;
            IntPtr unmanagedData;
 
            managedData = document;
            unmanagedData = Marshal.AllocCoTaskMem(managedData.Length);
            Marshal.Copy(managedData, 0, unmanagedData, managedData.Length);
 
            if (NativeMethods.StartPagePrinter(printerHandle))
            {
                NativeMethods.WritePrinter(
                    printerHandle,
                    unmanagedData,
                    managedData.Length,
                    out bytesWritten);
                NativeMethods.EndPagePrinter(printerHandle);
            }
            else
            {
                throw new Win32Exception();
            }
 
            Marshal.FreeCoTaskMem(unmanagedData);
 
            NativeMethods.EndDocPrinter(printerHandle);
        }
        else
        {
            throw new Win32Exception();
        }
 
        NativeMethods.ClosePrinter(printerHandle);
    }
    else
    {
        throw new Win32Exception();
    }
}

The P/Invoke declarations can be found in the sample code at the end of the article, and the documentation is available on MSDN. But, again, there’s nothing sexy going on here.

Conclusions and delusions

And that’s a wrap. I always find it interesting when I encounter a problem at work–which is a small business by any definition of the phrase–and the Google indicates that no one has ever encountered or documented this problem before. And it’s these types of integration problems that I find fascinating; after years of playing in castles in the sky, dealing with databases and exceptions and HTML forms, it’s nice to know that I can dust off the corners of my brain and my college degree and deal with bits and bytes when I need to. Writing software that makes hardware do stuff is always fun, too.

And after hours of watching the receipt printer spew out lines and lines of gibberish, there’s nothing like the feeling of seeing the bitmap print out and knowing that, finally, things have simply started to work.

Good luck!

Download the code used in this article.

There is nothing quite like dashing to work at night to restart the “SMC Comcast Business IP Gateway” because it stopped routing packets for one of our static IP addresses.

The reason? Oh, it just does that sort of thing every 25 days or so.

At work, I’ve written a small application called the “Fulfillment Manager.” From a user’s perspective, it’s an extremely simple application. It shows the current order counts for all of the stores that we ship for, and if you scan a barcode, it figures out what store that barcode belongs to, determines if the order the barcode corresponds to needs to be packed or shipped, and prints out a receipt/packing slip or USPS/FedEx shipping label and supporting shipping documentation automatically. Most operations involve just scanning the barcode and pressing enter.

Yes, it epitomizes Battleship Grey. You love it.

But, behind the scenes, it’s not quite as simple as all of that. It’s aggregating order data from heterogeneous data sources–some in our legacy database, some in our new fulfillment system, some in a custom integration with a third party. It has to figure out which postage account to pay for postage with or which FedEx account number to use to ship a package. For orders that aren’t coming from our new fulfillment system, it has to “cleanse” the address against the current USPS address database. It has to figure out the cheapest way to ship a package, compute customs values correctly, generate certificates of origin and commercial invoices for international shipments, and determine what box types an order is allowed to be packed in. And it has to write shipment information back to one of those three disparate data sources.

What this means is that I’m frequently making adjustments and bug fixes to the application. And managing the deployment and installation of those bug fixes had been, up until now, a pain.

A brief interlude on ClickOnce

The other internal application that we use (“Undies Client”) for our long-time running e-commerce store is deployed via ClickOnce, which is essentially the Java Web Start of the .NET world.

While ClickOnce is a neat technology and has its applications, to be sure, I probably wouldn’t use it again on Undies Client if I were starting that application over today, just as I decided not to use it for the Fulfillment Manager (which is an effort to divorce the processing and shipping features from Undies Client and make them simpler and applicable to multiple e-commerce stores).

First, there’s user confusion. If I deploy a Windows Installer MSI via Group Policy, then the application is magically there on all computers in the office. But for Undies Client, you have to go to a special Web page and click on a link. With ClickOnce, the installation happens per user, so employees can get confused if they go to another computer one day, log into their account, and see that the app isn’t there (“but Jennifer runs it on this computer so I thought it was already installed”).

Second, there’s deployment headaches. Like Java Web Start, you get a retarded warning if the deployment manifest wasn’t signed with an expensive code signing certificate. To mitigate that, you either buy one or start diddling with the self-signing certificate capability within the context of your own Active Directory domain. Not a show-stopper, and it makes sense, I guess, but it’s One More Thing that you have to deal with.

Third, when that certificate expires and needs to be renewed, you’re in for a world of hurt, because essentially all users will need to uninstall and re-visit the Web site download link and reinstall. Otherwise, the application simply stops seeing the newer updated versions and doesn’t update itself.

Fourth, the distribution of your app now has a dependency on an IIS installation somewhere, so that’s something else to maintain–both the configuration of that virtual directory in IIS as well as the shared drive to which Visual Studio dumps its files when clicking the “Publish” button.

Fifth, the installation can’t do much. Until recently, you couldn’t even create an icon on the desktop as part of the installation process. Nor can you do anything that would require elevated permissions for actions that you might typically do when running an installer, such as registering a COM DLL, or installing some third-party dependency. So the Web page at which you download the app usually contains things like “ooh be sure to install this that and the other first,” defeating the deployment simplicity of ClickOnce. And if you need to update one of those third party dependencies and your app because dependent on one of those updates, you have no way to update that dependency with ClickOnce, unless you take it upon yourself to have your application manage the upgrade during its next run. That’s just more work than you shouldn’t have to do.

After writing all of this, it may seem like I am saying that ClickOnce is a half-baked load of crap; it’s not half-baked. I’m saying it’s a fully-baked, complete load of crap. (Kidding. ClickOnce has its applications for applications that can be completely self-contained, but if at any point you become dependent on anything COM, it’s time to move on to real deployment technology.)

Using WiX to create a Windows Installer MSI file

I’ve blogged about WiX before. It’s a great open source tool put together by some guys who decided to write a reasonable mechanism for generating Windows installers because, for some reason, the Windows installer team has seemed to think that editing database tables in a cheeseball editor called Orca was sufficient. This would be like saying that our warehouse workers could ship orders by updating data in a Microsoft Excel spreadsheet.

You could also pay lots of money for InstallShield or something similar, which would create MSI installers for you, but installation is a convenience for me–as a small business whose primary focus is not end-user software, paying for that doesn’t make much sense. There’s also NSIS, but, oh–I just threw up all over myself. We’ll save NSIS for another post.

Additionally, the WiX guys have realized that installers usually need to do useful things, like install certificates, set up Web sites in IIS, and run database scripts, whereas the Windows Installer team seems to have been trying to make writing custom actions harder, not easier, with their subsequent releases, because that’s where most of the crashes and problems in setup packages happen. With WiX, we now have a suite of well-tested custom actions that lots of people are using; this should have been the Installer team’s original response instead of depending on the community and third parties to fill in this gap for them, but it is what it is.

The point is that WiX enables a whole class of small business developers like me to build first-class deployment methods into their applications. With a 300-odd line XML file, the Fulfillment Manager now builds to an MSI file. And since WiX integrates with Visual Studio, I can generate that XML file as part of my build process.

Indeed, I’ve set up TeamCity and use this as a continuous integration server. Whenever I commit a change to Subversion, TeamCity picks up the change, compiles the solution, runs the tests, and if they pass, copies the newly generated MSI file to a network share for potential deployment. It’s pretty sweet.

The missing piece of the puzzle, then, is actually getting this freshly baked MSI file onto all of the client machines in the office.

Deploying via the Group Policy Software Installation Extension

My first instinct was to use the Group Policy Software Installation Extension. This is the thingie where when you open up a group policy in the Group Policy Management Editor, you can drill down to the Software Installation thing under Computer Configuration, specify an MSI file for deployment, and (presto!) any computers linked to that GPO will install the MSI on next boot.

This worked swimmingly well for the first release of Fulfillment Manager.

We pause for another brief interlude on update strategies

Let me explain the design of the installation for a moment. I’ve written my WiX file such that each new MSI file that it generates is a “major upgrade” in the Windows Installer parlance–it has a different product code, a different package code, and a different version number (since my version numbers are a combination of an incrementing build number and the Subversion revision number). But they all have the same upgrade code and I schedule RemoveExistingProducts during the install.

This means that if you have an older version of the Fulfillment Manager on your machine and double-click the MSI file for a newer, updated version, you don’t have to do anything–the existing version is completely uninstalled and the new version is installed on top of it.

The Windows Installer has support for “minor upgrades” and “small updates”, but I can never keep the damn things straight. Can I add a new component? Can I change a file? Can I reorganize a feature? Do I really want to be thinking about this every time I press Build in Visual Studio? My application is small, so I think it’s far easier and more reassuring to just blow away the whole thing and install again during an update, starting with a clean slate each time. In fact, I think this is a reasonable approach for many reasonably-sized applications (Paint.NET, for example, does this) and only becomes a problem when you start getting really large (such as Visual Studio or Microsoft Word).

Getting the update out there

OK. So all I really need to do is get all of the client computers to run msiexec /i FulfillmentManager.msi on the MSI and I’ll be good to go.

You might think that I could just overwrite the old MSI file on the network share with the new one, and the computers would notice the change at the next boot. But you would be, as I was, a fool–machines that already had the install would not notice the change and machines that did not have the install would freak out because they could not find the correct MSI file. After positing my question on ServerFault, I discovered the way the Group Policy Software Installation Extension works is by creating an advertisement script (*.aas file) and referencing that script via an object sitting in the Active Directory. That LDAP entry and the script file both do annoying things like reference a specific package code and product code, both of which I change with every new build of my software. So this method is out for the count.

Similarly, lugging out the Group Policy Editor and trudging down to the package entry and clicking “Redeploy application…” won’t work for the reasons described above–except that it’ll break the machines that already have the software installed, too.

What works is lugging out the Group Policy Editor, trudging down to the package entry, clicking “Remove” and “Immediately remove”, and then adding the package right back. This creates an updated *.aas file and correspondingly updated LDAP entries, and the old LDAP entry is flagged with a “remove me now please” flag. This works, but there are three things that I don’t like about it:

• It’s a manual process, so I have to remember to do it every time I create a new deployable build.
• References to all of the old versions hang around by necessity, since it’s recording the fact that “hey, if I see this particular product code then I need to uninstall it.” Indeed, upon inspecting the SYSVOL share, I saw that there about 45 of such files sitting in there since development of this app started in early July.
• There is no way to perform this process programmatically. (Well, there is, it’s documented as part of the EU anti-trust settlement, but let’s get real now: if it has LDAP in the spec then I’m not touching it with a ten-foot pole. Plus, this would be work that is totally tangential to my problem, which is automating a 1-minute task that ignores the crap out of me. At several days’ worth of work, it’d take me a long time to climb out of that time deficit to realize any savings.)

Nirvana: Automatic updating

The Software Installation extension for Group Policy can do a lot of things that I don’t need, like using patches or transforms, or shifting installed software by just moving a computer to a different OU, when at the end of the day all I really wanted was something that looked like this:

Must this be so difficult?

Since my installer will uninstall any previous versions, all I need to do is run the installation package via msiexec. This is easy enough to do via batch script that I’ve configured to run at startup via group policy, since those batch scripts run as SYSTEM and will have the necessary privileges to complete successfully.

You would think that in addition to /i and /x (and the idiotic /vomus), msiexec would have a somewhat useful parameter called, oh, I don’t know, /install-it-only-if-the-damn-thing-isn't-already-installed, but that would be a useful feature, so of course the Windows Installer team didn’t actually implement it.

Now, granted, there is actually no harm in running my installer when my app is already installed. It just would check that all the components are indeed installed and exit. But this still leaves a bad taste in my mouth. Not all MSI’s that I want to use in this way might behave like this. And, if I have any custom actions, that means that they’ll also get run on every boot, which seems like a waste.

While I could spelunk through the registry to try and see if I can find my current product code in the list, I’d rather not do that, because (1) that might not work consistently on all versions of Windows and (2) really, I’m just not that comfortable with batch scripts to begin with.

What ended up doing is writing a little command-line program called msicheck. The source file is boring and decently commented. Because all of the Windows Installer APIs are in C, my utility is in C. I briefly thought about writing it in C# with some interop, because I hate C just that much, but that did seem awfully overkill for such a simple application.

Ah, C in Windows. I had long forgotten the days of Hungarian variable names that sound like Yosemite Sam cussing (“lpszPackageVer“) and the land of if ("swizzle" == "swizzle") returning false. Heck, I had forgotten the days of not evening having false! I digress; the code is probably awful, but seems to work well enough for my purposes.

> msicheck.exe /?
 
Determines if a given Windows Installer package is installed.
 
msicheck package
 
Exit Codes:
        0       Exact version installed
        1       General failure
        2       Path to MSI package not found or not accessible
        3       Error determining package status
        4       Newer version installed
        5       No version installed
        6       Older version installed

All msicheck does is take the full path to an MSI file as an argument and returns an exit code that indicates the status of that particular package by looking at the product code and the version. (That is, if the product code is not installed, it returns 5; if the same product code is found, it compares the version and returns 0, 4, or 6.) I call it in my batch script (which, I don’t do this for a living, it’s probably retarded) like so:

@ECHO OFF
 
REM ---------------------------------------------------------------------
REM VARIABLES
REM ---------------------------------------------------------------------
 
SET path=\\SKIVIEZSBS2008\Group Policy Installations\Fulfillment Manager\
SET package=%path%Skiviez.FulfillmentManager.Installer.WinForms.msi
SET msicheck=%path%msicheck.exe
 
ECHO Checking installed version of Fulfillment Manager...
 
"%msicheck%" "%package%"
 
IF ERRORLEVEL 0 IF NOT ERRORLEVEL 1 (
	echo Fulfillment Manager is up to date.
) ELSE (
	IF ERRORLEVEL 5 (
		echo Installing latest version...
		%SYSTEMROOT%\system32\msiexec /qn /i "%package%"
		IF ERRORLEVEL 0 (
			echo Fulfillment Manager successfully updated.
		) ELSE (
			echo Errors may have occurred during installation.
		)
	) ELSE (
		echo Newer version installed or error occurred.
	)
)

Keeping in mind that IF ERRORLEVEL 5 evaluates to true for error levels of 5 or greater, this means that I call msiexec only if the product isn’t installed or if an older version is installed.

Conclusions and Delusions

So, by using a batch script and my msicheck utility, I have things the way I want them. I commit a change; TeamCity builds it, runs the tests, and copies the resulting MSI to the network share that is referenced in my batch script; and the batch script runs at the next boot, uses msicheck to note that that package is not installed, and so runs msiexec to install the update (which removes the old version as part of the install process).

It’s certainly not applicable to every software deployment scenario, much like ClickOnce, but hopefully it’ll help someone out there who wanted to do something similar.

Download MsiCheck, which requires Windows Installer 4.5 to run. Complete with the “works on my machine!” guarantee of absolutely no warranties.

Good luck!

When attempting to re-integrate a feature branch into a trunk in my Subversion repository, I was getting a weird “Cannot reintegrate into a working copy not entirely at infinite depth” error.

To fix this, I had to fix two problems.

First, my working copy was very “old.” Like, two years of sitting on my desktop old. And I had upgraded Subversion since that time. Apparently, something was not happy with this, particularly related to some new checkout option not being set. So I created a new directory and made a fresh checkout, specifying the “Fully recursive” option in TortoiseSVN.

When I did this, I got a new message: “Retrieval of mergeinfo unsupported.” Still an error, but different!

I had upgraded my Subversion server to 1.6, but I never actually upgraded my repositories to the latest version, which was created way back in 1.4. So I had been operating in a kind of backwards-compatibility mode for two years. Whoops.

Running svnadmin upgrade c:\path\to\my\repo on the server fixed this right up. I also ran the recommended svn-populate-node-origins-index.exe program, which I’m sure did something magical.

Now I can complete the merge successfully in TortoiseSVN. Hope this helps someone.

So, as I’ve described previously, we’ve set up a Magento store to clear out some of the older inventory at the company I work for: consider Men’s Underwear Discounters to be the bargain bin, Sunday! Sunday! Sunday! closeout section for Skiviez. Using a free, open source platform like Magento has saved us a lot of time and money, especially for a store that we’re not really making huge amounts of money off of anyway (it’s there to recoup the costs of products that didn’t sell well, sort of like damage control).

Aside: You probably have configured your sales e-mails incorrectly

I have seen three live Magento installations. I have also seen three Magento installations where sales e-mails have been customized improperly. Since it has happened three times, by the engineer’s first law of truth, it must be true that many people have been configuring the sales e-mails in their Magento stores improperly.

By default, all of the sales e-mails are written in less-than-stellar English and contain references to things like “Magento Demo Store” and “(555) 555-5555.” And these sales e-mail templates live by default in the /app/design/frontend/default/default/template/email directory.

If you’re like us, you made the mistake of editing these files directly, replacing the references to “Magento Demo Store” with references to your own store, like “Men’s Underwear Discounters.” And this would work fine until you, like us, apply the next security update or upgrade via Magento Connect. If they’ve updated the default template, then you can be slightly bemused that, like us, all of your sales e-mails have reverted to the defaults, and you are now thanking customers for shopping at the “Magento Demo Store.” The result? You, like us, look like an idiot.

What we need is a way to make sure that our e-mail customizations persist across security updates and upgrades.

The way to do that is to set up new copies of each and every e-mail. From the Magento Admin screen, you need to

1. Choose System > Transactional E-mails from the menu.
2. Click “Add New Template.”
3. Under “Load default template”, select the template you would like to customize, then hit “Load Template”. The “Template Subject” and “Template Content” fields are populated with a copy of the template you selected.
4. Edit that template to your heart’s content, give it a name, and hit “Save Template”.
5. Under System > Configuration > Sales E-mails, be sure to select your new template instead of “Default from locale”.

Hooray! Now the changes that we make will stick around.

Formatting those e-mails as plain text

For Skiviez proper, we send multipart e-mails, where both HTML- and text-formatted messages are sent within the same e-mail. This works great because, depending on the customer’s preference and e-mail capabilities, they can view the e-mail in the format that they prefer.

Unfortunately, Magento doesn’t (at the time of this writing) have the capability for these multipart e-mails. Instead, you get to choose between HTML (the default) and plain text. I decided to go with plain text because

• I had seen issues on the forums where the HTML e-mails have problems displaying in some Web-based e-mail clients; and
• Nothing pisses me off more than HTML e-mail. I don’t know why. I just hate it.

Converting most of the e-mails to plain text is straight-forward; you just hit the “Convert to Plain Text” button and then reformat the template content so that it is not hideous. But when you get to the order e-mail, you’ll discover that some of the variables and references are returning the content in HTML; particularly, the address, payment, and item summary fields.

You could do something like changing the reference from

{{ var order.getBillingAddress().format('html') }}

to

{{ var order.getBillingAddress().format('text') }}

and that would work, but the output would not be what you expect:

 
Nicholas Piasecki
 
3005 Some Street
 
 
 
Sillytown,  California, 92683
United States
T: 5555555555

That is, if an element in the address is empty, it still receives a blank space, so you end up with the odd address formatting that you see in the above example.

You could then spend an hour and a half just trying to figure out what that magic ->format('html') or ->format('text') call actually does and where it’s coming from. That is an adventure in and of itself, but to make a long story short, there is a config.xml file in the /app/code/core/Mage/Customer/etc/ directory that contains entries like the following:

                    <text translate="title" module="customer">
                        <title>Text</title>
                        <defaultFormat><![CDATA[
{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}
{{depend company}}{{var company}}{{/depend}}
{{var street1}}
{{depend street2}}{{var street2}}{{/depend}}
{{depend street3}}{{var street3}}{{/depend}}
{{depend street4}}{{var street4}}{{/depend}}
{{depend city}}{{var city}},  {{/depend}}{{depend region}}{{var region}}, {{/depend}}{{var postcode}}
{{var country}}
T: {{var telephone}}
{{depend fax}}F: {{var fax}}{{/depend}}
                        ]]></defaultFormat>
                    </text>
                    <oneline translate="title" module="customer">
                        <title>Text One Line</title>
                        <htmlEscape>true</htmlEscape>
                        <defaultFormat>
<![CDATA[{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}, {{var street}}, {{var city}}, {{var region}} {{var postcode}}, {{var country}}]]>
                        </defaultFormat>
                    </oneline>
                    <html translate="title" module="customer">
                        <title>HTML</title>
                        <htmlEscape>true</htmlEscape>
                        <defaultFormat><![CDATA[
{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}<br/>
{{depend company}}{{var company}}<br />{{/depend}}
{{var street1}}<br />
{{depend street2}}{{var street2}}<br />{{/depend}}
{{depend street3}}{{var street3}}<br />{{/depend}}
{{depend street4}}{{var street4}}<br />{{/depend}}
{{depend city}}{{var city}},  {{/depend}}{{depend region}}{{var region}}, {{/depend}}{{var postcode}}<br/>
{{var country}}<br/>
{{depend telephone}}T: {{var telephone}}{{/depend}}
{{depend fax}}<br/>F: {{var fax}}{{/depend}}
            ]]></defaultFormat>
                    </html>

Once you stumble across that, it’s not so difficult to figure out what is going on. The Magento address object returns a Varien_Filter_Template object that takes one of the above defaultFormat entries as its template based on the parameter that you pass into the format() function. So you can see that valid parameters are ‘html’, ‘text’, ‘oneline’, and so on.

The {{depend}} syntax means that if the enclosed variable is empty, then that output won’t appear. For HTML, this works fine, since newlines aren’t signficant. But for the text template, this doesn’t work, because even though things like {{street2}} get omitted, the newline is still sitting there in the template. This is why we get the fugly output in our plain text e-mails.

My solution is to define my own block that I just reference in my e-mails. Here’s the e-mail template for Men’s Underwear Discounters, for example:

Dear {{var order.getCustomerName()}},
 
Thank you for your order from Men's Underwear Discounters.
Once your package ships, we will send an email with a link 
to track your order. You can check the status of your order
by logging into your account.
 
If you have any questions about your order please contact
us at support@mensunderweardiscounters.com.
 
Your order confirmation is below. Thank you again for your 
business.
 
ORDER {{var order.increment_id}} 
 
Placed on {{var order.getCreatedAtFormated('long')}}
 
BILLING INFORMATION
 
{{block type='core/template' area='frontend' template='email/order/addresstext.phtml' address=$order.getBillingAddress()}}
 
SHIPPING INFORMATION
 
{{block type='core/template' area='frontend' template='email/order/addresstext.phtml' address=$order.getShippingAddress()}}
 
via {{var order.getShippingDescription()}}
 
ORDER DETAILS
 
{{block type='core/template' area='frontend' template='email/order/itemstext.phtml' order=$order}} 
 
Thanks for shopping at Men's Underwear Discounters.
 
Sincerely,
The Men's Underwear Discounters Team
support@mensunderweardiscounters.com

If you compare this with the default e-mails, you can see that I’ve added these new {{block}} entries in several places. All they are saying is “hey, create a new static block with the file at /app/design/frontend/default/default/template/email/order/itemstext.phtml (or addresstext.phtml) and pass a variable into it. Whatever variable I pass in here will be accessible in the .phtml file by referencing $this->getWhatever(), where “Whatever” is the name of the parameter. For example, my itemstext.phtml file could reference $this->getOrder() and my addresstext.phtml file could reference $this->getAddress().

These “itemstext.phtml” and “addresstext.phtml” files don’t exist in the default installation. I had to create them and FTP them to that appropriate directory. The contents aren’t exotic; they’re just PHP files that emit output that’s suitable for dropping into a plain text e-mail.

Here’s the itemstext.phtml file:

<?php
	$order = $this->getOrder();
 
	if ($order)
	{
		foreach ($order->getAllItems() as $item)
		{
			if ($item->getParentItem())
			{
				continue;
			}
 
			echo '(';
			echo $item->getQtyOrdered() * 1;
			echo ')';
 
			echo ' ';
 
			echo $item->getName();
 
			echo ' @ ';
 
			echo number_format($item->getPrice(), 2);
 
			echo ' ea = ';
 
			echo number_format($item->getRowTotal(), 2);
 
			echo "\n";
		}
 
		echo "\n";
 
		echo 'Subtotal: $';
		echo number_format($order->getSubtotal(), 2);
		echo "\n";
 
		echo 'Discount: $';
		echo number_format(0.00 - $order->getDiscountAmount(), 2);
		echo "\n";
 
		echo 'Shipping & Handling: $';
		echo number_format($order->getShippingAmount(), 2);
		echo "\n";
 
		echo 'Tax: $';
		echo number_format($order->getTaxAmount(), 2);
		echo "\n";
 
		echo 'Grand Total: $';
		echo number_format($order->getGrandTotal(), 2);
		echo "\n";
	}
?>

And here’s the addresstext.html file:

<?php
	$address = $this->getAddress();
 
	if ($address) {
		if (strlen($address->getPrefix()) > 0) {
			echo $address->getPrefix() . ' ';
		}
 
		echo $address->getFirstname();
 
		if (strlen($address->getMiddlename()) > 0) {
			echo ' ' . $address->getMiddlename();
		}
 
		echo ' '. $address->getLastname();
 
		if (strlen($address->getSuffix()) > 0) {
			echo ' ' . $address->getSuffix();
		}
 
		echo "\n";
 
		if (strlen($address->getCompany()) > 0) {
			echo $address->getCompany() . "\n";
		}
 
		echo $address->getStreet1() . "\n";
 
		if (strlen($address->getStreet2()) > 0) {
			echo $address->getStreet2() . "\n";
		}
 
		if (strlen($address->getStreet3()) > 0) {
			echo $address->getStreet3() . "\n";
		}
 
		if (strlen($address->getStreet4()) > 0) {
			echo $address->getStreet4() . "\n";
		}
 
		if (strlen($address->getCity()) > 0) {
			echo $address->getCity() . ', ';
		}
 
		if (strlen($address->getRegion()) > 0) {
			echo $address->getRegion() . ' ';
		}
 
		echo $address->getPostcode() . "\n";
 
		echo $address->getCountry();
 
		if (strlen($address->getTelephone()) > 0) {
			echo "\n" . 'T: ' . $address->getTelephone();
		}
	}
?>

Conclusions and delusions

This story has been typical of my experiences with Magento. I eventually end up finding something that was thoughtfully designed (if slightly overarchitected), but poor documentation and difficulty testing impede progress more than necessary. Let’s hope that this gets better in future versions of the framework.

Good luck.

One of the pieces of WFS is an application that I’ve written that runs on our Honeywell Dolphin 7600. The Dolphin is a little mobile device than runs Windows CE 5.0, has 802.11g access, and has a barcode scanner. The application that I’ve written allows a warehouse employee to pick fulfillment requests, scan in inbound shipments against a previously recorded manifest, and process any returns that customers have sent back. Pretty neat stuff.

I’ve designed most of the interface so that it can be used with just the big green “SCAN” button and the numeric keypad, since it’s cumbersome to bust out a stylus and use two hands while using it in the warehouse. There are, however, a few cases where the user does need to type something in, typically via the soft input panel, which is the stupid Microsoft name for the on-screen keyboard. Examples where the on-screen keyboard are necessary include typing in a merchant’s identifier for an inbound shipment manifest or typing in a user name and password to sign into the application.

By default, the soft input panel displays “whereever it feels like,” typically at the bottom of the screen or whatever random location the user last left it in, and while the user can drag it out of the way with the stylus if it obstructs UI elements, I thought it would be nice if I showed, hid, and positioned the soft input panel automatically when necessary. For example, if the user focuses a textbox where I expect they’ll need to type in some text, I’ll pop up the soft input panel and position it automatically just underneath that textbox.

If you load up Visual Studio, you’ll see the Microsoft.WindowsCE.Forms.InputPanel available in the toolbox. But it doesn’t provide any way to position the input panel once it is displayed. For that, I had to P/Invoke into a native method.

/// <summary>
/// Shows the input panel relative to the given control.
/// </summary>
/// <param name="control">The control.</param>
private void ShowInputPanel(Control control)
{
	NativeMethods.SIPINFO sipInfo;
	var x = 0;
	var y = control.PointToScreen(new Point(0, control.Height)).Y;
 
	this.inputPanel.Enabled = true;
 
	sipInfo = new NativeMethods.SIPINFO();
	sipInfo.cbSize = (uint)Marshal.SizeOf(sipInfo);
	if (NativeMethods.SipGetInfo(ref sipInfo))
	{
		sipInfo.rcSipRect.left = x;
		sipInfo.rcSipRect.top = y;
 
		NativeMethods.SipSetInfo(ref sipInfo);
	}
}

The above method simply takes an instance of a Control on the form and shows the soft input panel right underneath it. The input panel is shown by setting Enabled to true; it’s the SipSetInfo() call that actually positions it. Since our screen is super tiny, I only worry about the y coordinate; no sense in positioning in the horizontal direction because otherwise I’d crop off part of the input panel from view. The NativeMethods class is where I dump all of my P/Invoke declarations:

/// <summary>
/// This structure contains information about the current state of the 
/// software-based input panel, such as the software-based input panel 
/// size, screen location, docked status, and visibility status.
/// </summary>
[StructLayout(LayoutKind.Sequential)]
public struct SIPINFO
{
	/// <summary>
	/// Size, in bytes, of the SIPINFO structure. This member must be 
	/// filled in by the application with the size of operator. Because
	/// the system can check the size of the structure to determine 
	/// the operating system version number, this member allows for
	/// future enhancements to the SIPINFO structure while maintaining 
	/// backward compatibility.
	/// </summary>
	public uint cbSize;
 
	/// <summary>
	/// Specifies flags representing state information of the 
	/// software-based input panel. The following table shows the
	/// possible bit flags. These flags can be used in combination. 
	/// </summary>
	public uint fdwFlags;
 
	/// <summary>
	/// Rectangle, in screen coordinates, that represents the area of 
	/// the desktop not obscured by the software-based input panel. 
	/// If the software-based input panel is floating, this rectangle 
	/// is equivalent to the working area. Full-screen applications 
	/// that respond to software-based input panel size changes can 
	/// set their window rectangle to this rectangle. If the
	/// software-based input panel is docked but does not occupy 
	/// an entire edge, then this rectangle represents the largest 
	/// rectangle not obscured by the software-based input panel. 
	/// If an application wants to use the screen space around the
	/// software-based input panel, it needs to reference rcSipRect.
	/// </summary>
	public RECT rcVisibleDesktop;
 
	/// <summary>
	/// Rectangle, in screen coordinates of the window rectangle and 
	/// not the client area, the represents the size and location of 
	/// the software-based input panel. An application does not
	/// generally use this information unless it needs to wrap
	/// around a floating or a docked software-based input panel 
	/// that does not occupy an entire edge.
	/// </summary>
	public RECT rcSipRect;
 
	/// <summary>
	/// Specifies the size of the data pointed to by the pvImData member.
	/// </summary>
	public uint dwImDataSize;
 
	/// <summary>
	/// Void pointer to IM-defined data. The IM calls the
	/// IInputMethod::GetImData and IInputMethod::SetImData methods to
	/// send and receive information from this structure.
	/// </summary>
	public IntPtr pvImData;
}
 
/// <summary>
/// This structure defines the coordinates of the upper-left and
/// lower-right corners of a rectangle. 
/// </summary>
[StructLayout(LayoutKind.Sequential)]
public struct RECT
{
	/// <summary>
	/// Specifies the x-coordinate of the upper-left corner of the rectangle. 
	/// </summary>
	public int left;
 
	/// <summary>
	/// Specifies the y-coordinate of the upper-left corner of the rectangle. 
	/// </summary>
	public int top;
 
	/// <summary>
	/// Specifies the x-coordinate of the lower-right corner of the rectangle. 
	/// </summary>
	public int right;
 
	/// <summary>
	/// Specifies the y-coordinate of the lower-right corner of the rectangle. 
	/// </summary>
	public int bottom;
}
 
/// <summary>
/// This function receives information including the state of the
/// software-based input panel, the area of the desktop that is not 
/// obscured by the software-based input panel, the screen coordinates
/// of the software-based input panel, and information about the input 
/// method (IM) that the software-based input panel is currently using.
/// </summary>
/// <param name="sipInfo">[out] Pointer to the SIPINFO structure that
/// contains information about the current software-based input panel.</param>
/// <returns>TRUE indicates success. FALSE indicates failure. To get 
/// extended error information, call GetLastError. </returns>
[DllImport("coredll.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SipGetInfo(
	ref SIPINFO sipInfo);
 
/// <summary>
/// This function sets information including the state of the 
/// software-based input panel, the area of the desktop that is not 
/// obscured by the software-based input panel, the screen coordinates 
/// of the software-based input panel, and application-defined information
/// about the input method (IM) that the software-based input panel is
/// currently using.
/// </summary>
/// <param name="sipInfo">ointer to the SIPINFO structure that contains
/// information about the current software-based input panel.</param>
/// <returns>TRUE indicates success. FALSE indicates failure. 
/// To get extended error information, call GetLastError. </returns>
[DllImport("coredll.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SipSetInfo(
	ref SIPINFO sipInfo);

Yes, they really did call it “coredll.dll”.

You can imagine how calling these methods is now pretty simple. Assume that these methods are attached to the GotFocus and LostFocus events of a TextBox control:

/// <summary>
/// Fired when the user name textbox gets focus.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="e">The <see cref="System.EventArgs"/> instance 
/// containing the event data.</param>
private void WhenUserNameGotFocus(object sender, EventArgs e)
{
	this.ShowInputPanel(this.userNameTextBox);
}
 
/// <summary>
/// Fired when the user name textbox loses focus.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="e">The <see cref="System.EventArgs"/> instance 
/// containing the event data.</param>
private void WhenUserNameLostFocus(object sender, EventArgs e)
{
	this.HideInputPanel();
}

I’ve only tested this on our Windows CE 5.0-based device, but I imagine that it’d work on any later platform that was supported by the InputPanel control. Good luck!

Consider the following:

string hql;
 
hql = @"
	from Item as i
	inner join i.Barcodes as b     
	inner join i.ItemGroup.Merchant as m   
	where b.Text = :barcode
	and m.Code = :merchantCode
";
 
query = session.CreateQuery(hql)
	.SetString("barcode", barcode)
	.SetString("merchantCode", merchantCode);
 
var items = query.List<Item>(); // explodes

Let’s assume that Barcodes is a composite-element of the Item.

This will explode with an exception that reads “Could not execute query[SQL: SQL not available]” while the inner exception reads “The value “System.Object[]” is not of type “Skiviez.Armadillo.Model.Item” and cannot be used in this generic collection. Parameter name: value”.

That is because I am calling List<Item>(), but as specified, my HQL query is returning a list of Items and Merchants.

To fix it, my HQL needs to be

string hql;
 
hql = @"
	select
		i
	from Item as i
	inner join i.Barcodes as b     
	inner join i.ItemGroup.Merchant as m   
	where b.Text = :barcode
	and m.Code = :merchantCode
";

The error message was a little vague so it took me a while to figure that out. Essentially, it means that the SQL query was probably executed fine, but NHibernate is blowing up while populating the list of result objects.

AVS Information In Magento

We’ve recently set up a new e-commerce store at Men’s Underwear Discounters that represents a division of the company that I work for, Skiviez. We decided to save time for this discount store by using Magento, the current darling child in the open source e-commerce platform market segment.

A brief interlude on consulting, documentation, and open source

Now, to be clear, it doesn’t take much to exceed expectations in the “open source e-commerce platform” market because prior to Magento, your options were pretty much limited to osCommerce, which is quite possibly the most reckless, idiotic excuse of an e-commerce platform that I have seen (and I don’t like to insult code), and its slightly-less-retarded sibling Zen Cart. (I will save the osCommerce rants for another post.)

Magento seems to be a dream come true: thousands of man-hours of development into an e-commerce platform for free. So what’s the catch? While Varien (“the Magento company”) has done a great service to the world and the open source community by creating Magento, they’re still in it to make a buck–and that’s not a bad thing–but you might not realize how they make those bucks until you’ve started playing around with your own Magento store. They make money on support, consulting, and customization because Magento is huge and poorly documented.

Actually, “poorly documented” would imply that there was at least some documentation, and while there truly is some documentation available in the source code and on Varien’s Magento Web site, in reality, that amount of documentation is a statistical rounding error. And when you actually view your Magento installation in an FTP program, you will be absolutely astounded at the number of directories and files. You can try inspecting the source files to try to put it all together, but a lot of it is “magic”: a framework erected at runtime by XML configuration files and PHP’s idiotic __call mechanism for magic functions that appear at runtime out of thin air. You can see references to a function called getOrder(), for example, grep the entire installation for “getOrder()”, and not get any results because the function doesn’t actually exist at compile time. Personally, I’ve always considered magic methods to be an extremely unwise idea, a “too clever” approach to programming. But for people new to PHP looking to do a little customization to their Magento store, it makes the codebase all the more impenetrable. That’s more consulting dollars.

You could go to the Magento forums, but you won’t find too many helpful answers because Magento obviously doesn’t want to give help for free, since giving help is central to their business model. So if you need help, your best bet is to generally (a) pay for it or (b) hope that you stumble across somebody’s blog who took the time to post a solution to your exact problem.

Why do I need to do this, anyway?

Magento has a lot of stuff built in. But we need to remember that Magento was built by software developers, and software developers generally don’t understand the problem domain of running an e-commerce business very well. Sure, every software developer has built the pet shop–products, categories, inventory, orders, how hard could it be?–but they stop or fail to understand that there’s a whole class of operations that need to happen after the order has been placed.

One of those operations is fraud analysis. The Address Verification System (AVS) and the Card Verification Number (CVN) system for credit cards enable merchants to help determine if a credit card has been fraudulently used on an order. AVS will tell the merchant whether or not the billing address specified by the customer matches or partially matches the billing address on the credit card’s billing statement. And the CVN number represents the number that (supposedly) exists in only two places in the world: on the physical credit card and in the issuing bank’s files. If a CVN doesn’t match and the AVS doesn’t match, then there’s a good bet that there is something wrong with the order.

Because software developers are nerds, this information doesn’t currently display by default in Magento’s administration section. So let’s add it.

Can’t I just edit the template?

I’m talking about adding this AVS display functionality in the context of the CyberSource extension for Magento. After spelunking through the myriad files and directories, I discovered that the Varien-provided CyberSource extension is saving the AVS (setCcAvsStatus()) and CVN (setCcCidStatus()) information when it authorizes the card, but it doesn’t display in the information block in the administration section.

While you could just modify app/design/adminhtml/default/default/template/cybersource/info.phtml directly (yes, that is the actual path to the file that displays the little “Payment” block), your changes would be lost and overwritten in the future if you ever upgraded the CyberSource extension via Magento Connect. That’s obviously less than ideal.

A better way would be to use Magento’s “rewrite” mechanism. We’ll create our own Magento module and instruct Magento to replace the CyberSource extension’s information block with our own.

Creating our own module

First, we’ll create a new empty directory to hold our files. Let’s call it avsdisplay. Then we’ll create code, etc, and design subdirectories. These directories mirror the subdirectories in the app folder of your Magento installation.

In the etc directory, we need to create an XML configuration file. Magento scans this directory at runtime for XML files and reads them to know which modules to load. I’ve named my configuration file Mud_Avsdisplay.xml, and the naming is very important. (Magento uses a mixture of convention and configuration.) The part of the filename before the underscore is your “namespace,” since PHP doesn’t support namespaces. And the part of the filename after the underscore is the name of your module. Mine is “Avsdisplay.” Don’t try to use something like “AvsDisplay,” however, because that would break Magento’s convention as we’ll see in a moment.

Inside this file, we tell Magento to load our module in the local pool and to actually run it (by setting “active” to “true”). (The other pools are core, for those modules provided by Varien, and community, for those modules that you can download via Magento Connect.)

<?xml version="1.0"?>
<config>
	<modules>
		<Mud_Avsdisplay>
			<active>true</active>
			<codePool>local</codePool>
		</Mud_Avsdisplay>
	</modules>
</config>

Now it’s time to create the meat of the module. In the code folder, create a local directory. Within that, create a directory named after your chosen namespace (Mud in my example) and within that yet another subdirectory named after your module (Avsdisplay in my example). (This is why naming was important. If I had called it “Mud_AvsDisplay,” with a capital ‘D’, then Magento would have looked in app/code/local/Mud/Avs/Display for my module instead of app/code/local/Mud/Avsdisplay.)

Within your module directory, you’ll need to create three more subdirectories: Block, etc, and Helper. When it’s all said and done, you’ll end up with a hierarchy that looks like this (ignore the “design” directory, we’ll get to that later):

Directory Structure

Creating the block

Next, we need to create the “block” that we’ll be replacing. After searching through the codebase, I figured out that I wanted to replace the Mage_Cybersource_Block_Info block (for reference, by Magento’s convention, that file lives in app/core/Mage/Cybersource/Block/Info.php — see the convention?). But I really only want to add stuff to it, not replace it outright, so I’ll create a new file in my Block folder called Info.php that extends that class. Something like:

<?php
 
class Mud_AvsDisplay_Block_Info extends Mage_Cybersource_Block_Info
{
	// Yes, these really have to be protected, not private, due to more Magento
	// "magic"
	protected $avsDescriptions = array(
		// Snipped for brevity; download file at end of blog post
	);
 
	protected $ccCidDescriptions = array(
		// Snipped for brevity; download file at end of blog post
	);
 
	protected function _construct()
	{
		parent::_construct();
		$this->setTemplate('avsdisplay/info.phtml');
	}
 
	public function getAvsStatusDescription()
	{
		$info = $this->getInfo();
		$avsCode = $info->getCcAvsStatus();
		$avsDescription = 'Unrecognized response code.';
 
		if (array_key_exists($avsCode, $this->avsDescriptions))
		{
			$avsDescription = $this->avsDescriptions[$avsCode];
		}
 
		return $avsDescription;
	}
 
	public function getCcCidDescription()
	{
		$info = $this->getInfo();
		$ccCidCode = $info->getCcCidStatus();
		$ccCidDescription = 'Unrecognized response code.';
 
		if (array_key_exists($ccCidCode, $this->ccCidDescriptions))
		{
			$ccCidDescription = $this->ccCidDescriptions[$ccCidCode];
		}
 
		return $ccCidDescription;
	}
}

Okay, this makes sense. I’m doing everything the old CyberSource info block did, except I added two new functions that give me human-friendly descriptions of the AVS and CVN codes, and I’m setting the template to something called avsdisplay/info.phtml instead of cybersouce/info.phtml. How this comes into play will make sense in a minute.

Creating the helper

In my Helper directory, I need to add a Data.php file that looks like this:

<?php
 
class Mud_Avsdisplay_Helper_Data extends Mage_Core_Helper_Abstract
{
}

That’s right, it does nothing except inherit from a Magento-provided abstract class. What is a helper? I have no idea, but you have to do it–Magento looks for this file.

Telling Magento to do the swap

In my etc directory, I need to add a config.xml file that will tell Magento some information about my module and, most importantly, that I want to be swapped with the CyberSource extension’s info block:

<?xml version="1.0"?>
<config>
	<modules>
		<Mud_Avsdisplay>
			<version>0.0.1</version>
		</Mud_Avsdisplay>
	</modules>
	<global>
		<helpers>
			<avsdisplay>
				<class>Mud_Avsdisplay_Helper_Data</class>
			</avsdisplay>
		</helpers>
		<blocks>
			<cybersource>
				<rewrite>
					<info>Mud_Avsdisplay_Block_Info</info>
				</rewrite>
			</cybersource>
		</blocks>
	</global>
</config>

Under <modules>, I’m just telling Magento the version number of my module. Under <global>, I say that I want to “rewrite” the CyberSource info block with my own class named Mud_Avsdisplay_Block_Info, which, by convention, Magento will look for in [module-root]/Mud/Avsdisplay/Block/Info.php. That is the file that I just created. Great.

Finally, writing the template that displays the information

But we haven’t actually changed any HTML at this point! Remember in my Info.php file where I set the template to something called info.phtml? Well, I need to create that file. At the top of my module folder–the one with the code and etc subdirectories–I’ll need to create some more subdirectories. A lot of them. In fact, I’ll need to create a bunch of empty directories that look like this:

In the bottom-most subdirectory, I’ll create the info.phtml file. This is a PHP file, except that when I use the $this pseudo-variable in this file, $this will be pointing to an instance of my Mud_Avsdisplay_Block_Info class. So I’ll be able to call my getAvsStatusDescription() method here among other things:

<?php
	if ($info = $this->getInfo())
	{
		?>
			<strong><?php echo $this->getMethod()->getTitle(); ?></strong><br />
			<strong>Type:</strong> <?php echo $this->__($this->getCcTypeName()); ?><br />
			<strong>Number:</strong> xxxx-<?php echo $this->__($info->getCcLast4()); ?><br />
			<strong>Expiry:</strong> <?php echo $this->__('%s/%s', $this->getCcExpMonth(), $info->getCcExpYear()); ?><br />
			<strong>AVS:</strong> (<?php echo $this->__($info->getCcAvsStatus()); ?>) <?php echo $this->__($this->getAvsStatusDescription()); ?><br />
			<strong>CCID:</strong> (<?php echo $this->__($info->getCcCidStatus()); ?>) <?php echo $this->__($this->getCcCidDescription()); ?><br />
		<?php
	}
?>

Conclusions and delusions

And that’s it. Upload this entire directory structure to your Magento installation’s app directory, merging the contents as necessary, and you should be able to see AVS information for your CyberSource orders in the administration section.

How did I figure out how to learn all of this? It was mostly by lots of Googling, picking apart existing modules to see how they were organized, and piecing together lots of blog posts. I hope this helps someone out there. Good luck!

Download the code used in this article.