Simply Does Not Work

Our latest shipping integration at work is near nirvana: you place a package on a scale, scan the barcode (the shipping program itself doesn’t even need to be focused!), choose the box type, and press enter. Then a label comes shooting out of the label printer, and maybe some customs forms documents, too.

Behind that magic <ENTER> press, though, a lot of magic is happening behind the scenes:

• I call our payment processing service to cleanse the address into a format suitable for mailing.
• I call the FedEx Web service to determine if the address is residential or commercial.
• If standard shipping, I call the USPS XML API, FedEx SOAP API, and UPS XML APIs simultaneously to find the cheapest rate and best transit time.
• I call the FedEx or UPS API directly to print a shipping label immediately OR invoke a Stamps.com COM object to pop up an extra dialog, which the user then presses enter to spit out the label interminably slowly.

It’s the last part that has been the bane of my existence. USPS does not offer any online mechanism to print postage-paid shipping labels in an automated fashion. You get to choose between Endicia and Stamps.com. While Endicia has an XML API that functions nearly identically to those offered by UPS and FedEx, it’s ridiculously expensive, and Stamps.com has no such API. Instead, I have to interact with a COM object that is popping up dialogs and forcing the warehouse worker to worry about buying postage when it runs low.

To add insult to injury, Stamps.com doesn’t understand EPL2 and instead sends its labels as gigantic GDI documents to the label printer, giving us print times of nearly 15 seconds for a 4″ x 6″ label when going over parallel. Compare to less than 2 seconds for a label sent as an EPL2 document. But Stamps.com is cheap.

And there are no alternatives.

Sure, FedEx has SmartPost and FIMS and UPS has Mail Innovations. These are services where we would put our non-postage paid parcels into a trash bag that FedEx or UPS would pick up, sort, route through their system to a post office near the parcel’s final destination, and affix postage there and let it enter the USPS system by mailing it for us at the “last mile post office.” But Mail Innovations is worthless (a statistically significant number of our packages shipped through Mail Innovations to Canada were tampered with), FIMS is mind-boggingly slow (up to 60 days to send a package to Finland, and FedEx reps seem to have no idea that it is actually taking that long for their service to actually work), and SmartPost is expensive unless you’re talking enterprise-level volumes, which I am not. Plus, if a package disappears, there’s plenty of finger pointing, trust me.

And sure, there’s Click-N-Ship at the USPS’s Web site. But Click-N-Ship provides zero integration capabilities and as such is useless when you’re talking more than, say, 3 packages per day. It prints lots of instructions that you just end up throwing away, uses PDF for label output (so you can’t really automate printing unless you want to rig up a harebrained routing-through-Ghostscript scheme), and is really intended for home users.

There’s permit imprint, which would work if we were mailing 200 identically weighted parcel post packages, but that is never going to happen. Some widgets are bigger than others.

And sure, there’s that moronic new commercial where they show the USPS employees touting the simplicity of the new priority mail flat-rate boxes for businesses. You still have to stamp them–that’s half my problem. And using a flat rate box means that you’re throwing money away a good 50% of the time. We’re not stupid. It does not cost $9.85 to send a 12″ x 9″ x 6″ box from Virginia to North Carolina. And where’s my delivery confirmation barcode? I’ll have to affix that too and record the number–how is this saving me any time? I want to throw my remote at the TV every time this commercial comes on.

So, I don’t understand why the post office doesn’t offer a postage Web service like UPS’s or FedEx’s XML shipping services, unless it’s just too much competition with their own licensees. The Visa business model does not make sense for everyone.

I don’t understand why the post office doesn’t sell laser printer versions of its CN-22 and CP-72 customs forms. We had to have our own specially made, and use the USPS’s API to generate the actual form. (More on this later.)

I don’t understand why the USPS has an address verification API available through their Web tools but doesn’t let anybody use it. (By anybody, I mean that only non-profits can use it. Not even local governments can use it.) Shouldn’t it be an obvious service to help customers verify that they’re shipping to a valid address? Well, perhaps I do understand, because there’s lots of money to be made selling CASS certification to vendors who then can charge ridiculously high fees for what is, at its core, just a giant database table that gets updated once a month. By giant, I mean it fits on a CD-ROM. CASS is beyond stupid.

I don’t understand why the USPS doesn’t get its act together and offer business accounts for its Priority mail and higher services. With the USPS, indicia is like currency: a misprinted label means you lose money, whereas with FedEx and UPS, a misprinted label means you throw it in the trash. But far, far more importantly, this means that businesses can’t leverage the “float” when using USPS: the shipping cost is incurred immediately instead of on net 30 terms.

I don’t understand why the USPS doesn’t allow scheduling a daily pickup for businesses. Sure, there’s the insane carrier request API, which we use, and have to remember to update every few months until we max out the pickup requests again. But it’s particularly insane that it requests the number of packages that will be picked up and that mail carriers treat this as some sort of contractual obligation. I am setting a date in the future–I can give you an estimated average, but please don’t complain when I have 90 packages instead of 50. I can’t see the future to know how many orders we will have to process that morning.

I don’t understand why the ICCC is staffed by weasels. When an API is not on your test server but you require us to test against the requested API on the test server before getting access to the production server, then something is really wrong, and someone at the USPS needs to be fired. Right now.

If anyone at the post office is listening, one of the greatest barriers to using the post office directly is a lack of integration points. Quit moaning about the number of parcels dropping and make it easier to actually ship packages, and maybe we’ll be more apt to use it.

Update 11/11/2009: See my comment below for more detailed information since this post was written.

Well, that’s a bit of an inflammatory title. But this screenshot speaks volumes:

I can't log in!

What has happened here is that my OpenID provider, Verisign Labs, is down for some reason while StackOverflow, the site that I’m trying to log on to, is still up.

A single point of failure

While this isn’t a big deal for a site that I use recreationally, it is kind of a big deal in principle: all sites that use OpenID are inaccessible to me, since I only have this single OpenID provider. It is a single point of failure.

Sure, I could have multiple OpenID accounts, such as a “backup” OpenID account at another provider, but this defeats the purpose of OpenID, of having a central point of authority regarding my online identity. If I have to create a second OpenID provider, I might as well just make a native username and password at the site that I’m trying to log into. I could also be my own OpenID provider, but I shouldn’t have to do that, and the average OpenID consumer is not going to know how to do that, anyway. It’s faster just to create a “native account” at the Web site that I’m trying to access and be done with it.

An unnecessary third-party dependency

For any e-commerce business, it’s usually unwise to take on any third-party dependency. Speaking from experience, that third-party will go down for some reason at some point and you will be the one fielding angry customer support calls for something that isn’t your problem. Even worse, if something as critical as authentication through OpenID goes down, then you’re losing sales, and that means that you’re not doing your job.

If you’re going to add OpenID to an e-commerce Web site or a “mission-critical” Web site, then you should implement it as a red-headed step-child, “convenience only” implementation; this way, your users can fall back onto their “native credentials” (those stored in your database) when their preferred OpenID provider explodes. In this way, OpenID is a shortcut, not a single authoritative source, much how like customers use PayPal’s Express Checkout to save time when entering payment and addressing information during checkout. If PayPal’s Web Service API goes down (and believe you me, it’s gone down several times this year alone), customers can still begrudgingly use the “native checkout” to complete their purchase. Some lost conversions due to the inconvenience–and for incorrectly faulting the integrity of your Web site for the third party’s failure–but at least the sale is still actually possible. If OpenID is your only authentication mechanism, and a user’s OpenID provider goes down, then that user is simply screwed. The onus should not be on the consumer to provide a backup authentication mechanism to your Web site.

It’s just complicated

OpenID is still just too complicated. I’m not the smartest software developer in the world by any means, but if I can barely wrap my mind around it enough just to get my first account, then I am 100% certain that our customers would have problems with it:

• Why do I have to go to some third-party provider to sign in? That doesn’t make any sense.
• Why should I trust that third-party provider? I’m not even sure that I trust you!
• I still don’t know which third-party provider to use! Oh well, I guess I’ll use Verisign, at least I’ve heard of them.
• What happens to my credentials when that third-party provider goes under?
• Can I call your business to reset my password when I forget it? I will anyway!

Even with our “native registration” of e-mail addresses and passwords, we had to use an Amazon.com style login form because users kept filling out the “new customer” form even when they already had an account:

Login screen. Amazon’s sign-in screen remains a model to be emulated, minimizing the common problem of new customers who try to log in without having registered. Amazon presents two questions in linear order: (1) “What is your email address?” and (2) “Do you have an Amazon.com password?” For the second question, users can select one of two radio buttons: “No, I am a new customer,” or “Yes, I have a password.” Many other sites present the new- and established-user sections side-by-side, and thereby divert new users to the established-user section through the magnetic attraction of type-in fields. — Jakob Niesen, useit.com

When our users have trouble navigating two fields and a radio button, you can imagine the hilarity that ensues when they are presented with multiple authentication mechanisms. Especially one that doesn’t ask for a password.

A leaky abstraction

While the idea of consolidating your online identity to a single source sounds like a good idea, it simply does not work in practice. Not even the United States government has a consistent picture of my identity: my identity is stored with the IRS, the Social Security Administration, the United States Postal service, the Virginia Department of Taxation, the Virginia Department of Motor Vehicles, and the City of Richmond’s Voter Registrar. These are all independent identity stores that, in general, agree that I am who I say that I am, but each identity is stored and updated independently of one another. Consider OpenID the RealID act of the Internet: what happens when your centralized identity gets screwed up? It’s an existentialist crisis and you just have to deal with it: identity is what other people say you are; it’s not what you say you are. Better to have multiple voices confirming your identity than just one.

Conclusions and Delusions

If you’re implementing a social-oriented Web site that is designed to integrate with Facebook or some of the Web-2.0-savvy consumers, then OpenID might make sense. Those nerds may have heard of OpenID and know how to use it, foibles and all. Until the dust settles on OpenID, though, I wouldn’t add it to a commercial site even if you paid for it: nobody has asked for it; it will cause problems.

My general recommendation would be to have the usual in house username and password mechanism that can be supplemented by OpenID. But realize that each alternative identification mechanism that you add runs the risk of customer confusion, bugs, security holes, and increased customer support. When in doubt, leave it out.

Well, this one was new to me, and I spent about ten minutes chasing my tail. Consider the following code (C#):

if (buffer != null)
{
	using (MemoryStream ms = new MemoryStream(buffer))
	using (GZipStream gs = new GZipStream(ms, CompressionMode.Decompress))
	{
		try
		{
			userProfile = new BinaryFormatter().Deserialize(gs) as UserProfile;
		}
		catch (SerializationException)
		{
			this.Clear();
		}
		catch (InvalidDataException)
		{
			this.Clear();
		}
	}
}

Looks pretty sane and cautious, right? This is part of one of my HTTP cookie classes, and it just takes a previously serialized object and attempts to deserialize it. If for some reason deserialization doesn’t work, big whoop, just clear the cookie and make the user log in again, because that probably means I just updated the software on the server anyway.

However, one of the constructors for the UserProfile class looks like this:

/// <summary>
/// Initializes a new instance of the UserProfile class.
/// </summary>
/// <param name="info">The <see cref="T:System.Runtime.Serialization.SerializationInfo"/>
/// to populate with data.</param>
/// <param name="context">The destination
/// <see cref="T:System.Runtime.Serialization.StreamingContext"/>) for this serialization.</param>
protected UserProfile(SerializationInfo info, StreamingContext context)
{
	string mailAddress;
	string merchantCode;
 
	this.Identity = info.GetValue("Identity", typeof(ArmadilloIdentity)) as ArmadilloIdentity;
	this.Culture = new CultureInfo(info.GetInt32("Culture"));
	this.Currency = Currency.GetCurrency(info.GetString("Currency"));
	this.SetTimeZone(info.GetString("TimeZone"));
 
	mailAddress = info.GetString("MailAddress");
	if (!string.IsNullOrEmpty(mailAddress))
	{
		this.SetMailAddress(mailAddress);
	}
 
	merchantCode = info.GetString("MerchantCode");
	if (!string.IsNullOrEmpty(merchantCode))
	{
		this.merchant = new MerchantLookup(merchantCode);
	}
 
	this.revision = (byte[])info.GetValue("Revision", typeof(byte[]));
}

This type of constructor allows me to work around the fact that some of the types that the UserProfile class is using aren’t marked as serializable, so I come up with my own way of serializing and deserializing them.

So what happens in my first code snippet when serialization fails? Well, the SerializationException doesn’t get caught. Why the hell not?

Well, the BinaryFormatter is obviously using reflection to new up an instance of my UserProfile. Typically, this is a non-issue because the constructor does not do anything interesting. But here, it does. So when my constructor bombs on a GetValue() call because I’ve changed the object since it was last deserialized, it tosses a SerializationException as we expect–but since the BinaryFormatter was using reflection to new up the object, that exception gets wrapped in a TargetInvocationException.

In other words, to be able to catch the exception, I need to

try
{
	userProfile = new BinaryFormatter().Deserialize(gs) as UserProfile;
}
catch (TargetInvocationException)
{
	this.Clear();
}
catch (SerializationException)
{
	this.Clear();
}

It’s the TargetInvocationException that gets caught. (I left the SerializationException in there because I’m paranoid.)

So metafiles in the GDI world of Windows are a vector format that is essentially a sequence of GDI drawing commands (we know it as System.Drawing.Graphics in the .NET world) serialized to disk. Because it’s a vector format, we can play back the metafile to different device contexts, all with different DPIs, and get more or less the same high-fidelity result on each device. But working with an in-memory metafile in C# isn’t as intuitive as one might think.

A problem that you will encounter is that while the metafile calculates its DPI according to the resolution of the reference device, the .NET Graphics instance that you get to draw on seems to always be set to 96 DPI. That simply will not work: the result, in fact, is a lot of cussing when your carefully measured measurements simply don’t line up when you play back the metafile later. Here’s a link to a guy who had the same exact problem. It gets really confusing because the amount of error will vary depending on the machine you run it on, since the reference device is the physical resolution (i.e., the desktop you’re looking at).

The secret is to apply a scale transform to your Graphics instance to fix the DPI mismatch.

Here’s some sample code:

/// <summary>
/// Adds a new metafile page, selects it as the current metafile, and
/// returns its graphics object. The caller is responsible for disposing
/// the returned Graphics instance. The unit of measurement in the
/// returned Graphics instance is GraphicsUnit.Inch.
/// </summary>
/// <param name="pageSize">The size of the page, in inches.</param>
/// <returns>a Graphics instance to render the page onto; the caller is
/// responsible for disposing of this Graphics instance</returns>
protected Graphics GetNextPage(SizeF pageSize)
{
    IntPtr deviceContextHandle;
    Graphics offScreenBufferGraphics;
    Graphics metafileGraphics;
    MetafileHeader metafileHeader;
    KeyValuePair<Metafile, MemoryStream> pair;
 
    this.currentStream = new MemoryStream();
    using (offScreenBufferGraphics = Graphics.FromHwndInternal(IntPtr.Zero))
    {
        deviceContextHandle = offScreenBufferGraphics.GetHdc();
        this.currentMetafile = new Metafile(
            this.currentStream,
            deviceContextHandle,
            new RectangleF(0, 0, pageSize.Width, pageSize.Height),
            MetafileFrameUnit.Inch,
            EmfType.EmfOnly);
 
        pair = new KeyValuePair<Metafile, MemoryStream>(
            this.currentMetafile,
            this.currentStream);
        this.metafiles.Add(pair);
 
        offScreenBufferGraphics.ReleaseHdc();
 
        metafileGraphics = Graphics.FromImage(this.currentMetafile);
        metafileHeader = this.currentMetafile.GetMetafileHeader();
        metafileGraphics.ScaleTransform(
            metafileHeader.DpiX / metafileGraphics.DpiX,
            metafileHeader.DpiY / metafileGraphics.DpiY);
 
        metafileGraphics.PageUnit = GraphicsUnit.Inch;
        metafileGraphics.SetClip(new RectangleF(0, 0, pageSize.Width, pageSize.Height));
    }
 
    return metafileGraphics;
}

Most of this is mumbo-jumbo related to my particular PrintDocumentBase implementation, but the real gem is the ScaleTransform() call: that’s where I account for the mismatch between the resolution of the metafile itself and Graphics instance that I’ll be doodling on. (My implementation also expects the page size to set so additional clipping information can be set, but that’s not important to this discussion.)

I can now doodle on the returned graphics object with abandon, resting assured that when I say “draw a line that is one inch long” I will get a line in my metafile that is one inch long. (My instance is storing the Metafile and MemoryStream pair in a dictionary. When I want to save the metafile to disk, I just have to call ToArray() on the MemoryStream.)

Bonus: Printing a serialized Metafile

Now let’s say that I have a byte[] array that corresponds to a serialized Metafile. How do I actually print that to my printer? We’ll have to do some P/Invokes. I have my own class called PrinterDocumentBase that basically returns a byte[] array on each call to GetPage(int pageNo). I assume that what I get back from that call in the function below is a serialized Metafile. (I have other PrinterBase inheritors that assume that byte[] array represents something different, like EPL2 data or ZPL data, and sends it to the printer appropriately.)

/// <summary>
/// Prints the given document.
/// </summary>
/// <param name="document">the document, which is guaranteed not to be
/// null in and in a language that the printer supports</param>
/// <param name="copies">the number of copies, which is guaranteed to be
/// a positive number</param>
protected override void DoPrint(PrinterDocumentBase document, int copies)
{
    IntPtr deviceContext;
    NativeMethods.DOCINFO documentInfo;
    int hardMarginLeft;
    int hardMarginTop;
    int pagesCount;
 
    documentInfo = new NativeMethods.DOCINFO();
    documentInfo.fwType = 0;
    documentInfo.lpszDatatype = null;
    documentInfo.lpszDocName = document.Name;
    documentInfo.lpszOutput = null;
    documentInfo.cbSize = Marshal.SizeOf(documentInfo);
 
    pagesCount = document.Prepare();
 
    deviceContext = NativeMethods.CreateDC(
        "WINSPOOL",
        this.Name.Normalize(),
        null,
        IntPtr.Zero);
 
    if (deviceContext != IntPtr.Zero)
    {
        hardMarginLeft = NativeMethods.GetDeviceCaps(
            deviceContext, 
            NativeMethods.DeviceCap.PHYSICALOFFSETX);
        hardMarginTop = NativeMethods.GetDeviceCaps(
            deviceContext, 
            NativeMethods.DeviceCap.PHYSICALOFFSETY);
 
        hardMarginLeft = (int)(hardMarginLeft * 100F / NativeMethods.GetDeviceCaps(deviceContext, NativeMethods.DeviceCap.LOGPIXELSX));
        hardMarginTop = (int)(hardMarginTop * 100F / NativeMethods.GetDeviceCaps(deviceContext, NativeMethods.DeviceCap.LOGPIXELSY));
 
        for (int copyIdx = 0; copyIdx < copies; ++copyIdx)
        {
            if (NativeMethods.StartDoc(deviceContext, documentInfo) > 0)
            {
                for (int i = 0; i < pagesCount; ++i)
                {
                    byte[] data;
 
                    data = document.GetPage(i);
 
                    using (var ms = new MemoryStream(data))
                    using (var metafile = Metafile.FromStream(ms))
                    {
                        using (var g = Graphics.FromHdcInternal(deviceContext))
                        {
                            g.TranslateTransform(-hardMarginLeft, -hardMarginTop);
 
                            if (NativeMethods.StartPage(deviceContext) > 0)
                            {
                                g.DrawImage(metafile, 0, 0);
                                g.Flush();
 
                                if (NativeMethods.EndPage(deviceContext) <= 0)
                                {
                                    throw new Win32Exception();
                                }
                            }
                            else
                            {
                                throw new Win32Exception();
                            }
                        }
                    }
                }
 
                if (NativeMethods.EndDoc(deviceContext) <= 0)
                {
                    throw new Win32Exception();
                }
            }
        }
    }
    else
    {
        throw new Win32Exception();
    }
 
    if (deviceContext != IntPtr.Zero)
    {
        NativeMethods.DeleteDC(deviceContext);
    }
}

The StartDoc and such functions are documented in MSDN and are pretty boring for this discussion.

Because when I specify I want text to print at a certain measurement in my metafile, I do have to do some dancing to shift back the hard margins of the printer–I’ll assume that I was smart enough to account for this when I was doodling on my metafile.

Keep in mind that when you do set GraphicsUnit.Inch as I have, your pen widths will be in whole inches, which can result in some pretty wide strokes! Took me a while to figure that one out, too, though it was blindingly obvious.

Hope that helps someone get on the right path!

So this week concluded several sleepless nights and much heartburn as I migrated Skiviez’s SBS 2003 machine (running as our domain controller and our mail server) to SBS 2008. As far as things go, it went relatively smoothly, and the remainder of the week was dealing with lots of small niceties that I had forgotten that I had set up on the 2003 server that I now needed to set up once again.

One of these was something that I used for my convenience over a VPN connection from home. You see, the internal order processing application that I wrote uses some shared folders to store some temporary data, such as e-mails that are generated but not yet released to Exchange, or a local copy of images that are available on the Web site. This software–and our users–are used to referring to Windows file shares as \\COMPUTER-NAME\SHARE-NAME; for example, \\CYRUS\Pickup Holding, because for some reason some of the older servers are named after my boss’s dead cats.

When connecting through VPN to SBS 2008, however, that “suffix-less” name resolution was not working. So when \\CYRUS\Pickup Holding failed to resolve to anything, \\cyrus.skiviez.com\Pickup Holding would work fine. This was super annoying.

The reason this worked previously with our SBS 2003 installation is that it was acting as a WINS server, which provided this type of computer name resolution for us. SBS 2008 finally retires this ancient technology by default, however, so I had two choices: I could either install the WINS server role on SBS 2008, or I could just figure out how to get the 015 DNS Domain Name option from DHCP to relay through the VPN connection.

I chose the latter option, since it’s certainly less confusing to be able to say to someone in the future “we don’t use WINS, DNS does everything.” So here’s how to do it:

1. On the SBS 2008 server, click Start > Administrative Tools > Routing and Remote Access.
2. In the tree view, drill down past the server name to IPV4 > General. Right-click the General option and choose “New Routing Protocol” and choose DHCP Relay Agent.
3. Now right-click the newly appended “DHCP Relay Agent” node and choose Properties. Add the IP address of your DHCP server (which is probably your SBS server itself), and click OK. Then click it again and choose “New Interface” and add the “Internal” interface.
4. Now if you connect through VPN, an ipconfig /all should show your domain name as a “Connection-specific DNS suffix” and pinging machines by their suffix-less computer names should work. (If it doesn’t, make sure your DHCP server is using that 015 DNS Domain Name option, which the SBS 2008 wizards set up by default.)

Happy file sharing!

Our mobile picking application at work runs on the .NET Compact Framework and typically connects to a SQL Server database. This is all nifty and supported with a vanilla install of NETCF and is as such uninteresting. But recently I’ve been tasked with an integration scenario where an alternative persistence implementation for the mobile picking application must communicate with a MySQL 5.1 database.

MySQL offers an ADO.NET driver called Connector/Net 6.0 at its Web site. Sweet. So you reference Mysql.Data.CF in your application and you’re good to go. But as soon as you call Open() on a MysqlConnection instance, you’ll get slammed with a System.IndexOutOfRangeException deep down in the innards of the of the Connector/Net library. If you take a look at the stack trace, you’ll see that it’s occurring in MemoryStream of all places. Apparently, upon calling Open(), the driver immediately issues a command to the database to get some metadata about its capabilities, and we blow up at trying to build that parameterized query.

Why is this broken?

The MySQL Connector/Net library is using the same code for the full framework as well as the compact framework, with little ifdefs for CF-specific or CF-omitted behavior sprinkled throughout the codebase. This isn’t a bad thing; in fact, it’s quite normal, and I do it for one of my common libraries that’s shared between our internal processing application and the mobile picker. The problem stems from an actual behavior difference between the MemoryStream implementations of the full and the compact frameworks. Here’s some sample code that demonstrates the problem:

MemoryStream ms = new MemoryStream();
ms.Position = 4;
ms.WriteByte((byte)0);
Console.WriteLine(ms.Capacity);
Console.ReadLine();

What we are doing is instancing a new MemoryStream via its default constructor; its internal byte[] buffer will be empty (capacity zero). We set its internal Position to somewhere and then try to write a byte.

In the full framework, the MemoryStream auto-expands the capacity by checking to see if the Position is greater than or equal to the Length. So after the above code executes, you end up with a MemoryStream instance whose internal byte[] buffer contains 5 bytes (all zeroes).

In the compact framework, the MemoryStream also auto-expands, but it only does it if the Position equals the Length–not if the Position is greater than the Length. So the above code results in a System.IndexOutOfRangeException. It’s not a documented exception for the WriteByte() method, so I would imagine it is an oversight in the compact framework. (Although the MySQL use case is a little strange; I would never use the MemoryStream class this way.)

The work around is to download the source distribution for Connector/Net 6.0 and edit the MySqlPacket.cs file. Around line 34, change

private MemoryStream buffer = new MemoryStream();

to

private MemoryStream buffer = new MemoryStream(5);

All we’re doing is starting the internal byte[] buffer with a capacity of 5, since the MySqlPacket.cs constructor sets the Position to 4. It shouldn’t affect actual behavior of the MySqlPacket since we’re only adding capacity and not length to the MemoryStream instance that it uses.

Other unrelated problems

If you’re upgrading the Connector/Net 6.0 source to NETCF v35 (just right-click the project in Visual Studio 2008 and choose “Upgrade Project,” you’ll need to delete the CompilerGeneratedAttribute.cs file. NETCF v35 already has one, but NETCF v20 didn’t; I guess that’s why the MySQL guys had created their own.

Finally, you may be getting a “Missing Manifest Exception” whenever Connector/Net 6.0 attempts to access its Resources object for error messages. That’s because they’re trying to share the same Resources.designer.cs file for the CF assembly that was created for the desktop assembly. The problem is that in line 42 of the generated Resources.designer.cs file, it’s trying to load the embedded resx file from the assembly–but it’s searching for “MySql.Data.MySqlClient.Properties.Resources”, which is the namespace for the desktop DLL. So it doesn’t find it in the NETCF assembly on your device and tosses the missing manifest error. The solution is to either fix up the name by editing it directly (e.g., “MySql.Data.CF.Properties.Resources”) or by deleting the file and letting Visual Studio regenerate it properly on the next recompile.

Conclusions and delusions

I’ve filed two bugs, one with MySQL and one on Connect, who knows what will happen with them:

• The bug at MySQL
• The bug at Microsoft Connect

The CF version of Connector/Net seems pretty alpha, but it’s what we’ve got to work with. Good luck!

Update 7/6/2009: Download my custom version of the file.

So our internal order processing application at work securely connects to our SQL Server database via an ADO.NET connection. And if you leave the application up and running for a long time period–say, an entire weekend–you sometimes come to work on Monday morning to see an error message like the following:

Something Didn't Work

A transport-level error has occurred while sending the request to the server. (provider: TCP Provider, error: 0 – An existing connection was forcibly closed by the remote host.) (Microsoft SQL Server, Error: 10054)

(Think that’s a sexy error dialog? It’s from the SQL Server 2005 SDK. Just Google it; MS lets you use it in your own applications. Pretty cool for displaying an exception and all of the inner exceptions that it wraps.)

If you dismiss the dialog and retry the command that failed, it works fine. What’s up?

The ADO.NET connection pool is great…

Well, we’re using the connection pooling feature of ADO.NET. Usually, connection pooling is a behind-the-scenes feature that makes our database connection code run faster. Instead of setting up and tearing down the connection to the database server with each call to Open() and Close() on a SqlConnection instance, ADO.NET is instead acquiring and releasing connections to and from a pool of connections. If, while executing some command against a connection and a fault occurs, ADO.NET will automatically remove the faulting connection from the connection pool and spin up a new connection. Sounds nifty, right?

…until the network farts

Well, sometimes the connections fault in a way that ADO.NET isn’t aware of until the next time you try to use that connection. In my case, the ADO.NET connection pool is happily diddling along when, for example, our office’s craptastic Comcast SMC unholy-marriage-of-router-and-modem device hiccups and reboots itself, severing any sockets that were once open.

My code then tries to do something innocuous like call Open() on a SqlConnection, execute a command, and then Close() the connection, as it has done a million times before. ADO.NET pulls a connection from the connection pool, notes that the socket is broken, clears out the connection pool, creates a new connection, and executes my command successfully, and I and my users are none the wiser of the unimportant detail that this connection pool repair job just happened.

Ha ha, no, that’s not what happens; that would make sense. What actually happens is that ADO.NET does all of those things but not before throwing a SqlException as described above instead of executing my command. Why I would care about this error, I’m not sure–my code isn’t even aware that the connection pool exists, its calling of Open() and Close() is quite explicit in terms of the behavior that it is expecting, and ADO.NET throwing error 10054 is, in my opinion, leaking implementation details that should be hidden beneath the SqlConnection abstraction. But this is what we’ve got.

Extension methods are great

When C# 3.0 introduced extension methods, I really thought they were a silly feature that would lead to developer confusion. I was instantly reminded of my PHP days where everyone in the world had devised their own unique set of HTML escape functions, and moving to any new Web site involved learning a whole new suite of strange extensions or redefinitions of built-in functions. But, in .NET, I’ve found myself using extension functions more and more often, especially since their usage is neatly held in check through the use of namespaces. It is through some extension methods that I work around the above error in my code.

Essentially, instead of calling command.ExecuteReader() or command.ExecuteScalar() on my SqlCommand instances, I call command.SafeExecuteReader() or command.SafeExecuteScalar() instead, with the SafeExecute* functions simply being extension methods that I defined as similar to the following, which I lifted with some modifications off some developer’s forum somewhere some time ago:

namespace Skiviez.Porcupine.Model
{
    using System.Data;
    using System.Data.SqlClient;
 
    /// <summary>
    /// Contains extensions to the SqlCommand class.
    /// </summary>
    internal static class SqlCommandExtensions
    {
        /// <summary>
        /// The maximum failed attempts that can occur before a standard
        /// SqlException is thrown.
        /// </summary>
        private const int MaxFailedAttempts = 3;
 
        /// <summary>
        /// Obtains a reader safely by automatically retrying the command a
        /// number of times in case an invalid connection in the connection pool
        /// is encountered.
        /// </summary>
        /// <param name="command">The command.</param>
        /// <param name="commandBehavior">The command behavior.</param>
        /// <returns>the data reader</returns>
        /// <exception cref="T:System.Data.SqlClient.SqlException">if a non-
        /// recoverable error is encountered</exception>
        internal static SqlDataReader SafeExecuteReader(
            this SqlCommand command, 
            CommandBehavior commandBehavior)
        {
            SqlDataReader reader;
            bool retry;
            int failedAttemptsCount;
 
            reader = null;
            retry = false;
            failedAttemptsCount = 0;
 
            do
            {
                try
                {
                    reader = command.ExecuteReader(commandBehavior);
                }
                catch (SqlException e)
                {
                    if (e.ErrorCode == 10054)
                    {
                        // Transport error occurred, meaning the connection pool
                        // had a bad entry. We'll probably be okay if we retry
                        // one more time.
                        retry = true;
                        failedAttemptsCount++;
                    }
                    else
                    {
                        throw;
                    }
                }
            } 
            while (retry && failedAttemptsCount <= MaxFailedAttempts);
 
            return reader;
        }
}

Essentially, if executing the command results in a SqlException, I check to see if it is this stupid 10054 error. If it is, I’ll retry execution of the command up to 3 times, even though a single retry almost always fixes the problem. If it still doesn’t work after that–or if the error wasn’t 10054 to begin with–I assume that something really is hosed and throw the exception anyway.

Conclusions and delusions

Thanks to an extension method, it’s pretty easy to safeguard my users against an error that is not really an error without resorting to lots of boilerplate code around each ExecuteReader() call. It’s one more step of making sure that my software simply works (most of the time!).

Update 5/5/2009: This is resolved in update 6.0.2. Original post follows.

My boss at work used Adobe InDesign CS4 to create a new, 20-page printed catalog for our customers. He discovered that InDesign has this nifty “Export to SWF” feature whereby, insto-presto, the same document can be converted into a snazzy Flash version, complete with super fancy page-turning effects. Neat!

Unfortunately, the file size of the generated Flash file is enormous–at 7.4 megabytes, it definitely wasn’t something that we could realistically put on our Web site.

Seems easy enough to fix! Let’s look at the options dialog that InDesign pops up during the export process:

Words, words, words...

Awesome! “JPEG Quality” sounds like just what we need, so let’s set it to “Minimum” and export the Flash–oh, wait, it’s the same size: 7.4 MB. Go back and set it to “Maximum”–7.4 MB. Set it to “Low”–7.4 MB.

Um.

A few wasted minutes of Googling gives us this gem on Adobe’s Forums from the purported Product Manager for InDesign:

First and foremost, unfortunately there is a bug in the JPEG Quality setting of the InDesign CS4 SWF Export dialog. The bug is that no matter what value you choose from the pop-up menu (Minimum, Low, Medium, High, Maximum), they will all end up as Medium quality. The bad news is that this bug was not discovered (internally) or reported (by pre-release and CS4 customers) before we shipped CS4 (6.0.0), or the 6.0.1 dot release. The good news is that it will be fixed in the next dot release (6.0.2). I can’t provide a specific date for the next dot release yet.

In other words, the feature Simply Does Not Work. This is the type of programming mistake that I make in our backend applications at a small business, sure: but it is not the type of mistake that you would expect from a multi-million dollar software development powerhouse who has been doing this for decades and charges over $600 for the product. Ridiculous.

Try exporting to a source file

Great. So the fix is not available, and I have to get this stupid Flash file up before I can go home. Maybe I can use the other InDesign option to export to XFL. Then we can open up this file in Adobe Flash proper and sort out the images there.

That works great, but when exporting to XFL, InDesign doesn’t export any of the hyperlinks or page animations, which is kind of half the point of the exporting feature. The Help manual even documents this as if this were useful behavior. Obviously InDesign had to generate some source file to generate the enormous SWF; why can’t I have that file, and not this half-baked XFL thing?

Now I’m getting irritated

Okay. Now what? Perhaps I can decompile the enormous SWF file so that I can access the image resources and lower the quality of the JPEGs. If you Google this long enough, you will find that generally all third-party Flash products (especially those relating to decompilation) fall into three categories:

• outdated open source software that probably never worked;
• poorly written shareware; and
• Trojan horses.

I finally downloaded a demo of Eltima’s Flash Decompiler Trillix. (What the hell does “Trillix” mean?) This application looks pretty, and it does let me see the image resources on the SWF and adjust their quality. But saving is disabled in the demo.

After roughly calculating out my hourly pay, I’d by that point determined that I had wasted much more of the company’s money than the $67 licensing fee, so I broke down and paid three tanks of gas for a magic license key number. I finally degraded the quality of all of the images and got the file size from 7.4 MB to 1.0 MB, which is good enough for us. Hopefully by the time the next catalog rolls around, Adobe will have released the 6.0.2 update.

The pièce de résistance

My boss asked me if I could use the new tool to just convert the SWF to a plain old FLA source file, the idea being that in the future we’d use this file as a template and bypass InDesign entirely.

Yep, you guessed it–the Eltiva software crashes with a bad pointer reference whenever it tries to decompile the scripts in the InDesign-generated Flash file. Why? Who knows–I don’t.

Conclusions and Delusions

This rant has some purpose: hopefully, it will help prevent some future poor sap from spending nearly half a day chasing his tale. The lessons learned: avoid the Export to SWF feature in Adobe InDesign CS4, and, if you must use it, use a decompiler to manually fix up the image resources.

This is why we drink.

The tried-and-true D7600.

So at work, I’ve written a customs Windows Forms application for our hand held mobile picker (a Hand Held Dolphin 7600 running Windows CE 5.0). I originally developed it under Visual Studio 2005 on XP and migrated my project to Visual Studio 2008 without a hitch.

When I tried to install the Hand Held D7600 SDK on my home machine running Windows Vista and Visual Studio 2008, everything seemed to install properly, but the D7600 was not registered as an available target platform in Visual Studio.

There are two problems at work here:

• Even if you’re developing just a .NET application, you need to install both the C++ SDK and the .NET SDK from Hand Held’s D7600 Web site. (Hand Held was recently acquired by Honeywell, and as is typical for large company mergers, the Web site and branding are both terribly confused at the moment.) It is the C++ SDK that actually performs the task of registering the target device with Visual Studio; the .NET installer just adds a few extra assemblies.
• The C++ SDK installer performs this registration via a Windows Installer custom action. The custom action appears to function properly but silently fails under Windows Vista unless the entire installer was launched with administrative privileges. The solution is to launch an elevated Command Prompt (right-click Command Prompt in the Start menu and choose “Run as administrator”) and execute the installer that way; e.g., msiexec /i "Windows CE C SDK for Visual Studio 2005 - Rev. 1.09.msi". Make sure you don’t have any instances of Visual Studio open while the installer is running. Then the D7600 should be available as a target platform the next time you start Visual Studio.

Hope this helps someone out there!

So you know that .NET has this nifty tracing framework built in; you just plug a few lines into your system.diagnostics section of your app.config or web.config file and now your Trace statements are being output to the listener that you specify. Neat.

Let’s say that you’ve gone the extra mile and you’ve implemented your own custom trace listener. Even better, let’s say that you’ve created a trace listener that extends from TextWriterTraceListener. After all, you’re probably logging to a text file, but perhaps you wanted to change the format around a little bit.

Recall that the way for us to specify the location of our trace logging file via configuration is to use the initializeData attribute as in the following example:

<system.diagnostics>
  <sharedListeners>
    <add
      name="Listener:ApplicationText"
      type="Skiviez.Hedgehog.Model.AlignedTextWriterTraceListener, Skiviez.Hedgehog.Model"
      initializeData="Media\Logs\Hedgehog.txt"
      traceOutputOptions="ThreadId, DateTime, ProcessId" />
  </sharedListeners>
</system.diagnostics>

Right. Looks pretty sane: I’ve got my custom trace listener type (we’ll assume that AlignedTextWriterTraceListener simply extends the built-in TextWriterTraceListener), I’m telling it to log to Media\Logs\Hedgehog.txt (relative to my working directory, I would presume), and I’m passing in some output options and giving it a name.

Nothing we have done would here would leave us to believe that we have broken the way TextWriterTraceListener works. We have, but it won’t be apparent until we try to run this in a ASP.NET Web site.

What have you got against Web sites?

So when we run it an ASP.NET Web site, we will note that while we receive no error, we also see no log file sitting in the Media\Logs directory as we specified. I can understand how the tracing code probably swallows exceptions–so as to not take down your entire Web site or application because some hoo hah misconfigured a tracing statement in the configuration file–so let’s assume that we’ve screwed something up. Check the permissions on the Media\Logs directory? Check. Check the working directory? Hmm.

The working directory of ASP.NET applications is usually strange, somewhere in the %SYSTEMDIR% area–that’s because your code is usually running from some temporary location where your ASP.NET Web site was compiled just-in-time to serve the first request. So our trace listener is trying to be relative to a highly privileged directory–not relative to our Web application’s root directory–and obviously the log file can’t be created there, failing silently.

Okay, that makes sense. But that seems astonishing somehow.

You’re not going crazy

The reason why this seems astonishing is that if you replace your configuration to use the standard TextWriterTraceListener instead of your custom type that extends from it, then the log file will magically appear in the expected location, relative to your Web application’s root directory, and not relative to the system directory.

<system.diagnostics>
  <sharedListeners>
    <add
      name="Listener:ApplicationText"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="Media\Logs\Hedgehog.txt"
      traceOutputOptions="ThreadId, DateTime, ProcessId" />
  </sharedListeners>
</system.diagnostics>

Okay. So you double-check your custom type to make sure you’re really not doing anything strange with configuration. Which of course you aren’t–you’re just forwarding constructors to the base TextWriterTraceListener’s constructors. You’re not supposed to care how they actually work.

So why is the path that the initializeData attribute seems to be relative to changing wildly between the two types?

Enter the Reflector

So after spelunking through Reflector for a little while, we stumble across this little gem in the TraceUtil’s class GetRuntimeObject() method:

internal static object GetRuntimeObject(
	string className, 
	Type baseType, 
	string initializeData)
{
	// ... snip ...
 
	if (string.IsNullOrEmpty(initializeData))
	{
		if (IsOwnedTextWriterTL(c))
		{
			throw new ConfigurationErrorsException(
				SR.GetString(
					"TextWriterTL_DefaultConstructor_NotSupported"));
		}
		ConstructorInfo constructor = c.GetConstructor(new Type[0]);
		if (constructor == null)
		{
			throw new ConfigurationErrorsException(
				SR.GetString(
					"Could_not_get_constructor", 
					new object[] { className }));
		}
		obj2 = constructor.Invoke(new object[0]);
	}
	else
	{
		ConstructorInfo info2 = 
			c.GetConstructor(new Type[] { typeof(string) });
		if (info2 != null)
		{
			if ((IsOwnedTextWriterTL(c) && 
				(initializeData[0] != Path.DirectorySeparatorChar)) && 
				((initializeData[0] != Path.AltDirectorySeparatorChar) && 
				!Path.IsPathRooted(initializeData)))
			{
				string configFilePath = DiagnosticsConfiguration.ConfigFilePath;
				if (!string.IsNullOrEmpty(configFilePath))
				{
					string directoryName = Path.GetDirectoryName(configFilePath);
					if (directoryName != null)
					{
						initializeData = Path.Combine(
							directoryName, 
							initializeData);
					}
				}
			}
			obj2 = info2.Invoke(new object[] { initializeData });
		}
		// ... snip ...
	}
		// ... snip ...
}
 
internal static bool IsOwnedTextWriterTL(Type type)
{
	if ((typeof(XmlWriterTraceListener) != type) && 
		(typeof(DelimitedListTraceListener) != type))
	{
		return (typeof(TextWriterTraceListener) == type);
	}
	return true;
}

Now, I know that I for one have written some strange and hackish code in my time, but whoever wrote this function was obviously wearing his silly pants that day. Here’s what the function does:

If the following are ALL true:

• The trace output listener type is one of the three concrete types explicitly provided by Microsoft (and not an inheritor of that type)
• The initializeData attribute is provided
• The initializeData path seems to be relative

Then the function mangles the initializeData attribute by combining it with the path of the configuration file that contained the attribute. It then instances the trace listener by providing this faked-just-in-time full path as the fileName argument.

So if I had configured a TextWriterTraceListener in my configuration file, and my configuration file were in C:\Foo, and I specified Media\Logs\Hedgehog.txt as the initializeData parameter, then the TraceUtils class would give the TextWriterTraceListener instance the value of C:\Foo\Media\Logs\Hedgehog.txt as the fileName argument.

But if I had configured anything but TextWriterTraceListener, XmlWriterTraceListener, or DelimitedListTraceListener as my output listener in my configuration file, and my configuration file were in C:\Foo, and I specified Media\Logs\Hedgehog.txt as the initializeData parameter, then the TraceUtils would kindly tell me to screw myself, pass the TextWriterTraceListener instance the value of Media\Logs\Hedgehog.txt as the fileName argument, which the instance would then match with the current working directory, which is something insane like C:\Windows\System32\Temporary ASP.NET Files if we’re talking about an ASP.NET Web site, and so I would end up with C:\Windows\System32\Temporary ASP.NET Files\Media\Logs\Hedgehog.txt, which is about as useful as sticking my thumb up my butt.

So just by extending a class, we have completely broken the way one of its configuration attributes works. That’s pretty astonishing when it comes to object-oriented design.

An ugly work-around

So I wasn’t about to specify a full path in initializeData because I don’t enjoy munging a configuration value on every single machine that I happen to build and deploy my solution to. As this is written in the .NET Framework’s internal TraceUtils class, I simply cannot avoid this behavior when extending a Microsoft-provided type.

So if I want to be able to specify a relative path in initializeData with a type that extends from TextWriterTraceListener and have it mimic the TextWriterTraceListener’s initializeData relative-path-resolution behavior, my only recourse is to copy TextWriterTraceListener out of Reflector and make it my own type with my own relative-path munging semantics.

I change the constructors of my new type, which I’ve christened TextWriterTraceListener in my own namespace, as follows:

/// <summary>
/// Initializes a new instance of the TextWriterTraceListener class, 
/// using the file as the recipient of the debugging and tracing output. 
/// </summary>
/// <param name="fileName">The name of the file the 
/// TextWriterTraceListener writes to. </param>
public TextWriterTraceListener(string fileName)
{
	this.fileName = MungeFileName(fileName);
}
 
/// <summary>
/// Initializes a new instance of the TextWriterTraceListener class 
/// with the specified name, using the file as the recipient of the 
/// debugging and tracing output. 
/// </summary>
/// <param name="fileName">the fileName to output to</param>
/// <param name="name">the name of the trace listener</param>
public TextWriterTraceListener(string fileName, string name) : base(name)
{
	this.fileName = MungeFileName(fileName);
}

and the gloriously-named MungeFileName() method is

/// <summary>
/// Munges the file name such that if it is a relative path, we go
/// relative from the configuration file and not from the current working
/// directory. This makes things work as expected on ASP.NET sites and
/// makes other applications similarly work with non-astonishing
/// behavior.
/// </summary>
/// <param name="fileName">the file name to munge</param>
/// <returns>the munged filename</returns>
private static string MungeFileName(string fileName)
{
	string configPath;
	string mungedFileName;
 
	mungedFileName = fileName;
 
	if (fileName[0] != Path.DirectorySeparatorChar &&
		fileName[0] != Path.AltDirectorySeparatorChar &&
		!Path.IsPathRooted(fileName))
	{
		ConfigurationSection configSection;
 
		configSection = (ConfigurationSection)ConfigurationManager
			.GetSection("system.diagnostics");
		if (configSection != null)
		{
			configPath = configSection.ElementInformation.Source;
 
			if (!string.IsNullOrEmpty(configPath))
			{
				string directoryName;
 
				directoryName = Path.GetDirectoryName(configPath);
 
				if (directoryName != null)
				{
					mungedFileName = Path.Combine(
						directoryName, 
						fileName);
				}
			}
		}
	}
 
	return mungedFileName;
}

What I’m doing here is saying, “Is the path relative? Then figure out where my system.diagnostics configuration came from, and make myself relative to THAT directory.” This way, I don’t have to have different code paths for this class depending on whether it’s running in a traditional Windows application or an ASP.NET Web site–the initializeData is always relative to the configuration file and not the current working directory, which is what I want 99% of the time, and it’s probably what most people want, too.

Conclusions and Delusions

I think the primary lesson to take away from this is to avoid doing magic configuration and baby-sitting for a class’s constructors. I have a feeling that TextWriterTraceListener was implemented by somebody at Microsoft and sometime much later in the development process they realized that specifying a relative initializeData was difficult for ASP.NET applications. Being unable to alter TextWriterTraceListener without breaking compatibility for non-ASP.NET applications that may have already been built, they devised an unusual TraceUtils class to munge the parameters read from configuration before they were passed to a new instance’s constructor.

The result? It worked well if you stuck to the built-in classes, but sure was damn surprising if you created a custom type and eventually used it on an ASP.NET project.

Hope this helps someone out there. You’re not insane, after all.

Here’s the implementation for your convenience.