Simply Does Not Work

When attempting to re-integrate a feature branch into a trunk in my Subversion repository, I was getting a weird “Cannot reintegrate into a working copy not entirely at infinite depth” error.

To fix this, I had to fix two problems.

First, my working copy was very “old.” Like, two years of sitting on my desktop old. And I had upgraded Subversion since that time. Apparently, something was not happy with this, particularly related to some new checkout option not being set. So I created a new directory and made a fresh checkout, specifying the “Fully recursive” option in TortoiseSVN.

When I did this, I got a new message: “Retrieval of mergeinfo unsupported.” Still an error, but different!

I had upgraded my Subversion server to 1.6, but I never actually upgraded my repositories to the latest version, which was created way back in 1.4. So I had been operating in a kind of backwards-compatibility mode for two years. Whoops.

Running svnadmin upgrade c:\path\to\my\repo on the server fixed this right up. I also ran the recommended svn-populate-node-origins-index.exe program, which I’m sure did something magical.

Now I can complete the merge successfully in TortoiseSVN. Hope this helps someone.

So, as I’ve described previously, we’ve set up a Magento store to clear out some of the older inventory at the company I work for: consider Men’s Underwear Discounters to be the bargain bin, Sunday! Sunday! Sunday! closeout section for Skiviez. Using a free, open source platform like Magento has saved us a lot of time and money, especially for a store that we’re not really making huge amounts of money off of anyway (it’s there to recoup the costs of products that didn’t sell well, sort of like damage control).

Aside: You probably have configured your sales e-mails incorrectly

I have seen three live Magento installations. I have also seen three Magento installations where sales e-mails have been customized improperly. Since it has happened three times, by the engineer’s first law of truth, it must be true that many people have been configuring the sales e-mails in their Magento stores improperly.

By default, all of the sales e-mails are written in less-than-stellar English and contain references to things like “Magento Demo Store” and “(555) 555-5555.” And these sales e-mail templates live by default in the /app/design/frontend/default/default/template/email directory.

If you’re like us, you made the mistake of editing these files directly, replacing the references to “Magento Demo Store” with references to your own store, like “Men’s Underwear Discounters.” And this would work fine until you, like us, apply the next security update or upgrade via Magento Connect. If they’ve updated the default template, then you can be slightly bemused that, like us, all of your sales e-mails have reverted to the defaults, and you are now thanking customers for shopping at the “Magento Demo Store.” The result? You, like us, look like an idiot.

What we need is a way to make sure that our e-mail customizations persist across security updates and upgrades.

The way to do that is to set up new copies of each and every e-mail. From the Magento Admin screen, you need to

1. Choose System > Transactional E-mails from the menu.
2. Click “Add New Template.”
3. Under “Load default template”, select the template you would like to customize, then hit “Load Template”. The “Template Subject” and “Template Content” fields are populated with a copy of the template you selected.
4. Edit that template to your heart’s content, give it a name, and hit “Save Template”.
5. Under System > Configuration > Sales E-mails, be sure to select your new template instead of “Default from locale”.

Hooray! Now the changes that we make will stick around.

Formatting those e-mails as plain text

For Skiviez proper, we send multipart e-mails, where both HTML- and text-formatted messages are sent within the same e-mail. This works great because, depending on the customer’s preference and e-mail capabilities, they can view the e-mail in the format that they prefer.

Unfortunately, Magento doesn’t (at the time of this writing) have the capability for these multipart e-mails. Instead, you get to choose between HTML (the default) and plain text. I decided to go with plain text because

• I had seen issues on the forums where the HTML e-mails have problems displaying in some Web-based e-mail clients; and
• Nothing pisses me off more than HTML e-mail. I don’t know why. I just hate it.

Converting most of the e-mails to plain text is straight-forward; you just hit the “Convert to Plain Text” button and then reformat the template content so that it is not hideous. But when you get to the order e-mail, you’ll discover that some of the variables and references are returning the content in HTML; particularly, the address, payment, and item summary fields.

You could do something like changing the reference from

{{ var order.getBillingAddress().format('html') }}

to

{{ var order.getBillingAddress().format('text') }}

and that would work, but the output would not be what you expect:

 
Nicholas Piasecki
 
3005 Some Street
 
 
 
Sillytown,  California, 92683
United States
T: 5555555555

That is, if an element in the address is empty, it still receives a blank space, so you end up with the odd address formatting that you see in the above example.

You could then spend an hour and a half just trying to figure out what that magic ->format('html') or ->format('text') call actually does and where it’s coming from. That is an adventure in and of itself, but to make a long story short, there is a config.xml file in the /app/code/core/Mage/Customer/etc/ directory that contains entries like the following:

                    <text translate="title" module="customer">
                        <title>Text</title>
                        <defaultFormat><![CDATA[
{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}
{{depend company}}{{var company}}{{/depend}}
{{var street1}}
{{depend street2}}{{var street2}}{{/depend}}
{{depend street3}}{{var street3}}{{/depend}}
{{depend street4}}{{var street4}}{{/depend}}
{{depend city}}{{var city}},  {{/depend}}{{depend region}}{{var region}}, {{/depend}}{{var postcode}}
{{var country}}
T: {{var telephone}}
{{depend fax}}F: {{var fax}}{{/depend}}
                        ]]></defaultFormat>
                    </text>
                    <oneline translate="title" module="customer">
                        <title>Text One Line</title>
                        <htmlEscape>true</htmlEscape>
                        <defaultFormat>
<![CDATA[{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}, {{var street}}, {{var city}}, {{var region}} {{var postcode}}, {{var country}}]]>
                        </defaultFormat>
                    </oneline>
                    <html translate="title" module="customer">
                        <title>HTML</title>
                        <htmlEscape>true</htmlEscape>
                        <defaultFormat><![CDATA[
{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}<br/>
{{depend company}}{{var company}}<br />{{/depend}}
{{var street1}}<br />
{{depend street2}}{{var street2}}<br />{{/depend}}
{{depend street3}}{{var street3}}<br />{{/depend}}
{{depend street4}}{{var street4}}<br />{{/depend}}
{{depend city}}{{var city}},  {{/depend}}{{depend region}}{{var region}}, {{/depend}}{{var postcode}}<br/>
{{var country}}<br/>
{{depend telephone}}T: {{var telephone}}{{/depend}}
{{depend fax}}<br/>F: {{var fax}}{{/depend}}
            ]]></defaultFormat>
                    </html>

Once you stumble across that, it’s not so difficult to figure out what is going on. The Magento address object returns a Varien_Filter_Template object that takes one of the above defaultFormat entries as its template based on the parameter that you pass into the format() function. So you can see that valid parameters are ‘html’, ‘text’, ‘oneline’, and so on.

The {{depend}} syntax means that if the enclosed variable is empty, then that output won’t appear. For HTML, this works fine, since newlines aren’t signficant. But for the text template, this doesn’t work, because even though things like {{street2}} get omitted, the newline is still sitting there in the template. This is why we get the fugly output in our plain text e-mails.

My solution is to define my own block that I just reference in my e-mails. Here’s the e-mail template for Men’s Underwear Discounters, for example:

Dear {{var order.getCustomerName()}},
 
Thank you for your order from Men's Underwear Discounters.
Once your package ships, we will send an email with a link 
to track your order. You can check the status of your order
by logging into your account.
 
If you have any questions about your order please contact
us at support@mensunderweardiscounters.com.
 
Your order confirmation is below. Thank you again for your 
business.
 
ORDER {{var order.increment_id}} 
 
Placed on {{var order.getCreatedAtFormated('long')}}
 
BILLING INFORMATION
 
{{block type='core/template' area='frontend' template='email/order/addresstext.phtml' address=$order.getBillingAddress()}}
 
SHIPPING INFORMATION
 
{{block type='core/template' area='frontend' template='email/order/addresstext.phtml' address=$order.getShippingAddress()}}
 
via {{var order.getShippingDescription()}}
 
ORDER DETAILS
 
{{block type='core/template' area='frontend' template='email/order/itemstext.phtml' order=$order}} 
 
Thanks for shopping at Men's Underwear Discounters.
 
Sincerely,
The Men's Underwear Discounters Team
support@mensunderweardiscounters.com

If you compare this with the default e-mails, you can see that I’ve added these new {{block}} entries in several places. All they are saying is “hey, create a new static block with the file at /app/design/frontend/default/default/template/email/order/itemstext.phtml (or addresstext.phtml) and pass a variable into it. Whatever variable I pass in here will be accessible in the .phtml file by referencing $this->getWhatever(), where “Whatever” is the name of the parameter. For example, my itemstext.phtml file could reference $this->getOrder() and my addresstext.phtml file could reference $this->getAddress().

These “itemstext.phtml” and “addresstext.phtml” files don’t exist in the default installation. I had to create them and FTP them to that appropriate directory. The contents aren’t exotic; they’re just PHP files that emit output that’s suitable for dropping into a plain text e-mail.

Here’s the itemstext.phtml file:

<?php
	$order = $this->getOrder();
 
	if ($order)
	{
		foreach ($order->getAllItems() as $item)
		{
			if ($item->getParentItem())
			{
				continue;
			}
 
			echo '(';
			echo $item->getQtyOrdered() * 1;
			echo ')';
 
			echo ' ';
 
			echo $item->getName();
 
			echo ' @ ';
 
			echo number_format($item->getPrice(), 2);
 
			echo ' ea = ';
 
			echo number_format($item->getRowTotal(), 2);
 
			echo "\n";
		}
 
		echo "\n";
 
		echo 'Subtotal: $';
		echo number_format($order->getSubtotal(), 2);
		echo "\n";
 
		echo 'Discount: $';
		echo number_format(0.00 - $order->getDiscountAmount(), 2);
		echo "\n";
 
		echo 'Shipping & Handling: $';
		echo number_format($order->getShippingAmount(), 2);
		echo "\n";
 
		echo 'Tax: $';
		echo number_format($order->getTaxAmount(), 2);
		echo "\n";
 
		echo 'Grand Total: $';
		echo number_format($order->getGrandTotal(), 2);
		echo "\n";
	}
?>

And here’s the addresstext.html file:

<?php
	$address = $this->getAddress();
 
	if ($address) {
		if (strlen($address->getPrefix()) > 0) {
			echo $address->getPrefix() . ' ';
		}
 
		echo $address->getFirstname();
 
		if (strlen($address->getMiddlename()) > 0) {
			echo ' ' . $address->getMiddlename();
		}
 
		echo ' '. $address->getLastname();
 
		if (strlen($address->getSuffix()) > 0) {
			echo ' ' . $address->getSuffix();
		}
 
		echo "\n";
 
		if (strlen($address->getCompany()) > 0) {
			echo $address->getCompany() . "\n";
		}
 
		echo $address->getStreet1() . "\n";
 
		if (strlen($address->getStreet2()) > 0) {
			echo $address->getStreet2() . "\n";
		}
 
		if (strlen($address->getStreet3()) > 0) {
			echo $address->getStreet3() . "\n";
		}
 
		if (strlen($address->getStreet4()) > 0) {
			echo $address->getStreet4() . "\n";
		}
 
		if (strlen($address->getCity()) > 0) {
			echo $address->getCity() . ', ';
		}
 
		if (strlen($address->getRegion()) > 0) {
			echo $address->getRegion() . ' ';
		}
 
		echo $address->getPostcode() . "\n";
 
		echo $address->getCountry();
 
		if (strlen($address->getTelephone()) > 0) {
			echo "\n" . 'T: ' . $address->getTelephone();
		}
	}
?>

Conclusions and delusions

This story has been typical of my experiences with Magento. I eventually end up finding something that was thoughtfully designed (if slightly overarchitected), but poor documentation and difficulty testing impede progress more than necessary. Let’s hope that this gets better in future versions of the framework.

Good luck.

One of the pieces of WFS is an application that I’ve written that runs on our Honeywell Dolphin 7600. The Dolphin is a little mobile device than runs Windows CE 5.0, has 802.11g access, and has a barcode scanner. The application that I’ve written allows a warehouse employee to pick fulfillment requests, scan in inbound shipments against a previously recorded manifest, and process any returns that customers have sent back. Pretty neat stuff.

I’ve designed most of the interface so that it can be used with just the big green “SCAN” button and the numeric keypad, since it’s cumbersome to bust out a stylus and use two hands while using it in the warehouse. There are, however, a few cases where the user does need to type something in, typically via the soft input panel, which is the stupid Microsoft name for the on-screen keyboard. Examples where the on-screen keyboard are necessary include typing in a merchant’s identifier for an inbound shipment manifest or typing in a user name and password to sign into the application.

By default, the soft input panel displays “whereever it feels like,” typically at the bottom of the screen or whatever random location the user last left it in, and while the user can drag it out of the way with the stylus if it obstructs UI elements, I thought it would be nice if I showed, hid, and positioned the soft input panel automatically when necessary. For example, if the user focuses a textbox where I expect they’ll need to type in some text, I’ll pop up the soft input panel and position it automatically just underneath that textbox.

If you load up Visual Studio, you’ll see the Microsoft.WindowsCE.Forms.InputPanel available in the toolbox. But it doesn’t provide any way to position the input panel once it is displayed. For that, I had to P/Invoke into a native method.

/// <summary>
/// Shows the input panel relative to the given control.
/// </summary>
/// <param name="control">The control.</param>
private void ShowInputPanel(Control control)
{
	NativeMethods.SIPINFO sipInfo;
	var x = 0;
	var y = control.PointToScreen(new Point(0, control.Height)).Y;
 
	this.inputPanel.Enabled = true;
 
	sipInfo = new NativeMethods.SIPINFO();
	sipInfo.cbSize = (uint)Marshal.SizeOf(sipInfo);
	if (NativeMethods.SipGetInfo(ref sipInfo))
	{
		sipInfo.rcSipRect.left = x;
		sipInfo.rcSipRect.top = y;
 
		NativeMethods.SipSetInfo(ref sipInfo);
	}
}

The above method simply takes an instance of a Control on the form and shows the soft input panel right underneath it. The input panel is shown by setting Enabled to true; it’s the SipSetInfo() call that actually positions it. Since our screen is super tiny, I only worry about the y coordinate; no sense in positioning in the horizontal direction because otherwise I’d crop off part of the input panel from view. The NativeMethods class is where I dump all of my P/Invoke declarations:

/// <summary>
/// This structure contains information about the current state of the 
/// software-based input panel, such as the software-based input panel 
/// size, screen location, docked status, and visibility status.
/// </summary>
[StructLayout(LayoutKind.Sequential)]
public struct SIPINFO
{
	/// <summary>
	/// Size, in bytes, of the SIPINFO structure. This member must be 
	/// filled in by the application with the size of operator. Because
	/// the system can check the size of the structure to determine 
	/// the operating system version number, this member allows for
	/// future enhancements to the SIPINFO structure while maintaining 
	/// backward compatibility.
	/// </summary>
	public uint cbSize;
 
	/// <summary>
	/// Specifies flags representing state information of the 
	/// software-based input panel. The following table shows the
	/// possible bit flags. These flags can be used in combination. 
	/// </summary>
	public uint fdwFlags;
 
	/// <summary>
	/// Rectangle, in screen coordinates, that represents the area of 
	/// the desktop not obscured by the software-based input panel. 
	/// If the software-based input panel is floating, this rectangle 
	/// is equivalent to the working area. Full-screen applications 
	/// that respond to software-based input panel size changes can 
	/// set their window rectangle to this rectangle. If the
	/// software-based input panel is docked but does not occupy 
	/// an entire edge, then this rectangle represents the largest 
	/// rectangle not obscured by the software-based input panel. 
	/// If an application wants to use the screen space around the
	/// software-based input panel, it needs to reference rcSipRect.
	/// </summary>
	public RECT rcVisibleDesktop;
 
	/// <summary>
	/// Rectangle, in screen coordinates of the window rectangle and 
	/// not the client area, the represents the size and location of 
	/// the software-based input panel. An application does not
	/// generally use this information unless it needs to wrap
	/// around a floating or a docked software-based input panel 
	/// that does not occupy an entire edge.
	/// </summary>
	public RECT rcSipRect;
 
	/// <summary>
	/// Specifies the size of the data pointed to by the pvImData member.
	/// </summary>
	public uint dwImDataSize;
 
	/// <summary>
	/// Void pointer to IM-defined data. The IM calls the
	/// IInputMethod::GetImData and IInputMethod::SetImData methods to
	/// send and receive information from this structure.
	/// </summary>
	public IntPtr pvImData;
}
 
/// <summary>
/// This structure defines the coordinates of the upper-left and
/// lower-right corners of a rectangle. 
/// </summary>
[StructLayout(LayoutKind.Sequential)]
public struct RECT
{
	/// <summary>
	/// Specifies the x-coordinate of the upper-left corner of the rectangle. 
	/// </summary>
	public int left;
 
	/// <summary>
	/// Specifies the y-coordinate of the upper-left corner of the rectangle. 
	/// </summary>
	public int top;
 
	/// <summary>
	/// Specifies the x-coordinate of the lower-right corner of the rectangle. 
	/// </summary>
	public int right;
 
	/// <summary>
	/// Specifies the y-coordinate of the lower-right corner of the rectangle. 
	/// </summary>
	public int bottom;
}
 
/// <summary>
/// This function receives information including the state of the
/// software-based input panel, the area of the desktop that is not 
/// obscured by the software-based input panel, the screen coordinates
/// of the software-based input panel, and information about the input 
/// method (IM) that the software-based input panel is currently using.
/// </summary>
/// <param name="sipInfo">[out] Pointer to the SIPINFO structure that
/// contains information about the current software-based input panel.</param>
/// <returns>TRUE indicates success. FALSE indicates failure. To get 
/// extended error information, call GetLastError. </returns>
[DllImport("coredll.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SipGetInfo(
	ref SIPINFO sipInfo);
 
/// <summary>
/// This function sets information including the state of the 
/// software-based input panel, the area of the desktop that is not 
/// obscured by the software-based input panel, the screen coordinates 
/// of the software-based input panel, and application-defined information
/// about the input method (IM) that the software-based input panel is
/// currently using.
/// </summary>
/// <param name="sipInfo">ointer to the SIPINFO structure that contains
/// information about the current software-based input panel.</param>
/// <returns>TRUE indicates success. FALSE indicates failure. 
/// To get extended error information, call GetLastError. </returns>
[DllImport("coredll.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool SipSetInfo(
	ref SIPINFO sipInfo);

Yes, they really did call it “coredll.dll”.

You can imagine how calling these methods is now pretty simple. Assume that these methods are attached to the GotFocus and LostFocus events of a TextBox control:

/// <summary>
/// Fired when the user name textbox gets focus.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="e">The <see cref="System.EventArgs"/> instance 
/// containing the event data.</param>
private void WhenUserNameGotFocus(object sender, EventArgs e)
{
	this.ShowInputPanel(this.userNameTextBox);
}
 
/// <summary>
/// Fired when the user name textbox loses focus.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="e">The <see cref="System.EventArgs"/> instance 
/// containing the event data.</param>
private void WhenUserNameLostFocus(object sender, EventArgs e)
{
	this.HideInputPanel();
}

I’ve only tested this on our Windows CE 5.0-based device, but I imagine that it’d work on any later platform that was supported by the InputPanel control. Good luck!

Consider the following:

string hql;
 
hql = @"
	from Item as i
	inner join i.Barcodes as b     
	inner join i.ItemGroup.Merchant as m   
	where b.Text = :barcode
	and m.Code = :merchantCode
";
 
query = session.CreateQuery(hql)
	.SetString("barcode", barcode)
	.SetString("merchantCode", merchantCode);
 
var items = query.List<Item>(); // explodes

Let’s assume that Barcodes is a composite-element of the Item.

This will explode with an exception that reads “Could not execute query[SQL: SQL not available]” while the inner exception reads “The value “System.Object[]” is not of type “Skiviez.Armadillo.Model.Item” and cannot be used in this generic collection. Parameter name: value”.

That is because I am calling List<Item>(), but as specified, my HQL query is returning a list of Items and Merchants.

To fix it, my HQL needs to be

string hql;
 
hql = @"
	select
		i
	from Item as i
	inner join i.Barcodes as b     
	inner join i.ItemGroup.Merchant as m   
	where b.Text = :barcode
	and m.Code = :merchantCode
";

The error message was a little vague so it took me a while to figure that out. Essentially, it means that the SQL query was probably executed fine, but NHibernate is blowing up while populating the list of result objects.

AVS Information In Magento

We’ve recently set up a new e-commerce store at Men’s Underwear Discounters that represents a division of the company that I work for, Skiviez. We decided to save time for this discount store by using Magento, the current darling child in the open source e-commerce platform market segment.

A brief interlude on consulting, documentation, and open source

Now, to be clear, it doesn’t take much to exceed expectations in the “open source e-commerce platform” market because prior to Magento, your options were pretty much limited to osCommerce, which is quite possibly the most reckless, idiotic excuse of an e-commerce platform that I have seen (and I don’t like to insult code), and its slightly-less-retarded sibling Zen Cart. (I will save the osCommerce rants for another post.)

Magento seems to be a dream come true: thousands of man-hours of development into an e-commerce platform for free. So what’s the catch? While Varien (“the Magento company”) has done a great service to the world and the open source community by creating Magento, they’re still in it to make a buck–and that’s not a bad thing–but you might not realize how they make those bucks until you’ve started playing around with your own Magento store. They make money on support, consulting, and customization because Magento is huge and poorly documented.

Actually, “poorly documented” would imply that there was at least some documentation, and while there truly is some documentation available in the source code and on Varien’s Magento Web site, in reality, that amount of documentation is a statistical rounding error. And when you actually view your Magento installation in an FTP program, you will be absolutely astounded at the number of directories and files. You can try inspecting the source files to try to put it all together, but a lot of it is “magic”: a framework erected at runtime by XML configuration files and PHP’s idiotic __call mechanism for magic functions that appear at runtime out of thin air. You can see references to a function called getOrder(), for example, grep the entire installation for “getOrder()”, and not get any results because the function doesn’t actually exist at compile time. Personally, I’ve always considered magic methods to be an extremely unwise idea, a “too clever” approach to programming. But for people new to PHP looking to do a little customization to their Magento store, it makes the codebase all the more impenetrable. That’s more consulting dollars.

You could go to the Magento forums, but you won’t find too many helpful answers because Magento obviously doesn’t want to give help for free, since giving help is central to their business model. So if you need help, your best bet is to generally (a) pay for it or (b) hope that you stumble across somebody’s blog who took the time to post a solution to your exact problem.

Why do I need to do this, anyway?

Magento has a lot of stuff built in. But we need to remember that Magento was built by software developers, and software developers generally don’t understand the problem domain of running an e-commerce business very well. Sure, every software developer has built the pet shop–products, categories, inventory, orders, how hard could it be?–but they stop or fail to understand that there’s a whole class of operations that need to happen after the order has been placed.

One of those operations is fraud analysis. The Address Verification System (AVS) and the Card Verification Number (CVN) system for credit cards enable merchants to help determine if a credit card has been fraudulently used on an order. AVS will tell the merchant whether or not the billing address specified by the customer matches or partially matches the billing address on the credit card’s billing statement. And the CVN number represents the number that (supposedly) exists in only two places in the world: on the physical credit card and in the issuing bank’s files. If a CVN doesn’t match and the AVS doesn’t match, then there’s a good bet that there is something wrong with the order.

Because software developers are nerds, this information doesn’t currently display by default in Magento’s administration section. So let’s add it.

Can’t I just edit the template?

I’m talking about adding this AVS display functionality in the context of the CyberSource extension for Magento. After spelunking through the myriad files and directories, I discovered that the Varien-provided CyberSource extension is saving the AVS (setCcAvsStatus()) and CVN (setCcCidStatus()) information when it authorizes the card, but it doesn’t display in the information block in the administration section.

While you could just modify app/design/adminhtml/default/default/template/cybersource/info.phtml directly (yes, that is the actual path to the file that displays the little “Payment” block), your changes would be lost and overwritten in the future if you ever upgraded the CyberSource extension via Magento Connect. That’s obviously less than ideal.

A better way would be to use Magento’s “rewrite” mechanism. We’ll create our own Magento module and instruct Magento to replace the CyberSource extension’s information block with our own.

Creating our own module

First, we’ll create a new empty directory to hold our files. Let’s call it avsdisplay. Then we’ll create code, etc, and design subdirectories. These directories mirror the subdirectories in the app folder of your Magento installation.

In the etc directory, we need to create an XML configuration file. Magento scans this directory at runtime for XML files and reads them to know which modules to load. I’ve named my configuration file Mud_Avsdisplay.xml, and the naming is very important. (Magento uses a mixture of convention and configuration.) The part of the filename before the underscore is your “namespace,” since PHP doesn’t support namespaces. And the part of the filename after the underscore is the name of your module. Mine is “Avsdisplay.” Don’t try to use something like “AvsDisplay,” however, because that would break Magento’s convention as we’ll see in a moment.

Inside this file, we tell Magento to load our module in the local pool and to actually run it (by setting “active” to “true”). (The other pools are core, for those modules provided by Varien, and community, for those modules that you can download via Magento Connect.)

<?xml version="1.0"?>
<config>
	<modules>
		<Mud_Avsdisplay>
			<active>true</active>
			<codePool>local</codePool>
		</Mud_Avsdisplay>
	</modules>
</config>

Now it’s time to create the meat of the module. In the code folder, create a local directory. Within that, create a directory named after your chosen namespace (Mud in my example) and within that yet another subdirectory named after your module (Avsdisplay in my example). (This is why naming was important. If I had called it “Mud_AvsDisplay,” with a capital ‘D’, then Magento would have looked in app/code/local/Mud/Avs/Display for my module instead of app/code/local/Mud/Avsdisplay.)

Within your module directory, you’ll need to create three more subdirectories: Block, etc, and Helper. When it’s all said and done, you’ll end up with a hierarchy that looks like this (ignore the “design” directory, we’ll get to that later):

Directory Structure

Creating the block

Next, we need to create the “block” that we’ll be replacing. After searching through the codebase, I figured out that I wanted to replace the Mage_Cybersource_Block_Info block (for reference, by Magento’s convention, that file lives in app/core/Mage/Cybersource/Block/Info.php — see the convention?). But I really only want to add stuff to it, not replace it outright, so I’ll create a new file in my Block folder called Info.php that extends that class. Something like:

<?php
 
class Mud_AvsDisplay_Block_Info extends Mage_Cybersource_Block_Info
{
	// Yes, these really have to be protected, not private, due to more Magento
	// "magic"
	protected $avsDescriptions = array(
		// Snipped for brevity; download file at end of blog post
	);
 
	protected $ccCidDescriptions = array(
		// Snipped for brevity; download file at end of blog post
	);
 
	protected function _construct()
	{
		parent::_construct();
		$this->setTemplate('avsdisplay/info.phtml');
	}
 
	public function getAvsStatusDescription()
	{
		$info = $this->getInfo();
		$avsCode = $info->getCcAvsStatus();
		$avsDescription = 'Unrecognized response code.';
 
		if (array_key_exists($avsCode, $this->avsDescriptions))
		{
			$avsDescription = $this->avsDescriptions[$avsCode];
		}
 
		return $avsDescription;
	}
 
	public function getCcCidDescription()
	{
		$info = $this->getInfo();
		$ccCidCode = $info->getCcCidStatus();
		$ccCidDescription = 'Unrecognized response code.';
 
		if (array_key_exists($ccCidCode, $this->ccCidDescriptions))
		{
			$ccCidDescription = $this->ccCidDescriptions[$ccCidCode];
		}
 
		return $ccCidDescription;
	}
}

Okay, this makes sense. I’m doing everything the old CyberSource info block did, except I added two new functions that give me human-friendly descriptions of the AVS and CVN codes, and I’m setting the template to something called avsdisplay/info.phtml instead of cybersouce/info.phtml. How this comes into play will make sense in a minute.

Creating the helper

In my Helper directory, I need to add a Data.php file that looks like this:

<?php
 
class Mud_Avsdisplay_Helper_Data extends Mage_Core_Helper_Abstract
{
}

That’s right, it does nothing except inherit from a Magento-provided abstract class. What is a helper? I have no idea, but you have to do it–Magento looks for this file.

Telling Magento to do the swap

In my etc directory, I need to add a config.xml file that will tell Magento some information about my module and, most importantly, that I want to be swapped with the CyberSource extension’s info block:

<?xml version="1.0"?>
<config>
	<modules>
		<Mud_Avsdisplay>
			<version>0.0.1</version>
		</Mud_Avsdisplay>
	</modules>
	<global>
		<helpers>
			<avsdisplay>
				<class>Mud_Avsdisplay_Helper_Data</class>
			</avsdisplay>
		</helpers>
		<blocks>
			<cybersource>
				<rewrite>
					<info>Mud_Avsdisplay_Block_Info</info>
				</rewrite>
			</cybersource>
		</blocks>
	</global>
</config>

Under <modules>, I’m just telling Magento the version number of my module. Under <global>, I say that I want to “rewrite” the CyberSource info block with my own class named Mud_Avsdisplay_Block_Info, which, by convention, Magento will look for in [module-root]/Mud/Avsdisplay/Block/Info.php. That is the file that I just created. Great.

Finally, writing the template that displays the information

But we haven’t actually changed any HTML at this point! Remember in my Info.php file where I set the template to something called info.phtml? Well, I need to create that file. At the top of my module folder–the one with the code and etc subdirectories–I’ll need to create some more subdirectories. A lot of them. In fact, I’ll need to create a bunch of empty directories that look like this:

In the bottom-most subdirectory, I’ll create the info.phtml file. This is a PHP file, except that when I use the $this pseudo-variable in this file, $this will be pointing to an instance of my Mud_Avsdisplay_Block_Info class. So I’ll be able to call my getAvsStatusDescription() method here among other things:

<?php
	if ($info = $this->getInfo())
	{
		?>
			<strong><?php echo $this->getMethod()->getTitle(); ?></strong><br />
			<strong>Type:</strong> <?php echo $this->__($this->getCcTypeName()); ?><br />
			<strong>Number:</strong> xxxx-<?php echo $this->__($info->getCcLast4()); ?><br />
			<strong>Expiry:</strong> <?php echo $this->__('%s/%s', $this->getCcExpMonth(), $info->getCcExpYear()); ?><br />
			<strong>AVS:</strong> (<?php echo $this->__($info->getCcAvsStatus()); ?>) <?php echo $this->__($this->getAvsStatusDescription()); ?><br />
			<strong>CCID:</strong> (<?php echo $this->__($info->getCcCidStatus()); ?>) <?php echo $this->__($this->getCcCidDescription()); ?><br />
		<?php
	}
?>

Conclusions and delusions

And that’s it. Upload this entire directory structure to your Magento installation’s app directory, merging the contents as necessary, and you should be able to see AVS information for your CyberSource orders in the administration section.

How did I figure out how to learn all of this? It was mostly by lots of Googling, picking apart existing modules to see how they were organized, and piecing together lots of blog posts. I hope this helps someone out there. Good luck!

Download the code used in this article.

When the following are all true with NHibernate 2.1, you’ll receive a NullReferenceException: Object reference not set to an instance of an object thrown from the innards of NHibernate’s AliasToBeanTransformer, mentioning some nonsense about a tuple:

• You’re writing an HQL query.
• You’re using aliases and the AliastoBeanTransformer.
• You’re executing the HQL query via a MultiQuery.

The problem is that, for reasons that are not clear to me, the alias names get stripped when they’re used in a MultiQuery. This results in NHibernate telling the AliasToBeanTransformer something like “here, go ahead and set the property called NULL to 42.” And so it explodes when it references the property name string that is null.

If you just use CreateQuery() instead of a multiquery, however, the aliases work fine. So that is the workaround. Alternatively, you could use a constructor-style projection in your HQL, but I think that’s a pain.

In short: AliasToBeanTransformer simply does not work in a multiquery.

Happy NHibernating.

I’ll continue the WiX-related discussion as I build the installers that maintain the system run at Skiviez’s new venture Warehouse Fulfillment Services. WiX has been fantastic, and without it, I probably wouldn’t be using the Windows Installer, but I’ve quickly found that having these installation packages makes deploying updates to the production server and much less tense process: for the Web site, the API server, and the background task worker, installing and updating to a newer version of my software is as simple as a double-click.

I also created an installer for the database. This is new territory for me; in the past, well, the database has largely been ignored as far as configuration management was concerned. It usually had grown “organically” over time, and by organically, I mean people adding things as they needed it directly in SQL Server Management Studio. Setting up a test database, to that end, usually meant grabbing a copy of the production database and using it with sensitive data removed. Blech.

Since I was dealing with a new system, I wanted to avoid this situation from the beginning, being able to release updates to the database in a sensible, maintainable way, and ensuring that the database could be installed on a new machine as a turn-key solution should the software ever become valuable enough to be sold. What I wanted was a WiX/MSI installation package that would do the following:

• On the first install, it should create the database and execute SQL scripts that
  • create any new users or logins required by the database;
  • create the schema;
  • create the “version 1.0″ tables, indexes, and foreign key relationships; and
  • populate the database with “required data,” such as those in lookup tables like Core.Countries or Core.AddressTypes.
• On any subsequent upgrade, it should just run SQL patch scripts that make changes to the database.

Now, I could make this simple or I could make this complicated. To keep things simple, I assume that the database versioning progresses linearly: that is, the SQL patch scripts that I add in the future (to add new tables, add indexes, and so on) will be expected to be executed in a certain order. I’m of the opinion that this is a pretty sensible restriction to have on a database.

The way I ended up doing this in WiX was by creating a project with two components. One component contains the database and the scripts that are run to create and populate the virgin database; it only ever runs once. The second component contains the patch scripts; it runs on every install or upgrade.

Wouldn’t having all those patch scripts run every time cause problems, like trying to create tables twice and such? Well, yes, so I’ve made sure to construct the scripts according to a certain pattern. My database has a table called Core.DatabasePatches which contains a GUID and a human-readable description field. Each of my SQL patch scripts, let’s take AddIndexToVariationTypes.sql as an example, is then coded to the following pattern:

BEGIN TRANSACTION;
 
DECLARE @PatchIdentifier UNIQUEIDENTIFIER;
DECLARE @PatchDescription NVARCHAR(255);
 
SET @PatchIdentifier = '{C2929CCB-54C7-4060-8CD5-D606B0BB2C77}';
SET @PatchDescription = 'Adds an index to the Core.VariationTypes table to support bulk import operations.';
 
IF (NOT EXISTS(SELECT * FROM Core.DatabasePatches WHERE PatchIdentifier = @PatchIdentifier))
BEGIN
 CREATE INDEX IX_VariationTypes_VariationTypeName ON
	Core.VariationTypes (VariationTypeName, MerchantId);
 
	INSERT INTO Core.DatabasePatches (
		PatchIdentifier,
		Description
	) VALUES (
		@PatchIdentifier,
		@PatchDescription
	);
END
 
COMMIT TRANSACTION;

You can see that all of the work is wrapped in a big IF statement that says “check the Core.DatabasePatches table to see if this patch has already been applied, and if it has, well don’t do it again.” This way, I can let WiX just run the damn scripts every time without me resorting to storing a “database version” registry key or some other value that I could use as a <Condition> on a WiX component. As time goes on, these scripts can get consolidated into the same file and merged back into a “baseline schema” for the next major upgrade, so I’m not too worried about scaling here.

There is still the problem of the component that sets up the database initially trying to run on every install, however. The Installed property isn’t good enough here because that will return false when we are doing updates. So what I ended up doing is creating a new BASELINE_INSTALLED property that checks for the Version that I’ve used in the initial version of my installer.

    <Upgrade Id="{SOME-GUID}">
      <UpgradeVersion
        Minimum="1.0.0"
        IncludeMinimum="yes"
        OnlyDetect="yes"
        Property="BASELINE_INSTALLED" />
    </Upgrade>

Then the component that creates the database has an appropriate <Condition> on it:

<Component Id="ComponentSqlDatabaseCore" Guid="{SOME-GUID}">
  <Condition><![CDATA[NOT BASELINE_INSTALLED]]></Condition>
  <CreateFolder>
    <Permission
      GenericAll="yes"
      User="SQLServerMSSQLUser$[ComputerName]$[SQLINSTANCE]" />
  </CreateFolder>
  <sql:SqlDatabase
    Id="SqlDatabaseCore"
    ConfirmOverwrite="yes"
    ContinueOnError="no"
    CreateOnInstall="yes"
    CreateOnReinstall="no"
    CreateOnUninstall="no"
    Database="Armadillo"
    DropOnInstall="no"
    DropOnReinstall="no"
    DropOnUninstall="no"
    Instance="[SQLINSTANCE]"
    Server="[SQLSERVER]">
    <sql:SqlFileSpec
      Id="SqlFileSpecCore"
      Filename="[ProgramFilesFolder]Skiviez\Armadillo\Database\Armadillo.mdf"
      Name="Armadillo" />
    <sql:SqlLogFileSpec
      Id="SqlLogFileSpecCore"
      Filename="[ProgramFilesFolder]Skiviez\Armadillo\Database\Armadillo.ldf"
      Name="ArmadilloLog" />
    <sql:SqlScript
      Id="SqlScriptCreateDatabaseUsers"
      BinaryKey="BinaryKeyCreateDatabaseUsers"
      ContinueOnError="no"
      ExecuteOnInstall="yes"
      ExecuteOnReinstall="no"
      ExecuteOnUninstall="no"
      Sequence="1" />
    <sql:SqlScript
      Id="SqlScriptCreateCoreSchema"
      BinaryKey="BinaryKeyCreateCoreSchema"
      ContinueOnError="no"
      ExecuteOnInstall="yes"
      ExecuteOnReinstall="no"
      ExecuteOnUninstall="no"
      Sequence="2" />
    <sql:SqlScript
      Id="SqlScriptInsertHarmonizedTariffSystemCodes"
      BinaryKey="BinaryKeyInsertHarmonizedTariffSystemCodes"
      ContinueOnError="no"
      ExecuteOnInstall="yes"
      ExecuteOnReinstall="no"
      ExecuteOnUninstall="no"
      Sequence="3" />
  </sql:SqlDatabase>
</Component>

For the sake of completeness, here’s what the component for the patch SQL scripts looks like:

<Component Id="ComponentSqlDatabasePatches" Guid="{SOME-GUID}">
  <CreateFolder />
  <sql:SqlScript
    Id="SqlScriptAddVariationTypeNameIndex"
    BinaryKey="BinaryKeyCreateVariationTypeNameIndex"
    ContinueOnError="no"
    ExecuteOnInstall="yes"
    ExecuteOnReinstall="yes"
    ExecuteOnUninstall="no"
    Sequence="4"
    SqlDb="SqlDatabaseCore" />
</Component>

When I have new patches to add in the future, I’ll just add them to that component, using the Sequence property make sure they run in my intended order on a fresh installation.

Now, I don’t have any scripts that un-apply any of these changes on uninstall or rollback, but since this is an installer used by one person in the world (me), I’m keeping that simple for now. Obviously something for a retail application would have that functionality, although I’ve seen recommendations that database management for those would be handled in the application itself, not in a Windows Installer. Caveat emptor!

I also don’t drop the database on uninstall because that seems terrifying to me, particularly if I have a blonde moment and do a major upgrade one day without realizing it.

This could be a totally insane approach, fair warning, but I don’t know of a better way (yet). Hope this helps someone else out there. Happy installing!

So for my employer, Skiviez, I’ve created a WiX installer for the Web site. It’s pretty sweet: just double-click the MSI, and insto presto, you have a Web site installed, with the directories created, permissions set correctly, and IIS configured correctly.

The Web site uses log4net for logging, and it uses the RollingFileAppender to log information in a Logs directory for the Web site. In WiX, this behavior is specified in the usual way:

  <Directory Id="DirectoryLogs" Name="Logs">
	<Component Id="ComponentCreateFolderLogs" Guid="{DAB6113C-83BA-422d-9D77-43F112BE7EF7}">
	  <CreateFolder>
		<Permission
		  GenericRead="yes"
		  GenericWrite="no"
		  GenericExecute="no"
		  Read="yes"
		  User="Authenticated Users" />
		<Permission
		  GenericAll="yes"
		  User="NT AUTHORITY\NETWORK SERVICE" />
	  </CreateFolder>
	  <RemoveFile
		Id="RemoveFileLogsAll"
		Name="*.*"
		On="uninstall" />
	</Component>
	<Component Id="ComponentConfigWebLogs" Guid="{82B14CA7-96B1-4c92-8E6E-846C8E17ED02}">
	  <File
		Id="FileConfigWebLogs"
		Name="Web.config"
		Source="$(var.Skiviez.Armadillo.UI.Web.ProjectDir)Logs\"
		KeyPath="yes" />
	</Component>
  </Directory>

There is nothing particularly exotic going on here. I tell Windows Installer to create the Logs folder, set some permissions on it so that NETWORK SERVICE (which the Web site runs as) can read and write to the folder and so that I (as an Authenticated User can easily read it), copy in a Web.config that locks down the folder from being served by IIS, and tell Windows Installer to remove all of the log files during uninstall. I’ve written this a bazillion times before and it works fine for most programs.

But when I tried to uninstall while the Web site was still running, the uninstallation would fail because the log file was still locked by the Web site. This is because WiX’s custom action that finagles with IIS runs after the standard RemoveFiles action in Windows Installer. Some poor soul seemed to be having my exact same problem, but no answers there.

Since this is the only Web site running on the server, an easy solution is to use another built-in WiX custom action to stop IIS during uninstallation and then start it back up again afterward. This has the advantage of spinning down the ASP.NET worker process so that the lock on the file is released, RemoveFiles completes successfully, the WiX IIS custom action removes the Web site, and finally IIS is gracefully started back up again.

I’m not sure why it wasn’t obvious to me; took me a good 45 minutes to figure this out.

Just drop this in the same Component that your Web site is defined in:

<ServiceControl
  Id="ServiceControlWebSiteMain"
  Name="W3SVC"
  Start="both"
  Stop="both"
  Wait="yes" />

(Side note: Windows Installer’s architecture is completely insane. I reckon I’ll save that rant for another post. Perhaps the most amusing bit is that when uninstallation fails for some reason, such as the situation described in this post, Windows Installer will happily re-install everything back to the way it was! Ha!)

Happy uninstalling!

Let’s say that you’ve written a super neat-o C# application that uses P/Invoke into a third-party native DLL. Let’s also say that, oh, 6 times out of 10, your application crashes hard with an “attempted to write to protected memory” error when run in release mode. And let’s finally say that when you run the application under the debugger, the crash doesn’t happen.

You might think that the third-party, native DLL that’s loaded into your process is, well, crashing your process.

And you’d be right. It is crashing there, but it’s not its fault. Because you’re causing it to walk off the end of an array.

Not that I did this or anything, spending a good 45 minutes to figure out what was going wrong. In my case, when I was P/Invoking to ZP4 (which, by the way, is awesome, for various reasons that I’ll have to enumerate in another post), I declared the extern to take a StringBuilder parameter (the underlying C DLL was expecting a char*). This is all very well and good and documented as “the right type” in MSDN.

Of course, I forgot to make sure that the Capacity of the StringBuilder was big enough for what the C DLL was going to try and put in the character array that interop was building under the hood. Unlike in .NET land where a StringBuilder can just make itself bigger when it runs out of room, in P/Invoke, the interop basically does this:

• Interop looks at the Capacity of the StringBuilder and allocates a char array that many chars wide. After all, it can’t read minds to know how much the function it is about to call is going to shove in there.
• Interop passes a pointer to that unmanaged char array by value to the C function.
• (The C function does whatever it does, referencing that pointer.)
• Keeping the initial Capacity in mind, Interop looks at the unmanaged char array and marshals it back into the StringBuilder. Then it frees the unmanaged memory it had allocated for the array since it’s no longer needed.

Because I didn’t set the Capacity before all this went down, I ended up with the default Capacity of 16 characters. It just so happens that the C function I was calling expected the character to array to be 928 characters. Oops.

Since the debugger adds a lot of record-keeping information to the stack, generally padding out the size and layout of my program in memory, I was “getting lucky” in debug mode by scribbling over 912 bytes of memory that weren’t very important. Without the debugger, though, I was scribbling on top of rather important things, eventually walking outside of my own memory space, causing Interop to delete memory it didn’t own. Most of the time.

For reasons that would bore most to tears, I had to swap out a workstation at work today. This workstation had one Zebra LP2844 laser printer attached to it, and along with the “new” (well, new-to-that-particular-desk, not new-to-the-world) workstation, it was getting another.

Simple! The “new” workstation already had the Zebra LP2844 printer drivers installed, so I’ll just Start > Printers > Add Printer > Yep, LPT1: > Mmhmm, ZEBRA EPL > (scrolling … scrolling … Why is this dialog box so small? … scrolling) ah, LP2844 > Continue Anyway (Why can’t a million dollar company get their drivers signed?) > Finish, insto presto, and….

Boom. “Printer Driver was not installed. The operation could not be completed.”

Hum.

I’ll try it again. This is an indication of insanity–expecting a different result from the same operation–but it’s 7:30 a.m. and I can’t be experiencing problems already. Come on.

Boom. “Printer Driver was not installed. The operation could not be completed.”

Okay, I’ll just download Zebra’s driver setup utility and install the printer that way. Next > Next > Add a printer > Next > Next, and….

Boom. “The data area passed to a system call is too small.”

Hum. That’s a new one. But different!

I know this printer works, why is this not working?

The Solution

After about 30 minutes of head-scratching, I stumbled upon the solution. These types of errors are apparently the result of corrupted or just plain buggy printer drivers already installed on the machine. They could be unrelated to the printer that you’re trying to install; it doesn’t matter. They’ll cause the error when the “add a printer” mechanism enumerates through the list of printer drivers already installed on the machine.

To fix it, it’s time to blow away some printer drivers. Go to Printers under the Control Panel. Click File > Server Properties and switch to the Drivers property sheet. Remove any suspicious-looking drivers and try to add your printer again. (For me, the problematic driver was a custom LP2844 driver that came on a UPS WorldShip disk.)