Let’s say that you’ve written a super neat-o C# application that uses P/Invoke into a third-party native DLL. Let’s also say that, oh, 6 times out of 10, your application crashes hard with an “attempted to write to protected memory” error when run in release mode. And let’s finally say that when you run the application under the debugger, the crash doesn’t happen.
You might think that the third-party, native DLL that’s loaded into your process is, well, crashing your process.
And you’d be right. It is crashing there, but it’s not its fault. Because you’re causing it to walk off the end of an array.
Not that I did this or anything, spending a good 45 minutes to figure out what was going wrong. In my case, when I was P/Invoking to ZP4 (which, by the way, is awesome, for various reasons that I’ll have to enumerate in another post), I declared the
extern to take a
StringBuilder parameter (the underlying C DLL was expecting a
char*). This is all very well and good and documented as “the right type” in MSDN.
Of course, I forgot to make sure that the
Capacity of the
StringBuilder was big enough for what the C DLL was going to try and put in the character array that interop was building under the hood. Unlike in .NET land where a
StringBuilder can just make itself bigger when it runs out of room, in P/Invoke, the interop basically does this:
- Interop looks at the
StringBuilderand allocates a
chararray that many
chars wide. After all, it can’t read minds to know how much the function it is about to call is going to shove in there.
- Interop passes a pointer to that unmanaged
chararray by value to the C function.
- (The C function does whatever it does, referencing that pointer.)
- Keeping the initial
Capacityin mind, Interop looks at the unmanaged
chararray and marshals it back into the
StringBuilder. Then it frees the unmanaged memory it had allocated for the array since it’s no longer needed.
Because I didn’t set the
Capacity before all this went down, I ended up with the default
Capacity of 16 characters. It just so happens that the C function I was calling expected the character to array to be 928 characters. Oops.
Since the debugger adds a lot of record-keeping information to the stack, generally padding out the size and layout of my program in memory, I was “getting lucky” in debug mode by scribbling over 912 bytes of memory that weren’t very important. Without the debugger, though, I was scribbling on top of rather important things, eventually walking outside of my own memory space, causing Interop to delete memory it didn’t own. Most of the time.