Asa is wrong, and he’s being obnoxious about it to boot. What Google, and Apple, and Microsoft is doing is called software integration, and in general it’s very good that iTunes, Google Earth, and Windows Live are adding Firefox integration into their software. They are installing plugins and/or extensions using the recommended methods so that all NPAPI-capable browsers can see and load them. This is not “Google being bad”, this is Google following Mozilla’s recommendations for browser integration!

It’s true that Firefox should give users more control over integration software that is found on the system, and we’re working on prompting users whether they want to include that integration as part of Firefox. But claiming that Google, Apple, and Microsoft are somehow being evil is stupid and short-sighted. The problem, if there is one, lies entirely with Firefox, not with the software which is doing exactly what we ask of them.

Since crash bugs are a top priority within Adobe, there is one relatively simple step Adobe should take which would make it much easier for everyone else to help Adobe track and diagnose crashes: implement a symbol server.

A symbol server is a public web server from which developers can fetch debugging information (PDB files) for released binaries. The Microsoft debuggers have excellent support for automatically pulling down symbols as they are needed in the debugger. Mozilla runs a symbol server for Firefox nightlies and releases, which is invaluable for people debugging and profiling Firefox without having to do a custom build. Microsoft runs a symbol server which contains debug information for Windows and many other Microsoft products, including the Silverlight plugin.

Debug information is not simply a way to get symbolic information from Flash. It is necessary in order to get any useful stack trace of the Mozilla code which is calling Flash. A common compiler optimization called frame pointer omission (FPO) avoids storing the frame pointer in the x86 EBP register, freeing that register up for general use. In order to walk the stack of this optimized code, the debugger has to query the frame size and frame pointer information from the PDB file. When debug information is not available, stack walking doesn’t produce usable results.

As an example, take the current #3 topcrash for nightly builds of Firefox (mozilla-central). The signature for this crash is NPSWF32.dll@0x1e7fe4. The stack traces from Mozilla’s crash reporting system are completely opaque:

Even worse, the crash signature depends on the particular version of Flash that is installed on the user’s computer. We can’t tell if a particular crash signature is fixed by a new revision of flash because without symbols we can’t correlate crashes between different versions.

As part of developing multi-process plugins for Firefox, we are constantly dealing with unexpected plugin behaviors. Whenever we encounter a problem which can be reproduced in both Silverlight and Flash, we’ll always test with silverlight, simply because Microsoft makes Silverlight symbols available through their symbol server and therefore we can actually step through their code and ours in a debugger.

Adobe should set up a symbol server for their three main plugins, Flash, Shockwave, and Acrobat. By implementing this simple tool, Adobe could help all browser vendors and interested hackers to help identify and fix bugs. If Adobe is concerned about using full debug information to reverse-engineer details of their code, there is a way to strip the PDB files so that only frame-pointer information and function names.

Stuart has been working on using an allocator for Mozilla which has much better performance characteristics, especially with memory fragmentation and heap growth over time. The allocator he chose is jemalloc, the default allocator for the FreeBSD libc. On Linux, intercepting and replacing malloc is fairly easy, because of the way dynamic symbol loading works. On Windows, however, it is difficult or impossible to intercept and redirect calls to malloc to a custom allocator. So instead of trying to hook to a prebuilt CRT, I spent most of today hacking the Windows C runtime (CRT), replacing the default allocator with jemalloc.

The Windows CRT sources come with VC8 professional edition so any licensed user may hack on them and redistribute the result as part of a larger program. It comes with a mostly-working nmakefile¹: I had to disable the code which builds the managed-code CRT because apparently I don’t have the right .NET headers installed. Getting the new jemalloc.c file to build wasn’t that hard, either: it required a few #defines, typedefs, and disabled warnings, but nothing serious. The hardest part about replacing It was figuring out what parts of the original CRT were heap-related and removing them correctly. It was a wild ride, but I think that I have a build of the Windows CRT that works… at least small programs like xpidl and shlibsign work.

Unfortunately, according to the EULA² I am not allowed to redistribute this modified CRT by itself. So the only way you can get it is by distributing it with a Firefox build. Also, I’m not allowed to post the patch queue which I used to develop the custom CRT, because those patches may contain copyrighted code in context. Do any of my readers know of a format that will alter a set of files according to a set of instructions without the instructions revealing the contents of the original files? I would really love it if Microsoft would release their C runtime code under a liberal open-source license… can someone suggest a good person to contact at Microsoft?

Stuart will have some builds posted soon, once a few kinks get ironed out.

For Mozilla2 we’re probably going to push the solution to an intermediary library: we will have a single allocator library which is used for both garbage-collected (managed) and explicitly allocated/freed (unmanaged) memory. We will switch back to the standard CRT, but we will try to avoid using the standard CRT allocator at all. See the “space management” thread on the tamarin-devel mailing list (December and January) for some background discussion.

1. nmake is one of the suckiest build systems on the planet. I could get better results with a .vbs.
2. From the EULA, section 3.1: …you agree: (i) except as otherwise noted in Section 2.1 (Sample Code), to distribute the Redistributables only in object code form and in conjunction with and as a part of a software application product developed by you that adds significant and primary functionality to the Redistributables…