Adobe Symbol Server: How Adobe Could Address Crash Issues
Since crash bugs are a top priority within Adobe, there is one relatively simple step Adobe should take which would make it much easier for everyone else to help Adobe track and diagnose crashes: implement a symbol server.
A symbol server is a public web server from which developers can fetch debugging information (PDB files) for released binaries. The Microsoft debuggers have excellent support for automatically pulling down symbols as they are needed in the debugger. Mozilla runs a symbol server for Firefox nightlies and releases, which is invaluable for people debugging and profiling Firefox without having to do a custom build. Microsoft runs a symbol server which contains debug information for Windows and many other Microsoft products, including the Silverlight plugin.
Debug information is not simply a way to get symbolic information from Flash. It is necessary in order to get any useful stack trace of the Mozilla code which is calling Flash. A common compiler optimization called frame pointer omission (FPO) avoids storing the frame pointer in the x86 EBP register, freeing that register up for general use. In order to walk the stack of this optimized code, the debugger has to query the frame size and frame pointer information from the PDB file. When debug information is not available, stack walking doesn’t produce usable results.
As an example, take the current #3 topcrash for nightly builds of Firefox (mozilla-central). The signature for this crash is NPSWF32.dll@0x1e7fe4. The stack traces from Mozilla’s crash reporting system are completely opaque:
| Frame | Signature |
| 0 | NPSWF32.dll@0x1e7fe4 |
| 1 | NPSWF32.dll@0x1ff471 |
| 2 | NPSWF32.dll@0x2005bd |
| 3 | NPSWF32.dll@0x1fb195 |
| 4 | NPSWF32.dll@0x1e02d1 |
| 5 | NPSWF32.dll@0x17c22a |
| 6 | NPSWF32.dll@0x2959d |
| 7 | NPSWF32.dll@0x30386 |
| 8 | @0x63aa15f |
| 9 | NPSWF32.dll@0x5bdef |
Even worse, the crash signature depends on the particular version of Flash that is installed on the user’s computer. We can’t tell if a particular crash signature is fixed by a new revision of flash because without symbols we can’t correlate crashes between different versions.
As part of developing multi-process plugins for Firefox, we are constantly dealing with unexpected plugin behaviors. Whenever we encounter a problem which can be reproduced in both Silverlight and Flash, we’ll always test with silverlight, simply because Microsoft makes Silverlight symbols available through their symbol server and therefore we can actually step through their code and ours in a debugger.
Adobe should set up a symbol server for their three main plugins, Flash, Shockwave, and Acrobat. By implementing this simple tool, Adobe could help all browser vendors and interested hackers to help identify and fix bugs. If Adobe is concerned about using full debug information to reverse-engineer details of their code, there is a way to strip the PDB files so that only frame-pointer information and function names.
February 18th, 2010 at 4:11 pm
I’ve been suggesting this in Flash crash bugs for a while now. Adobe’s people don’t seem to care.
cl
February 18th, 2010 at 4:11 pm
After searching through the Flash Player’s bug tracker and not finding a feature request for this, I’ve created one:
http://bugs.adobe.com/jira/browse/FP-3953
February 18th, 2010 at 5:11 pm
Probably worth pointing out that setting up a symbol server isn’t a technically-difficult task (AIUI), and it would have *immediate* payoffs.
I guess we’ll see if Adobe is all-talk or not.
February 18th, 2010 at 6:22 pm
Thanks, Till, for forwarding it directly.
jd/adobe
February 19th, 2010 at 2:37 pm
Also of note is that “setting up a symbol server” just involves getting the PDB files in the right directory structure, and exposing that via HTTP. We don’t use symstore.exe (which is probably the simplest way to do it, if that’s all you need), but we do just serve the resulting files via Apache.
February 19th, 2010 at 7:23 pm
Thanks for the post, Benjamin. The Flash Player team has an effort underway to create a public symbol server, though still will need to obfuscate some symbols such as for third party code. You are right — we have a lot to gain from implementing a symbol server. Thanks for bringing this issue to the surface, and we appreciate being able to work together with Mozilla to make these stack traces more usable.
Michelle Sintov
Flash Player Engineering
February 23rd, 2010 at 12:51 am
Thanks for setting that request up.
I’ve been constantly hitting a crash with the latest adobe update, and it would’ve helped if the symbols were available.
Basel/SDET, Microsoft
January 1st, 2011 at 2:28 am
Michelle, is this still being worked on?