Chart of the Day: Firefox Nightly Update Adoption Curves

Monday, April 15th, 2013

In general, people who are running the Firefox Nightly and Aurora channel are offered a new build every day. But users don’t update immediately, because Firefox does not interrupt you with an update prompt upon receiving an update. Instead it waits and applies the update at the next Firefox restart, or prompts the user to update only after significant idle time.

This means that there is a noticeable “delay” between a nightly build and when people start reporting bugs or crashes against the build. It also means that the number of users using any particular nightly build can vary widely. The following charts demonstrate this variability and the update adoption curves:

Per-build usage and adoption curves, Firefox nightly builds on Windows, 1-March to 14-April 2013
Overlapped adoption curves, 1-March to 14-April 2013

Because of this variability, engineers and QA should use care when using data from nightly builds. Note the following conclusions and recommendations:

  • Holidays, weekends, and other unexplained factors may mean that some nightly builds get below-average user totals.
  • Users often skip nightlies: reported regression ranges should be verified.
  • Reliable crash metrics will not be available for several days after a nightly build is released.
  • It may be necessary to correlate crash rates on particular builds against the user counts for that build in order to accurately measure crashes-per-user.
  • When multiple nightlies are built on the same day (for example, a respin for a bad regression), the user count for each build will be lower than an average nightly build.

This data was collected from ADU data provided by metrics and mirrored in the crash-stats database. The script used to collect this data is available in socorro-toolbox.

Make the Title Bar Clickable When Tabs are On Top

Thursday, March 24th, 2011

In Firefox 4, tab are “on top” by default. When you are in maximized mode, this also means that the tabs move up into the titlebar. I understand that the UI design team decided to do this to make the tabs easier to click (because of Fitts’ law), but I personally find this behavior annoying, because it means that you cannot use the title bar for its normal tasks. I almost always double-click the titlebar to switch between maximized and regular mode, or drag the window to the side to get half-screen mode.

Fortunately, there is a hidden preference which controls this behavior. You can restore normal title bar behavior in two ways:

  • Set the hidden preference “browser.tabs.drawInTitlebar” to false, using about:config
  • Install my extension Aero Window Title, which not only changes the default preference for you, but also puts the window title back onto the screen so that you can see the title of the current tab easily.

While I’m talking about it, I’d also like to plug my other new extension, Iconic Firefox Menu, which will turn the orange Firefox menu (on Windows) into a Firefox icon. Together, the two extensions look like this:

The Firefox menu is now an icon, instead of an ugly orange button. The title of the current tab is displayed in the title bar.

The Firefox Plugin Hang Detector

Wednesday, June 9th, 2010

If all goes well, Firefox 3.6.4 will be released with support for out-of-process plugins on Windows and Linux next week. As part of this project, Firefox now features a “hang detector” for plugins. The hang detector helps protect users from plugins and plugin scripts which stop responding for 10 seconds.

When Firefox makes an NPAPI call to a plugin which is being run in a separate plugin process, that call is translated into a message which is posted to the process. Firefox then waits for the response. If the plugin takes longer than 10 seconds to respond to a call, Firefox performs the following actions:

  • Stop the plugin process and collect a plugin-process “crash” minidump.
  • Collect a “crash” minidump from the browser process.
  • Terminate the plugin process.
  • Display the plugin-crashed UI, giving the user the opportunity to refresh the page and try again.

There are several reasons plugins might trigger the hang detector:

  • A plugin script (such as ActionScript run inside the Flash plugin) may be in an infinite loop or performing a very long computation.
  • The plugin itself may have a bug (such as a threading deadlock) which causes it to stop responding.
  • The plugin is not deadlocked, but is not processing events quickly enough, causing the event to linger waiting to be processed.
  • The implementation of Firefox out-of-process plugins may be causing a deadlock.

It is this last possibility that is most concerning, and we have pored over our Firefox crash stats studying the hang reports that we receive, trying to categorize the reports into one the categories above. During the long process of Firefox 3.6.4 release candidates, we have identified and fixed several “tophangs”: see bug 561817 for an example.

When hang reports were first submitted to crash-stats, it was very difficult to distinguish full crashes from hangs. It was also impossible to correlate the browser and plugin parts of a hang. Since then, the Socorro team has done an outstanding job of improving the crash-stats UI so that we can analyze hang reports. It is now possible, using the advanced query page, to search for only crash reports, only hang reports, and to limits searches to either the browser process or the plugin process. Individual hang reports are now cross-linked: see the following reports for an example:

In this particular hang-pair, it isn’t immediately clear what is causing the hang. The browser is painting, and during the process of painting a windowless Flash plugin sends the NPP_SetWindow NPAPI call. This is the hanging call. At the time the hang report was collected, the NPAPI thread of the Flash plugin is calling NPN_InvalidateRect, and inside that implementation locking a mutex. But we don’t know, looking at the stacks, whether this is a deadlock on that mutex, or whether the plugin just happened to be making that call at the time the hang detector collected its stack.

In some cases, developers may want to disable the hang detector. If you are using the Flash debugger, or if you are debugging Firefox, you can change the hang detector timeout by changing the preference dom.ipc.plugins.timeoutSecs. Setting a value of -1 disables the hang detector entirely. For more information on setting preferences, see about:config on MozillaZine.

Multi-Process Plugins on By Default

Wednesday, January 27th, 2010

Out-of-process plugins (OOPP) are now on by default in mozilla-central! Starting tomorrow morning, the mozilla-central nightly builds will load Flash and all other plugins in a separate process by default (on Windows and Linux). The Electrolysis team would love for people to test any plugins on their system, especially less-popular plugins.

Since we are moving relatively quickly with multi-process plugins, there are a few known issues to be aware of:

  • The plugin-crash UI is not finished. The current UI is just a non-localized dialog so that we can get crash reports from nightly testers. This will be changed soon!
  • On Windows, tearing/repainting issues when scrolling, bug 535295
  • On Linux, compiz effects and Flash don’t work together on some systems, bug 535612
  • On Windows, selecting “Print” option in Flash may lock up Firefox, bug 538918
  • On Windows, hulu won’t switch to full-screen mode, bug 539658
  • On Linux with GTK+-2.18 or later, GDK assertions and a fatal XError, bug 540197
  • Firefox-process crashes at NPObjWrapper_NewResolve with silverlight and sometimes Flash, bug 542263

If you discover crashes while running nightlies, please make sure you submit them, and check about:crashes for the crash ID and signature. We could use help making sure plugin-related crashes and instability are filed and tracked by searching for signatures here and filing bugs in the Core:Plug-Ins component.

If your browser hangs, you can probably recover by killing the mozilla-runtime process in the Windows task manager or via `kill` on Linux. If you are a developer with a debugger, please use the Mozilla symbol server and get stacks for both the Firefox process and the mozilla-runtime process and file a bug.

In some cases, it may be useful to the Electrolysis developers if you obtain a plugin log, which is a log of calls made between the plugin and the browser. Instructions for obtaining the log are available here.

I am very excited that we’ve made it this far, and I look forward to our next milestone release, which will backport these changes to the 1.9.2 release in preparation for Firefox Lorentz.

Flash in a separate process

If for some reason you need to disable multi-process plugins, set the pref dom.ipc.plugins.enabled to false.

Firefox 3.6

Thursday, January 21st, 2010

We released Firefox 3.6 today. If you are currently running Firefox, choose “Check for Updates” from the Help menu. If you aren’t, go get Firefox 3.6 now! One of our most popular new features is Personas, which you can use to style Firefox the way you want. We’ve also made Firefox faster, more responsive, and more secure than ever.

Mousewheel Zoom Eureka!

Wednesday, October 28th, 2009

In Firefox, you can make the page text larger and smaller by holding down the Control key and rolling your mouse scroll wheel up and down. Before continuing to read, I want you to think about which direction makes more sense: should scrolling down make the page get bigger or smaller?


How to disable the Comodo reseller root certificate in Firefox

Wednesday, December 24th, 2008

Slashdot is a-buzz (and rightly so!) with news that people have been able to obtain an SSL certificate for a domain they don’t own, by applying with one of Comodo’s certificate resellers. It is clear that there has been a major breach of trust, but we’re not sure of the best general solution. There has been a discussion in the newsgroup about what steps Mozilla should take for this breach.

In the meantime, I recommend disabling the root certificate used by this certificate authority, to avoid the possibility that other fraudulent certificates are floating around in the wild. Here’s how to disable the relevant CA root in Firefox:

  1. Open the preferences window
  2. Select the “Advanced” tab
  3. Select the “Encryption” sub-tab
  4. Choose “View Certificates”
    Firefox Preferences Window: Advanced -> Encryption -> Certificates

  5. Find and select the “AddTrust AB / AddTrust External CA Root” item
  6. Choose the “Edit’ button
    Root Certificates Dialog

  7. Remove all trust setting check-boxes.
    Edit Certificate Dialog

Note: disabling this root certificate will SSL websites validated by this Comodo reseller to stop working. That’s why you’re doing it, but if it’s your favorite website that stops working, please don’t blame me! If you’re really paranoid, you could also disable all Comodo roots: these include all the certificates with names like “AddTrust”, “Comodo CA Limited”, and “The UserTrust Network”.

Thanks to Eddy Nigg for first providing these instructions.

Firefox 1.0.7!?

Saturday, December 8th, 2007

I used my father-in-law’s laptop recently. I had installed Firefox on it for him when he bought it, and I was happy to see it was still the default browser. I thought it was a little odd that it opened new windows by default instead of new tabs, and then I had a terrible suspicion and checked “Help -> About Firefox” and discovered to some dismay that he was still running Firefox 1.0.7.

Needless to say I installed an up-to-date version immediately. I wonder how many other people might have no clue that their applications are incredibly out of date. I also wonder why his security software (virus scanner/firewall) wouldn’t warn him about such an important aspect of system security.