Chart of the Day: Firefox Nightly Update Adoption Curves
In general, people who are running the Firefox Nightly and Aurora channel are offered a new build every day. But users don’t update immediately, because Firefox does not interrupt you with an update prompt upon receiving an update. Instead it waits and applies the update at the next Firefox restart, or prompts the user to update only after significant idle time.
This means that there is a noticeable “delay” between a nightly build and when people start reporting bugs or crashes against the build. It also means that the number of users using any particular nightly build can vary widely. The following charts demonstrate this variability and the update adoption curves:
Because of this variability, engineers and QA should use care when using data from nightly builds. Note the following conclusions and recommendations:
- Holidays, weekends, and other unexplained factors may mean that some nightly builds get below-average user totals.
- Users often skip nightlies: reported regression ranges should be verified.
- Reliable crash metrics will not be available for several days after a nightly build is released.
- It may be necessary to correlate crash rates on particular builds against the user counts for that build in order to accurately measure crashes-per-user.
- When multiple nightlies are built on the same day (for example, a respin for a bad regression), the user count for each build will be lower than an average nightly build.
This data was collected from ADU data provided by metrics and mirrored in the crash-stats database. The script used to collect this data is available in socorro-toolbox.
April 16th, 2013 at 11:39 am
It would be helpful if you could include the explanation of the graphs in the main text of the blog post instead of putting it only on the page which appears when you click the graph.
Axis labels would also be helpful, as would be an explanation of what the gray / white areas in the first graph mean.
April 17th, 2013 at 2:43 am
“Instead it waits and applies the update at the next Firefox restart, or prompts the user to update only after significant idle time.”
I installed a nightly from an installer. On my machine firefox does not autoupdate. It shows the upgrade dialog, but it does not download updates until I tell to do so through the update prompt. (Going to about dialog does not work unlike in zip builds).
That might explain some of the slow adoption
April 22nd, 2013 at 3:40 pm
[…] crashes per build date. But they were very noisy, partly because they didn’t account for the differences in user population between nightly […]