Graph of the Day: Old Flash Versions and Blocklist Effectiveness
Today’s graph charts the percentage of Firefox users who have known-insecure versions of Flash. It also allows us to visually see the impact of various plugin blocks that have been staged over the past few months.
We are gradually rolling out blocks for more and more versions of Flash. In order to make sure that the blocklist was not causing significant user pain, we started out with the oldest versions of Flash that have the fewest users. We have since been expanding the block to include more recent versions of Flash that are still insecure. We hope to extend these blocks to all insecure versions of Flash in the next few months.
From the data, we see that users on very old versions of Flash (Flash 10.2 and earlier) are not changing their behavior because of the blocklist. This either means that the users never see Flash content, or that they always click through the warning. It is also possible that they attempted to upgrade but for some reason are unable.
Users with slightly newer versions seem more likely to upgrade. Over about a month, almost half of the users who had insecure versions of Flash 10.3-11.2 have upgraded.
Finally, it is interesting that these percentages drop down on the weekends. This indicates that work or school computers are more likely to have insecure versions of Flash than home computers. Because there are well-known exploits for all of these Flash versions, this represents a significant risk to organizations who are not keeping up with security updates!
View the chart in HTML version and the raw data. This data was brought to you by Telemetry, and so the standard cautions apply: telemetry is an opt-in sample on the beta/release channels, and may under-represent certain populations, especially enterprise deployments which may lock telemetry off by default. This data represents Windows users only, because we just recently started collecting Flash version information on Mac, and the Linux Flash player doesn’t expose its version at all.
Raw aggregates for Flash usage can be found in my dated directories on crash-analysis.mozilla.com, for example yesterday’s aggregate counts. You are welcome to scrape this data if you want to play with it; I am also willing to provide interested researchers with additional data dumps on request.
April 19th, 2013 at 12:43 pm
I’d need to do more formal regression analysis to be sure, but offhand, I think your conclusion is slightly off. The red line (11.*) already had a fairly steep obsolescence going on before the blocklist deployment. At a rough estimate, had he blocklist not been deployed, it probably would be at ~5% right now. So the block list only caused about 2% of the drop, effective mostly in the first two weeks. The before data for the yellow line (10.2) is too noisy and small to read effectively, but it looks like it dropped by at least a 0.5% if not a full 1% as a result of the blocklist. The blue line shows a weak ~3 week acceleration which comes out again to 2/3-1% drop.
In short, it looks like there is a fairly consistent drop-off of about 1/3 the remaining users when the blocklist is deployed.
April 19th, 2013 at 1:35 pm
It’s great that you did this analysis, but it seems that something is missing from the telemetry data. Surely it’s worth measuring the number of clickthroughs to help distinguish which users just ignore the warning.
Re: weekend drops, unless we normalize against the total volume of traffic I don’t think we can conclude that the proportion of insecure Flash usage drops on weekends, since overall traffic drops on weekends.
April 19th, 2013 at 1:58 pm
Monica, note that I’m not graphing total volume here, I’m graphing the *percent* of total volume for each day. So variations in total volume should not matter for this analysis.
I do agree that it would be interesting to see how many people actually chose to activate Flash.
April 19th, 2013 at 4:56 pm
The redline seems to only list 11.0 to 11.2. Would it be possible to add in a line for non-block listed versions (11.3 and up)? That may inform the overall upgrade effectiveness.
April 19th, 2013 at 4:59 pm
William, what exactly do you want it to show? https://crash-analysis.mozilla.com/bsmedberg/flash-distribution.html has a more general view of update rates, but unless I add another kind of slice, it’s difficult to construct a graph that isn’t just “the inverse of the data already presented”.
April 19th, 2013 at 7:52 pm
Another possibility for the missing upgrades may btw be that these are on operating systems, for which Adobe no longer supports upgrades, like for example Solaris.
April 20th, 2013 at 5:23 pm
Curt, as noted in the article, this data represents Windows only.
April 21st, 2013 at 2:07 pm
Another possibility.
I have installed Firefox, and numerous other software, as the Domain Admin for a company.
Unfortunately, even though I would *like* various software, including Firefox and Thunderbird, to auto-update, they don’t.
They (Flash/Firefox/etc. updaters) prompt for the admin password, so my users simply ignore the update request (or quietly sigh when told to update, the program believes it is but fails and reverts to the older version).
It may be that although the end-user would like to, and the admin would like end-users to, the OS-level security restriction conspire against it.
Anand