Graph of the Day: Virtual and Physical Memory Starvation
Today’s graph is a scatter plot of out-of-memory crashes. It categorizes crashes according to the smallest block of available VM and the amount of available pagefile space.
There were roughly 1000 crashes due to bug 829954 between 10-April and 15-April 2013. Click on individual crash plots to see memory details and a link to the crash report.
Direct link to SVG file. Link to raw data.
Conclusions
After graphing these crashes, it seems clear that there are two distinct issues:
- Crashes which are above the blue line and to the left have free space in their page file, but we have run out of contiguous virtual memory space. This is likely caused by the virtual memory leak from last week.
- Crashes which are below the blue line and to the right have available virtual memory, but don’t have any real memory for allocation. It is likely that the computer is already thrashing pretty heavily and appears very slow to the user.
I was surprised to learn that not all of these crashes were caused by the VM leak.
The short-term solution for this issue remains the same: the Mozilla graphics engine should stop using the infallible/aborting allocator for graphics buffers. All large allocations (network and graphics buffers) should use the fallible allocator and take extra effort to be OOM-safe.
Long-term, we need Firefox to be aware of the OS memory situation and continue to work on memory-shrinking behavior when the system starts paging or running out of memory. This includes obvious behaviors like throwing away the in-memory network and graphics caches, but it may also require drastic measures such as throwing away the contents of inactive tabs and reloading them later.
Charting Technique
With this post, I am starting to use a different charting technique. Previously, I was using the Flot JS library to generate graphs. Flot makes pretty graphs (although it doesn’t support labeling Axes without a plugin!). It also features a wide range of plugins which add missing features. But often, it doesn’t do exactly what I want and I’ve had to dig deep into its guts to get things to look good. It is also cumbersome to include dynamically generated JS graphs in a blog post, and the prior graphs have been screenshots.
This time around, I generated the graph as an SVG image using the svgwrite python library. This allows me to put the full SVG graph directly into the blog, and it also allows me to dynamic features such as rollovers directly in these blog posts. Currently I’m setting up the axes and labels manually in python, but I expect that this will turn into a library pretty quickly. I experimented with svgplotlib but the installation requirements were too complex for my needs.
I’m not sure whether or not the embedded SVG will make it through feed aggregators./readers or not. Leave comments if you see weird results.
April 24th, 2013 at 5:09 pm
> Long-term, we need Firefox to be aware of the OS memory situation and continue to work on memory-shrinking
> behavior when the system starts paging or running out of memory.
The code is all there for Windows; AvailableMemoryTracker.cpp. The issue isn’t being able to notice that we’re in a low-memory situation, but being able to do something about it in a timely fashion. I spent a long time on this, and I’ve concluded that it’s at best very difficult to get right. It’s very difficult to do right even on B2G, where we have a lot more control over the system. I’ve written extensively about this on the newsgroups, because it’s a popular idea; most recently: https://groups.google.com/d/msg/mozilla.dev.platform/NOCX_tDfFCA/C6NqiuMpK2kJ
It’s easier to do something sane when we’re running out of virtual memory than physical memory, though, because other processes on the system don’t cause us to run out of vmem. OTOH I don’t think low-memory notifications are necessarily the right way to tackle this particular vmem problem, because the issue does not appear to be under Firefox’s control.
April 24th, 2013 at 5:10 pm
It would be interesting to compare this data the data from non-OOM crashes.
April 24th, 2013 at 6:27 pm
Great graph. Gee I wish Win64 was a tier-1 platform.
April 29th, 2013 at 1:01 pm
I don’t know anything about out-of-memory crashes–they sound bad!–but I do know graphing data. If you’re looking for something JS-based, the Google Chart library (https://developers.google.com/chart/interactive/docs/gallery) isn’t too bad, though it may be too simple for some of your needs. On the server side: I still have loyalty to Gnuplot (http://gnuplot.info/), which is the grand master of professional-quality open-source plotting programs, but it has its own syntax you’d have to learn. matplotlib (http://matplotlib.org/) is IMHO the gold standard for producing graphs in Python, and definitely can produce SVG (amongst many others).
Also, labeling axes is not built into the core of Flot? Sigh. Am I going to have to start whacking people over the head with Edward Tufte’s “The Visual Display of Quantitative Information?” ;)
— Matt
By-the-by, the SVG passed through fine on Google Reader and on Feedly.