Today Joe Drew, Olli Pettay, and I have gotten Mobile Firefox (Fennec) working with a separate process for rendering. It’s a significant achievement, because even though we had to hack out some Fennec features, it’s already a fairly functional browser. Olli made a screencast showing the browser in action:

Getting Fennec working was difficult partly because the mobile Firefox code uses a different drawing system: instead of displaying a native scrollable widget, the mobile code uses a cache of “tiles” to display the web page. This allows them to display certain kinds of content over the web page, as well has have better control and speed when scrolling, zooming, and performing other interactions.

In order to get all this working with multiple processes, the group attacked pieces of the problem separately. Joe Drew implemented a new method on the canvas element: asyncDrawXULElement. This call, very similar to drawWindow, will asynchronously ask the content process to draw a tile (or part of a tile).

Olli implement various interaction fixes: forwarding mouse events from the tiles to the content process, forwarding some important events such as MozAfterPaint from the content process back to the chrome process, and fixing widget focus in the embedded browser so that keystrokes are sent to it correctly.

Finally, I modified the Mozilla frame loader and subdocument frame such that “remote frames” could work correctly even without a docshell. I then hacked up the Fennec sources so that it would also work without a docshell, mainly by commenting out the security UI and zoom-to-element features which require additional information from the content process.

Now that it’s working, we hope to be able to bring additional developers in to fix up the features which we hacked around, fix DOM features which are currently broken such as link targeting, and start getting much better measurements for interactive performance and memory usage.

In Firefox, you can make the page text larger and smaller by holding down the Control key and rolling your mouse scroll wheel up and down. Before continuing to read, I want you to think about which direction makes more sense: should scrolling down make the page get bigger or smaller?

(more…)

IPDL is the language that Mozilla is using to describe all the messages between processes. Invented by Chris Jones, the IPDL language makes it easier for us to write type-safe and secure code by generating a lot of the basic validation code involved with messages.

When Chris was at Mozilla Headquarters a few weeks ago, he presented a tech-talk on IPDL which has been recorded and is available for download or viewing:

In the presentation, Chris explains the motivations for IPDL, demonstrates basic usage, and answers many questions about the limitations and benefits of using IPDL. I encourage anyone who is interested in Mozilla’s Multi-Process work to watch the presentation.

There are also some IPDL protocols in the tree if you’d like to read them: see the dom/ipc directory in the Electrolysis branch.

When Taras Glek stepped up and made major improvements to the performance of libjar, we couldn’t resist making him the module owner. libjar hasn’t had an official owner for as long as I can remember. I would occasionally do reviews, as would the networking owners (biesi/bz), but the code was really orphaned code that was unmaintained. Taras is already changing that in a big way. In a quick IRC convention with Brendan today, Taras became the new owner of the JAR code. Congratulations Taras!

Mozilla, as an organization and product, has made implementing and developing open standards a core part of its mission from the beginning. But what is so important about standards?

I was listening to a presentation by Mitchell the other day, and one phrase in particular stood out: “Standards compliance is important because it ensures the portability of my data”. We don’t expend huge amounts of time and effort on open standards and standards compliance because of marketplace pressure. Open standards are a guarantee that the documents and data that our users are reading and writing today will continue to be available to us permanently in the future. Open standards are an essential service to the users of Firefox and to the web as a whole.

This is why Mozilla has spent so much effort implementing open video and the <video> element in HTML. And this is why open video is fundamentally better for users than Flash video or even patent-encumbered formats such as H.264.

Please welcome Todd Whiteman as the new owner of PyXPCOM.

PyXPCOM is a bridge between python and Mozilla XPCOM which allows app authors to write chrome script and XPCOM components in python. For a long time its source code has been in the main Mozilla codebase (CVS and then mozilla-central) in extensions/python. It is used by some fairly large projects such as the Komodo editor as well as the Sugar project.

Unfortunately, nobody ever created a bugzilla component for tracking PyXPCOM, and PyXPCOM bugs would languish in “Core: General” or “Core: XPCOM” unnoticed. As I was going through old bugs in XPCOM, I noticed the languishing PyXPCOM bugs. I asked Mark Hammond, the original PyXPCOM author and maintainer, whether he still had time to maintain the module. He did not, but he nominated Todd Whiteman (ActiveState) as his replacement.

In order to make PyXPCOM maintenance easier, Todd and I arranged for the following changes to PyXPCOM:

• The PyXPCOM code now lives at http://hg.mozilla.org/pyxpcom. It has its own configure script which builds against the XULRunner SDK. It is no longer necessary to build all of XULRunner or Firefox in order to build PyXPCOM. The PyXPCOM code in mozilla-central has been removed.
• A new bug component: PyXPCOM bugs now have their own bugzilla component in the “Other Applications” product:
  • View all open PyXPCOM bugs
  • File new PyXPCOM bug

Todd has nominated the following peers for PyXPCOM:

• Mark Hammond
• Tomeu Vizoso
• Shane Caraveo
• Trent Mick

Patches to PyXPCOM should be reviewed by Todd or one of the module peers. (Note that not all the peers appear on the module ownership page because of technical issues with the tool generating that page.)

I’m happy that PyXPCOM has active ownership and can continue to grow and serve the needs of its community.

Yesterday a group of developers were sifting through crash reports that might be related to the new JIT code in Firefox 3.5. We found an unusual set of crashes with EXCEPTION_ILLEGAL_INSTRUCTION. After sifting through these crashes, I found that they have very similar patterns:

• The crashes are all on old processors: the original Pentium, the original AMD K6 (not K6-II or K6-III), or a Via processor. See bug 500277 for the specific family/model/stepping numbers that appear to be affected.
• The crashes all have very low uptime: the browser appears to crash on startup.

Anyone with an original AMD K6 running Linux or Windows, could you please try launching Firefox 3.5 and browsing around the web with it? Please post your results in bug 500277. If you are willing to spend some time diagnosing the problem in a debugger, please email me or join #jsapi on IRC.

For a long while now (even before Google Chrome was announced), Mozilla has been examining ways to make Firefox better by splitting the work of displaying web pages up among multiple processes. There are several possible benefits of using multiple processes:

• Increased stability: if a plugin or webpage tries to use all the processor, memory, or even crashes, a process can isolate that bad behavior from the rest of the browser.
• Performance: By splitting work up among multiple processes, the browser can make use of multiple processor cores available on modern desktop computers and the next generation of mobile processors. The user interface can also be more responsive because it doesn’t need to block on long-running web page activities.
• Security: If the operating system can run a process with lower privileges, the browser can isolate web pages from the rest of the computer, making it harder for attackers to infect a computer.

Now that we’re basically done with Firefox 3.5 we’ve formed a project team. We’re calling the project “Electrolysis”. Because we can’t do everything at once, we are currently focusing on performance and stability; using a security sandbox will be implemented after the initial release. Details of the plan are available on the Mozilla wiki, but the outline is simple:

1. Sprint as fast as possible to get basic code working, running simple testcase plugins and content tabs in a separate process.
2. Fix the brokenness introduced in step one: shared networking, document navigation and link targeting, context menus and other UI functions, focus, drag and drop, and probably many other aspects of the code will need modifications. Many of these tasks can be performed in parallel by multiple people.
3. Profile for performance, and fix extension compatibility to the extent possible.
4. Ship!

We’re currently in the middle of stage one: Ben Turner and Chris Jones have borrowed the IPC message-passing and setup code from Chromium. We even have some very simple plugins loading across the process boundary! Most of the team is in Mountain View this week and we’re sprinting to see if we can implement a very basic tab in a separate process today and tomorrow.

For the moment we’re focusing on Windows and Linux, because the team is most familiar and comfortable on these environments. I sat down with Josh Aas on Friday and we discussed some of the unknowns/difficulties faced on mac. As soon as our initial sprint produces working code we’d love to have help from interested mac hackers!

If you’re interested in helping, or just lurking to see what’s going on, the Electrolysis team is using the #content channel on IRC and the mozilla.dev.tech.dom newsgroup for technical discussions and progress updates. We’ll also cross-post important status updates to mozilla.dev.platform.

If you’ve emailed me volunteering to help and I haven’t gotten back to you, I apologize! Until we get the stage-one sprint done there aren’t really any self-contained tasks which can be done in parallel.

Things I’ve learned recently:

• Using hg log on a file that was removed will not list the revision in which the file was removed. You want hg log --removed.
• Waist Watcher sodas are sweetened with Splenda, but don’t have the metallic taste that some other diet sodas do. I especially like the Citrus Frost (grapefruit) flavor. It’s like Fresca without the hidden Aspartame. (I have bad digestive reactions to Apartame.)
• Linking libxul on my Linux machine takes between 3 and 10 seconds, but apparently this is unusual. Several other people have reported link times that can range into the minutes. I recommend buying as much RAM as you can: if your entire object directory fits in the filesystem memory cache, build speed is much faster.
• When Microsoft Visual C++ mangles function symbols, the return type is encoded in the mangled name. When GCC mangles names, the return type is not encoded:
  

  	GCC	MSVC
  int testfunc(int)	_Z8testfunci	?testfunc@@YAHH@Z
  void testfunc(int)	_Z8testfunci	?testfunc@@YAXH@Z

  This means that separate functions in different translation units may not differ only by return type. We found this trying to combine the Chromium IPC code with Mozilla: const std::string& EmptyString() and const nsAFlatString& EmptyString() differ only by return type. On Windows this links correctly, but on Linux this causes multiple-definition errors.

pymake news:

• Bad news: pymake is still 5x slower than GNU make on Linux/Mac.
• Good news: pymake is 25% faster than msys make (GNU make on Windows)!
• Best news: there’s a lot of room to make performance better.

All measurements are do-nothing depend builds. Full rebuilds aren’t significantly affected because compiler speed overwhelms any time we spend in make.

Creating Windows processes is more expensive than creating processes on a unix-like operating system. Creating MSYS processes is hugely more expensive. Windows I/O in general is slow compared to Linux, at least for typical build tasks. Because pymake recurses in a single process, caches parsed makefiles such as rules.mk, and avoids many shell invocations, it can make up for slow parsing times by dramatically reducing time spent elsewhere.

How to use pymake on Windows

Don’t use pymake with client.mk on Windows, yet. pymake doesn’t understand MSYS-style paths, which is what configure substitutes for @srcdir@ and @topsrcdir@ when using client.mk. This will be fixed by the patches available from this bug tree.

Configuring manually isn’t hard: to build Firefox in c:/builds, follow this recipe:

$ mkdir /c/builds
$ hg clone http://hg.mozilla.org/mozilla-central /c/builds/mozilla-central
$ cd /c/builds/mozilla-central
$ autoconf-2.13 && (cd js/src && autoconf-2.13)
$ mkdir ff-debug
$ cd ff-debug
$ export MAKE='python -O c:/builds/mozilla-central/build/pymake/make.py'
$ ../configure --enable-application=browser --enable-debug --disable-optimize
$ python -O ../build/pymake/make.py -j4

How to use pymake on Linux/Mac

Configure manually as above, or add the following flags to your mozconfig file:

export MAKE="python -O $topsrcdir/build/pymake/make.py"
mk_add_options MAKE="python -O @TOPSRCDIR@/build/pymake/make.py"

Soon on all platforms this will be as simple as mk_add_options MOZ_ENABLE_PYMAKE=1

Thank you!

Special thanks to Arpad Borsos who wrote tests and an implementation of –keep-going for pymake.

Next plans

Immediate future plans for pymake reduce the process count even further, especially for depend builds:

Currently every invocation of nsinstall is a separate process, and we invoke nsinstall even when all its install targets are up to date. Simple tasks like this will instead be implemented as native python commands. Ted implemented a branch to do this, but the current implementation blocks the only thread. I think we’re going to switch and use shared-nothing threads and message passing to parallelize before making this the default behavior.

Every time Mozilla processes a makefile the build system combines all the compiler-generated dependencies into a single .all.pp file using mddepend.pl: this allows developers to move or remove header files without breaking depend builds. Running a perl script for every makefile invocation is silly, especially because all it does is parsing and rewrite makefile syntax. I will have pymake read these dependency files directly and ignore missing files (causing a rebuild without an error) using a syntax includedeps $(INCLUDEFILES)

Longer-term work that would make pymake much more useful:

• Build an object graph of the entire Mozilla tree recursively. I think I know how to do this, although there will be some issues with how to deal with local versus global variables.
• Warn and eventually force a more rigorous dependency graph: warn if a dependent file ‘appears’ without having a rule to create it.
• Make parsing a lot faster using mx.TextTools instead of native python regular expressions. Keep the regular expressions as a slow path for developers who don’t have TextTools installed.

Python Reference Cycles and Closures

While debugging pymake performance and memory usage I found an interesting fact, which in hind sight should have been obvious: functions which enclose themself in python create reference cycles which have to be cleaned up by the Python garbage collector:

def outerFunction(outerCallback):
  targetsToBuild = [1, 2, 3]
  def innerCallback():
    if len(targetsToBuild):
      # innerCallback closes on itself... this creates a reference cycle every time you call outerFunction
      # if you call outerFunction 100000 times per build, this can add up really quickly and cause large GC pauses
      targetsToBuild.pop(0).build(innerCallback)
    else:
      outerCallback()

After finding this problem, I refactored (1, 2, 3) the pymake code to use objects instead of closures to save asynchronous state while rebuilding. Also, OptionParser instances create cycles by default. There is a lightly-documented method OptionParser.destroy which can be used to manually break these cycles (thanks to Ted for finding it). pymake now runs without creating any reference cycles and I disabled the python garbage collector.

Environment Munging in MSYS

When MSYS goes from an MSYS process to a Windows process, and vice-versa, it munges certain environment variables to account for the path styles. I previously thought that it only munged PATH, but I discovered today that I was wrong: MSYS was munging the MAKEFLAGS environment variable in odd ways.

If MAKEFLAGS in the MSYS process was ‘ -j1 — PATH=e:/builds/mozilla-central’ it would be munged into ‘ -j1 — PATH=e;c:/mozilla-build/msys/builds/mozilla-central’ in a non-MSYS process. Without the leading space the value was not touched. I don’t know why this is, but I altered the pymake code slightly so that MAKEFLAGS would never start with a space (and would be more compatible with gmake in the process).