As I’ve mentioned before, for Firefox 3 we are planning on replacing the old and crusty talkback crash reporting system with a shiny new system based on Google Breakpad (formerly called Airbag). We have a set of milestones and a schedule, and a great team working on client and server pieces.

The Gran Paradiso 1.9a3 release has support for breakpad crash reporting on Windows. It is off by default because we don’t have a production server configured yet, but we would be happy for people to test using Ted‘s development server. To submit crashes to the server, follow this procedure:

• Download Gran Paradiso Setup Alpha 3.exe
• Disable talkback. You can do this either by choosing not to install talkback in the installer, or by disabling it in the Add-ons manager.
• Set MOZ_AIRBAG=1 in your environment. To make this a permanent setting, go to My Computer -> Properties -> Advanced -> Environment Variables.
• Run Firefox. Crash. You can crash 1.9a3 reliably by following this link.
• Unfortunately, the client doesn’t keep track of a crash report ID yet. You’ll have to find the report you submitted by date/time.

WARNINGS: Because this is Ted’s development server it may disappear at any time, or your crash data might disappear whenever he feels like it. Secondly, it is possible for passwords and other data to end up in minidump files. If you are paranoid about HTTP snooping (or you don’t trust Ted), don’t send crash reports.

NOTE: You could follow this procedure with any nightly build, and it would submit the report successfully. But since Ted doesn’t have symbol data for every nightly build, you wouldn’t get much useful information.

There is at least one thing this system already does better than talkback: it provides symbolic information for Windows system DLLs. Ted has uploaded the symbol information for Window XP SP2 and the VC8 CRT. In the future we can upload symbol information for other versions of Windows. For an example of a stack that walks through the system and the CRT, see this report from a memory-corruption bug.

Starting next week the Mozilla build system is going to require python on all platforms. This will allow us to gradually convert various build scripts which are currently written in perl, and start hacking on an autoconf replacement written in python. Any python 2.3 or higher will be acceptable.

In addition, I have made a production release of MozillaBuild, version 1.0. The MozillaBuild package is now the recommended build environment for all Mozilla developers on Windows. There were a few bugs with the 1.0 releases that have now been corrected in a MozillaBuild 1.1 release candidate.

After the Tinderboxes have been upgraded to use the MozillaBuild package on trunk, I am going to be discontinuing support for the cygwin build environment.

Please post questions and followups to the mozilla.dev.builds newsgroup.

This post is going to be a bunch of little things that irritate me about various software that I use. They’re not quite annoying enough for me to learn AppleScript/Songbird extensions/whatever and fix them myself, but I’m hoping that my readers will have solutions for at least some of these issues.

Unmute my computer every morning.

In the evenings I may turn down the sound or mute my MacBook Pro, to avoid annoying the kids or whatever. Every morning at 7AM (or whenever I wake my computer after 7am), I want the sound back to full volume and unmuted.

Play pieces randomly, not tracks.

I would like my music player to understand that the six movements of a Rheinberger Mass belong together, when playing in random mode. The cognitive dissonance of playing the Sanctus and then skipping to “She was a Hotel Detective” is painful. I’m happy to manually group the tracks when I first import them. Note: I’m currently using iTunes, but I don’t mind switching.

Disable music with words while writing

I would like my music player to avoid playing any music with words while I’m writing. I just can’t verbally multitask like that. I’m happy to tell the player which music has words, and tell it when I’m writing.

My feed reader should coalesce duplicate postings

Frequently I will get the same blog posting via Planet Mozilla, an individual blog feed, and even Google blogsearch feeds. I really desperately want my feed reader to know that these are the same post and display it once in my inbox, instead of multiple times.

Ted Mielczarek (luser on IRC) has been acting as a build-config peer for a while now, and I have been derelict and haven’t updated despot or made an announcement. Ted has been doing some great work integrating Breakpad (formerly airbag) into the toolkit, as well as learning and helping with the tangled web that is our build system. He pretty much single-handedly fixed our build system to support embedding CRT manifests for VC8.

It seems that whenever I challenge him to do something, he comes right back at me with a solution. For instance, I was musing on IRC how nice it would be to have cvsgraph produce SVG instead of a huge image. Four hours later, behold (the code is a wild-ass ugly hack, but it works)!

Belated welcome to Ted!

Suzanne and I are pleased to announce the birth of Micah Wilkes Smedberg, born 3 February, 2007 at 1:34 p.m. (EST). All are happy and healthy.

This is our fourth child, but our first boy. For those interested, take a look at Ellie, Claire, and Abbey‘s birth announcements.

I will be taking two weeks of paternity leave. While I may not be able to stay away from email entirely, I won’t be actively engaged in Mozilla activities. Here are some alternate contact points while I am away:

Build system

Ted Mielczarek (luser) or Mark Mentovai (mento)

Firefox partner builds

Dan Mills (v_thunder)

Toolkit

Gavin Sharp (gavin) or Mike Connor (mconnor)

Unit testing

Robert Campbell (robcee) or Robert Sayrer (sayrer)

Embedding/docshell

Boris Zbarsky (bz)

What VCS would you rather have?

The Tortoise: Bazaar (bzr)

A version control system, which appears to have a careful and well-planned development effort. The latest release is numbered 0.14. The import from Mozilla CVS is working so far; unfortunately, at the current import rate it will take more than a month to complete. And we have concerns about it’s performance even after import is complete.

The Hare: Mercurial (Hg)

A version control system, which appears to have a fast-and-loose development style. The latest release is numbered 0.9.3. If it would run to completion, importing Mozilla CVS would take a couple days. Unfortunately, every time we’ve tried to do an import we’ve run into bugs or undocumented features in the import tool, or odd edge-cases in the Mozilla CVS tree.

I have been helping preed bring Mozilla into the world of distributed version control systems. It sucks.

Learning It All Over

At least for me this is a very uncomfortable experience. I’m not going to pretend I like CVS, but at least the usage model for CVS is straightforward and fairly consistent. For my projects I’ve been using SVN without problems. Using a distributed VCS is a mind-bending exercise in which familiar terms like “repository”, “branch”, and “merge” no longer mean what they meant.

The documentation for these tools is in general quite painful. Partly this is because of terminology, but I have discovered that a lot of the confusion is that there is not a single usage model. There may be an official repository which hackers “push” to (a CVS-like model). There may be a central repository maintained by the project owner who “pulls” changes from others. There may not be an official repository at all, with a bunch of mostly-equal peers.

I think that I have a general understanding of how this is supposed to work, and I have a couple projects in Mercurial trees now. But I haven’t had to resolve merge conflicts yet, so I don’t feel that I’ve done more than scratch the surface of how these tools are supposed to be used.

Importing Mozilla CVS

Importing Mozilla’s CVS repository is a large task. The entire Mozilla CVS repository, with all its branches, has over 1 million file revisions which can be represented as about 200,000 change sets.

One of the things we decided early on is that Mozilla wanted a prepackaged solution as much as possible. Mozilla developers’ collective expertise is writing a web browser, not hacking version control. Any time that developers have to spend patching/hacking their VCS is collective waste. We are willing to hire outside experts to solve problems if necessary, and would very much like to do so. Of course, before hiring somebody you have to pick a system.

Unfortunately, neither of the two candidates has reached a 1.0 release yet, and the tools which import a CVS repository into the new system are even less complete. The two candidate VCS systems seem to have very different design philosophies, which show up quickly when performing import operations.

The Mercurial->CVS importer is very hacky code. The import process proceeds fairly quickly, but every time we tried it we ran into errors. We have yet to do a complete import that contains anything but the trunk:

Random CVS commit message character sets (patchset 565)

The commit message in CVS are all sorts of character sets. Initially the importer assumed that they were UTF8 and failed when it encountered invalid UTF8. I discovered a hidden HGENCODING environment variable which allowed the import to proceed by ignoring unknown character sets.

CVS backbranching (patchset 5155)

In CVS, branching is not an atomic operation, and there are several old branches that were performed at multiple times. The importer did not know how to handle this situation. Solution: don’t import those branches.

Commit messages containing cvsps-like output (patchset 47070)

Several CVS commit messages contained output that could be parsed as cvsps output. The cvsps parser contained the hg-cvs-import tool is terrible. Solution: I replaced the parser with the cvsps parser code contained in bzr.

AssertionError: failed to remove webshell/embed/ActiveX/tests/vbrowse/VBrowse.vbp from manifest (patchset 61380)

This one is still undiagnosed.

Bazaar’s CVS importer, on the other hand, feels like a carefully designed tool, with modules and unit tests. Unlike the Hg importer, the bzr->CVS importer has dealt with all odd input successfully. However, importing the trunk and all branches looks like it will take more than a month on a fast machine.

I like the fact that Bazaar focuses on correctness first and then deal with performance. Bazaar provides several desirable features, such as partial pull and a SVN frontend. But, until we can actually complete an import from CVS we don’t know whether bzr will perform acceptably. We have asked the bzr developers for assistance; we’ll see what happens.

To help speed the process, I created a tool to post-process cvsps output and limit it to branches which are still actively being developed. This should help reduce import times somewhat, as well as solving the backbranching issue which broke the Hg importer.

Other Issues

In addition to the import issues, there are other unsolved problems with deploying a distributed VCS system. We need to be able to control commit (aka “push”) access to the repository using our existing LDAP/ssh infrastructure. In addition, we want to be able to log who committed which changesets to the main repository: because a user can push other people’s commits, this is not as easy as it sounds, and I haven’t found a builtin log which saves this information.

Conclusion

Currently, we don’t have enough information to choose between Mercurial and Bazaar for a VCS, because we have not been able to complete a CVS import of either system. Right now we’re leaning heavily toward Hg, because of speed issues and because we have managed to get a trunk-only import kinda working. But the entire process is much more complicated than any of us would like, and is already turning into a major time-sucking adventure. Hopefully we can pick a system soon, and contract out the remaining work to developers who have done this before, or at least know the tools extremely well.

I have a version of MozillaBuild ready for widespread testing, packaged as an installer. I’m calling this RC1 because I don’t know of any bugs and unless we find some it is fine to put into production. It works with MSVC6, 7, 8, and MSVC 8 Express with an SDK. It builds the trunk, 1.8, and 1.8.0 branches.

Download MozillaBuildSetup.exe now

The installer was built using the patch in bug 366823, which hasn’t been reviewed or landed yet.

Please download and test this package (and start using it as your Windows development environment). If you have issues, please file bugs in Core->Build Config.

Changes From the Previous Release

• Reset CYGWIN and PATH environment variables so as not to pick up any VC paths from the environment.
• Support MSVC8 Express Edition.
• Use the atlthunk_compat code to support the ATL headers provided with the platform SDK.
• Set CVS_RSH=ssh automatically.
• Add an installer.

In addition, I added a script which converts a cygwin source tree with CRLF line endings to an MSYS tree using LF line endings, and a client.mk check to warn users of problems with line endings.

After playing with the code in my last post, and asking some apparently silly questions about the ES4 spec, I’ve found some techniques which can be used to implement python-style “import” using current JavaScript.

References to Bound Functions

In python, it is possible to hold a reference to an unbound method, or to a method which is bound to a particular object:

>>>class Foo:
...  def dumpMe(self, arg):
...    print "self: %s\narg: %s" % (self, arg)
...
>>> Foo.dumpMe
<unbound method Foo.dumpMe>
>>> Foo().dumpMe
<bound method Foo.dumpMe of <__main__.Foo instance at 0x2aaaaab33680>>
>>> Foo.dumpMe("something")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: unbound method dumpMe() must be called with Foo instance as first argument (got str instance instead)
>>> Foo().dumpMe("something")
self: <__main__.Foo instance at 0x2aaaaab33d40>
arg: something

The JavaScript language does not have a built-in syntax to obtain a bound method:

js> function Foo() { }
js> Foo.prototype.dumpMe = function(arg) {
  dump("this: " + this + "\narg: " + arg);
}
js> new Foo().dumpMe
function (arg) {
    print("this: " + this + "\narg: " + arg);
}
js> f = new Foo();
js> f.dumpMe("test")
this: [object Object]
arg: test
js> unbound = new Foo().dumpMe
js> unbound("test")
this: [object BackstagePass @ 0x6214e0 (native @ 0x513fc8)]
arg: test

(The BackstagePass object is the global object in xpcshell).

However, with a little extra code it is possible to emulate bound functions in JavaScript:

js> Function.prototype.bind = function(object) {
  var func = this;
  return function() { return func.apply(object, arguments); }
}
js> boundFunc = f.dumpMe.bind(f);
js> boundFunc("test")
this: [object Object]
arg: test

We can then use bound functions to emulate the from module import x, y, x statement of python:

js> function jsimport(module) {
  // this function has two forms:
  // jsimport(module) is equivalent to python "from module import *"
  // jsimport(module, "name" [, "name2"]) is equivalent to python "from module import name, name2"

  // NOTE: "this" is the global object
  function internal_import(name) {
    if (module[name] instanceof Function) {
      this[name] = module[name].bind(module);
    }
    else {
      this[name] = module[name];
    }
  }

  if (arguments.length == 1) {
    for (name in module) {
      if (module.hasOwnProperty(name)) {
        internal_import(name);
      }
    }
  }
  else {
    for (i = 1; i < arguments.length; ++i) {
      internal_import(arguments[i]);
    }
  }
}

You could use this technique today to hide away the IOService or PrefService a little:

jsimport(Components.classes["@mozilla.org/preferences/pref-service;1"].getService(Components.interfaces.nsIPrefBranch),
         "getCharPref", "getIntPref", "getBoolPref");

if (getBoolPref("dom.window.dump.enabled"))
  // do something dumpy

As part of MozillaBuild, I’ve been testing various combinations of compilers and SDKs. It turns out that it is possible to build Mozilla (trunk) using the Free Microsoft Visual C++ Express download with an appropriate SDK and a couple hacks. I have a bug to make MozillaBuild support the free compiler, with a patch that mostly-works.

The ATL headers have a special instruction which requires a library atlthunk.lib, which doesn’t ship with the SDK. The Mozilla Build FAQ has an ingenious solution to this problem which requires editing the SDK headers. Unfortunately, patching other people’s software is not a good idea in general. I’ve found another way around this problem by creating a replacement atlthunk.lib library which contains the two required functions. Source code is currently here. MSVC Express can build the replacement library, as long as you have your INCLUDE, BIN, LIB environment variables set up with SDK paths properly. I’m hoping to include the library as part of MozillaBuild, so that Mozilla code will build “out of the box” with MSVC Express.

The MFCEmbed example will not build with MSVC Express unless you build MFC yourself (which is painful). But MFCEmbed is not part of Firefox builds anyway (and should probably be disabled completely, since it’s an unmaintained bug-ridden mess).

Code that uses XPCOM is frequently verbose. Take, for instance, the relatively simple act of creating a URI object from a string:

var ioservice = Components.classes["@mozilla.org/network/io-service;1"].
  getService(Components.interfaces.nsIIOService);
var uri = ioservice.newURI(uristring, null, null);

What if this code looked a lot more like a Python import module statement?

const network = Components.modules["@mozilla.org/network"];
var uri = network.newURI(uristring, null, null);

This code is more readable, and is slightly more efficient. We could do this now, for Mozilla 1.9, in a backwards-compatible way that didn’t require any code changes for existing code (i.e. createInstance() and getService() would continue to work as they do today). We already have XPCOM modules, which currently only implement the nsIModule interface. To make the above code a reality we’d only need to give the module an identifier so that it could be accessed by name, and teach the necko module to implement nsIIOService, with a little classinfo throw in for automatic interface flattening.

With this technique, it is even be possible to load arbitrary files as XPCOM modules, without having to autoregister them in the global registry: extensionmodule = Components.loadModule(somefile).

There is at least one problem with this approach: it means that extensions could no longer override the IOService contractID. Back when XPCOM was being copied from MS-COM, this was considered a major advantage. I don’t believe that it ever worked well, and there are much better ways to achieve extensibility, for classes that are specifically designed to be overridden.

Perhaps JS could even grow a “from foo import X, Y, Z” statement, in imitation of Python:

from Components.modules["@mozilla.org/network"] import newURI, newFileURI, newChannelFromURI;

More Examples

Creating an nsIFile instance from a string path:

const XPCOM = Components.modules["@mozilla.org/xpcom"];
var file = XPCOM.File(spec);