Starting next week the Mozilla build system is going to require python on all platforms. This will allow us to gradually convert various build scripts which are currently written in perl, and start hacking on an autoconf replacement written in python. Any python 2.3 or higher will be acceptable.

In addition, I have made a production release of MozillaBuild, version 1.0. The MozillaBuild package is now the recommended build environment for all Mozilla developers on Windows. There were a few bugs with the 1.0 releases that have now been corrected in a MozillaBuild 1.1 release candidate.

After the Tinderboxes have been upgraded to use the MozillaBuild package on trunk, I am going to be discontinuing support for the cygwin build environment.

Please post questions and followups to the mozilla.dev.builds newsgroup.

This post is going to be a bunch of little things that irritate me about various software that I use. They’re not quite annoying enough for me to learn AppleScript/Songbird extensions/whatever and fix them myself, but I’m hoping that my readers will have solutions for at least some of these issues.

Unmute my computer every morning.

In the evenings I may turn down the sound or mute my MacBook Pro, to avoid annoying the kids or whatever. Every morning at 7AM (or whenever I wake my computer after 7am), I want the sound back to full volume and unmuted.

Play pieces randomly, not tracks.

I would like my music player to understand that the six movements of a Rheinberger Mass belong together, when playing in random mode. The cognitive dissonance of playing the Sanctus and then skipping to “She was a Hotel Detective” is painful. I’m happy to manually group the tracks when I first import them. Note: I’m currently using iTunes, but I don’t mind switching.

Disable music with words while writing

I would like my music player to avoid playing any music with words while I’m writing. I just can’t verbally multitask like that. I’m happy to tell the player which music has words, and tell it when I’m writing.

My feed reader should coalesce duplicate postings

Frequently I will get the same blog posting via Planet Mozilla, an individual blog feed, and even Google blogsearch feeds. I really desperately want my feed reader to know that these are the same post and display it once in my inbox, instead of multiple times.

Following in the tradition, here are some amusing search terms used to find this website in the past month:

• thunderbird is already running – good for it
• 10 oz baby – not quite
• greek mythology atlas shoulders
• different kinds of hammer – IANAC
• hares -delete in vcs – Version-controlled rabbit stew?
• ubuntu breasts – This shows up regularly… what are they expecting to find?
• mens nipple brestfeeding
• tortoise vs svk – tortoise wins?
• baby lojack and hospitals
• should we be able to design babies – no
• breastfeeding and firefighting – Not a combination I would recommend
• life should not be a journey to the grave – We’re all going to die somtime

Ted Mielczarek (luser on IRC) has been acting as a build-config peer for a while now, and I have been derelict and haven’t updated despot or made an announcement. Ted has been doing some great work integrating Breakpad (formerly airbag) into the toolkit, as well as learning and helping with the tangled web that is our build system. He pretty much single-handedly fixed our build system to support embedding CRT manifests for VC8.

It seems that whenever I challenge him to do something, he comes right back at me with a solution. For instance, I was musing on IRC how nice it would be to have cvsgraph produce SVG instead of a huge image. Four hours later, behold (the code is a wild-ass ugly hack, but it works)!

Belated welcome to Ted!

Suzanne and I are pleased to announce the birth of Micah Wilkes Smedberg, born 3 February, 2007 at 1:34 p.m. (EST). All are happy and healthy.

This is our fourth child, but our first boy. For those interested, take a look at Ellie, Claire, and Abbey‘s birth announcements.

I will be taking two weeks of paternity leave. While I may not be able to stay away from email entirely, I won’t be actively engaged in Mozilla activities. Here are some alternate contact points while I am away:

Build system

Ted Mielczarek (luser) or Mark Mentovai (mento)

Firefox partner builds

Dan Mills (v_thunder)

Toolkit

Gavin Sharp (gavin) or Mike Connor (mconnor)

Unit testing

Robert Campbell (robcee) or Robert Sayrer (sayrer)

Embedding/docshell

Boris Zbarsky (bz)

What VCS would you rather have?

The Tortoise: Bazaar (bzr)

A version control system, which appears to have a careful and well-planned development effort. The latest release is numbered 0.14. The import from Mozilla CVS is working so far; unfortunately, at the current import rate it will take more than a month to complete. And we have concerns about it’s performance even after import is complete.

The Hare: Mercurial (Hg)

A version control system, which appears to have a fast-and-loose development style. The latest release is numbered 0.9.3. If it would run to completion, importing Mozilla CVS would take a couple days. Unfortunately, every time we’ve tried to do an import we’ve run into bugs or undocumented features in the import tool, or odd edge-cases in the Mozilla CVS tree.

I have been helping preed bring Mozilla into the world of distributed version control systems. It sucks.

Learning It All Over

At least for me this is a very uncomfortable experience. I’m not going to pretend I like CVS, but at least the usage model for CVS is straightforward and fairly consistent. For my projects I’ve been using SVN without problems. Using a distributed VCS is a mind-bending exercise in which familiar terms like “repository”, “branch”, and “merge” no longer mean what they meant.

The documentation for these tools is in general quite painful. Partly this is because of terminology, but I have discovered that a lot of the confusion is that there is not a single usage model. There may be an official repository which hackers “push” to (a CVS-like model). There may be a central repository maintained by the project owner who “pulls” changes from others. There may not be an official repository at all, with a bunch of mostly-equal peers.

I think that I have a general understanding of how this is supposed to work, and I have a couple projects in Mercurial trees now. But I haven’t had to resolve merge conflicts yet, so I don’t feel that I’ve done more than scratch the surface of how these tools are supposed to be used.

Importing Mozilla CVS

Importing Mozilla’s CVS repository is a large task. The entire Mozilla CVS repository, with all its branches, has over 1 million file revisions which can be represented as about 200,000 change sets.

One of the things we decided early on is that Mozilla wanted a prepackaged solution as much as possible. Mozilla developers’ collective expertise is writing a web browser, not hacking version control. Any time that developers have to spend patching/hacking their VCS is collective waste. We are willing to hire outside experts to solve problems if necessary, and would very much like to do so. Of course, before hiring somebody you have to pick a system.

Unfortunately, neither of the two candidates has reached a 1.0 release yet, and the tools which import a CVS repository into the new system are even less complete. The two candidate VCS systems seem to have very different design philosophies, which show up quickly when performing import operations.

The Mercurial->CVS importer is very hacky code. The import process proceeds fairly quickly, but every time we tried it we ran into errors. We have yet to do a complete import that contains anything but the trunk:

Random CVS commit message character sets (patchset 565)

The commit message in CVS are all sorts of character sets. Initially the importer assumed that they were UTF8 and failed when it encountered invalid UTF8. I discovered a hidden HGENCODING environment variable which allowed the import to proceed by ignoring unknown character sets.

CVS backbranching (patchset 5155)

In CVS, branching is not an atomic operation, and there are several old branches that were performed at multiple times. The importer did not know how to handle this situation. Solution: don’t import those branches.

Commit messages containing cvsps-like output (patchset 47070)

Several CVS commit messages contained output that could be parsed as cvsps output. The cvsps parser contained the hg-cvs-import tool is terrible. Solution: I replaced the parser with the cvsps parser code contained in bzr.

AssertionError: failed to remove webshell/embed/ActiveX/tests/vbrowse/VBrowse.vbp from manifest (patchset 61380)

This one is still undiagnosed.

Bazaar’s CVS importer, on the other hand, feels like a carefully designed tool, with modules and unit tests. Unlike the Hg importer, the bzr->CVS importer has dealt with all odd input successfully. However, importing the trunk and all branches looks like it will take more than a month on a fast machine.

I like the fact that Bazaar focuses on correctness first and then deal with performance. Bazaar provides several desirable features, such as partial pull and a SVN frontend. But, until we can actually complete an import from CVS we don’t know whether bzr will perform acceptably. We have asked the bzr developers for assistance; we’ll see what happens.

To help speed the process, I created a tool to post-process cvsps output and limit it to branches which are still actively being developed. This should help reduce import times somewhat, as well as solving the backbranching issue which broke the Hg importer.

Other Issues

In addition to the import issues, there are other unsolved problems with deploying a distributed VCS system. We need to be able to control commit (aka “push”) access to the repository using our existing LDAP/ssh infrastructure. In addition, we want to be able to log who committed which changesets to the main repository: because a user can push other people’s commits, this is not as easy as it sounds, and I haven’t found a builtin log which saves this information.

Conclusion

Currently, we don’t have enough information to choose between Mercurial and Bazaar for a VCS, because we have not been able to complete a CVS import of either system. Right now we’re leaning heavily toward Hg, because of speed issues and because we have managed to get a trunk-only import kinda working. But the entire process is much more complicated than any of us would like, and is already turning into a major time-sucking adventure. Hopefully we can pick a system soon, and contract out the remaining work to developers who have done this before, or at least know the tools extremely well.

I have a version of MozillaBuild ready for widespread testing, packaged as an installer. I’m calling this RC1 because I don’t know of any bugs and unless we find some it is fine to put into production. It works with MSVC6, 7, 8, and MSVC 8 Express with an SDK. It builds the trunk, 1.8, and 1.8.0 branches.

Download MozillaBuildSetup.exe now

The installer was built using the patch in bug 366823, which hasn’t been reviewed or landed yet.

Please download and test this package (and start using it as your Windows development environment). If you have issues, please file bugs in Core->Build Config.

Changes From the Previous Release

• Reset CYGWIN and PATH environment variables so as not to pick up any VC paths from the environment.
• Support MSVC8 Express Edition.
• Use the atlthunk_compat code to support the ATL headers provided with the platform SDK.
• Set CVS_RSH=ssh automatically.
• Add an installer.

In addition, I added a script which converts a cygwin source tree with CRLF line endings to an MSYS tree using LF line endings, and a client.mk check to warn users of problems with line endings.

After playing with the code in my last post, and asking some apparently silly questions about the ES4 spec, I’ve found some techniques which can be used to implement python-style “import” using current JavaScript.

References to Bound Functions

In python, it is possible to hold a reference to an unbound method, or to a method which is bound to a particular object:

>>>class Foo:
...  def dumpMe(self, arg):
...    print "self: %s\narg: %s" % (self, arg)
...
>>> Foo.dumpMe
<unbound method Foo.dumpMe>
>>> Foo().dumpMe
<bound method Foo.dumpMe of <__main__.Foo instance at 0x2aaaaab33680>>
>>> Foo.dumpMe("something")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: unbound method dumpMe() must be called with Foo instance as first argument (got str instance instead)
>>> Foo().dumpMe("something")
self: <__main__.Foo instance at 0x2aaaaab33d40>
arg: something

The JavaScript language does not have a built-in syntax to obtain a bound method:

js> function Foo() { }
js> Foo.prototype.dumpMe = function(arg) {
  dump("this: " + this + "\narg: " + arg);
}
js> new Foo().dumpMe
function (arg) {
    print("this: " + this + "\narg: " + arg);
}
js> f = new Foo();
js> f.dumpMe("test")
this: [object Object]
arg: test
js> unbound = new Foo().dumpMe
js> unbound("test")
this: [object BackstagePass @ 0x6214e0 (native @ 0x513fc8)]
arg: test

(The BackstagePass object is the global object in xpcshell).

However, with a little extra code it is possible to emulate bound functions in JavaScript:

js> Function.prototype.bind = function(object) {
  var func = this;
  return function() { return func.apply(object, arguments); }
}
js> boundFunc = f.dumpMe.bind(f);
js> boundFunc("test")
this: [object Object]
arg: test

We can then use bound functions to emulate the from module import x, y, x statement of python:

js> function jsimport(module) {
  // this function has two forms:
  // jsimport(module) is equivalent to python "from module import *"
  // jsimport(module, "name" [, "name2"]) is equivalent to python "from module import name, name2"

  // NOTE: "this" is the global object
  function internal_import(name) {
    if (module[name] instanceof Function) {
      this[name] = module[name].bind(module);
    }
    else {
      this[name] = module[name];
    }
  }

  if (arguments.length == 1) {
    for (name in module) {
      if (module.hasOwnProperty(name)) {
        internal_import(name);
      }
    }
  }
  else {
    for (i = 1; i < arguments.length; ++i) {
      internal_import(arguments[i]);
    }
  }
}

You could use this technique today to hide away the IOService or PrefService a little:

jsimport(Components.classes["@mozilla.org/preferences/pref-service;1"].getService(Components.interfaces.nsIPrefBranch),
         "getCharPref", "getIntPref", "getBoolPref");

if (getBoolPref("dom.window.dump.enabled"))
  // do something dumpy