As I noted previously, I’ve been using our static analysis tools to generate documentation for the Mozilla string classes.

All of the code to generate this documentation is now checked in to mozilla-central. To regenerate documentation or hack the scripts, you will first need to build with static-checking enabled. Then, simply run the following command:

make -C xpcom/analysis classapi

To automatically upload the documentation to the Mozilla Developer Center, run the following command:

MDC_USER="Your Username" MDC_PASSWORD="YourPassword" make -C xpcom/analysis upload_classapi

One of the really exciting things about the Dehydra static-analysis project is that the analysis is not baked into any compiler. You can version your analysis scripts as part of your source code, run them from within your build system, and change them as your analysis needs change.

For example, I decided that a class inheritance diagram would help people understand the Mozilla string classes. So I modified the documentation script to produce graphviz output in addition to the standard XML markup. I then process the graphviz output to PNG with an imagemap and upload it to MDC along with the other output as an attachment¹.

The output is available now. I’m still looking for volunteers to improve the output as well as the source comments to make it all clearer!

1. There is a MediaWiki extension so you can put graphviz markup directly in a wiki page and it will be transformed automatically. However, this extension currently doesn’t work on the Mozilla Developer Center. It’s being tracked in bug 463464 if you’re interested. ^

One of the common complaints about the Mozilla string code is that it’s very difficult to know what methods are available on a given class. Reading the code is very difficult because it’s hidden behind a complex set of #defines, it’s parameterized for both narrow and wide strings, and because we have a deep and complex string hierarchy. The Mozilla Developer Center has a string guide, but not any useful reference documentation.

With a little hand-holding, static analysis tools can produce very useful reference documentation, which other tools simply cannot make. For example, because a static analysis tool knows the accessibility of methods, you can create a reference document that contains only the public API of a class. I spent parts of yesterday and today tweaking a Dehydra script to produce a string reference. I’m working with Eric Shepherd to figure out the best way to automatically upload the data to the Mozilla Developer Center, but I wanted to post a sample for comment. This is the public API of nsACString:

I am trying to keep the format of this document similar to the format we use for interfaces on MDC. It’s a bit challenging, because C++ classes have overloaded method names and frequently have many methods. In the method summary, I have grouped together all the methods with the same name.

Once the output and format are tweaked, I can actually hook the entire generation and upload process to a makefile target, and either run it on my local machine or hook it up to a buildbot. I used E4X to do the actual XML generation. It was a learning experience… I don’t think I’m a fan. I want Genshi for JavaScript. Making conditional constructs in E4X is slightly ugly, and making looping constructs is really painful: my kingdom for an XML generator so that I don’t have to loop and append to an XMLList.

When people get started with XPCOM, one of the most confusing rules is how to pass data across XPCOM boundaries. Take the following method:

IDL markup

string getFoo();

C++ generated method signature

nsresult GetFoo(char **aResult);

C++ Implementation

The aResult parameter is called an “out parameter”. The implementation of this method is responsible for allocating memory and setting *aResult:

nsresult
Object::GetFoo(char **aResult)
{
  // Allocate a string to pass back
  *aResult = NS_Alloc(4);

  // In real life, check for out-of-memory!
  strcpy(*aResult, “foo”);

  return NS_OK;
}

C++ Caller

The caller, after it is finished with the data, is responsible for freeing the data.

char *foo;
myIFace->GetFoo(&foo);
// do something with foo
NS_Free(foo);

The important thing to note is that the code doesn’t allocate memory with malloc, and doesn’t free it with free. All memory that is passed across XPCOM boundaries must be allocated with NS_Alloc and freed with NS_Free.

We have this rule because of mismatched allocators. Depending on your operating system and the position of the moon, each shared library may have its own malloc heap. If you malloc memory in one shared library and free it in a different library, the heap of each library may get corrupted and cause mysterious crashes. By forcing everyone to use the NS_Alloc/Free functions, we know that all code is using the same malloc heap.

Helper Functions

In most cases, there are helper functions which make following the rules much easier. On the implementation side, the ToNewUnicode and ToNewCString functions convert an existing nsAString/nsACString to an allocated raw buffer.

On the caller side, you should almost always use helper classes such as nsXPIDLString to automatically handle these memory issues:

Better C++ Caller

nsXPIDLCString foo;
myIFace->GetFoo(getter_Copies(foo));
// do something with foo

Impact on Extension Authors

It is especially important for extension authors to follow this advice on Windows. The Windows version of Firefox uses custom version of the Windows C runtime which we’ve patched to include the high-performance jemalloc allocator. Extension authors should link the C runtime statically, which guarantees that they will have a mismatched malloc/free heap.

Notes

• New code shouldn’t use “string” or “wstring” IDL types; use “AString” or “ACString” instead.
• See Choosing the right memory allocator on MDC.
• See the XPCOM string guide.

Occasionally, people will come on the #xulrunner or #extdev channel with a question about compiling XPCOM components. The question often goes something like this:

<IRCGuy> I’m following a tutorial on making XPCOM components, but I can’t seem to get them to compile. Can anyone tell me what my problem is?

Hint for asking a good question: IRCGuy needs to tell us some combination of 1) what tutorial he’s following, 2) what the failing command is or 3) what the error message is.

This time, IRCGuy’s compile command and error message are:

IRCGuy@IRCGuy-france:/mnt/data/IRCGuy/public/project/xpcom-test$ make
g++  -I/usr/local/include/xulrunner-1.9/unstable -I/usr/local/include/xulrunner-1.9/stable -L/usr/local/lib/xulrunner-devel-1.9/lib -Wl,-rpath-link,/usr/local/bin  -lxpcomglue_s -lxpcom -lnspr4 -fno-rtti -fno-exceptions -shared -Wl,-z,defs  france2.cpp -o france2.so
/tmp/cceFg2dD.o: In function `NSGetModule':
france2.cpp:(.text+0x38c): undefined reference to `NS_NewGenericModule2(nsModuleInfo const*, nsIModule**)'

IRCGuy’s problem is a problem of link ordering: with most unix-like linkers, it is very important to list object files and libraries in the correct order. The general order you want to follow is as follows:

1. Object files
2. Static libraries – specific to general
3. Dynamic libraries

If an object file needs a symbol, the linker will only resolve that symbol in static libraries that are later in the link line.

The corrected command:

g++  -I/usr/local/include/xulrunner-1.9/unstable -I/usr/local/include/xulrunner-1.9/stable -fno-rtti -fno-exceptions -shared -Wl,-z,defs  france2.cpp -L/usr/local/lib/xulrunner-devel-1.9/lib -Wl,-rpath-link,/usr/local/bin  -lxpcomglue_s -lxpcom -lnspr4 -o france2.so

Bonus tip: correct linker flags for linking XPCOM components can be found on the Mozilla Developer Center article on the XPCOM Glue. As noted in the article, xpcom components want to use the “Dependent Glue” linker strategy.

Automatically rewriting code is perilous business. Can you spot the bug in the following rewrite to remove stack nsCOMPtrs?

@@ -432,8 +432,8 @@ LoadAppDirIntoArray(nsIFile* aXULAppDir, if (!aXULAppDir) return; 
- nsCOMPtr<nsIFile> subdir; 
- aXULAppDir->Clone(getter_AddRefs(subdir)); 
+ nsIFile* subdir; 
+ aXULAppDir->Clone(&subdir);
  if (!subdir) return;

The patch produced by garburator matches the spec I originally wrote for Taras, bug and all… the bug is that nsCOMPtr automatically initializes itself to null, while a raw pointer doesn’t.

What makes automatic rewriting hard is that the specification of the rewrite is usually incomplete. I could say to Taras “get rid of nsCOMPtrs on the stack” or even provide sample patches, but reality is much more complicated. For the automatic rewriting of stack nsCOMPtrs, this was an iterative process. Taras would run the tool (named garburator), apply the patch, and test-build. When something didn’t work right, he’d analyze the problem (often with help from the IRC channel) and then refine the rewriting tool. Sometimes refining the tool wasn’t worth it, and he’d simply fix the code by hand or hide it from the rewriter.

Quick quiz: how do you deal with functions that take a nsCOMPtr<nsIFoo>&?

Answer: It depends… is this a hashtable enumerator callback function? If so, don’t rewrite the signature at all. If not, the correct rewriting is probably nsIFoo**… but then if the code passes a heap variable to the function, you have to wrap it in getter_AddRefs to get the correct write-barrier semantics.

Automatic rewriting is not yet a complete solution: in particular, it doesn’t attempt to rewriting templated functions, and it will often give up on items which are passed to templated functions. And of course, we can really only run the rewriting tools on Linux, which means that platform-specific mac and windows code will need to be rewritten manually. If you want horror stories, ask Taras about nsMaybeWeakPtr, the helper-class from hell! In any case, I now have an XPCOMGC tree which runs through component registration before crashing… it feels like quite an accomplishment, even though it hasn’t gotten past the first GC yet.

In XPCOMGC, the behavior of nsCOMPtr is very different than currently:

• nsCOMPtr should only be used as a class member, never on the stack. Taras is working on a rewriting script that will replace nsCOMPtr<nsIFoo> on the stack with a raw nsIFoo* (more on that later).
• the purpose of nsCOMPtr is not to ensure correct reference counting (there is no reference-counting!); instead it serves to enforce write barriers, so that MMgc can properly perform incremental GC.

I was able to rewrite nsCOMPtr so that existing code code mostly use the existing API: there is however one major difference: getter_AddRefs cannot return a pointer directly to the nsCOMPtr instance. Instead, it must save the value in a local variable and call nsCOMPtr.set() to preserve the write-barrier semantics. It does this using a temporary class:

/**
 * nsGetterAddRefs is used for XPCOM out parameters that need to be assigned
 * to nsCOMPtr members. We can't pass the address of nsCOMPtr.mRawPtr directly
 * because of the need to set the write barrier.
 */
template <class T>
class nsGetterAddRefs
{
public:
  explicit
  nsGetterAddRefs(nsCOMPtr<T> &aSmartPtr) :
    mTempPtr(aSmartPtr),
    mTargetSmartPtr(aSmartPtr)
  {
    // nothing else to do
  }

  ~nsGetterAddRefs()
  {
    mTargetSmartPtr = mTempPtr;
  }

  operator T**()
  {
    return &mTempPtr;
  }

private:
  T* mTempPtr;
  nsCOMPtr<T> &mTargetSmartPtr;
};

template <class T>
inline
nsGetterAddRefs<T>
getter_AddRefs(nsCOMPtr<T> &aSmartPtr)
{
  return nsGetterAddRefs<T>(aSmartPtr);
}

For the vast majority of cases where code makes a simple getter call, this works fine:

nsresult rv = something->GetAFoo(getter_AddRefs(mFoo));

However, if you test or use the returned value after you get it in the same statement, the value won’t be assigned yet:

Bad:

if (NS_SUCCEEDED(something->GetAFoo(getter_AddRefs(mFoo))) && mFoo)

Also bad:

NS_SUCCEEDED(something->GetAFoo(getter_AddRefs(mFoo))) && mFoo->DoSomething();

In the XPCOMGC world, both of these cases will fail because the ~nsGetterAddRefs destructor runs after the dereference of mFoo. Once we remove stack comptrs, this is not a common occurrence, but it does happen occasionally.

Checking for this kind of pattern is the perfect job for dehydra: check it out.

After playing with the code in my last post, and asking some apparently silly questions about the ES4 spec, I’ve found some techniques which can be used to implement python-style “import” using current JavaScript.

References to Bound Functions

In python, it is possible to hold a reference to an unbound method, or to a method which is bound to a particular object:

>>>class Foo:
...  def dumpMe(self, arg):
...    print "self: %s\narg: %s" % (self, arg)
...
>>> Foo.dumpMe
<unbound method Foo.dumpMe>
>>> Foo().dumpMe
<bound method Foo.dumpMe of <__main__.Foo instance at 0x2aaaaab33680>>
>>> Foo.dumpMe("something")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: unbound method dumpMe() must be called with Foo instance as first argument (got str instance instead)
>>> Foo().dumpMe("something")
self: <__main__.Foo instance at 0x2aaaaab33d40>
arg: something

The JavaScript language does not have a built-in syntax to obtain a bound method:

js> function Foo() { }
js> Foo.prototype.dumpMe = function(arg) {
  dump("this: " + this + "\narg: " + arg);
}
js> new Foo().dumpMe
function (arg) {
    print("this: " + this + "\narg: " + arg);
}
js> f = new Foo();
js> f.dumpMe("test")
this: [object Object]
arg: test
js> unbound = new Foo().dumpMe
js> unbound("test")
this: [object BackstagePass @ 0x6214e0 (native @ 0x513fc8)]
arg: test

(The BackstagePass object is the global object in xpcshell).

However, with a little extra code it is possible to emulate bound functions in JavaScript:

js> Function.prototype.bind = function(object) {
  var func = this;
  return function() { return func.apply(object, arguments); }
}
js> boundFunc = f.dumpMe.bind(f);
js> boundFunc("test")
this: [object Object]
arg: test

We can then use bound functions to emulate the from module import x, y, x statement of python:

js> function jsimport(module) {
  // this function has two forms:
  // jsimport(module) is equivalent to python "from module import *"
  // jsimport(module, "name" [, "name2"]) is equivalent to python "from module import name, name2"

  // NOTE: "this" is the global object
  function internal_import(name) {
    if (module[name] instanceof Function) {
      this[name] = module[name].bind(module);
    }
    else {
      this[name] = module[name];
    }
  }

  if (arguments.length == 1) {
    for (name in module) {
      if (module.hasOwnProperty(name)) {
        internal_import(name);
      }
    }
  }
  else {
    for (i = 1; i < arguments.length; ++i) {
      internal_import(arguments[i]);
    }
  }
}

You could use this technique today to hide away the IOService or PrefService a little:

jsimport(Components.classes["@mozilla.org/preferences/pref-service;1"].getService(Components.interfaces.nsIPrefBranch),
         "getCharPref", "getIntPref", "getBoolPref");

if (getBoolPref("dom.window.dump.enabled"))
  // do something dumpy

Code that uses XPCOM is frequently verbose. Take, for instance, the relatively simple act of creating a URI object from a string:

var ioservice = Components.classes["@mozilla.org/network/io-service;1"].
  getService(Components.interfaces.nsIIOService);
var uri = ioservice.newURI(uristring, null, null);

What if this code looked a lot more like a Python import module statement?

const network = Components.modules["@mozilla.org/network"];
var uri = network.newURI(uristring, null, null);

This code is more readable, and is slightly more efficient. We could do this now, for Mozilla 1.9, in a backwards-compatible way that didn’t require any code changes for existing code (i.e. createInstance() and getService() would continue to work as they do today). We already have XPCOM modules, which currently only implement the nsIModule interface. To make the above code a reality we’d only need to give the module an identifier so that it could be accessed by name, and teach the necko module to implement nsIIOService, with a little classinfo throw in for automatic interface flattening.

With this technique, it is even be possible to load arbitrary files as XPCOM modules, without having to autoregister them in the global registry: extensionmodule = Components.loadModule(somefile).

There is at least one problem with this approach: it means that extensions could no longer override the IOService contractID. Back when XPCOM was being copied from MS-COM, this was considered a major advantage. I don’t believe that it ever worked well, and there are much better ways to achieve extensibility, for classes that are specifically designed to be overridden.

Perhaps JS could even grow a “from foo import X, Y, Z” statement, in imitation of Python:

from Components.modules["@mozilla.org/network"] import newURI, newFileURI, newChannelFromURI;

More Examples

Creating an nsIFile instance from a string path:

const XPCOM = Components.modules["@mozilla.org/xpcom"];
var file = XPCOM.File(spec);

XPCOM technology, based on Microsoft COM, is fundamentally structured around the concept of binary object layouts and stylized calling conventions. XPCOM was a good technique for introducing modularity and extensibility to the Mozilla codebase, but it is showing its age. One of the interesting things about Mozilla 2 is that we can breaking API and binary compatibility.

There are several ways we should improve XPCOM:

1. Improve reference-counting (this could include universal support for cycle collection, or even allocating all XPCOM objects using MMgc; Graydon and I talked about this some at the Summit, and I’m sure he’ll take the lead determining what this means in practice.
2. Allow throwing complicated (object-type) exceptions from any XPCOM method, and reduce the verbosity and inefficiencies of nsresult return codes. C++ exceptions, as much as I dislike them, provide the shortest path to this goal. Taras has been working with oink to provide an automated way to convert method calls automatically.
3. Reduce the complexity and verbosity of using XPCOM. I’ve been spending a fair amount of time working in Python recently, and I’m very impressed with its use of module objects. Using XPCOM could be a lot easier from script with some very simple changes. I’ll blog about this soon!

In order to achieve these objectives, I’m convinced that Mozilla must break XPCOM binary compatibility, and should stop using XPCOM as the binary embedding solution:

• We may want the flexibility of making GCRoot or another abstract non-interface class the root type (nsISupports) for all XPCOM objects. We at least ought to add interfacerequestor and classinfo functionality to the root object type, and perhaps weak-reference support as well.
• C++ exceptions are very compiler dependent (and compiler-version dependent) and are not good candidates for binary freezing.

The implications of a change like this are considerable:

• It will no longer be possible (or desirable) to write binary XPCOM components in C++ that don’t live in the monolithic platform binary (libxul). At first this seemed like a significant challenge: Firefox and Thunderbird use binary components to do OS integration (profile migration and OS integration). Various extension also use binary components to integrate with external libraries. But most of these use-cases can be solved with a good foreign-function-interface library available from script. I’ll blog about this separately; I’ve been very impressed with the expressiveness and flexibility of the python ctypes library and I think it could be ported to SpiderMonkey rather easily.
• Binary embedders (e.g. gtkmozembed clients) will no longer be able to access DOM objects via their XPCOM interfaces.
  The simplest way to solve this problem is to extend the scriptable NPAPI object model to be accessible by binary embedders. This will give embedders access to the DOM that is straightforward and relatively complete.

Brainstorming Example

class nsISupports : virtual public RCObject
{
  inline void AddRef() { IncrementRef(); return 2; }
  inline void Release() { DecrementRef(); return 1; }

  virtual nsISupports* QueryInterface(REFNSIID aIID, PRBool aAddRef) = 0;

  /**
   * For ease of conversion, provide an old-style QI wrapper.
   */
  inline nsresult QueryInterface(REFNSIID aIID, void **aResult) {
    *aResult = QueryInterface(aIID, PR_TRUE);
    return (*aResult) ? NS_OK : NS_NOINTERFACE;
  }

  virtual nsISupports* GetInterface(REFNSIID aIID, PRBool aAddRef) = 0;

  virtual nsIClassInfo* GetClassInfo() = 0;
};

The virtual inheritance of RCObject could be a problem for xptcall. There are ways around that. I’m also a little concerned that objects won’t be storing pointers to the “root” GCObject, but rather vtables within that object. I hope that doesn’t mess up MMgc.