The Testing Matrix

Over the past few months, I have had a number of companies and individuals come to me and ask about enhanced or stripped-down versions of XULRunner. Replying to these questions has helped me hone my thoughts, and eventually reinforced my early decision that XULRunner (libxul) should ship a well-thought out set of component/features and should not be configurable. Optional features, whether “extra” or “stripped down”, come with significant and unjustifiable cost in terms of code maintenance, documentation, and QA.


The Mozilla suite has traditionally supported all sort of configuration flags that allowed browser customizers to disable or add optional components. There were build-time flags to disable/enable features such as XMLHttpRequest, webservices support, multi-profile support, or even cryptography (at one point this was required by law).

The various components were joined through XPCOM interfaces and XUL overlays. In theory, each component could be cleanly disabled without affecting other components. To some, this separation of components was (and is) one of the major benefits of the Mozilla codebase: embedders and redistributors are able to select the particular features they wish, without the burden of distributing unnecessary extras.

The Mozilla Platform

Any web browser is a miniature operating system, a “platform” on which web applications can be built. The Mozilla codebase is doubly a platform because of its ability to build rich-client applications and extensions using technologies such as XUL. One of the most important aspects of any platform is that it is all about developers.

Web developers do not want to have to figure out whether their customer is using a special gecko with XMLHttpRequest or XSLT disabled: they want to be able to test their app against a few versions of Firefox and against IE and say “it works”, and then move on to more important things like saving the world or making money.

XUL+XPCOM developers are in an even worse pickle: XPCOM interface documentation has traditionally been sparse to non-existent, and it is frequently impossible to tell what interfaces and XPCOM components would be available and functioning properly with various features configured off.

The Cost of Options

What seems at first glance a benefit to embedders and distributors turns out to have significant consequences; these consequences come in four forms:

  1. web developers;
  2. user and web developer documentation;
  3. quality assurance;
  4. the Mozilla codebase itself.

The cost to web authors is immediate and painful. Not only are they requied to deal with multiple version of gecko, each with its own bugs, but also with feature differences within gecko versions depending on the distributor or embedding context: will XMLHttpRequest be available to my web page? What about XSLT? box-model layout? Web authors need there to be “one gecko” with standard features that “just work”.

The cost to the mozilla codebase itself is twofold. First, the build system has developed undue complexity to dealing with combations of optional components. But more importantly, significant platform features have been unimplemented because of the “bad” dependencies they would introduce. The most obvious one that comes to mind is extending the XUL overlay syntax so that overlay elements can be inserted at arbitrary locations specified in xpath. Since xpath is part of an optional component, this change has been on hold for a long time due to resistance that XUL might depend on the optional XPath component.

Overall, the cost of build preferences is as great or greater in terms of developers than the cost of UI preferences is in terms of users.

The Testing Matrix

The hidden and probably most expensive costs incurred by optional featuritis are found in documentation and quality assurance. Every time you add an optional feature you add an exponential permutation to the feature testing matrix. It is certainly non-obvious that enabling the “typeaheadfind” extension in Firefox breaks the browser, but it is so! Is is also a non-obvious implementation detail that disable XSLT disables pretty-printing, and disabling “xmlextras” means that none of webservices works. Every permutation must also be documented in the web developer and XPCOM documentation. The sheer complexity of the task guarantees that neither the documentation nor the quality assurance tests can be both comprehensive and accurate.

A good Testing Matrix (which we have yet to achieve) requires intensive sets of testcases integrated with the Mozilla QA process and TestRunner. At the least, every frozen interface, and every web-developer API should have an automated testcase; visual features should also have a human-read testcase. I am excited about the TestRunner project and look forward to the future when this can be a regularly tested reality.

What Features?

The alternative to the extreme modularity of our codebase and ad-hoc featuritis is to have a definitive list of features that are part of our platform (both our web platform and our XUL+XPCOM platform). Our shipped features must integrate with each other and must not have dependencies which cannot be tested.

The list of features that I have specified for XULRunner can be found here. For the most part, this list was compiled as follows: every feature that will be part of our public “DOM” APIs is included. In addition, the core XUL language, and the features which are required to bootstrap a typical XUL application are included. Finally, it includes various features which seem to be desired by many XUL application authors (spellcheck APIs) or which would be very difficult to add as application-specific code (JS debugging).

But What About…

There are two camps of people who dislike my plan: developers who want more features in the XULRunner, and developers who want fewer.

If your favorite feature is not on the list, please don’t rant! You can probably include this feature in your application directly… it just won’t be shared with other applications. In most cases, extra features are be avoided for one of two reasons: 1) it’s not feasible to include it in the testing matrix or 2) not enough people would use it to justify bloating the XULRunner (and therefore Firefox) download weight.

Projects like Minimo and Chandler [1] [2], as well as several commercial groups, have expressed interest in a stripped-down XULRunner. Chandler would like basically a replacement for wxWidgets, without all the networking/SSL/etc code that comes with XULRunner. Minimo wants the networking/SSL code, but would like to strip out unnecessary XUL features and perhaps some of the more memory-hogging web features such as XSLT and (depending on the target device) xmlextras, webservices, etc.

I don’t think that either of these approaches will be successful in the long run. The strength of the Mozilla platform must remain in its innovation, stability, and documentation, and I do not believe it is possible to maintain a “split codebase” where some features are second-class or may not be present. To get a sense of what I mean, in terms of Minimo, see bug 302577.

Coda: Download Weight

Download weight is obviously important to a certain class of mass-market applications, though certainly not to all. The design requirements for XULRunner specify is that it maintain a small enough footprint for use as the runtime for Firefox, which therefore requires something smaller than 4.5MB (Windows) when optimally compressed (using 7zip). That is currently the case (the windows ZIP builds are 5.5MB, but that’s with significantly non-optimal compression), and we intend to stay well within that range and perhaps even improve the current Firefox target with various linking and code generation optimizations. The goal is not to be Java or .NET, with a fantastically heavy base runtime, but instead to be a runtime that allows for a range of networked applications, from existing web applications to semi-privileged offline web applications to fully-privileged client applications, using traditional web technologies and languages.

Atom Feed for Comments 6 Responses to “The Testing Matrix”

  1. Neil Stansbury Says:

    Ben, I don’t understand why XULRunner can’t make it’s components dynamically available as packages like XPIs? XULPlanet’s XPCOM Reference seems to provide a logical grouping to split them into individual packages.

    XULRunner could then be a tiny app with an ‘updater’ and ‘installer’ utility that downloads on demand the components needed by an implementor. All an implementor need do is specify em:minVersion/em:maxVersion for each dependant component as done currently in install.rdf.

    If a dependancy exists in install.rdf that’s been disabled then the app doesn’t install. With Firefox’s new binary update system, all these packages could then get updated like any other XPI.


  2. Heikki Toivonen Says:

    I agree that fully configurable, no-standard configuration platform is infeasible to test fully etc. However, I do think there is demand for, and it should be possible to maintain a couple of standard configurations. The current xulrunner is fine for its intended purpose. But I think a really minimal just-the-xul package would not be too difficult and I can see other projects besides Chandler being interested in it.

  3. Antonio Says:


    the testing-matrix (testrunner) looks a pretty promissory and useful tool indeed. But configurable options like this one your expose out (libxul) is critial once some popular webbrowsers that are built on the top of mozilla (or even minimo) don’t use XUL to get their UI, but GTK (gtkmozembed) either. What do you think about it (and also ?

    bte, i’m working on gtk-mozilla/minimo-based webbrowsers for the Nokia 770 device, and just worried about footprint and unnecessery runtime overload)


  4. BSBlog » Blog Archive » Mozilla, Embedding, Small Devices (and gtkmozembed) Says:

    […] The more difficult problem is that the toolkit (xulrunner) currently doesn’t support a lot of the “small-device” build options such as –disable-xul. We definitely want to continue to support small device embedding situations, while avoiding the costs of supporting an exponential combination of little build options that may have complex interactions. This provoked a spirited discussion among about what options are valuable to support and who should bear the support burden of these options. The provisional outcome of this discussion is as follows: Mozilla is interested in maintaining a couple of “small device profiles”, in cooperation with small-device embedders (cell phone manufacturers and others). I have written up a wiki document with a preliminary proposal describing three profiles. I would like to get input from small-device embedders about what features they actually need, and to what extent they are able to help in the maintenance and documentation of these small-device profiles. […]

  5. Enrico 'nekrad' Weigelt Says:

    Ah,drop one of the most important advantages of OSS – customizability in favour of equilibrium phantasies.
    Great deal.

    What comes next ? Merge mozilla w/ IE ?
    Yeah, live could be so easy if there was only one browser, only one GUI, only one OS, only one way of thinking, only one way of living. Hey it would be so easy, if there was only one way of dressing (yeah, we had some discussions about school uniforms in .de lately). And of course only one food for all, called Soylent.


    Actually, I’m really fed up of such discussions.

    Why not just going ahead and solving the real problem: lack of *real* modularity ?

    You talked about “mozilla’s extreme modularity”. I don’t see anything of that. I only see an gigantic blod with very unclear structures.
    In fact there’re things like “extensions” – just another word for “plugins”: some piece of code hooking in somewhere and changing a lot of things. Of course this is an QM nightmare.

    Modularity is something completely different. I suggest studying Wirth’s “9 thesis about simplicity”.
    Real modules are autonomous object with really clear and strict interfaces. Imports on the one side, and an export on the other side.

    For moz it would mean, ie. the whole storage stuff, GUI bindings, XUL engine, etc as *separate* modules.
    And in an strictly modularized mozilla, there would be separate libraries for handling the data storage stuff (ie. bookmarks, cookies, history, addresses, …) which can be built independently and have an clear an well-documented interfaces.

    Ah, as you wrote about bad/missing documentation of the xpcom stuff.
    Well, if *I* was the tree maintained, I would strictly reject undocumented modules.

    The big problem of moz development is: evryone just adds new features, the tree gows and grows, but nobody seriously tidies up. Such super-projects suffer from the same problems as super-companies like M$.

    Yeah, such things happen, if you only care about some “market” instead of concentrating on just building an good product ;-P

  6. Christopher Blizzard » Blog Archive » system components, fsync and distribution-specific changes: a cautionary tale Says:

    […] Because of the amount of work that we do on Linux and how closely we work with upstream projects (sqlite, cairo, etc) we’re still the experts on what works and what doesn’t. And because we have a pretty full set of tests that we go through we know what versions of upstream projects work well inside of Mozilla. Note that this doesn’t mean we know which versions don’t work with Mozilla, as I will illustrate later. We can’t be compatible with every single version of every upstream project with every single possible configuration, it just doesn’t work. […]

Leave a Reply