In June I was at the Mozilla Foundation for a week for an all-hands meeting. During that week I met a bunch of great folks I had only ever met on IRC, including dria, mconnor, pav, shaver, and vlad (in alphabetical order…). When we weren’t in official meeting of various sorts, there was time to get together in small ad-hoc working groups; these working groups were very productive and informative for me.

During one of WG sessions, vlad and mconnor and I came to a consensus: the current method of constructing XUL (Firefox/Tbird/whatever) themes is fragile and should be rethought. We came to the conclusion that most of the core styling instructions should ship in content packages, not skin packages. Skin packages would only contain theme-specific CSS overrides which alter the default theming instructions. The default theme will therefore contain empty skin CSS files.

This will make it much easier to skin applications in gecko 1.9; instead of duplicating every style rule in the classic theme, which is basically what theme authors do now, authors can simply override the elements that make their theme unique. This may simply be changes to a few icons, or colors, or whatever, but it means that themes are much more likely to be stable and work with multiple future versions of the application.

This work is being tracked in bug 305746.

People frequently send me e-mail to ask me questions or discuss XULRunner. While this is not an entirely bad thing (I like hearing what people want from XULRunner and the Mozilla “platform”), it gates a lot of discussion through me, which can slow things down considerably. I would like to recommend the following communication channels for people who have general things to ask about XULRunner:

• For general development questions pertaining to XULRunner applications, let’s use the netscape.public.dev.xul newsgroup;
• For questions about using XULRunner in embedding contexts, use netscape.public.mozilla.embedding;
• For discussion and planning about the development of XULRunner itself, I have created a semi-private mailing list XULRunner-dev@mozilla.org. This list is meant for serious developers who are working on XULRunner’s design and implementation. The archives are public, but list membership is by invitation only. If you would like to be part of this list, please email me privately and let me know.

I would also like to encourage XUL application developers to write articles for the XULRunner section of the Mozilla Developer Center. Writing XULRunner apps is pretty simple, but only if you have a good guide to get you started!

Update

The newsgroup hierarchy has changed. XULRunner discussion now belongs in mozilla.dev.platform, mozilla.dev.tech.xul, or mozilla.dev.embedding.

I have been trying to follow Joel Spolsky’s advice. Right now it is very hard to say “XULRunner 1.9 needs X hours of work to be ready for production use” because basically nobody (including myself) has any clue what concrete tasks are needed to get from here to there. I have a pretty good mental overview of the tasks, but that is a far cry from “it would take me 412 programming hours to complete them”, and even less the amount of debugging time to get to a final product.

Bugzilla is a pretty good bug tracking tool, but it’s painfully unsuited at task management and software scheduling. Joel recommends a simple spreadsheet, but that has some serious problems when I need to expose the data for lots of people on the web, and probably need to allow multiple programmers to fill in data. Instead of doing nothing, waiting for bugzilla to get better, I have written a little tool to help me manage XULRunner. It uses a custom XML dialect and some XBL magic to create a hierarchical list of tasks, with estimated and actual times for completion, and automatic totals.

Actually, I’m lying. I wrote two different versions of this tool, each with its own problems. I’m posting both URLs for comments and suggestions:

1. XUL version that uses server-side persistence – view. This version loads a XUL page directly into memory: the page can be edited inline and saved back to the server using the menu. If this were the final version, I would probably include auto-save functionality and maybe some other stuff. The big advantage of this version is that it persists the open/closed state of each item. The big problem with this version is when two people try to edit simultaneously: there is no collision detection, and there is no real possibility of smart-merge. Currently the “save” functionality is password-protected. If enough people are interested, I’ll make a “playground” version.
2. Wiki-based version – view | edit. This version cannot be edited inline: you have to edit the wikimarkup manually… but the markup is pretty readable all on its own, so you don’t even have to use the “view” version if you don’t want to. The big disadvantes of this version are the lack of inline editing, and no persistence of the open-closed state. This version makes use of a special mediawiki patch that allows wiki pages to be viewed “raw” in various mime types, see bug 300657.

Right now, I’m going to use the wiki-based version. In the future I may try again with some special database-oriented system in the future, but this version is “good enough” for my needs; I need to actually work on the data for XULRunner 1.9!

What would be really nice is to make bugzilla less painful to use, and integrate this kind of project-management data into it, but that is a topic for another day.

Over the past few months, I have had a number of companies and individuals come to me and ask about enhanced or stripped-down versions of XULRunner. Replying to these questions has helped me hone my thoughts, and eventually reinforced my early decision that XULRunner (libxul) should ship a well-thought out set of component/features and should not be configurable. Optional features, whether “extra” or “stripped down”, come with significant and unjustifiable cost in terms of code maintenance, documentation, and QA.

History

The Mozilla suite has traditionally supported all sort of configuration flags that allowed browser customizers to disable or add optional components. There were build-time flags to disable/enable features such as XMLHttpRequest, webservices support, multi-profile support, or even cryptography (at one point this was required by law).

The various components were joined through XPCOM interfaces and XUL overlays. In theory, each component could be cleanly disabled without affecting other components. To some, this separation of components was (and is) one of the major benefits of the Mozilla codebase: embedders and redistributors are able to select the particular features they wish, without the burden of distributing unnecessary extras.

The Mozilla Platform

Any web browser is a miniature operating system, a “platform” on which web applications can be built. The Mozilla codebase is doubly a platform because of its ability to build rich-client applications and extensions using technologies such as XUL. One of the most important aspects of any platform is that it is all about developers.

Web developers do not want to have to figure out whether their customer is using a special gecko with XMLHttpRequest or XSLT disabled: they want to be able to test their app against a few versions of Firefox and against IE and say “it works”, and then move on to more important things like saving the world or making money.

XUL+XPCOM developers are in an even worse pickle: XPCOM interface documentation has traditionally been sparse to non-existent, and it is frequently impossible to tell what interfaces and XPCOM components would be available and functioning properly with various features configured off.

The Cost of Options

What seems at first glance a benefit to embedders and distributors turns out to have significant consequences; these consequences come in four forms:

1. web developers;
2. user and web developer documentation;
3. quality assurance;
4. the Mozilla codebase itself.

The cost to web authors is immediate and painful. Not only are they requied to deal with multiple version of gecko, each with its own bugs, but also with feature differences within gecko versions depending on the distributor or embedding context: will XMLHttpRequest be available to my web page? What about XSLT? box-model layout? Web authors need there to be “one gecko” with standard features that “just work”.

The cost to the mozilla codebase itself is twofold. First, the build system has developed undue complexity to dealing with combations of optional components. But more importantly, significant platform features have been unimplemented because of the “bad” dependencies they would introduce. The most obvious one that comes to mind is extending the XUL overlay syntax so that overlay elements can be inserted at arbitrary locations specified in xpath. Since xpath is part of an optional component, this change has been on hold for a long time due to resistance that XUL might depend on the optional XPath component.

Overall, the cost of build preferences is as great or greater in terms of developers than the cost of UI preferences is in terms of users.

The Testing Matrix

The hidden and probably most expensive costs incurred by optional featuritis are found in documentation and quality assurance. Every time you add an optional feature you add an exponential permutation to the feature testing matrix. It is certainly non-obvious that enabling the “typeaheadfind” extension in Firefox breaks the browser, but it is so! Is is also a non-obvious implementation detail that disable XSLT disables pretty-printing, and disabling “xmlextras” means that none of webservices works. Every permutation must also be documented in the web developer and XPCOM documentation. The sheer complexity of the task guarantees that neither the documentation nor the quality assurance tests can be both comprehensive and accurate.

A good Testing Matrix (which we have yet to achieve) requires intensive sets of testcases integrated with the Mozilla QA process and TestRunner. At the least, every frozen interface, and every web-developer API should have an automated testcase; visual features should also have a human-read testcase. I am excited about the TestRunner project and look forward to the future when this can be a regularly tested reality.

What Features?

The alternative to the extreme modularity of our codebase and ad-hoc featuritis is to have a definitive list of features that are part of our platform (both our web platform and our XUL+XPCOM platform). Our shipped features must integrate with each other and must not have dependencies which cannot be tested.

The list of features that I have specified for XULRunner can be found here. For the most part, this list was compiled as follows: every feature that will be part of our public “DOM” APIs is included. In addition, the core XUL language, and the features which are required to bootstrap a typical XUL application are included. Finally, it includes various features which seem to be desired by many XUL application authors (spellcheck APIs) or which would be very difficult to add as application-specific code (JS debugging).

But What About…

There are two camps of people who dislike my plan: developers who want more features in the XULRunner, and developers who want fewer.

If your favorite feature is not on the list, please don’t rant! You can probably include this feature in your application directly… it just won’t be shared with other applications. In most cases, extra features are be avoided for one of two reasons: 1) it’s not feasible to include it in the testing matrix or 2) not enough people would use it to justify bloating the XULRunner (and therefore Firefox) download weight.

Projects like Minimo and Chandler [1] [2], as well as several commercial groups, have expressed interest in a stripped-down XULRunner. Chandler would like basically a replacement for wxWidgets, without all the networking/SSL/etc code that comes with XULRunner. Minimo wants the networking/SSL code, but would like to strip out unnecessary XUL features and perhaps some of the more memory-hogging web features such as XSLT and (depending on the target device) xmlextras, webservices, etc.

I don’t think that either of these approaches will be successful in the long run. The strength of the Mozilla platform must remain in its innovation, stability, and documentation, and I do not believe it is possible to maintain a “split codebase” where some features are second-class or may not be present. To get a sense of what I mean, in terms of Minimo, see bug 302577.

Coda: Download Weight

Download weight is obviously important to a certain class of mass-market applications, though certainly not to all. The design requirements for XULRunner specify is that it maintain a small enough footprint for use as the runtime for Firefox, which therefore requires something smaller than 4.5MB (Windows) when optimally compressed (using 7zip). That is currently the case (the windows ZIP builds are 5.5MB, but that’s with significantly non-optimal compression), and we intend to stay well within that range and perhaps even improve the current Firefox target with various linking and code generation optimizations. The goal is not to be Java or .NET, with a fantastically heavy base runtime, but instead to be a runtime that allows for a range of networked applications, from existing web applications to semi-privileged offline web applications to fully-privileged client applications, using traditional web technologies and languages.

In case you have not heard, one of the very important new Mozilla initiatives is the documentation project called the “Mozilla Developer Center”, or “devmo” for short. Mozilla has already proved successful at open software development and open community-based marketing, and this is the next logical step: an open, community-based documentation resource for web developers, extension authors, XUL application developers, and core Mozilla developers.

This documentation effort is in full swing right now; visit http://developer-test.mozilla.org/ to see it in action. It is powered by MediaWiki, a powerful and extensible piece of server software that allows for community collaboration in editing documents. Mediawiki has a long track record as the software that powers Wikipedia, the free encyclopedia.

I am personally very excited by the energy I already see in the devmo project, and it’s explosive potential when it is actually “launched”. One of the historical weaknesses in Mozilla as a development platform is the lack of good documentation, and devmo is taking up the challenge in a big way. Devmo is under the experienced leadership of Deb Richards, who I have gotten to know over the past few months as person who can keep an vast number of things going and not lose sight of the final goal.

Which is why I am writing this post not to rain on the parade, but to discuss the challenges that I see facing the project. Most importantly, how devmo presents dense reference material is going to be critically important to its most important “consumer”, web developers and XUL application authors. The wikitext format lends itself quite readily to tutorials and articles, and I am working at making code samples work without a lot of pain. But reference material is not easy to either write or present in traditional wiki format. For a point of reference, please visit the msdn.microsoft.com page about the html <applet> tag. To understand what I really want out of devmo, visit that page in IE.

The property list is presented as a dense and highly-hyperlinked table. But the most important part of that table is that it is absolutely complete. You don’t have to go visit “the properties of DOM Objects” and “the properties of HTML Objects” to see what is available on this DOM element. But I want more! I want to be able to say “show me which properties were available in mozilla 1.x”.

When the gecko developers add a new method to a DOM object, it is unlikely that anyone is going to systematically go through and add this property to every one of several hundred individual element pages with the correct descriptive text and versioning information. Reference material is a recipe for out-of-date or incomplete documentation; documentation which does not list the minimum runtime for which a particlar method/property/style became available is almost worse than useless.

I believe that creating this kind of dense documentation is going to require that these cross-references are created in some kind of XML syntax. I believe that this XML should live in a wiki “page” so that it can be edited by the community like any other page. The advanced sorting capabilities that you see on the IE version can be provided through the use of XBL bindings in such a way that clients which do not support XBL will get the full property table (there are other possible solutions involving cross-browser scripting which would be more complicated but would be compatible with more browsers).

Finally, one of the major features that I want out of devmo is to provide context-sensitive help in rich-client editing tools. Everything from the XUL Application Creator slated to be part of the XULRunner Developer Kit to NVU could really use context-sensitive help when editing markup of various kinds including HTML/XHTML/XUL/CSS/JS. For this to be practical, large parts of the metadata provided by devmo will need to be in machine-readable formats.

I know that I am asking a lot out of devmo. It’s not going to get there overnight, and that should not stop anyone from contributing an article or helping edit right now. I especially encourage people with editing skills to get involved and clean up the articles that people like me might write.

I was stuck in a funk yesterday, trying to get XULRunner application bundles working properly on Mac OS X. The basic problem is that I’m trying to run the xulrunner-bin executable from the XUL.framework, but have the launch manager think that it is running as the application bundle (using the application bundle dock icon/application name/etc). The basic strategy is to have a little stub binary inside the application bundle which will execv to the xulrunner-bin binary. However, this wasn’t working… Launch Services was convinced that the xulrunner-bin was running on its own, and because it is part of a framework instead of an application bundle, it was running as a background application (no access to the dock or menus).

In my frustration I visited my infrequent hangout freenode #opendarwin, who were quick to inform me that launch services was not their domain, but that I should visit #macdev.

Thank you Mike Ash and arwyn (whose real name I don’t know) for your invaluable help! I would probably be rearchitechting how mac bundles launched the XULRunner if it weren’t for you.

There has been much confusion over what the XULRunner short-term plan is, and when various features are supposed to be available. This post is my attempt to define my short-term goals and plans and outline my timeline for important changes in the next milestone:

Update: The XULRunner roadmap is now being maintained at http://wiki.mozilla.org/XULRunner:Roadmap.

XULRunner 1.8

For the current milestone (gecko 1.8/Firefox 1.1), my immediate goal is to get a developer preview release of XULRunner that can be used to demonstrate capabilities and determine future feature needs. This release will probably not be especially useful for actual production deployment because it will lack some of the (very important) installer capabilities. The included libxul has many problems, such as exported nonfrozen symbols and a very nonfrozen API.

This developer preview release will be shipped as a framework DMG on mac, and a zip/tarball build on win/unix. There are experimental RPMs available for SuSE (and there hopefully will be some fedora-compatible RPMs as well). The XULRunner binary has the ability to register itself as a GRE for embedding purposes.

Assuming all goes smoothly, XULRunner will have the ability to install a xulapp from a directory or a zipfile from the commandline, as well as ship with a GUI xulapp to perform the same function. For the 1.8 timeframe this will not add the program to the Windows Add/Remove program list or set up a dependency tree very well (automated uninstall may not be supported, you would have to just delete the installation directory).

The major tasks left for XULRunner 1.8 are linking changes to the standalone XPCOM glue so that it does not depend on NSPR, the stub binary which is used by installed xul applications and the abovementioned xulapp installer code. Finally, the Great and Mightily Overworked Chase has promised that there are new tinderboxen on the way which will build XULRunner with SVG+canvas support and will actually start producing mac nightly builds at all.

XULRunner 1.9

In the 1.9 timeframe my goal is to make XULRunner a completely workable production solution for XUL applications including Firefox and hopefully Thunderbird, as well as for embedding applications such as Camino and Epiphany. There are many steps to this process, but my general approach is incremental:

1. Make many useful XPCOM utility classes available from the XPCOM glue.
2. Allow the build system to build Firefox as a xulapp. This will not be the default (yet), while work is being completed on the installation mechanisms.
3. Get the windows installer working
4. Build Firefox as a XULApp by default.
5. Get the various platform-specific embedding mechanisms integrated tightly and freeze the basic necessities. Windows Mac GTK QT.

I have fallen into the most dangerous trap of a computer programmer: doing nothing in the pursuit of perfection. When I left my job at Saint Patrick’s I promised that I would have my new weblog running very soon, and promised an exciting new development to boot… which was silly of me, because I had great plans but no working code to go with those plans. So I have not had a weblog for almost six months, because I kept telling myself that I would get up one day and my new weblog software would magically write itself.

I have given up those silly perfectionist dreams and returned to WordPress on my new website, and found a nice surprise: WordPress is a lot better than it was, and it has solved a lot of the complaints I originally had with it. I will be looking into WordPress extensions to make this blog even easier for you (my readers) to use: please let me know if you have suggestions.

It is good to be blogging again, because I have lots of ideas and decisions that I need to talk about. I have a backlog of ideas that I’ve wanted to blog about for the past six months, so expect some serious blog output in the next few days while I get caught up with myself.

Most importantly, the Mozilla Foundation has hired me (working remotely from Pennsylvania) to take XULRunner and libxul by the horns and wrestle them into a workable software solution. I am also spending a significant amount of time on the build system and such for localizations.

I’m aware that my website has broken links here and there, and that the import of the old blog database has left some dangling links. I’ll fixup what I can when I can, but it’s a low-priority task ;-) .

Dear readers,

As some of you may already know, I am leaving Washington D.C. and my job at St. Patrick’s, effective this Sunday 30-January. I will be joining the expanding biosphere of hackers who are paid to work on the Mozilla products and platform. I will be working full-time for the mozdev group, inc., a consulting company that specializes in software solutions based on mozilla technologies.

I am saddened to be leaving St. Patrick’s; I have made many good friends here, and learned a tremendous amount about music, worship, and life in general. I could not have asked for a better job coming right out of college, or a more beautiful church or organ.

I am excited to be moving on to the new challenges of software development. As most of you are aware, I have been significantly involved with the development of Mozilla Firefox, serving as the volunteer coordinator of localizations for the 1.0 release, and writing a decent amount of the backend code that makes loading extensions possible. My work with mozdev group is full of exciting challenges. And I intend to continue development of the XULrunner, the next-generation runtime for XUL applications, or applications which wish to embed the gecko rendering engine.

Suzanne and I will be moving to Johnstown, PA on 3-Februrary, and so I will probably be out of touch for most of next week. Our new mailing adress will be

221 Hemlock St.
Windber, PA 15963

I also have a new email address, effective immediately: benjamin@smedbergs.us. Please update your address book accordingly.

Also, the location of my weblog will be moving soon to http://benjamin.smedbergs.us/. I will try to set up a decent number of redirects so that this current location continues to get you somewhere useful. However, old blog post links will not continue to work, because I do not intend to use WordPress for the new blog. Stay tuned for some exciting developments there.

I have reversed the configure flag --enable-crypto to be --disable-crypto, so that crypto is on by default. See bug 272189 for the patch etc.

Since some countries (not in the US) may still have use or export restrictions on cryptology, builders in these countries should be aware of their local laws and be careful to abide by them.

In addition, this change means that Firefox builders no longer need to use browser/config/mozconfig, but can just specify the following options in their mozconfig or on the command line:

mk_add_options MOZ_CO_PROJECT=browser
ac_add_options --enable-application=browser

To build manually, follow this basic pattern:

cvs co mozilla/client.mk
cd mozilla
make -f client.mk checkout MOZ_CO_PROJECT=browser
cd my_objdir; # this is optional
/path/to/mozilla/configure --enable-application=browser

Please post somments and replies on the newsgroup netscape.public.mozilla.builds. Happy hacking!