I’m hoping to start blogging more regularly, and model my blog after The Old New Thing. So I’m planning on posting in pairs: one technical post related to my work or Mozilla, and one non-technical post relating to personal posts about my family, music, or other things I find interesting.

I think I am raising Food Network junkies. I was making BLT sandwiches for lunch the other day, and my three-year-old daughter Claire was hungry. She asked me “are you going to start plating soon?”

She adores Michael Symon, and often when we turn on Dinner: Impossible she says “I love Michael Symon, he’s beeeaautiful.” Claire wants to be an Iron Chef when she grows up. Ellie really enjoys when Cat Cora is the Iron Chef, or when there’s a female challenger in general. And they all watch Good Eats with rapt attention.

It’s amazing to me how much and how quickly they learn. We play a “how do we cook it” game: I pick an ingredient, and they tell me how you’d cook it. Ellie, who is four years old, recently said to me, “Daddy, if you don’t heat the carrot pieces in a pan with oil, they won’t be soft in your carrot stew.” In their play-kitchen, they are always concocting soups and baked goods, and arguing about ingredients.

I’m taking a break from garbage collection for a week or so: I got stuck, and there are lots of other things going on I wanted to help out on. Yesterday and today’s project was profiling some DOM testcases.

Two days ago, Jason recently landed a great patch to minimize the XPConnect overhead of DOM calls (fast-path DOM). Prior to this patch, many profiles of DOM scripting were dominated by XPConnect overhead (marshaling calls from JS to binary XPCOM). So I decided to re-do some of these profiles and see if there were any easy wins lurking, now that the noise was gone. I first ran the Dromaeo tests in a build from mozilla-central and compared the results to Safari on the same machine. Now, I’m taking some of the comparatively worst performers and using Shark to profile the tests.

I figured that getting shark to profile individual tests would require some major hacking. But it turns out that Dromaeo already has support for wrapping tests with calls to generate Shark profiles! All I needed to do was hack a little bit to generate a single profile at a time.

I started by profiling the following test: DOM Modification (Prototype): update(). mozilla-central was 8x slower than Safari on this test.

1. Start with a shark-enabled Firefox.
2. Download or clone Dromaeo from here.
3. Type `make web` to build a local copy of Dromaeo.
4. Start shark for programmatic control as documented here.
5. Point your browser at the test like so:

   file:///builds/dromaeo/web/index.html?dom-modify-prototype&shark=update&numTests=1

6. Shark should do a little dance and pop up a profile viewer. For a quick overview on using the Shark profile viewer, see Vlad’s blog.
7. By using the top-down view, I quickly discovered that over 70% of runtime was spent in a single function:

   

8. By double-clicking this function, I could see a heatmap of execution within the function: just two lines of code were responsible for most of the time!:
   .

9. This was more than enough evidence to file a bug.
10. After a bit of conversation with Brian Crower on IRC, I found that my initial hypothesis was wrong: The JS_ISSPACE
    macro is not really to blame. Every time it encountered a \s or \S in a regular expression character class, the code would loop over all 65,536 characters in the unicode basic plane and ask a series of lookup tables “is this character a space?” Because there are a small number of actual whitespace characters, I could replace this large loop with a small table of whitespace character ranges.

11. The patch made this particular test 77% faster, from 850ms to 195ms.

I’ve already filed a bug on another test and will be working through at least four more significant slowdowns. Doing this profiling has been a lot of fun, and a nice change of pace from the garbage collection slog. I really encourage anyone who has a mac to spend a little time with Shark and a performance issue: it actually makes visualizing and analyzing performance problems fun.

I am setting up some temporary tinderboxes to repack localization builds. Because I don’t trust the DNS service from my home ISP, I wanted to download builds from ftp.mozilla.org using HTTPS. It turns out this was quite the challenging task, due to the following cute and relatively useless error message:

ERROR: Certificate verification error for ftp.mozilla.org: unable to get local issuer certificate
To connect to ftp.mozilla.org insecurely, use '--no-check-certificate'.

What this really means is “your copy of wget/OpenSSL didn’t come with any root certificates, and HTTPS just isn’t going to work until you get them and I know about them.”

Getting Root Certificates

The best way to get the root certificates you need is at this website. It has a tool that will convert the root certificates built-in to Mozilla NSS into the PEM format that OpenSSL expects. It also has pre-converted PEM files available for download if you’re lazy.

Installing cacert.pem into MozillaBuild (Windows)

To install cacert.pem so that it works with MozillaBuild:

1. Copy cacert.pem to c:/mozilla-build/wget/cacert.pem
2. Create the following configuration file at c:/mozilla-build/wget/wget.ini:

   ca_certificate=c:/mozilla-build/wget/cacert.pem

Ted filed a bug about setting this up automatically for a future version of MozillaBuild.

Installing cacert.pem on Mac:

The following instructions assume you got your wget from macports using port install wget.

1. Copy cacert.pem to /opt/local/etc/cacert.pem
2. Create the following configuration file at /opt/local/etc/wgetrc:

   ca_certificate=/opt/local/etc/cacert.pem

One of the important communication mechanisms in the Mozilla project is IRC. IRC is a great tool for instant communication among large groups of diverse people. However, it’s easy to mis-use IRC, so I’d like to propose some etiquette rules:

Think/search before your ask

Google can answer a fair number of questions. Keep the signal/noise ratio high on IRC by checking FAQs and google before asking questions.

Ask questions in the right channel

Some channels welcome newbies: #xulrunner and #extdev specifically welcome new XULRunner application authors and extension developers. Some channels (#developers) are used for serious/deep project communication, and don’t really welcome novices. If you’re not sure, feel free to silently watch the channel for a few minutes. If you don’t know which channel is right, feel free to ask “is this the right channel to ask questions about X”. The channel residents will let you know!

IRC is not good for some questions

Complicated questions are difficult to answer on IRC: “When I configure with –enable-XX while cross-compiling from YY, I have problem ZZ.” To answer this question you need to sort through all sorts of issues such as why you’re using –enable-XX, what the actual error message is, and details about the cross-compile setup. You’d be much better off posting this question to the appropriate newsgroup (mozilla.dev.builds in this case).

If I say to you “please post details about this question to the newsgroup”, it’s not because I don’t like you or don’t want to help you… it’s because IRC isn’t a good medium for answering your particular question.

IRC doesn’t take up my whole attention

Most of the people on IRC are also doing work (coding, reviewing, writing, whatever). If somebody doesn’t have time to talk to you right now, feel free to wait for somebody else to come along, or send email/post to the newsgroups. Demanding somebody’s attention on IRC is very rude. If you really need their attention, send email.

Don’t send uninvited private messages

Asking questions in the appropriate channel is nice behavior. Other people in the channel can look at the conversation and even provide help. By sending me a private message, you are demanding my attention (see above), and drastically limiting the number of people who can help you. Unless you know me really well, don’t send me private messages.

IRC is not email

If I don’t respond to you right away, don’t assume that I’ve seen your message. If I don’t respond to your message and you need to get in touch with me, please send me email. Please don’t ping every four hours until I respond to you.

In it’s basic form, JSON cannot serialize cyclic graphs of objects, or graphs where multiple paths can lead to the same object. In a project I’m working on, I wanted to move such a graph of highly-interconnected objects from JS to python. So I have invented a format built on top of JSON that can be used to serialize/deserialize such graphs.

Basically, the JSON comes across as a large list:

[
  /* list[0] is the base object at the root of the eventual object graph. */
  {
    /* string, number, true/false, and null properties are serialized directly */
    "stringprop": "stringvalue",
    "numprop": 3.1415,
    /* but lists and objects are not serialized directly. Instead, they are represented by an index
       into the base list. "sharp" is a nod to JS sharp variables, from which this was originally inspired */
    "complexprop": {"sharp": 1}
  },
  /* list[1] is referenced from list[0].complexprop. It also references itself, see below */
  [
    "simplestring",
    3,
    {"sharp": 1}
  ]
]

You can find JS for serializing these types of graphs here, and python for deserializing them here.

It turns out that I probably don’t actually need this code: I’ve found a simpler solution for my particular problem, but I wanted to share this solution in case other people might find it useful.

I was intrigued to read about a new search engine, cuil.com, which is being launched by former Google employees. So I tried it out by searching for XULRunner. The top hits were:

• http://en.wikipedia.org/wiki/XULRunner
• http://wiki.mozilla.org/XULRunner:Roadmap
• http://blogs.acceleration.net/ryan/archive/2005/05/06/1073.aspx
• http://chatzilla.rdmsoft.com/xulrunner/
• http://wiki.mozilla.org/XUL:Xul_Runner
• http://developer.yahoo.net/blog/archives/2008/02/flickr-uploadr-open-source-xulrunner.html
• http://benjamin.smedbergs.us/blog/2007-05-15/xulrunner-what-we-are-doing/
• http://www.songbirdnest.com/node/1771

There were also suggested “categories”:

Mozilla

Mozilla Foundation, SpiderMonkey, MozillaZine, Mitchell Baker, SeaMonkey, Mozilla Corporation, Mozilla Public License, Bugzilla

Mozilla Developers

Asa Dotzler, Mitchell Baker, Daniel Glazman, Window Snyder, Brendan Eich, Mike Shaver, Ben Goodger, Tristan Nitot

Mozilla Extensions

ChatZilla, Adblock, Greasemonkey, CustomizeGoogle, DOM Inspector, Venkman, Flashblock, ColorZilla

Netscape

Mozilla, Netscape Navigator, Netscape Browser, The Book of Mozilla, Daniel Glazman, Ben Goodger, Netscape Communicator, Tristan Nitot

W3C Standards

MathML, Resource Description Framework, XSL Transformations, Document Object Model, Cascading Style Sheets, Scalable Vector Graphics, XHTML, … (hidden)

I’m a bit disturbed by the pattern here: a search for XULRunner turns up interesting results, but the primary page on XULRunner (http://developer.mozilla.org/en/docs/XULRunner) is not present. Suggested terms for Mozilla has interesting results, but the most important (Firefox) is not present. The suggested categories are interesting, but the sub-results aren’t relevant to the specific topic I searched for: what I really want is “Mozilla Developers who do XULRunner”.

Is cuil.com built on a strategy of “list all the search results except for the one I really wanted”?

Oh, and a search for “tamarin javascript” turns up 0 results… something must be fishy with their search index.

I’m going to be at OSCON next week. Taras and I will be hosting a BoF session on using automatic analysis and rewriting tools for open-source projects on Wednesday evening. You’re welcome to come and learn about our tools, watch demos, or just just heckle and meet!

I’ve been to OSCON once before. I got a chance to meet people in person I only knew via email, and also meet some new people who had read my blog or otherwise knew of me. It was a blast. I hope to meet even more people this time around. Many of the official sessions don’t look that exciting to me, so I might spend a decent amount of time at OSCamp or just talking to interesting people. If you’d like to meet me and can’t make the BoF session, you can probably catch me at the Mozilla booth.

I’ll be staying the following weekend in Portland, so if you have recommendations of restaurants or sights that I shouldn’t miss while I’m there, comments welcome.

I am a relative newcomer to python, and have been blown away by the flexibility of some operations in Python. The string-formatting operator, %, is really wonderful and flexible. This is my attempt at implementing something similar in JavaScript.

Obviously, you can’t create a new operator in JavaScript, and in addition you can’t use % as a JavaScript identifier. So I went for the next-best thing:

String.prototype.format = function string_format(d) {
  // there are two modes of operation... unnamed indices are read in order;
  // named indices using %(name)s. The two styles cannot be mixed.
  // Unnamed indices can be passed as either a single argument to this function,
  // multiple arguments to this function, or as a single array argument
  let curindex = 0;

  if (arguments.length > 1)
    d = arguments;
  
  function r(s, key, type) {
    let v;
    if (key == "") {
      if (curindex == -1)
        throw Error("Cannot mix named and positional indices in string formatting.");

      if (curindex == 0 && (!(d instanceof Object) || !(0 in d)))
        v = d;
      else if (!(curindex in d))
        throw Error("Insufficient number of items in format, requesting item %i".format(curindex));
      else
        v = d[curindex];

      ++curindex;
    }
    else {
      key = key.slice(1, -1);
      if (curindex > 0)
        throw Error("Cannot mix named and positional indices in string formatting.");
      curindex = -1;
      
      if (!(key in d))
        throw Error("Key '%s' not present during string substitution.".format(key));
      v = d[key];
    }
    switch (type) {
    case "s":
      return v.toString();
    case "r":
      return v.toSource();
    case "i":
      return parseInt(v);
    case "f":
      return Number(v);
    case "%":
      return "%";
    default:
      throw Error("Unexpected format character '%s'.".format(type));
    }
  }
  return this.replace(/%(\([^)]+\))?(.)/g, r);
};
String.prototype.ø = String.prototype.format;

If you are at all familiar with the python string-formatting operator, this should be very similar:

"%s %s".ø("angry", "monkeys"); == "angry monkeys";
"%(key)s: %(value)s".ø({key: 'bananas', value: 'tasty'}) == "bananas: tasty";
"%i - %i".ø([1, 3]) == "1 - 3";
"%r".ø(['α', 'ω'] == '["α", "ω"]';

I know that there are many sprintf-like libraries out there for JavaScript: I just happen to like mine best. Caution: this code requires JavaScript 1.7… if you replace some “let”s with “var”s it may work in older browsers, but I haven’t tested it.

Being able to pass a function as a replacement in String.replace is a very powerful feature!

Today I set up a buildbot/tinderbox for the mozilla-central codebase built with static checking. This allows us to enforce annotations such as NS_FINAL_CLASS and NS_STACK_CLASS throughout our codebase. See the static-analysis-bsmedberg tree on the mozilla-central tinderbox.

Stack-only Classes

As an example, I today annotated nsAutoString as a stack class. If someone commits code which allocates an nsAutoString on the heap, the static-checking tinderbox will turn red.

I have a set of similar patches in the works which mark various helper classes as stack-only. These patches are needed because the XPCOMGC rewrites treat stack-only classes differently from regular heap-allocated types.

Help Wanted

A while back Dave Mandelin wrote an analysis of outparam usage. This is now running and producing warnings. I would like to find some help to go through the fairly large number of warnings this analysis produces and find the real bugs and fix the bogus warnings in the analysis.

The most important warning to check is

warning: outparam not written on NS_SUCCEEDED

This indicates a condition where the analyzer can’t prove that an outparam was written, but the method returned NS_SUCCEEDED anyway. This can lead to uninitialized memory errors and odd latent bugs. If you’d like to help, please hop over to the #mmgc IRC channel, and dmandelin or I can help walk you through the analysis/fixing process.

Local Machine

Because the dehydra/treehydra codebase is still in flux prior to the 1.0 release, I am currently maintaining this on one of my local machines, so if it goes down please don’t pester Mozilla’s IT or release teams. Once dehydra 1.0 is released, we will get turn on static checking on the main unit-test tinderboxes maintained by the Mozilla release team.

More in the occasional series of tricks you can do with the Mozilla build system.

Making Preprocessed Sources

If you have a compile error and you can’t figure out what macros are being expanded, you can make a preprocessed version of any C/C++ file:

$ cd $OBJDIR/some/directory
$ make nsMyFilename.i

Using Doxygen to Generate Documentation

You can use the documentation-generation tool Doxygen to automatically generate interface/class/method documentation from the Mozilla sources.

$ cd $OBJDIR
$ make documentation

Warning: doxygen isn’t especially precise. Your mileage may vary.

Displaying a Build Variable

Sometimes when hacking a Makefile it may not be obvious what the final value of a variable is. You can check this by using the echo-variable-% rule:

$ cd $OBJDIR/some/directory
$ make echo-variable-EXTRA_DSO_LDOPTS