Please welcome Todd Whiteman as the new owner of PyXPCOM.

PyXPCOM is a bridge between python and Mozilla XPCOM which allows app authors to write chrome script and XPCOM components in python. For a long time its source code has been in the main Mozilla codebase (CVS and then mozilla-central) in extensions/python. It is used by some fairly large projects such as the Komodo editor as well as the Sugar project.

Unfortunately, nobody ever created a bugzilla component for tracking PyXPCOM, and PyXPCOM bugs would languish in “Core: General” or “Core: XPCOM” unnoticed. As I was going through old bugs in XPCOM, I noticed the languishing PyXPCOM bugs. I asked Mark Hammond, the original PyXPCOM author and maintainer, whether he still had time to maintain the module. He did not, but he nominated Todd Whiteman (ActiveState) as his replacement.

In order to make PyXPCOM maintenance easier, Todd and I arranged for the following changes to PyXPCOM:

• The PyXPCOM code now lives at http://hg.mozilla.org/pyxpcom. It has its own configure script which builds against the XULRunner SDK. It is no longer necessary to build all of XULRunner or Firefox in order to build PyXPCOM. The PyXPCOM code in mozilla-central has been removed.
• A new bug component: PyXPCOM bugs now have their own bugzilla component in the “Other Applications” product:
  • View all open PyXPCOM bugs
  • File new PyXPCOM bug

Todd has nominated the following peers for PyXPCOM:

• Mark Hammond
• Tomeu Vizoso
• Shane Caraveo
• Trent Mick

Patches to PyXPCOM should be reviewed by Todd or one of the module peers. (Note that not all the peers appear on the module ownership page because of technical issues with the tool generating that page.)

I’m happy that PyXPCOM has active ownership and can continue to grow and serve the needs of its community.

Yesterday a group of developers were sifting through crash reports that might be related to the new JIT code in Firefox 3.5. We found an unusual set of crashes with EXCEPTION_ILLEGAL_INSTRUCTION. After sifting through these crashes, I found that they have very similar patterns:

• The crashes are all on old processors: the original Pentium, the original AMD K6 (not K6-II or K6-III), or a Via processor. See bug 500277 for the specific family/model/stepping numbers that appear to be affected.
• The crashes all have very low uptime: the browser appears to crash on startup.

Anyone with an original AMD K6 running Linux or Windows, could you please try launching Firefox 3.5 and browsing around the web with it? Please post your results in bug 500277. If you are willing to spend some time diagnosing the problem in a debugger, please email me or join #jsapi on IRC.

For a long while now (even before Google Chrome was announced), Mozilla has been examining ways to make Firefox better by splitting the work of displaying web pages up among multiple processes. There are several possible benefits of using multiple processes:

• Increased stability: if a plugin or webpage tries to use all the processor, memory, or even crashes, a process can isolate that bad behavior from the rest of the browser.
• Performance: By splitting work up among multiple processes, the browser can make use of multiple processor cores available on modern desktop computers and the next generation of mobile processors. The user interface can also be more responsive because it doesn’t need to block on long-running web page activities.
• Security: If the operating system can run a process with lower privileges, the browser can isolate web pages from the rest of the computer, making it harder for attackers to infect a computer.

Now that we’re basically done with Firefox 3.5 we’ve formed a project team. We’re calling the project “Electrolysis”. Because we can’t do everything at once, we are currently focusing on performance and stability; using a security sandbox will be implemented after the initial release. Details of the plan are available on the Mozilla wiki, but the outline is simple:

1. Sprint as fast as possible to get basic code working, running simple testcase plugins and content tabs in a separate process.
2. Fix the brokenness introduced in step one: shared networking, document navigation and link targeting, context menus and other UI functions, focus, drag and drop, and probably many other aspects of the code will need modifications. Many of these tasks can be performed in parallel by multiple people.
3. Profile for performance, and fix extension compatibility to the extent possible.
4. Ship!

We’re currently in the middle of stage one: Ben Turner and Chris Jones have borrowed the IPC message-passing and setup code from Chromium. We even have some very simple plugins loading across the process boundary! Most of the team is in Mountain View this week and we’re sprinting to see if we can implement a very basic tab in a separate process today and tomorrow.

For the moment we’re focusing on Windows and Linux, because the team is most familiar and comfortable on these environments. I sat down with Josh Aas on Friday and we discussed some of the unknowns/difficulties faced on mac. As soon as our initial sprint produces working code we’d love to have help from interested mac hackers!

If you’re interested in helping, or just lurking to see what’s going on, the Electrolysis team is using the #content channel on IRC and the mozilla.dev.tech.dom newsgroup for technical discussions and progress updates. We’ll also cross-post important status updates to mozilla.dev.platform.

If you’ve emailed me volunteering to help and I haven’t gotten back to you, I apologize! Until we get the stage-one sprint done there aren’t really any self-contained tasks which can be done in parallel.

Things I’ve learned recently:

• Using hg log on a file that was removed will not list the revision in which the file was removed. You want hg log --removed.
• Waist Watcher sodas are sweetened with Splenda, but don’t have the metallic taste that some other diet sodas do. I especially like the Citrus Frost (grapefruit) flavor. It’s like Fresca without the hidden Aspartame. (I have bad digestive reactions to Apartame.)
• Linking libxul on my Linux machine takes between 3 and 10 seconds, but apparently this is unusual. Several other people have reported link times that can range into the minutes. I recommend buying as much RAM as you can: if your entire object directory fits in the filesystem memory cache, build speed is much faster.
• When Microsoft Visual C++ mangles function symbols, the return type is encoded in the mangled name. When GCC mangles names, the return type is not encoded:
  

  	GCC	MSVC
  int testfunc(int)	_Z8testfunci	?testfunc@@YAHH@Z
  void testfunc(int)	_Z8testfunci	?testfunc@@YAXH@Z

  This means that separate functions in different translation units may not differ only by return type. We found this trying to combine the Chromium IPC code with Mozilla: const std::string& EmptyString() and const nsAFlatString& EmptyString() differ only by return type. On Windows this links correctly, but on Linux this causes multiple-definition errors.

Suzanne and I are pleased to announce the birth of Rebekah Elizabeth Smedberg, born 18 April A.D. 2009 at 12:43 a.m. 8 lbs. 3 oz., 20 inches. All are well.

Christian worship has a history of stunningly beautiful hymnody. Many of the hymns and chants of the Latin rite form the musical and poetical foundations of Western culture. I have a habit of posting some of the most beatiful or neglected hymns on this blog. I recently posted the Pange Lingua for Holy Thursday and the Exultet for Easter vigil.

After these posts I received several comments about how these posts were inappropriate for Planet Mozilla. Asa and the other people who run Planet Mozilla have decided that it should be an aggregation of the people who work on Mozilla, not just Mozilla-related posts. I agree with their decision. We already have topical newsgroups; Planet Mozilla serves a different purpose of broadening and connecting the Mozilla community. This is why Planet Mozilla syndicates all of my posts, not just the Mozilla-related ones.

It’s surprising to me that the religious posts receive so much negative reaction. I have never received any objections to the book reviews, random political musings, Pennsylvania trivia, or other posts which are equally “off topic”.

For those of you who wish to read only my Mozilla-related posts, follow the mozilla category or its feed.

For more information, see prior discussions in the mozilla.governance newsgroup.

Rejoice, heavenly powers! Sing choirs of angels!
Exult, all creation around God’s throne!
Jesus Christ, our King is risen!
Sound the trumpet of salvation!

Rejoice, O earth, in shining splendor,
radiant in the brightness of your King!
Christ has conquered! Glory fills you!
Darkness vanishes for ever!

Rejoice, O Mother Church! Exult in glory!
The risen Savior shines upon you!
Let this place resound with joy,
echoing the mighty song of all God’s people!

My dearest friends,
standing with me in this holy light,
join me in asking God for mercy,
that he may give his unworthy minister
grace to sing his Easter praises.

The Lord be with you.
And also with you.

Lift up your hearts.
We lift them up to the Lord.

Let us give thanks to the Lord our God.
It is right to give him thanks and praise.

It is truly right that with full hearts and minds and voices
we should praise the unseen God, the all-powerful Father,
and his only Son, our Lord Jesus Christ.

For Christ has ransomed us with his blood,
and paid for us the price of Adam’s sin to our eternal Father!

This is our passover feast,
When Christ, the true Lamb, is slain,
whose blood consecrates the homes of all believers.

This is the night,
when first you saved our fathers:
you freed the people of Israel from their slav’ry,
and led them dry-shod through the sea.

This is the night,
when the pillar of fire destroyed the darkness of sin.

This is night,
when Christians ev’rywhere,
washed clean of sin and freed from all defilement,
are restored to grace and grow together in holiness.

This is the night,
when Jesus broke the chains of death
and rose triumphant from the grave.

What good would life have been to us,
had Christ not come as our Redeemer?

Father, how wonderful your care for us!
How boundless your merciful love!
To ransom a slave you gave away your Son.

O happy fault, O necessary sin of Adam,
which gained for us so great a Redeemer!

Most blessed of all nights,
chosen by God to see Christ rising from the dead!

Of this night scripture says:
“The night will be as clear as day:
it will become my light, my joy.”

The power of this holy night dispels all evil,
washes guilt away, restores lost innocence,
brings mourners joy;
it casts out hatred, brings us peace,
and humbles earthly pride.

Night truly blessed,
when heaven is wedded to earth
and we are reconciled to God!

Therefore, heavenly Father, in the joy of this night,
receive our evening sacrifice of praise,
your Church’s solemn offering.

Accept this Easter candle,
a flame divided but undimmed,
a pillar of fire that glows to the honor of God.

Let it mingle with the lights of heaven
and continue bravely burning
to dispel the darkness of this night!

May the Morning Star which never sets
find this flame still burning:
Christ, that Morning Star,
who came back from the dead,
and shed his peaceful light on all mankind,
your Son, who lives and reigns for ever and ever.

Amen.

The hymn of the solemn proclamation of Easter.

St. Thomas Aquinas, sequence for the Solemnity of the Body and Blood of Christ and processional for the Mass of the Lord’s Supper.

pymake news:

• Bad news: pymake is still 5x slower than GNU make on Linux/Mac.
• Good news: pymake is 25% faster than msys make (GNU make on Windows)!
• Best news: there’s a lot of room to make performance better.

All measurements are do-nothing depend builds. Full rebuilds aren’t significantly affected because compiler speed overwhelms any time we spend in make.

Creating Windows processes is more expensive than creating processes on a unix-like operating system. Creating MSYS processes is hugely more expensive. Windows I/O in general is slow compared to Linux, at least for typical build tasks. Because pymake recurses in a single process, caches parsed makefiles such as rules.mk, and avoids many shell invocations, it can make up for slow parsing times by dramatically reducing time spent elsewhere.

How to use pymake on Windows

Don’t use pymake with client.mk on Windows, yet. pymake doesn’t understand MSYS-style paths, which is what configure substitutes for @srcdir@ and @topsrcdir@ when using client.mk. This will be fixed by the patches available from this bug tree.

Configuring manually isn’t hard: to build Firefox in c:/builds, follow this recipe:

$ mkdir /c/builds
$ hg clone http://hg.mozilla.org/mozilla-central /c/builds/mozilla-central
$ cd /c/builds/mozilla-central
$ autoconf-2.13 && (cd js/src && autoconf-2.13)
$ mkdir ff-debug
$ cd ff-debug
$ export MAKE='python -O c:/builds/mozilla-central/build/pymake/make.py'
$ ../configure --enable-application=browser --enable-debug --disable-optimize
$ python -O ../build/pymake/make.py -j4

How to use pymake on Linux/Mac

Configure manually as above, or add the following flags to your mozconfig file:

export MAKE="python -O $topsrcdir/build/pymake/make.py"
mk_add_options MAKE="python -O @TOPSRCDIR@/build/pymake/make.py"

Soon on all platforms this will be as simple as mk_add_options MOZ_ENABLE_PYMAKE=1

Thank you!

Special thanks to Arpad Borsos who wrote tests and an implementation of –keep-going for pymake.

Next plans

Immediate future plans for pymake reduce the process count even further, especially for depend builds:

Currently every invocation of nsinstall is a separate process, and we invoke nsinstall even when all its install targets are up to date. Simple tasks like this will instead be implemented as native python commands. Ted implemented a branch to do this, but the current implementation blocks the only thread. I think we’re going to switch and use shared-nothing threads and message passing to parallelize before making this the default behavior.

Every time Mozilla processes a makefile the build system combines all the compiler-generated dependencies into a single .all.pp file using mddepend.pl: this allows developers to move or remove header files without breaking depend builds. Running a perl script for every makefile invocation is silly, especially because all it does is parsing and rewrite makefile syntax. I will have pymake read these dependency files directly and ignore missing files (causing a rebuild without an error) using a syntax includedeps $(INCLUDEFILES)

Longer-term work that would make pymake much more useful:

• Build an object graph of the entire Mozilla tree recursively. I think I know how to do this, although there will be some issues with how to deal with local versus global variables.
• Warn and eventually force a more rigorous dependency graph: warn if a dependent file ‘appears’ without having a rule to create it.
• Make parsing a lot faster using mx.TextTools instead of native python regular expressions. Keep the regular expressions as a slow path for developers who don’t have TextTools installed.

Python Reference Cycles and Closures

While debugging pymake performance and memory usage I found an interesting fact, which in hind sight should have been obvious: functions which enclose themself in python create reference cycles which have to be cleaned up by the Python garbage collector:

def outerFunction(outerCallback):
  targetsToBuild = [1, 2, 3]
  def innerCallback():
    if len(targetsToBuild):
      # innerCallback closes on itself... this creates a reference cycle every time you call outerFunction
      # if you call outerFunction 100000 times per build, this can add up really quickly and cause large GC pauses
      targetsToBuild.pop(0).build(innerCallback)
    else:
      outerCallback()

After finding this problem, I refactored (1, 2, 3) the pymake code to use objects instead of closures to save asynchronous state while rebuilding. Also, OptionParser instances create cycles by default. There is a lightly-documented method OptionParser.destroy which can be used to manually break these cycles (thanks to Ted for finding it). pymake now runs without creating any reference cycles and I disabled the python garbage collector.

Environment Munging in MSYS

When MSYS goes from an MSYS process to a Windows process, and vice-versa, it munges certain environment variables to account for the path styles. I previously thought that it only munged PATH, but I discovered today that I was wrong: MSYS was munging the MAKEFLAGS environment variable in odd ways.

If MAKEFLAGS in the MSYS process was ‘ -j1 — PATH=e:/builds/mozilla-central’ it would be munged into ‘ -j1 — PATH=e;c:/mozilla-build/msys/builds/mozilla-central’ in a non-MSYS process. Without the leading space the value was not touched. I don’t know why this is, but I altered the pymake code slightly so that MAKEFLAGS would never start with a space (and would be more compatible with gmake in the process).

Part of an occasional series of less-well-known books I love.

Lost in the Cosmos: The Last Self-Help Book by Walker Percy

Despite the subtitle, Lost in the Cosmos is only a mock self-help book. It doesn’t pretend to have answers, but instead has many questions, a multi-chapter diversion about the theory of symbols and language, and two science-fiction short stories.

In a recent issue of a home-and-garden magazine, an article listed fifty ways to make a coffee table.

One table was made of an old transom of stained glass supported by an antique brass chandelier cut ingeniously to make the legs.
Another was a cypress stump, waxed and highly polished.
Another was a big spool used for telephone cable set on end.
Another was a lobster trap.
Another was a Coca-Cola sign propped on Coke crates.
Another was a stone slab from an old morgue, the blood runnel used as an ash tray.
Another was a hayloft door set on cut-down sawhorses.
Another was the hatch of a sailboat mounted on halves of ships’ wheels.
Another was a cobbler’s bench.

Not a single one was a table designed as such, that is, a horizontal member with four legs.

Question: Why was not a single table designed as such rather than being a non-table doing duty as a table?

(a) Because people have gotten tired of ordinary tables.
(b) Because the fifty non-tables converted to use as tables make good conversation pieces.
(c) Because it is a chance to make use of valuable odds and ends which would otherwise gather dust in the attic.
(d) Because the self in the twentieth century is a voracious nought which expands like the feeding vacuole of an amoeba seeking to nourish and inform its own nothingness by ingesting new objects in the world but, like a vacuole, only succeeds in emptying them out.

(Check One)

The first half of the story is questions such as this, interspersed with hypothetical scenarios. Each question focuses on some aspect of who we are, both individually and collectively. Then, in an odd but effective twist, the book veers off into a chapter about “semiotics”: how we use symbols to communicate, how these symbols gain meaning, and how we use symbols differently than any other creature.

The end of Lost in the Cosmos contains two science-fiction short stories. The first story is particularly compelling because it deals with the fall of man: that man gained consciousness of himself and soon after was destroyed by that very consciousness. The alien race, on the other hand, has preternatural consciousness: they have avoided “the malformations of self consciousness”. The aliens refuse to let man land, because they are fallen and have not asked for help. The state of fallen mankind described in this story is different but has striking parallels to the story of the fall of Adam.

For a book which bounces quickly through language, transcendence, evolution, boredom, and sexuality, and pretends to classify everyone’s self-understanding in 16 categories, Lost in the Cosmos is a surprisingly coherent and enjoyable story. It is sometimes aggravating, especially when none of the multiple-choice answers comes close to your answer, but it will always provoke thought about how man knows himself, and how you personally know yourself.

Available at amazon.com.

If you’ve read Lost in the Cosmos and want to dive deeper, Walker Percy wrote a less-fiction on language: The Message in the Bottle: How Queer Man Is, How Queer Language Is, and What One Has to Do with the Other (also available at amazon.com).