Archive for the 'untagged' Category
Taras Owns libjar
Wednesday, August 19th, 2009When Taras Glek stepped up and made major improvements to the performance of libjar, we couldn’t resist making him the module owner. libjar hasn’t had an official owner for as long as I can remember. I would occasionally do reviews, as would the networking owners (biesi/bz), but the code was really orphaned code that was unmaintained. Taras is already changing that in a big way. In a quick IRC convention with Brendan today, Taras became the new owner of the JAR code. Congratulations Taras!
Why Bother With Standards?
Wednesday, August 19th, 2009Mozilla, as an organization and product, has made implementing and developing open standards a core part of its mission from the beginning. But what is so important about standards?
I was listening to a presentation by Mitchell the other day, and one phrase in particular stood out: “Standards compliance is important because it ensures the portability of my data”. We don’t expend huge amounts of time and effort on open standards and standards compliance because of marketplace pressure. Open standards are a guarantee that the documents and data that our users are reading and writing today will continue to be available to us permanently in the future. Open standards are an essential service to the users of Firefox and to the web as a whole.
This is why Mozilla has spent so much effort implementing open video and the <video> element in HTML. And this is why open video is fundamentally better for users than Flash video or even patent-encumbered formats such as H.264.
What’s Most Important?
Wednesday, August 19th, 2009David Brooks’ Advice for High School Graduates:
The most important decision any of us make is who we marry. Yet there are no courses on how to choose a spouse. There’s no graduate department in spouse selection studies. Institutions of higher learning devote more resources to semiotics than love.
The most important talent any person can possess is the ability to make and keep friends. And yet here too there is no curriculum for this.
The most important skill a person can possess is the ability to control one’s impulses. Here too, we’re pretty much on our own.
These are all things with a provable relationship to human happiness. Instead, society is busy preparing us for all the decisions that have a marginal effect on human happiness. There are guidance offices to help people in the monumental task of selecting a college. There are business schools offering lavish career placement services. There is a vast media apparatus offering minute advice on how to furnish your home or expand your deck.
I’ve never heard it put so succinctly.
Things I’ve Learned
Wednesday, May 27th, 2009Things I’ve learned recently:
- Using hg log on a file that was removed will not list the revision in which the file was removed. You want hg log --removed.
- Waist Watcher sodas are sweetened with Splenda, but don’t have the metallic taste that some other diet sodas do. I especially like the Citrus Frost (grapefruit) flavor. It’s like Fresca without the hidden Aspartame. (I have bad digestive reactions to Apartame.)
- Linking libxul on my Linux machine takes between 3 and 10 seconds, but apparently this is unusual. Several other people have reported link times that can range into the minutes. I recommend buying as much RAM as you can: if your entire object directory fits in the filesystem memory cache, build speed is much faster.
- When Microsoft Visual C++ mangles function symbols, the return type is encoded in the mangled name. When GCC mangles names, the return type is not encoded:
GCC MSVC int testfunc(int) _Z8testfunci ?testfunc@@YAHH@Z void testfunc(int) _Z8testfunci ?testfunc@@YAXH@Z
This means that separate functions in different translation units may not differ only by return type. We found this trying to combine the Chromium IPC code with Mozilla:
const std::string& EmptyString()
andconst nsAFlatString& EmptyString()
differ only by return type. On Windows this links correctly, but on Linux this causes multiple-definition errors.
Rebekah Elizabeth Smedberg
Saturday, April 18th, 2009Religion and Planet Mozilla
Tuesday, April 14th, 2009Christian worship has a history of stunningly beautiful hymnody. Many of the hymns and chants of the Latin rite form the musical and poetical foundations of Western culture. I have a habit of posting some of the most beatiful or neglected hymns on this blog. I recently posted the Pange Lingua for Holy Thursday and the Exultet for Easter vigil.
After these posts I received several comments about how these posts were inappropriate for Planet Mozilla. Asa and the other people who run Planet Mozilla have decided that it should be an aggregation of the people who work on Mozilla, not just Mozilla-related posts. I agree with their decision. We already have topical newsgroups; Planet Mozilla serves a different purpose of broadening and connecting the Mozilla community. This is why Planet Mozilla syndicates all of my posts, not just the Mozilla-related ones.
It’s surprising to me that the religious posts receive so much negative reaction. I have never received any objections to the book reviews, random political musings, Pennsylvania trivia, or other posts which are equally “off topic”.
For those of you who wish to read only my Mozilla-related posts, follow the mozilla category or its feed.
For more information, see prior discussions in the mozilla.governance newsgroup.
Rejoice!
Saturday, April 11th, 2009Rejoice, heavenly powers! Sing choirs of angels!
Exult, all creation around God’s throne!
Jesus Christ, our King is risen!
Sound the trumpet of salvation!
Rejoice, O earth, in shining splendor,
radiant in the brightness of your King!
Christ has conquered! Glory fills you!
Darkness vanishes for ever!
Rejoice, O Mother Church! Exult in glory!
The risen Savior shines upon you!
Let this place resound with joy,
echoing the mighty song of all God’s people!
My dearest friends,
standing with me in this holy light,
join me in asking God for mercy,
that he may give his unworthy minister
grace to sing his Easter praises.
The Lord be with you.
And also with you.
Lift up your hearts.
We lift them up to the Lord.
Let us give thanks to the Lord our God.
It is right to give him thanks and praise.
It is truly right that with full hearts and minds and voices
we should praise the unseen God, the all-powerful Father,
and his only Son, our Lord Jesus Christ.
For Christ has ransomed us with his blood,
and paid for us the price of Adam’s sin to our eternal Father!
This is our passover feast,
When Christ, the true Lamb, is slain,
whose blood consecrates the homes of all believers.
This is the night,
when first you saved our fathers:
you freed the people of Israel from their slav’ry,
and led them dry-shod through the sea.
This is the night,
when the pillar of fire destroyed the darkness of sin.
This is night,
when Christians ev’rywhere,
washed clean of sin and freed from all defilement,
are restored to grace and grow together in holiness.
This is the night,
when Jesus broke the chains of death
and rose triumphant from the grave.
What good would life have been to us,
had Christ not come as our Redeemer?
Father, how wonderful your care for us!
How boundless your merciful love!
To ransom a slave you gave away your Son.
O happy fault, O necessary sin of Adam,
which gained for us so great a Redeemer!
Most blessed of all nights,
chosen by God to see Christ rising from the dead!
Of this night scripture says:
“The night will be as clear as day:
it will become my light, my joy.â€
The power of this holy night dispels all evil,
washes guilt away, restores lost innocence,
brings mourners joy;
it casts out hatred, brings us peace,
and humbles earthly pride.
Night truly blessed,
when heaven is wedded to earth
and we are reconciled to God!
Therefore, heavenly Father, in the joy of this night,
receive our evening sacrifice of praise,
your Church’s solemn offering.
Accept this Easter candle,
a flame divided but undimmed,
a pillar of fire that glows to the honor of God.
Let it mingle with the lights of heaven
and continue bravely burning
to dispel the darkness of this night!
May the Morning Star which never sets
find this flame still burning:
Christ, that Morning Star,
who came back from the dead,
and shed his peaceful light on all mankind,
your Son, who lives and reigns for ever and ever.
Amen.
The hymn of the solemn proclamation of Easter.
Sing, My Tongue, the Savior’s Glory
Thursday, April 9th, 2009Pange, lingua, gloriosi Corporis mysterium, Sanguinisque pretiosi, quem in mundi pretium fructus ventris generosi Rex effudit Gentium. | Sing, my tongue, the Savior’s glory, of His flesh the mystery sing; of the Blood, all price exceeding, shed by our immortal King, destined, for the world’s redemption, from a noble womb to spring. |
Nobis datus, nobis natus ex intacta Virgine, et in mundo conversatus, sparso verbi semine, sui moras incolatus miro clausit ordine. | Of a pure and spotless Virgin born for us on earth below, He, as Man, with man conversing, stayed, the seeds of truth to sow; then He closed in solemn order wondrously His life of woe. |
In supremae nocte coenae recumbens cum fratribus observata lege plene cibis in legalibus, cibum turbae duodenae se dat suis manibus. | On the night of that Last Supper, seated with His chosen band, He the Pascal victim eating, first fulfills the Law’s command; then as Food to His Apostles gives Himself with His own hand. |
Verbum caro, panem verum verbo carnem efficit: fitque sanguis Christi merum, et si sensus deficit, ad firmandum cor sincerum sola fides sufficit. | Word-made-Flesh, the bread of nature by His word to Flesh He turns; wine into His Blood He changes; what though sense no change discerns? Only be the heart in earnest, faith her lesson quickly learns. |
Tantum ergo Sacramentum veneremur cernui: et antiquum documentum novo cedat ritui: praestet fides supplementum sensuum defectui. | Down in adoration falling, Lo! the sacred Host we hail; Lo! o’er ancient forms departing, newer rites of grace prevail; faith for all defects supplying, where the feeble senses fail. |
Genitori, Genitoque laus et jubilatio, salus, honor, virtus quoque sit et benedictio: Procedenti ab utroque compar sit laudatio. | To the everlasting Father, and the Son who reigns on high, with the Holy Ghost proceeding forth from Each eternally, be salvation, honor, blessing, might and endless majesty. |
Amen. Alleluia. |
St. Thomas Aquinas, sequence for the Solemnity of the Body and Blood of Christ and processional for the Mass of the Lord’s Supper.
pymake: 25% faster than msys make
Thursday, April 2nd, 2009pymake news:
- Bad news: pymake is still 5x slower than GNU make on Linux/Mac.
- Good news: pymake is 25% faster than msys make (GNU make on Windows)!
- Best news: there’s a lot of room to make performance better.
All measurements are do-nothing depend builds. Full rebuilds aren’t significantly affected because compiler speed overwhelms any time we spend in make.
Creating Windows processes is more expensive than creating processes on a unix-like operating system. Creating MSYS processes is hugely more expensive. Windows I/O in general is slow compared to Linux, at least for typical build tasks. Because pymake recurses in a single process, caches parsed makefiles such as rules.mk, and avoids many shell invocations, it can make up for slow parsing times by dramatically reducing time spent elsewhere.
How to use pymake on Windows
Don’t use pymake with client.mk on Windows, yet. pymake doesn’t understand MSYS-style paths, which is what configure substitutes for @srcdir@ and @topsrcdir@ when using client.mk. This will be fixed by the patches available from this bug tree.
Configuring manually isn’t hard: to build Firefox in c:/builds, follow this recipe:
$ mkdir /c/builds $ hg clone http://hg.mozilla.org/mozilla-central /c/builds/mozilla-central $ cd /c/builds/mozilla-central $ autoconf-2.13 && (cd js/src && autoconf-2.13) $ mkdir ff-debug $ cd ff-debug $ export MAKE='python -O c:/builds/mozilla-central/build/pymake/make.py' $ ../configure --enable-application=browser --enable-debug --disable-optimize $ python -O ../build/pymake/make.py -j4
How to use pymake on Linux/Mac
Configure manually as above, or add the following flags to your mozconfig file:
export MAKE="python -O $topsrcdir/build/pymake/make.py" mk_add_options MAKE="python -O @TOPSRCDIR@/build/pymake/make.py"
Soon on all platforms this will be as simple as mk_add_options MOZ_ENABLE_PYMAKE=1
Thank you!
Special thanks to Arpad Borsos who wrote tests and an implementation of –keep-going for pymake.
Next plans
Immediate future plans for pymake reduce the process count even further, especially for depend builds:
Currently every invocation of nsinstall is a separate process, and we invoke nsinstall even when all its install targets are up to date. Simple tasks like this will instead be implemented as native python commands. Ted implemented a branch to do this, but the current implementation blocks the only thread. I think we’re going to switch and use shared-nothing threads and message passing to parallelize before making this the default behavior.
Every time Mozilla processes a makefile the build system combines all the compiler-generated dependencies into a single .all.pp file using mddepend.pl: this allows developers to move or remove header files without breaking depend builds. Running a perl script for every makefile invocation is silly, especially because all it does is parsing and rewrite makefile syntax. I will have pymake read these dependency files directly and ignore missing files (causing a rebuild without an error) using a syntax includedeps $(INCLUDEFILES)
Longer-term work that would make pymake much more useful:
- Build an object graph of the entire Mozilla tree recursively. I think I know how to do this, although there will be some issues with how to deal with local versus global variables.
- Warn and eventually force a more rigorous dependency graph: warn if a dependent file ‘appears’ without having a rule to create it.
- Make parsing a lot faster using mx.TextTools instead of native python regular expressions. Keep the regular expressions as a slow path for developers who don’t have TextTools installed.
Python Reference Cycles and Closures
While debugging pymake performance and memory usage I found an interesting fact, which in hind sight should have been obvious: functions which enclose themself in python create reference cycles which have to be cleaned up by the Python garbage collector:
def outerFunction(outerCallback): targetsToBuild = [1, 2, 3] def innerCallback(): if len(targetsToBuild): # innerCallback closes on itself... this creates a reference cycle every time you call outerFunction # if you call outerFunction 100000 times per build, this can add up really quickly and cause large GC pauses targetsToBuild.pop(0).build(innerCallback) else: outerCallback()
After finding this problem, I refactored (1, 2, 3) the pymake code to use objects instead of closures to save asynchronous state while rebuilding. Also, OptionParser instances create cycles by default. There is a lightly-documented method OptionParser.destroy which can be used to manually break these cycles (thanks to Ted for finding it). pymake now runs without creating any reference cycles and I disabled the python garbage collector.
Environment Munging in MSYS
When MSYS goes from an MSYS process to a Windows process, and vice-versa, it munges certain environment variables to account for the path styles. I previously thought that it only munged PATH, but I discovered today that I was wrong: MSYS was munging the MAKEFLAGS environment variable in odd ways.
If MAKEFLAGS in the MSYS process was ‘ -j1 — PATH=e:/builds/mozilla-central’ it would be munged into ‘ -j1 — PATH=e;c:/mozilla-build/msys/builds/mozilla-central’ in a non-MSYS process. Without the leading space the value was not touched. I don’t know why this is, but I altered the pymake code slightly so that MAKEFLAGS would never start with a space (and would be more compatible with gmake in the process).