Debian Versioning of Mozilla Libraries Harmful
It has come to my attention that the maintainer of the Debian Mozilla packages (Mike Hommey) is using patches that significantly alter the XULRunner linking strategy (first discussed here).These patches cause various Mozilla libraries to be soversioned and installed in the library path (/usr/lib). Although I have emailed Mr. Hommey privately on a number of occasions, I have not received a reply, and the linking strategy that he has implemented is poorly thought out and will lead to instability and frustration.
Background
On Linux systems, traditional shared libraries are installed into the system path (typically /usr/lib). In addition, there is a mechanism to support versioning, such that multiple versions of a library can be installed at the same time. This mechanism records information about the symbols provided by the library: incompatible changes are marked by a revision of the library’s major version number, backwards-compatible additions are marked by a change in the library’s minor version number, and minor bugfixes are marked by a revision of the micro (or release) version number. When an application is linked against a shared library, the version numbers are stored in the library and the dynamic linker ensures that when the application is loaded a library that is compatible is loaded.
Mozilla is not just ABI
For libraries where the provided functionality matches the exported functions, this simple solution works well. But the Mozilla libraries are an entirely different situation: XPCOM libraries export a small set of frozen APIs, but the stability of these initialization and accessor APIs has little to do with the stability of the actual functionality provided by gecko (XULRunner). Almost all XPCOM functionality is exposed through interfaces, which are versioned using UUIDs. As functionality is completed, these interfaces and the contractIDs that allow access to components are frozen, which guarantees that they will not change in future releases. Thus the functionality exposed by Mozilla is a set of frozen functionality exposed through interfaces, and a much larger set of nonfrozen functionality.
XPCOM also has the ability to load an extensible binary XPCOM components: these components, if carefully written, can work with multiple versions of XPCOM and the various applications written on top of XPCOM. It is important that XPCOM components can continue to be compiled which work with multiple applications and versions. It is also important that when these component are loaded they use the XPCOM symbols from the currently running version of XPCOM library; if these components were to load an alternate XPCOM library, they would quickly crash or at least fail in unexpected ways.
The Proven Solution
In order to allow multiple versions of XPCOM to coexist correctly on a machine, Mozilla has developed a linking strategy which allows applications to find a compatible version of XPCOM (XULRunner) and load it at runtime. This is called the standalone XPCOM glue. The standalone glue linking strategy has the following characteristics:
- Multiple versions of XULRunner are installed in separate directories.
- System XULRunners are typically installed in /usr/lib/xulrunner-1.8.0.1, /usr/lib/xulrunner-1.8.0.2, etc. Each version is registered in a *.conf file in /etc/gre.d.
- XULRunner can also be installed in a per-user location, typically ~/.lib/xulrunner-1.8.0.1 etc, and is registered in a configuration file within ~/.gre.d.
- An application which wants to use XPCOM APIs does not link directly against libxpcom.so (which would not be found, because it is not in the default library search path). Instead, it links against the standalone glue (libxpcomglue.a), a static library without mozilla shared library dependencies. During startup, the application calls the function GRE_GetGREPathWithProperties to locate a compatible XULRunner. It then calls XPCOMGlueStartup(), which dynamically loads the correct XULRunner libraries.
A great deal of effort was put into developing this linking strategy so that it actually works properly; and it does work properly on all three of the major platforms that Mozilla supports (Windows, Mac OS X, and Linux). It allows for relocatable installs of both the XULRunner platform and the applications that depend on the platform, which is an important part of Mozilla’s user-centric software model. XULRunner packages which use this linking model have been prepared on SuSE and other distros.
Debian Going Their Own Way
Unfortunately, the Debian pacakagers have decided to go their own way and try to create XPCOM libraries which can be versioned and installed in the system library path. This decision is ill-conceived: it means that XPCOM components built for other Linux systems are almost guaranteed not to work correctly on Debian systems: they will in many cases load the wrong XPCOM library and fail spectacularly. It also means that it will be impossible to compile XPCOM components on Debian systems that will work with multiple versions of an application, or on any other Linux system.
I am sure that the Debian team believes that this change in linking strategy is better and correct and that they are not acting maliciously; but the fact remains that they have not proposed or discussed their changes with me (as both build-sytem and embedding API owner in Mozilla), nor have they responded to my mails. I don’t think I have any choice but to publicly point out the significant flaws in their linking strategy and encourage Debian via peer pressure to rethink their poorly thought out decision.
February 22nd, 2006 at 8:48 pm
Looks like he’s has trouble communicating with you guys too:
http://web.glandium.org/blog/?p=48
February 22nd, 2006 at 9:20 pm
Post should be updated as “Debian Mozilla packages” = Debian Mozilla Suite maintained by Takuo KITAME, and Debian’s Firefox is maintained by Eric Dorland and Mike Hommey(co-maintainer).
February 22nd, 2006 at 9:24 pm
Nevertheless, all the best, Debian team!
February 23rd, 2006 at 2:14 am
Benjamin, I only received one mail and am on my way to write you one. It just takes a bit more time that I’d like. Or I’ll just do it on my blog…
February 23rd, 2006 at 5:07 am
Perhaps he is fixing it to be more unix like.
http://web.glandium.org/blog/?p=71
February 23rd, 2006 at 9:52 am
“I am sure that the Debian team believes that this change in linking strategy is better and correct” – sounds like “I know you think you’re right, but you’re not” – maybe when building apps to follow the OS’s conventions wouldn’t be a bad idea, and it seems they are just trying to do squeeze “your” libs to fit
February 23rd, 2006 at 10:03 am
“I know you think you’re right, but you’re not.” is exactly what I’m saying. The OS convention in this case produces incorrect results and is therefore wrong.
February 23rd, 2006 at 10:27 am
Sorry, but making the xulrunner libraries incompatible with hypothetical binaries built on other systems (which would fail to run anyway for random reasons as your build system is utterly fragile) is a minor issue at best.
I’ve stated it many times, and I repeat it: the mozilla developers haven’t proven in any way that they understand what is an ABI and how to install software properly on an Unix system. The entire build system was built on Windows, for Windows, without any thought about integrating in an all-in-one Unix system. Glandium’s work allows us to build packages that integrate seamlessly, like they already do for all other libraries. We can now envision a decent upgrade path and security support for the future. It is incompatible with other systems, so what? The sonames are Debian-specific anyway (hence the d at the end of the soname) and non-Debian binaries won’t even notice them.
I know you think your build system is good, but it’s not. Throwing away all standards to come up with something incompatible and fragile is not the right thing to do. We’re not going to abandon 30 years of Unix history just because a group of Windows programmers comes up with a “revolutionary” solution to problems they created at first. A library is a library. It has an interface, a SONAME, and installs with other libraries. Please keep your XPCOM talk for marketing guys.
February 23rd, 2006 at 11:44 am
Debian routinely “improves” on upstream in ways that break compatibility with the rest of the Linux world and sometimes breaks the software itself in subtle ways. Many, many projects have suffered from this ridiculous mentality, which extends even to patching security “holes” that don’t exist.
Put bluntly, I would expect nothing less from them. They care nothing for respecting the decisions of others and have proven it time and time again. The correct solution is to bypass them entirely – provide XULRunner packages yourself, from mozilla.org, and recommend Debian users install that instead.
February 23rd, 2006 at 11:52 am
Oh, and Josselin, the ELF soname versioning scheme is far, far from being well designed. I’ve seen many different versioning schemes over the last few years, and have designed one myself, and ELF library versioning is so far from being perfect it’s not even funny.
A few of the more obvious problems:
* The linker cannot tell what minor version of a library is needed, without symbol versions. But symbol versions are a proprietary GNU extension, virtually undocumented and as a result almost nothing uses them. Adding them to libraries retroactively is extremely hard. Worse, it’s a terribly over-engineered design for such a simple thing.
* The soname system prevents a binary from being able to “I need version 1.2.3 or above”, for instance because it relies on bugfixes on that version. As a result, packagers have to guess and often get this wrong. Worse, because of the first problem I described, Debians shlibdeps system just assumes that whatever is installed at the time the package is built is the minimum required version, leading to package dependencies that bear no relation to reality.
* They exist in an unmanaged namespace, and the UNIX culture encourages short, incomprehensible names. Conflicts occur, and when they do distributions (assuming they package the software at all) rename them with no co-ordination so further breaking binary compatibility.
* Symbols exist in a global scope, meaning that multiple versions can’t correctly co-exist in a single process image.
Basically, ELF is shit, and claiming the new system MozCorp have designed sucks because it’s “marketing talk” reveals that you don’t know what you’re talking about.
February 23rd, 2006 at 12:10 pm
Mike,
1) This is one of the problems the shlibs system solves.
2) Package dependencies can be overriden to a lower version if needed; in the real world, it is never needed.
3) Soname conflicts? That’s quite infrequent, and when it happens it is most of the time caused by a code fork.
4) Hey, didn’t you tell a few lines earlier about symbol versioning?
You seem to forget ELF has one major advantage over the rest: it works. A simple system that works is much superior to a complex system that doesn’t work. Having had to deal with mozilla’s random ABI breakages, I wouldn’t say XPCOM (or whatever contains the Mozilla ABI) works correctly. Not for what *we* want to achieve.
February 24th, 2006 at 2:32 am
Cross platform compatibility is essential. Debian folks’ bug fixes and other compatible improvements are appreciated. However, their changes to package interfaces that break compatiblity with upstream are *not*. Breaking cross-platform compatibility for the sake of cross-package similarity is a horrible idea. It fractures the user community, creates separate pockets of knowledge and experience for each system, and dramatically increases the costs of cross-platform support.
It doesn’t matter what downstream integrators think about ABIs, SONAMEs, XPCOM, build systems, or UNIX history. The only important “standard” is the upstream package interface, and throwing it away to come up with something incompatible and fragile is not the right thing to do. We’re not going to abandon the efficiencies of the mass market just because a group of Debian integrators comes up with “revolutionary” plan to make all the packages within their system look the same. A package is a package. It has an interface. Period. Note that this interface *includes* the full names by which files are accessed. So if you add SONAMEs to libraries that don’t have SONAMEs, or if you change names of the form /usr/lib/xulrunner-1.8.0.1/foo to /usr/lib/foo, you’re breaking the interface. Don’t do that.
Benjamin and Mike Hearn: You are indeed right. Stick to your guns. Mozilla Corporation should build its own cross-platform compatible Debian packages and recommend that Debian users install these instead.
February 25th, 2006 at 12:38 pm
Josselin,
1) shlibs doesn’t solve this problem at all, because it replaces precise and correct versioning in the dynamic linker with imprecise and frequently incorrect knowledge inside the package manager, which may not be used (for instance, commercial games, source code installs etc).
shlibs also has no way of knowing that a particular library is being dlopened and so optional.
2) Package dependencies *can* be overriden, but in practice, 3rd party packagers are often unaware of these requirements and miss them out. Yes, I’m talking from experience here, Benjamin isn’t the first one to have this problem with Debian. You said “In the real world, it is never needed” and I’m struggling to believe this. I can think of several cases in which the exact version of a program is critical to know/control.
3) Soname conflicts between entirely separate projects may be infrequent but they can happen. If you want an example of a binary name conflict in Debian look at epiphany the card game vs epiphany the browser. Worse, when distributions patch packages, or compile them differently, the same Soname can in fact refer to different exported ABIs. Look at the mess libpython unicode support is in for an example of that … it basically makes Python programs that contain C extension modules entirely unportable.
4) Symbol versioning is hardly ever used in practice. Even when it is use (typically only by the GNU project) it is hard to get right. Look at libstdc++, it uses symbol versioning but this versioning is worthless because unversioned symbols can still enter the process image via header inlines. If even the GNU guys can’t get this right, what makes you think symbol versioning is going to be a solution anytime soon. I’d prefer global application of Michael Meeks’ direct linkage patches, but unfortunately this was ignored by the glibc people (surprise surprise) and is now a SUSE extension :(
I’m not convinced ELF works. I have too much experience of dealing with cases where it falls flat on its face to believe this. In Debian you mostly work around these problems either with huge hacks (shlibdeps), or by breaking binary compatibility with the rest of the Linux community (eg library renaming, downstream application of symvers, patching/overriding upstream source).
As to Mozilla ABI breakages … the biggest problems I’ve seen with this are apps that actually build entirely new browsers on top of the Mozilla codebase. Do you seriously expect MozCorp to freeze so much that major surgery on their codebase (completely new GUI for instance) will never break? Are there many cases of an XPCOM interface that has been marked as frozen changing accidentally, or do these breakages only affect programs that used unfrozen interfaces knowingly?
February 26th, 2006 at 6:25 am
Alan, you haven’t understood anything of what I wrote, have you? I see you’re trying to be funny by mimicking my message, but the very idea of implying that we break some compatibility shows you should document yourself first. Why? Because there is no such thing as cross-platform compatibility for Mozilla. For example, trying to launch a distribution’s epiphany binary with another distribution’s Mozilla installation is doomed to fail. It will fail because the Mozilla ABI is way too fragile. It changes without warning from time to time, and even when it doesn’t change, it will break because of C++ ABI changes.
Oh, and I love that idea of providing packages of *libraries* and recommend our users to use them instead of those that are required by the packages they will install. Please go ahead and do this. Thanks to the Debian-specific versioning, they won’t ever be noticed by installed packages and won’t do any harm.
Mike, your lack of understanding of how things are done in Debian is simply insulting.
February 26th, 2006 at 6:36 am
(oops, I clicked on submit too fast)
To follow on the reply to Mike’s babbling: we’ve been making changes to library packages for a very long time, and not breaking upstream compatibility, especially for LSB packages, is at top priority when making such changes. We’ve been adding symbol versioning where it was needed, without breaking this compatibility. Any kind of breakage with upstream interfaces is considered a bug, most of the time a release-critical bug. If we have to render ourselves incompatible, we use specific SONAMEs. Which was done by glandium for the xulrunner packages.
And the important thing to us is of course applications that build entirely new browsers on top of the Mozilla codebase. More important than a few applications written in XUL that are so widespread that nobody even asked for their inclusion. Our default browsers are Epiphany and Konqueror, and with an increasing number of GNOME applications relying on gtkmozembed, it became a library we couldn’t go on treating specifically.
February 27th, 2006 at 7:12 am
Hi,
These are all some general remarks, completely biased by the fact I’m a Debian developer (read package maintainer).
1. This whole discussion isn’t very productive in its current form, but it’s a good summary of various gripes Debian people had with Mozilla stuff, and Mozilla people had with Debian stuff.
2. I find it a bit insulting though that the Mozilla people imagine the ELF and the shlibs systems so broken when it has been used in a scale as large as Debian.
3. From an user perspective, Mozilla has a long track of security issues, and of incompatibilities between minor releases. The efforts glandium pushed should help a lot in these respects.
4. It might be a bit sad not to discuss the architecture of a new library layout for xulrunner with upstream prior to using it in Debian. The current discussion shows that it wouldn’t have been integrated upstream anyway, glandium made the effort to implement it fully in Debian, with the relevant Galeon, Epiphany, Devhelp, Yelp and other patches to support his claim that it was feasible and useful.
5. I’m astonished that Mozilla doesn’t consider the XPCOM components they ship *not* covered by ABI. It’s 100% compatible or it’s not. You have to update other applications or not.
6. The shlibs system is fine, I don’t see any valid critic of it’s work besides the dlopen() issue. shlibs work perfectly when SONAMEs are correct. shlibs works perfectly with symbols versionning, with API additions, and with versionned libraries. shlibs can be overriden if that’s needed (at both ends, provider and consumer). The net result is that you’re guaranteed to be able to launch an application when you’ve installed it: at the very least at the ELF level, but usally it covers all levels, no dependency hell, no missing packages, no version issue.
Concerning the dlopen() remark, the behavior of the application can vary so much in handling of the dlopen() that you can not make a general assummption on the type of dependency this is. For example the application might popup for configuration of a module’s repository, can fetch it remotely, can miserably fail; I don’t think it makes sense to cover such things in the shlibs system, or any other dependencies management solution, it is simply the job of the package maintainer to handle this manually or to write a system to handle such deps if these are numerous.
7. The compatibility between GNU/Linux distros is a valid point, which should disappear once upstream is convinced the Debian system is the best one, or vice-versa.
8. Please, could someone give a complete and detailed example of how the system chosen in Debian would fail miserably and the Mozilla system wouldn’t? If Mozilla people can think of an usal scenario which is clearly not covered by the current Debian xulrunner layout but is by their’s, it might give some substance to judge that the Debian layout is not worth it.
9. I’m curious of why static linking is needed: what’s the advantage instead of using a libxpcompglue shared library?
Cheers,
February 28th, 2006 at 2:16 pm
Josselin: Sorry if my last message came across a bit rude. This wasn’t intended. I understand what you’re saying and I understand your frustration with Mozilla’s changing interfaces.
I think you misunderstood me—I’m not talking about cross-platform compatibility in terms of relocating compiled packages from one distribution to another. I’m talking about it in a more general sense: a package with a particular name should behave the same way on all platforms. This ensures that if I write a package that depends on it, my package will compile and run on these platforms without patches or extra effort.
Ideally, Debian could simply change the package name (e.g. `dexulrunner’) and mozilla.org packages could be installed alongside. However, in practice, the FHS’s design requires a lot of extra work be done to prevent naming conflicts between the two.
March 1st, 2006 at 6:54 pm
As I understand it – and that’s at least the case for all GNOME packages – the build process isn’t changed with glandium’s packages. It’s just the matter of using –with-mozilla=xulrunner. The code behind is the same, what changes is how it gets linked. Any source code incompatibility would be unintentional, and don’t think that’s likely to happen.
March 2nd, 2006 at 2:36 pm
To clarify, I’m not a Mozilla person, but rather somebody who has extensive experience of dealing with binary (in)compatibility on Linux.
Josselin, I have specific examples of where Debian is not binary compatible with the rest of the world. How can you claim that Debian care oh so much about this when I have had to implement workarounds for your distribution so many times? Trivial example, you applied Debian specific downstream symbol versions to libpng. Now any program compiled on Debian that uses libpng and run on any other distro generates scary warnings. If any other distro does the same thing but chooses a different tag name, the binaries will break completely. Fedora is just as bad.
And for what it’s worth I do not misunderstand how things are done in Debian. You haven’t shown that, or even tried. I understand your policies with regards to this, and I think they’re wrong, and I’ve had to deal with the mess they’ve created many times before. Look at what happened with trying to make GTKspell dlopened – the Debian packagers complained that it would “break” shlibdeps and so shouldn’t be applied. What kind of insanity is that? Rather than improve Gaim they said it should be rejected simply to work around fundamental design faults in Debian.
The problems you guys have with Mozilla upgrades breaking Epiphany or Galeon are entirely your own fault. There is no law saying that there must be a single Mozilla installation on the system, programs that replace the UI like alternative browsers could easily have their own private copies. This is how it’s done on other platforms like the Mac. But you guys refuse to use such static linking, even though it has been suggested in the past, and instead bitch at MozCorp for not being perfectly compatible. Ridiculous.
And I’m sorry but I’ve made several specific points as to why ELF and shlibdeps does not work. Brushing them off as “Debian is large, so it must work” is not good enough.
March 2nd, 2006 at 2:47 pm
Incidentally this quote is very telling:
“Oh, and I love that idea of providing packages of *libraries* and recommend our users to use them instead of those that are required by the packages they will install. Please go ahead and do this. Thanks to the Debian-specific versioning, they won’t ever be noticed by installed packages and won’t do any harm.”
This is the second time now I’ve seen a Debian packager say basically “You do what you like, we’ll override you anyway and you can’t stop us, so f$*k you”. Which is what you’re saying here Josselin, don’t try to deny it.
This attitude makes me EXTREMELY ANGRY. It is NOT your software and you do NOT have carte blanche to modify it as you package it against upstreams wishes. If you truly believe your way is better than make a bone-fide fork, with a new name, new website, and so on. Then see if people use your XULRunner or MozCorps XULRunner. Anything else is simply underhand and not acceptable.
March 3rd, 2006 at 12:47 pm
[…] The ugly. Some distribution people, notably Debian developers, outright hate us or insult us. This is part of a wider theme of upstream/downstream tension. Debian is by far the worst here and deserves to be singled out – their packagers routinely anger high profile developers, and the practice of subtly breaking software continues unabated. When autopackage 1.0 was released and we got a lot of media attention, one Debian developer implied that they would deliberately add autopackage runtimes modified against our wishes to their repositories. This behaviour is unacceptable, but is the status quo and as such is not seriously challenged. […]
March 4th, 2006 at 6:01 am
About libpng: being the one having applied symbol versioning to libpng, I can only say you’re living completely out of reality. The patch’s interoperatibility was extensively tested to prove that warnings are only warnings, and it hasn’t been done without upstream or other major distributors knowing it. As it is the ONLY sane way to provide smooth upgrades between major libpng versions, libpng’s upstream is finally considering to adopt it. Yes, even libpng is getting better than Mozilla.
About GTKspell: using dlopen on a library with a rapidly-changing ABI like GTKspell is indeed a very bad idea, especially because it breaks the holy interoperability you are promoting.
About Mozilla: if you want to make suggestions as ridiculous as static linking, please, go away. Mozilla security upgrades are already painful enough to deal with.
About overriding stupid upstream decisions: yes, we know we’re doing it, and as long as upstream maintainers will make wrong decisions we can work around, we’re still going to do it. We will not ship a broken system just to keep it like upstream. This is free software, and we do have carte blanche to modify it.
About the general feeling I have about you: if your idol is D.J. Bernstein, I doubt you truly understand what “quality releases”, “free software” or “security updates” mean.
March 4th, 2006 at 12:55 pm
> It is NOT your software and you do NOT have carte
> blanche to modify it as you package it against
> upstreams wishes.
Interesting theory. I don’t remember reading anything in the Mozilla license that requires permission to modify, but I’ll check again.
(Without taking a side on this ridiculous flame, neither side of which will ever move an inch, here’s a bit of perspective: if one is using Debian, they should be able to use the Debian package, and the packagers should do what they feel is necessary to make the package work in Debian. If you don’t want to take advantage of Debian’s packaging system, why on earth would you be using Debian?)
March 4th, 2006 at 6:57 pm
Huh? When did I say my idol was DJ Bernstein? Random.
March 5th, 2006 at 12:17 am
Sorry, someone posted something from DJB earlier, but it was Alan, not you.
November 20th, 2006 at 5:34 pm
I’ve had my share of Debian problems, some were easy some were difficult(time consumming)! I don’t use Debian anymore, not because it’s not a good distro, I just do not have the time to fix a package install ( even the one’s marked as “Debian”).
I do a lot of programming for linux and sometimes especially for reason of time versus cost I’ve used the gtkmozembed module and have recieved several complaints from the Debian community because my program doesn’t work with their Distro. All I can give is my apology, because I haven’t the time to go back and hack my programming for one Distro.
For this most just curse and flame with the “I’m not going to use your software and my friends won’t either”. To which I say “that’s your choice and it’s fine with me, their are a multitude of platforms it does work on”. Just as a side if anyone is interested, do a google on “segmentation fault wiith gtkmozembed” and look at the number of Debian
complaint forums that come up, don’t believe me or anyone else just do a simple google and you’ll see what I mean.
Probably get some flame for this but who cares. I create programs because I love programming and mostly because it puts a roof over my childrens heads, cloths on their backs and FOOD in their bellies,(watch for the “You need to go program for Microsoft and stay out of Linux FLAME”). I program for Linux because I believe IT IS a superior OS and someday it will knock Microsoft off its high horse.
Am I gripping at Debian…, NO. Debian is a Linux Distro and in the spirit of a Free and Open community it has the write to do things as they see fit. Will I and most other programmers who make a living at it put in more hours just to make our program work for one Distro…, I doubt it, bottom line is if it doesn’t work for your distro don’t use it you won’t hurt my feelings or my wallet, just don’t e-mail me to make changes just for your Distro when it works perfectly fine on many others, you see where I’m comming from.
(For a clarification, I’m a contract programmer, even though every program is licensed under the GPL, I write programs for Companies who would like their product to work on Linux and I tell them up front that some Distros’ are different but the program will work on a majority of Systems and they’re fine with it. Just as you wouldn’t go to work at your job for FREE neither will I. So considering the time with an agreed price this will determine if I need to use a shortcut library like gtkmozembed or gtkembedmoz depending. Then once the programs run on the clients machine the contract is completed and I receive the contract price).
June 11th, 2008 at 6:23 pm
[…] Mike on pretty much all his points. I googled for more details on this controversy and discovered this which explains some more details about the whole incident. As near as I can see, the Debian […]
June 18th, 2008 at 2:13 am
[…] strictly), and applied his own patches to fix it. This apparently led to a lot of problems, and a mozilla developer blamed debian for their poor collaboration with the upstream developers, and pointed out to debian […]
January 30th, 2009 at 7:13 am
I come away from reading this post and its comments with the distinct impression that people in the Debian project do not play well with others. In this exchange, each person associated with Debian resorted to taunting the other side, whereas the other interlocutors attempted to keep to the decision points of the discussion.