The danger of extensions (16:25 -0800)

I've been concerned for a while about the quality of Firefox extensions, and I'd like to explain why.

To start, I think one of the original motivations for having an extensions system in Firefox was to reduce demands for feature additions that are only used by a small number of people or are experimental. I have no problem with this. What I'm concerned about is that extensions are being promoted to large numbers of users as one of the advantages of Firefox. I think this may come back to haunt the Mozilla community.

I'm under the impression that extensions are used quite commonly, perhaps even beyond the more technical users of Firefox. This means that problems with extensions change user perception of quality of Mozilla products as a whole. But extensions are not of the same quality as the applications they extend.

But what do I mean by quality? Everybody has different definitions. And that's actually the key point here. It could mean that the program does what it's supposed to do. Or that its interface is easy for users to understand. Or that it is free of security holes. Or that it doesn't use excessive amounts of memory, or leak memory over time. Or that it is fast. Or that it fits well with the conventions of the operating environment (Windows, MacOS, GNOME). Or many other things.

The Mozilla applications are developed in an environment with tools and conventions that lead to openness and accountability. This helps different people work on improving different measures of quality and thus improves user perception of overall quality. The source tree is accessible in many ways, people can easily monitor changes being made to it (while keeping in mind the reputations of those making the changes), and most of those changes have been discussed publicly on bugs and code reviewed by reviewers who are accountable for their reviews. It's generally clear what's changing, why it's changing, and who is responsible for those changes. And enough people (eyeballs) with these different ideas of quality pay attention to current changes that lapses below the expected quality in many of these areas are often caught quite quickly.

Furthermore, with the Mozilla source code all in one repository, it is possible to improve the level of quality in one of these areas or in a new area. It's easy to search the entire tree for uses of strcat that cause buffer overflows. It's easy to search the entire tree for memory leaks caused by calls to addObserver not matched by calls to removeObserver. And so on, for many of the measures of quality that I listed above.

Extensions don't benefit from these community mechanisms. There's no central source code repository for all the extensions hosted on addons.mozilla.org. There's no general mechanism for monitoring all the changes. There's no way of knowing who reviewed the code of an extension, although I've heard there is a review process before extensions can be listed. (Review for what? But remember that formal pre-checkin review is only a small part of the review that happens for the Mozilla core and applications.)

Thus the pressures to improve extension quality come from fewer sources: the programming knowledge of just the author and the reviewer, and feedback about issues the extension's users know are related to the extension rather than the application (generally only its correct operation and user interface). Thus many dimensions of quality will be missed. And many of these dimensions can be spoiled by one rotten apple. If a user has ten extensions installed, one of them crashes often, and one of them leaks a lot of memory, then the user's perception of the application as a whole will be that it crashes a lot and leaks a lot of memory.

There are further problems with testing. Just as some people look over source code for a specific type of problem, other people test in specific areas, such as speed, memory use, various types of testing for security problems, or just basic functioning and usability. Extensions pose a different problem here: there's a combinatorial explosion of configurations to test. The person who cares enough about memory leaks to do a bit of memory leak testing might not have the leakiest extension installed. The person who tests for security bugs using mangleme might not have the extension with the most serious vulnerabilities installed.

And this problem gets even worse because extension versions are separate from application versions. There's no need to test the View Source window from Firefox 1.0.3 with the browser from Firefox 1.5. But users can use different versions of an extension with the same version of the browser, so each combination might get only a fraction of the total combined testing.

And it's worse still because extensions can interfere with each other. In the most obvious case, two extensions might try to add a button in the same place and cause broken user interface, if only because the buttons don't all fit anymore. But the same thing could happen with memory leaks, or slowdowns loading pages, or any of the other measures of quality. I doubt there's been much, if any, serious testing of memory leaks, performance, and many other important dimensions of quality across combinations of extensions.

This combinatorial explosion leads to large numbers of small problems. And large numbers of small problems are less likely to be noticed and fixed than small numbers of large problems.

I worry that these aren't just theoretical concerns. How many of the reports of huge memory leaks in Firefox 1.5 are because N of the M different versions/forks of AdBlock leak every page you visit? (I've been told N is nonzero, but haven't tested it for myself.) And I worry that the Mozilla community is staking too much of its communal reputation on something that the community as a whole has too little control over.