2009-11/12

A Belorussian translation of this article is provided by Patric Conrad.

Imagine: A world where CSS support across browsers creates a stable, consistent platform for web design. A world in which you can develop in one browser and reasonably expect your design to reproduce truly in all other browsers running with your system settings. A world where every new release of a feature matches the spec precisely and is compatible with all other existing implementations.

Our access to this world is interoperability testing. With a solid spec foundation for the CSS standards, a comprehensive test suite to measure it, and most of all, true collaboration among browser developers and web designers to create it, we can make the CSS specs and test suites a powerful tool in improving the interoperability of layout engines and web sites everywhere.

My goal in this article is to give some background on the roles CSS2.1 and its test suite play on the Web, and why those roles are important. Then I want to show how powerful and useful a CSS2.1 test suite could be for the Web if we took on the challenge of making it a valuable resource for implementors and web authors as well as for the W3C.

What is a standard?

To quote the British Standards Institution:

A standard is an agreed, repeatable way of doing something. It is a published document that contains a technical specification or other precise criteria designed to be used consistently as a rule, guideline, or definition. Standards help to make life simpler and to increase the reliability and the effectiveness of many goods and services we use. Standards are created by bringing together the experience and expertise of all interested parties such as the producers, sellers, buyers, users and regulators of a particular material, product, process or service.

At W3C, a "Recommendation" is a completed standard. Its goal is to state the precise criteria that define an agreed, repeatable way of doing something—in our case, processing a particular web technology like CSS.

What is interoperability?

The IEEE defines interoperability as the ability of two or more systems or components to exchange information and to use the information that has been exchanged. At W3C it means that when two or more systems are given the same input (e.g. HTML+CSS source code) under the same conditions (e.g. screen res, color depth, font availability, and viewport size), they produce the same output.

Interoperability among CSS implementations is the Holy Grail of CSS2.1. It means the ability to test your web design in one browser on your system, and automatically get the same result in all the other browsers on your system. This is worth pursuing.

The W3C REC Track Process

In the early days of the World Wide Web Consortium, a spec in the Working Draft stage became a full-fledged Recommendation simply after passing the Last Call for Comments gauntlet. Specs are like software, however: you don't notice all the bugs until you start trying to use them. It became clear that W3C specs needed a testing phase to prove that they were interoperably implementable. Thus the Candidate Recommendation phase was created.

To become a full W3C Recommendation, a Candidate Recommendation must prove itself by passing a series of tests: two independent implementations must pass a test for every testable assertion in the spec. In other words, the Working Group must prove that every thing their spec says the browser must do could be programmed correctly by two independent software teams. If the spec is self-contradicting, lacks enough detail, or requires Web-incompatible behavior that the implementors refuse to implement, it won't pass the Candidate Recommendation phase.

What is CSS2.1?

CSS2.1 is the CSS Working Group's attempt to put CSS2 and CSS1 through the Candidate Recommendation phase, correcting it in response to experience actually using the spec. The process has uncovered hundreds of problems in the original Recommendations. In a real sense, even though CSS2 is a Recommendation and CSS2.1 is only a Candidate Recommendation, CSS2.1 is a far more mature specification.

CSS2.1 is important to the Web because it defines the foundation of CSS as a technology. Every CSS3 module builds on top of the definitions in CSS2.1: CSS as a whole is dependent on CSS2.1 as a platform. Instability in this platform could be simply a spec issue, but more often it shows up as rendering inconsistencies across browsers.

What's the difference between a Test Suite and an Acid Test?

An acid test like Acid 1 and Acid 2 is designed to highlight a few major errors and key missing features in a single, intensely complex test page. It's not a very comprehensive test of the technology, but often triggers focus on areas that have been poorly implemented in the past.

A test suite on the other hand, is intended to be a comprehensive battery of tests for every aspect of the specification. Since CSS combinations are infinite, a CSS test suite can never be 100% complete. But it can set a baseline for support across all of the specification's features, and ideally that baseline includes all of the common cases that authors would need to rely on.

Why build a CSS2.1 test suite at W3C?

Well, the simple answer is that building a test suite at W3C allows the completion of CSS2.1's candidacy stage. Microsoft by itself could probably put together a test suite and implementation reports that are just enough to complete the W3C requirements. But to actually build a test suite that is intrinsically useful to the entire Web community takes collaboration, not just contribution.

Besides allowing the completion of CSS2.1's candidacy stage, building a test suite through collaboration at W3C can provide:

• A standard way to measure CSS2.1 support across implementations. No more vague claims of CSS2.1 support: we can measure conformance against the test suite as an agreed-upon yardstick. With adequate tools for reporting results, the CSS2.1 test suite can become a game for implementors to compete in.
• A central repository to house tests. There are many tests on sites scattered throughout the Web. A W3C test suite provides a well-known place to collect them so that they can all be found in one shot and run on a common testing platform.
• Wide review of tests for correctness. Currently each browser team must rely on its own internal experts to evaluate bug reports and their testcases against the spec. Submitting tests through W3C does not replace that process, but because the tests are shared with other browser teams and CSS authors and experts, they are exposed to a much wider review. Team-internal assumptions can be noticed and corrected much earlier in the implementation process if the team publishes its tests for public review rather than waiting for people to notice bugs after a release.
• Better communication on bugs between authors and implementors. The CSS authoring community can submit tests demonstrating CSS bugs for inclusion in the test suite, and these tests are then automatically on the radar for all CSS implementations. On the flip side, implementors' pass/fail reports can communicate to authors which features are well-implemented and which are still a little shaky (and under exactly which conditions they're shaky).
• Broader and deeper coverage of the technology. Different testers focus on different things, and approach testing in different ways. By combining multiple test collections into a single pool, we get a test suite with more comprehensive coverage.
• Reduced duplication of effort. By collaborating on a common test suite instead of each creating their own, implementors get a better test suite with less effort. This means they can spend those saved resources on improving their implementation and thus the state of CSS support on the Web.

Testing at W3C is an opportunity to create more interoperability and better communication throughout the web community, and collaborating on test suites can make them more powerful and useful to everyone, authors and implementors alike. By taking full advantage of the opportunity, we can build the test suites at W3C into a fundamental tool for improving CSS as a technology and the Web as a platform.

Interested? Next up, I'm going to report the current status of our testing infrastructure, outline a plan for the future of CSS testing, and explain what's missing to get us there.