2011-01
In an earlier article I explained
the role CSS2.1 and how powerful and useful a CSS2.1 test suite could be for
the Web.
In this article I'm going to explain the current situation in CSS testing
at W3C, what my goals are for each aspect of the testing process, and what's
missing to get us there. It's an update of my 2009 edition.
Submitting Tests
The goal here is to allow multiple people to submit and update tests in
ways that are straightforward and easy for them, and to store the tests in
a way that both works for W3C and is compatible with what our contributors
want to do with the tests elsewhere.
We currently have:
- A standard format
for tests that includes the metadata we need to index the tests and
generate useful reports.
- A Subversion server to collect
tests and track their development
- Submissions from HP, Microsoft, Mozilla, and individual volunteers.
- A volunteer (Gérard Talbot)
who is dedicated to collecting tests from the web authoring community.
What we're missing:
- A friendlier way to submit tests. Subversion has a bit of a learning
curve; a WebDAV+Subversion setup might be easier for contributors if it
is capable of transparently handling file moves and renames. A customized
Web interface to SVN might be another option.
- Increased participation from Opera and WebKit's quality assurance teams: they
are already writing tests for internal use, but their efforts are
totally disconnected from W3C, which means they don't benefit
the Web community as a whole. Establishing collaboration at the CSS2.1
level would also carry over into CSS3 modules, where the availability of
their tests would allow faster adoption of new CSS3 technologies.
- Awareness of W3C test formats at Mozilla and sync-up between W3C and
Mozilla's test repositories. (Mozilla contributors already write tests
in reftest format.)
- Best practices and examples for converting tests to
self-describing reftests
format. The reftest format allows script-driven testing for many
(though not all) CSS features, which makes test result reporting
much faster. Self-describing tests are easy to analyze by a human
and can catch reftest false positives when both the test and its
reference fail in the same way.
- Reftestification: For historic reasons, most of the tests in
the CSS2.1 test suite are self-describing tests that are not
reftests. Converting the tests to reftest format would make it
easy for browser teams to run the tests as part of their usual
QA process, and to keep their results in the W3C database
up-to-date. (Self-describing reftests are the official format
for CSS3 tests.)
- Automatic builds of the test suite, so that contributors can
see their work in between snapshot publications.
Publishing Tests
The goal here is to present the tests for each module on an official W3C
server in the formats needed to run on all CSS implementations; to have the
tests indexed so that it is easy for a developer to find the tests he is
interested in running; and to have a coverage report that shows which parts
of the spec are tested and which tests are missing.
What we have:
- A place to host tests.
- An official test format that requires several bits of information:
- Links to the specific sections of the spec(s) being tested.
- The title of the test, intended for a table of contents.
- Optionally, a statement explaining what, specifically, the
test is asserting if passed.
- A set of build scripts that can pull both
reftests and
self-describing tests
from multiple directories, index their metadata, convert them
to XHTML and HTML output, and merge them into a single test
suite publication.
What we're missing:
- A more understandable human index, since the design of the
current one is a little obscure.
- Database-driven test searches.
- An easy way to find the source file of a given generated test.
- A publication system that can gracefully handle intersecting
test suites. (For example, many, but not all, CSS2.1 tests are
also CSS3 Backgrounds tests, and many, but not all CSS3 Backgrounds
tests are also CSS2.1 tests.)
Reviewing Tests
The goal here is to make it straightforward and easy for reviewers to
review and comment on the tests they're interested in reviewing and to
keep track of the required next steps for each test.
What we have:
- A review process
and checklist.
- Not much else, other than some experience showing that mailing lists
and wikis are not efficient ways to track test reviews.
What we're missing:
- A way to track and search by the status of submitted tests: whether they
have been reviewed, need to be reviewed, have errors, need further work,
are waiting for spec clarifications, etc. Right now I get questions like
"I want to help review some tests, which tests should I review?" and I can't
come up with an answer. It would also allow people to see which tests are
unreliable, or wrong, and either improve them or skip them when evaluating
an implementation.
- An efficient way to submit and track review comments associated with a test.
- Mailing lists allow easy submission, but can't track. If the test
author is not subscribed and engaged when the comment comes in, it
just gets lost in the archives and may or may not get found the next
time someone works with that test.
- Bug trackers allow easy tracking, but are difficult for commenters.
For one thing, it's not clear how to map problems with a test into
the bug report model. Some of that can maybe be fixed by heavily
customizing the bug tracker's templates, but they're just not designed
for a test-reviewing workflow. For example, we've noticed that the same
comment often applies to an entire series of tests (or even to
all tests submitted by a particular contributor), and a system with one
comment form per test is very inefficient for dealing with that. On the
flip side, filing a bug per issue rather than a bug per test makes it
hard for a contributor sit down with a particular test, integrate all
the feedback, and fix the test.
- An automated way to catch machine-checkable errors. Many format violations
are machine-checkable, and having those checked by machine would free up
reviewers' time to focus on the aspects of the test that must be
human-verified.
Ideally we want it to be:
- Easy for anyone to submit comments on errors they find in the test suite.
- Efficient for a dedicated reviewer to work straight through a list of
related tests.
- Straightforward for a test contributor to find all the comments that
pertain to his tests, improve them as necessary, and report back the
changes to his reviewers for confirmation.
What we're thinking:
Melinda Grant (HP), Eira Monstad, Peter Linss (HP / CSSWG Chair) and I
spent some time working out the requirements and design
for a system. One of the key ideas was to integrate with the Subversion
repository so that changes there automatically show up in the test's record
in the system. Another idea was to detect and cache the metadata in the tests
to make it possible to search for tests on a particular topic.
We have a half-finished prototype of such a system, but it
hasn't been getting much priority.
Collecting and Reporting Test Results
The goal for collecting is to make it easy and efficient to collect accurate
test results for as many implementations as we can, both for the human-verified
tests and the reftests.
The goal for reporting is to generate up-to-date reports from the test results
that are useful and insightful for implementors, web designers, and W3C.
What we have:
- A test harness
that allows submission of test results for the human-verified tests by
simply clicking the Pass/Fail/Skip buttons associated with a test shown on
the screen. (It records the UA string and the result.) A subset of tests
can be run and you can stop at any time, so it's not required to run
thousands of tests before submitting any results.
- Some rudimentary output capabilities in the test harness for
reporting those results.
What we're missing:
- Designs for useful reports that we could create from the test results,
that would scale well to thousands of tests and myriad implementations.
Peter Linss has created the bare minimum the CSSWG needs to prepare
a spec for REC, but there is a lot of useful data in the databases
that is currently not being useful to either browser developers or web
designers.
- Code that automatically creates those reports from the collected test
results.
- Easy and straightforward ways to generate and incorporate scripted test
results into the harness result data.
Get Involved
The CSS Working Group's wiki has
more information on CSS testing and how you can get involved.
