Cascading Test Sheets

By Ashok Argent-Katwala

Testing Web pages is a pretty complex task. Very often we settle for too little, checking little more than:

• that our markup validates against the spec
• that a simple link-checker doesn't find dead links
• & that some monkey-testing doesn't turn up any glaring errors

Those are good checks to make, but we need to do a lot better.

I want to make some declarative statements about what is expected of different pages, and have them run routinely. For complicated pages, that depend on user-supplied, database-held or offsite data, I'd like to run the tests on any pages I might ever ship, and give the administrators a decent stab at recreating the error and damn-well fixing it.

Thankfully, the CSS people have done a fine job of allowing you to pick out parts of an HTML document and then apply styles to them. Better yet, we've just about reached the point where smart designers can express what they want in CSS, without needing to write anything terribly complicated.

This is a little idea for how to do that. There's no implementation yet, but I'm looking for feedback on doing it this way. The basic idea is to express some useful, human-level tests in a CSS-like language that make sense to more than code-nerds, and use them to test individual Web pages, or entire sites, and be more confident that they do all that you expect, and nothing that you don't.

---

Before getting stuck into the full description, here's a short example. We use CSS selectors, but replace the properties with details of the test we want to run. The intent should be pretty obvious to anyone familiar with CSS:

 DIV.footer P {
 title: "Footer must have maintenance info";
 begins: "Maintained by";
 }
 
 DIV.footer P A[href^='mailto:'] {
 title: "Footer must have a mailto link.";
 }
 
 DIV.header A[href='/'] {
 title: "The header navigation must include a link to the home page.";
 }
 
 H2:not([id]) {
 title: "Every H2 must have an ID attribute set.";
 occurs: never;
 }

Or, in English:

• The first rule says that there must be a paragraph in the footer that begins with the text 'Maintained by'.
• The second rule insists that there is a mailto: link in a paragraph in the footer.
• The third rule checks that the header always links to the front of the site. Naturally, you wouldn't want that rule on the front page itself.
• The last rule checks that every H2 has an id set, perhaps because we're planning on linking to the sections of some long doc.

I'm using some parts of the CSS3 draft selectors here, and you would obviously want to be clear about what selectors your testing engine supported when writing your tests.

So what can you put in the body of one of these tests? The aim here is to keep the cleverness in the selectors as much as possible, then I can borrow the code for doing the matching from a decent free browser.

title required

Human-readable title. Remember that you will mainly see the titles when they fail, so you probably want to phrase them that way.

description

Single-paragraph description of the failure.

link

Link to better explanation of the requirement. In a sophisticated environment this would link back to the original requirement gathered. In my use I sometimes just link to an appropriate guideline or Web article making the point.

begins, ends, contains

Just what you expect. This tests the flattened textual content of the matched element, ignoring any intervening tags. This is straightforward string comparison, no clever globs or regular expressions. Including several of these mean you expect them all to be true to pass the test (together with any matches).

matches

Tests the content against a given regular expression. I expect this will be the usual 'extended' RE syntax of sed et al.

occurs

The number of occurences we need to find in the document to pass the test. Must be 'never', 'every' or a strictly positive integer. Defaults to 1, which means we expect to see a match (although there may be more than one). 'every' means that any tag matching the rule must also pass the conditions of the test for the whole test to pass.

limit

The maximum number of occurences we can find in the document and still pass the test. Must be a non-negative integer. Defaults to 0, meaning 'no limit'.

That's about it. It's a little cheeky to call these 'cascading' as we don't have the full cascade of CSS. For now I'm just using these locally, and not publishing link elements in the pages that point you to the tests; I think that works best. So, we can lose the author-, user- and browser-level rules, nor do we have the '!important' flag. The rules for specificity are probably too much for our purposes too, as we don't really want to accidentally fail to run one test because a different, more specific test has been added.

Let's suppose we have a Web site with a tree of pages like this where _username_ is a variable:

 /
 /about
 /users/
 /users/_username_/
 /users/_username_/recommends/

We could set up a similar tree for the test sheets, with the convention that _ could match a single part of the path. For example:

 tests/.cts
 tests/about.cts
 tests/users/.cts
 tests/users/_/.cts
 tests/users/_/recommends/.cts

This means you can take a URI like:

 http://example.com/users/bob/recommends/

and look in the tests directory for all the test sheets which 'match'. I rarely have GET parameters on my sites, but you could just remove them before generating the list of test sheets. In our example, the following tests would apply:

• tests/.cts
• tests/users/.cts
• tests/users/_/.cts
• tests/users/_/recommends/.cts

You would then apply them, in that order, most general first. If you came upon a later rule with the exact same selector, you take the later declaration. That allows you to have site-wide rules, and explicitly replace them later. This isn't so much a cascade, but a handy convention for generating one long list of tests for a particular page.

That's just the basic thinking on how this might work. I welcome your thoughts on how this might be useful for you. I plan on writing a simple implementation using one of the open & free browsers. Once I have that, I'll write it up more formally.