How should tests report their outcome?
All tests in the test suite should report their outcome in a consistent manner, making it easy for humans or computer programs to understand and to process them. The following test states, defined by EARL (the Evaluation And Report Language), have proved useful: • cannotTell • fail • notApplicable • notTested • pass The more information (within reason) reported by failing tests, the more useful the tests. If your users know that one test out of one thousand fails, but they don’t know what it was testing or why it failed, that is not very helpful. If they know what the test was testing, what behavior it was expecting from the implementation under test, and how the implementation failed to conform to these expectations, this makes it easier for users to find and fix the problem. The more useful your test suite is, the more it will be used. See, for example, the CSS1 Test Suite which describes clearly the output that can be expected from each test. The HTML 4 Test Suite uses a similar app
