3.7. What's the difference between the "fail" and "error" return status?

If a test failed, then the harness was able to actually run the test code. The failure could either indicate that the test truly failed (i.e. it threw an exception) or that there was some problem running the test due to security restrictions, lack of resources, etc.

If error is the final result of a test, then the harness was unable to run the test. An error is most commonly associated with problems in the test description (typos, missing required arguments, etc.).

In either case the result contains a short message intended to provide hints as to where the problem lies.