www.delorie.com/gnu/docs/greg/greg_7.html   search  
Buy GNU books!

Greg testing framework

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.4 A POSIX conforming test framework

This section is copied almost directly from the DejaGNU documentation with minor modifications -

Greg is believed to conform to the POSIX standard for test frameworks.

POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance.

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. Greg supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

A test has succeeded. That is, it demonstrated that the assertion is true.

POSIX 1003.3 does not incorporate the notion of unexpected passes, so for strict POSIX, PASS, instead of UPASS, is returned for test cases which were not expected to pass but did. This means that PASS is in some sense more ambiguous than if UPASS is also used.

A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework.

POSIX 1003.3 does not incorporate the notion of expected failures, so for strict POSIX, FAIL, instead of XFAIL, is returned for test cases which were expected to fail and did not. This means that FAIL is in some sense more ambiguous than if XFAIL is also used.

A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished.

Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. With Greg this is not a problem - any unexpected termination of a greg-testcase procedure will produce UNRESOLVED.

Here are some of the ways a test may wind up UNRESOLVED:

A test was not run. This is a placeholder, used when there is no real test case yet.

The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented.
Greg uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. For a Greg test suite to conform to POSIX 1003.3, you must set the variable greg-posix to be true (or run the `greg' command with the --posix flag). Doing this will ensure that non-posix extensions are not used.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

  webmaster     delorie software   privacy  
  Copyright 2003   by The Free Software Foundation     Updated Jun 2003