Spec-driven test suite

This is a somewhat detailed implementation plan for lynX's "Spec-driven test" idea on the Talk:Strategy page. The theory behind this proposal is that the client server and interserver interactions can be simplified for the purpose of testing if state information for the transactions being tested can be known and treated as inputs, thereby reducing the complexity of the test situation by treating each connection as a finite state machine.

Requirements for test client

The test client should be able to:

1. Initialize its state from a configuration file that defines connection and test information as psyc variables.
2. Be capable of implementing a simple metalanguage (so that tests can be embedded in a plain text spec).
3. Make multiple simultaneous client connections to instances of the server being tested.
4. Perform psyctext variable substitution on the spec text to generate the packets it needs to send and the responses it expects to receive.
5. Provide transaction logging and diff output comparing the responses received on each connection to those expected.
6. Permit a lexical engine to be replaced by a state comparison without a major rewrite (if the initial version does not implement PSYCstate).
7. Parse routing and create a new client connection for client roles defined in the spec.

The following features are desirable but may be deferred if necessary to expedite an initial release of the test client:

1. Implement the metalanguage (embeddable aspect of the spec) requirement.
2. Compare state changes communicated by packets rather than making direct lexical comparisons.
3. Implement a listening interface to permit the test client to emulate a server.

The test client that does not implement these would still be useful for developing the spec using the current psyced (below).

Requirements for spec development

The spec should:

1. Provide a list of vars to be defined for a given test and a human readable description for each.
2. Provide a psyctext transcript of each connection anticipated for a given test.
3. Order the spec in a way that successful tests initialize the state for subsequent tests.
4. Label individual test specifications as obsolete, deprecated, current, future or experimental with an emphasis on maintaining compatibility with the current spec.

Procedure

This section describes boot-up and maintenance for an on-going specification and testing process.

1. List the features and commands to be tested from the client.
2. Draft a client-side script (psyc packets) to be to be used for the test.
3. Formalize the metalanguage and markup to be used for spec documents.
4. Prepare client.
5. Freeze features.
6. Prepare spec (client scripts only).
7. Run client spec in test client to generate psyc packets for server and inter-server dialog.
8. Debug spec.
9. Freeze spec.
10. Debug reference server.
11. Repeat as necessary or desired

See also

There is an interesting tool, formerly known as idx-tsunami, now called Tsung, coming from the ejabberd company.. http://tsung.erlang-projects.org/