Regression Testing

General rules

Keep tests small, simple and fast. It doesn't help, if one test takes 10 hours.

Results of tests should be reproduceable. Thats not always possible, but at n runs it should return n times the same.

The game has to be available with a rapid tag

Single tests can be run on demand, means: (when EngineTesting=1)

/runtest 1

will run test one.

How to create a test

A test will be run on a spring-headless, which means it has no visual output.

The test has to be active, when the modoption "EngineTesting = 1" in script.txt is set, it should default to 0.

In the lua widget / gadget, check if this tag is set, and then start the tests.

A test consits of four parts:

Setup

spawn units, create features, ... initialize. move/reclaim/... command to units

Run

mostly the engine will do the work. maybe the run a is just some lua function.

End condition

- test run > frames
- unit died
- unit created

...

Validate

- test if unit reached goal - test if all units are idle / did their work - ...

Results should be shown this way when it failed:

Spring.Log(string.format("Test %d result: failed",testnumber, result), LOGLEVEL.ERROR)

or on success:

Spring.Log(string.format("Test %d result: ok",testnumber, result))