Regression Testing

Regression tests are made to not let things break again. If something broke, a regression test should be made.

General rules

Keep tests small, simple and fast. It doesn't help, if one test takes 10 hours.

Results of tests should be reproduceable. Thats not always possible, but at n runs it should return n times the same.

The game has to be available with a rapid tag

Single tests can be run on demand, means: (when testsrun=1) Also, tests can be disabled with "testsdisable=1,2,3", which will disable test 1,2 and 3.

/runtest 1

will run test one.

also, tests should be written, that they can run in parallel. tests shouldn't rely on a specific map.

modoptions

testsrun=1
testsdisable=1,2,3
test1areax = <map cordinate>
test1areay = <map cordinate>

How to create a test

A test will be run on a spring-headless, which means it has no visual output.

The test has to be active, when the modoption "EngineTesting = 1" in script.txt is set, it should default to 0.

In the lua widget / gadget, check if this tag is set, and then start the tests.

A test consits of five parts:

Setup

spawn units, create features, ... initialize. move/reclaim/... command to units

Run

mostly the engine will do the work. maybe the run a is just some lua function.

End condition

- test run > frames
- unit died
- unit created

...

Validate

- test if unit reached goal
- test if all units are idle / did their work
- ...

Results should be shown this way when it failed:

Spring.Log(string.format("Test %d result: failed",testnumber, result), LOGLEVEL.ERROR)

or on success:

Spring.Log(string.format("Test %d result: ok",testnumber, result))

Cleanup

• remove created units
• unset variables

Example Gadget

links

http://springrts.com/phpbb/viewtopic.php?f=21&t=27928