Debugging sync errors

Development < Debugging Sync Errors

Desync happens when simulation gives different results for different clients. Currently it is detected by means of checksumming crucial data on every write; this checksum is then transferred to the host, who compares them to his own checksum.

For purposes of debugging, the sync debugger stores every such write with some debugging information. When a desync happens, the write history is compared on every client, who then sends locations where computation has given different results. The location is then resolved to line number of the engine source, which should provide information about the nature of the desync.

Compiling

To enable sync debugging, you have use a special build of Spring.

Using buildbot

Go to http://springrts.com:7778/builders/full-windows set "Property 1 Name" to "config" and "value" to "syncdebug" If the build works, an executable will be uploaded to the usual place; at the time of this writing, it's http://springrts.com/dl/buildbot/syncdebug/. The generated archive will have [syncdebug] in its name.

Self-Compile

1. To compile it, first get the same revision from git, configure with
   • cmake -DCMAKE_BUILD_TYPE=DEBUG2 -DTRACE_SYNC=true -DSYNCDEBUG=true

Running

1. Host a game. Make sure every player who joins has this exe; it's not needed to have it at the time of joining the battle, but when the game starts, everyone must have it. Expect random undefined behaviour otherwise.
2. If you want engine source lines, the host has to download two more files: debug symbols and addr2line. Debug symbols are generated and uploaded by buildserv; as of now, they can be found at http://springrts.com/dl/buildbot/syncdebug/ . addr2line is a part of MinGW32 binutils package.
   1. Download debug symbols
   2. Download addr2line
   3. Extract both to where your syncdebug-enabled spring.exe lives
   4. Rename debug symbols to spring.dbg

Only the host needs to download addr2line and debug symbols.

When everybody is in game, test your setup by enabling cheats (/cheat) and having one of the players do a test desync (/desync). The game should pause and syncdebugger should do its work. If the game pauses, but you won't see SD messages, it's likely that one of the clients doesn't have the correct exe.

In addition to syncdebugger logs, syncdebug engine builds also have synctrace enabled. This option makes spring.exe write a file called trace<playernum>.log which logs important simulation events in a diff-friendly format. After a desync, get this file from the desynced player(s) and do a diff -u yourtrace histrace to find the first logged sync mismatch. The real desync probably happened somewhere in between the two log lines.

If you're a game developer, please be aware that Lua may be a source of desyncs. E.g. table iteration using pairs when you have tables, coroutines, or functions as keys is not a sync-safe operation, see mantis #1050 for example of such a desync.

available commands=

/desync This command modifies some game data which causes the game to desync. Use this to test the sync debugger.

/fakedesync This just triggers the sync debugger, without desyncing the game. The sync debugger will print an error “Huh, all checksums are equal?!?” to the logfile after issuing this command. Use this to test the sync debugger. Server only.

/reset Reset the sync debugger to it initial state. Use this if you want to continue playing after the sync debugger finished outputting the debugging info.

Post processing

All previously required postprocessing has now been built into the sync debugger, so this step is quickly done: just check that syncdebug-server.log looks sane and if it does, send it to the team.