Rapid: Difference between revisions
fix svn url |
mod/mods -> game/games |
||
| Line 26: | Line 26: | ||
here is a short description how it works: | here is a short description how it works: | ||
take a | take a game archive | ||
atm the archive is a single blob | atm the archive is a single blob | ||
rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder. | rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder. | ||
games can share files as long as the md5 is the same. | |||
A | A game, is a binary file that is what can be seen in ~/.spring/pool/ | ||
~/.spring/packages containts all | ~/.spring/packages containts all games with the associated md5s | ||
The filesize is useless, since it assumes same md5 == same file, ie assumes no collision. | The filesize is useless, since it assumes same md5 == same file, ie assumes no collision. | ||
Then, there's a repository of | Then, there's a repository of games: | ||
ie, a list of available | ie, a list of available games, and the pool files they need | ||
since now on, i'll refer to a | since now on, i'll refer to a game as the file which is list of md5 that it needs | ||
unless specified otherwise | unless specified otherwise | ||
spring can read those | spring can read those games and pool folder directly | ||
unpacking the files on the fly and reading them like it'd be a monolithic archive | unpacking the files on the fly and reading them like it'd be a monolithic archive | ||
then, there's yet another repository, which contains the list of repositories | then, there's yet another repository, which contains the list of repositories | ||
| Line 47: | Line 47: | ||
there's list of repositories on top | there's list of repositories on top | ||
then splits to multiple repos | then splits to multiple repos | ||
then each repos has multiple | then each repos has multiple games | ||
http://repos.springrts.com/repos.gz is the master | http://repos.springrts.com/repos.gz is the master | ||
| Line 54: | Line 54: | ||
download the list of repositories | download the list of repositories | ||
for each repository | for each repository | ||
download the list of available | download the list of available games | ||
the each repo has also info about latest version of a | the each repo has also info about latest version of a game | ||
then, give a list of available | then, give a list of available games ( packages in linux package manager terms ), and eventually some meta package that represents the latest version of a game | ||
on download request, download the | on download request, download the game descriptor | ||
parse it to get the list of md5 | parse it to get the list of md5 | ||
check the available md5 local in the pool folder, and download the missing | check the available md5 local in the pool folder, and download the missing | ||
| Line 64: | Line 64: | ||
if the test is successful, move the md5 files to the pool folder | if the test is successful, move the md5 files to the pool folder | ||
the | the game descriptor | ||
belongs to the packages folder instead | belongs to the packages folder instead | ||
with filename == md5 hash | with filename == md5 hash | ||
and sdp extension | and sdp extension | ||
extra functionality, which is optional: | extra functionality, which is optional: | ||
add possibility to delete | add possibility to delete games ( since the filenames use md5 hash, it's quite unclear to the user which files he should remove ) | ||
and a garbage collector which removes the files from the pool which aren't linked by | and a garbage collector which removes the files from the pool which aren't linked by game | ||
simple reference counting should suffice for garbace collecting | simple reference counting should suffice for garbace collecting | ||
Revision as of 02:37, 23 November 2012
server-side
Made by CarRepairer:
http://packages.springrts.com/
Or svn, developed by licho & det[THE]threat (instantly creates the rapid entry after an svn commit) http://code.google.com/p/zero-k/source/browse/trunk#trunk%2Ftools%2Fbuild_ca
To get+compile the server-side-tools:
sudo aptitude install subversion libextlib-ocaml-dev libxml-light-ocaml-dev \
libzip-ocaml-dev libpcre-ocaml-dev libsvn-dev liblua5.1-0-dev
svn co http://zero-k.googlecode.com/svn/trunk/tools/build_ca
cd build_ca
./add-archive <file.sdz> </path/to/repo> <tag1 tag2 ...>
filename of the .sdp is:
md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)
general description
Rapid is used to list all games in the (online-) repositry
as spring supports also reading such a structure, there are some docs in the sourcecode
here is a short description how it works:
take a game archive
atm the archive is a single blob rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder. games can share files as long as the md5 is the same. A game, is a binary file that is what can be seen in ~/.spring/pool/
~/.spring/packages containts all games with the associated md5s
The filesize is useless, since it assumes same md5 == same file, ie assumes no collision.
Then, there's a repository of games:
ie, a list of available games, and the pool files they need since now on, i'll refer to a game as the file which is list of md5 that it needs unless specified otherwise spring can read those games and pool folder directly unpacking the files on the fly and reading them like it'd be a monolithic archive then, there's yet another repository, which contains the list of repositories if you picture in a tree graph there's list of repositories on top then splits to multiple repos then each repos has multiple games
http://repos.springrts.com/repos.gz is the master
what a rapid downloader needs to do: download the list of repositories for each repository download the list of available games the each repo has also info about latest version of a game then, give a list of available games ( packages in linux package manager terms ), and eventually some meta package that represents the latest version of a game
on download request, download the game descriptor parse it to get the list of md5 check the available md5 local in the pool folder, and download the missing upon download of each file, the rapid tool should deflate each archive and verify that the md5 is the same as the file
if not, reject it
if the test is successful, move the md5 files to the pool folder
the game descriptor belongs to the packages folder instead with filename == md5 hash and sdp extension extra functionality, which is optional: add possibility to delete games ( since the filenames use md5 hash, it's quite unclear to the user which files he should remove ) and a garbage collector which removes the files from the pool which aren't linked by game
simple reference counting should suffice for garbace collecting
in short
rapid downloads the master file with all repos:
http://repos.springrts.com/repos.gz
then it downloads from all repos /versions.gz for example here http://packages.caspring.org/versions.gz
ca-1faction:revision:6355,054cf15829e79c357a3be50236e00c4f,Complete Annihilation test-6355,Complete Annihilation 1 faction test-6355 the first section is the name: "ca-1faction:revision:6355"
the package will be downloaded from hash +.sdp so 054cf15829e79c357a3be50236e00c4f.sdp that file belongs to the local packages folder that file contains the list of md5 that the mod requires so you should open it and then check the local pool folder for all the missing files
http://packages.caspring.org/packages/054cf15829e79c357a3be50236e00c4f.sdp this file contains binary md5 + filename of the original file
remember: you must keep the file names untouched as in, everything is indexed by md5
to download the md5 files there's a streamer which will speed up the download
