Download Systems

The Lobbys currently use two download systems: Torrent and Rapid

Sorry, this file is from a chatlog (+slightly improved) use it only as a start.

Rapid

server-side

Made by CarRepairer:

http://packages.springrts.com/

Or svn, developed by licho & det[THE]threat (instantly creates the rapid entry after an svn commit)

To get+compile the server-side-tools:

sudo aptitude install subversion libextlib-ocaml-dev libxml-light-ocaml-dev libzip-ocaml-dev libpcre-ocaml-dev libsvn-dev svn co svn://svn.caspring.org/trunk/tools/build_ca cd build_ca

clients

• SpringDownloader
• Rapid (python): http://pypi.python.org/pypi/rapid-spring/

general description

Rapid is used to list all mods in the (online-) repositry

as spring supports also reading such a structure, there are some docs in the sourcecode

here is a short description how it works:

take a mod archive

atm the archive is a single blob rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder. mods can share files as long as the md5 is the same. A mod, is a binary file that is what can be seen in ~/.spring/pool/

~/.spring/packages containts all mods with the associated md5s

The filesize is useless, since it assumes same md5 == same file, ie assumes no collision.

Then, there's a repository of mods:

ie, a list of available mods, and the pool files they need
since now on, i'll refer to a mod as the file which is list of md5 that it needs
unless specified otherwise
spring can read those mods and pool folder directly
unpacking the files on the fly and reading them like it'd be a monolithic archive
then, there's yet another repository, which contains the list of repositories
if you picture in a tree graph
there's list of repositories on top
then splits to multiple repos
then each repos has multiple mods

http://repos.caspring.org/repos.gz is the master

what a rapid downloader needs to do: download the list of repositories for each repository download the list of available mods the each repo has also info about latest version of a mod then, give a list of available mods ( packages in linux package manager terms ), and eventually some meta package that represents the latest version of a mod

on download request, download the mod descriptor
parse it to get the list of md5
check the available md5 local in the pool folder, and download the missing
upon download of each file, the rapid tool should deflate each archive and verify that the md5 is the same as the file

if not, reject it

if the test is successful, move the md5 files to the pool folder

the mod descriptor
belongs to the packages folder instead
with filename == md5 hash
and sdp extension
extra functionality, which is optional:
add possibility to delete mods ( since the filenames use md5 hash, it's quite unclear to the user which files he should remove )
and a garbage collector which removes the files from the pool which aren't linked by mod

simple reference counting should suffice for garbace collecting

in short

rapid downloads the master file with all repos:

 http://repos.caspring.org/repos.gz

then it downloads from all repos /versions.gz for example here http://packages.caspring.org/versions.gz

ca-1faction:revision:6355,054cf15829e79c357a3be50236e00c4f,Complete Annihilation test-6355,Complete Annihilation 1 faction test-6355 the first section is the name: "ca-1faction:revision:6355"

the package will be downloaded from hash +.sdp so 054cf15829e79c357a3be50236e00c4f.sdp that file belongs to the local packages folder that file contains the list of md5 that the mod requires so you should open it and then check the local pool folder for all the missing files

http://packages.caspring.org/packages/054cf15829e79c357a3be50236e00c4f.sdp this file contains binary md5 + filename of the original file

remember: you must keep the file names untouched as in, everything is indexed by md5

to download the md5 files there's a streamer which will speed up the download

Torrent

The torrent system can be used to download a specific file by filename

first you request the list of available files to plasma server:

(http://planet-wars.eu/PlasmaServer/Service.asmx)

iirc it uses XML + SOAP it will give a list of archive names + path to the torrent for a given file you download the torrent and query the server for the http paths as in, will return a list of http mirrors of the file

the global list
gives a link to the torrent
the single file query
gives a list of http mirrors for the file itself
ie the map or te mod

so basically, it works like this:
get the list of files from the server
dowload the torrent file
at the same time request the list of http seeds
for the same file
open the torrent file as soon as it's downloaded
and start it
as soon as the http seeds arrive
add them to the torrent itself
so it will use them as additional source

this will speed up greatly the download