|
|
| (49 intermediate revisions by 4 users not shown) |
| Line 1: |
Line 1: |
| The Lobbys currently use two download systems: Torrent and Rapid
| | There are curently three systems for spring content files: |
| | *[[rapid]] is currently for games only (rsync like) |
| | *[[upq]] allows to search for games maps and engine |
| | *[[plasma]] allows to search for games and maps |
|
| |
|
| | These systems allow to automaticly download files. |
|
| |
|
| Sorry, this file is from a chatlog (+slightly improved) use it only as a start.
| | ==clients/implementations== |
| | | *[[SpringLobby]] (c++, cross-platform) |
| ==Rapid== | | *[[chobby]] (lua, cross-platform) |
| (developed by licho & det)
| | *[[pr-downloader]] (c++, cross-platform) |
| | | [[Category: Download_Systems]] |
| ===Clients===
| |
| *SpringDownloader | |
| *Rapid (python): http://pypi.python.org/pypi/rapid-spring/
| |
| | |
| | |
| Rapid is used to list all mods in the (online-) repositry
| |
| | |
| as spring supports also reading such a structure, there are some docs in the [http://github.com/spring/spring/blob/master/rts/System/FileSystem/ArchivePool.h sourcecode]
| |
| | |
| here is a short description how it works:
| |
| | |
| take a mod archive
| |
| | |
| atm the archive is a single blob
| |
| rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder.
| |
| mods can share files as long as the md5 is the same.
| |
| A mod, is a binary file that is what can be seen in ~/.spring/pool/
| |
| | |
| ~/.spring/packages containts all mods with the associated md5s
| |
| | |
| The filesize is useless, since it assumes same md5 == same file, ie assumes no collision.
| |
| | |
| Then, there's a repository of mods:
| |
| ie, a list of available mods, and the pool files they need
| |
| since now on, i'll refer to a mod as the file which is list of md5 that it needs
| |
| unless specified otherwise
| |
| spring can read those mods and pool folder directly
| |
| unpacking the files on the fly and reading them like it'd be a monolithic archive
| |
| then, there's yet another repository, which contains the list of repositories
| |
| if you picture in a tree graph
| |
| there's list of repositories on top
| |
| then splits to multiple repos
| |
| then each repos has multiple mods
| |
| | |
| http://repos.caspring.org/repos.gz is the master
| |
| | |
| what a rapid downloader needs to do:
| |
| download the list of repositories
| |
| for each repository
| |
| download the list of available mods
| |
| the each repo has also info about latest version of a mod
| |
| then, give a list of available mods ( packages in linux package manager terms ), and eventually some meta package that represents the latest version of a mod
| |
| on download request, download the mod descriptor
| |
| parse it to get the list of md5
| |
| check the available md5 local in the pool folder, and download the missing
| |
| upon download of each file, the rapid tool should deflate each archive and verify that the md5 is the same as the file
| |
| if not, reject it
| |
| | |
| if the test is successful, move the md5 files to the pool folder
| |
| the mod descriptor
| |
| belongs to the packages folder instead
| |
| with filename == md5 hash
| |
| and sdp extension
| |
| extra functionality, which is optional:
| |
| add possibility to delete mods ( since the filenames use md5 hash, it's quite unclear to the user which files he should remove )
| |
| and a garbage collector which removes the files from the pool which aren't linked by mod
| |
| | |
| simple reference counting should suffice for garbace collecting
| |
| | |
| ===in short===
| |
| rapid downloads the master file with all repos:
| |
| http://repos.caspring.org/repos.gz
| |
| then it downloads from all repos /versions.gz
| |
| for example here http://packages.caspring.org/versions.gz
| |
| | |
| ca-1faction:revision:6355,054cf15829e79c357a3be50236e00c4f,Complete Annihilation test-6355,Complete Annihilation 1 faction test-6355
| |
| the first section is the name: "ca-1faction:revision:6355"
| |
| | |
| the package will be downloaded from hash +.sdp
| |
| so 054cf15829e79c357a3be50236e00c4f.sdp
| |
| that file belongs to the local packages folder
| |
| that file contains the list of md5
| |
| that the mod requires
| |
| so you should open it
| |
| and then check the local pool folder
| |
| for all the missing files
| |
| | |
| http://packages.caspring.org/packages/054cf15829e79c357a3be50236e00c4f.sdp
| |
| this file contains binary md5 + filename of the original file
| |
| | |
| remember: you must keep the file names untouched
| |
| as in, everything is indexed by md5
| |
| | |
| to download the md5 files there's a streamer which will speed up the download
| |
| | |
| ==Torrent==
| |
| | |
| The torrent system can be used to download a specific file by filename
| |
| | |
| | |
| first you request the list of available files to plasma server:
| |
| | |
| (http://planet-wars.eu/PlasmaServer/Service.asmx)
| |
| | |
| iirc it uses XML + SOAP
| |
| it will give a list of archive names + path to the torrent
| |
| for a given file
| |
| you download the torrent
| |
| and query the server for the http paths
| |
| as in, will return a list of http mirrors of the file
| |
| | |
| the global list
| |
| gives a link to the torrent
| |
| the single file query
| |
| gives a list of http mirrors for the file itself
| |
| ie the map or te mod
| |
| | |
| so basically, it works like this:
| |
| get the list of files from the server
| |
| dowload the torrent file
| |
| at the same time request the list of http seeds
| |
| for the same file
| |
| open the torrent file as soon as it's downloaded
| |
| and start it
| |
| as soon as the http seeds arrive
| |
| add them to the torrent itself
| |
| so it will use them as additional source
| |
| | |
| this will speed up greatly the download
| |
There are curently three systems for spring content files:
- rapid is currently for games only (rsync like)
- upq allows to search for games maps and engine
- plasma allows to search for games and maps
These systems allow to automaticly download files.
clients/implementations
