Rapid: Difference between revisions

From Spring
Jump to navigationJump to search
← Older edit
Abma (talk | contribs)
0 edits
Abma (talk | contribs)
0 edits
 
(51 intermediate revisions by 5 users not shown)
Line 1: Line 1:
===server-side===
"Rapid" is a concept for storing "archives" for spring. It consists of multiple indexes:


Made by det, source: https://github.com/det/BuildRapid rewrite in c++: https://github.com/spring/BuildRapidCpp
* main index of all known rapid repos: https://repos.springrts.com/repos.gz
* index of all known archives in a repo, i.e.: https://packages.springrts.com/versions.gz, https://repos.springrts.com/zk/versions.gz, ...
* index of a single game/map/content archive: .sdp which lists all files inside the "pool" directory


usage:
spring can directly read .sdp and the pool directory which contains the real files of an archive. multiple .sdp can share the same files in a pool which makes "rapid" so efficient in storage.
<code><pre>
./add-archive <file.sdz> </path/to/repo> <tag1 tag2 ...>
</pre></code>


filename of the .sdp is:
Serverside [[RapidTools]] are used to manage the repository of all games.
  md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)




Admin interface made by CarRepairer:
=Using rapid as Gamedev=
If a Gamedev wants to use rapid, he needs to use a git repository for his game where the game archive is in the .sdd format stored:


http://packages.springrts.com/
The git repository needs to contains a folder which is in the structre of a .sdd. in modinfo.lua version has to be
 
<pre>
===preparing the svn repository===
The svn repository needs to contains a folder which is in the structre of a .sdd. in modinfo.lua version has to be
<code><pre>
version = '$VERSION',
version = '$VERSION',
</pre></code>
</pre>
$VERSION later on will be replaced by the svn commit number or, when a release is made, the version number of the release.
$VERSION later on will be replaced by the git commit number or, when a release is made, the version number of the release. modinfo.lua needs to be lowercase!


===creating a release===
==Creating a release==
when the svn repository is connected to rapid, a release can be made by making a svn-commit with a commit message in the format:
When the git repository is connected to rapid, a release can be made by making a git-commit with a commit message in the format:
<code><pre>
<pre>
VERSION{x.xx}
VERSION{x.xx}
</pre></code>
</pre>
 
Where everything in the brackets will replace $VERSION in modinfo.lua.
 
This will create a tag :stable for this commit and create an archive at https://''gamename''.repo.springrts.com/builds/, i.e. https://i18n.repo.springrts.com/builds/ which can be distributed/uploaded to different sites if needed.
It is also automatically published to upq at https://springfiles.springrts.com/ which all current download systems have access to; you can validate what happened there, too.
 
===Adding your game to the rapid download system===
Please create a new issue at https://github.com/spring/RapidTools/issues and report the location of your git repository and the wanted rapid tag.


everything in the brakets will be replace $VERSION in modinfo.lua.
To download you can use one of the [[Rapid_Tags]].


this will create a tag :stable for this commit and create an archive at http://packages.springrts.com/builds/ which can be distributed/uploaded to different sites if needed.
=Files structures=
also it is automaticly published to upq at http://api.springfiles.com/ where all current download systems have access to, there you can validate what happened, too.


===general description===
file structure of rapid is:
Rapid is used to list all games in the (online-) repositry


as spring supports also reading such a structure, there are some docs in the [http://github.com/spring/spring/blob/master/rts/System/FileSystem/ArchivePool.h sourcecode]
<pre>
/versions.gz
/repos.gz
/packages/
/pool/
</pre>


here is a short description how it works:
==list of rapid repositories: repos.gz==
gziped text file which has a list of all rapid repositories.


take a game archive
<pre>
main,https://packages.springrts.com,,
</pre>
for springrts it is:
https://repos.springrts.com/repos.gz


atm the archive is a single blob
==list of archives on a mirror: versions.gz==
rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder.
gziped text file which has a list of all sdp archives in this repository.
games can share files as long as the md5 is the same.
<pre>
A game, is a binary file that is what can be seen in ~/.spring/pool/  
ba-tronic:version:7.12-tronic1,baafb4fb215c60873e7d10e7ec600dc5,Balanced Annihilation V7.12,Balanced Annihilation V7.12-tronic1
</pre>


~/.spring/packages containts all games with the associated md5s
the md5 hash is the filename of the sdp of the archive. so for example:


The filesize is useless, since it assumes same md5 == same file, ie assumes no collision.
https://packages.springrts.com/packages/baafb4fb215c60873e7d10e7ec600dc5.sdp


Then, there's a repository of games:
==folder with index archives: /packages/==
ie, a list of available games, and the pool files they need
this folder contains all available packages (~games) in the rapid file structure.
since now on, i'll refer to a game as the file which is list of md5 that it needs
unless specified otherwise
spring can read those games and pool folder directly
unpacking the files on the fly and reading them like it'd be a monolithic archive
then, there's yet another repository, which contains the list of repositories
if you picture in a tree graph
there's list of repositories on top
then splits to multiple repos
then each repos has multiple games


http://repos.springrts.com/repos.gz is the master
==an index archive: .sdp==
gziped index file with a list of all files in this archive. this is for example a game.


what a rapid downloader needs to do:
Format of the .sdp (index) files:
download the list of repositories
* There is one entry per indexed file. These are repeated until EOF.
for each repository
* The format of one such entry:
download the list of available games
** 1 byte real file name length
the each repo has also info about latest version of a game
** real file name
then, give a list of available games ( packages in linux package manager terms ), and eventually some meta package that represents the latest version of a game
** 16 byte MD5 digest
on download request, download the game descriptor
** 4 byte CRC32
parse it to get the list of md5
** 4 byte file size
check the available md5 local in the pool folder, and download the missing
* The 16 bytes MD5 digest is the reference to the 32 Hex char file name
upon download of each file, the rapid tool should deflate each archive and verify that the md5 is the same as the file
* under pool, which contains the content.
if not, reject it


if the test is successful, move the md5 files to the pool folder
filename of the .sdp is:
the game descriptor
belongs to the packages folder instead
with filename == md5 hash
and sdp extension
extra functionality, which is optional:
add possibility to delete games ( since the filenames use md5 hash, it's quite unclear to the user which files he should remove )
and a garbage collector which removes the files from the pool which aren't linked by game


simple reference counting should suffice for garbace collecting
  md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)


===in short===
rapid downloads the master file with all repos:
  http://repos.springrts.com/repos.gz
then it downloads from all repos /versions.gz
for example here http://packages.caspring.org/versions.gz


ca-1faction:revision:6355,054cf15829e79c357a3be50236e00c4f,Complete Annihilation test-6355,Complete Annihilation 1 faction test-6355
see also void PoolArchiveT::load(DigestT const & Digest):
the first section is the name: "ca-1faction:revision:6355"


the package will be downloaded from hash +.sdp
https://github.com/spring/RapidTools/blob/master/Rapid/PoolArchive.cpp#L35
so 054cf15829e79c357a3be50236e00c4f.sdp
that file belongs to the local packages folder
that file contains the list of md5
that the mod requires
so you should open it
and then check the local pool folder
for all the missing files


http://packages.caspring.org/packages/054cf15829e79c357a3be50236e00c4f.sdp
or
this file contains binary md5 + filename of the original file


remember: you must keep the file names untouched
https://github.com/spring/spring/blob/master/rts/System/FileSystem/Archives/PoolArchive.h
as in, everything is indexed by md5


to download the md5 files there's a streamer which will speed up the download
==the pool folder==
===adding your game to the rapid download system===
this folder contains all files referenced by the .sdp's in gzipped form. the filename is the md5 of the file. this way files which are used by multiple archives can be shared.
[http://springrts.com/phpbb/viewtopic.php?f=67&t=22077 forum thread]


The pool system uses two directories, to be found in the root of a spring
data directory, pool and packages. They may look as follows:
<pre>
/pool/00/00756ec29fe8fc9d3da9b711e76bc9.gz
/pool/00/3427d26f419dabe74eaf7b865407b8.gz
...
/pool/01/
/pool/02/
...
/pool/ff/
</pre>


To download you can use one of the [[Rapid_Tags]].
basicly its in this form:
 
<pre>
/pool/<first 2 hex chars>/<last 30 hex chars>.gz
</pre>
 
The splitting up into the 00 till ff sub-dirs is only to prevent possible
problems with file-system limits for maximum files per directory (eg. FAT32).


[[Category: Download Systems]]
[[Category: Download Systems]]

Latest revision as of 19:54, 19 September 2021

"Rapid" is a concept for storing "archives" for spring. It consists of multiple indexes:

spring can directly read .sdp and the pool directory which contains the real files of an archive. multiple .sdp can share the same files in a pool which makes "rapid" so efficient in storage.

Serverside RapidTools are used to manage the repository of all games.


Using rapid as Gamedev

If a Gamedev wants to use rapid, he needs to use a git repository for his game where the game archive is in the .sdd format stored:

The git repository needs to contains a folder which is in the structre of a .sdd. in modinfo.lua version has to be 

version = '$VERSION',

$VERSION later on will be replaced by the git commit number or, when a release is made, the version number of the release. modinfo.lua needs to be lowercase!

Creating a release

When the git repository is connected to rapid, a release can be made by making a git-commit with a commit message in the format: 

VERSION{x.xx}

Where everything in the brackets will replace $VERSION in modinfo.lua.

This will create a tag :stable for this commit and create an archive at https://gamename.repo.springrts.com/builds/, i.e. https://i18n.repo.springrts.com/builds/ which can be distributed/uploaded to different sites if needed. It is also automatically published to upq at https://springfiles.springrts.com/ which all current download systems have access to; you can validate what happened there, too.

Adding your game to the rapid download system

Please create a new issue at https://github.com/spring/RapidTools/issues and report the location of your git repository and the wanted rapid tag.

To download you can use one of the Rapid_Tags.

Files structures

file structure of rapid is: 

/versions.gz
/repos.gz
/packages/
/pool/

list of rapid repositories: repos.gz

gziped text file which has a list of all rapid repositories. 

main,https://packages.springrts.com,,

for springrts it is: https://repos.springrts.com/repos.gz

list of archives on a mirror: versions.gz

gziped text file which has a list of all sdp archives in this repository. 

ba-tronic:version:7.12-tronic1,baafb4fb215c60873e7d10e7ec600dc5,Balanced Annihilation V7.12,Balanced Annihilation V7.12-tronic1

the md5 hash is the filename of the sdp of the archive. so for example:

https://packages.springrts.com/packages/baafb4fb215c60873e7d10e7ec600dc5.sdp

folder with index archives: /packages/

this folder contains all available packages (~games) in the rapid file structure.

an index archive: .sdp

gziped index file with a list of all files in this archive. this is for example a game.

Format of the .sdp (index) files:

  • There is one entry per indexed file. These are repeated until EOF.
  • The format of one such entry:
    • 1 byte real file name length
    • real file name
    • 16 byte MD5 digest
    • 4 byte CRC32
    • 4 byte file size
  • The 16 bytes MD5 digest is the reference to the 32 Hex char file name
  • under pool, which contains the content.

filename of the .sdp is: 

 md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)


see also void PoolArchiveT::load(DigestT const & Digest):

https://github.com/spring/RapidTools/blob/master/Rapid/PoolArchive.cpp#L35

or

https://github.com/spring/spring/blob/master/rts/System/FileSystem/Archives/PoolArchive.h

the pool folder

this folder contains all files referenced by the .sdp's in gzipped form. the filename is the md5 of the file. this way files which are used by multiple archives can be shared.

The pool system uses two directories, to be found in the root of a spring data directory, pool and packages. They may look as follows: 

/pool/00/00756ec29fe8fc9d3da9b711e76bc9.gz
/pool/00/3427d26f419dabe74eaf7b865407b8.gz
...
/pool/01/
/pool/02/
...
/pool/ff/

basicly its in this form: 

/pool/<first 2 hex chars>/<last 30 hex chars>.gz

The splitting up into the 00 till ff sub-dirs is only to prevent possible problems with file-system limits for maximum files per directory (eg. FAT32).

Retrieved from "https://springrts-wiki.p2004a.com/mediawiki/index.php?title=Rapid&oldid=15611"