Rapid: Difference between revisions

From Spring
Jump to navigationJump to search
← Older edit
Abma (talk | contribs)
0 edits
Abma (talk | contribs)
0 edits
 
(50 intermediate revisions by 5 users not shown)
Line 1: Line 1:
===server-side===
"Rapid" is a concept for storing "archives" for spring. It consists of multiple indexes:


Made by det, source: https://github.com/det/BuildRapid rewrite in c++: https://github.com/spring/RapidTools
* main index of all known rapid repos: https://repos.springrts.com/repos.gz
usage:
* index of all known archives in a repo, i.e.: https://packages.springrts.com/versions.gz, https://repos.springrts.com/zk/versions.gz, ...
<code><pre>
* index of a single game/map/content archive: .sdp which lists all files inside the "pool" directory
./add-archive <file.sdz> </path/to/repo> <tag1 tag2 ...>
</pre></code>


filename of the .sdp is:
spring can directly read .sdp and the pool directory which contains the real files of an archive. multiple .sdp can share the same files in a pool which makes "rapid" so efficient in storage.
  md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)


Serverside [[RapidTools]] are used to manage the repository of all games.


Admin interface made by CarRepairer:


http://packages.springrts.com/
=Using rapid as Gamedev=
If a Gamedev wants to use rapid, he needs to use a git repository for his game where the game archive is in the .sdd format stored:


===preparing the svn repository===
The git repository needs to contains a folder which is in the structre of a .sdd. in modinfo.lua version has to be
The svn repository needs to contains a folder which is in the structre of a .sdd. in modinfo.lua version has to be
<pre>
<code><pre>
version = '$VERSION',
version = '$VERSION',
</pre></code>
</pre>
$VERSION later on will be replaced by the svn commit number or, when a release is made, the version number of the release.
$VERSION later on will be replaced by the git commit number or, when a release is made, the version number of the release. modinfo.lua needs to be lowercase!


===creating a release===
==Creating a release==
when the svn repository is connected to rapid, a release can be made by making a svn-commit with a commit message in the format:
When the git repository is connected to rapid, a release can be made by making a git-commit with a commit message in the format:
<code><pre>
<pre>
VERSION{x.xx}
VERSION{x.xx}
</pre></code>
</pre>


everything in the brakets will be replace $VERSION in modinfo.lua.
Where everything in the brackets will replace $VERSION in modinfo.lua.


this will create a tag :stable for this commit and create an archive at http://packages.springrts.com/builds/ which can be distributed/uploaded to different sites if needed.
This will create a tag :stable for this commit and create an archive at https://''gamename''.repo.springrts.com/builds/, i.e. https://i18n.repo.springrts.com/builds/ which can be distributed/uploaded to different sites if needed.
also it is automaticly published to upq at http://api.springfiles.com/ where all current download systems have access to, there you can validate what happened, too.
It is also automatically published to upq at https://springfiles.springrts.com/ which all current download systems have access to; you can validate what happened there, too.


===general description===
===Adding your game to the rapid download system===
Rapid is used to list all games in the (online-) repositry
Please create a new issue at https://github.com/spring/RapidTools/issues and report the location of your git repository and the wanted rapid tag.


as spring supports also reading such a structure, there are some docs in the [http://github.com/spring/spring/blob/master/rts/System/FileSystem/ArchivePool.h sourcecode]
To download you can use one of the [[Rapid_Tags]].


here is a short description how it works:
=Files structures=


take a game archive
file structure of rapid is:


atm the archive is a single blob
<pre>
rapid takes all the files in an archive it zips them separately and gives to each zip, as filename, the md5 of the file itself those files belong to the pool folder.
/versions.gz
games can share files as long as the md5 is the same.
/repos.gz
A game, is a binary file that is what can be seen in ~/.spring/pool/  
/packages/
/pool/
</pre>


~/.spring/packages containts all games with the associated md5s
==list of rapid repositories: repos.gz==
gziped text file which has a list of all rapid repositories.


The filesize is useless, since it assumes same md5 == same file, ie assumes no collision.
<pre>
main,https://packages.springrts.com,,
</pre>
for springrts it is:
https://repos.springrts.com/repos.gz


Then, there's a repository of games:
==list of archives on a mirror: versions.gz==
ie, a list of available games, and the pool files they need
gziped text file which has a list of all sdp archives in this repository.
since now on, i'll refer to a game as the file which is list of md5 that it needs
<pre>
unless specified otherwise
ba-tronic:version:7.12-tronic1,baafb4fb215c60873e7d10e7ec600dc5,Balanced Annihilation V7.12,Balanced Annihilation V7.12-tronic1
spring can read those games and pool folder directly
</pre>
unpacking the files on the fly and reading them like it'd be a monolithic archive
then, there's yet another repository, which contains the list of repositories
if you picture in a tree graph
there's list of repositories on top
then splits to multiple repos
then each repos has multiple games


http://repos.springrts.com/repos.gz is the master
the md5 hash is the filename of the sdp of the archive. so for example:


what a rapid downloader needs to do:
https://packages.springrts.com/packages/baafb4fb215c60873e7d10e7ec600dc5.sdp
download the list of repositories
for each repository
download the list of available games
the each repo has also info about latest version of a game
then, give a list of available games ( packages in linux package manager terms ), and eventually some meta package that represents the latest version of a game
on download request, download the game descriptor
parse it to get the list of md5
check the available md5 local in the pool folder, and download the missing
upon download of each file, the rapid tool should deflate each archive and verify that the md5 is the same as the file
if not, reject it


if the test is successful, move the md5 files to the pool folder
==folder with index archives: /packages/==
the game descriptor
this folder contains all available packages (~games) in the rapid file structure.
belongs to the packages folder instead
with filename == md5 hash
and sdp extension
extra functionality, which is optional:
add possibility to delete games ( since the filenames use md5 hash, it's quite unclear to the user which files he should remove )
and a garbage collector which removes the files from the pool which aren't linked by game


simple reference counting should suffice for garbace collecting
==an index archive: .sdp==
gziped index file with a list of all files in this archive. this is for example a game.


===in short===
Format of the .sdp (index) files:
rapid downloads the master file with all repos:
* There is one entry per indexed file. These are repeated until EOF.
  http://repos.springrts.com/repos.gz
* The format of one such entry:
then it downloads from all repos /versions.gz
** 1 byte real file name length
for example here http://packages.caspring.org/versions.gz
** real file name
** 16 byte MD5 digest
** 4 byte CRC32
** 4 byte file size
* The 16 bytes MD5 digest is the reference to the 32 Hex char file name
* under pool, which contains the content.


ca-1faction:revision:6355,054cf15829e79c357a3be50236e00c4f,Complete Annihilation test-6355,Complete Annihilation 1 faction test-6355
filename of the .sdp is:
the first section is the name: "ca-1faction:revision:6355"


the package will be downloaded from hash +.sdp
  md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)
so 054cf15829e79c357a3be50236e00c4f.sdp
that file belongs to the local packages folder
that file contains the list of md5
that the mod requires
so you should open it
and then check the local pool folder
for all the missing files


http://packages.caspring.org/packages/054cf15829e79c357a3be50236e00c4f.sdp
this file contains binary md5 + filename of the original file


remember: you must keep the file names untouched
see also void PoolArchiveT::load(DigestT const & Digest):
as in, everything is indexed by md5


to download the md5 files there's a streamer which will speed up the download
https://github.com/spring/RapidTools/blob/master/Rapid/PoolArchive.cpp#L35
===adding your game to the rapid download system===
[http://springrts.com/phpbb/viewtopic.php?f=67&t=22077 forum thread]


or


To download you can use one of the [[Rapid_Tags]].
https://github.com/spring/spring/blob/master/rts/System/FileSystem/Archives/PoolArchive.h
 
==the pool folder==
this folder contains all files referenced by the .sdp's in gzipped form. the filename is the md5 of the file. this way files which are used by multiple archives can be shared.
 
The pool system uses two directories, to be found in the root of a spring
data directory, pool and packages. They may look as follows:
<pre>
/pool/00/00756ec29fe8fc9d3da9b711e76bc9.gz
/pool/00/3427d26f419dabe74eaf7b865407b8.gz
...
/pool/01/
/pool/02/
...
/pool/ff/
</pre>
 
basicly its in this form:
 
<pre>
/pool/<first 2 hex chars>/<last 30 hex chars>.gz
</pre>
 
The splitting up into the 00 till ff sub-dirs is only to prevent possible
problems with file-system limits for maximum files per directory (eg. FAT32).


[[Category: Download Systems]]
[[Category: Download Systems]]

Latest revision as of 19:54, 19 September 2021

"Rapid" is a concept for storing "archives" for spring. It consists of multiple indexes:

spring can directly read .sdp and the pool directory which contains the real files of an archive. multiple .sdp can share the same files in a pool which makes "rapid" so efficient in storage.

Serverside RapidTools are used to manage the repository of all games.


Using rapid as Gamedev

If a Gamedev wants to use rapid, he needs to use a git repository for his game where the game archive is in the .sdd format stored:

The git repository needs to contains a folder which is in the structre of a .sdd. in modinfo.lua version has to be 

version = '$VERSION',

$VERSION later on will be replaced by the git commit number or, when a release is made, the version number of the release. modinfo.lua needs to be lowercase!

Creating a release

When the git repository is connected to rapid, a release can be made by making a git-commit with a commit message in the format: 

VERSION{x.xx}

Where everything in the brackets will replace $VERSION in modinfo.lua.

This will create a tag :stable for this commit and create an archive at https://gamename.repo.springrts.com/builds/, i.e. https://i18n.repo.springrts.com/builds/ which can be distributed/uploaded to different sites if needed. It is also automatically published to upq at https://springfiles.springrts.com/ which all current download systems have access to; you can validate what happened there, too.

Adding your game to the rapid download system

Please create a new issue at https://github.com/spring/RapidTools/issues and report the location of your git repository and the wanted rapid tag.

To download you can use one of the Rapid_Tags.

Files structures

file structure of rapid is: 

/versions.gz
/repos.gz
/packages/
/pool/

list of rapid repositories: repos.gz

gziped text file which has a list of all rapid repositories. 

main,https://packages.springrts.com,,

for springrts it is: https://repos.springrts.com/repos.gz

list of archives on a mirror: versions.gz

gziped text file which has a list of all sdp archives in this repository. 

ba-tronic:version:7.12-tronic1,baafb4fb215c60873e7d10e7ec600dc5,Balanced Annihilation V7.12,Balanced Annihilation V7.12-tronic1

the md5 hash is the filename of the sdp of the archive. so for example:

https://packages.springrts.com/packages/baafb4fb215c60873e7d10e7ec600dc5.sdp

folder with index archives: /packages/

this folder contains all available packages (~games) in the rapid file structure.

an index archive: .sdp

gziped index file with a list of all files in this archive. this is for example a game.

Format of the .sdp (index) files:

  • There is one entry per indexed file. These are repeated until EOF.
  • The format of one such entry:
    • 1 byte real file name length
    • real file name
    • 16 byte MD5 digest
    • 4 byte CRC32
    • 4 byte file size
  • The 16 bytes MD5 digest is the reference to the 32 Hex char file name
  • under pool, which contains the content.

filename of the .sdp is: 

 md5(md5(filename1) + md5(content of file1) + md5(filename2) + md5(content of file2) ...)


see also void PoolArchiveT::load(DigestT const & Digest):

https://github.com/spring/RapidTools/blob/master/Rapid/PoolArchive.cpp#L35

or

https://github.com/spring/spring/blob/master/rts/System/FileSystem/Archives/PoolArchive.h

the pool folder

this folder contains all files referenced by the .sdp's in gzipped form. the filename is the md5 of the file. this way files which are used by multiple archives can be shared.

The pool system uses two directories, to be found in the root of a spring data directory, pool and packages. They may look as follows: 

/pool/00/00756ec29fe8fc9d3da9b711e76bc9.gz
/pool/00/3427d26f419dabe74eaf7b865407b8.gz
...
/pool/01/
/pool/02/
...
/pool/ff/

basicly its in this form: 

/pool/<first 2 hex chars>/<last 30 hex chars>.gz

The splitting up into the 00 till ff sub-dirs is only to prevent possible problems with file-system limits for maximum files per directory (eg. FAT32).

Retrieved from "https://springrts-wiki.p2004a.com/mediawiki/index.php?title=Rapid&oldid=15611"