Docs: added detailed docs about how to interact with the datastorage #5059

Jarno458 · 2025-05-28T21:54:26Z

What is this fixing or adding?

Added details docks on how to interact with the data storage, this was split off from #2244

How was this tested?

by many peer reviews, and locally testing with a data storage

BadMagic100

My concerns from the other PR look to have been addressed, found a few minor things on a once-over but otherwise looks good

BadMagic100 · 2025-05-29T04:35:56Z

docs/server datastorage.md

+Because any client can read and write to the data storage at any time, you can never assume that value you read is still the same 1 second later. So an easy mistake is for example retrieving a value using a `Get` command, then processing the value, altering it, and then writing it back to the store with an `Set`-`replace` command. The `Set`-`replace` command will simply override the value discarding any possible changes by any other clients.
+
+There are a few aspects to the data storage that we can use to our advantage to ensure thread safety:
+* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command. 


Suggested change

* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command.

* Any additional data given to the `Set` command will be passed along to the `SetReply` response. This can be used to "tag" a certain `SetReply` by, for example, adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command.

BadMagic100 · 2025-05-29T04:38:18Z

docs/server datastorage.md

+So we only were able to subtract `original_value` - `value` = 20 energy.
+
+### Gifting
+Gifting allows players to gift items they don't need to other games. For example, if you get a health potion but you are at no risk of dying, someone else might have more use for it. Gifting works by slot-specific giftboxes, any client can add a gift to the giftbox, and only the client of the slot the giftbox belongs to is supposed to remove gifts from it. Now it might seem easy with an array of gift objects however while appending to an array is thread safe, removing entries from an array cannot be done thread safe. Therefore, giftboxes are implemented as dictionaries where each gift has its own unique key. New gifts can safely be added using the `update` operation and removal of specific gifts can be done using their unique keys with the `pop` operation. For more information visit the [gifting api documentation](https://github.com/agilbert1412/Archipelago.Gifting.Net/blob/main/Documentation/Gifting%20API.md).


I'm not sure exactly how I'd want to phrase this but it's probably worth some sort of mention that the gifting docs should be treated as the authoritative source if the content here is in conflict (ie, out of date)

ScipioWright

Some minor wording changes, did not read through the second half

ScipioWright · 2025-05-29T13:16:14Z

docs/server datastorage.md

+This document covers some of the patterns and tips and tricks used to communicate with data storage. Communication with the data storage is done through the [`Get`](network%20protocol.md#Get) \ [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) \ [`SetReply`](network%20protocol.md#SetReply) and [`SetNotify`](network%20protocol.md#SetNotify) commands described inside the [Network Protocol](network%20protocol.md). The data storage is meant to preserve some data on the server that spans across the lifecycle of the server and is shared with all clients (on any team). 
+
+## Keys
+The data storage works by keys and is internally stored as a dictionary. Note that the whole data storage is accessible by all clients so if you want to store data specifically for your team, game or slot you will have to make your key unique; this is often done by adding the team number, game name or slot id to the key. Some examples:


Suggested change

The data storage works by keys and is internally stored as a dictionary. Note that the whole data storage is accessible by all clients so if you want to store data specifically for your team, game or slot you will have to make your key unique; this is often done by adding the team number, game name or slot id to the key. Some examples:

Data storage is internally stored as a dictionary, so it is accessed by its keys. Data storage is not slot-specific, so if you want to store data specifically for your team, game, or slot, you will have to make your key unique. This is often done by adding the team number, game name, or slot number to the key. Some examples:

Some minor cleanup

ScipioWright · 2025-05-29T13:18:38Z

docs/server datastorage.md

@@ -0,0 +1,115 @@
+# Serverside Data Storage
+
+This document covers some of the patterns and tips and tricks used to communicate with data storage. Communication with the data storage is done through the [`Get`](network%20protocol.md#Get) \ [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) \ [`SetReply`](network%20protocol.md#SetReply) and [`SetNotify`](network%20protocol.md#SetNotify) commands described inside the [Network Protocol](network%20protocol.md). The data storage is meant to preserve some data on the server that spans across the lifecycle of the server and is shared with all clients (on any team). 


Suggested change

This document covers some of the patterns and tips and tricks used to communicate with data storage. Communication with the data storage is done through the [`Get`](network%20protocol.md#Get) \ [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) \ [`SetReply`](network%20protocol.md#SetReply) and [`SetNotify`](network%20protocol.md#SetNotify) commands described inside the [Network Protocol](network%20protocol.md). The data storage is meant to preserve some data on the server that spans across the lifecycle of the server and is shared with all clients (on any team).

This document covers some of the patterns, tips, and tricks used to communicate with data storage. Communication with data storage is done through the [`Get`](network%20protocol.md#Get) / [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) / [`SetReply`](network%20protocol.md#SetReply), and [`SetNotify`](network%20protocol.md#SetNotify) commands described in the [Network Protocol](network%20protocol.md).

The purpose of data storage is to store data for a multiworld that can be accessed by the clients connected to that multiworld.

Some more minor rewording

ScipioWright · 2025-05-29T13:23:09Z

docs/server datastorage.md

+| `GiftBox;5;13`        | Slot specific giftbox for slot 13 on team 5 |
+
+## Thread Safety
+Because any client can read and write to the data storage at any time, you can never assume that value you read is still the same 1 second later. So an easy mistake is for example retrieving a value using a `Get` command, then processing the value, altering it, and then writing it back to the store with an `Set`-`replace` command. The `Set`-`replace` command will simply override the value discarding any possible changes by any other clients.


Suggested change

Because any client can read and write to the data storage at any time, you can never assume that value you read is still the same 1 second later. So an easy mistake is for example retrieving a value using a `Get` command, then processing the value, altering it, and then writing it back to the store with an `Set`-`replace` command. The `Set`-`replace` command will simply override the value discarding any possible changes by any other clients.

Any client can read from or write to data storage at any time, so you can never assume the value you read will be the same 1 second later. An easy mistake to make is retrieving a value using the `Get` command, processing and altering the value, then writing back to data storage with a `Set`-`replace` command. This will override the value, discarding possible changes by other clients made between when you read and wrote to data storage.

More minor wording changes

ScipioWright · 2025-05-29T13:23:47Z

docs/server datastorage.md

+## Thread Safety
+Because any client can read and write to the data storage at any time, you can never assume that value you read is still the same 1 second later. So an easy mistake is for example retrieving a value using a `Get` command, then processing the value, altering it, and then writing it back to the store with an `Set`-`replace` command. The `Set`-`replace` command will simply override the value discarding any possible changes by any other clients.
+
+There are a few aspects to the data storage that we can use to our advantage to ensure thread safety:


Suggested change

There are a few aspects to the data storage that we can use to our advantage to ensure thread safety:

There are a few aspects of data storage that we can use to ensure thread safety:

ScipioWright · 2025-05-29T13:27:37Z

docs/server datastorage.md

+Because any client can read and write to the data storage at any time, you can never assume that value you read is still the same 1 second later. So an easy mistake is for example retrieving a value using a `Get` command, then processing the value, altering it, and then writing it back to the store with an `Set`-`replace` command. The `Set`-`replace` command will simply override the value discarding any possible changes by any other clients.
+
+There are a few aspects to the data storage that we can use to our advantage to ensure thread safety:
+* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command. 


Suggested change

* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command.

* Additional data given to the `Set` command will be passed to the `SetReply` response. This can be used to "tag" a certain `SetReply` by adding a specific random value to your `Set` that, when you receive the `SetReply`, you can check that value to match it back up to your `Set` command.

This should probably continue on with an example of why you would want to do this.

ScipioWright · 2025-05-29T13:28:09Z

docs/server datastorage.md

+
+There are a few aspects to the data storage that we can use to our advantage to ensure thread safety:
+* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command. 
+* All operations in your `Set` command are atomic; that is, they are executed in order without interruption from others and the final result will only trigger one `SetReply`.


Suggested change

* All operations in your `Set` command are atomic; that is, they are executed in order without interruption from others and the final result will only trigger one `SetReply`.

* All operations in your `Set` command are atomic; they are executed in order without interruption from others and the final result will only trigger one `SetReply`.

ScipioWright · 2025-05-29T13:29:39Z

docs/server datastorage.md

+* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command. 
+* All operations in your `Set` command are atomic; that is, they are executed in order without interruption from others and the final result will only trigger one `SetReply`.
+
+It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys just holding a basic value like a number in each. For example, if you want to safely update the following data structure 


Suggested change

It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys just holding a basic value like a number in each. For example, if you want to safely update the following data structure

It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys that hold basic values like integers. For example, let's say you have the following data structure:

ScipioWright · 2025-05-29T13:33:36Z

docs/server datastorage.md

+
+It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys just holding a basic value like a number in each. For example, if you want to safely update the following data structure 
+
+Key `Fruit`: `{ "Apple": 3, "Cherry": 7 }` it will be much easier to safely update as two separate keys `FruitApple`: `3` and `FruitCherry`: `7`.


Suggested change

Key `Fruit`: `{ "Apple": 3, "Cherry": 7 }` it will be much easier to safely update as two separate keys `FruitApple`: `3` and `FruitCherry`: `7`.

`Fruit`: `{ "Apple": 3, "Cherry": 7 }`

It will be much safer to update numbers if the this entry is split up into two separate keys:

`FruitApple`: `3` and `FruitCherry`: `7`.

This way, you can update these values by using `Set`-`add` instead of `Set`-`replace`.

el-u · 2025-06-02T19:38:31Z

docs/network protocol.md

@@ -469,7 +469,7 @@ The following operations can be applied to a datastorage key
 | left_shift | Applies a bitwise left-shift to the current value of the key by `value`. |
 | right_shift | Applies a bitwise right-shift to the current value of the key by `value`. |
 | remove | List only: removes the first instance of `value` found in the list. |
-| pop | List or Dict: for lists it will remove the index of the `value` given. for dicts it removes the element with the specified key of `value`. |
+| pop | List or Dict: for lists it will remove the index of the `value` given. for dicts it removes the element with the specified key of `value`. Will error if index or key does not exist |


Suggested change

| pop | List or Dict: for lists it will remove the index of the `value` given. for dicts it removes the element with the specified key of `value`. Will error if index or key does not exist |

| pop | List or Dict: for lists it will remove the index of the `value` given. for dicts it removes the element with the specified key of `value`. Will error if index or key does not exist. |

el-u · 2025-06-02T19:40:44Z

docs/server datastorage.md

+Let's look at a few examples:
+
+### EnergyLink
+EnergyLink lets you share you excess energy across multiple clients and games. It uses a team-specific key of `EnergyLink{team}`. The energy value is a numeric value and it's important that games can add and take energy from it in a thread safe way. A little warning, over the course of a long game, many clients can contribute to the energy value making it larger then an int64, while python and json have no issues with this, some programming languages might need to take extra care.


Suggested change

EnergyLink lets you share you excess energy across multiple clients and games. It uses a team-specific key of `EnergyLink{team}`. The energy value is a numeric value and it's important that games can add and take energy from it in a thread safe way. A little warning, over the course of a long game, many clients can contribute to the energy value making it larger then an int64, while python and json have no issues with this, some programming languages might need to take extra care.

EnergyLink lets you share you excess energy across multiple clients and games. It uses a team-specific key of `EnergyLink{team}`. The energy value is a numeric value and it's important that games can add and take energy from it in a thread safe way. A little warning, over the course of a long game, many clients can contribute to the energy value making it larger than an int64. While Python and JSON have no issues with this, some programming languages might need to take extra care.

el-u · 2025-06-02T19:45:10Z

docs/server datastorage.md

+    ]
+}
+```
+Removing gifts, note by first updating to an empty dict we are avoiding a possible error that occurs when you try to remove a non existing key as pop-ing an non existing key will raise an error otherwise.


Suggested change

Removing gifts, note by first updating to an empty dict we are avoiding a possible error that occurs when you try to remove a non existing key as pop-ing an non existing key will raise an error otherwise.

Removing gifts:

Note that by first updating to an empty dict we are avoiding a possible error that occurs when you try to remove a non-existing key, as pop-ing a non-existing key will raise an error otherwise.

qwint · 2025-06-08T03:31:11Z

added waiting on author to address review comments

Split of from data storage error handling

2e4bd7e

github-actions bot added the is: documentation Improvements or additions to documentation. label May 28, 2025

Jarno458 added the waiting-on: peer-review Issue/PR has not been reviewed by enough people yet. label May 28, 2025

Jarno458 requested review from BadMagic100 and el-u May 28, 2025 22:12

Jarno458 changed the title ~~Docs: added detailed docs about how to interact with the daa~~ Docs: added detailed docs about how to interact with the datastorage May 29, 2025

BadMagic100 suggested changes May 29, 2025

View reviewed changes

ScipioWright reviewed May 29, 2025

View reviewed changes

el-u approved these changes Jun 2, 2025

View reviewed changes

qwint added the waiting-on: author Issue/PR is waiting for feedback or changes from its author. label Jun 8, 2025

	* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command.
	* Any additional data given to the `Set` command will be passed along to the `SetReply` response. This can be used to "tag" a certain `SetReply` by, for example, adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command.

	The data storage works by keys and is internally stored as a dictionary. Note that the whole data storage is accessible by all clients so if you want to store data specifically for your team, game or slot you will have to make your key unique; this is often done by adding the team number, game name or slot id to the key. Some examples:
	Data storage is internally stored as a dictionary, so it is accessed by its keys. Data storage is not slot-specific, so if you want to store data specifically for your team, game, or slot, you will have to make your key unique. This is often done by adding the team number, game name, or slot number to the key. Some examples:

		@@ -0,0 +1,115 @@
		# Serverside Data Storage

		This document covers some of the patterns and tips and tricks used to communicate with data storage. Communication with the data storage is done through the [`Get`](network%20protocol.md#Get) \ [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) \ [`SetReply`](network%20protocol.md#SetReply) and [`SetNotify`](network%20protocol.md#SetNotify) commands described inside the [Network Protocol](network%20protocol.md). The data storage is meant to preserve some data on the server that spans across the lifecycle of the server and is shared with all clients (on any team).

	This document covers some of the patterns and tips and tricks used to communicate with data storage. Communication with the data storage is done through the [`Get`](network%20protocol.md#Get) \ [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) \ [`SetReply`](network%20protocol.md#SetReply) and [`SetNotify`](network%20protocol.md#SetNotify) commands described inside the [Network Protocol](network%20protocol.md). The data storage is meant to preserve some data on the server that spans across the lifecycle of the server and is shared with all clients (on any team).
	This document covers some of the patterns, tips, and tricks used to communicate with data storage. Communication with data storage is done through the [`Get`](network%20protocol.md#Get) / [`Retrieved`](network%20protocol.md#Retrieved), [`Set`](network%20protocol.md#Set) / [`SetReply`](network%20protocol.md#SetReply), and [`SetNotify`](network%20protocol.md#SetNotify) commands described in the [Network Protocol](network%20protocol.md).
	The purpose of data storage is to store data for a multiworld that can be accessed by the clients connected to that multiworld.

	There are a few aspects to the data storage that we can use to our advantage to ensure thread safety:
	There are a few aspects of data storage that we can use to ensure thread safety:

	* Any additional data given to the `Set` command will be passed along to the `SetReply` response. this can be used to "tag" a certain `SetReply` by for-example adding a specific random value to your `Set` then when you receive the `SetReply` you can check that value to match it back up to your `Set` command.
	* Additional data given to the `Set` command will be passed to the `SetReply` response. This can be used to "tag" a certain `SetReply` by adding a specific random value to your `Set` that, when you receive the `SetReply`, you can check that value to match it back up to your `Set` command.

	* All operations in your `Set` command are atomic; that is, they are executed in order without interruption from others and the final result will only trigger one `SetReply`.
	* All operations in your `Set` command are atomic; they are executed in order without interruption from others and the final result will only trigger one `SetReply`.

	It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys just holding a basic value like a number in each. For example, if you want to safely update the following data structure
	It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys that hold basic values like integers. For example, let's say you have the following data structure:


		It's a lot simpler to stay thread safe with basic types such a numbers as shown below in [EnergyLink](#EnergyLink), therefore it can useful to spread your data across multiple data storage keys just holding a basic value like a number in each. For example, if you want to safely update the following data structure

		Key `Fruit`: `{ "Apple": 3, "Cherry": 7 }` it will be much easier to safely update as two separate keys `FruitApple`: `3` and `FruitCherry`: `7`.

	\| pop \| List or Dict: for lists it will remove the index of the `value` given. for dicts it removes the element with the specified key of `value`. Will error if index or key does not exist \|
	\| pop \| List or Dict: for lists it will remove the index of the `value` given. for dicts it removes the element with the specified key of `value`. Will error if index or key does not exist. \|

	EnergyLink lets you share you excess energy across multiple clients and games. It uses a team-specific key of `EnergyLink{team}`. The energy value is a numeric value and it's important that games can add and take energy from it in a thread safe way. A little warning, over the course of a long game, many clients can contribute to the energy value making it larger then an int64, while python and json have no issues with this, some programming languages might need to take extra care.
	EnergyLink lets you share you excess energy across multiple clients and games. It uses a team-specific key of `EnergyLink{team}`. The energy value is a numeric value and it's important that games can add and take energy from it in a thread safe way. A little warning, over the course of a long game, many clients can contribute to the energy value making it larger than an int64. While Python and JSON have no issues with this, some programming languages might need to take extra care.

	Removing gifts, note by first updating to an empty dict we are avoiding a possible error that occurs when you try to remove a non existing key as pop-ing an non existing key will raise an error otherwise.
	Removing gifts:
	Note that by first updating to an empty dict we are avoiding a possible error that occurs when you try to remove a non-existing key, as pop-ing a non-existing key will raise an error otherwise.

Docs: added detailed docs about how to interact with the datastorage #5059

Are you sure you want to change the base?

Docs: added detailed docs about how to interact with the datastorage #5059

Conversation

Jarno458 commented May 28, 2025

What is this fixing or adding?

How was this tested?

Uh oh!

BadMagic100 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ScipioWright left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qwint commented Jun 8, 2025

Uh oh!

Uh oh!