add swip-35: negative incentives epic WIP #69
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zelig
added
the
improvement
headers
extended text,and structure still draft
more elab
SWIPs/swip-35.md
Outdated
|
|
||
| ## Abstract | ||
|
|
||
| This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, swarm will offer a wider range of constructions for its users to secure the persistance of chunks and their metadata and at the same time allow a low barrier of entry service that node operators with extra storage or bzz future providers are able to increase their revenue with. |
There was a problem hiding this comment.
Suggested change
| This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, swarm will offer a wider range of constructions for its users to secure the persistance of chunks and their metadata and at the same time allow a low barrier of entry service that node operators with extra storage or bzz future providers are able to increase their revenue with. | |
| This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, swarm will offer a wider range of constructions for its users to secure the persistence of chunks and their metadata and at the same time allow a low barrier of entry service that node operators with extra storage or bzz future providers are able to increase their revenue with. |
SWIPs/swip-35.md
Outdated
|
|
||
| The current storage incentive system has a few shortcomings mainly related to the problem that postage batches are difficult to interpret. This interpretation problem leads to quite a bit of misunderstanding and misguided (also potentially failing) expectations regarding both the stored amount and the storage period. | ||
|
|
||
| These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. |
There was a problem hiding this comment.
Suggested change
| These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. | |
| These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak meta-data in the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. |
SWIPs/swip-35.md
Outdated
|
|
||
| These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. | ||
|
|
||
| The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots. |
There was a problem hiding this comment.
Suggested change
| The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots. | |
| The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce a *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots. |
SWIPs/swip-35.md
Outdated
| In this SWIP we introduce a scheme in which uploads are not necessarily using a stamp; even though uploaders can purchase storage for arbitrary long periods. | ||
| No per-chunk metadata needs recording and made available in this case, meaning that the end-user case no longer raises portability concerns. | ||
|
|
||
| Last, but not least, enforcing payments for each upload does not really make sense: firstly, users had some difficulty to grasp that given the free-tier for bandwidth inentives allowing free downlad and upload with throttled throughput, how come they need to pay for the storage. This problem was most obvious for really short-lived messages that are used in chat or notifications. Since, in these cases, preservation is not meaningful or even actively not desired, paying for postage sounds not only a unnecessary but outright wasteful. Secondly, the fact that expired chunks are not deleted but are put to the *cache*[^5] poses the question how come there is no direct entry to it. |
There was a problem hiding this comment.
Suggested change
| Last, but not least, enforcing payments for each upload does not really make sense: firstly, users had some difficulty to grasp that given the free-tier for bandwidth inentives allowing free downlad and upload with throttled throughput, how come they need to pay for the storage. This problem was most obvious for really short-lived messages that are used in chat or notifications. Since, in these cases, preservation is not meaningful or even actively not desired, paying for postage sounds not only a unnecessary but outright wasteful. Secondly, the fact that expired chunks are not deleted but are put to the *cache*[^5] poses the question how come there is no direct entry to it. | |
| Last, but not least, enforcing payments for each upload does not really make sense: firstly, users had some difficulty to grasp that given the free-tier for bandwidth inentives allowing free download and upload with throttled throughput, how come they need to pay for the storage. This problem was most obvious for really short-lived messages that are used in chat or notifications. Since, in these cases, preservation is not meaningful or even actively not desired, paying for postage sounds not only a unnecessary but outright wasteful. Secondly, the fact that expired chunks are not deleted but are put to the *cache*[^5] poses the question how come there is no direct entry to it. |
SWIPs/swip-35.md
Outdated
|
|
||
| #### Contractual terms of storage | ||
|
|
||
| The most natural way to pay for storage is by the *amount* and *time period*. Surely, the unit price of storage (per chunk, per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now, expiring after a fixed period or on a fixed datte). Experience shows that any deviation from this expectation comes as a surprise for the user. |
There was a problem hiding this comment.
Suggested change
| The most natural way to pay for storage is by the *amount* and *time period*. Surely, the unit price of storage (per chunk, per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now, expiring after a fixed period or on a fixed datte). Experience shows that any deviation from this expectation comes as a surprise for the user. | |
| The most natural way to pay for storage is by the *amount* and *time period*. Surely, the unit price of storage (per chunk, per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now, expiring after a fixed period or on a fixed date). Experience shows that any deviation from this expectation comes as a surprise for the user. |
SWIPs/swip-35.md
Outdated
|
|
||
| The most natural way to pay for storage is by the *amount* and *time period*. Surely, the unit price of storage (per chunk, per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now, expiring after a fixed period or on a fixed datte). Experience shows that any deviation from this expectation comes as a surprise for the user. | ||
|
|
||
| The current primitive of storage compensation, the postage batch falls short on these. The primary reason is that it was designed with the simplicity in mind pertaining to the intergration into the redistribution system. |
There was a problem hiding this comment.
Suggested change
| The current primitive of storage compensation, the postage batch falls short on these. The primary reason is that it was designed with the simplicity in mind pertaining to the intergration into the redistribution system. | |
| The current primitive of storage compensation, the postage batch falls short on these. The primary reason is that it was designed with the simplicity in mind pertaining to the integration into the redistribution system. |
SWIPs/swip-35.md
Outdated
|
|
||
| On the other hand, the redistribution game is a *probabilistic outpayment scheme* in the sense that instead of paying -- on each game all the contributors for a recurring fixed proportion of used storage we just pay out the entire reward pot for the period to a random winner at every game. This allows for central pricing sensitive to supply and offers automatic and optimal price discovery as long as we assume a liquid market of operators providing storage. As a consequence, users do not need to speculate on the future price of storage at all, they can just trust the oracle to discover the best price of rent. Users provide the compensation for their quota as a "hot wallet" from which the price of rent is automatically deduced. The postage batch is such a construct: it serves as a hot wallet. The expiry is defined as the onset of insolvency, i.e., whenever the balance goes to zero. | ||
|
|
||
| Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period `[s, e]` the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` |
There was a problem hiding this comment.
Suggested change
| Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period `[s, e]` the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` | |
| Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then `A` assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period `[s, e]` the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` |
SWIPs/swip-35.md
Outdated
| 2. end; datetime of expiry of storage | ||
| 3. start; arrival time of request | ||
|
|
||
| As long as 3. is undifferentiating due to a large number of (near) simultaneous requests |
There was a problem hiding this comment.
Suggested change
| As long as 3. is undifferentiating due to a large number of (near) simultaneous requests | |
| As long as 3 is undifferentiating due to a large number of (near) simultaneous requests |
SWIPs/swip-35.md
Outdated
| 3. start; arrival time of request | ||
|
|
||
| As long as 3. is undifferentiating due to a large number of (near) simultaneous requests | ||
| 1 and 2 will not help align identity to chunk or even chunk to chunk easily. This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally, the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonimity set, thus, the better the anonimity that you get. |
There was a problem hiding this comment.
Suggested change
| 1 and 2 will not help align identity to chunk or even chunk to chunk easily. This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally, the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonimity set, thus, the better the anonimity that you get. | |
| 1 and 2 will not help align identity to chunk or even chunk to chunk easily. This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally, the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the volume of traffic is not great, especially traffic relating to long term storage (more expensive), publishing may be delayed until a time after which the chunks of the file have already disappeared from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonymity set, thus, the better the anonymity that you get. |
SWIPs/swip-35.md
Outdated
|
|
||
| ## Evaluation | ||
|
|
||
| Even though treating the postage batch as a hot wallet has quite a few advantages, such an insurance scheme bring improvement in several areas: |
There was a problem hiding this comment.
Suggested change
| Even though treating the postage batch as a hot wallet has quite a few advantages, such an insurance scheme bring improvement in several areas: | |
| Even though treating the postage batch as a hot wallet has quite a few advantages, such an insurance scheme brings improvements in several areas: |
SWIPs/swip-35.md
Outdated
| - no leakage of shared ownership through signature by the same batch owner | ||
| - **precision**: | ||
| - exact period | ||
| - exacy amount |
There was a problem hiding this comment.
Suggested change
| - exacy amount | |
| - exact byte amount |
SWIPs/swip-35.md
Outdated
| - no benefit from rent price drop | ||
| - **portability**: | ||
| - only providers need to keep track of batches and batch utilisation | ||
| - since users got all they need from just knowing that a file/root was insured/kept alive |
There was a problem hiding this comment.
Suggested change
| - since users got all they need from just knowing that a file/root was insured/kept alive | |
| - users get all the assurance they need from just knowing that a file/root was insured/kept alive |
SWIPs/swip-35.md
Outdated
| - free postage | ||
| - global pinning, incentives, notifications | ||
| - stamp mixing, requests, receipts and address indexes | ||
| - challanges with inclusion proofs etc |
There was a problem hiding this comment.
Suggested change
| - challanges with inclusion proofs etc | |
| - challenges with inclusion proofs etc |
nugaon
reviewed
Jun 4, 2025
|
|
||
| ### Negative incentives | ||
| The lack of individual accountability makes Swarm's storage incentivisation limited as a security measure against data loss. Introducing non-compensatory insurance, on the other hand, adds an additional layer of *negative incentives*. In other words, the threat of punitive measures (losing the stake deposited against foul play) is introduced | ||
| in order to compell storage providers to ensure reliability of data retention and/or prompt retrievability for users. |
There was a problem hiding this comment.
Data retrievability is not yet outlined in the spec (just for note since we touched upon it before)
|
|
||
| In other words, increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (*upload and disappear experience*). If the intended expiry passes without the batch running out of funds, then storage just continues, or alternatively all funds remaining on the batch can be used for more storage service.[^12] | ||
|
|
||
| [^12]: Using the current balance to create another batch (potenitally different size, period and owner) is currently not supported but it should be. |
There was a problem hiding this comment.
what would be the upside? aren't the topup/dilute functions sort out such needs?
|
|
||
| #### Distribution of responsibility: providers in the overlay address space | ||
|
|
||
| A liquid quota market is best provided by a *decentralised service network*: served by swarm nodes themselves. Ideally providers will have a density not too dissimilar to the nodes providing the reserve, but the system also needs to be prepared to provide service in the case of a lot (a couple of orders of magnitude) fewer or more nodes. This is taken care of by the swear contract, which maintains a balanced set of providers according to *smart neighbourhood management* ([described in a separate SWIP (SWIP-39)](https://github.com/ethersphere/SWIPs/pull/74)) |
There was a problem hiding this comment.
in case of insurers, it is unknown what would be the "garbage collection" of not participating actors.
|
|
||
| We propose that chunks have a *secondary address*, defined as the (legacy Keccak) hash of the primary chunk address with or without the first bit flipped (bitwise XOR the first byte with 0x80). This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. | ||
|
|
||
| So when nodes swear on to provide insurance (or insurance/availability level X service), they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that dictated by the reserve. |
There was a problem hiding this comment.
the proposal does not detail the stake prices for running an insurance service. Is it the same all across the network? Does it influence in any way how many chunks they can stamp? Does it vary over time?
| #### Punitive measures | ||
|
|
||
| Unlike in the case of bandwidth incentives where retrievals are immediately accounted and settled, long-term storage guarantees are promissory in nature and do not allow the instant gratification of forwarding/backwarding. It is only after the end of its validity that one can check if a promise has been kept. | ||
| Nodes wanting to sell storage space should have a stake verified and locked-in at the time of making their promise. This implies that nodes need to register in advance, agreeing to the contract by placing a security deposit. Once registered, a node may start selling storage promises. While their registration is active, if they are found to have lost a chunk that was covered by their promise, they stand to lose their deposit. |
There was a problem hiding this comment.
There is also a service reliability concern: what happens with the insured chunks of different users if the provider got slashed? They cannot challange the provider anymore since there is no stake to lose. It implies that one provider slash affects multiple users in the same time.
| In case an escrow is used, the reward is vested over a period of | ||
| the storer is only awarded the locked funds if they are able | ||
| to provide *proof of entitlement*. This procedure is analogous to the postage | ||
| stamp redistribution game. Out-payments conditional on proofs of custody can be implemented in a way similar to the reserve redistribution game. |
There was a problem hiding this comment.
if there are multiple beneficiaries, the users can overissue cheques over their deposited balance anytime, so maybe providers could have the option to drop commitment in that case (similarly closing a connection/blacklisting in the case of bandwidth incentives).
| do not keep their promise to store). The registration is open-ended Users of Swarm should be able to count on the loss of deposit as | ||
| a disincentive against foul play for as long as the insurer node has open commitments. | ||
| Therefore, even a well-performing Swarm node is only eligible to reclaim their | ||
| deposit after all their commitments are closed or taken over by new nodes. |
There was a problem hiding this comment.
how is it verifiable? These are off-chain commitments...
my first shoot is that to have a withdrawal challange period when insurer issues a reclaim requests with a state root hash and anyone can provide a fraud proof against it.
But what is the structure and logic between commitment state versions of the provider? Because a record of a commitment should appear in every version of commitment state versions until its lifetime.
If all previous approach is correct, then this system needs active watchtowers/verifier actors scanning for fraudalence between every epoch... and that is not really a well incentivized or accountable role.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP