[EPIC] Accurate metrics when client-side sampling is on #73
Description
Problem
Our customers frequently use client-side sampling to control costs, but we also don’t feel confident suggesting a 100% client-side sampling rate on all platforms because of performance concerns. At the same time, we tend to assume a 100% client-side sampling rate when we talk about the completeness of metrics.
Accurate metrics must be a key goal regardless of client-side or server-side sampling rates.
User Stories
- As a user, I want my metrics to be accurate, even if I sample on the client.
Possible Solutions
- Improve our SDKs to sample 100% in all cases (does not solve cost control by customers).
- Always extract and aggregate complete metrics in our SDK regardless of the sampling rate (will create cost on our end, complicated to maintain on all SDKs)
- Report the client-side sample rate with metrics sent to the backend and extrapolate them based on this rate.
Clarifications
- While it's a technical requirement to strive for, we will never be able to assume that any given client samples with 100% and as such, tuning our SDKs for this, won't solve the problem completely.
- Whichever approach we choose, this will increase COGS for us.
- We need to re-validate all SDKs and implement better backpressure management to guarantee even if customers set 100% sampling, especially in a very high throughput environment, we never negatively impact the customers infrastructure