Cohort Dimension Support in Superposition

[knutties]

Problem statement

Superposition has moved towards strict equality in contexts to support completely unambiguous resolution while performing operations. This removes the operator flexibility that was earlier available in constructing context expressions. This flexibility allowed users to construct context expressions like dimension1 in ['val1' , 'va2']. As mentioned earlier, this flexibility causes ambiguous resolution behaviour as Superposition does not yet assign priority to operators (it does only for dimensions).

So the problem statement at hand is how to bring back this feature while retaining the unambiguous resolution semantics.

Proposed Solution

To solve this problem, we introduce a concept called Cohort Dimensions. Cohort Dimensions are dimensions which are derived out of existing dimensions (we will refer to the existing dimensions as Primary Dimensions going forward.). Cohort dimensions have the following characteristics:

• Each cohort dimension is defined in terms a one Primary Dimension.
• Each cohort dimension consists of one or more variants (just like experiments). Variants are values that the cohort dimensions can take. Each variant is defined by logical expressions involving the primary dimension. These logical expressions can have ranges, pattern matches, array membership etc.
• Each cohort dimension will have a default variant auto-created.
• Cohort dimension variants are an ordered set - the user is expected to order them because the matching of primary dimension to the cohort dimension will happen in the order that the user has declared and the first matching variant will be assigned as the value of cohort dimension.
• During resolution of a config, cohort dimensions are computed from primary dimensions and added to the context for final resolution.
• For the purpose of creating experiments and overrides - cohort dimensions behave the same way as primary dimensions.
• Once a cohort dimension is created from a primary dimension, a dependency relationship is established between primary and the corresponding cohort dimension. The primary dimension depends on the cohort dimension i.e. whenever the primary dimension is introduced in an experiment or an override - the cohort dimension is automatically added to the experiment or override with its value filled in.
• The cohort dimension should be lower in priority to the primary dimension (is this really needed ?? - what happens if this constraint is not there)
• Changes in cohort variant definitions are not protected by experimentation - they will be binary roll-outs i.e. a full-flip of configuration values will happen if the cohort variant definitions change.

Technical implementation

[Datron]

Some thoughts -

• Cohorts don't seem to derive a dimension, rather abstracts a context so it can be reused again and again. We should define cohorts in terms of context, this way you can define a cohort recursively (a cohort abstracting a cohort)
• If we have a cohort that is defined with multiple enums - lets say user cohort has admin, qa, default. Can I define a context with both admin and qa? I'm assuming the cohort definition will have to contain a 3rd enum - admin_qa which encompasses both admin and QA
• Inside a cohort enum, what are the constraints on the operators that can be used inside the enum definition?

[knutties]

We should define cohorts in terms of context, this way you can define a cohort recursively (a cohort abstracting a cohort)

That is an interesting way to look at it. Context is a slightly wider concept than dimension. By definition a dimension expression is a context - and cohorts are a set of arms each with its own dimension expression involving the same dimension.

Atleast the reason for calling it dimension is that it appears so in the experiments and overrides - which is the main part of Superposition.

If we have a cohort that is defined with multiple enums - lets say user cohort has admin, qa, default. Can I define a context with both admin and qa

No - it is a dimension for the purposes of an experiment/override - in a context - one dimension can appear only once - the same applies to cohort dimensions. In your example - yes a third enum would be required - typically cohorts should have variants which capture mutually exclusive sets - but not necessary. That is left to users to decide. If they overlap then the user will have to order the variants appropriately.

Inside a cohort enum, what are the constraints on the operators that can be used inside the enum definition?

I think we can allow all the operators that are supported today - I don't see why we cannot. Atleast that is where we should start off till we see that an operator cannot be supported.

[Datron]

Table structure

CREATE TABLE IF NOT EXISTS cohorts(
id BIGINT PRIMARY KEY,
name TEXT NOT NULL,
description TEXT NOT NULL,
change_reason TEXT NOT NULL,
definition JSON NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
created_by TEXT NOT NULL,
last_modified_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
last_modified_by TEXT NOT NULL
)

The definition JSON field represents the cohorts enums. For example, a cohort defined as:

Cohort users
admin -> user_id in ['kara', 'ankit']
qa -> user_id in ['c', 'd']

Will be written as a map to the DB

{
	"admin": {
		"and": [
			"in": [
				{"var": ["user_id"]},
				["kara", "ankit"]
			]
		]
	},
	"qa": {
		"and": [
			"in": [
				{"var": ["user_id"]},
				["c", "d"]
			]
		]
	} 
}

Why write it like this? It makes matching enums a lookup in a map and lets us append the jsonlogic to an existing context or amend it after evaluation in contexts without jsonlogic.

API Design

API	request	response	description
GET /cohorts	None, will take filters and pagination params as query params per standard	list of cohorts	cohorts listing API
POST /cohorts	takes cohort name, description, change reason and definition	created cohort object or error	create a new cohort
PUT /cohorts/{id}	Update cohort definition, description or change reason	updated cohort object or error	update a cohort
GET /cohorts/{id}	None	fetched cohort object or error	fetch a cohort
DELETE /cohorts/{id}	None	deleted cohort object or error	delete a cohort

[knutties]

@Datron - do we need the and and the in - or is it basically the same strategy to help with adding conditions later ?

[knutties]

id BIGINT PRIMARY KEY,
name TEXT NOT NULL,

cohort name itself is id ? I am assuming it will become a dimension as well - the uniqueness need to be across two tables in this case.

[Datron]

Our discussion so far:

Cohorts can be a type of dimension, and stored as a dimension with a column to differentiate between a cohort dimension and regular dimension. The cohort dimensions will have the following:

• A weight defined for the cohort
• it will derive based a single dimension
• the cohort will have a set of enums
• every enum maps to a context in jsonlogic

The evaluation happens as follows, taking an example:

Cohort Users
admin -> user_id in ['a', 'b']
qa -> user_id in ['c', 'd', 'e']

Assume the application is querying with the context -> user_id == 'a'

• First do a cohort computation step where the user_id is matched against all cohorts that derive it, find the first cohort enum that matches the given dimension value
• Plug the discovered cohort dimensions into the evaluation context
• Run the config resolution as usual

Some questions still left

• Should underlying dimension depend on cohort dimension? (dependent dimensions)
• Should the priority of cohort dimension be lower than the underlying dimension?

[Datron]

If cohorts behave as a dimension, can autocomplete and validation functions work on it?

[Datron]

After some more discussion on point 1:

Should underlying dimension depend on cohort dimension? (dependent dimensions)

the following concept was reached -

Cohorts consist of two kinds:

• Local cohorts where the cohort branches are defined in superposition
• Remote cohorts where the cohort branches are defined in another system

Hypothesis - A dimension never depends on another dimension, it always depends on a cohort (local or remote). A dependency implies membership in a cohort - a dimension A depending on another dimension B means that dimension B is a cohort.

Example -
if dimension city depends on state -> state is a cohort of cities
if dimension state depends on country -> country is a cohort of states

There are relationships between entities in a data model that show us that they have a STRONG DEPENDENCY which means the dependency dimension is a cohort or should be modeled as such. If the relationship is a weak dependency, the choice is up to the user.

Action items:

• Introduce local and remote cohorts
• rework dependent dimensions to work between dimension and cohorts
• Introduce membership functions for to set values for remote cohorts

[Datron]

Some more points:

• Include a default stub for values that do not fall into any cohort in a cohort dimension
• Introduce a cohort recompute handler that updates existing contexts when a value is added/removed from a cohort
• Introduce a cleanup function that removes unreachable contexts because the cohort dimensions changed

[ayushjain17]

JSONSchema value for the dimensions:

clientId

schema: {
    "type": "string"
}

client_group

schema: {
    "type": "string",
    "enum": ["group1", "group2", "group3", "otherwise"],
    "definitions": {
        "group1": {
            "in": [{"var": "clientId"}, ["asdadf", "client2" ]]
        },
        "group2": {
            "in": [{"var": "clientId"}, ["test", "client3" ]]
        },
        "group3": {
            "in": [{"var": "clientId"}, ["client5", "client4" ]]
        }
    }
}

UI default for schema:

{
    "enum": [
        "otherwise"
    ],
    "type": "string",
    "definitions": {}
}

backend verifies if all the values mentioned in the enum other than otherwise is present in definitions section, and also run validation on all the definitions present under the definition key

[ayushjain17]

dimension type ADT:

enum DimensionType {
   Regular,
   LocalCohort(string),
   RemoteCohort(string),
}

columns needed for this in DB:

1. dependency_graph (as is)
2. dimension_type - string
   a. regular
   b. local-cohort:<based_on>
   c. remote-cohort:<based_on>

columns to remove:

1. dependencies
2. dependents

dimension type in API to probably go ahead with externally tagged ser-de

[Datron]

Further additional discussions around cohort dimensions:

The core problem revolves around managing cohort definitions and associations, particularly concerning data duplication, maintenance overhead, and inconsistencies when relationships change. The current implementation often requires recording both a cohort and its "based on" dimension, leading to several issues:

• Data Duplication and Maintenance: Explicitly recording both the cohort and its dependent dimension creates two sources of data, making maintenance difficult when definitions change. This can lead to unreachable overrides and require manual cleanup.
• Managing Cohort Changes: If a cohort's definition changes, overrides associated with previous definitions may become stale or unreachable. Similarly, if a cohort is defined after contexts are written, updating all existing contexts becomes a significant challenge.
• Entity Membership: While an entity cannot be in two arms of the same cohort simultaneously due to ordered resolution (only the first match is considered), the possibility of an entity belonging to multiple separate cohorts based on the same dimension introduces complexity in understanding which overrides will apply.
• User Input for Local Cohorts: For local cohorts, the system should ideally disregard user input in the evaluation context as the system calculates these values. Allowing user input for local cohorts can lead to inconsistencies and compromise system integrity.
• Control Population in Remote Cohorts: In remote cohorts, where the system doesn't inherently know the value of a remote cohort for a given client ID (e.g., merchant ID from client ID), duplicating data in the context was a surrogate way to facilitate control population. However, this reintroduces the same maintenance issues.
• Uniqueness and Composite Keys: Client IDs may not be uniquely identified across different merchants, leading to ambiguity if only the client ID is used. This suggests a need for composite primary keys (e.g., client ID + merchant ID + reseller ID + tenant ID) as dimensions, which contrasts with the current hierarchical remote cohort model.
• Debugging Complexity: Introducing more complex concepts like remote cohorts could make debugging difficult for users when issues arise.

Solutions and Their Pros and Cons

1. Restrict Context to Either Cohort or Based-On Dimension (Not Both)

   Pros:
   Reduces data duplication and maintenance overhead.
   Simplifies the context and makes it cleaner.
   Aligns with the principle of a single source of truth for relationships.
   Cons:
   Challenges for control population in remote cohorts where the system cannot derive the remote cohort value. This would require an API to expose the relationship.

2. Allow Multiple Local Cohorts per Dimension with Explicit Priority Management

   Pros:
   Provides flexibility for users to create different groupings based on a single dimension.
   By forcing users to explicitly define relative priorities, it makes them conscious of potential overrides.
   Cons:
   Can lead to subverted user expectations if overrides are unintentionally applied due to complex priority rules.
   Requires a clear user experience to guide users in setting and understanding relative priorities.

3. Derivation Functions for Remote Cohorts

   Pros:
   Allows the system to derive remote cohort values based on a dependent dimension (e.g., reseller ID from merchant ID via an API call).
   Facilitates control population by providing the necessary relationship information.
   Cons:
   Requires an external API for derivation, potentially introducing authentication and performance considerations.
   The derivation function needs to return a single, deterministic value, which can be problematic if the underlying relationship is not unique (e.g., client ID not unique across merchants).
   Cannot create remote cohort-only contexts without the dependent dimension if the derivation function is the only source.

4. Mandatory Validation Functions for Remote Cohorts

   Pros:
   Ensures the validity of remote cohort values (e.g., valid reseller IDs).
   Can simplify debugging by clearly identifying issues with invalid inputs.
   Cons:
   Adds more work for the consumer system to implement and maintain validation logic.
   The platform's flexibility might make universal mandatory validation difficult without significant user effort.

5. Modeling Composite Keys as Dimensions

   Pros:
   Addresses the issue of non-unique identifiers by combining multiple values (e.g., client ID, merchant ID, reseller ID) to form a unique dimension.
   Cons:
   Requires a re-modeling of the system to explicitly capture the composite nature of keys, which the current system (Superposition) lacks.
   Composite keys can become problematic in large-scale, distributed systems where multiple disparate systems need to manage all keys and their relationships.
   Contrasts with the current hierarchical structure of remote cohorts.

6. Allow Explicit Additions to Remote Cohorts (with Stale Data Cleanup)

   Pros:
   Provides flexibility for owners to manage data.
   Avoids blocking valid use cases by restricting functionality.
   Cons:
   Can lead to ambiguity and stale data if relationships change.
   Requires cleanup mechanisms to manage dead data or unreachable overrides.

Final Action Items

Natarajan Kannan will park the discussion about remote cohorts for now.
Natarajan Kannan will set up time later to catch up on migration and other topics.
Natarajan Kannan will independently think about and revisit the validation for remote cohorts.
Natarajan Kannan will document and capture all functions, including context validation and derivation functions, to determine their implementation in cohorts.
The group will monitor metrics to understand the usage of remote cohorts.
The team will proceed with the current understanding of remote cohorts, allowing explicit additions as a flexible option, and will articulate any underlying assumptions about ID relationships.