Create new `operation_distinctiveness` signal, and revamp Semantic Similarity normalisation

[frankkilcommins]

Revamp: https://github.com/jentic/api-ai-readiness-framework/blob/main/docs/specification/spec.md#semantic-similarity

Define new signal to look for overlapping operations which make it difficult for AI systems to select the appropriate operation.

[char0n]

We can gather path, http verb and tags

From those using deterministic heuristics for common APIs, we'll be able to infer more context and penalize consistency less.
The more API is aligned with REST standards, the less penalized in distinctiveness it gets

[frankkilcommins]

Proposal for Operation Distinctiveness below. Subsequently, the Semantic Similarity normalisation rule will also be updated.

Operation Distinctiveness (operation_distinctiveness)

The operation_distinctiveness signal measures how easily an AI agent can tell similar operations apart. It focuses on semantic separation between operations that share the same HTTP method and resource family.

When operations are named and described in ways that are too similar, agents have difficulty choosing the correct one. This leads to incorrect routing and brittle orchestration.

The signal is computed in two stages:

1. Compute a per-operation distinctiveness score distinctiveness_i based on nearest-neighbour semantic similarity.
2. Aggregate per-operation scores into a single dimension signal, with reduced weight for deprecated operations.

Comparable operations

Two operations are considered comparable if they:

• use the same HTTP method, and
• belong to the same resource family.

The resource family is derived from the path template, with version-like segments stripped.

Implementations MUST:

• Strip a leading version segment from the path if present. Examples include:

  • v1, v2, v3
  • v1.0, v2.1
  • date-style versions such as 2023-01-01, 2024-06
  • vendor-style versions such as v1beta, v1alpha
  • semver versioning strategies

• Then extract the next one or two path segments that represent the resource.

Examples:

• GET /v1/users → resource family ["users"]
• GET /v2/users → resource family ["users"]
• GET /v1/users/{id} → resource family ["users"]
• GET /v1/users/{id}/orders → resource family ["users", "orders"]

Two operations i and j are comparable if method_i = method_j AND family_i = family_j

If no other operation is comparable with operation i, it is considered unambiguous for this signal and MUST receive a per-operation distinctiveness score of 1.0.

Note: versioning is intentionally normalised out so that GET /v1/users and GET /v2/users are treated as variants of the same conceptual capability for distinctiveness evaluation.

Semantic representation of an operation

For each operation, the implementation MUST construct a textual representation that captures its intent.

The textual representation MUST include:

• operationId (if present).
• HTTP method.
• path template (including parameter placeholders).
• summary (if present).
• the first sentence or a fixed length extract of description (if present).
• primary tags or resource nouns inferred from the path.

An example:

T_i = operationId + " | " + method + " " + path
      + " | " + summary
      + " | " + leading_sentence(description)

The exact concatenation format can be implementation-defined, but MUST be consistent per run.

Per-operation distinctiveness

The implementation MUST use a semantic similarity function over these representations. The RECOMMENDED approach is:

• Embed each T_i into a vector x_i using an embedding model.
• Use cosine similarity as sim(x_i, x_j) ∈ [0, 1].

For each operation i:

• Let C_i be the set of comparable operations (same method and resource family, excluding i itself).
• If C_i is empty, set distinctiveness_i = 1.0.
• Otherwise, compute the nearest-neighbour similarity as nn_sim_i = max_{j ∈ C_i} sim(x_i, x_j).
• Then define distinctiveness_i = 1 - nn_sim_i.

This yields:

• distinctiveness_i ≈ 1.0 when the closest neighbour is clearly different in meaning.
• distinctiveness_i ≈ 0.0 when the closest neighbour is almost identical in meaning.

This is an instance of inverse semantic similarity using a nearest-neighbour strategy.

Deprecation-aware aggregation

Once per-operation scores are computed, operation_distinctiveness aggregates them across all operations, with reduced weight for deprecated ones.

For each operation i:

w_i = 1.0   if deprecated != true
w_i = 0.25  if deprecated == true

Then:

distinctiveness = ( Σ_i (w_i × distinctiveness_i) ) / Σ_i w_i

This aggregation:

• Treats multiple active variants such as GET /v1/users and GET /v2/users as genuinely more confusing.
• Reduces the long-term impact of legacy operations marked deprecated: true, while still reflecting that they exist in the surface.

If there are no operations, distinctiveness MUST be set to 1.0.

[frankkilcommins]

Proposed update for Semantic Similarity Normalisation Rule:

Semantic Similarity

Semantic similarity is used for signals that require comparing the meaning of two or more textual artefacts (for example, operation identifiers, summaries, or descriptions) rather than their exact string form.

It provides a normalised similarity score similarity(i, j) ∈ [0, 1], where:

• 0 means “completely unrelated”
• 1 means “effectively the same meaning”

Many signals then derive a distinctiveness or distance score as:

distinctiveness = 1 - similarity

Text representation

For each item i to be compared, the implementation MUST construct a textual representation T_i. This representation SHOULD be composed from the most relevant fields for the signal, for example:

• operationId
• HTTP method and path
• summary
• a short extract of description
• relevant tags or resource nouns

The exact concatenation format MAY be implementation-defined, but MUST be consistent within a run.

Individual signal definitions (such as operation_distinctiveness and intent_legibility) specify which fields MUST be included for that signal.

Embedding-based similarity

The RECOMMENDED approach for computing semantic similarity is:

• Encode each T_i into a vector x_i using an embedding model.
• Compute cosine similarity between vectors:

similarity(i, j) = cosine(x_i, x_j) ∈ [0, 1]

Implementations MAY combine multiple channels internally (for example, separate embeddings for operationId and summary), but this MUST be treated as an implementation detail. The final similarity MUST still be a single scalar in [0, 1].

String-based Similarity (fallback)

If embeddings are not available, an implementation MAY fall back to a deterministic string similarity function over T_i, for example:

• normalised Levenshtein distance
• token Jaccard similarity
• character n-gram similarity

The fallback MUST:

• produce values in [0, 1], where larger values mean “more similar”, and
• be applied consistently for all compared items.

Behavioural expectations

Regardless of the underlying method, the evaluator SHOULD ensure the following qualitative behaviour:

• Identical or near-identical meanings SHOULD produce similarity(i, j) ≥ 0.85.
• Clearly distinct or opposite purposes SHOULD produce similarity(i, j) ≤ 0.20.

Exact thresholds are signal-defined, but MUST be chosen so that:

• duplicate or near-duplicate items are treated as highly similar, and
• clearly different items are treated as low similarity.

Usage

Signals that rely on semantic similarity (for example, operation_distinctiveness, intent_legibility) MUST explicitly state:

• how T_i is constructed for that signal, and
• how similarity values are aggregated (for example, nearest neighbour, mean across pairs, etc.).

The Semantic Similarity normalisation rule only defines how to compute similarity(i, j) in a normalised way. The scoring logic that uses those similarity values is defined per-signal.

[aberming]

A few comments on above. I understand what we are trying to do here is ring an alarm for the developer where we predict that an AI agent may not be able to disambiguate between two operations or tools and is hence likely to choose the wrong one. This is motivated by observations that in APIs certain endpoints can be very minor variants on one another, often within same resource family and supplementary metadata (description, summary) but be noisy, absent or otherwise poor quality. So we are really concerned about penalising APIs where we find cases of high semantic closeness/adjacency (not an overall characteristic of the semantic distribution of the API).

Note this is related but different to what we worked on last quarter in search which was more oriented towards the info overload problem - how do we find the right tools over a larger surface area. Of course they may come into play here too with large/many private APIs! Naturally APIs with many endpoints will be likely to score lower.. maybe that's a good thing.

This work has a core assumption that the semantic adjacency is an approximation of the differentiability. Given this implementation, most developers should be able to raise their signal through best practice in metadata alone (intent, terse, explicit, technical only when important). This would dovetail nicely with how we have implement Search in the Directory API.

The approach looks good overall - some comments:

1. I would generally encourage us to think binary - this is distinct or not distinct. We are really trying to label difficult-to-disambiguate pairs.
2. I suspect that the fallback methods will work functionally but probably not hold up very well in practice with lots of false positives.
3. General disclaimer on interpreting semantic distances. They can be compared within a dataset for semantic adjacency as we propose. However this is not stable for different agent queries or for different models. So any threshold we decide would need to be retuned if we change the model and different models cannot be compared with the same threshold; the space itself is different. Without knowing the query we cannot know for sure if the endpoints are sufficiently distinct ... just a statistical limitation. We of course are being query independent here so have to make a general assumption. I would pick a common small embedding model and stick to it.
4. One for @frankkilcommins maybe, in the absence of more scientific rigour it would be good to spell out the kind of real world test cases, ideally including some stress tests/edge cases we think are acceptably distinguished and those which are not. This can be used as gating tests and possibly to help tune any threshold.
5. For semantic projection, we have an internal implementation in Directory API that may be helpful as reference, already having encountered some edge cases. Though I would say use as a reference only - here we should emulate what we think a common basic semantic implementation would be in an agent/MCP system as per the spec.

[frankkilcommins]

Thank @aberming,

I would generally encourage us to think binary - this is distinct or not distinct. We are really trying to label difficult-to-disambiguate pairs.

This might be a little course grained but I'm not against it. Let's start with the scaling approach and if we're not happy then we can revert to binary

I suspect that the fallback methods will work functionally but probably not hold up very well in practice with lots of false positives.

We can remove String-based Similarity (fallback) and just proceed with embedding.

absence of more scientific rigour it would be good to spell out the kind of real world test cases

Yes agree, we should have a set of APIs that act as a useful basis for our tests