[Feature]: account for recently active keys in --assignment-mode min_group load

[lawrence-harmonic]

Problem Statement

Suppose we want to do RL with N concurrent agents and K inference instances. Ideally, we would like each inference instance to handle exactly N / K agents (this can allow for exact planning of the KV cache size). --policy manual --assignment-mode min_group is close to this, but min_group only counts keys which have a currently running requests, and thus will not count all active agents (e.g. agents which are currently running a tool call).

Proposed Solution

Support a TTL parameter for --assignment-mode min_group. Recently active keys within this TTL would be counted in the load.

Alternatives Considered

No response

Feature Area

Routing & Load Balancing

Affected Component(s)

No response

Use Case

See Problem Statement

Priority to You

Nice to have

Contribution

• I am willing to contribute this feature (with guidance)
• I am willing to help test this feature
• I can provide more detailed requirements if needed

Additional Context

No response

[ChetanXpro]

hey @slin1237 , i’d like to work on this. should the ttl be a new option or reuse max-idle-secs? also should active requests always count even after the ttl? happy to pick this up if nobody else is working on it.