[serve] Transform replica level metrics to AutoScalingContext constructor args by arcyleung · Pull Request #57202 · ray-project/ray

arcyleung · 2025-10-06T01:55:38Z

Changes

Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function.
dropped requests_per_replica since its expensive to compute
renamed queued_requests to total_queued_requests for consistency with total_num_requests
added total_running_requests
added tests assert new fields are populated correctly
run custom metrics tests with RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER = 0 and 1
updated docs

gemini-code-assist

Code Review

This pull request wires up replica-level metrics to the AutoscalingContext for use in custom autoscaling policies. The overall change is in the right direction, but the implementation for transforming the metrics has several critical issues, including potential KeyError and IndexError exceptions, as well as type mismatches with the AutoscalingContext definition. I've provided a detailed comment with a suggested fix to make the metric collection robust and correct.

python/ray/serve/_private/autoscaling_state.py

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh · 2025-10-11T12:31:48Z

@arcyleung I pushed some changes to your PR. Mainly

removed and renamed some fields on autoscaling context
added tests

Signed-off-by: abrar <abrar@anyscale.com>

python/ray/serve/_private/autoscaling_state.py

Signed-off-by: abrar <abrar@anyscale.com>

arcyleung · 2025-10-11T17:08:10Z

I've just added an additional test case corresponding to the example in docs, since the counter one might not be intuitive to showcase both up-and-downscaling.

arcyleung · 2025-10-11T17:09:01Z

@arcyleung I pushed some changes to your PR. Mainly
1. removed and renamed some fields on autoscaling context

2. added tests

Thanks! Appreciate the help with the refactor and docs

arcyleung · 2025-10-11T19:58:22Z

python/ray/serve/tests/BUILD.bazel

+            "env": {
+                "RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER": "1",
+                "RAY_SERVE_COLLECT_AUTOSCALING_METRICS_ON_HANDLE": "0",
+                "RAY_SERVE_REPLICA_AUTOSCALING_METRIC_RECORD_INTERVAL_S": "0.1",


This test also fails locally for me when I set "RAY_SERVE_REPLICA_AUTOSCALING_METRIC_RECORD_INTERVAL_S": "0.1" but passes with "0.5"

…RVAL_S to 0.5 Signed-off-by: Arthur Leung <arcyleung@gmail.com>

Signed-off-by: Arthur Leung <arcyleung+github@gmail.com>

Signed-off-by: Arthur Leung <arcyleung@gmail.com>

python/ray/serve/_private/autoscaling_state.py

Signed-off-by: abrar <abrar@anyscale.com>

zcin · 2025-10-16T02:16:26Z

@abrarsheikh I think tests are failing.

arcyleung · 2025-10-16T03:12:53Z

@abrarsheikh I think tests are failing.

Will update, the test cases were just missing the look_back_period_s autoscaling config

…configs Signed-off-by: Arthur Leung <arcyleung@gmail.com>

Signed-off-by: Arthur Leung <arcyleung@gmail.com>

…-metrics

…ctor args (ray-project#57202) ## Changes 1. Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function. 2. dropped `requests_per_replica` since its expensive to compute 3. renamed `queued_requests` to `total_queued_requests` for consistency with `total_num_requests` 4. added `total_running_requests` 5. added tests assert new fields are populated correctly 6. run custom metrics tests with `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` = 0 and 1 7. updated docs --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Arthur Leung <arcyleung+github@gmail.com> Co-authored-by: abrar <abrar@anyscale.com> Co-authored-by: Arthur Leung <arcyleung@gmail.com>

…ctor args (ray-project#57202) ## Changes 1. Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function. 2. dropped `requests_per_replica` since its expensive to compute 3. renamed `queued_requests` to `total_queued_requests` for consistency with `total_num_requests` 4. added `total_running_requests` 5. added tests assert new fields are populated correctly 6. run custom metrics tests with `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` = 0 and 1 7. updated docs --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Arthur Leung <arcyleung+github@gmail.com> Co-authored-by: abrar <abrar@anyscale.com> Co-authored-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: xgui <xgui@anyscale.com>

…ctor args Original PR #57202 by arcyleung Original: ray-project/ray#57202

…oScalingContext constructor args Merged from original PR #57202 Original: ray-project/ray#57202

…ctor args (#57202) ## Changes 1. Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function. 2. dropped `requests_per_replica` since its expensive to compute 3. renamed `queued_requests` to `total_queued_requests` for consistency with `total_num_requests` 4. added `total_running_requests` 5. added tests assert new fields are populated correctly 6. run custom metrics tests with `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` = 0 and 1 7. updated docs --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Arthur Leung <arcyleung+github@gmail.com> Co-authored-by: abrar <abrar@anyscale.com> Co-authored-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…ctor args (ray-project#57202) ## Changes 1. Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function. 2. dropped `requests_per_replica` since its expensive to compute 3. renamed `queued_requests` to `total_queued_requests` for consistency with `total_num_requests` 4. added `total_running_requests` 5. added tests assert new fields are populated correctly 6. run custom metrics tests with `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` = 0 and 1 7. updated docs --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Arthur Leung <arcyleung+github@gmail.com> Co-authored-by: abrar <abrar@anyscale.com> Co-authored-by: Arthur Leung <arcyleung@gmail.com>

…ctor args (ray-project#57202) ## Changes 1. Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function. 2. dropped `requests_per_replica` since its expensive to compute 3. renamed `queued_requests` to `total_queued_requests` for consistency with `total_num_requests` 4. added `total_running_requests` 5. added tests assert new fields are populated correctly 6. run custom metrics tests with `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` = 0 and 1 7. updated docs --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Arthur Leung <arcyleung+github@gmail.com> Co-authored-by: abrar <abrar@anyscale.com> Co-authored-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…ctor args (ray-project#57202) ## Changes 1. Wire up the AutoScalingContext constructor args to make metrics readable in the custom AutoScalingPolicy function. 2. dropped `requests_per_replica` since its expensive to compute 3. renamed `queued_requests` to `total_queued_requests` for consistency with `total_num_requests` 4. added `total_running_requests` 5. added tests assert new fields are populated correctly 6. run custom metrics tests with `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` = 0 and 1 7. updated docs --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Arthur Leung <arcyleung+github@gmail.com> Co-authored-by: abrar <abrar@anyscale.com> Co-authored-by: Arthur Leung <arcyleung@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

arcyleung requested a review from a team as a code owner October 6, 2025 01:55

arcyleung changed the title ~~[serve] Transform replica level metrics to AutoScalingContext constructor args~~ [serve][Draft] Transform replica level metrics to AutoScalingContext constructor args Oct 6, 2025

This comment was marked as outdated.

Sign in to view

gemini-code-assist bot reviewed Oct 6, 2025

View reviewed changes

python/ray/serve/_private/autoscaling_state.py Outdated Show resolved Hide resolved

ray-gardener bot added serve Ray Serve Related Issue community-contribution Contributed by the community labels Oct 6, 2025

abrarsheikh reviewed Oct 6, 2025

View reviewed changes

python/ray/serve/_private/autoscaling_state.py Outdated Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

arcyleung force-pushed the autoscaling-context-metrics branch from f2b9a74 to 6243438 Compare October 9, 2025 15:42

This comment was marked as outdated.

Sign in to view

abrarsheikh mentioned this pull request Oct 10, 2025

Add ray docs for custom autoscaling in serve #57600

Merged

This comment was marked as outdated.

Sign in to view

arcyleung force-pushed the autoscaling-context-metrics branch from 6ffe2e3 to a242e68 Compare October 10, 2025 22:39

enrich autoscaling context

3fbea82

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh force-pushed the autoscaling-context-metrics branch from c3fc84f to 3fbea82 Compare October 11, 2025 12:20

abrarsheikh added the go add ONLY when ready to merge, run all tests label Oct 11, 2025

This comment was marked as outdated.

Sign in to view

reuse

6823c84

Signed-off-by: abrar <abrar@anyscale.com>

This comment was marked as outdated.

Sign in to view

abrarsheikh added 2 commits October 11, 2025 12:43

add back requests

793bc8a

Signed-off-by: abrar <abrar@anyscale.com>

dedicated key for queued

3671aef

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh changed the title ~~[serve][Draft] Transform replica level metrics to AutoScalingContext constructor args~~ [serve] Transform replica level metrics to AutoScalingContext constructor args Oct 11, 2025

abrarsheikh added 2 commits October 11, 2025 13:08

update docs

a35a176

Signed-off-by: abrar <abrar@anyscale.com>

update docs

e86f3e9

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh requested a review from a team as a code owner October 11, 2025 13:08

This comment was marked as outdated.

Sign in to view

arcyleung commented Oct 11, 2025

View reviewed changes

python/ray/serve/_private/autoscaling_state.py Show resolved Hide resolved

fix test

8656f95

Signed-off-by: abrar <abrar@anyscale.com>

arcyleung commented Oct 11, 2025

View reviewed changes

Increase interval of RAY_SERVE_REPLICA_AUTOSCALING_METRIC_RECORD_INTE…

7f05034

…RVAL_S to 0.5 Signed-off-by: Arthur Leung <arcyleung@gmail.com>

abrarsheikh requested a review from zcin October 14, 2025 21:36

arcyleung and others added 2 commits October 14, 2025 21:01

Merge branch 'master' into autoscaling-context-metrics

377ca28

Signed-off-by: Arthur Leung <arcyleung+github@gmail.com>

Fix lint

dbd575f

Signed-off-by: Arthur Leung <arcyleung@gmail.com>

arcyleung force-pushed the autoscaling-context-metrics branch from d8b26d5 to dbd575f Compare October 15, 2025 03:05

zcin reviewed Oct 16, 2025

View reviewed changes

python/ray/serve/_private/autoscaling_state.py Outdated Show resolved Hide resolved

python/ray/serve/_private/autoscaling_state.py Outdated Show resolved Hide resolved

python/ray/serve/_private/autoscaling_state.py Outdated Show resolved Hide resolved

abrarsheikh added 3 commits October 16, 2025 00:53

refactor code for simplicity

cca67f2

Signed-off-by: abrar <abrar@anyscale.com>

remove redundant function

ceab7f7

Signed-off-by: abrar <abrar@anyscale.com>

fix docs

556d2e9

Signed-off-by: abrar <abrar@anyscale.com>

zcin approved these changes Oct 16, 2025

View reviewed changes

Add missing look_back_period_s to TestCustomServeMetrics autoscaling …

9594687

…configs Signed-off-by: Arthur Leung <arcyleung@gmail.com>

This comment was marked as outdated.

Sign in to view

Return types

53b490f

Signed-off-by: Arthur Leung <arcyleung@gmail.com>

abrarsheikh approved these changes Oct 16, 2025

View reviewed changes

Merge remote-tracking branch 'origin/master' into autoscaling-context…

fa95474

…-metrics

zcin merged commit 3763246 into ray-project:master Oct 16, 2025
6 checks passed

snorkelopstesting2-coder mentioned this pull request Oct 22, 2025

[serve] Transform replica level metrics to AutoScalingContext constructor args snorkel-marlin-repos/ray-project_ray_pr_57202_e39dfb53-78b5-4294-b765-088dcf6a5717#1

Merged

Conversation

arcyleung commented Oct 6, 2025 • edited by abrarsheikh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

This comment was marked as outdated.

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

abrarsheikh commented Oct 11, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

arcyleung commented Oct 11, 2025

Uh oh!

arcyleung commented Oct 11, 2025

Uh oh!

arcyleung Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zcin commented Oct 16, 2025

Uh oh!

arcyleung commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arcyleung commented Oct 6, 2025 •

edited by abrarsheikh

Loading

arcyleung commented Oct 16, 2025 •

edited

Loading