Skip to content

Support context argument in batch and unbatch methods#711

Open
discobot wants to merge 2 commits into
Lightning-AI:mainfrom
discobot:fix/617-batch-unbatch-context
Open

Support context argument in batch and unbatch methods#711
discobot wants to merge 2 commits into
Lightning-AI:mainfrom
discobot:fix/617-batch-unbatch-context

Conversation

@discobot

Copy link
Copy Markdown

What does this PR do?

Fixes #617.

LitAPI.batch and LitAPI.unbatch are the only hooks that never receive the per-request context. It is worse than a missing option: declaring batch(self, inputs, context) with max_batch_size > 1 currently fails every batched request with TypeError: batch() missing 1 required positional argument: 'context', surfaced as HTTP 500.

Root cause: the batched loops (regular and streaming) call lit_api.batch/lit_api.unbatch directly instead of through the _inject_context helper that decode_request/predict/encode_response already go through — even though the contexts list is in scope at all four call sites. This PR wraps those call sites with _inject_context(contexts, ...) and documents the optional context argument in the batch/unbatch docstrings. _inject_context only passes context when the method signature declares it, so existing implementations are unaffected.

Added test_inject_context_in_batch_and_unbatch covering both loops: batch stores each request's input in its per-request context and unbatch reads it back, so the test fails with the TypeError above without the fix. tests/unit/test_lit_server.py, tests/unit/test_batch.py and tests/unit/test_loops.py pass locally.

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

The batched loops called lit_api.batch and lit_api.unbatch directly, so user implementations declaring a context parameter failed with a TypeError. Route both calls through _inject_context, the same way decode_request, predict and encode_response are already handled, in both the regular and streaming batched loops. Document the optional context argument in the LitAPI.batch/unbatch docstrings and add regression tests covering both loops.

Fixes Lightning-AI#617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Option to use the context variable from the batch and unbatch methods.

2 participants