[mangeshraut712]

Summary

Adds rt.llm.EchoLLM, an offline model for tests that do not have an API key.

On each chat call it returns the full message history it received, one message per line (role: content), then the line This is an echo of the test model. It never requests a tool call, so an agent with tools still finishes in one turn. structured and astructured raise NotImplementedError. Usage is reported as zero tokens, zero cost, and zero latency, with the model name echo.

I left the provider as ModelProvider.UNKNOWN. This is not a real provider, so a new enum value did not seem useful.

Fixes #1682

Type of change

• Feature
• Docs

Checklist

• Lint & format pass (ruff check . and ruff format --check .)
• Tests added and pass locally (pytest -c pyproject.toml packages/railtracks/tests: 3485 passed, 8 skipped)
• Docs updated (mkdocs build --strict succeeded)
• Breaking changes include migration notes

Notes

The docs section is on the LLM Setup page. The runnable snippet is docs/scripts/documentation/echo_llm.py.

  

[CoronRing]

Thanks for this, @mangeshraut712! The implementation is clean and matches the issue spec. Tests, lint, mypy, and mkdocs build --strict all pass on my side.

One addition before we merge: streaming support. At the moment, rt.astream on an agent that uses EchoLLM fails with NotImplementedError: EchoLLM does not support streamed chat calls. A test model is especially useful for testing streaming flows offline, so we'd like it to support them too.

Suggested behaviour ( do not need 100%, just something I though reasonable)

• A fixed 5-second total. Every streamed call should take 5 seconds in total, however long the echo text is.
• One chunk per second. Split the full echo text into 5 chunks and yield one each second. For example, a 10-character reply streams as 2 characters per second. If the length doesn't divide evenly, give the extra characters to the first chunks, so sizes differ by at most one (12 characters becomes 3, 3, 2, 2, 2).
• Chunks, not single characters. Don't yield character by character.
• No token concept. Chunks are plain slices of the text. Usage stays at zero, as it is now.
• Final Response. After the last chunk, yield the same Response that _chat returns. Joining the chunks must give exactly that response's content.

Where it goes

• Override _astream_chat and _astream_chat_with_tools in EchoLLM. Both stream the same echo, and the tools version still never requests a tool call.
• Wait with await asyncio.sleep(...) between chunks, not time.sleep, so the event loop isn't blocked.
• Leave _astream_structured alone. The NotImplementedError it inherits from ModelBase is correct, since structured output stays unsupported.

A rough shape:

async def _astream_chat(self, messages: MessageHistory) -> AsyncGenerator[str | Response, None]:
    response = self._echo(messages)
    for chunk in _split_evenly(response.message.content, _STREAM_CHUNKS):
        await asyncio.sleep(_STREAM_SECONDS / _STREAM_CHUNKS)
        yield chunk
    yield response

Tests

Please monkeypatch asyncio.sleep in the tests so the suite doesn't actually wait 5 seconds per test. Then check that:

• exactly 5 chunks are yielded, their sizes differ by at most one, and joined together they equal the chat content;
• the sleep durations add up to 5 seconds;
• the final item is a Response matching chat;
• rt.astream on an agent works both with and without tools.
• No need for doc on streaming support, as streaming is very much expected in test llm

Thanks again! Once this is in, we're happy to merge.

[CoronRing]

@Amir-R25 Want your quick second eye on this when ur back, as it adds another component of llm module, though I thought really reasonable.

[mangeshraut712]

Thanks! Added streaming in 879aea8. Each call sends the echo in 5 chunks, one per second, then the same Response as chat. Tests use a fake sleep and cover rt.astream with and without tools. Structured streaming still raises NotImplementedError.

[CoronRing]

Great work; all is clear as day on my side! If you enjoyed the framework, please tell your friends and star the repo so it gets a bit of attention.

Thanks & love from the maintenance team >_<