[tokenizer] Guard communicator handle_recv against stale messages

[brucechanglongxu]

This PR fixes a race in tokenizer communicator receive handling. Late messages can arrive after watching-call cleanup; previously that path could append into cleared state and throw. I added a guard in handle_recv so it only processes messages when active watch state exists, plus a regression test that reproduces the late-message case and verifies no crash. Local test run: /usr/bin/python3 -m unittest discover -s test/registered/tokenizer -p test_tokenizer_communicator.py -v (pass).

Motivation

Fix a race in tokenizer communicator receive handling. Late messages can arrive after a watch call has already cleaned up internal state. Previously, that path could try to append into cleared state and throw.

Modifications

1. Added a guard in handle_recv so it only appends/signals when active watch state exists.
2. Added a regression test that completes a watch call, sends a late stale message, and verifies no crash.

Accuracy Tests

Ran:

/usr/bin/python3 -m unittest discover -s test/registered/tokenizer -p test_tokenizer_communicator.py -v

Pass:

test_handle_recv_ignores_stale_message_after_watch_completes

Speed Tests and Profiling

Not applicable. This is a control-flow safety fix only.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!