chore(deps): update submissions

[renovate]

This PR contains the following updates:

Package	Change	Age	Confidence	Type	Update
dask	==2024.12.0 -> ==2025.11.0				major
ddtrace (changelog)	==2.1.4 -> ==4.0.0				major
numba	==0.59.1 -> ==0.62.1				minor
numpy (changelog)	<2 -> <3				major
public.ecr.aws/lambda/python	3.10 -> 3.14			final	minor
python	3.10 -> 3.14			stage	minor
python-json-logger	==2.0.7 -> ==4.0.0				major
pyvips (changelog)	==2.2.2 -> ==3.0.0				major
s3fs	==0.4.2 -> ==2025.10.0				major
scanpy	==1.9.8 -> ==1.11.5				minor
tiledb	==0.25.0 -> ==0.35.2				minor

---

Release Notes

DataDog/dd-trace-py (ddtrace)

v4.0.0: 4.0.0

Compare Source

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.

dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.

This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview

Breaking Changes

• Support for ddtrace with Python 3.8 is removed after being deprecated in the 3.0 release line. Use ddtrace 4.x with Python 3.9 or newer.
• 32-bit linux is no longer supported. Please contact us if this blocks upgrading dd-trace-py.
• mongoengine
  • Drops support for the ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.
• CI Visibility
  • Removed deprecated entry points for the pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.
• dynamic instrumentation
  • removed the deprecated DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.
• exception replay
  • removed the deprecated DD_EXCEPTION_DEBUGGING_ENABLED variable.
• tracing
  • Deprecated methods have been removed
    • Span.set_tag_str has been removed, use Span.set_tag instead.
    • Span.set_struct_tag has been removed.
    • Span.get_struct_tag has been removed.
    • Span._pprint has been removed
    • Span.finished setter was removed, please use Span.finish() method instead.
    • Tracer.on_start_span method has been removed.
    • Tracer.deregister_on_start_span method has been removed.
    • ddtrace.trace.Pin has been removed.
    • Span.finish_with_ancestors was removed with no replacement.
  • Some methods have had their type signatures changed
    • Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> None
    • Span.get_tag typing is now get_tag(key: str) -> Optional[str]
    • Span.set_tags typing is now set_tags(tags: dict[str, str]) -> None
    • Span.get_tags typing is now get_tags() -> dict[str, str]
    • Span.set_metric typing is now set_metric(key: str, value: int | float) -> None
    • Span.get_metric typing is now get_metric(key: str) -> Optional[int | float]
    • Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> None
    • Span.get_metrics typing is now get_metrics() -> dict[str, int | float]
  • Span.record_exception's timestamp and escaped parameters are removed
• LLM Observability
  • manual instrumentation methods, including LLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.
  • LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
    • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
    • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• profiling
  • this updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.
  • The V1 stack profiler is removed. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED is now removed.
• freezegun
  • The deprecated freezegun integration is now removed.
• opentracer
  • This change removes the deprecated opentracer package
• aioredis
  • The aioredis integration has been removed.
• google_generativeai
  • The google_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
    As an alternative, you can use the recommended google_genai library and corresponding integration instead.
• openai
  • Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
    will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.
• django
  • This upgrades the default tracing behavior to enable minimal tracing mode by default (DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.
  • When DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.
• Other
  • This change removes the ddtrace.settings package. Environment variables should be used to adjust settings.
  • This change removes the deprecated non_active_span parameter to HttpPropagator.inject
  • This change removes the deprecated environment variable DEFAULT_RUNTIME_METRICS_INTERVAL.

Deprecation Notes

• Support for ddtrace with Python 3.9 is deprecated after Python 3.9 reached its end-of-life.
• Deprecates support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
• Deprecates programmatic tracing configuration via the ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling
  • Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
  • This fix resolves performance issue affecting coverage collection for Python 3.12+
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • add support for HTTPS_PROXY.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Corrected the description of the assessment argument in submit_evaluation().
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• dynamic instrumentation
  • fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.
  • This fix resolves an issue where AssertionError exceptions were silently suppressed in the _acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)
• kafka
  • This fix resolves an issue where only the first message in a batch was dispatched to Data Streams Monitoring (DSM) when consuming multiple Kafka messages
• langchain
  • This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
    The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.
  • Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.
• openai
  • This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #​14574.
• opentelemetry
  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.
• pytest plugin
  • fix for potential KeyError exceptions in test runs when gevent is detected within the environment.
• code origin
  • ensure that code location information is added to entry spans when Code Origin is enabled remotely.
• ray
  • This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().
  • This fix resolves an issue where exceptions raised in Ray child spans were not properly recorded in the trace.
  • This fix stops instrumenting internal Ray actors (those starting with underscore) that were causing excessive noise, and adds ray.data._internal to the module denylist.
• IAST
  • Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.
• tracer
  • This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0
  • This fix ensures compatibility with wrapt 2.0.0
• logging
  • Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.
• Other
  • Fix a potential race condition in the tracer.
  • Fix the Python Detector regular expression so it also detects paths ending with only the major version number.
  • Prevent a potential ResourceWarning in multiprocess scenarios.
  • Prevent startup failure when a temporary directory is not available.

Other Changes

• profiling
  • This removes the wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.

v3.19.0: 3.19.0

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• profiling
  • This updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.

Deprecation Notes

• tracing
  • Span.finished setter is deprecated, use Span.finish() method instead.
  • Span.finish_with_ancestors() is deprecated with no alternative.
• LLM Observability
  • The ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
• LLM Observability
  • Experiments can now be run multiple times by using the optional runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.
  • Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • Adds additional tags to MCP client session and tool call spans to power LLM Observability MCP tool call features.
• mcp
  • Marks client mcp tool call spans as errors when the corresponding server tool call errored

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • Add support for HTTPS_PROXY.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
• other
  • Prevent a potential ResourceWarning in multiprocess scenarios.
  • Prevent startup failure when a temporary directory is not available.

Other Changes

• profiling
  • This removes the wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.

v3.18.1: 3.18.1

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: this fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.

• LLM Observability: fix support for HTTPS_PROXY.

• Error Tracking: Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.

v3.18.0: 3.18.0

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• openai: Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
  will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.

Deprecation Notes

• tracing
  • Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
  • Span.set_tag_str is deprecated and will be removed in version 4.0.0.
    As an alternative to Span.set_tag_str, you can use Span.set_tag instead.
• profiling: The V1 stack profiler is deprecated and will be removed in 4.0. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.

New Features

• AAP
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling: Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+
• kafka: This fix resolves an issue where only the first message in a batch was dispatched to Data Streams Monitoring (DSM) when consuming multiple Kafka messages
• langchain
  • This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.
  • Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.
• LLM Observability
  • Corrected the description of the assessment argument in submit_evaluation().
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
    The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.
• openai: This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #​14574.
• pytest plugin: fix for potential KeyError exceptions in test runs when gevent is detected within the environment.
• code origin: ensure that code location information is added to entry spans when Code Origin is enabled remotely.
• ray
  • This fix resolves an issue where exceptions raised in Ray child spans were not properly recorded in the trace.
  • This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().
  • This fix stops instrumenting internal Ray actors (those starting with underscore) that were causing excessive noise, and adds ray.data._internal to the module denylist.
• IAST: Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.
• otel
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.
  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
• Fix a potential race condition in the tracer.
• profiling
  • This fix resolves an issue where AssertionError exceptions were silently suppressed in the _acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.
• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.
• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.
• tracer
  • This fix ensures compatibility with wrapt 2.0.0
  • This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0

v3.17.3: 3.17.3

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix addresses a performance issue where repository tags were fetched during the unshallow process to extract commit metadata, causing slowdowns in repositories with many tags.

• LLM Observability: Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.

• opentelemetry:

  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.

• ray: This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().

• tracer: This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0

v3.17.2: 3.17.2

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• langchain: This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.

• openai: This fix resolves an issue where using async iteration with paginated methods (e.g., async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #​14574.

• langchain: Fixes an issue where streamed responses that end before the first chunk is received would result in an IndexError.

• Fix a potential race condition in the tracer.

• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.

• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.

v3.17.1: 3.17.1

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+

• LLM Observability

  • Corrected the description of the assessment argument in submit_evaluation().
    assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled. The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.

• IAST: Fixed an issue where using weak hashing or cipher algorithms outside of a request context (e.g., during application startup) could raise an unhandled exception. The fix ensures proper error handling when IAST operations are performed without an active request context.

• profiling: Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.

v3.17.0: 3.17.0

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• LLM Observability: Experiments can now be created to be stored under a different project from the project defined in LLMObs.enable

Deprecation Notes

• LLM Observability: LLMObs.submit_evaluation_for() has been deprecated and will be removed in a future version. It will be replaced with LLMObs.submit_evaluation() which will take the signature of the original LLMObs.submit_evaluation_for() method in ddtrace version 4.0. Please use LLMObs.submit_evaluation() for submitting evaluations moving forward.
  To migrate:
  • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
  • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• tracing: Tracer.on_start_span and Tracer.deregister_on_start_span are deprecated and will be removed in v4.0.0 with no planned replacement.
• Support for ddtrace with Python 3.8 is deprecated and will be removed in version 4.0.0.

New Features

• CI Visibility: This introduces Test Impact Analysis code coverage support for Python 3.13.
• azure_eventhubs: Add support for Azure Event Hubs producers.
• azure_functions: Add support for Event Hubs triggers.
• LLM Observability
  • Introduces automatic tracing context propagation for LLM Observability traces involving asynchronous tasks created via asyncio.create_task().
  • The asyncio and futures integrations are now enabled by default on LLMObs.enable(), which enables asynchronous context propagation for those libraries.
  • The LLMObs.submit_evaluation() and LLMObs.submit_evaluation_for() methods now accept a reasoning argument to denote an explanation of the evaluation results.
  • The OpenAI integration now submits LLM spans to LLM Observability for parse() methods used for structured outputs.
  • The LLMObs.submit_evaluation_for() method now accepts a assessment argument to denote
    whether or not the evaluation is valid or correct. Accepted values are either "pass" or "fail".
• openai: Adds support for tracing the parse() methods for structured outputs on chat.completions and responses endpoints (available in OpenAI SDK >= 1.92.0).
• AAP
  • This introduces track_user_id in the ATO SDK, which is equivalent to track_user but does not require the login, only the user id.
  • This introduces supports for custom scanners for data classification.

Bug Fixes

• AAP
  • This fix resolves an issue where downstream request analysis would not match headers in rules when using requests with urllib3\<2.
  • This PR is a tentative fix for rare memory problems with libddwaf that we were unable to reproduce for now.
• Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
• CI Visibility
  • This fix resolves an issue where tests would be incorrectly detected as third-party code if a third-party package containing a folder with the same name as the tests folder was installed. For instance, the sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.
  • This fix resolves an issue with our coverage implementation for Python versions 3.12+ that affects generated bytecode that isn't mapped to a line in the code
• LLM Observability: Resolves an issue with the Google GenAI integration where processing token metrics would sometimes be skipped if the LLM message had no text part.
• grpc: This fix resolves an issue where the internal span was left active in the caller when using the future interface.
• Profiling: prevent potential deadlocks with thread pools.
• ray
  • This fix resolves an issue where submitting Ray jobs caused an AttributeError crash in certain configurations.
  • This fix resolves an issue where long-running job spans could remain unfinished when an exception occurred during job submission.
  • This fix resolves an issue where long-running spans did not preserve the correct resource name when being recreated.
• otel: Ensures the /v1/logs path is correctly added to prevent log payloads from being dropped by the Agent when using OTEL_EXPORTER_OTLP_ENDPOINT configuration. Metrics and traces are unaffected.

v3.16.4: 3.16.4

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• langchain: This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.

• Fix a potential race condition in the tracer.

• Fix the Python Detector regular expression so it also detects paths ending with only the major version number.

• logging: Fixed ddtrace internal logging when trace-log correlation is disabled. Prevents ValueError: Formatting field not found in record: 'dd.service'.

v3.16.3: 3.16.3

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+

• LLM Observability: Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled. The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.

• profiling: Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.

v3.16.2: 3.16.2

Compare Source

Bug Fixes

• Pin to wrapt<2 until we can ensure full compatibility with the breaking changes

v3.16.1: 3.16.1

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix resolves an issue where tests would be incorrectly detected as third-party code if a third-party package containing a folder with the same name as the tests folder was installed. For instance, the sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.

• grpc: This fix resolves an issue where the internal span was left active in the caller when using the future interface.

• ray: This fix resolves an issue where submitting Ray jobs caused an AttributeError crash in certain configurations.

• ray: This fix resolves an issue where long-running spans did not preserve the correct resource name when being recreated.

• ray: This fix resolves an issue where long-running job spans could remain unfinished when an exception occurred during job submission.

• AAP: This PR is a tentative fix for rare memory problems with libddwaf that we were unable to reproduce for now.

• Internal: Fix some modules being unloaded too soon when using pytest + ddtrace + gevent.

v3.16.0: 3.16.0

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• This change updates library injection logic to work under Python 3.14.
• This change adds support and tests for Python 3.14 to much of the library's functionality. The following products and integrations still do not work with Python 3.14:
  • Profiling
  • IAST
  • datastreams
  • ci_visibility
  • pytest
  • django - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django_hosts - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • djangorestframework - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • django:celery - django version 6.1, which will be compatible with Python 3.14, is not yet released
  • dramatiq - dramatiq doesn't yet have a release supporting 3.14
  • grpc_aio - some tests in the suite don't work with pytest-asyncio >= 1.0
  • rq - rq doesn't work with python 3.14
  • sqlite3 - pysqlite3-binary doesn't yet support python 3.14
  • opentelemetry - opentelemetry-exporter-otlp doesn't yet work with Python 3.14
  • openai - tiktoken doesn't yet work with Python 3.14
  • ai_guard_langchain - tiktoken doesn't yet work with Python 3.14
  • openai_agents
  • langchain
  • langgraph - tiktoken doesn't yet work with Python 3.14
  • litellm - tiktoken doesn't yet work with Python 3.14
  • google_generativeai - protobuf doesn't yet work with Python 3.14
  • vertexai
  • crewai - tiktoken doesn't yet work with Python 3.14
  • ray - ray doesn't yet work with Python 3.14
  • kafka - confluent-kafka doesn't yet work with Python 3.14
  • aws_lambda - datadog-lambda doesn't yet work with Python 3.14
  • llmobs - ragas doesn't yet work with Python 3.14
  • appsec_integrations_fastapi

Deprecation Notes

• vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9.

New Features

• opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not found
  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to delta
  • OTEL_METRIC_EXPORT_INTERVAL is set to 10000
  • OTEL_METRIC_EXPORT_TIMEOUT is set to 7500

• LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.

• ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.

  To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.

Bug Fixes

• AAP: This fix resolves an issue where stream endpoints with daphne/django where unresponsive due to an asyncio error.
• CI Visibility: This fix resolves an issue where code imported at module level but not executed during a test would not be considered by Test Impact Analysis as impacting the test. For example, a test using a constant imported from some other module would not count the constant definition among its impacting lines, because the constant definition is not executed during the test, but rather when the module was imported. With this change, code executed at import time is also included among the impacted lines of a test.
• google-adk: Fixes an AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.
• opentelemetry: Fixes the parsing of OTLP metrics exporter configurations and the operation to automatically append the v1/metrics path to HTTP OTLP endpoints.
• langchain: Resolves an issue where langchain patching would throw an ImportError for when using langchain_core>=0.3.76.
• LLM Observability
  • ensures APM is disabled when DD_APM_TRACING_ENABLED=0 when using LLM Observability.
  • Resolves an issue where model IDs were not being parsed correctly if the model ID was an inference profile ID in the bedrock integration.
  • enable the backend to differentiate AI Obs spans from other DJM spans, so that customers are not billed for AI Observability spans as part of their APM bill.

Other Changes

• sampling: Add more debug logs to help debug sampling issues.

v3.15.1: 3.15.1

Compare Source

Bug Fixes

• Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.

• CI Visibility: This fix resolves an issue where tests would be incorrectly detected as third-party code if a third-party package containing a folder with the same name as the tests folder was installed. For instance, the sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.

• langchain: Resolves an issue where langchain patching would throw an ImportError for when using langchain_core>=0.3.76.

v3.15.0: 3.15.0

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

New Features

• google-adk: Adds APM tracing and LLM Observability support for the Google ADK library (google-adk). Support includes APM tracing and LLM Observability for agent runs, tool calls, and code execution.
• django: This introduces the DD_DJANGO_TRACING_MINIMAL environment variable for performance-sensitive applications. When enabled, this disables Django ORM, cache, and template instrumentation while keeping middleware instrumentation enabled. This significantly reduces overhead by removing Django-specific spans while preserving visibility into the underlying database drivers, cache clients, and other integrations. For example, with this enabled, Django ORM query spans are disabled but database driver spans (e.g., psycopg, MySQLdb) will still be created. To enable minimal tracing, set DD_DJANGO_TRACING_MINIMAL=true.
• AWS: adds aws.partition tag onto AWS traces based on the region for the boto, botocore, and aiobotocore integrations.
• AAP: This extends downstream request analysis (API10) to the requests package. Previously, downstream request analysis was only supported in the standard cpython api (urllib).
• dynamic instrumentation/exception replay/code origin for spans: added support for the latest Datadog agent intake for snapshots. This requires a minimum agent version of 7.49.0.
• CI Visibility: This introduces the env var DD_CIVISIBILITY_ENABLED (with default value True) so it can be disabled to avoid sending traces to the Test Visibility product from the test runners.
• azure_servicebus: Add distributed tracing support for sending batches with Azure Service Bus producers.
• azure_functions: Use span links to connect Service Bus trigger consumers to the producers that send the messages.
• tracing: Added support for resource renaming, an experimental feature that lets the Datadog platform adjust the resource field on web request spans when the endpoint cannot be correctly deduced. Enable the feature by setting DD_TRACE_RESOURCE_RENAMING_ENABLED="true"
• Code Security (IAST)
  • Untrusted Serialization detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
  • Reduce false positives if md5 or sha1 functions have the parameter usedforsecurity=False.
• LLM Observability: Extends the prompt structure to add tags and chat_template, and a new Prompt TypedDict class that would be used in annotation and annotation_context.
• LLM Observability: Datasets & Experiments SDK now has summary evaluators support.

Bug Fixes

• CI Visibility: This fix solves an issue where the ITR skip count metric was aggregating skipped tests even when skipping level was set to suite. It will now count appropriately (skipped suites or skipped tests) depending on ITR skip level.
• sampling: This change prevents the DatadogSampler from getting recreated whenever the SpanAggregator is reset, and instead updates the rate limiter that the sampler uses.
• dynamic instrumentation: fix an issue that prevented multiple probes on the same location from being instrumented.
• exception replay
  • prevent Celery from crashing when a task raises a custom exception with mandatory arguments.
  • ensure that value capture starts from the leaf frame of the innermost exception.
• tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.
• AAP
  • This fix resolves an issue where the endpoint discovery feature could generate a crash for flask at startup.
  • This fix disables grpc threat monitoring, as it could generate false positives.
• libinjection: allow python module executed with -m entries in the denylist.
• profiling
  • Upgrades echion to resolve segmentation faults that can happen on services with a lot of asyncio.Tasks.
  • Fix crash in memory profiling when garbage collection is triggered while sampling a PyObject_Realloc call, which can lead to accessing freed memory.
  • Profiling won't load if --skip-atexit is not set when --lazy or --lazy-apps is set on uWSGI<2.0.30. This is to prevent crashes from profiling native extension modules. See unbit/uwsgi#2726 for details.
• RemoteConfig: Fixes an issue introduced in Python 3.13 where creating a shared array with the c_char type raised a TypeError, this now uses the 'c' typecode for better compatibility across versions.
• source code integration: check that DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.

v3.14.4: 3.14.4

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• AAP:

  • This fix resolves an issue where stream endpoints with daphne/django where unresponsive due to an asyncio error.
  • This fix resolves an issue where the endpoint discovery feature could generate a crash for flask at startup.

• CI Visibility: This fix solves an issue where the ITR skip count metric was aggregating skipped tests even when skipping level was set to suite. It will now count appropriately (skipped suites or skipped tests) depending on ITR skip level.

v3.14.3: 3.14.3

Compare Source

Estimated end-of-life date, accurate to within three months: 09-2026
See the support level definitions for more information.

Bug Fixes

• exception replay: prevent Celery from crashing when a task raises a custom exception with mandatory arguments.

• tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.

• libinjection: allow deny listing python modules executed with python -m and deny py_compile.

v3.14.2: 3.14.2

Compare Source

Estimated end-of-life date, accurate to within three months: 09-2026
See the support level definitions for more information.

Bug Fixes

• sampling: This change prevents the DatadogSampler from getting recreated whenever the SpanAggregator is reset, and
  instead updates the rate limiter that the sampler uses.

• dynamic instrumentation: fix an issue that prevented multiple probes on the same location from being instrumented.

• source code integration: check that DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.

v3.14.1: 3.14.1

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• AAP:
  • This fix disables grpc threat monitoring, as it could generate false positives.

v3.14.0: 3.14.0

Compare Source

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

New Features

• ai_guard
  • add AI Guard evaluations support to LangChain instrumentation
  • add evaluation support to streaming LangChain APIs
• LLM Observability
  • ml_app is now optional, defaulting to service. while it is still recommended to set ml_app, enabling LLM Observability will no longer throw if one is not provided or propagated from an upstream service.
  • This introduces a tool_definitions parameter to the LLMObs.annotate() method for tool calling scenarios. Users can now pass a list of tool definition dictionaries directly to annotate LLM spans with available tools. Each tool definition must include a name (string) field, with optional description (string) and schema (JSON-serializable dictionary) fields.
  • Experiments (beta) dataset methods (create_dataset, create_dataset_from_csv and pull_dataset) argument name is changed to dataset_name.

Bug Fixes

• AAP
  • make sure the status code for downstream requests is properly sent to libddwaf.
  • Resolves an incompatibility with gevent>=25.8.1 that would cause a deadlock when starting the waf via remote config.
• django: Fixes issue causing ValueError: coroutine already executing on Python 3.13+ with django.utils.decorators.async_only_middleware.
• asyncpg: Fix the error "Error: expected pool connect callback to return an instance of 'asyncpg.connection.Connection', got 'ddtrace.contrib.internal.asyncpg.patch._TracedConnection'" due to using the custom connect option. With this fix, postgres.connect spans will be created wh

[github-actions]

Deployment Summary

• backend_url: https://pr-7100-backend.rdev.single-cell.czi.technology
• frontend_url: https://pr-7100-frontend.rdev.single-cell.czi.technology
• delete_db_task_definition_arn: arn:aws:ecs:us-west-2:***:task-definition/dp-rdev-pr-7100-deletion:1
• migrate_db_task_definition_arn: arn:aws:ecs:us-west-2:***:task-definition/dp-rdev-pr-7100-migration:1