-
Notifications
You must be signed in to change notification settings - Fork 0
Add Datadog Serverless Compatibility Layer #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 11 commits
a81c037
1ad20c4
bc7f58a
1930d43
6b5d864
e47a8e8
069364e
f1ebdbb
c2ec90d
7038747
8be2c68
bd6753f
ebec5b9
eaf0a55
bafae31
30c486e
848c0d5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
[flake8] | ||
max-line-length = 120 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
name: Publish packages on PyPI | ||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
publish-destination: | ||
type: choice | ||
description: Publish to PyPI or TestPyPI? | ||
default: "TestPyPI" | ||
options: | ||
- "TestPyPI" | ||
- "PyPI" | ||
|
||
permissions: {} | ||
|
||
jobs: | ||
download-binaries: | ||
runs-on: ubuntu-latest | ||
outputs: | ||
package-version: ${{ steps.package.outputs.package-version }} | ||
serverless-compat-version: ${{ steps.serverless-compat-binary.outputs.serverless-compat-version }} | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
- id: package | ||
run: | | ||
PACKAGE_VERSION=$(awk -F '"' '/version = / {print $2}' pyproject.toml) | ||
echo "package-version=$PACKAGE_VERSION" >> "$GITHUB_OUTPUT" | ||
- id: serverless-compat-binary | ||
run: | | ||
LIBDATADOG_RESPONSE=$(curl -s "https://api.github.com/repos/datadog/libdatadog/releases") | ||
SERVERLESS_COMPAT_VERSION=$(echo "$LIBDATADOG_RESPONSE" | jq -r --arg pattern "sls-v[0-9]*\.[0-9]*\.[0-9]*" '.[] | select(.tag_name | test($pattern)) | .tag_name' | sort -V | tail -n 1) | ||
|
||
echo "Using version ${SERVERLESS_COMPAT_VERSION} of Serverless Compatibility Layer binary" | ||
echo "serverless-compat-version=$(echo "$SERVERLESS_COMPAT_VERSION" | jq -rR 'ltrimstr("sls-")')" >> "$GITHUB_OUTPUT" | ||
|
||
curl --output-dir ./temp/ --create-dirs -O -s -L "https://github.com/DataDog/libdatadog/releases/download/${SERVERLESS_COMPAT_VERSION}/datadog-serverless-agent.zip" | ||
unzip ./temp/datadog-serverless-agent.zip -d ./temp/datadog-serverless-agent | ||
|
||
mkdir -p datadog_serverless_compat/bin/linux-amd64 datadog_serverless_compat/bin/windows-amd64 | ||
cp ./temp/datadog-serverless-agent/datadog-serverless-agent-linux-amd64/datadog-serverless-trace-mini-agent datadog_serverless_compat/bin/linux-amd64/datadog-serverless-compat | ||
cp ./temp/datadog-serverless-agent/datadog-serverless-agent-windows-amd64/datadog-serverless-trace-mini-agent.exe datadog_serverless_compat/bin/windows-amd64/datadog-serverless-compat.exe | ||
- uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3 | ||
with: | ||
name: bin | ||
path: datadog_serverless_compat/bin | ||
build: | ||
runs-on: ubuntu-latest | ||
needs: [download-binaries] | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
- uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 | ||
with: | ||
path: datadog_serverless_compat | ||
- run: | | ||
chmod +x datadog_serverless_compat/bin/linux-amd64/datadog-serverless-compat | ||
chmod +x datadog_serverless_compat/bin/windows-amd64/datadog-serverless-compat.exe | ||
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0 | ||
with: | ||
python-version: "3.9 - 3.12" | ||
- uses: snok/install-poetry@v76e04a911780d5b312d89783f7b1cd627778900a # v1.4.1 | ||
duncanpharvey marked this conversation as resolved.
Show resolved
Hide resolved
|
||
with: | ||
version: 1.8.5 | ||
- run: poetry build | ||
- uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3 | ||
with: | ||
name: dist | ||
path: dist | ||
publish: | ||
runs-on: ubuntu-latest | ||
needs: [build] | ||
steps: | ||
- uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 | ||
- uses: pypa/gh-action-pypi-publish@67339c736fd9354cd4f8cb0b744f2b82a74b5c70 # v1.12.3 | ||
with: | ||
packages-dir: dist/ | ||
password: ${{ github.event.inputs.publish-destination == 'PyPI' && secrets.PYPI_SERVERLESS_COMPAT_API_TOKEN || secrets.PYPI_SERVERLESS_COMPAT_API_TOKEN_TEST }} | ||
repository-url: ${{ github.event.inputs.publish-destination == 'PyPI' && 'https://upload.pypi.org/legacy/' || 'https://test.pypi.org/legacy/' }} | ||
release: | ||
if: github.event.inputs.publish-destination == 'PyPI' | ||
runs-on: ubuntu-latest | ||
needs: [publish] | ||
permissions: | ||
contents: write | ||
env: | ||
PACKAGE_VERSION: ${{ needs.download-binaries.outputs.package-version }} | ||
SERVERLESS_COMPAT_VERSION: ${{ needs.download-binaries.outputs.serverless-compat-version }} | ||
steps: | ||
- uses: softprops/action-gh-release@01570a1f39cb168c169c802c3bceb9e93fb10974 # v2.1.0 | ||
with: | ||
body: "Uses [${{ env.SERVERLESS_COMPAT_VERSION }}](https://github.com/DataDog/libdatadog/releases/tag/sls-${{ env.SERVERLESS_COMPAT_VERSION }}) of the Serverless Compatibility Layer binary." | ||
draft: true | ||
tag_name: "v${{ env.PACKAGE_VERSION }}" | ||
generate_release_notes: true | ||
make_latest: true |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
# Byte-compiled / optimized / DLL files | ||
__pycache__/ | ||
|
||
# Compiled Binaries | ||
bin | ||
|
||
# Distribution / packaging | ||
dist/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
Datadog datadog-serverless-compat-js | ||
Copyright 2025 Datadog, Inc. | ||
|
||
This product includes software developed at Datadog (https://www.datadoghq.com/). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,54 @@ | ||
# datadog-serverless-compat-py | ||
Datadog Serverless Compatibility Layer for Python | ||
# Datadog Serverless Compatibility Layer for Python | ||
|
||
Datadog library for Python to enable tracing and custom metric submission from Azure Functions and Google Cloud Run Functions (1st gen). | ||
|
||
## Installation | ||
|
||
1. Install the Datadog Serverless Compatibility Layer. | ||
``` | ||
pip install datadog-serverless-compat | ||
``` | ||
|
||
2. Install the Datadog Tracing Library following the official documentation for [Tracing Python Applications](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/python). | ||
|
||
3. Add the Datadog Serverless Compatibility Layer and the Datadog Tracer in code. | ||
|
||
``` | ||
from datadog_serverless_compat import start | ||
from ddtrace import tracer, patch_all | ||
|
||
start() | ||
patch_all() | ||
``` | ||
|
||
## Configuration | ||
|
||
1. Set Datadog environment variables | ||
- `DD_API_KEY` = `<YOUR API KEY>` | ||
- `DD_SITE` = `datadoghq.com` | ||
- `DD_ENV` = `<ENVIRONMENT` | ||
- `DD_SERVICE` = `<SERVICE NAME>` | ||
- `DD_VERSION` = `<VERSION>` | ||
- `DD_TRACE_STATS_COMPUTATION_ENABLED` = `true` | ||
duncanpharvey marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
The default Datadog site is **datadoghq.com**. To use a different site, set the `DD_SITE` environment variable to the desired destination site. See [Getting Started with Datadog Sites](https://docs.datadoghq.com/getting_started/site/) for the available site values. | ||
|
||
The `DD_SERVICE`, `DD_ENV`, and `DD_VERSION` settings are configured from environment variables in Azure and are used to tie telemetry together in Datadog as tags. Read more about [Datadog Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging). | ||
|
||
[Trace Metrics](https://docs.datadoghq.com/tracing/metrics/metrics_namespace/) are enabled by default but can be disabled with the `DD_TRACE_STATS_COMPUTATION_ENABLED` environment variable. | ||
|
||
Enable debug logs for the Datadog Serverless Compatibility Layer with the `DD_LOG_LEVEL` environment variable: | ||
|
||
``` | ||
DD_LOG_LEVEL=debug | ||
``` | ||
|
||
Alternatively disable logs for the Datadog Serverless Compatibility Layer with the `DD_LOG_LEVEL` environment variable: | ||
|
||
``` | ||
DD_LOG_LEVEL=off | ||
``` | ||
|
||
2. For additional tracing configuration options, see the [official documentation for Datadog trace client](https://datadoghq.dev/dd-trace-js/). | ||
|
||
3. If installing to Azure Functions, install the [Datadog Azure Integration](https://docs.datadoghq.com/integrations/azure/#setup) and set tags on your Azure Functions to further extend unified service tagging. This allows for Azure Function metrics and other Azure metrics to be correlated with traces. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
from datadog_serverless_compat.logger import initialize_logging | ||
|
||
initialize_logging(__name__) | ||
|
||
from datadog_serverless_compat.main import start # noqa: E402 F401 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
import logging | ||
import os | ||
|
||
try: | ||
# Added in version 3.11 | ||
level_mapping = logging.getLevelNamesMapping() | ||
except AttributeError: | ||
level_mapping = {name: num for num, name in logging._levelToName.items()} | ||
|
||
# https://docs.datadoghq.com/agent/troubleshooting/debug_mode/?tab=agentv6v7#agent-log-level | ||
level_mapping.update( | ||
{ | ||
"TRACE": 5, | ||
"WARN": logging.WARNING, | ||
"OFF": 100, | ||
} | ||
) | ||
|
||
|
||
def initialize_logging(name): | ||
logger = logging.getLogger(name) | ||
str_level = (os.environ.get("DD_LOG_LEVEL") or "INFO").upper() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The behavior is slightly different when
On further review I prefer your suggestion because it does result in the warning to show that an empty string is an invalid log level.
I also added a test case to cover when There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Regarding the test runner workflow - I'm planning on working on that in a separate PR in the interest of getting this initial package release completed. |
||
level = level_mapping.get(str_level) | ||
|
||
if level is None: | ||
logger.setLevel(logging.INFO) | ||
logger.warning("Invalid log level: %s Defaulting to INFO", str_level) | ||
else: | ||
logger.setLevel(level) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,96 @@ | ||
from enum import Enum | ||
import logging | ||
import os | ||
from subprocess import Popen | ||
import sys | ||
|
||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
class CloudEnvironment(Enum): | ||
AZURE_FUNCTION = "Azure Function" | ||
GOOGLE_CLOUD_RUN_FUNCTION_1ST_GEN = "Google Cloud Run Function 1st gen" | ||
GOOGLE_CLOUD_RUN_FUNCTION_2ND_GEN = "Google Cloud Run Function 2nd gen" | ||
UNKNOWN = "Unknown" | ||
|
||
|
||
def get_environment(): | ||
if ( | ||
os.environ.get("FUNCTIONS_EXTENSION_VERSION") is not None | ||
and os.environ.get("FUNCTIONS_WORKER_RUNTIME") is not None | ||
): | ||
return CloudEnvironment.AZURE_FUNCTION | ||
|
||
if ( | ||
os.environ.get("FUNCTION_NAME") is not None | ||
and os.environ.get("GCP_PROJECT") is not None | ||
): | ||
return CloudEnvironment.GOOGLE_CLOUD_RUN_FUNCTION_1ST_GEN | ||
|
||
if ( | ||
os.environ.get("K_SERVICE") is not None | ||
and os.environ.get("FUNCTION_TARGET") is not None | ||
): | ||
return CloudEnvironment.GOOGLE_CLOUD_RUN_FUNCTION_2ND_GEN | ||
|
||
return CloudEnvironment.UNKNOWN | ||
|
||
|
||
def get_binary_path(): | ||
# Use user defined path if provided | ||
binary_path = os.getenv("DD_SERVERLESS_COMPAT_PATH") | ||
if binary_path is not None: | ||
return binary_path | ||
|
||
binary_path_os_folder = os.path.join( | ||
os.path.dirname(__file__), | ||
"bin/windows-amd64" if sys.platform == "win32" else "bin/linux-amd64", | ||
) | ||
binary_extension = ".exe" if sys.platform == "win32" else "" | ||
binary_path = os.path.join( | ||
binary_path_os_folder, f"datadog-serverless-compat{binary_extension}" | ||
) | ||
|
||
return binary_path | ||
|
||
|
||
def start(): | ||
environment = get_environment() | ||
logger.debug(f"Environment detected: {environment}") | ||
|
||
if environment == CloudEnvironment.UNKNOWN: | ||
logger.error( | ||
f"{environment} environment detected, will not start the Datadog Serverless Compatibility Layer" | ||
) | ||
return | ||
|
||
logger.debug(f"Platform detected: {sys.platform}") | ||
|
||
if sys.platform not in {"win32", "linux"}: | ||
logger.error( | ||
( | ||
f"Platform {sys.platform} detected, the Datadog Serverless Compatibility Layer is only supported", | ||
" on Windows and Linux", | ||
) | ||
) | ||
return | ||
|
||
binary_path = get_binary_path() | ||
logger.debug(f"Spawning process from binary at path {binary_path}") | ||
|
||
if not os.path.exists(binary_path): | ||
logger.error( | ||
f"Serverless Compatibility Layer did not start, could not find binary at path {binary_path}" | ||
) | ||
return | ||
|
||
try: | ||
logger.debug( | ||
f"Trying to spawn the Serverless Compatibility Layer at path: {binary_path}" | ||
) | ||
Popen(binary_path) | ||
except Exception as e: | ||
logger.error( | ||
f"An unexpected error occurred while spawning Serverless Compatibility Layer process: {repr(e)}" | ||
) |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
[tool.poetry] | ||
name = "datadog-serverless-compat" | ||
version = "0.1.0" | ||
description = "Datadog Serverless Compatibility Layer for Python" | ||
authors = ["Datadog, Inc. <[email protected]>"] | ||
license = "Apache-2.0" | ||
readme = "README.md" | ||
repository = "https://github.com/DataDog/datadog-serverless-compat-py" | ||
keywords = [ | ||
"datadog", | ||
"azure", | ||
"google", | ||
"functions" | ||
] | ||
include = [ | ||
{ path = "datadog_serverless_compat/bin/**/*", format = ["sdist", "wheel"] } | ||
] | ||
classifiers = [ | ||
"Development Status :: 4 - Beta", | ||
duncanpharvey marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"Programming Language :: Python :: 3", | ||
"Programming Language :: Python :: 3.9", | ||
"Programming Language :: Python :: 3.10", | ||
"Programming Language :: Python :: 3.11", | ||
"Programming Language :: Python :: 3.12", | ||
] | ||
|
||
[tool.poetry.dependencies] | ||
python = "^3.9" | ||
|
||
|
||
[build-system] | ||
requires = ["poetry-core"] | ||
build-backend = "poetry.core.masonry.api" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, it looks to me like this is grabbing the current version number. Meaning, that in order to do a proper release, you've gotta first bump the version number in the pyproject.toml file, then run the gh action.
I'm wondering if instead maybe we could pass the new package version in as an option? You'd then need the gh action to open a PR though. Maybe another idea is to confirm in the input options that the user knows that the pyproject.toml version must first be updated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's correct
Yeah I think some sort of validation is needed. I see the version for
datadog-lambda-python
is bumped and committed prior to a release. Is it just a matter of documenting the release steps internally?DataDog/datadog-lambda-python#543
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you get to define the release steps however you want. We certainly don't stick to a single way of doing it across all our repos. I think you should do what you think will be most appropriate for this repo. Just make sure to be explicit as to when you think the version should be bumped in the pyproject.toml file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this workflow fail if we accidentally try to publish the same version twice?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@apiarian-datadog According to the PyPI help docs, the same version will not be allowed to be published twice.
https://pypi.org/help/#:~:text=To%20avoid%20this%20situation%20in,then%20upload%20the%20new%20distribution.