Skip to content

Add Datadog Serverless Compatibility Layer #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jan 14, 2025
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[flake8]
max-line-length = 120
94 changes: 94 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
name: Publish packages on PyPI

on:
workflow_dispatch:
inputs:
publish-destination:
type: choice
description: Publish to PyPI or TestPyPI?
default: "TestPyPI"
options:
- "TestPyPI"
- "PyPI"

permissions: {}

jobs:
download-binaries:
runs-on: ubuntu-latest
outputs:
package-version: ${{ steps.package.outputs.package-version }}
serverless-compat-version: ${{ steps.serverless-compat-binary.outputs.serverless-compat-version }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- id: package
run: |
PACKAGE_VERSION=$(awk -F '"' '/version = / {print $2}' pyproject.toml)
echo "package-version=$PACKAGE_VERSION" >> "$GITHUB_OUTPUT"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, it looks to me like this is grabbing the current version number. Meaning, that in order to do a proper release, you've gotta first bump the version number in the pyproject.toml file, then run the gh action.

I'm wondering if instead maybe we could pass the new package version in as an option? You'd then need the gh action to open a PR though. Maybe another idea is to confirm in the input options that the user knows that the pyproject.toml version must first be updated?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, it looks to me like this is grabbing the current version number. Meaning, that in order to do a proper release, you've gotta first bump the version number in the pyproject.toml file, then run the gh action.

That's correct

I'm wondering if instead maybe we could pass the new package version in as an option? You'd then need the gh action to open a PR though. Maybe another idea is to confirm in the input options that the user knows that the pyproject.toml version must first be updated?

Yeah I think some sort of validation is needed. I see the version for datadog-lambda-python is bumped and committed prior to a release. Is it just a matter of documenting the release steps internally?

DataDog/datadog-lambda-python#543

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you get to define the release steps however you want. We certainly don't stick to a single way of doing it across all our repos. I think you should do what you think will be most appropriate for this repo. Just make sure to be explicit as to when you think the version should be bumped in the pyproject.toml file.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this workflow fail if we accidentally try to publish the same version twice?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apiarian-datadog According to the PyPI help docs, the same version will not be allowed to be published twice.

https://pypi.org/help/#:~:text=To%20avoid%20this%20situation%20in,then%20upload%20the%20new%20distribution.

Screenshot 2025-01-14 at 3 32 32 PM

- id: serverless-compat-binary
run: |
LIBDATADOG_RESPONSE=$(curl -s "https://api.github.com/repos/datadog/libdatadog/releases")
SERVERLESS_COMPAT_VERSION=$(echo "$LIBDATADOG_RESPONSE" | jq -r --arg pattern "sls-v[0-9]*\.[0-9]*\.[0-9]*" '.[] | select(.tag_name | test($pattern)) | .tag_name' | sort -V | tail -n 1)

echo "Using version ${SERVERLESS_COMPAT_VERSION} of Serverless Compatibility Layer binary"
echo "serverless-compat-version=$(echo "$SERVERLESS_COMPAT_VERSION" | jq -rR 'ltrimstr("sls-")')" >> "$GITHUB_OUTPUT"

curl --output-dir ./temp/ --create-dirs -O -s -L "https://github.com/DataDog/libdatadog/releases/download/${SERVERLESS_COMPAT_VERSION}/datadog-serverless-agent.zip"
unzip ./temp/datadog-serverless-agent.zip -d ./temp/datadog-serverless-agent

mkdir -p datadog_serverless_compat/bin/linux-amd64 datadog_serverless_compat/bin/windows-amd64
cp ./temp/datadog-serverless-agent/datadog-serverless-agent-linux-amd64/datadog-serverless-trace-mini-agent datadog_serverless_compat/bin/linux-amd64/datadog-serverless-compat
cp ./temp/datadog-serverless-agent/datadog-serverless-agent-windows-amd64/datadog-serverless-trace-mini-agent.exe datadog_serverless_compat/bin/windows-amd64/datadog-serverless-compat.exe
- uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: bin
path: datadog_serverless_compat/bin
build:
runs-on: ubuntu-latest
needs: [download-binaries]
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
path: datadog_serverless_compat
- run: |
chmod +x datadog_serverless_compat/bin/linux-amd64/datadog-serverless-compat
chmod +x datadog_serverless_compat/bin/windows-amd64/datadog-serverless-compat.exe
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.9 - 3.12"
- uses: snok/install-poetry@v76e04a911780d5b312d89783f7b1cd627778900a # v1.4.1
with:
version: 1.8.5
- run: poetry build
- uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: dist
path: dist
publish:
runs-on: ubuntu-latest
needs: [build]
steps:
- uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
- uses: pypa/gh-action-pypi-publish@67339c736fd9354cd4f8cb0b744f2b82a74b5c70 # v1.12.3
with:
packages-dir: dist/
password: ${{ github.event.inputs.publish-destination == 'PyPI' && secrets.PYPI_SERVERLESS_COMPAT_API_TOKEN || secrets.PYPI_SERVERLESS_COMPAT_API_TOKEN_TEST }}
repository-url: ${{ github.event.inputs.publish-destination == 'PyPI' && 'https://upload.pypi.org/legacy/' || 'https://test.pypi.org/legacy/' }}
release:
if: github.event.inputs.publish-destination == 'PyPI'
runs-on: ubuntu-latest
needs: [publish]
permissions:
contents: write
env:
PACKAGE_VERSION: ${{ needs.download-binaries.outputs.package-version }}
SERVERLESS_COMPAT_VERSION: ${{ needs.download-binaries.outputs.serverless-compat-version }}
steps:
- uses: softprops/action-gh-release@01570a1f39cb168c169c802c3bceb9e93fb10974 # v2.1.0
with:
body: "Uses [${{ env.SERVERLESS_COMPAT_VERSION }}](https://github.com/DataDog/libdatadog/releases/tag/sls-${{ env.SERVERLESS_COMPAT_VERSION }}) of the Serverless Compatibility Layer binary."
draft: true
tag_name: "v${{ env.PACKAGE_VERSION }}"
generate_release_notes: true
make_latest: true
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Byte-compiled / optimized / DLL files
__pycache__/

# Compiled Binaries
bin

# Distribution / packaging
dist/
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright [yyyy] [name of copyright owner]
Copyright 2025 Datadog, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
4 changes: 4 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Datadog datadog-serverless-compat-js
Copyright 2025 Datadog, Inc.

This product includes software developed at Datadog (https://www.datadoghq.com/).
56 changes: 54 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,54 @@
# datadog-serverless-compat-py
Datadog Serverless Compatibility Layer for Python
# Datadog Serverless Compatibility Layer for Python

Datadog library for Python to enable tracing and custom metric submission from Azure Functions and Google Cloud Run Functions (1st gen).

## Installation

1. Install the Datadog Serverless Compatibility Layer.
```
pip install datadog-serverless-compat
```

2. Install the Datadog Tracing Library following the official documentation for [Tracing Python Applications](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/python).

3. Add the Datadog Serverless Compatibility Layer and the Datadog Tracer in code.

```
from datadog_serverless_compat import start
from ddtrace import tracer, patch_all

start()
patch_all()
```

## Configuration

1. Set Datadog environment variables
- `DD_API_KEY` = `<YOUR API KEY>`
- `DD_SITE` = `datadoghq.com`
- `DD_ENV` = `<ENVIRONMENT`
- `DD_SERVICE` = `<SERVICE NAME>`
- `DD_VERSION` = `<VERSION>`
- `DD_TRACE_STATS_COMPUTATION_ENABLED` = `true`

The default Datadog site is **datadoghq.com**. To use a different site, set the `DD_SITE` environment variable to the desired destination site. See [Getting Started with Datadog Sites](https://docs.datadoghq.com/getting_started/site/) for the available site values.

The `DD_SERVICE`, `DD_ENV`, and `DD_VERSION` settings are configured from environment variables in Azure and are used to tie telemetry together in Datadog as tags. Read more about [Datadog Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging).

[Trace Metrics](https://docs.datadoghq.com/tracing/metrics/metrics_namespace/) are enabled by default but can be disabled with the `DD_TRACE_STATS_COMPUTATION_ENABLED` environment variable.

Enable debug logs for the Datadog Serverless Compatibility Layer with the `DD_LOG_LEVEL` environment variable:

```
DD_LOG_LEVEL=debug
```

Alternatively disable logs for the Datadog Serverless Compatibility Layer with the `DD_LOG_LEVEL` environment variable:

```
DD_LOG_LEVEL=off
```

2. For additional tracing configuration options, see the [official documentation for Datadog trace client](https://datadoghq.dev/dd-trace-js/).

3. If installing to Azure Functions, install the [Datadog Azure Integration](https://docs.datadoghq.com/integrations/azure/#setup) and set tags on your Azure Functions to further extend unified service tagging. This allows for Azure Function metrics and other Azure metrics to be correlated with traces.
5 changes: 5 additions & 0 deletions datadog_serverless_compat/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
from datadog_serverless_compat.logger import initialize_logging

initialize_logging(__name__)

from datadog_serverless_compat.main import start # noqa: E402 F401
29 changes: 29 additions & 0 deletions datadog_serverless_compat/logger.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import logging
import os

try:
# Added in version 3.11
level_mapping = logging.getLevelNamesMapping()
except AttributeError:
level_mapping = {name: num for num, name in logging._levelToName.items()}

# https://docs.datadoghq.com/agent/troubleshooting/debug_mode/?tab=agentv6v7#agent-log-level
level_mapping.update(
{
"TRACE": 5,
"WARN": logging.WARNING,
"OFF": 100,
}
)


def initialize_logging(name):
logger = logging.getLogger(name)
str_level = (os.environ.get("DD_LOG_LEVEL") or "INFO").upper()

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: os.environ.get("DD_LOG_LEVEL", "INFO") usually? unless there's something more subtle going on with that or statement?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behavior is slightly different when DD_LOG_LEVEL is set to an empty string versus not being set (ie. None).

  • os.environ.get("DD_LOG_LEVEL") or "INFO" returns INFO when DD_LOG_LEVEL is an empty string
  • os.environ.get("DD_LOG_LEVEL", "INFO") returns an empty string when DD_LOG_LEVEL is an empty string

On further review I prefer your suggestion because it does result in the warning to show that an empty string is an invalid log level.

WARNING  tests.test_logger:logger.py:25 Invalid log level:  Defaulting to INFO

I also added a test case to cover when DD_LOG_LEVEL is not set.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the test runner workflow - I'm planning on working on that in a separate PR in the interest of getting this initial package release completed.

level = level_mapping.get(str_level)

if level is None:
logger.setLevel(logging.INFO)
logger.warning("Invalid log level: %s Defaulting to INFO", str_level)
else:
logger.setLevel(level)
96 changes: 96 additions & 0 deletions datadog_serverless_compat/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
from enum import Enum
import logging
import os
from subprocess import Popen
import sys


logger = logging.getLogger(__name__)


class CloudEnvironment(Enum):
AZURE_FUNCTION = "Azure Function"
GOOGLE_CLOUD_RUN_FUNCTION_1ST_GEN = "Google Cloud Run Function 1st gen"
GOOGLE_CLOUD_RUN_FUNCTION_2ND_GEN = "Google Cloud Run Function 2nd gen"
UNKNOWN = "Unknown"


def get_environment():
if (
os.environ.get("FUNCTIONS_EXTENSION_VERSION") is not None
and os.environ.get("FUNCTIONS_WORKER_RUNTIME") is not None
):
return CloudEnvironment.AZURE_FUNCTION

if (
os.environ.get("FUNCTION_NAME") is not None
and os.environ.get("GCP_PROJECT") is not None
):
return CloudEnvironment.GOOGLE_CLOUD_RUN_FUNCTION_1ST_GEN

if (
os.environ.get("K_SERVICE") is not None
and os.environ.get("FUNCTION_TARGET") is not None
):
return CloudEnvironment.GOOGLE_CLOUD_RUN_FUNCTION_2ND_GEN

return CloudEnvironment.UNKNOWN


def get_binary_path():
# Use user defined path if provided
binary_path = os.getenv("DD_SERVERLESS_COMPAT_PATH")
if binary_path is not None:
return binary_path

binary_path_os_folder = os.path.join(
os.path.dirname(__file__),
"bin/windows-amd64" if sys.platform == "win32" else "bin/linux-amd64",
)
binary_extension = ".exe" if sys.platform == "win32" else ""
binary_path = os.path.join(
binary_path_os_folder, f"datadog-serverless-compat{binary_extension}"
)

return binary_path


def start():
environment = get_environment()
logger.debug(f"Environment detected: {environment}")

if environment == CloudEnvironment.UNKNOWN:
logger.error(
f"{environment} environment detected, will not start the Datadog Serverless Compatibility Layer"
)
return

logger.debug(f"Platform detected: {sys.platform}")

if sys.platform not in {"win32", "linux"}:
logger.error(
(
f"Platform {sys.platform} detected, the Datadog Serverless Compatibility Layer is only supported",
" on Windows and Linux",
)
)
return

binary_path = get_binary_path()
logger.debug(f"Spawning process from binary at path {binary_path}")

if not os.path.exists(binary_path):
logger.error(
f"Serverless Compatibility Layer did not start, could not find binary at path {binary_path}"
)
return

try:
logger.debug(
f"Trying to spawn the Serverless Compatibility Layer at path: {binary_path}"
)
Popen(binary_path)
except Exception as e:
logger.error(
f"An unexpected error occurred while spawning Serverless Compatibility Layer process: {repr(e)}"
)
7 changes: 7 additions & 0 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[tool.poetry]
name = "datadog-serverless-compat"
version = "0.1.0"
description = "Datadog Serverless Compatibility Layer for Python"
authors = ["Datadog, Inc. <[email protected]>"]
license = "Apache-2.0"
readme = "README.md"
repository = "https://github.com/DataDog/datadog-serverless-compat-py"
keywords = [
"datadog",
"azure",
"google",
"functions"
]
include = [
{ path = "datadog_serverless_compat/bin/**/*", format = ["sdist", "wheel"] }
]
classifiers = [
"Development Status :: 4 - Beta",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]

[tool.poetry.dependencies]
python = "^3.9"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"