Skip to content

[aws-lambda] POC - Request for Comments - Add support to carrier pre-processing in aws-lambda instrumentation #9300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

rapphil
Copy link
Contributor

@rapphil rapphil commented Aug 24, 2023

This PR is a POC for the proposal FaaS: Change requirements regarding handling of AWS Lambda-provided SpanContext

In this PR we:

  • Change the default behaviour of lambda to always try to include AWS X-Ray environment traces keys in the carrier that will be offered to the extract operation of the global propagators.
    * Adds a new option otel.instrumentation.aws-lambda.link-xray-traces (also as env var OTEL_INSTRUMENTATION_AWS_LAMBDA_LINK_XRAY_TRACES) to allow users to include X-Ray traces originated from the lambda environment as span links.

p.s.: there is no way to configure the global context propagators to include the X-Ray propagator to fully validate this change.

@laurit
Copy link
Contributor

laurit commented Aug 25, 2023

@tylerbenson could you help reviewing this

@rapphil rapphil changed the title Add support to carrier pre-processing in aws-lambda instrumentation [aws-lambda] POC - Request for Comments =Add support to carrier pre-processing in aws-lambda instrumentation Aug 25, 2023
@rapphil rapphil changed the title [aws-lambda] POC - Request for Comments =Add support to carrier pre-processing in aws-lambda instrumentation [aws-lambda] POC - Request for Comments - Add support to carrier pre-processing in aws-lambda instrumentation Aug 25, 2023
@tylerbenson
Copy link
Member

I'm trying to get clarification/confirmation from the TC. Once I get that, I'll review.

@bryan-aguilar
Copy link

I'm trying to get clarification/confirmation from the TC. Once I get that, I'll review.

Can you provide more information on what clarification/confirmation you are seeking?

@jsuereth
Copy link
Contributor

So there's a few fundamental decisions that were made in OTEL:

  1. Message passing tracing will use links as default - see approved OTEP 220.
  2. The TC weighed in on AWS lambda discussions from FAAS WG here.

Given these two, there's one central problem I see with this PR:

We should be preparing users for the new default parenting of spans in message passing scenarios to be links, not spans. We need to be working on a transition period and configuraiton to help users perserve the behavior they have today, but also let them know the default will be changing in OTEL (OTEP 220 guarantees this).

Given we are talking about breaking changes for users, I'd rather not churn how the instrumentation works any more than is strictly necessary. This PR does not move in the direction of OTEP 220, and I'm concerned the new environment variable that's added has the wrong default for the future of OTEL.

Signed-off-by: Raphael Silva <[email protected]>
Signed-off-by: Raphael Silva <[email protected]>
Signed-off-by: Raphael Silva <[email protected]>
Signed-off-by: Raphael Silva <[email protected]>
@rapphil rapphil force-pushed the rapphil-poc-semantic-conventions-277 branch from 7e12eda to 833392c Compare August 30, 2023 05:33
Signed-off-by: Raphael Silva <[email protected]>
@rapphil rapphil requested a review from trask August 30, 2023 22:24
Comment on lines 23 to 33
return new AwsLambdaFunctionInstrumenter(
openTelemetry,
InstrumenterBuilder<AwsLambdaRequest, Object> otelInstrumenterBuilder =
Instrumenter.builder(
openTelemetry,
"io.opentelemetry.aws-lambda-events-2.2",
AwsLambdaEventsInstrumenterFactory::spanName)
.addAttributesExtractor(new AwsLambdaFunctionAttributesExtractor())
.addAttributesExtractor(new ApiGatewayProxyAttributesExtractor())
.buildInstrumenter(SpanKindExtractor.alwaysServer()));
.addAttributesExtractor(new ApiGatewayProxyAttributesExtractor());

return new AwsLambdaFunctionInstrumenter(
openTelemetry, otelInstrumenterBuilder.buildInstrumenter(SpanKindExtractor.alwaysServer()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think can revert these changes now (just to clean up the diff and make the POC review easier for others)

Comment on lines 20 to 29
return new AwsLambdaFunctionInstrumenter(
openTelemetry,
InstrumenterBuilder<AwsLambdaRequest, Object> otelInstrumenterBuilder =
Instrumenter.builder(
openTelemetry,
"io.opentelemetry.aws-lambda-core-1.0",
AwsLambdaFunctionInstrumenterFactory::spanName)
.addSpanLinksExtractor(new AwsXrayEnvSpanLinksExtractor())
.addAttributesExtractor(new AwsLambdaFunctionAttributesExtractor())
.buildInstrumenter(SpanKindExtractor.alwaysServer()));
.addAttributesExtractor(new AwsLambdaFunctionAttributesExtractor());

return new AwsLambdaFunctionInstrumenter(
openTelemetry, otelInstrumenterBuilder.buildInstrumenter(SpanKindExtractor.alwaysServer()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Signed-off-by: Raphael Silva <[email protected]>
@rapphil rapphil requested a review from trask August 30, 2023 22:53

private static final String AWS_TRACE_ENV_VAR = "_X_AMZN_TRACE_ID";
private static final String AWS_TRACE_PROPERTY = "com.amazonaws.xray.traceHeader";
private static final String AWS_TRACE_PROPAGATOR_KEY = "x-amzn-trace-id";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my preference would be for this to be a separate key in the carrier, e.g. x-amzn-env-trace-id or x-amzn-vended-trace-id (instead of overwriting the AwsLambdaRequest header), so that the propagator chain can have access to all of the different propagated trace contexts

but this is a question that needs to be answered on the spec side

@jaydeluca
Copy link
Member

is this PR still relevant?

@tylerbenson
Copy link
Member

I don't think so.

@trask trask closed this Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants