Sending traces to AWS X-Ray from containerized Lambda function using OpenTelemetry

This post explains how to send traces to AWS X-Ray from a containerized AWS Lambda function using OpenTelemetry. Since the standard ADOT Lambda layer cannot be used with container images, the solution involves building a custom OpenTelemetry Collector image, pushing it to Amazon ECR, and copying it into the Lambda function’s container to serve as a Lambda extension.

Introduction

Observability plays a crucial role in distributed applications, providing insights into performance bottlenecks, error rates, and overall application health. When deploying containerized AWS Lambda functions and instrumenting with OpenTelemetry SDK, sending traces to AWS X-Ray becomes challenging. This blog post presents a solution to this problem by building an OpenTelemetry Collector and pushing it to an Amazon ECRrepository, allowing you to copy it into your Lambda function container image.

The proposed solution leverages the AWS Distro for OpenTelemetry(ADOT), a secure and production-ready distribution of the OpenTelemetry project. With this approach, you can instrument your application code using the OpenTelemetry SDK, and the OpenTelemetry Collector running inside the Lambda function will collect and send traces to observability backends like AWS X-Ray. This enables you to monitor your containerized Lambda functions effectively, identifying performance issues and understanding the application’s overall health.

By following the steps outlined in this blog post, you can seamlessly integrate OpenTelemetry with your containerized AWS Lambda functions, unlocking the power of distributed tracing and observability. This solution empowers you to gain valuable insights into your distributed applications, ensuring optimal performance and a superior user experience.

Architecture Overview

In this section, we demonstrate the overall architecture for sending traces from a containerized Lambda function.

  1. When you upload an image file to S3 bucket, a Lambda function is invoked.
  2. The main handler function makes external API call against the external components (in this case Amazon Rekognition).
  3. The function is instrumented with OpenTelemetry SDK and it generates traces.
  4. The OTLP receiver inside OpenTelemetry Collector Lambda extension receives traces and the AWS X-Ray Tracing Exporter for OpenTelemetry Collector exports them to AWS X-Ray.

I use AWS X-Ray as the monitoring backend to visualize telemetry data of API requests to Amazon Rekognition. It is also possible to switch to other monitoring backends to visualize telemetry data of API requests or HTTP requests to other services.

System design for sending traces from a containerized Lambda function

When deploying Lambda functions as .zip file archives, you can send telemetry data using ADOT Lambda layer. When you use container tools like Docker for increased portability and deploy Lambda functions as container images, however, you cannot use ADOT Lambda layer and have to package your preferred runtime and all code dependencies into the container image.

In this blog you build an OpenTelemetry Collector AWS Lambda Extension layer and pushing it to an Amazon ECR repository, then copying it into your Lambda function. Creating separate container images (one for OpenTelemetry Collector and the other for a handler function) allows you to add them to multiple functions, and share them in a similar way as Lambda layers.

Walkthrough

Requirements

  • AWS CLI v2
  • Node.js
  • docker

Cloning the repository

The OpenTelemetry Collector is responsible for collecting and exporting trace data from the Lambda function to one or more backends, such as AWS X-Ray in this blog post. The OpenTelemetry Collector AWS Lambda Extension layer does not include the AWS X-Ray Tracing Exporter for OpenTelemetry Collector, so you need to build a custom Collector image with the necessary configuration and the AWS X-Ray Tracing Exporter. Clone the repository and follow the instructions in the README.md.

git clone https://github.com/aws-samples/opentelemetry-lambda-container.git
cd opentelemetry-lambda-container
git clone https://github.com/open-telemetry/opentelemetry-lambda.git -b layer-collector/0.4.0
cp -r opentelemetry-lambda/collector/ otel-collector-lambda-extension
rm -rf opentelemetry-lambda/

Building OpenTelemetry Collector

First, modify the OpenTelemetry Collector configuration to include the AWS X-Ray exporter, which allows the Collector to send trace data to the AWS X-Ray service.

  1. Add the awsxrayexporter dependency to otel-collector-lambda-extension/lambdacomponents/go.mod.
cd otel-collector-lambda-extension/lambdacomponents/
go mod edit --require=github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]
  1. Edit the otel-collector-lambda-extension/lambdacomponents/default.go file to import the awsxrayexporter and add it to the exporters list.
import (
    // import awsxrayexporter
    "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter"
)

func Components(extensionID string) (otelcol.Factories, error) {
    ...
    exporters, err := exporter.MakeFactoryMap(
        // add awsxrayexporter here
        awsxrayexporter.NewFactory(),
        
        loggingexporter.NewFactory(),
        otlpexporter.NewFactory(),
        otlphttpexporter.NewFactory(),
        prometheusremotewriteexporter.NewFactory(),
    )
    ...

    return factories, multierr.Combine(errs...)
}
  1. Add a Dockerfile in otel-collector-lambda-extension/Dockerfile to build the OpenTelemetry Collector.
FROM public.ecr.aws/docker/library/golang:1.20 as collector-builder
WORKDIR /src
COPY . .
RUN go mod tidy
RUN GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath -o collector .

FROM scratch
COPY --from=collector-builder /src/collector src/collector

Creating an ECR repository and pushing the collector image to the repository

After building the OpenTelemetry Collector image, you need to push it to an Amazon Elastic Container Registry (ECR) repository, which allows the Lambda function to use the Collector as a Lambda extension. The steps in this section demonstrate how to create an ECR repository, log in to the registry, and push the Collector image to the repository. You deploy AWS resources in us-east-1, but you can change the region by setting AWS_DEFAULT_REGION.

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_DEFAULT_REGION=us-east-1
cd opentelemetry-lambda-container/otel-collector-lambda-extension
aws ecr create-repository --repository-name lambda-extension/otel-collector
aws ecr get-login-password | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com
docker build -t lambda-extension/otel-collector .
docker tag lambda-extension/otel-collector:latest ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/lambda-extension/otel-collector:v1
docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/lambda-extension/otel-collector:v1

Copying the collector into a Lambda function

Replace and in lambda-function/Dockerfile.

FROM public.ecr.aws/docker/library/rust:1.76.0 as rust-builder

WORKDIR /rust/rust_app
COPY src/ /rust/rust_app/src/
COPY Cargo.toml /rust/rust_app/

RUN apt-get update
RUN apt-get install musl-tools -y
RUN rustup update && \
    rustup target add x86_64-unknown-linux-musl
RUN cargo build --release --target x86_64-unknown-linux-musl

FROM public.ecr.aws/lambda/provided:al2023

COPY --from=rust-builder /rust/rust_app/target/x86_64-unknown-linux-musl/release/bootstrap ${LAMBDA_RUNTIME_DIR}/bootstrap
COPY --from=<Your AWS account id>.dkr.ecr.<AWS Region>.amazonaws.com/lambda-extension/otel-collector:v1 /src/collector /opt/extensions/collector
COPY adot-config.yaml /opt/collector-config/config.yaml
CMD [ "lambda-handler" ]

The Dockerfile has three separate stages: one for building a binary, and another where the binary gets copied from the first stage into the next stage, and the last where the collector gets copied from the ECR repository into the function.

Deploying AWS resources using AWS CDK

Deploy the Lambda function instrumented with the OpenTelemetry SDK, using the OpenTelemetry Collector as a Lambda extension with the AWS CDK (Cloud Development Kit) . The Lambda function is triggered when an image being uploaded to an S3 bucket. The RekognitionSourceBucketName output provides the name of the S3 bucket, which you can use to upload images and invoke the Lambda function.

Run the following commands to deploy AWS resources.

cd opentelemetry-lambda-container/lambda-trace
npm install
cdk deploy

Invoking the Lambda function

Run the following command to upload an image to the S3 bucket.

aws s3api put-object --bucket <RekognitionSourceBucketName> --key image.png --body <Some image file>

You can see detect-label span generated here.

detect-label span has label-num metadata inserted here.

Conclusion

In this post, I explained how to build a custom OpenTelemetry Collector image with the necessary configuration and the AWS X-Ray Tracing Exporter. You can copy the collector into a Lambda function, allowing it to send telemetry data to OpenTelemetry backends including AWS X-Ray

aws  observability  lambda