Manually generation of TRACEPARENT

[sfc-gh-kbregula]

Hello,

I was working on integrating Docker buildx with OpenTelemetry on Jenkins and unfortunately I couldn't use otel exec because I needed to start a new span and finish it once it had built all the Docker images.

Here is my script that create a new builder:

  docker buildx create \
      --use \
      --bootstrap \
      --name "${BUILDER_NAME}" \
      --driver docker-container \
      --config ./buildkitd.toml \
      --driver-opt "network=host" \
      --driver-opt "env.TRACEPARENT=${TRACEPARENT:-}" \
      --driver-opt "env.OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:${OTEL_GRPC_PORT:-4317}" \
      --driver-opt "network=host" \
      --driver-opt "default-load=true"

I had to generate a new TRACEPARENT manually, as described in this guide: Caller explicitly tracing.

private def startGlobalSpan() {
    // First, get a trace and span ID. We need to get one now so we can propagate it to the child
    // Get trace ID from TRACEPARENT, if present
    spanTid = sh(script: 'echo ${TRACEPARENT:-} | cut -d- -f2', returnStdout: true).trim()
    spanTid = spanTid ?: sh(script: "LC_ALL=C tr -dc 'a-f0-9' < /dev/urandom | head -c 32", returnStdout: true).trim()
    // Always generate a new span ID
    spanSid = sh(script: "LC_ALL=C tr -dc 'a-f0-9' < /dev/urandom | head -c 16", returnStdout: true).trim()
    def traceParent = "00-${spanTid}-${spanSid}-01"

    // Get start time before executing the code
    spanStart = sh(script: 'python3 -c "import time; print(time.time())"', returnStdout: true).trim()
    env.TRACEPARENT = traceParent
}

It would be great if OTEL-cli supported a simpler way to create TRACEPARENT without the need to use sockets.

Best regards,
Kamil

[tobert]

Oh, neat idea. I suppose I could add a subcommand like otel-cli traceparent or something like that. That way you could replace your function with TRACEPARENT=$(otel-cli traceparent). Do I understand you correctly?

[foutoucour]

I am also interested.
I need to implement otel-cli in Jenkins. We experimented with https://plugins.jenkins.io/opentelemetry/ for several months, but the plugin doesn't handle our scale and crashes them. Our controllers build 2000+ jobs a day.
I would need something to create and close a traceparent.

For now, I do something like:

pipeline {
    agent any
    environment {
        OTEL_CLI_PRINT_TRACEPARENT = "true"
        OTEL_CLI_CARRIER_FILE = "${env.WORKSPACE}/otel-carrier.txt"
        OTEL_SERVICE_NAME = "jenkins"
        OTEL_CLI_VERBOSE = "true"
        OTEL_CLI_SOCKDIR = "${env.WORKSPACE}/otel-cli-socket"
    }

    options {
        timestamps()
    }
    stages {
        stage("setUp") {
            steps {
                script {
                    sh "touch ${OTEL_CLI_CARRIER_FILE}"
                    dir(env.OTEL_CLI_SOCKDIR) {}
                    sh "otel-cli span background --name 'otel-cli-example background span' --sockdir $OTEL_CLI_SOCKDIR &"
                }
            }
        }
        stage("install") {
            steps {
                sh "otel-cli exec --name 'install' make install"
            }
        }
    }
    post {
         always {
                sh "otel-cli span end --sockdir $OTEL_CLI_SOCKDIR"
         }
    }
}

@tobert Would you advise another approach? Would you see the improvement otel-cli tracesparent would get us? Or that's a good way?

My current challenge is about the background span I want to have for the entire job. Jenkins kills the shell (even with the &). I am still fishing for solutions that scale.
🥂

[foutoucour]

I also wonder what approach you would suggest for Windows. Does the otel-cli span background --name 'otel-cli-example background span' --sockdir $OTEL_CLI_SOCKDIR & work?

[tobert]

I also wonder what approach you would suggest for Windows. Does the otel-cli span background --name 'otel-cli-example background span' --sockdir $OTEL_CLI_SOCKDIR & work?

@foutoucour I don't think the background socket will work on Windows. I've assumed most Windows users would reach for Powershell to trace. I'll re-check that assumption if you have a use case in mind.

[tobert]

@tobert Would you advise another approach? Would you see the improvement otel-cli tracesparent would get us? Or that's a good way?

My current challenge is about the background span I want to have for the entire job. Jenkins kills the shell (even with the &). I am still fishing for solutions that scale. 🥂

For your usecase, it might make sense to generate your root span with otel-cli and propagate that trace/span id. Simply: otel-cli span --name "build ${BUILD_ID} trace root" --tp-carrier "${TRACEPARENT_FILE}" and then have your CI system include that file in the build assets.

The unsent traceparent feature would be for things like the OP is doing, where they want a trace to be rooted in some process that can't propagate back to its parent easily. Even in that case, I strongly recommend generating a span from the parent process to act as root to stay aligned with the assumptions built into the OpenTelemetry ecosystem.

That said, you can try the synthetic id approach using pure shell without any changes to otel-cli. Check out this post for some ideas of how: https://blog.howardjohn.info/posts/shell-tracing/ Please let me know if this ends up working out and I'll try to make it easier for folks.

[foutoucour]

Hey @tobert
Thank you for the replies.
I worked my case more with your tool and don't need anything more now.
I still cannot do the background span, as I am on Jenkins pipelines (no background task) and need to support Windows.
So, I worked around the situation.
I create a sandwich of spans. I make an "empty" span representing the process's beginning. I use this empty span as an anchor to the actual processes. Then, I close the sandwich with a summary span. The summary span includes the duration of the process, the exit code, and such.

It is imperfect, but it allows us to have our traces on Jenkins pipelines, which is essential work.

Thank you for OTEL-CLI!