Skip to content

Is OTLP file exporter required for SDKs? #3817

Open
Open
Is OTLP file exporter required for SDKs?#3817
Labels
spec:protocolRelated to the specification/protocol directorytriage:accepted:ready-with-sponsorReady to be implemented and has a specification sponsor assigned

Description

@jack-berg
jack-berg
opened on Jan 11, 2024

The OTLP File exporter seems like it could be an implementation requirement for SDKs, but on closer inspection:

  • It does not appear as a known value for OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER
  • It only appears in the spec compliance matrix under logs. Its not implemented by any SDKs so this may have been a mistake.

On one hand, it would be nice to be able to configure SDKs to export to files. On the other hand, supporting file export requires a variety of additional un-specified configuration options including the the name of the file, whether a file rotation strategy is used, and more. Additionally, supporting this in the SDK when the functionality exists in the collector duplicates effort.

Activity

  1. jack-berg
    added
    spec:protocolRelated to the specification/protocol directory
    on Jan 11, 2024
  2. github-actions
    assigned
    bogdandrutu
    on Jan 11, 2024
  3. reyang
    added
    triaged-needmoreinfo
    on Jan 24, 2024
  4. austinlparker
    added
    triage:deciding:community-feedbackOpen to community discussion. If the community can provide sufficient reasoning, it may be accepted
    and removed
    triaged-needmoreinfo
    on Apr 30, 2024
  5. austinlparker
    unassigned
    bogdandrutu
    on Apr 30, 2024

  6. austinlparker commented on Apr 30, 2024

    @austinlparker
    austinlparker
    on Apr 30, 2024
    Member

    It would be good if we could get more specific feedback about the intended usage, use cases, and related work on this.

  7. zeitlinger commented on Jul 19, 2024

    @zeitlinger
    zeitlinger
    on Jul 19, 2024
    Member

    The intended usage is to be to counterpart of the otlpjsonfilereceiver component - which is currently under development:

    open-telemetry/opentelemetry-collector-contrib#33846

  8. zeitlinger commented on Jul 19, 2024

    @zeitlinger
    zeitlinger
    on Jul 19, 2024
    Member

    Regarding the spec, I'd propose

    OTEL_LOGS_EXPORTER=stdout

    and

    OTEL_LOGS_EXPORTER=file
OTEL_LOGS_EXPORTER_FILE=/path/to/file

    wdyt?

  9. austinlparker commented on Jul 23, 2024

    @austinlparker
    austinlparker
    on Jul 23, 2024
    Member

    Which SIGs would this impact? All of them (outside of collector?)

  10. zeitlinger commented on Jul 24, 2024

    @zeitlinger
    zeitlinger
    on Jul 24, 2024
    Member

    Which SIGs would this impact? All of them (outside of collector?)

    yes, all language sigs

  11. pellared commented on Jul 24, 2024

    @pellared
    pellared
    on Jul 24, 2024
    Member

    @zeitlinger, I would like to have otlp in the name.

    I propose to

    Add otlpstdout, otlpstderr, otlpfile to known values of OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER`.

    Add OTLP_FILE_EXPORTER_PATH, OTLP_FILE_TRACES_EXPORTER_PATH, OTLP_FILE_METRICS_EXPORTER_PATH, OTLP_FILE_LOGS_EXPORTER_PATH which works only for otlpfile (based on https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md#configuration-options but names have a little different convention to have the names shorter). The per-signals env vars take precedence over OTLP_FILE_EXPORTER_PATH.

  12. zeitlinger commented on Jul 24, 2024

    @zeitlinger
    zeitlinger
    on Jul 24, 2024
    Member

    good point, so the question is

    • OTLP_FILE_METRICS_EXPORTER_PATH vs. OTEL_EXPORTER_OTLP_METRICS_FILE (or _PATH)
    • OTLP_FILE_EXPORTER_PATH vs. OTEL_EXPORTER_OTLP_FILE (or _PATH)

    same number of chars, so I'd prefer the second, more consistent option

  13. pellared commented on Sep 19, 2024

    @pellared
    pellared
    on Sep 19, 2024
    Member

    @open-telemetry/technical-committee, can you please "re-triage" the issue given the community feedback we got #4183 (comment) + comments below?

    I can be assigned as a sponsor as I am already trying to assist @zeitlinger.

  14. github-actions
    added
    triage:followupNeeds follow up during triage
    on Sep 19, 2024

  15. 3 remaining items

  16. trask
    added
    triage:accepted:ready-with-sponsorReady to be implemented and has a specification sponsor assigned
    on Sep 24, 2024
  17. github-project-automation
    added this to Specification + OTEPson Sep 24, 2024

  18. jsuereth commented on Nov 5, 2024

    @jsuereth
    jsuereth
    on Nov 5, 2024
    Contributor

    I realize I'm a bit late to this party (around file export). I just want to call out that I think writing JSON is a "simple" exporter, but possibly not the most efficient way to do file-based communication between Process -> Collector.

    I put together a quick (trace-only) prototype e.g. of a more sophisticated file format (designed as an IPC mechanism): https://github.com/jsuereth/otlp-mmap/

    I'm not sure this is the right issue for this discussion, but I do think we should create a strategy for this kind of architecture:

    flowchart LR
SDK-->|writes| file
Collector-->|reads| file
Collector-->|otlp| Backend
    Loading

    I do think a naive "json" export is fine, but I think we should also be considering something that allows us to "get data off process" as fast as possible, e.g. for dealing with crash scenarios.

    Posting here to better understand if this is the intended use case of the file exporter and if we think we should be designing a more efficient / robust solution that would eventually be the "default recommended" path for writing to files.

  19. zeitlinger commented on Nov 6, 2024

    @zeitlinger
    zeitlinger
    on Nov 6, 2024
    Member

    I think it's great to have more eyes on the topic - and I agree that the current iteration I'm working on (json over stdout) can probably be improved - but I still think it's a good first step as it unlocks many use cases and is easy to use with k8s.

  20. jsuereth
    added a commit that references this issue on Nov 26, 2024

    Add OTLP stdout exporter (#4183)

    79380c6

  21. github-actions commented on Aug 3, 2026

    @github-actions
    github-actionsBot
    on Aug 3, 2026 – with GitHub Actions

    This issue was marked stale due to lack of activity. Please leave a comment if you would like to keep this issue open.

  22. github-actions
    added
    Stale
    on Aug 3, 2026

  23. github-actions commented on Aug 21, 2026

    @github-actions
    github-actionsBot
    on Aug 21, 2026 – with GitHub Actions

    Closed due to inactivity.

    We appreciate your input, and if this issue is still relevant,
    please reopen it, or comment "/reopen" and we'll do it for you.

    Thank you for your contribution!

  24. github-actions
    closed this as not plannedon Aug 21, 2026
  25. github-project-automation
    moved this from Spec - Accepted to Spec - Closed in Specification + OTEPson Aug 21, 2026

  26. edmorley commented on Aug 21, 2026

    @edmorley
    edmorley
    on Aug 21, 2026

    /reopen

  27. edmorley commented on Aug 21, 2026

    @edmorley
    edmorley
    on Aug 21, 2026

    The reopen didn't work, since the bot only allows the original author or project maintainers to reopen:
    https://github.com/open-telemetry/opentelemetry-specification/actions/runs/32457812703/workflow#L11-L12

  28. pellared
    reopened this on Aug 21, 2026
  29. pellared
    removed
    Stale
    on Aug 21, 2026
  30. pellared
    removed their assignment
    on Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    spec:protocolRelated to the specification/protocol directorytriage:accepted:ready-with-sponsorReady to be implemented and has a specification sponsor assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions