filter out logs based on ActivityTraceFlags

[cataggar]

I want to prevent logs from being sent to Azure.Monitor. Based on the filtering processor example, I created:

open System.Diagnostics
open OpenTelemetry
open OpenTelemetry.Logs

/// Remove Heartbeat from Host.Results
type HeartbeatLogFilter() =
    inherit BaseProcessor<LogRecord>()
    override _.OnEnd log =
        if log.CategoryName = "Host.Results" then
            let fullName =
                log.StateValues |> Seq.tryPick(fun kv ->
                    if kv.Key = "FullName" then
                        Some kv.Value
                    else
                        None
                )
            if fullName = Some "Heartbeat" then
                log.TraceFlags <- log.TraceFlags &&& ~~~ ActivityTraceFlags.Recorded

It removes ActivityTraceFlags.Recorded from the LogRecord.TraceFlags.

Expected behavior

I was expecting the log records with ActivityTraceFlags.Recorded removed to not be exported to Azure Monitor.

Actual behavior

The log records still end up in Azure Monitor.

[cijothomas]

TraceFlags are part of LogRecord, but they are not used to make any decision about export vs not export for logs. (They are used for traces)

As of today, there is not easy mechanism to filter logs based on the log content itself. You can write own FilteringProcessor wrapping the exporting processor, but that'll only work if the Exporter being target has public ctor.

https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/logs/extending-the-sdk

[cijothomas]

Tagging @CodeBlanch to see if we can offer a doc/feature to filter logs easily.

[cataggar]

As of today, there is not easy mechanism to filter logs based on the log content itself. You can write own FilteringProcessor wrapping the exporting processor, but that'll only work if the Exporter being target has public ctor.

@cijothomas, I see a public constructor, but the class is internal.

    internal class AzureMonitorLogExporter : BaseExporter<LogRecord>
    {
        private readonly ITransmitter _transmitter;
        private readonly string _instrumentationKey;
        private readonly ResourceParser _resourceParser;
        private readonly AzureMonitorPersistentStorage _persistentStorage;

        public AzureMonitorLogExporter(AzureMonitorExporterOptions options, TokenCredential credential = null) : this(new AzureMonitorTransmitter(options, credential))
        {
        }

https://github.com/Azure/azure-sdk-for-net/blob/c3fc45c5d59afcadca84e49e291d33ff439751a0/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/AzureMonitorLogExporter.cs#L15-L24

I reopened Azure/azure-sdk-for-net#32276 (comment).

[cataggar]

@cijothomas, the possibility of making AzureMonitorLogExporter public was rejected. There is still no way to filter logs easily that I'm aware of. Our logs of full of "Heartbeat"s that we do not need to be logging and I'd like to filter out. Azure/azure-functions-host#4742 (comment)

[cijothomas]

Are you not able to filter the logs using https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/logs/customizing-the-sdk#log-filtering ?

[cataggar]

@cijothomas, unfortunately no. That is limited to filtering on log.CategoryName. The logic is shown in the issue description. For the category Host.Results, I want to filter out any logs that have a FullName of Heartbeat.

[cijothomas]

@cijothomas, unfortunately no. That is limited to filtering on log.CategoryName. The logic is shown in the issue description. For the category Host.Results, I want to filter out any logs that have a FullName of Heartbeat.

Could you ask the log producer to use a different category for Heartbeats?

[cataggar]

That is the category that Azure Functions to log all requests. We need the other requests to be logged, just not certain ones like Heartbeat. I'm pursuing that angle here, but no traction on it yet. Back to this angle, why can't we filter LogRecord in an an exporter? Why limit it to its CategoryName?

[cijothomas]

why can't we filter LogRecord in an an exporter

No reason. Its upto individual exporters to offer such capabilities. From OTEL, there should be some solution to filter logs - this just do not exist today, unless the Exporter itself has public ctors.