Python Functions overwrites Custom Dimensions in logs sent to App Insights

[priyaananthasankar]

Steps to reproduce

• Create an Azure Python Function that is connected to Application Insights.

• Create a log message (via logger.Info) and set a customDimension (see opencensus example here: https://github.com/census-instrumentation/opencensus-python/blob/master/contrib/opencensus-ext-azure/examples/logs/properties.py)

• Look at the message in App Insights and notice that customDimensions is overwritten by Azure.

[NazariyIvasyuk-Hayfin]

Guys, is there any progress with this issue? I have run into this problem too. Everything works fine when I run the code from the notebook, but the same code doesn't work in an azure function. Logs are populated, no errors/warnings are raised, but customDimensions are completely overwritten.

[christoffer-hk]

I am also encountering this problem and would love a fix for this.

[Hazhzeng]

We're currently under active development on opencensus integration to Python Worker.
Meanwhile, please try using the following snippet code to manually integrate opencensus custom dimension into your Python function app.

import json
import logging
import requests

import azure.functions as func

from opencensus.ext.azure.trace_exporter import AzureExporter
from opencensus.ext.azure.log_exporter import AzureLogHandler
from opencensus.trace import config_integration
from opencensus.trace.samplers import ProbabilitySampler
from opencensus.trace.tracer import Tracer
from opencensus.trace import execution_context
from opencensus.trace.propagation.trace_context_http_header_format import TraceContextPropagator

config_integration.trace_integrations(['logging'])

# instrucment_key = os.getenv('APPINSIGHTS_INSTRUMENTATIONKEY')
exporter = AzureExporter(connection_string='InstrumentationKey=')
handler = AzureLogHandler(connection_string='InstrumentationKey=')
logger = logging.getLogger(__name__)
logger.addHandler(handler)

def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    span_context = TraceContextPropagator().from_headers({
        "traceparent": context.trace_context.Traceparent,
        "tracestate": context.trace_context.Tracestate
    })
    tracer = Tracer(
        span_context=span_context,
        exporter=exporter,
        sampler=ProbabilitySampler(1.0)
    )
    execution_context.set_opencensus_tracer(tracer)

    logger.warning('Before the span')
    with tracer.span("custom_dimensions_span"):
        properties = {'custom_dimensions': {'ABCD': 'EFG'}}
        logger.info('inside the span', extra=properties)
    logger.warning('After the span')

    return json.dumps({
        'method': req.method,
        'ctx_func_name': context.function_name,
        'ctx_func_dir': context.function_directory,
        'ctx_invocation_id': context.invocation_id,
        'ctx_trace_context_Traceparent': context.trace_context.Traceparent,
        'ctx_trace_context_Tracestate': context.trace_context.Tracestate,
    })

[Anmol007]

Hi @lzchen @Hazhzeng, I tried the above example with ServiceBusTrigger (keeping all the same) but it leads to duplicate traces (pls refer to the image below).

When I remove the explicit addition of AzureLogHandler by removing logger.addHandler(handler)
(as pointed by @brettsam in his comment to avoid double logs) then the traces are not duplicated but custom_dimensions information is lost (pls refer to the image below)

How to avoid the log duplication here while retaining the custom_dimensions property set in the python script?

[Hazhzeng]

Hi @Anmol007,

This is due to a behavior where the root logger has a default handler sending logs to application insight.
So right before the line config_integration.trace_integrations(['logging']). You can clear out the default logging by inserting the following two lines:

root_logger = logging.getLogger()
root_logger.handlers = []

[Anmol007]

Hi @Anmol007,

This is due to a behavior where the root logger has a default handler sending logs to application insight.
So right before the line config_integration.trace_integrations(['logging']). You can clear out the default logging by inserting the following two lines:

root_logger = logging.getLogger()
root_logger.handlers = []

Hi @Hazhzeng, I tried the solution you proposed. However I still see double logs (only difference now is that there is no operationName, operationId and customDimensions.Category value changed from Function.ServiceBusTopicTrigger to
Host.Function.Console). Is this expected?
Note: I am able to reduce one duplication by moving the logger.addHandler(handler) inside main i.e making it local. Could you help me explain why this is leading to one less log entry and is it recommended this way?

[thatscotdatasci]

I was able to remove the duplicates by removing the handlers on the root logger as follows:

if (logger.hasHandlers()):
    logger.handlers.clear()

However, as shown in @Anmol007's screenshots, the AzureLogHandler traces have several different field values - operation_Id, operation_ParentId, etc. The root logger has an AsyncLoggingHandler from this library - why can this not be updated to accept custom dimensions? Should it not also be using the Opencensus Python SDK?

[pacorreia]

I stumbled upon same issue as @thatscotdatasci.

The AsyncLoggingHandler seems to be incomplete, and I could not find a way to override it yet.

will this require some contact with PG to speed up things?

[frankShih]

I'm facing a similar issue.

Is there any workaround yet?