Bug: sys.stderr is used as default logging stream

[bram-tv]

Expected Behaviour

The API documentation of Logger() contains:

stream : sys.stdout, optional
   valid output for a logging stream, by default sys.stdout

This is no longer the case.

Current Behaviour

The sys.stderr stream is used as default logging stream.

Code snippet

from aws_lambda_powertools import Logger

logger = Logger()
logger.info("Example log message")

Possible Solution

No response

Steps to Reproduce

Run the above code but redirect stdout to /dev/null:

$ python3 example.py >/dev/null
{"level":"INFO","location":"<module>:4","message":"Example log message","timestamp":"2023-07-10 16:03:20,348+0000","service":"service_undefined"}
$

Message still shows up, i.e. it's not being logged to stdout.

Run the above code but redirect stderr to /dev/null

$ python3 example.py 2>/dev/null
$ 

Another way to see it is by using strace:

$ strace -s 65536 -e write python3 example.py
write(2, "{\"level\":\"INFO\",\"location\":\"<module>:4\",\"message\":\"Example log message\",\"timestamp\":\"2023-07-10 16:05:46,196+0000\",\"service\":\"service_undefined\"}\n", 146{"level":"INFO","location":"<module>:4","message":"Example log message","timestamp":"2023-07-10 16:05:46,196+0000","service":"service_undefined"}
) = 146
+++ exited with 0 +++
$

It's writing to stderr..

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.10

Packaging format used

PyPi

Debugging logs

This issue was introduced in commit 0dc5e1b.
More specific:
old code:

        self._handler = logging.StreamHandler(stream) if stream is not None else logging.StreamHandler(sys.stdout)

new code:

        self.logger_handler = logger_handler or logging.StreamHandler(stream)

Extra context: by default stream is None and using logging.StreamHandler(None) by default uses stderr.

https://docs.python.org/3.10/library/logging.handlers.html

Returns a new instance of the StreamHandler class. If stream is specified, the instance will use it for logging output; otherwise, sys.stderr will be used.

[heitorlessa]

Thank you @bram-tv for reporting it - looking into it. Out of curiosity, how does that affect you when running on AWS Lambda runtime? Or are you running outside of Lambda?

Thank you

[heitorlessa]

Confirmed, working on a fix soon

[heitorlessa]

Fix is up - kept stream as a private property lemme know if you have an use case that warrant otherwise: #2736

took slightly longer than expected due to a bug in pytest when asserting stderr

[bram-tv]

Out of curiosity, how does that affect you when running on AWS Lambda runtime? Or are you running outside of Lambda?

It affected me during development of a Lambda function, which I suppose is technically outside of Lambda.

Concrete: when running the code using serverless in combination with :

• the serverless-offline plugin: the log message would not show/not always show (TLDR: serverless-offline normally logs both stdout and stderr but it looks for a special string on stdout and if that appears it stop processing stderr. In my case the stdout was processed first which caused the stderr output not to be logged at all)
• serverless invoke local --function ...: the log messages would be out of order with print statements (because one went to stderr and the other to stdout); [the stdout output was during debugging of why the log messages weren't showing; normally there are no prints and everything happens via the logger]

[heitorlessa]

Gotcha, that makes sense. It'll be available in the next release (either this Friday or the next Friday).

Thanks for clarifying and for reporting it one more time :)

[github-actions]

This is now released under 2.20.0 version!

[willbengtson]

This seems like a breaking change for folks that currently use the library and expect certain logs to go to stderr. I see you still allow a stream to be passed in, but couldn't you have done stream or sys.stderr to keep existing functionality, but provide the ability to have all system logs and diagnostic messages go to stdout. Now you mix output and system messages in the same stream to be parsed by default.