Is it possible to use Powertools outside of lambda?

[demiurg]

I have asked this on Discord, but I should mirror my question here:

I really like Powertools for its envelopes and metrics, and it works great in lambda. However, I have some Fargate ECS tasks, and I'd like to re-use same libraries/code between lambdas and ECS, but the Powertools page mentions that some functionality "automatically" gets disabled without lambda context. Is that correct? Is it possible to use some functionality outside of lambda?

[heitorlessa]

Thank you for creating it @demiurg ! I'm gonna copy/paste what I shared on Discord earlier, so I can edit them later as bandwidth permits.

---

A brief response now is that most of the utilities can work outside Lambda. These are the exceptions:

• Tracer. It relies on a Root Segment being created (Lambda does this transparently)
• Metrics. It relies on Lambda native integration with CloudWatch Logs EMF. Because we simply print to the standard output, you might be able to do this with CloudWatch Logs Agent configured with EMF[1] (haven't tried myself). Fargate and ECS have built-in integration with EMF so this should work out of the box too (if logs are sent to CloudWatch).
• Batch. It relies on Lambda SQS/DynamoDB Stream/Kinesis Event Source. You'd need to poll yourself. For Partial failure mechanisms, you can use the output to decide when to delete successful messages.
• Idempotency. Don't use idempotent decorator as it's meant for Lambda handler. Use idempotent_function instead. You can also pass a custom function_name when configuring the Persistency layer to not end up with test-func .

Everything else works out of the box. One caution is that we optimize certain parts of the code for Lambda short-lived/single-concurrent request behaviour - there are methods you can all like clear_cache here and there (Parameters, Metrics, Logger state).

[1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html

[ohadbenita]

Would it be possible to provide a way for the current metrics util to output its EMF JSONs to another output? say the TCP port of the Cloudwatch agent that can be installed on an ECS task for instance in order to support EMF metric reporting?

[leandrodamascena]

Would it be possible to provide a way for the current metrics util to output its EMF JSONs to another output? say the TCP port of the Cloudwatch agent that can be installed on an ECS task for instance in order to support EMF metric reporting?

Hello @ohadbenita. The short answer to this question is no, so far Powertools does not support output other than the default stdout. But we are in the process of supporting external providers and consequently other outputs. Below are some other information about this issue.

Support for other providers

• We have an open RFC where we are discussing development experience and customer needs. If you're interested, we'd love to hear from you.

ECS + CloudWatch

• As I understand your question, you want to configure Cloudwatch manually in your definition task (using a sidecar) in ECS, right? So, I don't see the need to change anything in Powertools for this, as you can set the Cloudwatch agent port in containerDefinitions and your app container will automatically send logs/emf to this destination. The same goes if you run the Cloudwatch agent + app in any other location (even on-premises) and want to send EMF metrics. Please correct me if I'm wrong, but this is what I understand.
• ECS has native integration with Cloudwatch for logs, both in EC2 mode and in Fargate mode application logs can be written directly to Cloudwatch (EMF logs too) without any sidecar container. Does this not work for you?
• In the next release of Powertools (we'll release this week), we will include a manual flush method, which will bring a better experience using the Metrics utility outside of Lambda.

If I missed any information or misunderstood something, please let me know and I can create a hello world app to show how it can be used.

Thank you

[ohadbenita]

Hi @leandrodamascena, your response has proven to be super-useful as I just found out that the awslogs logs driver for ECS now support parsing EMF JSONs into Cloudwatch metrics. I've just removed the Cloudwatch agent from my task and was able, while only printing of the EMF JSON to report metrics to Cloudwatch.

Thanks again.