v2 Ideas - Risks / Mitigations for Agentic AI systems

[ColinEberhardt]

For v2 of the framework we want to expand the scope to incorporate agentic AI systems. This issue is a place to collect ideas.

---

Looks like a really good paper with some genuine insight and patterns people can employ:

https://simonwillison.net/2025/Jun/13/prompt-injection-design-patterns/

[alvin-c-shih]

It is a good paper, but focused on agents:

• The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment.

Would be good to spend more time on it for v2.

[ColinEberhardt]

Good piont @alvin-c-shih - I've re-named this issue to focus on v2/agentic ideas

[ColinEberhardt]

The lethal trifecta for AI agents: private data, untrusted content, and external communication - lots of great points about security issues with agentic AI / MCP

[ColinEberhardt]

https://www.anthropic.com/engineering/building-effective-agents

[MKQuantum]

AI Governance will need to be augmented a bit for models like AI Agentic Meshes and AI-Native Systems.

In fact, it may make Governance and Audit/Compliance easier if these models and architectures are extending primitives from Kubernetes and Service Mesh. The "ideal state" of these AI Agent Systems should be able to be described declaratively and there should be capabilities such as continuous reconciliation/audit/inference and etc.

Count me in to help with this in the next version of the framework!