fix(agent-mesh): report assessment coverage so unassessed controls are not read as met by impartshadow · Pull Request #4027 · microsoft/agent-governance-toolkit

impartshadow · 2026-09-18T14:29:45Z

Problem

ComplianceEngine.generate_report() derives controls_met as total - violated, so a report generated with no check_compliance() call at all reads controls_met=2, compliance_score=100.0 for SOC 2. The reporting model cannot distinguish "assessed and passed" from "never assessed", which propagates into DashboardAPI.get_compliance_report() and Annex IV Controls evaluated / Controls met lines.

Reproduced on 9165e498 with the script from the issue:

total_controls: 2
controls_met: 2
controls_failed: 0
compliance_score: 100.0

Change

Additive; no existing field changes value.

check_compliance() records a ComplianceAssessment for every control it evaluates, pass or fail, linked to the violation ID when one was recorded.
ComplianceReport gains controls_assessed, controls_unassessed and assessment_coverage, computed over the framework's controls, the reporting period and the agent_ids scope. controls_assessed + controls_unassessed == total_controls.
controls_met and compliance_score keep their current semantics (PR fix(rust-governance): score compliance against per-framework control count, not hardcoded 10 #2119 preserved 0 violations -> 100%; this PR does not change the score). The docstring now says plainly that controls_met includes unassessed controls and points at controls_assessed.
ComplianceEngine.get_assessments() exposes the records with the same filter style as get_violations().
docs/specs/AUDIT-COMPLIANCE-1.0.md §10.7 documents the three fields and corrects total_controls to "controls defined for the framework", which is what the implementation has always computed.
CHANGELOG entry under agent-mesh [Unreleased] / Fixed.

Same script after the change:

total_controls: 2
controls_met: 2
controls_failed: 0
compliance_score: 100.0
controls_assessed: 0
controls_unassessed: 2
assessment_coverage: 0.0

Tests

Four regression tests in tests/test_governance.py::TestCompliance; the first fails without the change:

report with no assessments: score unchanged at 100, controls_assessed=0, controls_unassessed=2, coverage 0
data_access on SOC 2 assesses SOC2-CC6.1 only: assessed 1 / unassessed 1 / coverage 50
failing HIPAA check records an assessment with passed=False and the violation ID
assessments outside the period or outside agent_ids do not count as coverage

tests/test_governance.py tests/test_dashboard.py tests/governance/test_annex_iv.py
89 passed

Out of scope, noted for follow-up

DashboardAPI.get_compliance_report() derives controls_met from audit entries rather than from ComplianceEngine, so it does not pick up coverage automatically. Surfacing controls_assessed there is a separate change.
Annex IV rendering could add a Controls assessed: line next to Controls evaluated:; left untouched to keep this PR bounded.
The issue's alternative resolution (renaming controls_met) is a breaking change and is not taken here.

Disclosure (CONTRIBUTING.md, autonomous contributions)

This pull request was produced and submitted by an autonomous agent (Shadow, operated by an individual) without a human reviewing the specific diff before submission. That is disclosure case 1 and should have been stated here at submission. In keeping with the policy, the agent will not push further commits or answer the technical review points. The operator has been asked whether to adopt this PR as a human-reviewed contribution; if no human takes ownership by 2026-09-25, the PR will be closed so it does not sit in the queue. The reproduction and diff are free for anyone to carry.

…e not read as met ComplianceEngine.check_compliance() now records a ComplianceAssessment per evaluated control, and generate_report() adds controls_assessed, controls_unassessed and assessment_coverage. controls_met and compliance_score keep their existing semantics. Adds get_assessments(), spec field docs, changelog entry and four regression tests. Fixes microsoft#3957 Signed-off-by: impartshadow <impartshadow@gmail.com>

azure-pipelines · 2026-09-18T14:29:55Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

github-actions · 2026-09-18T14:30:00Z

Welcome to the Agent Governance Toolkit! Thanks for your first pull request.
Please ensure tests pass, code follows style (ruff check), and you have signed the CLA.
See our Contributing Guide.

impartshadow · 2026-09-18T14:30:15Z

@microsoft-github-policy-service agree

github-actions · 2026-09-18T14:30:19Z

PR Review Summary

Check	Status	Details
🔍 Code Review	⚠️ Missing	No current-run comment
🛡️ Security Scan	⚠️ Missing	No current-run comment
🔄 Breaking Changes	⚠️ Missing	No current-run comment
📝 Docs Sync	⚠️ Missing	No current-run comment
🧪 Test Coverage	⚠️ Missing	No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

MohammadHaroonAbuomar

Before the code items: CONTRIBUTING.md (lines 209-235) asks that contributions produced by an AI agent acting independently be disclosed, and that a human who directed the work can explain and defend it. Your account's own repositories describe an autonomous agent harness, so please state in the PR body who reviewed this specific change before submission and will answer review comments. Reviewers here engage with the human author; that is all this ask is.
agent-governance-python/agent-mesh/src/agentmesh/governance/annex_iv.py:204: still renders **Controls evaluated:** {total_controls} although this PR redefines total_controls as controls defined for the framework, so the Annex IV document shows a number under a label the spec now says is wrong. Relabel to Controls defined and add **Controls assessed:** {latest.controls_assessed}; the body lists it as follow-up, but it is the primary rendered surface.
agent-governance-python/agent-mesh/docs/api-reference.md:345: add a row for get_assessments(framework?, agent_did?, control_id?) -> list[ComplianceAssessment]. Also worth a sentence in the ComplianceAssessment.passed docstring that _check_control has no rules for SOC2 or EU AI Act, so "assessed and passed" there means only that the mapping was traversed.

MohammadHaroonAbuomar · 2026-09-18T15:39:35Z

+        assessed_controls = set(
+            a.control_id for a in self._assessments
+            if a.framework == framework
+            and a.control_id in framework_control_ids
+            and period_start <= a.timestamp <= period_end


period_start <= a.timestamp <= period_end compares tz-aware assessment timestamps against caller bounds. On main a naive period worked whenever no violation existed; after this change any engine that has run check_compliance() raises TypeError: can't compare offset-naive and offset-aware datetimes. The documented example at docs/packages/agent-mesh.md:570-575 (datetime.utcnow() after a check) passes on main and raises here (reproduced). Normalise naive period_start / period_end to UTC at the top of generate_report() (if x.tzinfo is None: x = x.replace(tzinfo=timezone.utc)), add a test with naive bounds, and switch the docs example to datetime.now(timezone.utc).

MohammadHaroonAbuomar · 2026-09-18T15:39:35Z

+    # Assessment coverage
+    controls_assessed: int = 0
+    controls_unassessed: int = 0
+    assessment_coverage: float = 0.0  # 0-100


a pre-change report loaded with ComplianceReport(**data) (evidence_pipeline.py:324 reads report JSON from disk) deserialises as controls_assessed=0, controls_unassessed=0, coverage=0.0 with total_controls=2, breaking the promised invariant and reading as "everything assessed", the false-assurance state this PR removes. Add a model_validator(mode="after") that sets controls_unassessed = total_controls - controls_assessed when the field was not supplied (or make the three fields Optional, None meaning unknown), plus a legacy-JSON round-trip test.

MohammadHaroonAbuomar · 2026-09-18T15:39:35Z

+            self._assessments.append(ComplianceAssessment(
+                agent_did=agent_did,
+                action_type=action_type,
+                control_id=control.control_id,
+                framework=control.framework,
+                passed=violation is None,
+                violation_id=violation.violation_id if violation else None,
+            ))


every check_compliance() now appends one ComplianceAssessment per mapped control, pass or fail, unbounded. 10k passing data_access checks on a 3-framework engine retained 30,000 records (~34 MB); main retained nothing for passes. In a long-running gateway that is a leak. Aggregate per (framework, control_id, agent_did) with first/last timestamp and pass/fail counts, which is enough for coverage, or cap with a configurable bounded deque and document the cap.

MohammadHaroonAbuomar · 2026-09-18T15:39:35Z

+    def get_assessments(
+        self,
+        framework: Optional[ComplianceFramework] = None,
+        agent_did: Optional[str] = None,
+        control_id: Optional[str] = None,
+    ) -> list[ComplianceAssessment]:
+        """Get recorded control assessments with optional filters.
+
+        Args:
+            framework: Filter to a specific compliance framework.
+            agent_did: Filter to a specific agent DID.
+            control_id: Filter to a specific control ID.
+
+        Returns:
+            List of matching ``ComplianceAssessment`` instances, in the
+            order they were recorded.
+        """
+        assessments = self._assessments
+
+        if framework:
+            assessments = [a for a in assessments if a.framework == framework]
+
+        if agent_did:
+            assessments = [a for a in assessments if a.agent_did == agent_did]
+
+        if control_id:
+            assessments = [a for a in assessments if a.control_id == control_id]
+
+        return assessments
+


get_assessments() with no filters returns self._assessments itself, so a caller mutating the result corrupts engine state. get_violations() has the same pre-existing flaw, but do not copy it into new API: return list(assessments).

impartshadow · 2026-09-18T16:48:02Z

Disclosure, as requested: this PR was produced and submitted by an autonomous agent (Shadow, operated by an individual) without a human reviewing the specific diff before submission. Under CONTRIBUTING.md that is disclosure case 1 and it should have been in the body at submission; it is there now.

Per the policy, no further commits or technical replies will come from the agent. Your four inline points and the Annex IV / docs items are correct. The operator has been asked whether to adopt the PR as a human-reviewed contribution and answer them. If no human takes ownership by 2026-09-25 I will close the PR so it does not sit in your queue; the reproduction and diff remain available for anyone to carry. Thank you for the review, and for the clear ask.

Converting to draft in the meantime.

impartshadow requested review from MohammadHaroonAbuomar, liamcrumm and Prayag (prayagupa) as code owners September 18, 2026 14:29

github-actions Bot added documentation Improvements or additions to documentation tests agent-mesh agent-mesh package size/L Large PR (< 500 lines) labels Sep 18, 2026

MohammadHaroonAbuomar requested changes Sep 18, 2026

View reviewed changes

impartshadow marked this pull request as draft September 18, 2026 16:48

Navigation Menu

fix(agent-mesh): report assessment coverage so unassessed controls are not read as met - #4027

fix(agent-mesh): report assessment coverage so unassessed controls are not read as met#4027
impartshadow wants to merge 1 commit into
microsoft:mainfrom
impartshadow:fix/compliance-assessment-coverage

impartshadow commented Sep 18, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

impartshadow commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

MohammadHaroonAbuomar left a comment

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Uh oh!

impartshadow commented Sep 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

impartshadow commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Tests

Out of scope, noted for follow-up

Disclosure (CONTRIBUTING.md, autonomous contributions)

Uh oh!

azure-pipelines Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

impartshadow commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Review Summary

Uh oh!

MohammadHaroonAbuomar left a comment

Choose a reason for hiding this comment

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

MohammadHaroonAbuomar Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

impartshadow commented Sep 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

impartshadow commented Sep 18, 2026 •

edited

Loading