Ethical and Responsible AI Review

docs/_mitigations/mi-10.md

@@ -9,6 +9,7 @@ mitigates:
   - ri-5
   - ri-6
   - ri-11
+  - ri-25
 ---
 
 #### Supplier Controls:

docs/_mitigations/mi-11.md

@@ -9,6 +9,9 @@ mitigates:
   - ri-5
   - ri-6
   - ri-11
+  - ri-24
+  - ri-28
+  - ri-29
 --- 
 
 #### Human Feedback Loop

docs/_mitigations/mi-13.md

@@ -7,6 +7,8 @@ type:
  - Detective
 mitigates:
  - ri-12
+ - ri-25
+ - ri-27
 ---
 
-- Provide citations / linkage to the source data in Confluence
+- Provide citations / linkage to the source data in Confluence

docs/_mitigations/mi-15.md

@@ -7,6 +7,7 @@ type:
  - Detective
 mitigates:
  - ri-1
+ - ri-15
 ---
 
 Testing (evaluating model responses against a set of test cases) and monitoring (continuous evaluation in production) are vital elements in the process of the development and continued deployment of an LLM System, they ensure that your system is functioning properly and that your changes to the system bring a positive improvement, and more. 

docs/_mitigations/mi-18.md

@@ -0,0 +1,13 @@
+---
+sequence: 18
+title: Bias Audits
+layout: mitigation
+doc-status: Draft
+type:
+  - Detective
+mitigates:
+  - ri-24
+  - ri-28
+---
+
+Periodically evaluate RAG outputs using fairness benchmarks (e.g., demographic parity, equal opportunity) to detect and mitigate systemic bias across user groups or content types.

docs/_mitigations/mi-19.md

@@ -0,0 +1,13 @@
+---
+sequence: 19
+title: Explainability Layer
+layout: mitigation
+doc-status: Draft
+type:
+  - Preventative
+mitigates:
+  - ri-25
+  - ri-27
+---
+
+Enforce output-level explainability by integrating mechanisms (e.g., citation generators or evidence highlighting) to make source attribution mandatory in RAG responses.

docs/_mitigations/mi-2.md

@@ -7,6 +7,8 @@ type:
   - Preventative
 mitigates:
   - ri-1
+  - ri-26
+  - ri-30
 ---
 
 To mitigate the risk of sensitive data leakage and tampering in the vector store, the data filtering control ensures that sensitive information from internal knowledge sources, such as Confluence, is anonymized and/or entirely excluded before being processed by the model. This control aims to limit the exposure of sensitive organizational knowledge when creating embeddings that feed into the vector store, thus reducing the likelihood of confidential information being accessible or manipulated.

docs/_mitigations/mi-20.md

@@ -0,0 +1,12 @@
+---
+sequence: 20
+title: Automated PII Detection
+layout: mitigation
+doc-status: Draft
+type:
+  - Preventative
+mitigates:
+  - ri-26
+---
+
+Use NLP-based automated tools to scan and flag personally identifiable information in documents before they are embedded into vector stores.

docs/_mitigations/mi-21.md

@@ -0,0 +1,12 @@
+---
+sequence: 21
+title: Data Retention Policies
+layout: mitigation
+doc-status: Draft
+type:
+  - Preventative
+mitigates:
+  - ri-26
+---
+
+Establish clear rules and mechanisms to define how long data used in embeddings is retained, ensuring users are informed and consent to the use of their data in compliance with legal standards.

docs/_mitigations/mi-22.md

@@ -0,0 +1,12 @@
+---
+sequence: 22
+title: UX Controls for Traceability
+layout: mitigation
+doc-status: Draft
+type:
+  - Preventative
+mitigates:
+  - ri-29
+---
+
+Design user interfaces to alert or block when model outputs lack source traceability, enabling users to question or validate the origin of information.

docs/_mitigations/mi-23.md

@@ -0,0 +1,12 @@
+---
+sequence: 23
+title: Model Drift Monitoring
+layout: mitigation
+doc-status: Draft
+type:
+  - Detective
+mitigates:
+  - ri-28
+---
+
+Monitor RAG output trends over time using statistical or semantic methods to detect unintended shifts or biases caused by embedding or model updates.

docs/_mitigations/mi-24.md

@@ -0,0 +1,14 @@
+---
+sequence: 24
+title: Embedding Refresh Strategy
+layout: mitigation
+doc-status: Draft
+type:
+  - Preventative
+mitigates:
+  - ri-24
+  - ri-28
+  - ri-30
+---
+
+Implement a scheduled review and re-ingestion strategy for vector embeddings to ensure outdated or biased representations are periodically updated or removed.

docs/_mitigations/mi-4.md

@@ -15,6 +15,7 @@ mitigates:
   - ri-8
   - ri-10
   - ri-13
+  - ri-25
 ---
 
 #### What to log/monitor

docs/_mitigations/mi-5.md

@@ -10,6 +10,8 @@ mitigates:
  - ri-5
  - ri-6
  - ri-12
+ - ri-24
+ - ri-29
 ---
 
 System Acceptance Testing is the final phase of the software testing process where the complete system is tested against the specified requirements to ensure it meets the criteria for deployment. For non-AI systems, this will involve creating a number of test cases which are executed, with an expectation that when all tests pass the system is guaranteed to meet its requirements. 
@@ -28,4 +30,4 @@ System Acceptance Testing is a highly effective control for understanding the ov
 * [GitHub - openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.](https://github.com/openai/evals)
 * [Evaluation / 🦜️🔗 LangChain](https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/)
 * [Promptfoo](https://www.promptfoo.dev/)
-* [Inspect](https://inspect.ai-safety-institute.org.uk/) 
+* [Inspect](https://inspect.ai-safety-institute.org.uk/) 

docs/_mitigations/mi-6.md

@@ -8,6 +8,9 @@ type:
 mitigates:
   - ri-1
   - ri-2
+  - ri-26
+  - ri-28
+  - ri-30
 ---
 
 - Data is classified within the Confluence data store, and filtered prior to ingestion

docs/_mitigations/mi-7.md

@@ -7,6 +7,7 @@ type:
   - Preventative
 mitigates:
   - ri-1
+  - ri-25
 ---
 
 This control is about legal agreements between the SaaS inference provider and the organization. Those legal agreements not only have to exists, but have to be understood by the organization to make sure they comply with all requirements.

docs/_risks/ri-24.md

@@ -0,0 +1,11 @@
+---
+sequence: 24
+title: Unintended Societal Impact
+layout: risk
+doc-status: Pre-Draft
+type: Bias
+---
+
+- RAG outputs may unintentionally reinforce harmful stereotypes or social biases.
+- RAG systems may retrieve or synthesize content that reflects historical biases, leading to reputational damage and ethical violations. 
+- Especially risky in decision-support contexts (e.g., hiring, loans).

docs/_risks/ri-25.md

@@ -0,0 +1,11 @@
+---
+sequence: 25
+title: Non-compliance with AI Regulation
+layout: risk
+doc-status: Pre-Draft
+type: Integrity
+---
+
+- Non-compliance with AI Regulation
+- Lack of transparency, traceability, and explainability in RAG outputs can violate AI and data privacy laws.
+- Audits may fail due to absent model versioning or unclear data flows.

docs/_risks/ri-26.md

@@ -0,0 +1,10 @@
+---
+sequence: 26
+title: Inadequate Consent Management
+layout: risk
+doc-status: Pre-Draft
+type: Confidentiality
+---
+
+- Embeddings may include PII or sensitive content without proper consent.
+- Documents containing sensitive employee or client information may be embedded without user consent, violating privacy norms and regulations.

docs/_risks/ri-27.md

@@ -0,0 +1,10 @@
+---
+sequence: 27
+title: Lack of Explainability in Responses
+layout: risk
+doc-status: Pre-Draft
+type: Integrity
+---
+
+- Users receive untraceable outputs lacking cited sources.
+- When the model response doesn't show source documents, users cannot verify accuracy, undermining transparency and responsible AI practices.

docs/_risks/ri-28.md

@@ -0,0 +1,10 @@
+---
+sequence: 28
+title: Bias Propagation via Embedding Drift
+layout: risk
+doc-status: Pre-Draft
+type: Bias
+---
+
+- Embeddings evolve over time, inadvertently favoring dominant or biased narratives.
+- As RAG embeddings are updated, they may begin to reflect and reinforce the same institutional or historical biases, particularly in high-volume document repositories.

docs/_risks/ri-29.md

@@ -0,0 +1,10 @@
+---
+sequence: 29
+title: Inappropriate Automation of Ethical Judgments
+layout: risk
+doc-status: Pre-Draft
+type: Bias / Integrity
+---
+
+- AI-generated content may be treated as final advice or decision-making without oversight.
+- Users may misinterpret RAG responses as ethical or legal truth, especially in contexts like HR, compliance, or customer advice, leading to unaccountable automation of moral decisions.

docs/_risks/ri-30.md

@@ -0,0 +1,10 @@
+---
+sequence: 30
+title: Unreviewed Data Inclusion
+layout: risk
+doc-status: Pre-Draft
+type: Integrity
+---
+
+- Non-curated or outdated content may be added to vector stores, leading to misinformation or ethical risks.
+- If internal documents or outdated guidance are added to vector stores without review, the RAG system may retrieve misleading or low-quality information that affects users' decisions.