TLM-Demo-Notebook/TLM-Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6e4f227a-db80-4d19-a907-5035e25754f6",
   "metadata": {},
   "source": [
    "# Trustworthy Language Model (TLM) Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a468ee8-cba2-4c0e-8991-077d7d438a99",
   "metadata": {},
   "source": [
    "[![Run in Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cleanlab/cleanlab-tools/blob/main/TLM-Demo-Notebook/TLM-Demo.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "394348f6-b428-4607-a838-6f3368dc5d0e",
   "metadata": {},
   "source": [
    "### Imports and Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6fe8db7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install cleanlab-tlm pandas scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "035734ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set your API key\n",
    "import os\n",
    "os.environ[\"CLEANLAB_TLM_API_KEY\"] = \"<API key>\"  # Get your API key from: https://tlm.cleanlab.ai/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d8d5d4b0-c6c9-453c-ac58-19aa85f56837",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 150)\n",
    "from cleanlab_tlm import TLM\n",
    "from sklearn.metrics import accuracy_score\n",
    "import requests\n",
    "\n",
    "\n",
    "# Helper to selectively print lines from customer conversation\n",
    "def print_conversation(conversation, indices=None):\n",
    "    lines = conversation.split('\\n')\n",
    "    conversation_entries = []\n",
    "    prefix = \"\\n.\\n.\\n.\\n\"\n",
    "    for line in lines:\n",
    "        if line.startswith('Customer:') or line.startswith('Agent:'):\n",
    "            conversation_entries.append(line.strip())\n",
    "    if indices is None:\n",
    "        indices = list(range(len(conversation_entries)))\n",
    "        prefix = \"\\n\\n\"\n",
    "    print(prefix.join([conversation_entries[i] for i in indices if i < len(conversation_entries)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1707d06d-677c-4528-a8d5-1cc9d7d8edb3",
   "metadata": {},
   "source": [
    "## What is TLM?\n",
    "\n",
    "TLM provides a trustworthiness score and explanation to any LLM generated response. \n",
    "\n",
    "This enables GenAI powered applications to enter production with speed and reliability."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d593ec58-a051-4f12-a16d-7b8b51a28b50",
   "metadata": {},
   "source": [
    "### TLM\n",
    "Example of TLM with a simple prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8e160f38",
   "metadata": {},
   "outputs": [],
   "source": [
    "tlm = TLM(options={\"log\": [\"explanation\"]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f6824052-b1a3-4e13-b3ff-96babdeee063",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: The third month of the year alphabetically is \"March.\" The months in alphabetical order are:\n",
      "\n",
      "1. April\n",
      "2. August\n",
      "3. December\n",
      "4. February\n",
      "5. January\n",
      "6. July\n",
      "7. June\n",
      "8. March\n",
      "9. May\n",
      "10. November\n",
      "11. October\n",
      "12. September\n",
      "Trustworthiness Score: 0.4979648802626605\n",
      "Explanation: The answer provided states that \"March\" is the third month of the year alphabetically. However, when listing the months in alphabetical order, \"March\" is actually the eighth month. Therefore,  incorrect. \n",
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "December.\n"
     ]
    }
   ],
   "source": [
    "question = \"What's the third month of the year alphabetically?\"\n",
    "output = tlm.prompt(question)\n",
    "\n",
    "print(f'Response: {output[\"response\"]}')\n",
    "print(f'Trustworthiness Score: {output[\"trustworthiness_score\"]}')\n",
    "print(f'Explanation: {output[\"log\"][\"explanation\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76afd5f8",
   "metadata": {},
   "source": [
    "Example of TLM evaluating existing prompt, response pair:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ca036dfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same question, different responses\n",
    "question = \"Who wrote Harry Potter? Respond with only their name.\"\n",
    "# Correct response\n",
    "response_1 = \"J.K. Rowling\"\n",
    "# Incorrect response\n",
    "response_2 = \"Albus Dumbledore\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dbe298ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trustworthiness Score: 0.9936905078946259\n"
     ]
    }
   ],
   "source": [
    "trust_score_response_1 = tlm.get_trustworthiness_score(question, response_1)\n",
    "print(f'Trustworthiness Score: {trust_score_response_1[\"trustworthiness_score\"]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0b6c14df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trustworthiness Score: 0.0012339607050982507\n"
     ]
    }
   ],
   "source": [
    "trust_score_response_2 = tlm.get_trustworthiness_score(question, response_2)\n",
    "print(f'Trustworthiness Score: {trust_score_response_2[\"trustworthiness_score\"]}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e0be29c-74aa-40bd-90a6-b2a4c29a4153",
   "metadata": {},
   "source": [
    "## TLM offers many native models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "736cc840-1654-4f5e-80a3-0170a1eb3c7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': '3 billion', 'trustworthiness_score': 0.6751875172590791}"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tlm = TLM(options={\"model\": \"gpt-4.5-preview\"})\n",
    "\n",
    "tlm.prompt(\"How many parameters does the o3-mini LLM have? Answer in numerical format.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "1343c5c4-6609-4c38-a8f9-37c574aac6d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': '2.8 billion', 'trustworthiness_score': 0.20813908670322}"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tlm = TLM(options={\"model\": \"claude-3.7-sonnet\"})\n",
    "\n",
    "tlm.prompt(\"How many parameters does the o3-mini LLM have? Answer in numerical format.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "06e5b282-dce5-425a-a2ff-10991645aba4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'I couldn’t find any publicly available information that specifies a parameter count for the OpenAI o3‑mini model. Without an official disclosure, it isn’t possible to provide a numerical answer.',\n",
       " 'trustworthiness_score': 0.8630028862775448}"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tlm = TLM(options={\"model\": \"o3-mini\"})\n",
    "\n",
    "prompt = \"How many parameters does the OpenAI o3-mini LLM have? Answer in numerical format.\"\n",
    "\n",
    "tlm.prompt(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "d831c27a-9981-49bd-b1bd-37e5f74d5c88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': '0.05', 'trustworthiness_score': 0.9921643823071055}"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt = \"A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? Answer in numeric format\"\n",
    "\n",
    "tlm.prompt(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "febea3f0-9c94-4714-ad52-87a21efc8256",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'Let x be the price of the Super De Duper Amazing Willy Wonka Extra Cocoa Trimethylsiloxysilicate-Free Chocolate Bar. Then, the Holographic Mint Condition Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch-Produced Super De Duper Big Black Bug Pokemon Trading Card costs x + 1.00.\\n\\nThe total cost is given by:\\nx + (x + 1.00) = 1.10\\n\\nCombine like terms:\\n2x + 1.00 = 1.10\\n\\nSubtract 1.00 from both sides:\\n2x = 0.10\\n\\nDivide both sides by 2:\\nx = 0.05\\n\\nThus, the Super De Duper Amazing Willy Wonka Extra Cocoa Trimethylsiloxysilicate-Free Chocolate Bar costs 0.05.',\n",
       " 'trustworthiness_score': 0.8593818064195563}"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt = \"A Holographic Mint Condition Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch-Produced Super De Duper Big Black Bug Pokemon Trading Card and a Super De Duper Amazing Willy Wonka Extra Cocoa Trimethylsiloxysilicate-Free Chocolate Bar cost $1.10 in total. The Holographic Mint Condition Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch-Produced Super De Duper Big Black Bug Pokemon Trading Card costs $1.00 more than the Super De Duper Amazing Willy Wonka Extra Cocoa Trimethylsiloxysilicate-Free Chocolate Bar. How much does the Super De Duper Amazing Willy Wonka Extra Cocoa Trimethylsiloxysilicate-Free Chocolate Bar cost? Answer in numeric format\"\n",
    "\n",
    "tlm.prompt(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97fd5146-c4ae-44ed-8f1d-2a59c28a5615",
   "metadata": {},
   "source": [
    "## Use Cases\n",
    "\n",
    "1. Compute Trustworthiness Scores for any LLM\n",
    "   - Turn your own LLM into a TLM to flag hallucinations\n",
    "2. Question Answering / RAG\n",
    "   - Answer specific questions about your knowledge base. \n",
    "3. Data Extraction\n",
    "   - Extract relevant information from your knowledge base. \n",
    "4. Classification\n",
    "   - Classify and categorize your knowledge base. \n",
    "   - Auto-boost accuracy, increase the accuracy of your LLM responses.\n",
    "   - Auto-labeling, label your dataset faster and more accurately.\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dfa0e6d-c9c9-4f9c-b071-3bb2c726c5d4",
   "metadata": {},
   "source": [
    "### Initialize and View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "535a62cf-1f14-4ed8-93e6-d4f08a4979ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>YR96</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Lisa. How may I assist you today?\\n\\nCustomer: Hi, Lisa. I need help with a ret...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>XZ39</td>\n",
       "      <td>Customer: Hello, I am calling to inquire about my return and exchange order.\\n\\nAgent: Good afternoon, thank you for contacting BrownBox customer ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "3  YR96   \n",
       "4  XZ39   \n",
       "\n",
       "                                                                                                                                            conversation  \n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...  \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...  \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...  \n",
       "3  Agent: Thank you for calling BrownBox Customer Support. My name is Lisa. How may I assist you today?\\n\\nCustomer: Hi, Lisa. I need help with a ret...  \n",
       "4  Customer: Hello, I am calling to inquire about my return and exchange order.\\n\\nAgent: Good afternoon, thank you for contacting BrownBox customer ...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize TLM\n",
    "tlm = TLM(quality_preset = \"low\", options={\"log\": [\"explanation\"]})\n",
    "\n",
    "# Read in data\n",
    "df = pd.read_csv(\"https://raw.githubusercontent.com/cleanlab/cleanlab-tools/refs/heads/main/TLM-Demo-Notebook/customer-service-conversation.csv\")\n",
    "df_original = df.copy()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b4a5c79d-78de-40ec-9216-37a9814bbe5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\n",
      "\n",
      "Customer: Hi Rachel, I recently purchased a water purifier from your website, and I have a question about the warranty. Can you help me with that?\n",
      "\n",
      "Agent: Of course, I'd be happy to help. Can you please provide me with your order number and the name of the water purifier you purchased?\n",
      "\n",
      "Customer: My order number is BB12345678, and the water purifier is the AquaMax Pro.\n",
      "\n",
      "Agent: Thank you for the information. Now, can you please let me know what your question is regarding the warranty?\n",
      "\n",
      "Customer: I received the water purifier a few days ago, and I'm not sure when the warranty period starts. Can you tell me the start date of the warranty?\n",
      "\n",
      "Agent: Sure, I can help you with that. The warranty for your AquaMax Pro water purifier starts from the date of delivery. So, the warranty period for your water purifier started on the day it was delivered to you.\n",
      "\n",
      "Customer: Okay, that's good to know. And how long is the warranty period for the water purifier?\n",
      "\n",
      "Agent: The warranty period for the AquaMax Pro water purifier is one year from the date of delivery. During this period, if there are any defects or issues with the product, you can get it repaired or replaced free of charge.\n",
      "\n",
      "Customer: Alright, thank you for the information, Rachel. I appreciate your help.\n",
      "\n",
      "Agent: You're welcome. Is there anything else you need assistance with?\n",
      "\n",
      "Customer: No, that's all. Thank you again for your help.\n",
      "\n",
      "Agent: It was my pleasure assisting you. If you have any further questions or concerns, please feel free to reach out to us. Have a great day!\n"
     ]
    }
   ],
   "source": [
    "print(df.iloc[0].conversation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54aef40d-05f4-4016-81f0-b4b1db310f9d",
   "metadata": {},
   "source": [
    "## Use Case: Compute Trustworthiness Scores for any LLM\n",
    "Use TLM to get a trustworthiness score for any (prompt, response) pair!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30ce5d3b-21cc-403f-8183-d40db0c4ebf0",
   "metadata": {},
   "source": [
    "#### Single Conversation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "34e0d94f-40c5-4550-90fa-efd3ec83d380",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\n",
      "\n",
      "Customer: Hi Sarah, I placed an order for a Smart Watch, and I want to confirm if it has been processed.\n",
      "\n",
      "Agent: Sure, I can help you with that. May I know your name and order number, please?\n",
      "\n",
      "Customer: My name is Jane, and my order number is BB123456.\n",
      "\n",
      "Agent: Thank you, Jane. Let me check the order status for you. (puts customer on hold for a minute) I can confirm that your order has been processed, and it is currently in transit. You should receive it within the next three business days.\n",
      "\n",
      "Customer: Great! Thank you for checking that.\n",
      "\n",
      "Agent: You're welcome, Jane. Is there anything else I can assist you with?\n",
      "\n",
      "Customer: No, that's all. Thank you for your help.\n",
      "\n",
      "Agent: You're welcome. If you have any further questions, please don't hesitate to call us back. Have a great day!\n",
      "\n",
      "Customer: You too. Goodbye!\n",
      "\n",
      "Agent: Goodbye!\n"
     ]
    }
   ],
   "source": [
    "conversation = df[df.id==\"NQ67\"].conversation.values[0]\n",
    "print(conversation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36323287-79ea-40db-877d-625de1d15c69",
   "metadata": {},
   "source": [
    "#### Summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0d8635d2-738a-46c9-832b-7495c4da2a54",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Score for Response 1: 0.9837152780689016 \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 2: 0.6784452368897584 \n",
      "\n",
      "TLM Explanation for Response 2: The proposed answer inaccurately summarizes the conversation. The customer, Jane, did not specifically inquire about the shipping speed; she asked for confirmation on whether her order had been processed. The agent confirmed that the order was processed and mentioned that it was in transit, with an expected delivery within three business days. While the answer does mention the three-day shipping timeframe, it misrepresents the customer's original question, which was about the order's processing status rather than the speed of shipping. Therefore, the answer does not accurately reflect the content of the conversation. \n",
      "\n",
      "----------------------------------------------------------------------------\n",
      "\n",
      "TLM Score for Response 3: 0.3596345957131949 \n",
      "\n",
      "TLM Explanation for Response 3: The proposed answer incorrectly states that the customer wants to cancel their order and get a refund, which is not supported by the conversation. In fact, the customer, Jane, inquired about the status of her order for a Smart Watch and was informed by the agent that it had been processed and was in transit. There was no mention of cancellation or a refund request at any point in the dialogue. Therefore, the proposed answer does not accurately reflect the content of the conversation. \n",
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Jane confirms her Smart Watch order status with Sarah, who states it’s processed and in transit, expected in three days. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"Given the following conversation, respond with a very short summarization. Conversation: {}\".format(conversation)\n",
    "\n",
    "response_1 = \"The customer confirmed her Smart Watch order was processed and is in transit, expected in three days.\"\n",
    "response_2 = \"Customer wants to check the shipping speed of her Smart Watch. Agent confirmed the shipping speed is set to 3 days.\"\n",
    "response_3 = \"Customer wants to cancel their order and get a refund. Agent processed the cancellation.\"\n",
    "\n",
    "trust_score_1 = tlm.get_trustworthiness_score(prompt, response_1)\n",
    "trust_score_2 = tlm.get_trustworthiness_score(prompt, response_2)\n",
    "trust_score_3 = tlm.get_trustworthiness_score(prompt, response_3)\n",
    "\n",
    "print(f\"TLM Score for Response 1: {trust_score_1['trustworthiness_score']}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 2: {trust_score_2['trustworthiness_score']}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 2: {trust_score_2['log']['explanation']}\", \"\\n\")\n",
    "print(\"----------------------------------------------------------------------------\\n\")\n",
    "print(f\"TLM Score for Response 3: {trust_score_3['trustworthiness_score']}\", \"\\n\")\n",
    "print(f\"TLM Explanation for Response 3: {trust_score_3['log']['explanation']}\", \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6073a73e-5c97-4bef-9171-0d2761838135",
   "metadata": {},
   "source": [
    "## Use Case: Question Answering (RAG)\n",
    "\n",
    "In this example, we will use TLM to answer a customer's question about the company's customer service policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "57bf6e25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The following is the customer service policy of ACME Inc.\n",
      "# ACME Inc. Customer Service Policy\n",
      "\n",
      "## Table of Contents\n",
      "1. Free Shipping Policy\n",
      "2. Free Returns Policy\n",
      "3. Fraud Detection Guidelines\n",
      "4. Customer Interaction Tone\n",
      "\n",
      "## 1. Free Shipping Policy\n",
      "\n",
      "### 1.1 Eligibility Criteria\n",
      "- Free shipping is available on all orders over $50 within the continental United States.\n",
      "- For orders under $50, a flat rate shipping fee of $5.99 will be applied.\n",
      "- Free shipping is not available for expedited shipping methods (e.g., overnight or 2-day shipping).\n",
      "\n",
      "### 1.2 Exclusions\n",
      "- Free shipping does not apply to orders shipped to Alaska, Hawaii, or international destinations.\n",
      "- Oversized or heavy items may incur additional shipping charges, which will be clearly communicated to the customer before purchase.\n",
      "\n",
      "### 1.3 Handling Customer Inquiries\n",
      "- If a customer inquires about free shipping eligibility, verify the order total and shipping destination.\n",
      "- Inform customers of ways to qualify for free shipping (e.g., adding items to reach the $50 threshold).\n",
      "- For orders just below the threshold, you may offer a one-time courtesy free shipping if it's the customer's first purchase or if they have a history of large orders.\n",
      "\n",
      "## 2. Free Returns Policy\n",
      "\n",
      "### 2.1 Eligibility Criteria\n",
      "- Free returns are available for all items within 30 days of the delivery date.\n",
      "- Items must be unused, unworn, and in their original packaging with all tags attached.\n",
      "- Free returns are limited to standard shipping methods within the continental United States.\n",
      "\n",
      "### 2.2 Exclusions\n",
      "- Final sale items, as marked on the product page, are not eligible for free returns.\n",
      "- Customized or personalized items are not eligible for free returns unless there is a manufacturing defect.\n",
      "- Undergarments, swimwear, and earrings are not eligible for free returns due to hygiene reasons.\n",
      "\n",
      "### 2.3 Process for Handling Returns\n",
      "1. Verify the order date and ensure it falls within the 30-day return window.\n",
      "2. Ask the customer about the reason for the return and document it in the system.\n",
      "3. Provide the customer with a prepaid return label if they qualify for free returns.\n",
      "4. Inform the customer of the expected refund processing time (5-7 business days after receiving the return).\n",
      "\n",
      "### 2.4 Exceptions\n",
      "- For items damaged during shipping or with manufacturing defects, offer an immediate replacement or refund without requiring a return.\n",
      "- For returns outside the 30-day window, use discretion based on the customer's history and the reason for the late return. You may offer store credit as a compromise.\n",
      "\n",
      "## 3. Fraud Detection Guidelines\n",
      "\n",
      "### 3.1 Red Flags for Potential Fraud\n",
      "- Multiple orders from the same IP address with different customer names or shipping addresses.\n",
      "- Orders with unusually high quantities of the same item.\n",
      "- Shipping address different from the billing address, especially if in different countries.\n",
      "- Multiple failed payment attempts followed by a successful one.\n",
      "- Customers pressuring for immediate shipping or threatening to cancel the order.\n",
      "\n",
      "### 3.2 Verification Process\n",
      "1. For orders flagging as potentially fraudulent, place them on hold for review.\n",
      "2. Verify the customer's identity by calling the phone number on file.\n",
      "3. Request additional documentation (e.g., photo ID, credit card statement) if necessary.\n",
      "4. Cross-reference the shipping address with known fraud databases.\n",
      "\n",
      "### 3.3 Actions for Confirmed Fraud\n",
      "- Cancel the order immediately and refund any charges.\n",
      "- Document the incident in the customer's account and flag it for future reference.\n",
      "- Report confirmed fraud cases to the appropriate authorities and credit card companies.\n",
      "\n",
      "### 3.4 False Positives\n",
      "- If a legitimate customer is flagged, apologize for the inconvenience and offer a small discount or free shipping on their next order.\n",
      "- Document the incident to improve our fraud detection algorithms.\n",
      "\n",
      "## 4. Customer Interaction Tone\n",
      "\n",
      "### 4.1 General Guidelines\n",
      "- Always maintain a professional, friendly, and empathetic tone.\n",
      "- Use the customer's name when addressing them.\n",
      "- Listen actively and paraphrase the customer's concerns to ensure understanding.\n",
      "- Avoid negative language; focus on what can be done rather than what can't.\n",
      "\n",
      "### 4.2 Specific Scenarios\n",
      "\n",
      "#### Angry or Frustrated Customers\n",
      "- Remain calm and do not take comments personally.\n",
      "- Acknowledge the customer's feelings and apologize for their negative experience.\n",
      "- Focus on finding a solution and clearly explain the steps you'll take to resolve the issue.\n",
      "- If necessary, offer to escalate the issue to a supervisor.\n",
      "\n",
      "#### Confused or Indecisive Customers\n",
      "- Be patient and offer clear, concise explanations.\n",
      "- Ask probing questions to better understand their needs.\n",
      "- Provide options and explain the pros and cons of each.\n",
      "- Offer to send follow-up information via email if the customer needs time to decide.\n",
      "\n",
      "#### VIP or Loyal Customers\n",
      "- Acknowledge their status and thank them for their continued business.\n",
      "- Be familiar with their purchase history and preferences.\n",
      "- Offer exclusive deals or early access to new products when appropriate.\n",
      "- Go above and beyond to exceed their expectations.\n",
      "\n",
      "### 4.3 Language and Phrasing\n",
      "- Use positive language: \"I'd be happy to help you with that\" instead of \"I can't do that.\"\n",
      "- Avoid technical jargon or abbreviations that customers may not understand.\n",
      "- Use \"we\" statements to show unity with the company: \"We value your feedback\" instead of \"The company values your feedback.\"\n",
      "- End conversations on a positive note: \"Is there anything else I can assist you with today?\"\n",
      "\n",
      "### 4.4 Written Communication\n",
      "- Use proper grammar, spelling, and punctuation in all written communications.\n",
      "- Keep emails and chat responses concise and to the point.\n",
      "- Use bullet points or numbered lists for clarity when providing multiple pieces of information.\n",
      "- Include a clear call-to-action or next steps at the end of each communication.\n",
      "\n",
      "The following dialogue features a discussion between a user and a customer service bot. The bot attempts to help the customer but must respect the guidelines in the customer service policy. The bot provides very accurate and concise answers. The bot does not tell the user to contact customer service\n",
      "Remember, as a representative of ACME Inc., you are often the first point of contact for our customers. Your interactions should always reflect our commitment to exceptional customer service and satisfaction.\n"
     ]
    }
   ],
   "source": [
    "# Read customer service policy\n",
    "url = \"https://raw.githubusercontent.com/cleanlab/cleanlab-tools/refs/heads/main/TLM-Demo-Notebook/customer-service-policy.md\"\n",
    "response = requests.get(url)\n",
    "customer_service_policy = response.text\n",
    "print(customer_service_policy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7ec2414d",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_template = '''You are a customer service agent for ACME Inc. Your task is to answer the following customer question based on the customer service policy.\n",
    "\n",
    "Customer Service Policy: {}\n",
    "Customer Question: {}\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ce9fed9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thank you for your question! At ACME Inc., we offer free returns for items within 30 days of delivery, provided they are unused, unworn, and in their original packaging with all tags attached. However, if the item is not defective and you are simply disappointed with your purchase, it may not qualify for a refund under our policy. \n",
      "\n",
      "If you would like to return the item, please ensure it meets the eligibility criteria, and I can guide you through the return process. If the return is approved, you will receive a refund within 5-7 business days after we receive the item. If you have any further questions or need assistance, feel free to ask!\n",
      "\n",
      "Trustworthiness Score:  0.8652735152398818\n"
     ]
    }
   ],
   "source": [
    "customer_question = '''Can I get a refund if I was dissapointed with my purchase, even if it was not defective?'''\n",
    "customer_question_prompt = prompt_template.format(customer_service_policy, customer_question)\n",
    "response = tlm.prompt(customer_question_prompt)\n",
    "print(response[\"response\"])\n",
    "print()\n",
    "print(\"Trustworthiness Score: \", response[\"trustworthiness_score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "81d2c954",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thank you for your question! Unfortunately, under our Free Returns Policy, undergarments and earrings are not eligible for free returns due to hygiene reasons. If the noserings are unworn and in their original packaging, you may still be able to return them, but please note that you would be responsible for the return shipping costs. If you have any further questions or need assistance, feel free to ask!\n",
      "\n",
      "Trustworthiness Score:  0.7615646194326461\n",
      "\n",
      "Explanation:  incorrect because it states that the customer may return the noserings if they are unworn and in their original packaging, which contradicts the customer service policy. According to the Free Returns Policy, earrings, including noserings, are not eligible for free returns due to hygiene reasons. Therefore, the customer cannot return the noserings regardless of their condition. The answer should have clearly communicated that the noserings cannot be returned at all, rather than suggesting a possibility of return with shipping costs.\n"
     ]
    }
   ],
   "source": [
    "customer_question = '''Can I return a pair of noserings if they dont fit?'''\n",
    "customer_question_prompt = prompt_template.format(customer_service_policy, customer_question)\n",
    "response = tlm.prompt(customer_question_prompt)\n",
    "print(response[\"response\"])\n",
    "print()\n",
    "print(\"Trustworthiness Score: \", response[\"trustworthiness_score\"])\n",
    "print()\n",
    "print(\"Explanation: \", response[\"log\"][\"explanation\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5561a3b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm here to assist you through this chat, but I understand that you may prefer to speak with someone directly. Unfortunately, I can't provide a phone number or direct you to customer service. However, I'm happy to help you with any questions or concerns you have right here! What can I assist you with today?\n",
      "\n",
      "Trustworthiness Score:  0.7684822273449248\n",
      "\n",
      "Explanation:  The proposed answer does not align with the customer service policy, which emphasizes maintaining a professional and empathetic tone while addressing customer inquiries. The customer is asking for a way to contact customer service by phone, and while the bot cannot provide a phone number or direct the customer to customer service, it should acknowledge the customer's request more positively. The response could have included an expression of understanding regarding the customer's preference for phone communication and reassured them that their concerns can be addressed through chat. The answer should focus on providing assistance rather than stating limitations. Therefore, the response is not appropriate as it does not fully adhere to the guidelines for customer interaction.\n"
     ]
    }
   ],
   "source": [
    "customer_question = '''Is there any way to contact customer service by phone instead of through this chat?'''\n",
    "customer_question_prompt = prompt_template.format(customer_service_policy, customer_question)\n",
    "\n",
    "response = tlm.prompt(customer_question_prompt)\n",
    "print(response[\"response\"])\n",
    "print()\n",
    "print(\"Trustworthiness Score: \", response[\"trustworthiness_score\"])\n",
    "print()\n",
    "print(\"Explanation: \", response[\"log\"][\"explanation\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af33eb42-433d-42c4-b175-19f4ac3b0e14",
   "metadata": {},
   "source": [
    "## Use Case: Data Extraction\n",
    "Extract customer name, contact information, and order number from conversation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb076560-cb07-4dfa-9cb6-738d2c762a06",
   "metadata": {},
   "outputs": [],
   "source": [
    "extraction = ''' Please extract these details from the following conversation: \n",
    "customer name, customer phone number, customer email, and order number.\n",
    "If a detail is not present, respond with None for that detail.\n",
    "Here is the conversation: {}'''\n",
    "extraction_prompts = [extraction.format(conversation) for conversation in df.conversation.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fc05a0f9-8533-409f-97c1-6d8f9c737d7e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|██████████|\n"
     ]
    }
   ],
   "source": [
    "# Generate answers and trustworthiness scores\n",
    "extractions = tlm.prompt(extraction_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d800ee87-62fb-4643-a94d-3e347235cc82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add results to df\n",
    "df_extr = df.copy()\n",
    "df_extr[\"extraction\"] = [entry['response'] for entry in  extractions]\n",
    "df_extr[\"extraction_score\"] = [entry['trustworthiness_score'] for entry in  extractions]\n",
    "df_extr[\"extraction_expl\"] = [entry['log']['explanation'] for entry in  extractions]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f5366ec-5785-470d-a7af-1eda3b95b8b6",
   "metadata": {},
   "source": [
    "#### TLM Responses with Lowest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "490b5bcc-c937-4842-a55a-47450382b635",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>extraction</th>\n",
       "      <th>extraction_score</th>\n",
       "      <th>extraction_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>BC62</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How can I assist you today?\\n\\nCustomer: Hi Rachel, I received a wrong ...</td>\n",
       "      <td>Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: BB987654321</td>\n",
       "      <td>0.720271</td>\n",
       "      <td>The answer correctly extracts the order number from the conversation, which is BB987654321. However, it fails to provide any information regarding...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>SC68</td>\n",
       "      <td>Customer: Hi, I need some help with my BrownBox account.\\n\\nAgent: Hello, thank you for contacting BrownBox customer support. My name is Jack. May...</td>\n",
       "      <td>Customer Name: John  \\nCustomer Phone Number: None  \\nCustomer Email: john@email.com  \\nOrder Number: 123456789</td>\n",
       "      <td>0.749987</td>\n",
       "      <td>The proposed answer correctly identifies the customer name as \"John\" based on the conversation. However, it incorrectly lists the order number as ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>GU39</td>\n",
       "      <td>Agent: Thank you for calling BrownBox customer support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I received a damaged ...</td>\n",
       "      <td>Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: 123456789</td>\n",
       "      <td>0.893278</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "79  BC62   \n",
       "39  SC68   \n",
       "13  GU39   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "79  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How can I assist you today?\\n\\nCustomer: Hi Rachel, I received a wrong ...   \n",
       "39  Customer: Hi, I need some help with my BrownBox account.\\n\\nAgent: Hello, thank you for contacting BrownBox customer support. My name is Jack. May...   \n",
       "13  Agent: Thank you for calling BrownBox customer support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I received a damaged ...   \n",
       "\n",
       "                                                                                                           extraction  \\\n",
       "79          Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: BB987654321     \n",
       "39  Customer Name: John  \\nCustomer Phone Number: None  \\nCustomer Email: john@email.com  \\nOrder Number: 123456789     \n",
       "13            Customer Name: None  \\nCustomer Phone Number: None  \\nCustomer Email: None  \\nOrder Number: 123456789     \n",
       "\n",
       "    extraction_score  \\\n",
       "79          0.720271   \n",
       "39          0.749987   \n",
       "13          0.893278   \n",
       "\n",
       "                                                                                                                                          extraction_expl  \n",
       "79  The answer correctly extracts the order number from the conversation, which is BB987654321. However, it fails to provide any information regarding...  \n",
       "39  The proposed answer correctly identifies the customer name as \"John\" based on the conversation. However, it incorrectly lists the order number as ...  \n",
       "13                                                                                                        Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_trust_extractions = df_extr.sort_values(by=\"extraction_score\", ascending=True).head(3)\n",
    "low_trust_extractions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ffd5c9d-47d5-4677-a4f2-728aa16d79d5",
   "metadata": {},
   "source": [
    "#### LLM responds with incorrect model number that was corrected by agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ba92c4b5-bb3d-4727-bba6-9e13823ac7ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.720271354745226 \n",
      "\n",
      "TLM Response:\n",
      " Customer Name: None  \n",
      "Customer Phone Number: None  \n",
      "Customer Email: None  \n",
      "Order Number: BB987654321   \n",
      "\n",
      "TLM Explanation:  The answer correctly extracts the order number from the conversation, which is BB987654321. However, it fails to provide any information regarding the customer's name, phone number, or email address, all of which are marked as None. The response is clear and straightforward, but it lacks completeness since it does not include all requested details. The extraction of the order number is accurate, but the absence of the other details makes the overall response less useful. Therefore, while the answer is partially correct, it does not fully meet the user's request for all specified details. Given these factors, I would rate the response as a 3, as it addresses part of the user's question but lacks confidence in its completeness and accuracy regarding the other details. \n",
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Customer Name: None  \n",
      "Customer Phone Number: None  \n",
      "Customer Email: None  \n",
      "Order Number: BB987654324  .\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", low_trust_extractions.extraction_score.values[0], \"\\n\")\n",
    "print(\"TLM Response:\\n\",low_trust_extractions.extraction.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", low_trust_extractions.extraction_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "393139a0-3c19-4ef0-bfa7-cd1f9dd10deb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: My order number is BB987654321. I received a Mixer Grinder with model number MG1234 instead of the Wet Grinder I ordered.\n",
      ".\n",
      ".\n",
      ".\n",
      "Agent: Thank you for providing that information, sir. That one is wrong, it is actually BB987654324. Let me check the availability of the Wet Grinder for you. Please hold on for a moment while I access the information.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(low_trust_extractions.conversation.values[0], [3,4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17959c4-9de6-41da-b590-e7ecccbade04",
   "metadata": {},
   "source": [
    "#### TLM Responses with Highest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "596cfe98-43bb-491b-a229-280d7d5b0d6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>extraction</th>\n",
       "      <th>extraction_score</th>\n",
       "      <th>extraction_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>CF85</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I recently purchased ...</td>\n",
       "      <td>Customer Name: John  \\nCustomer Phone Number: None  \\nCustomer Email: john.doe@email.com  \\nOrder Number: 123456</td>\n",
       "      <td>0.991019</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>BC81</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Tom, how may I assist you today?\\n\\nCustomer: Hi, I'm having trouble log...</td>\n",
       "      <td>Customer Name: John Doe  \\nCustomer Phone Number: 555-123-4567  \\nCustomer Email: johndoe@email.com  \\nOrder Number: None</td>\n",
       "      <td>0.990308</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>NI19</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Max. How may I assist you today?\\n\\nCustomer: Hi Max, I recently purchased a wr...</td>\n",
       "      <td>Customer Name: Jane Doe  \\nCustomer Phone Number: None  \\nCustomer Email: janedoe123@example.com  \\nOrder Number: BB987654321</td>\n",
       "      <td>0.988496</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "92  CF85   \n",
       "45  BC81   \n",
       "18  NI19   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "92  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I recently purchased ...   \n",
       "45  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Tom, how may I assist you today?\\n\\nCustomer: Hi, I'm having trouble log...   \n",
       "18  Agent: Thank you for calling BrownBox Customer Support. My name is Max. How may I assist you today?\\n\\nCustomer: Hi Max, I recently purchased a wr...   \n",
       "\n",
       "                                                                                                                         extraction  \\\n",
       "92               Customer Name: John  \\nCustomer Phone Number: None  \\nCustomer Email: john.doe@email.com  \\nOrder Number: 123456     \n",
       "45      Customer Name: John Doe  \\nCustomer Phone Number: 555-123-4567  \\nCustomer Email: johndoe@email.com  \\nOrder Number: None     \n",
       "18  Customer Name: Jane Doe  \\nCustomer Phone Number: None  \\nCustomer Email: janedoe123@example.com  \\nOrder Number: BB987654321     \n",
       "\n",
       "    extraction_score                                  extraction_expl  \n",
       "92          0.991019  Did not find a reason to doubt trustworthiness.  \n",
       "45          0.990308  Did not find a reason to doubt trustworthiness.  \n",
       "18          0.988496  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_trust_extractions = df_extr.sort_values(by=\"extraction_score\", ascending=False).head(3)\n",
    "high_trust_extractions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fcc98ce9-5845-4a22-8a25-f187685c0f8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLM Trust Score:  0.9910193319362357 \n",
      "\n",
      "TLM Response:\n",
      " Customer Name: John  \n",
      "Customer Phone Number: None  \n",
      "Customer Email: john.doe@email.com  \n",
      "Order Number: 123456   \n",
      "\n",
      "TLM Explanation:  Did not find a reason to doubt trustworthiness.\n"
     ]
    }
   ],
   "source": [
    "print(\"TLM Trust Score: \", high_trust_extractions.extraction_score.values[0], \"\\n\")\n",
    "print(\"TLM Response:\\n\",high_trust_extractions.extraction.values[0], \"\\n\")\n",
    "print(\"TLM Explanation: \", high_trust_extractions.extraction_expl.values[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a87c55-a93f-4479-9577-7b7a84233cdd",
   "metadata": {},
   "source": [
    "## Use Case: Classification, Auto-boost Accuracy, and Auto-labeling "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758ca868-316d-4fbc-8c9d-58d74ad77097",
   "metadata": {},
   "source": [
    "#### Question: Classify conversations into one of 6 issue areas:\n",
    "1. Warranty and Product Support\n",
    "2. Returns and Exchanges\n",
    "3. Shipping and Delivery\n",
    "4. Order and Payment\n",
    "5. Account Management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "07073bd8-76df-45b9-b8d1-fc00126d0022",
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = '''Please classify the following conversation between a customer and service agent\n",
    "into one of the following support categories: {}\n",
    "Please only respond with the support category and nothing else. You may only respond with one of 'Warranty and Product Support', 'Returns and Exchanges', 'Shipping and Delivery', 'Order and Payment', or 'Account Management'.\n",
    "Here is the conversation: {}'''\n",
    "category_descriptions = '''\n",
    "Warranty and Product Support: Handles warranty claims, product registration, support for service issues, and warranty term disputes.\n",
    "Returns and Exchanges: Covers returning or exchanging items, refund timelines, shipping issues, and non-returnable products.\n",
    "Shipping and Delivery: Focuses on delivery options, failed deliveries, shipping restrictions, and pickup or shipping problems.\n",
    "Order and Payment: Involves order placement, cancellations, pricing issues, payment options, and refund or invoice concerns.\n",
    "Account Management: Addresses login issues, account updates, verification processes, reactivation, and loyalty program questions.\n",
    "'''\n",
    "categories_prompts = [categories.format(category_descriptions, conversation) for conversation in df.conversation.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c3a97e3a-db4d-456f-96d6-8bb7ad630ca7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|██████████|\n"
     ]
    }
   ],
   "source": [
    "categories = tlm.prompt(categories_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "83504a6d-85e9-4d20-aca4-19a4e0166ccb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add results to df\n",
    "df_cat = df.copy()\n",
    "df_cat[\"category\"] = [entry['response'] for entry in categories]\n",
    "df_cat[\"category_score\"] = [entry['trustworthiness_score'] for entry in categories]\n",
    "df_cat[\"category_expl\"] = [entry['log']['explanation'] for entry in categories]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fa26a60-f092-4fe8-87c2-1d91fc6ee27d",
   "metadata": {},
   "source": [
    "#### TLM Responses with Lowest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8221023b-0154-497f-8af8-f2dc9f4b4b4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>category_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>KP59</td>\n",
       "      <td>Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\\n\\nAgent: Hello, thank you for calling Bro...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.479531</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>NQ67</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I placed an order for...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.488012</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>PR33</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.564650</td>\n",
       "      <td>This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "22  KP59   \n",
       "38  NQ67   \n",
       "51  PR33   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "22  Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\\n\\nAgent: Hello, thank you for calling Bro...   \n",
       "38  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I placed an order for...   \n",
       "51  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...   \n",
       "\n",
       "                 category  category_score  \\\n",
       "22  Shipping and Delivery        0.479531   \n",
       "38  Shipping and Delivery        0.488012   \n",
       "51  Returns and Exchanges        0.564650   \n",
       "\n",
       "                                                                                                                                            category_expl  \n",
       "22  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "38  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  \n",
       "51  This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that th...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_trust_categories = df_cat.sort_values(by=\"category_score\", ascending=True).head(3)\n",
    "low_trust_categories.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "384e7c75-c384-41a2-97b0-e9270c5abacb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shipping and Delivery\n"
     ]
    }
   ],
   "source": [
    "print(low_trust_categories.category.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a744411d-c9ea-4198-8d1a-458b07b933c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): \n",
      "Order and Payment.\n"
     ]
    }
   ],
   "source": [
    "print(low_trust_categories.category_expl.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "021fd469-3f0a-492c-b8f1-c968f495b3f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(low_trust_categories.conversation.values[0], [0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fad0008-c914-4e43-b4c9-c36257651fd2",
   "metadata": {},
   "source": [
    "#### TLM Responses with Highest Trustworthiness Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ae18a96a-b3b2-46c4-a3e8-d61a30307b2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>category_expl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>XZ51</td>\n",
       "      <td>Agent: Hello, thank you for contacting BrownBox customer support. How may I assist you today?\\n\\nCustomer: Hi, I am trying to reactivate my accoun...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>JN62</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex, I recently bought a pai...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>BL44</td>\n",
       "      <td>Agent: Hello, thank you for contacting BrownBox customer support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah, I'm trying ...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Did not find a reason to doubt trustworthiness.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "28  XZ51   \n",
       "9   JN62   \n",
       "52  BL44   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "28  Agent: Hello, thank you for contacting BrownBox customer support. How may I assist you today?\\n\\nCustomer: Hi, I am trying to reactivate my accoun...   \n",
       "9   Agent: Thank you for calling BrownBox Customer Support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex, I recently bought a pai...   \n",
       "52  Agent: Hello, thank you for contacting BrownBox customer support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah, I'm trying ...   \n",
       "\n",
       "              category  category_score  \\\n",
       "28  Account Management        0.999632   \n",
       "9   Account Management        0.999632   \n",
       "52  Account Management        0.999632   \n",
       "\n",
       "                                      category_expl  \n",
       "28  Did not find a reason to doubt trustworthiness.  \n",
       "9   Did not find a reason to doubt trustworthiness.  \n",
       "52  Did not find a reason to doubt trustworthiness.  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_trust_categories = df_cat.sort_values(by=\"category_score\", ascending=False).head(3)\n",
    "high_trust_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e6a97e2e-2894-4688-8dd6-a08d5fcfbe83",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Account Management\n"
     ]
    }
   ],
   "source": [
    "print(high_trust_categories.category.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "347ddb32-eb6a-4cf5-bfe5-c22f4aa1ac10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customer: Hi, I am trying to reactivate my account, but it's not working.\n"
     ]
    }
   ],
   "source": [
    "print_conversation(high_trust_categories.conversation.values[0], [1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef52a307-8e29-46f9-86b9-fdd6a63b43bc",
   "metadata": {},
   "source": [
    "## Accuracy Improvement\n",
    "How does TLM's Trustworthiness Score increase LLM accuracy?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d89cbb1-61e1-45bf-8281-0b712a65fabb",
   "metadata": {},
   "source": [
    "#### Let's revisit the category classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6e6a2a4f-5c2a-4171-83f6-106841a47579",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "      <td>0.999231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.999358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.999483</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "\n",
       "                                                                                                                                            conversation  \\\n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...   \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...   \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...   \n",
       "\n",
       "                       category  category_score  \n",
       "0  Warranty and Product Support        0.999231  \n",
       "1         Returns and Exchanges        0.999358  \n",
       "2         Shipping and Delivery        0.999483  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset = df_cat[[\"id\", \"conversation\", \"category\", \"category_score\"]]\n",
    "subset.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbbd09d3-f2b1-49c4-8081-79ee4a09a0a1",
   "metadata": {},
   "source": [
    "#### Import and add ground truth category labels\n",
    "In this example, we have ground truth labels for the correct category for all of the conversations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "6a954358-b367-4446-a086-4f9b401fb903",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>ground_truth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RU91</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "      <td>0.999231</td>\n",
       "      <td>Warranty and Product Support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XL37</td>\n",
       "      <td>Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.999358</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OJ95</td>\n",
       "      <td>Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.999483</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id  \\\n",
       "0  RU91   \n",
       "1  XL37   \n",
       "2  OJ95   \n",
       "\n",
       "                                                                                                                                            conversation  \\\n",
       "0  Agent: Thank you for calling BrownBox Customer Support. My name is Rachel. How may I assist you today?\\n\\nCustomer: Hi Rachel, I recently purchase...   \n",
       "1  Customer: Hello, I have an issue with my recent purchase from BrownBox.\\n\\nAgent: Good afternoon! Thank you for contacting BrownBox customer suppo...   \n",
       "2  Agent: Hello, thank you for calling BrownBox Customer Support. My name is Emily, how may I assist you today?\\n\\nCustomer: Hi, I recently purchased...   \n",
       "\n",
       "                       category  category_score                  ground_truth  \n",
       "0  Warranty and Product Support        0.999231  Warranty and Product Support  \n",
       "1         Returns and Exchanges        0.999358         Returns and Exchanges  \n",
       "2         Shipping and Delivery        0.999483         Shipping and Delivery  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_ground_truth = pd.read_csv(\"https://raw.githubusercontent.com/cleanlab/cleanlab-tools/refs/heads/main/TLM-Demo-Notebook/customer-service-chat-categories.csv\")\n",
    "category_df = pd.merge(subset, category_ground_truth, on='id', how='left')\n",
    "category_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1faaee7f-6c10-49cd-952f-073fa8e02e06",
   "metadata": {},
   "source": [
    "#### Compute Baseline Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bcb6a354-b125-43f6-862f-ca6decb8ccd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base Accuracy:  93.6%\n"
     ]
    }
   ],
   "source": [
    "base_acc = accuracy_score(category_df[\"category\"], category_df[\"ground_truth\"])\n",
    "print(\"Base Accuracy: \", f\"{base_acc:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00216641",
   "metadata": {},
   "source": [
    "#### Auto-boost Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "b59fcfef",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Querying TLM... 100%|██████████|"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Preset Accuracy:  94.5%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tlm = TLM(quality_preset=\"best\")\n",
    "\n",
    "# Generate answers and trustworthiness scores using best quality preset\n",
    "best_preset_categories = tlm.prompt(categories_prompts)\n",
    "\n",
    "# Add results and compute accuracy\n",
    "category_df[\"best_preset_category\"] = [entry['response'] for entry in best_preset_categories]\n",
    "category_df[\"best_preset_category_score\"] = [entry['trustworthiness_score'] for entry in best_preset_categories]\n",
    "best_preset_acc = accuracy_score(category_df[\"best_preset_category\"], category_df[\"ground_truth\"])\n",
    "print(\"Best Preset Accuracy: \", f\"{best_preset_acc:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee303e3-b8cb-4d83-b712-e7af98907935",
   "metadata": {},
   "source": [
    "#### Compute Accuracy at Various TLM Score Thresholds\n",
    "The TLM trust score threshold leads to higher classification accuracy as the threshold becomes more stringent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "38fd073e-f3e5-411b-9116-9a71187036b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of all predictions:  93.6%\n",
      "Accuracy of predictions with trust score >70%:  97.2%\n",
      "Accuracy of predictions with trust score >80%:  98.1%\n",
      "Accuracy of predictions with trust score >90%:  99.0%\n"
     ]
    }
   ],
   "source": [
    "category_df_70 = category_df[category_df.category_score > 0.7]\n",
    "category_df_80 = category_df[category_df.category_score > 0.8]\n",
    "category_df_90 = category_df[category_df.category_score > 0.9]\n",
    "acc_70 = accuracy_score(category_df_70[\"category\"], category_df_70[\"ground_truth\"])\n",
    "acc_80 = accuracy_score(category_df_80[\"category\"], category_df_80[\"ground_truth\"])\n",
    "acc_90 = accuracy_score(category_df_90[\"category\"], category_df_90[\"ground_truth\"])\n",
    "print(\"Accuracy of all predictions: \", f\"{base_acc:.1%}\")\n",
    "print(\"Accuracy of predictions with trust score >70%: \", f\"{acc_70:.1%}\")\n",
    "print(\"Accuracy of predictions with trust score >80%: \", f\"{acc_80:.1%}\")\n",
    "print(\"Accuracy of predictions with trust score >90%: \", f\"{acc_90:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9cdbf19",
   "metadata": {},
   "source": [
    "## Auto-labeling\n",
    "Use TLM to label your dataset faster and more accurately."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "948ec151",
   "metadata": {},
   "source": [
    "#### TLM Responses with High Trustworthiness Score\n",
    "These responses have high trustworthiness scores, indicating that they are likely to be correct and can be labeled automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a54079fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>best_preset_category</th>\n",
       "      <th>best_preset_category_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>XZ51</td>\n",
       "      <td>Agent: Hello, thank you for contacting BrownBox customer support. How may I assist you today?\\n\\nCustomer: Hi, I am trying to reactivate my accoun...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>JN62</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex, I recently bought a pai...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>BL44</td>\n",
       "      <td>Agent: Hello, thank you for contacting BrownBox customer support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah, I'm trying ...</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>Account Management</td>\n",
       "      <td>0.999632</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "28  XZ51   \n",
       "9   JN62   \n",
       "52  BL44   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "28  Agent: Hello, thank you for contacting BrownBox customer support. How may I assist you today?\\n\\nCustomer: Hi, I am trying to reactivate my accoun...   \n",
       "9   Agent: Thank you for calling BrownBox Customer Support. My name is Alex. How can I assist you today?\\n\\nCustomer: Hi Alex, I recently bought a pai...   \n",
       "52  Agent: Hello, thank you for contacting BrownBox customer support. My name is Sarah. How can I assist you today?\\n\\nCustomer: Hi Sarah, I'm trying ...   \n",
       "\n",
       "              category  category_score        ground_truth  \\\n",
       "28  Account Management        0.999632  Account Management   \n",
       "9   Account Management        0.999632  Account Management   \n",
       "52  Account Management        0.999632  Account Management   \n",
       "\n",
       "   best_preset_category  best_preset_category_score  \n",
       "28   Account Management                    0.999632  \n",
       "9    Account Management                    0.999632  \n",
       "52   Account Management                    0.999632  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_df.sort_values(by=\"category_score\", ascending=False).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "484cb468",
   "metadata": {},
   "source": [
    "#### TLM Responses with High Trustworthiness Score\n",
    "These responses have low trustworthiness scores, indicating that they are likely to be incorrect and should be reviewed manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3b545411",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>conversation</th>\n",
       "      <th>category</th>\n",
       "      <th>category_score</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>best_preset_category</th>\n",
       "      <th>best_preset_category_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>KP59</td>\n",
       "      <td>Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\\n\\nAgent: Hello, thank you for calling Bro...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.479531</td>\n",
       "      <td>Order and Payment</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.547593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>NQ67</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I placed an order for...</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.488012</td>\n",
       "      <td>Order and Payment</td>\n",
       "      <td>Shipping and Delivery</td>\n",
       "      <td>0.549392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>PR33</td>\n",
       "      <td>Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.564650</td>\n",
       "      <td>Order and Payment</td>\n",
       "      <td>Returns and Exchanges</td>\n",
       "      <td>0.781469</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  \\\n",
       "22  KP59   \n",
       "38  NQ67   \n",
       "51  PR33   \n",
       "\n",
       "                                                                                                                                             conversation  \\\n",
       "22  Customer: Hi, I am calling to check the status of my order for a pram/stroller that I placed last week.\\n\\nAgent: Hello, thank you for calling Bro...   \n",
       "38  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I placed an order for...   \n",
       "51  Agent: Thank you for calling BrownBox Customer Support. My name is Sarah. How may I assist you today?\\n\\nCustomer: Hi Sarah, I am trying to club o...   \n",
       "\n",
       "                 category  category_score       ground_truth  \\\n",
       "22  Shipping and Delivery        0.479531  Order and Payment   \n",
       "38  Shipping and Delivery        0.488012  Order and Payment   \n",
       "51  Returns and Exchanges        0.564650  Order and Payment   \n",
       "\n",
       "     best_preset_category  best_preset_category_score  \n",
       "22  Shipping and Delivery                    0.547593  \n",
       "38  Shipping and Delivery                    0.549392  \n",
       "51  Returns and Exchanges                    0.781469  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_df.sort_values(by=\"category_score\", ascending=True).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80dd698f",
   "metadata": {},
   "source": [
    "#### You only need to manually label a small percentage of your dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9cbb5195",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Examples with trust score ≤ 80%: 5 (4.5% of the dataset)\n"
     ]
    }
   ],
   "source": [
    "# Number of datapoints below 80% trustworthiness score\n",
    "count = len(category_df[category_df.category_score <= 0.8])\n",
    "total = len(category_df)\n",
    "percentage = count/total * 100\n",
    "print(f\"Examples with trust score ≤ 80%: {count} ({percentage:.1f}% of the dataset)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e524914",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "- [Advanced Usage notebook](https://help.cleanlab.ai/tlm/tutorials/tlm_advanced/) which includes:\n",
    "  - Generating explanations of low trustworthiness scores\n",
    "  - Running TLM over large datasets\n",
    "  - Using quality presets to control latency/cost vs. response accuracy and trustworthiness score reliability\n",
    "  - Reducing latency/cost without sacrificing response-quality via a `TLMLite` option that allows different models for producing the response vs. scoring its trustworthiness.\n",
    "- Add your own [custom evaluation](https://help.cleanlab.ai/tlm/tutorials/tlm_custom_eval/) metrics to TLM and calibrate trustworthiness scores to human quality annotations\n",
    "- TLM API [reference](https://help.cleanlab.ai/tlm/python/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe25332d",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}