Skip to content

tracking.md

Latest commit

 

History

History
443 lines (339 loc) · 13.7 KB

tracking.md

File metadata and controls

443 lines (339 loc) · 13.7 KB

Tracking & Performance

Introduction

Prompt Deck includes an optional database tracking system that logs prompt versions and execution data. This enables:

  • Performance monitoring — Track token usage, latency, and cost per prompt version.
  • A/B testing — Compare performance metrics across different prompt versions.
  • Audit trails — Record what was sent to and received from AI providers.
  • Cost analysis — Monitor API spending per prompt, model, and provider.
  • User feedback — Attach ratings and comments to individual executions.

Tracking is entirely optional. Prompt Deck works fully without it.

Setup

Enable Tracking

Set the tracking configuration in config/prompt-deck.php:

'tracking' => [
    'enabled'    => env('PROMPTDECK_TRACKING_ENABLED', true),
    'connection' => env('PROMPTDECK_DB_CONNECTION'),
],

Or via environment variable:

PROMPTDECK_TRACKING_ENABLED=true

By default, tracking is disabled when APP_DEBUG=true and enabled in production.

Publish Migrations

Publish and run the migrations to create the required tables:

php artisan vendor:publish --tag=prompt-deck-migrations
php artisan migrate

Database Connection

By default, tracking uses your application's default database connection. To store tracking data on a separate database:

PROMPTDECK_DB_CONNECTION=analytics

The connection name must match a connection defined in config/database.php.

Database Schema

prompt_versions Table

Stores prompt version records and tracks which version is active.

Column Type Nullable Description
id bigint No Auto-incrementing primary key.
name string No The prompt name (e.g. order-summary).
version unsigned int No The version number (e.g. 1, 2, 3).
system_prompt text Yes The system prompt content.
user_prompt text No The user prompt content.
metadata json Yes Version metadata as JSON.
is_active boolean No Whether this version is the active version. Default: false.
created_at timestamp Yes Creation timestamp.
updated_at timestamp Yes Last update timestamp.

Indexes:

  • Unique index on (name, version) — ensures no duplicate versions per prompt.
  • Index on (name, is_active) — fast lookups for the active version.

prompt_executions Table

Logs individual prompt executions with performance data.

Column Type Nullable Description
id bigint No Auto-incrementing primary key.
prompt_name string No The prompt name.
prompt_version unsigned int No The prompt version used.
input json Yes The input sent to the AI provider.
output text Yes The response received from the AI provider.
tokens unsigned int Yes Total token usage (input + output).
latency_ms unsigned int Yes Round-trip time in milliseconds.
cost decimal(8,6) Yes Estimated cost in your currency.
model string Yes The AI model used (e.g. gpt-4o).
provider string Yes The AI provider (e.g. openai, anthropic).
feedback json Yes User feedback (ratings, comments, etc.).
created_at timestamp Yes Execution timestamp.

Indexes:

  • Index on (prompt_name, prompt_version, created_at) — fast filtering by prompt and time range.

Note The prompt_executions table does not have an updated_at column. Records are insert-only.

Recording Executions

Manual Tracking

Use the PromptDeck::track() method to record an execution manually:

use Veeqtoh\PromptDeck\Facades\PromptDeck;

PromptDeck::track('order-summary', 2, [
    'input'    => ['message' => 'Summarise order #1234'],
    'output'   => 'Your order contains 3 items totalling $150.00...',
    'tokens'   => 150,
    'latency'  => 234.5,
    'cost'     => 0.002,
    'model'    => 'gpt-4o',
    'provider' => 'openai',
    'feedback' => ['rating' => 5, 'comment' => 'Accurate summary'],
]);

All fields in the data array are optional. You can track as little or as much as you need:

// Minimal tracking — just record that the prompt was used
PromptDeck::track('order-summary', 2, []);

// Track only tokens and cost
PromptDeck::track('order-summary', 2, [
    'tokens' => 150,
    'cost'   => 0.002,
]);

If tracking is disabled in configuration, the track() method is a safe no-op — you can call it without checking configuration first.

Automatic Tracking via Middleware

When using the Laravel AI SDK integration, the TrackPromptMiddleware handles tracking automatically. See AI SDK — Performance Tracking Middleware for setup details.

Tracked Fields

Field Type Description
input array|null The input data sent to the AI provider. Stored as JSON.
output string|null The text response from the AI provider.
tokens int|null Total token count (input + output tokens).
latency float|null Round-trip time in milliseconds. Stored as latency_ms.
cost float|null Estimated cost. Stored with up to 6 decimal places.
model string|null AI model identifier (e.g. gpt-4o, claude-3-sonnet).
provider string|null AI provider name (e.g. openai, anthropic).
feedback array|null Arbitrary feedback data. Stored as JSON.

Eloquent Models

Prompt Deck provides two Eloquent models for interacting with tracking data.

PromptVersion

Veeqtoh\PromptDeck\Models\PromptVersion

Represents a prompt version record in the prompt_versions table.

use Veeqtoh\PromptDeck\Models\PromptVersion;

// Find the active version
$active = PromptVersion::where('name', 'order-summary')
    ->where('is_active', true)
    ->first();

// Get all versions for a prompt
$versions = PromptVersion::where('name', 'order-summary')
    ->orderBy('version')
    ->get();

Fillable attributes: name, version, system_prompt, user_prompt, metadata, is_active

Casts:

Attribute Cast
version integer
is_active boolean
metadata array

PromptExecution

Veeqtoh\PromptDeck\Models\PromptExecution

Represents an execution record in the prompt_executions table.

use Veeqtoh\PromptDeck\Models\PromptExecution;

// Get recent executions
$recent = PromptExecution::where('prompt_name', 'order-summary')
    ->latest()
    ->limit(100)
    ->get();

// Get executions for a specific version
$v2Executions = PromptExecution::where('prompt_name', 'order-summary')
    ->where('prompt_version', 2)
    ->get();

Fillable attributes: prompt_name, prompt_version, input, output, tokens, latency_ms, cost, model, provider, feedback

Casts:

Attribute Cast
prompt_version integer
tokens integer
latency_ms integer
cost decimal:6
input array
feedback array

Note PromptExecution sets UPDATED_AT = null since execution records are insert-only and never updated.

Factories

Both models include factories for testing.

PromptVersionFactory

use Veeqtoh\PromptDeck\Models\PromptVersion;

// Create a basic version
$version = PromptVersion::factory()->create();

// Create an active version
$version = PromptVersion::factory()->active()->create();

// Create with a specific name and version
$version = PromptVersion::factory()
    ->named('order-summary')
    ->version(2)
    ->active()
    ->create();

Available states:

Method Description
active() Mark as the active version (is_active = true).
version(int $v) Set a specific version number.
named(string $name) Set a specific prompt name.

PromptExecutionFactory

use Veeqtoh\PromptDeck\Models\PromptExecution;

// Create a basic execution
$execution = PromptExecution::factory()->create();

// Create with feedback
$execution = PromptExecution::factory()
    ->withFeedback(['rating' => 5, 'comment' => 'Great'])
    ->create();

// Create a minimal execution (only required fields)
$execution = PromptExecution::factory()->minimal()->create();

// Create for a specific prompt
$execution = PromptExecution::factory()
    ->forPrompt('order-summary', 2)
    ->create();

Available states:

Method Description
withFeedback(array $feedback = []) Include user feedback. Generates random feedback if no argument given.
minimal() Set all optional fields to null.
forPrompt(string $name, int $version) Set a specific prompt name and version.

Querying Execution Data

Basic Queries

use Veeqtoh\PromptDeck\Models\PromptExecution;

// Total executions for a prompt
$count = PromptExecution::where('prompt_name', 'order-summary')->count();

// Most recent execution
$latest = PromptExecution::where('prompt_name', 'order-summary')
    ->latest()
    ->first();

// Executions in the last 24 hours
$recent = PromptExecution::where('prompt_name', 'order-summary')
    ->where('created_at', '>=', now()->subDay())
    ->get();

Performance Analysis

// Average latency for a prompt version
$avgLatency = PromptExecution::where('prompt_name', 'order-summary')
    ->where('prompt_version', 2)
    ->avg('latency_ms');

// Average token usage
$avgTokens = PromptExecution::where('prompt_name', 'order-summary')
    ->where('prompt_version', 2)
    ->avg('tokens');

// 95th percentile latency (approximate)
$p95 = PromptExecution::where('prompt_name', 'order-summary')
    ->orderBy('latency_ms')
    ->limit(1)
    ->offset((int) (PromptExecution::where('prompt_name', 'order-summary')->count() * 0.95))
    ->value('latency_ms');

A/B Testing

Compare metrics across prompt versions:

// Compare average latency between v1 and v2
$comparison = PromptExecution::where('prompt_name', 'order-summary')
    ->whereIn('prompt_version', [1, 2])
    ->groupBy('prompt_version')
    ->selectRaw('prompt_version, AVG(latency_ms) as avg_latency, AVG(tokens) as avg_tokens, COUNT(*) as executions')
    ->get();

// Compare feedback ratings
$ratings = PromptExecution::where('prompt_name', 'order-summary')
    ->whereNotNull('feedback')
    ->whereIn('prompt_version', [1, 2])
    ->groupBy('prompt_version')
    ->selectRaw("prompt_version, AVG(JSON_EXTRACT(feedback, '$.rating')) as avg_rating")
    ->get();

Cost Analysis

// Total cost for a prompt
$totalCost = PromptExecution::where('prompt_name', 'order-summary')
    ->sum('cost');

// Cost by model
$costByModel = PromptExecution::where('prompt_name', 'order-summary')
    ->groupBy('model')
    ->selectRaw('model, SUM(cost) as total_cost, COUNT(*) as executions')
    ->get();

// Daily cost trend
$dailyCost = PromptExecution::where('prompt_name', 'order-summary')
    ->where('created_at', '>=', now()->subDays(30))
    ->groupByRaw('DATE(created_at)')
    ->selectRaw('DATE(created_at) as date, SUM(cost) as daily_cost')
    ->orderBy('date')
    ->get();

Version Management via Database

When tracking is enabled, version activation is managed through the prompt_versions table instead of metadata.json files. This provides a centralised, queryable record of version history.

The PromptDeck::activate() method and prompt:activate command automatically use the appropriate storage (database or file) based on your tracking configuration.

// Activate programmatically
PromptDeck::activate('order-summary', 2);

// Query active versions
$activeVersions = PromptVersion::where('is_active', true)->get();

// Version history
$history = PromptVersion::where('name', 'order-summary')
    ->orderBy('version')
    ->get()
    ->map(fn ($v) => [
        'version'  => $v->version,
        'active'   => $v->is_active,
        'metadata' => $v->metadata,
    ]);