Portkey AI Go SDK

Go SDK for the Portkey AI Gateway - A blazing fast AI Gateway for routing to 200+ LLMs with a unified API.

This SDK is designed for self-hosted deployments of the Portkey Gateway. You manage your own API keys and infrastructure.

Features

• 🚀 Universal LLM Access: Connect to 200+ LLM providers through a single API
• 🔄 Automatic Retries: Built-in exponential backoff for resilient requests
• 🌊 Streaming Support: Full SSE streaming support for real-time responses
• 🎯 Smart Routing: Fallback and load balancing strategies
• 📊 Observability: Complete request tracing and metadata support
• 💾 Caching: Semantic and simple caching for cost optimization
• 🛡️ Type-Safe: Comprehensive type definitions with full IDE support
• 🎨 Fluent API: Intuitive configuration builder for complex routing

Installation

go get github.com/youssefsiam38/portkeyai

Quick Start

Prerequisites

1. Deploy the Portkey Gateway (self-hosted)
2. Get your LLM provider API keys (e.g., OpenAI, Anthropic, etc.)

Basic Chat Completion

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/youssefsiam38/portkeyai"
)

func main() {
    // Create a client pointing to your self-hosted gateway
    client, err := portkeyai.NewClient(
        "",  // No Portkey API key needed for self-hosted
        portkeyai.WithBaseURL("http://localhost:8787"),  // Your gateway URL
        portkeyai.WithProvider("openai"),
        portkeyai.WithAuthorization("Bearer " + os.Getenv("OPENAI_API_KEY")),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Create a chat completion request
    req := portkeyai.NewChatCompletionRequest(
        "gpt-4",
        portkeyai.NewSystemMessage("You are a helpful assistant."),
        portkeyai.NewUserMessage("What is the capital of France?"),
    )

    // Send the request
    resp, err := client.Chat.Create(context.Background(), req, nil)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(resp.Choices[0].Message.Content)
}

Streaming Chat Completion

stream, err := client.Chat.CreateStream(ctx, req, nil)
if err != nil {
    log.Fatal(err)
}
defer stream.Close()

for {
    chunk, err := stream.Recv()
    if err == io.EOF {
        break
    }
    if err != nil {
        log.Fatal(err)
    }

    if len(chunk.Choices) > 0 {
        fmt.Print(chunk.Choices[0].Delta.Content)
    }
}

Authentication

For self-hosted Portkey Gateway, you manage your own API keys:

Direct Provider Authentication (Recommended for Self-Hosted)

Pass your provider API key directly to the gateway:

client, err := portkeyai.NewClient(
    "",  // Empty for self-hosted (or omit Portkey API key validation)
    portkeyai.WithBaseURL("http://localhost:8787"),  // Your gateway URL
    portkeyai.WithProvider("openai"),
    portkeyai.WithAuthorization("Bearer sk-..."),
)

Per-Request Provider Override

You can also override the provider per request:

// Create client without default provider
client, err := portkeyai.NewClient(
    "",
    portkeyai.WithBaseURL("http://localhost:8787"),
)

// Specify provider in request options
opts := &portkeyai.RequestOptions{
    Provider:      "openai",
    Authorization: "Bearer sk-...",
}

resp, err := client.Chat.Create(ctx, req, opts)

Using Gateway Configuration

For advanced routing (fallback, load balancing), use config:

// Build a fallback config with your API keys
config := portkeyai.NewConfig().
    WithFallbackStrategy().
    AddTarget(portkeyai.NewTarget().
        WithProvider("openai").
        WithAPIKey("sk-openai-key").
        Build()).
    AddTarget(portkeyai.NewTarget().
        WithProvider("anthropic").
        WithAPIKey("sk-ant-key").
        Build())

configJSON, _ := config.Build()

client, err := portkeyai.NewClient(
    "",
    portkeyai.WithBaseURL("http://localhost:8787"),
    portkeyai.WithConfig(configJSON),
)

For Portkey Hosted Platform (Optional)

If using Portkey's hosted platform instead of self-hosted, virtual keys are supported:

client, err := portkeyai.NewClient(
    "your-portkey-api-key",
    // BaseURL defaults to https://api.portkey.ai/v1
    portkeyai.WithVirtualKey("your-virtual-key"),
)

Note: Virtual keys are a feature of Portkey's hosted platform. For self-hosted gateways, use direct provider authentication.

Core Features

Chat Completions

Basic Request

req := portkeyai.NewChatCompletionRequest(
    "gpt-4",
    portkeyai.NewSystemMessage("You are a helpful assistant."),
    portkeyai.NewUserMessage("Hello!"),
)

resp, err := client.Chat.Create(ctx, req, nil)

With Options

req := portkeyai.ChatCompletionRequest{
    Model: "gpt-4",
    Messages: []portkeyai.ChatCompletionMessage{
        portkeyai.NewUserMessage("Write a poem"),
    },
    Temperature: float64Ptr(0.7),
    MaxTokens:   intPtr(150),
    TopP:        float64Ptr(0.9),
}

Function Calling / Tools

tool := portkeyai.Tool{
    Type: "function",
    Function: portkeyai.Function{
        Name:        "get_weather",
        Description: "Get the current weather",
        Parameters: map[string]interface{}{
            "type": "object",
            "properties": map[string]interface{}{
                "location": map[string]interface{}{
                    "type": "string",
                    "description": "City name",
                },
            },
            "required": []string{"location"},
        },
    },
}

req := portkeyai.ChatCompletionRequest{
    Model: "gpt-4",
    Messages: []portkeyai.ChatCompletionMessage{
        portkeyai.NewUserMessage("What's the weather in Paris?"),
    },
    Tools: []portkeyai.Tool{tool},
}

Vision / Images

msg := portkeyai.NewUserMessageWithImages(
    "What's in this image?",
    "https://example.com/image.jpg",
    "https://example.com/image2.jpg",
)

req := portkeyai.NewChatCompletionRequest("gpt-4-vision-preview", msg)

Embeddings

Single Text

req := portkeyai.NewEmbeddingRequest(
    "text-embedding-ada-002",
    "The quick brown fox jumps over the lazy dog",
)

resp, err := client.Embeddings.Create(ctx, req, nil)
fmt.Printf("Embedding: %v\n", resp.Data[0].Embedding)

Batch Embeddings

texts := []string{
    "First text to embed",
    "Second text to embed",
    "Third text to embed",
}

req := portkeyai.NewEmbeddingRequest("text-embedding-ada-002", texts)
resp, err := client.Embeddings.Create(ctx, req, nil)

Custom Dimensions

dimensions := 512
req := portkeyai.NewEmbeddingRequestWithOptions(
    "text-embedding-3-small",
    "Text to embed",
    "float",
    &dimensions,
)

Advanced Routing & Resilience

Fallback Configuration

Automatically fallback to alternative providers when the primary fails:

// Simple fallback
config, err := portkeyai.NewFallbackConfig(
    "openai-key",
    "anthropic-key",
    "google-key",
).Build()

client, err := portkeyai.NewClient(
    "portkey-api-key",
    portkeyai.WithConfig(config),
)

With Model Overrides

config := portkeyai.NewConfig().
    WithFallbackStrategy().
    WithDefaultRetry().
    AddTarget(portkeyai.NewTarget().
        WithVirtualKey("openai-key").
        WithModelOverride("gpt-4").
        Build()).
    AddTarget(portkeyai.NewTarget().
        WithVirtualKey("anthropic-key").
        WithModelOverride("claude-3-opus-20240229").
        Build())

Load Balancing

Distribute requests across multiple providers:

// Equal distribution
config, err := portkeyai.NewLoadBalanceConfig(
    "provider-1",
    "provider-2",
    "provider-3",
).Build()

Weighted Distribution

// 70% cheap, 30% premium
config, err := portkeyai.NewLoadBalanceConfigWeighted(map[string]float64{
    "cheap-provider":   0.7,
    "premium-provider": 0.3,
}).Build()

Retry Configuration

// Custom retry settings
client, err := portkeyai.NewClient(
    "portkey-api-key",
    portkeyai.WithVirtualKey("virtual-key"),
    portkeyai.WithMaxRetries(5),
    portkeyai.WithRetryableStatusCodes([]int{429, 500, 502, 503, 504}),
)

Config-Level Retries

config := portkeyai.NewConfig().
    WithRetry(5, []int{429, 500, 502, 503}).
    AddVirtualKeyTarget("provider-key")

Observability & Tracing

Trace IDs

Track requests across your application:

opts := &portkeyai.RequestOptions{
    TraceID:  "user-session-12345",
    SpanName: "question-answering",
}

resp, err := client.Chat.Create(ctx, req, opts)

Hierarchical Tracing

// Parent request
parentOpts := &portkeyai.RequestOptions{
    TraceID:  "workflow-123",
    SpanID:   "parent-span",
    SpanName: "Main Workflow",
}

// Child request
childOpts := &portkeyai.RequestOptions{
    TraceID:      "workflow-123",
    SpanID:       "child-span",
    ParentSpanID: "parent-span",
    SpanName:     "Sub-task",
}

Metadata

Attach custom metadata to requests:

opts := &portkeyai.RequestOptions{
    Metadata: map[string]interface{}{
        "user_id":      "user_123",
        "environment":  "production",
        "feature_flag": "new_model_v2",
    },
}

Caching

Cache Namespace

Partition your cache by custom namespaces:

opts := &portkeyai.RequestOptions{
    CacheNamespace: "math-questions",
}

// First request - cached
resp1, _ := client.Chat.Create(ctx, req, opts)

// Second request - from cache
resp2, _ := client.Chat.Create(ctx, req, opts)

Force Cache Refresh

opts := &portkeyai.RequestOptions{
    CacheNamespace:    "questions",
    CacheForceRefresh: true,
}

Client Configuration

Complete Options

client, err := portkeyai.NewClient(
    "api-key",
    portkeyai.WithVirtualKey("virtual-key"),
    portkeyai.WithBaseURL("https://custom.gateway.ai"),
    portkeyai.WithConfig("pc-config-id"),
    portkeyai.WithMaxRetries(5),
    portkeyai.WithRetryDelay(2 * time.Second),
    portkeyai.WithMaxRetryDelay(60 * time.Second),
    portkeyai.WithHTTPClient(customHTTPClient),
    portkeyai.WithDefaultRequestOptions(&portkeyai.RequestOptions{
        Metadata: map[string]interface{}{
            "app": "my-app",
        },
    }),
)

Runtime Configuration Updates

client.SetVirtualKey("new-virtual-key")
client.SetConfig("new-config")
client.SetProvider("anthropic")
client.SetAuthorization("Bearer new-token")

Streaming Utilities

Stream to Completion

Convert a stream to a complete response:

stream, err := client.Chat.CreateStream(ctx, req, nil)
if err != nil {
    log.Fatal(err)
}

// Consume entire stream and get complete response
resp, err := portkeyai.StreamToCompletion(ctx, stream)
if err != nil {
    log.Fatal(err)
}

fmt.Println(resp.Choices[0].Message.Content)

Chat Completion Accumulator

stream, _ := client.Chat.CreateStream(ctx, req, nil)
defer stream.Close()

acc := portkeyai.NewChatCompletionAccumulator()

for {
    chunk, err := stream.Recv()
    if err == io.EOF {
        break
    }
    if err != nil {
        log.Fatal(err)
    }

    acc.AddChunk(chunk)
    fmt.Print(chunk.Choices[0].Delta.Content)
}

// Get complete message
message := acc.ToMessage()
fmt.Printf("\n\nFull content: %s\n", acc.GetContent())
fmt.Printf("Finish reason: %s\n", acc.GetFinishReason())

Error Handling

The SDK provides type-safe error handling:

resp, err := client.Chat.Create(ctx, req, nil)
if err != nil {
    switch e := err.(type) {
    case *portkeyai.APIError:
        fmt.Printf("API Error %d: %s\n", e.StatusCode, e.Message)
        if e.IsRetryable() {
            // Handle retryable error
        }
    case *portkeyai.NetworkError:
        fmt.Printf("Network Error: %v\n", e.Err)
    case *portkeyai.ValidationError:
        fmt.Printf("Validation Error: %s - %s\n", e.Field, e.Message)
    default:
        fmt.Printf("Unknown error: %v\n", err)
    }
    return
}

Opinionated Defaults

The SDK comes with production-ready defaults:

• ✅ Automatic Retries: 3 attempts with exponential backoff
• ✅ Retryable Status Codes: 429, 500, 502, 503, 504
• ✅ No Timeout: Supports long-running streaming requests
• ✅ Connection Pooling: Reuses HTTP connections for performance
• ✅ Exponential Backoff: With jitter to prevent thundering herd

Customizing Defaults

client, err := portkeyai.NewClient(
    "api-key",
    portkeyai.WithMaxRetries(0), // Disable retries
    portkeyai.WithRetryDelay(500 * time.Millisecond),
    portkeyai.WithMaxRetryDelay(10 * time.Second),
)

Examples

Comprehensive examples are available in the /examples directory:

• examples/chat/basic - Basic chat completion
• examples/chat/streaming - Streaming responses
• examples/chat/with_tools - Function calling
• examples/embeddings/basic_example - Text embeddings
• examples/config/fallback - Fallback configuration
• examples/config/loadbalance - Load balancing
• examples/config/retry - Retry strategies
• examples/advanced/custom_headers - Tracing & metadata

Run examples with:

go run examples/chat/basic/main.go
go run examples/chat/streaming/main.go

Testing

Run the test suite:

go test ./tests/...

With coverage:

go test -cover ./tests/...

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Resources

• Portkey Documentation
• API Reference
• Portkey Dashboard
• GitHub Repository

Support

• Documentation
• Discord Community
• GitHub Issues

Changelog

v0.1.0 (Initial Release)

• ✅ Chat completions API (streaming & non-streaming)
• ✅ Embeddings API
• ✅ Automatic retries with exponential backoff
• ✅ Fluent configuration builder
• ✅ Gateway routing (fallback & load balancing)
• ✅ Full observability support (traces, spans, metadata)
• ✅ Caching with namespaces
• ✅ Type-safe error handling
• ✅ Comprehensive examples and documentation

---

Built with ❤️ by the Portkey community