Overview

Pay-i Instrumentation is the foundation of effective GenAI management, providing comprehensive visibility into your AI system's usage, performance, costs, and business value. Instrumentation is what enables you to track, monitor, analyze, optimize, and measure the success of your GenAI operations through quantifiable metrics and KPIs.

This guide introduces the core instrumentation capabilities of Pay-i and directs you to specialized documentation for implementation details.

What is Pay-i Instrumentation?

Instrumentation is the process of adding monitoring and tracking code to your application that captures data about your GenAI API calls and connects them to business outcomes. Pay-i's instrumentation capabilities allow you to:

• Track GenAI usage and costs across all major providers (OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, etc.)
• Associate business context with technical API calls
• Monitor performance metrics like latency and token usage
• Apply budget controls to manage spending
• Measure business value and ROI through custom KPIs and success metrics
• Evaluate the effectiveness of different GenAI implementations
• Track success metrics that align with business objectives
• Analyze patterns to optimize AI usage for both technical efficiency and business outcomes

Pay-i offers two levels of instrumentation to meet different needs:

Auto-Instrumentation

Auto-Instrumentation provides the fastest path to tracking your GenAI usage with minimal code changes. By simply initializing Pay-i instrumentation, you get immediate visibility into:

• Comprehensive usage metrics across providers and models
• Detailed cost tracking and analysis
• Performance data (latency, token usage, etc.)
• Reliability metrics and error tracking
• Foundational metrics needed for ROI calculations

Auto-instrumentation is the recommended starting point for all Pay-i users, as it provides valuable insights with almost no implementation effort.

from payi.lib.instrument import payi_instrument

# One line to enable tracking across all supported providers
payi_instrument()

Learn more about Auto-Instrumentation →

Custom Instrumentation

Custom Instrumentation builds on auto-instrumentation by adding business context and value metrics to your API calls. This enhanced level of tracking connects technical usage to business outcomes through custom annotations and KPI tracking.

Custom instrumentation enables you to:

• Associate API calls with specific business use cases
• Track usage by individual users or customer segments
• Apply granular spending limits and controls
• Organize requests with meaningful tags for analysis
• Measure business ROI through custom success metrics
• Tie GenAI activities to specific KPIs and value drivers
• Track success rates and quality metrics for AI-generated content
• Quantify business value through custom metrics and indicators
• Connect technical performance to real-world business outcomes

# Adding business context and success metrics to API calls
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    extra_headers=create_headers(
        use_case_name="customer_service",
        user_id="jane_doe",
        limit_ids=["department_budget"],
        request_tags=["greeting"],
        success_metrics={"customer_satisfaction": True, "resolution_time": "fast"}
    )
)

Learn more about Custom Instrumentation →

Business Value Measurement

A key advantage of Pay-i's instrumentation is the ability to measure and quantify the business value generated by your GenAI investments. This includes:

• KPI Tracking: Associate GenAI usage with specific key performance indicators
• Success Metrics: Define and track custom metrics that indicate successful AI implementation
• ROI Calculation: Combine cost data with value metrics to calculate return on investment
• Value Attribution: Understand which AI use cases deliver the most business value
• Quality Assessment: Measure the effectiveness and quality of AI-generated outputs

Through these capabilities, Pay-i transforms GenAI from a technical implementation into a measurable business asset with quantifiable return on investment.

Implementation Approaches

Pay-i supports two different operational approaches for instrumentation:

1. Direct Provider Call with Telemetry - Your application communicates directly with the GenAI provider while Pay-i's SDK captures and reports usage data asynchronously.

2. Proxy Routing - Your application routes calls through Pay-i, which forwards them to the provider and returns the results with additional metadata.

For a detailed explanation of these approaches and when to use each, see Pay-i Operational Modes.

Getting Started with Instrumentation

The recommended approach for most users is:

1. Start with Auto-Instrumentation to establish your GenAI spend Statement of Record
2. Add Custom Instrumentation to provide business context
3. Configure Budget Limits to control spending
4. Implement Use Cases to organize and analyze your GenAI operations
5. Define Success Metrics to measure business value
6. Set up KPI Tracking to evaluate business impact

This incremental approach allows you to quickly gain visibility while gradually adding more sophisticated tracking, controls, and value measurement.

Related Resources

• Auto-Instrumentation - Establishing your complete GenAI spend and usage Statement of Record
• Custom Instrumentation - Adding business context
• Pay-i Operational Modes - Understanding technical implementation approaches
• Pay-i Proxy Configuration - For when you need Block limits
• Budget Limits - Setting and monitoring budget limits
• Use Cases - Creating and managing use cases
• Use Case KPIs - Setting up KPI tracking
• Value of Pay-i - Understanding the business value of instrumentation