Overview

Key Performance Indicators (KPIs) in Pay-i allow you to define and track custom business metrics associated with your GenAI-infused applications. Pay-i already provides many standard metrics for use cases out of the box (such as cost, usage, failure rates, and performance metrics), but KPIs let you add your own business-specific measurements.

KPIs are always associated with a specific use case instance (identified by a use_case_id). This connection ensures that your custom metrics are linked to the exact context in which they were generated, providing a comprehensive view alongside the standard metrics Pay-i already tracks.

When to Use KPIs

KPIs are valuable when you need to track metrics beyond what Pay-i automatically provides, such as:

• Quality scores for AI-generated outputs: Track metrics like Deflection Rate in chatbots to measure how often the AI successfully handles queries without human intervention
• User satisfaction ratings from feedback systems: Implement Customer Satisfaction scores on a Likert scale (1-5) to measure user sentiment about AI interactions
• Business outcomes like conversion rates or revenue generated: Create metrics like Clickthrough Rate, Close Rate, and Conversion $ Value to directly measure business impact
• Custom performance criteria specific to your application: Define boolean KPIs (true/false) for application-specific success conditions
• Internal benchmarks for your AI features: Set up goals for each KPI (e.g., "5.00% True" for Clickthrough Rate) and track goal attainment
• Input components for business value calculations: Create metrics that feed into Value metrics for ROI calculation (conversion rates, satisfaction scores, etc.)

Types of KPIs

Pay-i supports several KPI types to accommodate different measurement approaches:

The KPI type helps contextualize the measurement and ensures proper display in dashboards and reports.

Implementation Note

When submitting KPI values through the Python SDK, all values are provided as floating-point numbers (float), regardless of the KPI type:

# All KPI values are provided as float
client.use_cases.kpis.create(
    kpi_name="Clickthrough Rate",  # Boolean KPI
    use_case_id=use_case_id,
    score=1.0  # Boolean values: 1.0 = true, 0.0 = false
)

client.use_cases.kpis.create(
    kpi_name="Customer Satisfaction",  # Likert5 KPI
    use_case_id=use_case_id,
    score=4.5  # Rating on 1-5 scale
)

For boolean KPIs specifically, use:

• 1.0 to represent true/success outcomes
• 0.0 to represent false/failure outcomes

The Pay-i system interprets these values according to the KPI type, ensuring proper display in dashboards and reports.

Real-World KPI Examples

Based on production implementations, here are common KPIs used in successful GenAI applications:

These KPIs represent proven metrics that help quantify both the technical performance and business impact of your GenAI applications. The typical goal values represent benchmarks from successful implementations, though your specific targets may vary based on your industry and use case.

Implementation Considerations

When implementing KPIs in your applications, consider these important factors:

1. Measurement strategy: Determine how you'll collect the data for each KPI (user actions, feedback forms, expert validation)
2. Validation method: For boolean KPIs like Error Detection, decide whether validation comes from users, experts, or automated systems
3. Collection timing: Determine when in the user journey you'll collect and record each KPI value
4. Interpretation context: Consider what additional properties you'll track alongside KPIs to provide context

For Python SDK implementation examples of all KPI types, see:

• Python SDK KPIs Guide - For comprehensive implementation patterns and code samples
• Python SDK KPIs API Reference - For detailed method parameters and return types

Funnel Conversion KPIs

A powerful application of KPIs is tracking multi-step user journeys through funnel conversion tracking. By defining a series of boolean KPIs that represent each stage in a conversion funnel, you can measure drop-off rates and identify opportunities for optimization.

Example funnel KPIs for a product recommendation system:

By tracking these sequential KPIs for each use case instance, you can:

• Visualize the conversion rates between stages in your AI-powered user journeys
• Compare funnel performance across different GenAI models (e.g., GPT-4o vs. Claude-3)
• Measure the impact of prompt engineering changes on conversion rates
• Identify which version of your AI application drives the highest completion rates
• Quantify the business impact of different LLM strategies through completed purchases

This approach is particularly valuable when A/B testing different GenAI implementations to determine which model, prompt strategy, or retrieval method delivers the best business outcomes.

Dashboard Visualization of KPI Data

Pay-i automatically adjusts the visualization of KPI data based on the KPI type to provide the most meaningful representation:

Boolean KPIs

KPIs like Clickthrough Rate, Close Rate, and Deflection Rate are displayed as:

• Bar charts showing the distribution of true vs. false values
• Percentage calculations showing the proportion of true values (e.g., "5.23% True")
• Goal attainment indicators comparing actual performance to thresholds (e.g., "Goal: 5.00% True")

These visualizations make it easy to see how often the desired outcome is occurring and whether you're meeting your targets.

Numeric KPIs

KPIs like Conversion $ Value, Time Saved, and Money Saved are displayed as:

• Histograms showing the distribution across value ranges
• Average value calculations with goal comparisons (e.g., "$12.47 (Goal: $10.00)")
• Aggregate totals for organization-wide impact assessment

These visualizations help identify patterns in your numeric data and track the overall financial or operational impact of your AI features.

Likert Scale KPIs

KPIs like Customer Satisfaction are displayed as:

• Histograms showing response distribution across the scale points (1-5, 1-7, or 1-10)
• Average score calculations with goal comparisons (e.g., "3.85 / 5.00 (Goal: 3.50)")
• Trend analysis to track changes in sentiment over time

These visualizations provide insight into user sentiment and satisfaction with your AI features, helping you identify areas for improvement.

Relationship to Value Metrics

While KPIs focus on measuring specific performance metrics, Pay-i also offers Value metrics as a complementary measurement system focused on financial ROI and business impact calculation.

Important Distinction

It's important to understand that:

• KPIs measure individual components that contribute to business value (clicks, conversions, satisfaction scores)
• Value metrics calculate comprehensive ROI and business impact by combining multiple components

You would not create a direct "ROI KPI" - instead, ROI calculation is handled through Value metrics that use your KPIs as inputs.

How Value Metrics Work

Value metrics are not KPIs, but rather a separate measurement approach that:

• Uses Business Equations to calculate overall impact and ROI
• Incorporates both standard metrics (like Instance Cost) and your custom KPIs
• Is configured exclusively through the Pay-i UI
• Provides specialized dashboards for financial reporting

Example: Building ROI Calculations

Consider this practical example of how KPIs and Value metrics work together:

1. Define KPIs that measure individual business outcomes:

   • "Clickthrough Rate" (boolean KPI)
   • "Conversion $ Value" (numeric KPI)
   • "Customer Satisfaction" (likert5 KPI)

2. Create Value metrics in the UI that use these KPIs in Business Equations:

   • ROI = (Sum of Conversion $ Values) / (Total Instance Cost)
   • Customer Value = (Clickthrough Rate %) × (Average Customer Satisfaction) × (Fixed Multiplier)

This approach separates the measurement of individual components (KPIs) from the calculation of comprehensive business value (Value metrics).

For complete details on implementing Value metrics, see Value Metrics.

Related Resources

• For detailed guidance on creating and managing KPIs, see Working with KPIs
• For implementation strategies and best practices, see KPI Best Practices
• For concrete implementation examples for different application types, see KPI Examples from Real Applications
• To understand the foundational concept of use cases, see Use Cases
• To learn about organizing use case instances, explore Use Case Versioning
• For information on measuring business value through system KPIs, see Value Metrics