KPIs
Overview
Key Performance Indicators (KPIs) in Pay-i allow you to define and track custom business metrics associated with your GenAI-infused applications. Pay-i already provides many standard metrics for use cases out of the box (such as cost, usage, failure rates, and performance metrics), but KPIs let you add your own business-specific measurements.
KPIs are always associated with a specific use case instance (identified by a use_case_id
). This connection ensures that your custom metrics are linked to the exact context in which they were generated, providing a comprehensive view alongside the standard metrics Pay-i already tracks.
When to Use KPIs
KPIs are valuable when you need to track metrics beyond what Pay-i automatically provides, such as:
- Quality scores for AI-generated outputs: Track metrics like Deflection Rate in chatbots to measure how often the AI successfully handles queries without human intervention
- User satisfaction ratings from feedback systems: Implement Customer Satisfaction scores on a Likert scale (1-5) to measure user sentiment about AI interactions
- Business outcomes like conversion rates or revenue generated: Create metrics like Clickthrough Rate, Close Rate, and Conversion $ Value to directly measure business impact
- Custom performance criteria specific to your application: Define boolean KPIs (true/false) for application-specific success conditions
- Internal benchmarks for your AI features: Set up goals for each KPI (e.g., "5.00% True" for Clickthrough Rate) and track goal attainment
- Input components for business value calculations: Create metrics that feed into Value metrics for ROI calculation (conversion rates, satisfaction scores, etc.)
Types of KPIs
Pay-i supports several KPI types to accommodate different measurement approaches:
KPI Type | Description | Example Use |
---|---|---|
boolean | Binary measurements (true/false) | Pass/fail assessments, success indicators |
number | Numeric measurements with no specific scale | Time saved, revenue generated |
percentage | Measurements on a 0-100% scale | Accuracy rates, completion rates |
likert5 | Measurements on a 5-point Likert scale | User satisfaction ratings (1-5) |
likert7 | Measurements on a 7-point Likert scale | More nuanced satisfaction scales (1-7) |
likert10 | Measurements on a 10-point Likert scale | Detailed rating systems (1-10) |
The KPI type helps contextualize the measurement and ensures proper display in dashboards and reports.
Implementation Note
When submitting KPI values through the Python SDK, all values are provided as floating-point numbers (float
), regardless of the KPI type:
# All KPI values are provided as float
client.use_cases.kpis.create(
kpi_name="Clickthrough Rate", # Boolean KPI
use_case_id=use_case_id,
score=1.0 # Boolean values: 1.0 = true, 0.0 = false
)
client.use_cases.kpis.create(
kpi_name="Customer Satisfaction", # Likert5 KPI
use_case_id=use_case_id,
score=4.5 # Rating on 1-5 scale
)
For boolean KPIs specifically, use:
1.0
to represent true/success outcomes0.0
to represent false/failure outcomes
The Pay-i system interprets these values according to the KPI type, ensuring proper display in dashboards and reports.
Real-World KPI Examples
Based on production implementations, here are common KPIs used in successful GenAI applications:
KPI Name | KPI Type | Typical Goal | Description | Common Application |
---|---|---|---|---|
Clickthrough Rate | boolean | 5.00% True | Tracks when users click on AI-generated content | Content generation, ads |
Close Rate | boolean | 1.00% True | Measures conversion after engaging with AI content | E-commerce, sales |
Deflection Rate | boolean | 25.00% True | Tracks when AI resolves issues without human intervention | Chat bots, customer support |
Conversion $ Value | number | $10.00 | Measures monetary value of conversions from AI interactions | Ad generation, product recommendations |
Time Saved | number | 2 minutes | Quantifies efficiency gains from using AI solutions | Document processing, content creation, customer support |
Money Saved | number | $5.00 | Measures direct cost reduction from AI automation | Workflow automation, content generation, customer support |
Customer Satisfaction | likert5 | 3.50 / 5.00 | Captures user sentiment about AI interaction quality | Chat bots, content generation |
These KPIs represent proven metrics that help quantify both the technical performance and business impact of your GenAI applications. The typical goal values represent benchmarks from successful implementations, though your specific targets may vary based on your industry and use case.
Implementation Considerations
When implementing KPIs in your applications, consider these important factors:
- Measurement strategy: Determine how you'll collect the data for each KPI (user actions, feedback forms, expert validation)
- Validation method: For boolean KPIs like Error Detection, decide whether validation comes from users, experts, or automated systems
- Collection timing: Determine when in the user journey you'll collect and record each KPI value
- Interpretation context: Consider what additional properties you'll track alongside KPIs to provide context
For Python SDK implementation examples of all KPI types, see:
- Python SDK KPIs Guide - For comprehensive implementation patterns and code samples
- Python SDK KPIs API Reference - For detailed method parameters and return types
Funnel Conversion KPIs
A powerful application of KPIs is tracking multi-step user journeys through funnel conversion tracking. By defining a series of boolean KPIs that represent each stage in a conversion funnel, you can measure drop-off rates and identify opportunities for optimization.
Example funnel KPIs for a product recommendation system:
KPI Name | KPI Type | Typical Goal | Description |
---|---|---|---|
Recommendation Viewed | boolean | 90.00% True | User saw the AI recommendations |
Recommendation Clicked | boolean | 20.00% True | User clicked on at least one recommendation |
Added to Cart | boolean | 5.00% True | User added a recommended item to cart |
Purchased | boolean | 1.00% True | User completed purchase of a recommended item |
By tracking these sequential KPIs for each use case instance, you can:
- Visualize the conversion rates between stages in your AI-powered user journeys
- Compare funnel performance across different GenAI models (e.g., GPT-4o vs. Claude-3)
- Measure the impact of prompt engineering changes on conversion rates
- Identify which version of your AI application drives the highest completion rates
- Quantify the business impact of different LLM strategies through completed purchases
This approach is particularly valuable when A/B testing different GenAI implementations to determine which model, prompt strategy, or retrieval method delivers the best business outcomes.
Dashboard Visualization of KPI Data
Pay-i automatically adjusts the visualization of KPI data based on the KPI type to provide the most meaningful representation:
Boolean KPIs
KPIs like Clickthrough Rate, Close Rate, and Deflection Rate are displayed as:
- Bar charts showing the distribution of true vs. false values
- Percentage calculations showing the proportion of true values (e.g., "5.23% True")
- Goal attainment indicators comparing actual performance to thresholds (e.g., "Goal: 5.00% True")
These visualizations make it easy to see how often the desired outcome is occurring and whether you're meeting your targets.

Numeric KPIs
KPIs like Conversion $ Value, Time Saved, and Money Saved are displayed as:
- Histograms showing the distribution across value ranges
- Average value calculations with goal comparisons (e.g., "$12.47 (Goal: $10.00)")
- Aggregate totals for organization-wide impact assessment
These visualizations help identify patterns in your numeric data and track the overall financial or operational impact of your AI features.

Likert Scale KPIs
KPIs like Customer Satisfaction are displayed as:
- Histograms showing response distribution across the scale points (1-5, 1-7, or 1-10)
- Average score calculations with goal comparisons (e.g., "3.85 / 5.00 (Goal: 3.50)")
- Trend analysis to track changes in sentiment over time
These visualizations provide insight into user sentiment and satisfaction with your AI features, helping you identify areas for improvement.

Relationship to Value Metrics
While KPIs focus on measuring specific performance metrics, Pay-i also offers Value metrics as a complementary measurement system focused on financial ROI and business impact calculation.
Important Distinction
It's important to understand that:
- KPIs measure individual components that contribute to business value (clicks, conversions, satisfaction scores)
- Value metrics calculate comprehensive ROI and business impact by combining multiple components
You would not create a direct "ROI KPI" - instead, ROI calculation is handled through Value metrics that use your KPIs as inputs.
How Value Metrics Work
Value metrics are not KPIs, but rather a separate measurement approach that:
- Uses Business Equations to calculate overall impact and ROI
- Incorporates both standard metrics (like Instance Cost) and your custom KPIs
- Is configured exclusively through the Pay-i UI
- Provides specialized dashboards for financial reporting
Example: Building ROI Calculations
Consider this practical example of how KPIs and Value metrics work together:
-
Define KPIs that measure individual business outcomes:
- "Clickthrough Rate" (boolean KPI)
- "Conversion $ Value" (numeric KPI)
- "Customer Satisfaction" (likert5 KPI)
-
Create Value metrics in the UI that use these KPIs in Business Equations:
- ROI = (Sum of Conversion $ Values) / (Total Instance Cost)
- Customer Value = (Clickthrough Rate %) × (Average Customer Satisfaction) × (Fixed Multiplier)
This approach separates the measurement of individual components (KPIs) from the calculation of comprehensive business value (Value metrics).
For complete details on implementing Value metrics, see Value Metrics.
Related Resources
- For detailed guidance on creating and managing KPIs, see Working with KPIs
- For implementation strategies and best practices, see KPI Best Practices
- For concrete implementation examples for different application types, see KPI Examples from Real Applications
- To understand the foundational concept of use cases, see Use Cases
- To learn about organizing use case instances, explore Use Case Versioning
- For information on measuring business value through system KPIs, see Value Metrics
Updated 17 days ago