Guides

Overview

Key Performance Indicators (KPIs) in Pay-i allow you to define and track custom business metrics associated with your GenAI-infused applications. Pay-i already provides many standard metrics for use cases out of the box (such as cost, usage, failure rates, and performance metrics), but KPIs let you add your own business-specific measurements.

KPIs are always associated with a specific use case instance (identified by a use_case_id). This connection ensures that your custom metrics are linked to the exact context in which they were generated, providing a comprehensive view alongside the standard metrics Pay-i already tracks.

When to Use KPIs

KPIs are valuable when you need to track metrics beyond what Pay-i automatically provides, such as:

  • Quality scores for AI-generated outputs: Track metrics like Deflection Rate in chatbots to measure how often the AI successfully handles queries without human intervention
  • User satisfaction ratings from feedback systems: Implement Customer Satisfaction scores on a Likert scale (1-5) to measure user sentiment about AI interactions
  • Business outcomes like conversion rates or revenue generated: Create metrics like Clickthrough Rate, Close Rate, and Conversion $ Value to directly measure business impact
  • Custom performance criteria specific to your application: Define boolean KPIs (true/false) for application-specific success conditions
  • Internal benchmarks for your AI features: Set up goals for each KPI (e.g., "5.00% True" for Clickthrough Rate) and track goal attainment
  • Input components for business value calculations: Create metrics that feed into Value metrics for ROI calculation (conversion rates, satisfaction scores, etc.)

Types of KPIs

Pay-i supports several KPI types to accommodate different measurement approaches:

KPI TypeDescriptionExample Use
booleanBinary measurements (true/false)Pass/fail assessments, success indicators
numberNumeric measurements with no specific scaleTime saved, revenue generated
percentageMeasurements on a 0-100% scaleAccuracy rates, completion rates
likert5Measurements on a 5-point Likert scaleUser satisfaction ratings (1-5)
likert7Measurements on a 7-point Likert scaleMore nuanced satisfaction scales (1-7)
likert10Measurements on a 10-point Likert scaleDetailed rating systems (1-10)

The KPI type helps contextualize the measurement and ensures proper display in dashboards and reports.

Implementation Note

When submitting KPI values through the Python SDK, all values are provided as floating-point numbers (float), regardless of the KPI type:

# All KPI values are provided as float
client.use_cases.kpis.create(
    kpi_name="Clickthrough Rate",  # Boolean KPI
    use_case_id=use_case_id,
    score=1.0  # Boolean values: 1.0 = true, 0.0 = false
)

client.use_cases.kpis.create(
    kpi_name="Customer Satisfaction",  # Likert5 KPI
    use_case_id=use_case_id,
    score=4.5  # Rating on 1-5 scale
)

For boolean KPIs specifically, use:

  • 1.0 to represent true/success outcomes
  • 0.0 to represent false/failure outcomes

The Pay-i system interprets these values according to the KPI type, ensuring proper display in dashboards and reports.

Real-World KPI Examples

Based on production implementations, here are common KPIs used in successful GenAI applications:

KPI NameKPI TypeTypical GoalDescriptionCommon Application
Clickthrough Rateboolean5.00% TrueTracks when users click on AI-generated contentContent generation, ads
Close Rateboolean1.00% TrueMeasures conversion after engaging with AI contentE-commerce, sales
Deflection Rateboolean25.00% TrueTracks when AI resolves issues without human interventionChat bots, customer support
Conversion $ Valuenumber$10.00Measures monetary value of conversions from AI interactionsAd generation, product recommendations
Time Savednumber2 minutesQuantifies efficiency gains from using AI solutionsDocument processing, content creation, customer support
Money Savednumber$5.00Measures direct cost reduction from AI automationWorkflow automation, content generation, customer support
Customer Satisfactionlikert53.50 / 5.00Captures user sentiment about AI interaction qualityChat bots, content generation

These KPIs represent proven metrics that help quantify both the technical performance and business impact of your GenAI applications. The typical goal values represent benchmarks from successful implementations, though your specific targets may vary based on your industry and use case.

Implementation Considerations

When implementing KPIs in your applications, consider these important factors:

  1. Measurement strategy: Determine how you'll collect the data for each KPI (user actions, feedback forms, expert validation)
  2. Validation method: For boolean KPIs like Error Detection, decide whether validation comes from users, experts, or automated systems
  3. Collection timing: Determine when in the user journey you'll collect and record each KPI value
  4. Interpretation context: Consider what additional properties you'll track alongside KPIs to provide context

For Python SDK implementation examples of all KPI types, see:

Funnel Conversion KPIs

A powerful application of KPIs is tracking multi-step user journeys through funnel conversion tracking. By defining a series of boolean KPIs that represent each stage in a conversion funnel, you can measure drop-off rates and identify opportunities for optimization.

Example funnel KPIs for a product recommendation system:

KPI NameKPI TypeTypical GoalDescription
Recommendation Viewedboolean90.00% TrueUser saw the AI recommendations
Recommendation Clickedboolean20.00% TrueUser clicked on at least one recommendation
Added to Cartboolean5.00% TrueUser added a recommended item to cart
Purchasedboolean1.00% TrueUser completed purchase of a recommended item

By tracking these sequential KPIs for each use case instance, you can:

  • Visualize the conversion rates between stages in your AI-powered user journeys
  • Compare funnel performance across different GenAI models (e.g., GPT-4o vs. Claude-3)
  • Measure the impact of prompt engineering changes on conversion rates
  • Identify which version of your AI application drives the highest completion rates
  • Quantify the business impact of different LLM strategies through completed purchases

This approach is particularly valuable when A/B testing different GenAI implementations to determine which model, prompt strategy, or retrieval method delivers the best business outcomes.

Dashboard Visualization of KPI Data

Pay-i automatically adjusts the visualization of KPI data based on the KPI type to provide the most meaningful representation:

Boolean KPIs

KPIs like Clickthrough Rate, Close Rate, and Deflection Rate are displayed as:

  • Bar charts showing the distribution of true vs. false values
  • Percentage calculations showing the proportion of true values (e.g., "5.23% True")
  • Goal attainment indicators comparing actual performance to thresholds (e.g., "Goal: 5.00% True")

These visualizations make it easy to see how often the desired outcome is occurring and whether you're meeting your targets.

Numeric KPIs

KPIs like Conversion $ Value, Time Saved, and Money Saved are displayed as:

  • Histograms showing the distribution across value ranges
  • Average value calculations with goal comparisons (e.g., "$12.47 (Goal: $10.00)")
  • Aggregate totals for organization-wide impact assessment

These visualizations help identify patterns in your numeric data and track the overall financial or operational impact of your AI features.

Likert Scale KPIs

KPIs like Customer Satisfaction are displayed as:

  • Histograms showing response distribution across the scale points (1-5, 1-7, or 1-10)
  • Average score calculations with goal comparisons (e.g., "3.85 / 5.00 (Goal: 3.50)")
  • Trend analysis to track changes in sentiment over time

These visualizations provide insight into user sentiment and satisfaction with your AI features, helping you identify areas for improvement.

Relationship to Value Metrics

While KPIs focus on measuring specific performance metrics, Pay-i also offers Value metrics as a complementary measurement system focused on financial ROI and business impact calculation.

Important Distinction

It's important to understand that:

  • KPIs measure individual components that contribute to business value (clicks, conversions, satisfaction scores)
  • Value metrics calculate comprehensive ROI and business impact by combining multiple components

You would not create a direct "ROI KPI" - instead, ROI calculation is handled through Value metrics that use your KPIs as inputs.

How Value Metrics Work

Value metrics are not KPIs, but rather a separate measurement approach that:

  • Uses Business Equations to calculate overall impact and ROI
  • Incorporates both standard metrics (like Instance Cost) and your custom KPIs
  • Is configured exclusively through the Pay-i UI
  • Provides specialized dashboards for financial reporting

Example: Building ROI Calculations

Consider this practical example of how KPIs and Value metrics work together:

  1. Define KPIs that measure individual business outcomes:

    • "Clickthrough Rate" (boolean KPI)
    • "Conversion $ Value" (numeric KPI)
    • "Customer Satisfaction" (likert5 KPI)
  2. Create Value metrics in the UI that use these KPIs in Business Equations:

    • ROI = (Sum of Conversion $ Values) / (Total Instance Cost)
    • Customer Value = (Clickthrough Rate %) × (Average Customer Satisfaction) × (Fixed Multiplier)

This approach separates the measurement of individual components (KPIs) from the calculation of comprehensive business value (Value metrics).

For complete details on implementing Value metrics, see Value Metrics.

Related Resources