Overview

Key Performance Indicators (KPIs) are essential for measuring the success of your GenAI applications. This guide provides practical instructions for creating and managing KPIs in Pay-i, with a focus on using the web interface to configure and view KPIs.

Whether you're evaluating a chatbot's effectiveness, measuring the ROI of AI-generated content, or tracking user satisfaction with your GenAI features, proper KPI implementation is critical for demonstrating business value.

For a conceptual understanding of KPIs in Pay-i, see Use Case KPIs. For comprehensive SDK usage documentation, see Python SDK KPIs.

Understanding the KPI Workflow

Pay-i implements KPIs using a two-tiered approach:

KPI Definitions: Created at the use case type level (identified by use_case_name)
KPI Values: Recorded at the use case instance level (identified by use_case_id)

This important distinction means implementing KPIs is a two-step process:

First, you define what KPIs exist for a particular use case type (e.g., "Chat-Bot" or "Image-Ad")
Then, you record actual values for those KPIs on specific instances of that use case type

This approach ensures consistent metrics across all instances of a given use case type, while allowing for instance-specific measurements.

Creating and Managing KPIs

Pay-i offers two ways to create and manage KPIs: through the web portal for a convenient visual interface, or programmatically via the SDK for automation and integration with your applications.

Managing KPIs in the Pay-i Portal

The Pay-i web interface provides a straightforward way to create, edit, and manage KPIs:

Creating KPIs

To create a new KPI:

Navigate to the use case configuration page
Select the Configuration tab
In the Key Performance Indicators section, click the + KPI button

In the New Use Case KPI dialog:

Enter a name for the KPI
Select a type from the dropdown (Boolean, Number, Percentage, Likert (5), Likert (7), or Likert (10))
Set a goal value appropriate for the KPI type
Click Create

The KPI will appear in the list with its name, type, and goal value.

Editing and Deleting KPIs

To modify existing KPIs:

Find the KPI you want to modify in the list
Click the three-dot menu (⋮) on the right side of the KPI row
Select one of the following options:
- Edit: Opens a dialog to modify the KPI's name, type, or goal value
- Delete: Removes the KPI from the use case

These management capabilities allow you to adapt your KPIs as your measurement needs evolve or correct any settings if needed.

Python SDK Integration

While this guide focuses on UI-based workflows, Pay-i also provides comprehensive SDK support for automating KPI creation and data collection. If you're looking to integrate KPIs into your applications programmatically, we recommend the following resources:

Python SDK KPIs - A complete guide to KPI implementation with code examples
Python SDK KPIs API Reference - Detailed API documentation for all KPI methods
KPI Best Practices - Guidance on implementing effective KPIs

The Python SDK provides methods for:

Creating KPI definitions for use case types
Recording KPI values for specific use case instances
Retrieving, updating, and deleting KPI data

For example, to define a KPI and record a value:

from payi import Payi

client = Payi()

# Define a KPI for a use case type
client.use_cases.definitions.kpis.create(
    use_case_name="Chat-Bot",
    kpi_name="Deflection Rate",
    description="Tracks when AI resolves issues without human intervention",
    kpi_type="boolean",
    goal=0.25
)

# Later, record a value for a specific instance
client.use_cases.kpis.create(
    kpi_name="Deflection Rate",
    use_case_id="uc_123456789",
    score=1.0
)

For complete SDK documentation and extensive examples, see the Python SDK KPIs guide.

KPI Examples from Real Applications

Based on actual Pay-i implementations, here are examples of KPIs used in production GenAI applications. These examples show both the definition step and the value recording step.

Chat Bot KPIs

For conversational AI applications, common KPIs include:

# Step 1: Define the KPIs for the Chat-Bot use case type (done once)
client.use_cases.definitions.kpis.create(
    use_case_name="Chat-Bot",
    kpi_name="Deflection Rate",
    description="Tracks when AI resolves issues without human intervention",
    kpi_type="boolean",
    goal=0.25  # Goal: 25% of queries resolved by AI
)

client.use_cases.definitions.kpis.create(
    use_case_name="Chat-Bot",
    kpi_name="User Satisfaction",
    description="Measures if users were satisfied with the interaction",
    kpi_type="boolean",
    goal=0.75  # Goal: 75% of users satisfied
)

# Step 2: Record values for specific chat sessions (done per instance)
chat_session = client.use_cases.create(use_case_name="Chat-Bot")
chat_session_id = chat_session.use_case_id

# Track whether the AI successfully handled the query without human intervention
client.use_cases.kpis.create(
    kpi_name="Deflection Rate",
    use_case_id=chat_session_id,
    score=1.0  # Boolean: 1.0 = True (deflected), 0.0 = False (escalated)
)

# Record user satisfaction with the AI interaction
client.use_cases.kpis.create(
    kpi_name="User Satisfaction",
    use_case_id=chat_session_id,
    score=1.0  # Boolean: 1.0 = True (satisfied), 0.0 = False (unsatisfied)
)

Image-Ad Generation KPIs

For AI-generated advertising content, typical KPIs include:

# Step 1: Define the KPIs for the Image-Ad use case type (done once)
client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Clickthrough Rate",
    description="Tracks when users click on AI-generated content",
    kpi_type="boolean",
    goal=0.05  # Goal: 5% clickthrough rate
)

client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Close Rate",
    description="Measures conversion after engaging with AI content",
    kpi_type="boolean",
    goal=0.01  # Goal: 1% conversion rate
)

client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Conversion $ Value",
    description="Measures monetary value of conversions",
    kpi_type="number",
    goal=10.00  # Goal: $10 per conversion
)

client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Customer Satisfaction",
    description="User rating of content quality",
    kpi_type="likert5",
    goal=3.5  # Goal: 3.5 out of 5 rating
)

# Step 2: Record values for specific ad campaigns (done per instance)
ad_campaign = client.use_cases.create(use_case_name="Image-Ad")
ad_campaign_id = ad_campaign.use_case_id

# Track user engagement with the generated ad
client.use_cases.kpis.create(
    kpi_name="Clickthrough Rate",
    use_case_id=ad_campaign_id,
    score=1.0  # Boolean: 1.0 = True (clicked), 0.0 = False (not clicked)
)

# Record if the interaction led to conversion
client.use_cases.kpis.create(
    kpi_name="Close Rate",
    use_case_id=ad_campaign_id,
    score=1.0  # Boolean: 1.0 = True (converted), 0.0 = False (not converted)
)

# Quantify the financial value of the conversion
client.use_cases.kpis.create(
    kpi_name="Conversion $ Value",
    use_case_id=ad_campaign_id,
    score=10.14  # Numeric value representing dollars
)

# Measure user satisfaction with the generated content
client.use_cases.kpis.create(
    kpi_name="Customer Satisfaction",
    use_case_id=ad_campaign_id,
    score=3.89  # Likert5 scale (1-5)
)

In the Pay-i dashboard, these KPIs are automatically aggregated to show metrics like average scores, goal attainment percentage, and trends over time. You can also view what percentage of use cases has been reviewed with KPI data (e.g., Use Cases Reviewed: 88.87%).

Combining KPIs with Use Case Properties

For more context around your KPIs, combine them with use case properties. First define your KPI, then attach properties to instances:

# Define the KPI for your use case type
client.use_cases.definitions.kpis.create(
    use_case_name="Content-Generator",
    kpi_name="Response Quality",
    description="Quality rating of generated content",
    kpi_type="likert5",
    goal=4.0
)

# Create a use case instance
content_generation = client.use_cases.create(use_case_name="Content-Generator")
use_case_id = content_generation.use_case_id

# Add context to the use case instance
client.use_cases.properties.create(
    property_name="model_version",
    use_case_id=use_case_id,
    value="gpt-4o"
)

# Then add KPI value for that specific instance
client.use_cases.kpis.create(
    kpi_name="Response Quality",
    use_case_id=use_case_id,
    score=4.8
)

This combination allows you to analyze KPI performance across different conditions, models, or configurations.

Analyzing KPI Data in the Pay-i Dashboard

Pay-i provides sophisticated visualization tools for analyzing your KPI data:

Dashboard Visualizations by KPI Type

The Pay-i dashboard automatically adjusts visualizations based on the KPI type to provide the most meaningful representation of your performance metrics:

Boolean KPIs (like Deflection Rate and Clickthrough Rate):
- Displayed as true/false distributions with bar charts showing count of true vs false values
- Automatically calculates percentage of true values as the average score
- Example: A Clickthrough Rate KPI might show 5.23% True (Goal: 5.00%)
- Visual representation highlights both the raw count and percentage of successful outcomes
Numeric KPIs (like Conversion $ Value):
- Displayed as histograms showing distribution across value ranges
- The x-axis represents value ranges (e.g., $0-$5, $5-$10, $10-$15, etc.)
- The y-axis shows count of occurrences within each range
- Average value is calculated and displayed compared to goal (e.g., $12.47 (Goal: $10.00))
- Distribution patterns help identify outliers and concentration areas
Likert Scale KPIs (like Customer Satisfaction):
- Displayed as histograms showing distribution across the scale points (1-5, 1-7, or 1-10)
- Shows how many responses fall into each rating category
- Automatically calculates the average score (e.g., 3.85 / 5.00 (Goal: 3.50))
- Visual representation makes it easy to see sentiment distribution

For each KPI type, the dashboard also provides:

Goal attainment indicators (whether you're meeting your targets)
Trend analysis to track changes over time
Filtering options by date range, use case version, and other parameters

Dashboard Elements

The Pay-i KPI dashboard provides:

Time-series graphs showing KPI performance over time
Distribution histograms showing value patterns
Summary metrics including:
- Average score
- Goal values
- Percentage of goal met
- Use cases reviewed percentage
Filtering options by date range and use case type

Trend Analysis

Track changes in KPIs over time to identify:

Performance improvements
Degradation patterns
Correlation with system changes
Seasonal variations

Comparative Analysis

Compare KPIs across different dimensions:

Between different use cases
Across different time periods
Before and after system changes
Against established benchmarks

Related Resources

Learn about the core concepts of KPIs in Pay-i in the Use Case KPIs guide
For implementation strategies and best practices, see KPI Best Practices
For information on measuring business value through system KPIs, see Value Metrics