# Working with KPIs

Overview Key Performance Indicators (KPIs) are essential for measuring the success of your GenAI applications. This guide provides practical instructions for creating and managing KPIs in Pay-i, with a focus on using the web interface to configure and view KPIs. Whether you're evaluating a chatbot's...

# Overview

Key Performance Indicators (KPIs) are essential for measuring the success of your GenAI applications. This guide provides practical instructions for creating and managing KPIs in Pay-i, with a focus on using the web interface to configure and view KPIs.

Whether you're evaluating a chatbot's effectiveness, measuring the ROI of AI-generated content, or tracking user satisfaction with your GenAI features, proper KPI implementation is critical for demonstrating business value.

For a conceptual understanding of KPIs in Pay-i, see [Use Case KPIs](/docs/use-case-kpis). For comprehensive SDK usage documentation, see [Python SDK KPIs](/docs/python-sdk-kpis).

# Understanding the KPI Workflow

Pay-i implements KPIs using a two-tiered approach:

1. **KPI Definitions**: Created at the use case type level (identified by `use_case_name`)
2. **KPI Values**: Recorded at the use case instance level (identified by `use_case_id`)

This important distinction means implementing KPIs is a two-step process:

- First, you define what KPIs exist for a particular use case type (e.g., "Chat-Bot" or "Image-Ad")
- Then, you record actual values for those KPIs on specific instances of that use case type

This approach ensures consistent metrics across all instances of a given use case type, while allowing for instance-specific measurements.

# Creating and Managing KPIs

Pay-i offers two ways to create and manage KPIs: through the web portal for a convenient visual interface, or programmatically via the SDK for automation and integration with your applications.

# Managing KPIs in the Pay-i Portal

The Pay-i web interface provides a straightforward way to create, edit, and manage KPIs:

## Creating KPIs

[block:image]
{
  "images": [
    {
      "image": [
        "https://files.readme.io/7c48f41311dc2b3ceb5d159ae8925a25777896e3ffe61cc3198dd47e9188d3a2-image.png",
        null,
        ""
      ],
      "align": "center",
      "border": true
    }
  ]
}
[/block]


To create a new KPI:

1. Navigate to the use case configuration page
2. Select the **Configuration** tab
3. In the **Key Performance Indicators** section, click the **+ KPI** button

[block:image]
{
  "images": [
    {
      "image": [
        "https://files.readme.io/7d8b52d807e596a0dccca3f317f97893d2b75c768c72aed11cff05d921cf3dbf-image.png",
        null,
        ""
      ],
      "align": "center",
      "border": true
    }
  ]
}
[/block]


4. In the **New Use Case KPI** dialog:

- Enter a name for the KPI
- Select a type from the dropdown (**Boolean**, **Number**, **Percentage**, **Likert (5)**, **Likert (7)**, or **Likert (10)**)
- Set a goal value appropriate for the KPI type
- Click **Create**

The KPI will appear in the list with its name, type, and goal value.

## Editing and Deleting KPIs

To modify existing KPIs:

1. Find the KPI you want to modify in the list
2. Click the three-dot menu (**⋮**) on the right side of the KPI row
3. Select one of the following options:
   - **Edit**: Opens a dialog to modify the KPI's name, type, or goal value
   - **Delete**: Removes the KPI from the use case

These management capabilities allow you to adapt your KPIs as your measurement needs evolve or correct any settings if needed.

# Python SDK Integration

While this guide focuses on UI-based workflows, Pay-i also provides comprehensive SDK support for automating KPI creation and data collection. If you're looking to integrate KPIs into your applications programmatically, we recommend the following resources:

- [Python SDK KPIs](/docs/python-sdk-kpis) - A complete guide to KPI implementation with code examples
- [Python SDK KPIs API Reference](/docs/python-sdk-kpis-api-reference) - Detailed API documentation for all KPI methods
- [KPI Best Practices](/docs/kpi-best-practices) - Guidance on implementing effective KPIs

The Python SDK provides methods for:

- Creating KPI definitions for use case types
- Recording KPI values for specific use case instances
- Retrieving, updating, and deleting KPI data

For example, to define a KPI and record a value:

```python
from payi import Payi

client = Payi()

# Define a KPI for a use case type
client.use_cases.definitions.kpis.create(
    use_case_name="Chat-Bot",
    kpi_name="Deflection Rate",
    description="Tracks when AI resolves issues without human intervention",
    kpi_type="boolean",
    goal=0.25
)

# Later, record a value for a specific instance
client.use_cases.kpis.create(
    kpi_name="Deflection Rate",
    use_case_id="uc_123456789",
    score=1.0
)
```

For complete SDK documentation and extensive examples, see the [Python SDK KPIs guide](/docs/python-sdk-kpis).

# KPI Examples from Real Applications

Based on actual Pay-i implementations, here are examples of KPIs used in production GenAI applications. These examples show both the definition step and the value recording step.

## Chat Bot KPIs

For conversational AI applications, common KPIs include:

```python
# Step 1: Define the KPIs for the Chat-Bot use case type (done once)
client.use_cases.definitions.kpis.create(
    use_case_name="Chat-Bot",
    kpi_name="Deflection Rate",
    description="Tracks when AI resolves issues without human intervention",
    kpi_type="boolean",
    goal=0.25  # Goal: 25% of queries resolved by AI
)

client.use_cases.definitions.kpis.create(
    use_case_name="Chat-Bot",
    kpi_name="User Satisfaction",
    description="Measures if users were satisfied with the interaction",
    kpi_type="boolean",
    goal=0.75  # Goal: 75% of users satisfied
)

# Step 2: Record values for specific chat sessions (done per instance)
chat_session = client.use_cases.create(use_case_name="Chat-Bot")
chat_session_id = chat_session.use_case_id

# Track whether the AI successfully handled the query without human intervention
client.use_cases.kpis.create(
    kpi_name="Deflection Rate",
    use_case_id=chat_session_id,
    score=1.0  # Boolean: 1.0 = True (deflected), 0.0 = False (escalated)
)

# Record user satisfaction with the AI interaction
client.use_cases.kpis.create(
    kpi_name="User Satisfaction",
    use_case_id=chat_session_id,
    score=1.0  # Boolean: 1.0 = True (satisfied), 0.0 = False (unsatisfied)
)
```

## Image-Ad Generation KPIs

For AI-generated advertising content, typical KPIs include:

```python
# Step 1: Define the KPIs for the Image-Ad use case type (done once)
client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Clickthrough Rate",
    description="Tracks when users click on AI-generated content",
    kpi_type="boolean",
    goal=0.05  # Goal: 5% clickthrough rate
)

client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Close Rate",
    description="Measures conversion after engaging with AI content",
    kpi_type="boolean",
    goal=0.01  # Goal: 1% conversion rate
)

client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Conversion $ Value",
    description="Measures monetary value of conversions",
    kpi_type="number",
    goal=10.00  # Goal: $10 per conversion
)

client.use_cases.definitions.kpis.create(
    use_case_name="Image-Ad",
    kpi_name="Customer Satisfaction",
    description="User rating of content quality",
    kpi_type="likert5",
    goal=3.5  # Goal: 3.5 out of 5 rating
)

# Step 2: Record values for specific ad campaigns (done per instance)
ad_campaign = client.use_cases.create(use_case_name="Image-Ad")
ad_campaign_id = ad_campaign.use_case_id

# Track user engagement with the generated ad
client.use_cases.kpis.create(
    kpi_name="Clickthrough Rate",
    use_case_id=ad_campaign_id,
    score=1.0  # Boolean: 1.0 = True (clicked), 0.0 = False (not clicked)
)

# Record if the interaction led to conversion
client.use_cases.kpis.create(
    kpi_name="Close Rate",
    use_case_id=ad_campaign_id,
    score=1.0  # Boolean: 1.0 = True (converted), 0.0 = False (not converted)
)

# Quantify the financial value of the conversion
client.use_cases.kpis.create(
    kpi_name="Conversion $ Value",
    use_case_id=ad_campaign_id,
    score=10.14  # Numeric value representing dollars
)

# Measure user satisfaction with the generated content
client.use_cases.kpis.create(
    kpi_name="Customer Satisfaction",
    use_case_id=ad_campaign_id,
    score=3.89  # Likert5 scale (1-5)
)
```

In the Pay-i dashboard, these KPIs are automatically aggregated to show metrics like average scores, goal attainment percentage, and trends over time. You can also view what percentage of use cases has been reviewed with KPI data (e.g., **Use Cases Reviewed: 88.87%**).

# Combining KPIs with Use Case Properties

For more context around your KPIs, combine them with use case properties. First define your KPI, then attach properties to instances:

```python
# Define the KPI for your use case type
client.use_cases.definitions.kpis.create(
    use_case_name="Content-Generator",
    kpi_name="Response Quality",
    description="Quality rating of generated content",
    kpi_type="likert5",
    goal=4.0
)

# Create a use case instance
content_generation = client.use_cases.create(use_case_name="Content-Generator")
use_case_id = content_generation.use_case_id

# Add context to the use case instance
client.use_cases.properties.create(
    property_name="model_version",
    use_case_id=use_case_id,
    value="gpt-4o"
)

# Then add KPI value for that specific instance
client.use_cases.kpis.create(
    kpi_name="Response Quality",
    use_case_id=use_case_id,
    score=4.8
)
```

This combination allows you to analyze KPI performance across different conditions, models, or configurations.

# Analyzing KPI Data in the Pay-i Dashboard

Pay-i provides sophisticated visualization tools for analyzing your KPI data:

## Dashboard Visualizations by KPI Type

The Pay-i dashboard automatically adjusts visualizations based on the KPI type to provide the most meaningful representation of your performance metrics:

- **Boolean KPIs** (like Deflection Rate and Clickthrough Rate):
  - Displayed as true/false distributions with bar charts showing count of true vs false values
  - Automatically calculates percentage of true values as the average score
  - Example: A Clickthrough Rate KPI might show **5.23% True (Goal: 5.00%)**
  - Visual representation highlights both the raw count and percentage of successful outcomes

- **Numeric KPIs** (like Conversion $ Value):
  - Displayed as histograms showing distribution across value ranges
  - The x-axis represents value ranges (e.g., $0-$5, $5-$10, $10-$15, etc.)
  - The y-axis shows count of occurrences within each range
  - Average value is calculated and displayed compared to goal (e.g., **$12.47 (Goal: $10.00)**)
  - Distribution patterns help identify outliers and concentration areas

- **Likert Scale KPIs** (like Customer Satisfaction):
  - Displayed as histograms showing distribution across the scale points (1-5, 1-7, or 1-10)
  - Shows how many responses fall into each rating category
  - Automatically calculates the average score (e.g., **3.85 / 5.00 (Goal: 3.50)**)
  - Visual representation makes it easy to see sentiment distribution

For each KPI type, the dashboard also provides:

- Goal attainment indicators (whether you're meeting your targets)
- Trend analysis to track changes over time
- Filtering options by date range, use case version, and other parameters

## Dashboard Elements

The Pay-i KPI dashboard provides:

1. Time-series graphs showing KPI performance over time
2. Distribution histograms showing value patterns
3. Summary metrics including:
   - Average score
   - Goal values
   - Percentage of goal met
   - Use cases reviewed percentage
4. Filtering options by date range and use case type

## Trend Analysis

Track changes in KPIs over time to identify:

- Performance improvements
- Degradation patterns
- Correlation with system changes
- Seasonal variations

## Comparative Analysis

Compare KPIs across different dimensions:

- Between different use cases
- Across different time periods
- Before and after system changes
- Against established benchmarks

# Related Resources

- Learn about the core concepts of KPIs in Pay-i in the [Use Case KPIs](/docs/use-case-kpis) guide
- For implementation strategies and best practices, see [KPI Best Practices](/docs/kpi-best-practices)
- For information on measuring business value through system KPIs, see [Value Metrics](/docs/value-metrics)
- <br />