Overview

Key Performance Indicators (KPIs) in Pay-i allow you to define and track custom business metrics associated with your GenAI-infused applications. Pay-i already provides many standard metrics for use cases out of the box (such as cost, usage, failure rates, and performance metrics), but KPIs let you add your own business-specific measurements.

KPIs are always associated with a specific use case instance (identified by a use_case_id). This connection ensures that your custom metrics are linked to the exact context in which they were generated, providing a comprehensive view alongside the standard metrics Pay-i already tracks.

When to Use KPIs

KPIs are valuable when you need to track metrics beyond what Pay-i automatically provides, such as:

Quality scores for AI-generated outputs: Track metrics like Deflection Rate in chatbots to measure how often the AI successfully handles queries without human intervention
User satisfaction ratings from feedback systems: Implement Customer Satisfaction scores on a Likert scale (1-5) to measure user sentiment about AI interactions
Business outcomes like conversion rates or revenue generated: Create metrics like Clickthrough Rate, Close Rate, and Conversion $ Value to directly measure business impact
Custom performance criteria specific to your application: Define boolean KPIs (true/false) for application-specific success conditions
Internal benchmarks for your AI features: Set up goals for each KPI (e.g., "5.00% True" for Clickthrough Rate) and track goal attainment
ROI measurements for specific business use cases: Implement Value metrics (a special type of system KPI) to calculate financial return on investment

Types of KPIs

Pay-i supports several KPI types to accommodate different measurement approaches:

KPI Type	Description	Example Use
`boolean`	Binary measurements (true/false)	Pass/fail assessments, success indicators
`number`	Numeric measurements with no specific scale	Time saved, revenue generated
`percentage`	Measurements on a 0-100% scale	Accuracy rates, completion rates
`likert5`	Measurements on a 5-point Likert scale	User satisfaction ratings (1-5)
`likert7`	Measurements on a 7-point Likert scale	More nuanced satisfaction scales (1-7)
`likert10`	Measurements on a 10-point Likert scale	Detailed rating systems (1-10)

The KPI type helps contextualize the measurement and ensures proper display in dashboards and reports.

Real-World KPI Examples

Based on production implementations, here are common KPIs used in successful GenAI applications:

KPI Name	KPI Type	Typical Goal	Description	Common Application
Clickthrough Rate	boolean	5.00% True	Tracks when users click on AI-generated content	Content generation, ads
Close Rate	boolean	1.00% True	Measures conversion after engaging with AI content	E-commerce, sales
Deflection Rate	boolean	25.00% True	Tracks when AI resolves issues without human intervention	Chat bots, customer support
Conversion $ Value	number	$10.00	Measures monetary value of conversions from AI interactions	Ad generation, product recommendations
Time Saved	number	2 minutes	Quantifies efficiency gains from using AI solutions	Document processing, content creation, customer support
Money Saved	number	$5.00	Measures direct cost reduction from AI automation	Workflow automation, content generation, customer support
Customer Satisfaction	likert5	3.50 / 5.00	Captures user sentiment about AI interaction quality	Chat bots, content generation

These KPIs represent proven metrics that help quantify both the technical performance and business impact of your GenAI applications. The typical goal values represent benchmarks from successful implementations, though your specific targets may vary based on your industry and use case.

Dashboard Visualization of KPI Data

Pay-i automatically adjusts the visualization of KPI data based on the KPI type to provide the most meaningful representation:

Boolean KPIs

KPIs like Clickthrough Rate, Close Rate, and Deflection Rate are displayed as:

Bar charts showing the distribution of true vs. false values
Percentage calculations showing the proportion of true values (e.g., "5.23% True")
Goal attainment indicators comparing actual performance to thresholds (e.g., "Goal: 5.00% True")

These visualizations make it easy to see how often the desired outcome is occurring and whether you're meeting your targets.

Numeric KPIs

KPIs like Conversion $ Value, Time Saved, and Money Saved are displayed as:

Histograms showing the distribution across value ranges
Average value calculations with goal comparisons (e.g., "$12.47 (Goal: $10.00)")
Aggregate totals for organization-wide impact assessment

These visualizations help identify patterns in your numeric data and track the overall financial or operational impact of your AI features.

Likert Scale KPIs

KPIs like Customer Satisfaction are displayed as:

Histograms showing response distribution across the scale points (1-5, 1-7, or 1-10)
Average score calculations with goal comparisons (e.g., "3.85 / 5.00 (Goal: 3.50)")
Trend analysis to track changes in sentiment over time

These visualizations provide insight into user sentiment and satisfaction with your AI features, helping you identify areas for improvement.

User-Defined vs. System KPIs

Pay-i has two categories of KPIs:

User-Defined KPIs: These are custom metrics that you create and manage through the SDK, as described in this document. They allow you to track business-specific measurements of your choosing.
System KPIs: These are specialized metrics managed by Pay-i itself. The primary type of system KPI is the Value metric, which specifically quantifies business impact and ROI from your GenAI-infused applications.

Unlike user-defined KPIs, Value metrics are configured exclusively through the Pay-i UI rather than the SDK. This centralized approach ensures consistent calculation methodologies across your organization and provides specialized dashboards for financial reporting.

For comprehensive details on implementing Value metrics for ROI measurement, see Value Metrics.

Related Resources

For detailed guidance on creating and managing KPIs, see Working with KPIs
For implementation strategies and best practices, see KPI Best Practices
To understand the foundational concept of use cases, see Use Cases
To learn about organizing use case instances, explore Use Case Versioning
For information on measuring business value through system KPIs, see Value Metrics