Overview

Key Performance Indicators (KPIs) in Pay-i allow you to define and track custom business metrics associated with your GenAI use cases. Pay-i already provides many standard metrics for use cases out of the box (such as cost, usage, failure rates, and performance metrics), but KPIs let you add your own business-specific measurements. As you’re iterating on your use cases, you want to understand not just how its performing from a cost, latency, and failure perspective, but also whether it is meeting the goals of your business.

There are two steps to working with KPIs in Pay-i, defining them, and then providing the data (called "scores"). Once defined, Pay-i will correlate them with the versions of your Use Case so that you can see how implementation changes (such as new models) are impacting your business.

Further, Pay-i lets you define Value Policies that use KPI data to measure quantifiable ROI.

When to Use KPIs

KPIs are valuable when you need to track metrics beyond what Pay-i automatically provides, such as:

• Success/Failure Rates for Specific Outcomes: Track metrics like Deflection Rate in chatbots to measure how often the AI successfully handles queries without human intervention
• User Satisfaction Ratings from Feedback Systems: Customer Satisfaction scores, such as a Likert scale (1-5), to measure user sentiment about AI interactions
• Business outcomes like conversion rates or revenue generated: Track metrics like Clickthrough Rate, Close Rate, and Conversion Value of AI outreach to directly measure business impact

Defining KPIs

Pay-i makes defining KPIs very straightforward. There are four values for each KPI definition. Not all fields can be edited once the KPI is created, but you can define any number of KPIs for your use cases and remove them at any time.

KPI Types

Pay-i supports several KPI types to accommodate different measurement approaches. If there is a KPI Type that you feel is missing, please contact [email protected].

Defining Goals

Apart from being able to track whether your Use Cases are meeting their targets, goals for KPIs are used when creating Value Policies. For each KPI Type, goals are defined slightly differently.

Providing KPI Scores

Once defined, the next step is to provide Pay-i with the KPI data. Details for how this is done technically are described in the development section, but conceptually this is very straightforward.

KPIs are always defined for a Use Case. Every time that use case is used (e.g., every time an agent is executed), an Instance is created with a unique identifier. From there, a single API call lets you set the KPI scores for that Instance.

KPI Score Example

Let's take a chatbot as example use case. After each chat, the user is given a thumbs up/thumbs down button to rate their experience as a whole.

To model this, the chatbot use case has a single KPI defined in Pay-i. For this example, we will name the KPI User Satisfaction. This KPI's type is Boolean, to model the thumbs up as True and thumbs down as False.

Each chat with this chatbot is an Instance of the chatbot use case, and has a unique ID which can be provided either by your service or auto-generated by Pay-i. Let's imagine that the ID for our example chat is chat1 and the user gave it a thumbs up.

Setting the score for the User Satisfaction KPI for a given chat is as simple as calling Pay-i and providing the Instance ID, chat1 in this example, the name of the KPI User Satisfaction and the score True. You can call this Pay-i API any number of times, setting and resetting the scores as needed for any number of KPIs on the chatbot use case.

Pay-i knows which version of your use case the Instance belongs to, and it knows which KPIs have been defined for that use case. From there, Pay-i handles the rest, aggregating across all Instances of that use case and integrating the data with Value Policies to calculate ROI.

Real-World KPI Examples

Here are example KPIs based on real world production implementations.