This guide explains how the Conversation Score is calculated to measure the overall success of your AI Employee's interactions. It details the penalty values associated with auto and manual star ratings and provides calculation examples to help you interpret the metric.
Overview
The Conversation Score is a performance metric calculated as:
Conversation Score = 1 - (Total Penalties / Total Sessions)
It aggregates data across all sessions to indicate friction or success. Higher scores (closer to 100% or 1.0) indicate better outcomes, while lower scores suggest that the AI Employee may need attention or logic adjustments.
Penalty determination
Session penalties are derived from the CSS rating assigned to a session (see Conversation Success Score). The penalty is calculated using the following logic:
Conversation Penalty = 1 - CSS_rating_of_session
The following table defines the penalty values for each rating:
Star rating | Penalty value |
5 stars | 0.02 |
4 stars | 0.2 |
3 stars | 0.4 |
2 stars | 0.6 |
1 star | 0.8 |
Unassessed | 0 |
Example 1: High performance
In this scenario, an AI Employee has completed 28 total sessions with generally positive ratings.
Calculate Total Penalties: Sum the penalties from all 28 sessions (e.g., a total of 1.42).
Calculate Average Penalty: Divide the total penalties by the total sessions (e.g., 1.42 / 28 = 0.051).
Determine Score: Subtract the average penalty from 1 (1 - 0.051 = 0.949).
Result: The Conversation Score is approximately 95%. This indicates the agent is performing well.
Example 2: Low performance
In this scenario, an AI Employee has completed 28 total sessions, but all sessions received a 1-star rating.
Calculate Total Penalties: Sum the penalties from all 28 sessions (e.g., a total of 22.4).
Calculate Average Penalty: Divide the total penalties by the total sessions (e.g., 22.4 / 28 = 0.8).
Determine Score: Subtract the average penalty from 1 (1 - 0.8 = 0.2).
Result: The Conversation Score is 20%. This indicates the agent requires immediate attention to resolve interaction friction.