The absolute worst thing an AI agent can do is hallucinate a false solution when it is unsure. It is far better for an agent to say, "I'm sorry, I don't have that information. Let me connect you with a human representative," than to make up an incorrect price.
But how do you define "unsure"? If you configure your fallback threshold too high, the agent will escalate half of all inquiries, rendering the automation useless. If you set it too low, customers will receive incorrect answers.
The Metric: Semantic Distance and Grounding Scores
Inside trained.chat, we evaluate every response on three telemetry markers:
- RAG Grounding Score: How well does the generated text map to the retrieved document facts?
- Vector Similarity Distance: Does the user's question actually map closely to the knowledge base index?
- User Frustration Signals: Checking message inputs for irritation patterns or explicit requests for human agent involvement.
If the calculated grounding score drops below 65%, the agent immediately triggers a human handoff alert, transferring the conversation context directly to the dashboard Inbox where a team member can take over. Post-implementation, our deflection settled at a healthy 88% while maintaining a 22% lift in customer satisfaction (CSAT).