An AI assistant on a website can answer questions, recommend a suitable service, work with available appointments or guide a customer through an order. None of this automatically means that it is producing a measurable business result.
To understand its actual value, counting opened chats or sent messages is not enough. We need to know what the visitor wanted to achieve, how far they progressed and whether they successfully completed the intended action. In our first article, we explained the difference between a chatbot and a connected AI assistant. Now we will look at how to determine whether the solution is working.
First, define what success means
Every business may expect a different outcome from an AI assistant. A cinema may care about completed ticket orders. A medical practice may focus on booked examinations. A service company might measure qualified enquiries, while a customer support team may track requests resolved without staff involvement.
Before deployment, the business should therefore select one primary outcome. It should be specific and technically measurable. Instead of a broad objective such as “improve communication”, it is better to measure:
- a completed order or booking;
- a submitted qualified enquiry;
- a successful search for a suitable appointment;
- a request handed to the correct member of staff;
- a common question resolved without further contact.
If success has not been clearly defined, even a large number of conversations will not reveal whether the assistant is genuinely helping the business.
Measure the entire journey, not only the final step
Several steps take place between opening a chat and completing an order. Measuring them shows where customers continue and where they leave the process. A simple measurement funnel might look like this:
- the visitor opened the chat;
- they asked a relevant question;
- they showed interest in a specific service;
- they started the booking process;
- they selected an appointment or product;
- they submitted the required information;
- the booking system confirmed completion.
This breakdown helps identify the actual source of a problem. If many people begin a booking but do not finish it, the chat itself may not be responsible. The cause might be limited availability, unclear conditions, a technical error or the amount of information requested.
Completion should be confirmed by the system in which the order or booking is genuinely recorded. A customer writing “I would like to book” is not yet a completed booking.
Which metrics have practical value?
The exact metrics depend on the process, but several indicators are useful across most deployments.
Relevant conversations
Not every opened chat represents genuine interest. Test messages, spam and accidental openings without a question should not be mixed into the main evaluation.
Booking processes started
This figure shows how many visitors moved from asking a question to taking a specific action.
Completed orders or bookings
An order should only be counted when it has been confirmed by the system in which it is actually recorded. Reports should also state whether “completed” means created, confirmed, paid or later fulfilled. These stages are not interchangeable.
Completion rate
Calculate it as completed bookings ÷ booking processes started × 100. The metric shows what proportion of people who started the process also completed it. It is not a universal benchmark. The result is influenced by the industry, price, availability and complexity of the decision.
Human handover rate
A handover is not automatically a failure. It may be the correct outcome for a sensitive, unusual or unclear request. Reports should distinguish between an intended handover and a situation in which the assistant failed to answer a routine question.
Unresolved and incorrect answers
The business should regularly review cases in which the assistant did not respond, supplied inaccurate information or produced an answer the customer could not understand. A small number of serious mistakes can matter more than a large number of successful answers to simple questions.
Without a comparable baseline, an increase cannot be established
If an assistant processes 50 orders in one month, it is safe to say that 50 orders passed through the assistant. It is not automatically safe to claim that the business gained 50 additional orders. Some customers might have ordered without the assistant by using a form, telephone or another channel.
A comparison should account for:
- website traffic and traffic sources;
- marketing campaigns and seasonality;
- available appointments or capacity;
- changes to prices and services;
- a consistent definition of a completed order.
Where technically and commercially appropriate, a controlled test can compare a version with the assistant against one without it under equivalent conditions. If such a test is unsuitable, results should be reported as the volume processed through the assistant, not as proven incremental growth.
Performance must be assessed alongside quality and safety
An AI assistant could achieve a high completion rate while still creating problems if it supplies inaccurate information or directs customers to the wrong service. Evaluation should therefore include a regular review of a sample of conversations. This review can assess whether:
- the answer was based on current information;
- the assistant correctly understood the customer’s intention;
- it avoided offering an unavailable appointment;
- it acknowledged uncertainty when appropriate;
- it handed sensitive requests to a person correctly;
- it avoided collecting information that was unnecessary for the task.
For AI assistants used by clinics and diagnostic centres, the boundary between organisational assistance and medical advice must remain clear. An assistant may help someone choose a booking category, but it must not pretend to diagnose a condition or replace a professional examination.
Measurement should use only the information required for the defined purpose. The business should understand which events are recorded, how long the information is retained and whether useful analysis can be performed without unnecessarily storing conversation content or personal data.
How to estimate the financial value
Once the technical metrics are available, the next question is commercial: does the solution justify its cost? A basic model may consider the value of completed and subsequently fulfilled orders, demonstrable staff time savings, installation and integration costs, monthly operating costs and ongoing management.
The calculation should distinguish between an order and actual revenue. An appointment can be cancelled, and a created order may never be paid. Where possible, the later status of the order should therefore be measured rather than recording its creation alone. Time savings are most useful when based on a defined task and a comparable view of staff effort before and after deployment.
A practical measurement plan for deployment
Before the assistant goes live, agree on:
- one primary business outcome;
- the precise definition of a completed action;
- the events that will be recorded;
- a baseline period or another comparison method;
- quality review rules;
- a regular evaluation schedule.
After launch, first verify that events are being recorded correctly. There is little value in evaluating results before the measurement itself has been tested. The initial operating period may also reveal new customer questions, unclear website content or parts of the booking process that require improvement.
A useful report should always state the measurement period, data source, definition of the outcome and limitations of the analysis. This gives the business more than an impressive headline figure: it provides information that supports a genuine decision. You can see transparently presented outcomes in our portfolio.
A good AI assistant should produce a demonstrable result
A successful AI assistant is not the one that generates the largest number of messages. It is a solution that safely helps customers complete the intended task while giving the business enough reliable information to evaluate its performance honestly.
Measure from day one
What could be measured on your website?
Send us a link and a short description of your current customer journey. During a free initial consultation, we can propose a specific objective, the required integration and a suitable measurement approach.
Book a free consultation