A chatbot is a form of interaction

A chatbot is a system that communicates with users through text or voice. It may work with predefined answers or generate responses using AI. Some chatbots can also connect to a system and perform actions. For this reason, drawing a firm distinction by saying that “a chatbot only talks” would be inaccurate.

If your business wants to answer frequently asked questions, direct users to the right document, or collect initial request details, a conversational assistant may be enough to get started. Success should be measured by the accuracy of its answers, how up to date its sources are, and its ability to transfer the user to an employee when needed.

Understand the difference between a workflow and an agent

Anthropic distinguishes between workflows that follow predefined steps and agents in which the model dynamically directs the process and the use of tools. This distinction is more informative than a product name. Anthropic: Building effective agents

For example, if a service request always follows the sequence “collect information, determine the category, create a record,” a controlled workflow can be built. If resolving the request requires examining different sources and choosing the next step according to the situation, an agent-based approach can be considered. This does not mean that every complex task must be assigned to an agent.

Narrow down the choice with three questions

A system offering more capabilities is not automatically a better fit for your business. Unnecessary use of tools can increase response time, cost, and the need for oversight.

Grant action permissions gradually

Giving an assistant permission to read order information is not the same as giving it permission to cancel an order. Set separate boundaries for actions such as making payments, changing prices, or deleting customer records. You can start the first trial with read-only access and the ability to suggest actions.

It must be possible to trace on whose behalf the application acts, which record it changes, and how the action can be reversed. Asking the user for additional information when a request is unclear and requiring human approval at critical steps should be part of the design.

The same customer request can be resolved in three different ways

Consider a customer asking, “When will my order arrive?” A simple information assistant can show the delivery policy and provide the relevant order-tracking link. This approach may be sufficient if the customer is looking for a general explanation. However, reporting the status of a particular order requires verified record information; the general policy must not be confused with the actual order status.

A workflow with defined rules can first verify the customer, obtain the order number, look up the shipping record, and communicate the result in a predefined format. The order of the steps is known in advance. The design can also specify what to do when the order cannot be found or the customer information does not match. This structure can provide a clear framework for control in most standard inquiries.

With an agent-based approach, the system can choose, within defined limits, which records to examine and in what order to reach the goal. For example, in a support case, it might be asked to assess the relevant order, previous conversations, and shipping information together. Such flexibility is meaningful only when there is a genuine need for it; greater access and more decision-making freedom also broaden the scope of evaluation.

Draw a boundary between generating answers and changing records

A system telling a user, “I can create a return request,” does not show that it is authorized to do so. It must be established on whose account the action will be performed, which records the customer can access, and where approval will be obtained. A request made by the user during a conversation must not override the organization’s authorization rules.

One suggested approach is to divide actions into three groups: reading information, drafting a change, and applying the change. In the first pilot, the system can be limited to reading records and making suggestions to an employee. At a later stage, suitable actions can be carried out after explicit approval. Human oversight may continue depending on the impact of the task; not every step has to be automated.

This distinction is particularly useful in processes where there is a risk of changing the wrong record. For example, if the system cannot choose between two similar company names, it should ask for more information before acting. Successful use includes knowing when to pause in the face of uncertainty, as well as completing requests.

Assign responsibility for the information sources to be used

Both chatbots and agents can provide incorrect information if they work with outdated documents. Determine who updates sources such as the product catalog, delivery policy, and scope of services. Ask the provider when changes to a document will be reflected in the system and how the old version will be handled.

In an example support assistant, the document on which an answer is based could be visible to the employee. This makes it easier to check the information that will be sent to the user. It may not be necessary to show a long list of sources with every answer, but examining where the information comes from makes evaluation easier, particularly for internal use.

Also distinguish between adding an answer to a document and changing a business rule. The text describing the return period may be updated, while the system’s rule for determining return eligibility may require a separate change. Content, integration, and permissions should be managed as distinct responsibilities.

What questions should you ask a provider?

After a presentation claiming “We have an AI agent,” ask for a concrete example. Ask when the system chooses its own next step and when it follows a predefined path. Examine what it does when a connected system is unavailable and whether it gives the user a success message before the outcome of an action has been confirmed.

Have the provider explain pricing using your own task as an example, too. A conversation may involve numerous queries, tool calls, or handoffs to a human. The number of visible messages may not match the volume of activity in the background. Seeing the costs of typical, lengthy, and unsuccessful tasks in separate scenarios provides a more realistic comparison.

Finally, find out the terms under which your business can leave the system or change providers. Discuss how information sources, conversation logs, configurations, and integration documentation will be handed over. A good demo does not remove the need to ask about maintenance and continuity.

Do not judge success by the number of conversations

A high volume of messages does not mean that a large number of tasks have been resolved. If users repeat the same question or constantly ask to speak to an employee, the experience may deteriorate even as the conversation count grows. For this reason, track correctly completed tasks, incorrect actions, repeat inquiries, and the need for human support together.

For an information assistant, answer accuracy and appropriate routing may be the priorities. For a system that performs actions, acting on the correct record, maintaining an audit trail, and recovering from errors must also be measured. It is normal for evaluation criteria to change as the intended use changes.

Regularly review examples of success and failure for a small test group. Investigate whether employees are making extra corrections behind a conversation that appears smooth to the user. This allows you to assess the solution’s benefits across the whole workflow, rather than through the interface alone.

Test the demo with your own examples

Do not ask the provider only easy questions. Also try situations involving a missing order number, conflicting documents, an unauthorized action request, or a broken integration. A system’s ability to say that it does not know is more valuable than acting with misplaced certainty.

Base your decision on cost per answer, correctly completed tasks, human intervention, and recovery after errors. If you need a simple response system, starting there and moving to more extensive automation when the work genuinely requires it makes the investment more measurable.