Skip to content

Conversational AI vs. Traditional Chatbot: The 15-Minute Test

Four practical tests to find out whether your bot only answers or actually resolves: free-text language, system lookup, result verification and handoff with a summary.

Marlos Carmo

Founder of Tolky

Updated on November 15, 2026

6 min read
Conversational AI vs. Traditional Chatbot: The 15-Minute Test

Share

Almost every service platform now says it has "AI". The real difference shows up when you put the bot in front of a concrete situation. This article does not repeat the concept (for that, read Conversational AI is not a chatbot). The goal here is practical: a 15-minute test you can apply to any vendor, Tolky included.

Two models, one rule of thumb

  • Fixed-flow chatbot (rules): the conversation design is mapped in advance. Menus, buttons and keywords. Predictable and easy to audit, but rigid outside the script.
  • Conversational AI: uses language models to understand what the customer meant, query agents and systems and act within business rules.

In practice, the market is hybrid. Platforms such as Manychat and Blip combine visual flows with AI steps, and CRM suites (HubSpot, Salesforce, RD Station) already offer AI agents. So the useful question is not "does it have AI?" but "where does the AI come in and what can it do?".

The 15-minute test

Ask for a demo (or use the test environment) and run the four tests below. Score each from 0 to 2.

Test 1: free text (3 minutes)

Write the way customers write: with typos, slang and, if the channel allows, a short voice message. Example: "send me this months invoice, its due tomorow".

  • 0: the bot returns the menu or "I didn't understand".
  • 1: it understands, but only with exact words or loses context by the second message.
  • 2: it understands the intent, asks only for what is missing and keeps context.

Test 2: a request that requires a system lookup (4 minutes)

Ask something whose answer exists only in your system: an order status, an invoice amount, a deal stage.

  • 0: it answers with generic instructions ("go to the portal").
  • 1: it queries a knowledge base (RAG), but not the customer's own data.
  • 2: it queries the system (via API or integration) and answers with that customer's real data, after identifying them.

Test 3: check the result in the system (4 minutes)

Request an action (open a ticket, update a field, log an interaction) and go check the system, not the conversation.

  • 0: the bot says it did it, but nothing shows up in the system.
  • 1: it records something, but without category, owner or history.
  • 2: the record exists, is correct and is linked to the conversation. Sensitive actions (amounts, refunds) go through rules and authorization limits.

Test 4: handoff with a summary (4 minutes)

Ask to speak to a person, or simulate irritation and a threat to cancel.

  • 0: the conversation drops into the queue with no context, and the customer repeats everything.
  • 1: it transfers with the raw transcript.
  • 2: it transfers with a summary, the reason and the data already collected, so the agent continues where the bot left off.

How to read the score: from 0 to 3 you have a fixed-flow bot or an FAQ with a chat interface. From 4 to 6 there is AI, but it is still disconnected from operations. From 7 to 8 it is operational conversational AI.

The test works for any vendor. The decision criteria for the IT team are in how to choose an enterprise AI automation platform.

When a rules-based chatbot is still the right choice

Not every flow needs AI. Fixed flows still make sense when:

  • the process is regulated and each step must be identical and auditable;
  • data collection is fixed (for example, a registration form with required fields);
  • volume is low and predictable, and maintaining a simple flow costs less than configuring an agent;
  • you want a three-button entry menu, with AI coming in only from the option that requires interpretation.

The combination is often best: a simple menu at the door and an integrated agent behind the option that calls for lookup and action.

What conversational AI does differently, in practice

  • Free text: the customer writes however they like; the AI interprets intent instead of searching for keywords.
  • Service memory: conversation and customer history live in the CRM, and the agent uses it to answer (memory comes from the recorded history; vector databases serve to search the knowledge base).
  • Action with rules: the agent can query and update systems via API, within defined permissions and limits. A product exchange request, for example, can be taken as far as the payment link for the difference if the business rule allows it and the amount is within the authorization.
  • Smart handoff: when the case falls outside scope or the tone calls for a person, the human agent receives the summary.
  • Several specialized agents: sales, support and collections can have distinct agents working in the same environment.

At Tolky, these elements live alongside CRM, tickets and the human team in the same panel, with service on WhatsApp, webchat, Microsoft Teams and voice. Integration with your systems is done via API, depending on the project. To see what each plan includes, check pricing, and if you are comparing vendors, see the comparisons.

From test to calculation: return on your own assumptions

Instead of a generic ROI, use your own math:

Estimated monthly savings = (cost per human interaction × monthly volume × % resolved by the agent) − the platform's monthly fee

The resolved percentage is the variable that weighs most, and only a pilot with your real conversations gives that number. Also track repeat-contact rate and CSAT, not just volume. For the full method, read AI automation ROI: how to measure the return. Plans and service allowances are on pricing; Meta's WhatsApp fees are charged separately.

Run the same test on Tolky

Bring your four hardest conversations, the ones your current bot cannot resolve, and run the tests in the demo. You leave with a score and a map of what would need to be integrated to reach 8.

Book a demo and take the 15-minute test. If you prefer to start on your own, see the plans.

Share

About the author

Marlos Carmo

Founder of Tolky

Marlos Carmo is an AI entrepreneur and founder of Tolky, the conversational-era infrastructure and AI CRM that unifies intelligent service, multi-channel support (such as WhatsApp and voice), live CRM, and operational intelligence in a single ecosystem. He is a finalist for the SXSW Innovation Awards and a member of Francesco's Economy, a global network of young entrepreneurs focused on innovation and social impact. He works connecting Artificial Intelligence and digital transformation in projects for large organizations.

Next step

Want to bring this to your operation?

Tati, Tolky's AI assistant, understands your scenario in a few questions, suggests the best path and connects you with someone from the team, with full context.

Read also