Why most demos look the same

Every chatbot demo opens with the visitor asking a question the bot was prepped for. The answer comes back fast, the styling looks clean, the conversation feels natural. That's the demo. Two weeks after a real install on your site, the same bot is fabricating prices, asking the same visitor for their email five times in one conversation, never sending you a lead notification, or quietly burning through a per-message budget you didn't know you had.

The five questions below are designed to surface those failure modes before you sign up. None of them require technical knowledge to ask, but the answers tell you whether the vendor has thought through the problems that actually break small-business chat in production.

1Where does the answer come from?

Ask the vendor: "Does the bot answer only from content I provide, or can it search the open internet, or generalize from what the underlying language model already knows?"

The right answer is some version of only from content you provide. Your website, your FAQ, your policy docs, optionally a connected inventory feed or calendar. Nothing else.

Two failure modes hide behind any other answer. The first is fabrication: a bot that's allowed to generalize from training data will confidently invent prices, hours, return policies, and services you don't offer. The second is open-internet surface: a bot that can search the web is one Google result away from quoting your competitor's pricing at your visitors. Neither is acceptable on a small-business site.

The fifteen-second test: ask the demo something it shouldn't know. A price you never published. A service you don't offer. A competitor's policy. A grounded chat assistant admits it. An ungrounded one invents a confident, plausible-sounding answer. That single test tells you more than the sales pitch.

2What happens when the bot can't answer?

Ask: "When a visitor asks something the bot doesn't have an answer for, what happens next?"

Two answers are unacceptable. "It guesses based on similar businesses" is the fabrication failure mode from question 1. "It says 'I don't know' and the conversation ends" is a dead-end: the visitor leaves, you never hear about it, and the lead is gone.

The right behavior is a chain. The bot acknowledges the unknown honestly. It offers to have a real person follow up. It asks for the visitor's name. Then it asks for their email or phone. Then it fires one notification to you with the full transcript so you can pick up the thread on your terms. This is lead-recovery behavior. It's also where most small-business chat ROI actually comes from: the visitor who had one off-FAQ question and would have bounced without it.

Verify by asking the demo something obscure about the vendor's own demo business and watch what happens. Do you get an email afterwards? If not, ask the vendor where that email would have gone in a real install.

3How am I billed, and who absorbs the cost when the model has a bad week?

Pricing models for chat assistants fall into three shapes, and only one of them protects a small business from surprise bills.

The diagnostic question: "If the model my bot uses gets deprecated next quarter and the replacement costs more or behaves differently, what changes for me?" A flat-fee vendor's answer should be "nothing — that's our problem." Any other answer means you're going to be paying surprise bills, doing engineering work you didn't sign up for, or both.

4How does the bot decide when to ask for contact info — and when to stop?

This is the policy-layer question, and it's the one most chatbot vendors fail.

Symptoms of a bot without a real policy layer: it asks for the visitor's email three turns in a row, ignores a "no thanks, just browsing" reply, keeps pitching after the visitor says they're not buying, fires four duplicate lead notifications when the same person mentions their email twice in one chat. These aren't edge cases. They're the routine failures of trying to control behavior with prompt instructions to a language model.

The right answer sounds like: "A rule-based decision per turn, separate from the language model. The model writes the words; a layer of code decides what shape the next reply should take." Should the bot just answer? Answer and offer to connect? Ask for a name? Wind down because the visitor said no? Acknowledge frustration before anything else? These are policy decisions, made deterministically, the same way every time.

A specific marker that separates careful systems from thin wrappers: ask whether the bot can qualify a visitor with a few criteria questions before sharing your calendar link, and route the not-a-fit visitors to a polite decline (or a helpful next-best resource) instead of the booking page. Pure-prompt bots can't reliably do this. A real policy layer makes it deterministic: the calendar link literally cannot be shared until the criteria check completes.

5What can I see in the dashboard a week after launch?

A chatbot is only as useful as your visibility into what it's doing. The dashboard is where you find out whether the bot is helping or quietly failing.

A serious vendor gives you:

Red flags: "We'll send you a monthly summary" (manual reports are not real reporting). "You can ask us anytime" (you shouldn't need to). No dashboard at all (you'll find out the bot stopped working when traffic dries up).

The bonus question: who answers when the bot can't?

One more, because most buyers forget it: "Can I jump into a conversation live if I'm at my phone and see a hot lead?"

The pure-AI chatbot is a useful default. But the moment a visitor with money asks something nuanced, you want to be able to take over the keyboard yourself for two minutes. A vendor that doesn't support that is selling you a fire-and-forget bot. A vendor that does support it is selling you a chat system where the bot handles 80% and you handle the 20% where it matters.

How Simple Business Bots answers each of these

For the sake of being direct about where we fit:

If a competitor can answer all five questions the same way we do, that's a good vendor and you should pick the one whose UI and support model you prefer. If they can't, you've saved yourself a migration in six months.

Related reading