Not every chatbot needs a language model behind it. The decision affects cost, predictability, and how much ongoing work the bot needs.
"Chatbot" covers two genuinely different things now, and the distinction gets blurred in casual conversation. A rule-based bot follows a fixed decision tree — if the user says X, respond with Y. An LLM-powered bot understands intent more flexibly and generates responses dynamically. Neither is strictly better; they're suited to different shapes of problem.
Most production bots end up as a hybrid rather than a pure pick of one approach, and it's usually a cleaner design than either extreme. Deterministic tasks — "where's my order," "what are your hours," "cancel my subscription" — go straight to rule-based lookups against a real system, because the answer is a known fact and a rule is faster, cheaper, and impossible to get wrong. Everything that doesn't match a known pattern falls through to the LLM layer, which handles the open-ended half of the conversation and hands back to a rule the moment it recognizes a structured task hiding inside a messy sentence.
The dividing line is usually "does this have one correct answer that lives in a database." If yes, a rule gets there faster and can't hallucinate it. If the question requires actually understanding what the customer is asking, that's the LLM's job. Most teams that start pure-LLM end up adding rules back in for exactly this reason — not because the model was wrong, but because a rule is simply the right tool for a lookup.
A rule-based bot matches user input against predefined patterns or button-driven menus and follows a scripted conversation flow. There's no model generating novel responses — every possible path through the conversation was designed in advance. That makes behavior completely predictable: the same input reliably produces the same output, and every possible conversation path can be tested and audited.
The tradeoff is brittleness outside the scripted paths. A question phrased slightly differently than anticipated, or one that doesn't fit any predefined branch, either fails silently or forces an awkward "I don't understand" response. Rule-based bots work best when the set of things users ask is genuinely narrow and predictable.
An LLM-powered bot uses a language model to interpret the user's actual intent, regardless of exact phrasing, and generate a response dynamically — typically grounded in your own content via retrieval so it's answering from real information rather than guessing. It handles the long tail of ways people actually phrase questions, which a fixed decision tree can't realistically cover.
The tradeoff is predictability and cost. Responses are generated, not scripted, so testing needs to cover behavior patterns rather than exact conversation paths, and every conversation carries a real inference cost that a rule-based bot doesn't have. It also needs genuine guardrail work — grounding, escalation rules, tone control — to behave reliably in production.
The set of things users ask is genuinely narrow and well understood.
Complete predictability and easy auditability matter more than natural conversation.
Per-conversation cost needs to stay as low as possible at high volume.
Users phrase the same question in genuinely varied, unpredictable ways.
The knowledge base is large enough that scripting every path isn't realistic.
The value of resolving more conversations outweighs the added inference cost.
Yes, and it's a common evolution — many teams start with a rule-based bot to handle the highest-volume, most predictable questions, then add LLM-powered handling for the long tail of questions the rules don't cover.
Per-conversation, usually yes — there's inference cost involved that a rule-based bot doesn't have. Whether that's actually more expensive overall depends on how much human agent time it saves, which is often the larger cost in the comparison.
Not necessarily. If the questions are genuinely predictable and the phrasing variation is limited, a well-built rule-based flow can handle them reliably and more cheaply — the LLM earns its cost when the phrasing and intent variation gets wide enough that rules can't keep up.
Rule-based bots cost more to maintain as scope grows — every new scenario means new rules — while LLM-powered bots scale to new questions without a proportional maintenance increase, at the cost of ongoing API spend.
Yes, and it's a common pattern — rules handle deterministic tasks like order status lookups, while an LLM layer handles open-ended conversation and hands off to rules when it recognizes a structured task.
A practical checklist for taking a chatbot from a working demo to a production deployment that holds up against real customers.
Read the guide