3 min read

Why Our Telegram Support Agent Gave the Wrong Refund Policy and How I Fixed It

Why Our Telegram Support Agent Gave the Wrong Refund Policy and How I Fixed It

Introduction

Last month our support agent in Telegram told a customer we do refunds within 14 days. The real window is 30. The customer had a screenshot and asked for the refund.

The agent was not broken. It answered confidently from whatever context it had, and our actual policies were not in that context.

Background

We run a small B2B service at 1it.pro. Three people handle around 80 Telegram messages a day: pre-sales questions, onboarding, and support. In March I put a Hermes agent in front of that queue to take the routine half.

The setup took an afternoon. I installed it, configured it in the AgentForge web UI, connected the Telegram bot, and it started replying. For the first week it looked great, because most questions were simple.

Problem

Then the answers started to drift. The agent knew our product name from the system prompt, but not our refund policy, not our SLA, not how we handle a failed deployment. It filled the gaps with plausible general text.

The root cause was boring. Our knowledge lived in four places: a 40-page PDF nobody updated, a Notion page, a pinned message in a team chat, and my head. Every policy has an exact number in it — days, hours, limits — and the agent could not reach any of them.

Why it matters

A wrong answer from an AI agent does not read like a wrong answer. It reads like a confident one. Customers act on it. One asked for a refund outside our window because we told them it was fine.

The cost is not that one ticket. It is the review. Every reply needed a human to read it before we could trust it, which killed the reason we deployed it in the first place. We were spending the same three people on the same queue, plus a new job of checking the bot.

Solution

I stopped trying to make the agent smarter and made its context smaller and more honest. Instead of one big document, I built a knowledge folder of short markdown files, each with one job.

  • 📌 instructions.md — who the agent is, what it may decide on its own, when it must escalate
  • 📌 policies.md — exact numbers only, each line with an owner and a last-reviewed date
  • 📌 examples.md — ten real question and answer pairs taken from our own chat history
  • 📌 notes.md — current edge cases, updated weekly, nothing older than a month

Every file stays under a page. If a file needs a table of contents, it is two files. The agent reads the instructions on every message and pulls policy or example fragments when a question matches.

The examples file did most of the work. Instructions tell the agent what to do. Examples show it the shape of a good answer, including the part where we say we cannot help and hand the conversation to a human.

An agent does not learn your business. It reads your context, every single time. If a number is not in a file it can reach, the agent will guess.

I also added one hard rule to the system prompt: if the answer depends on a policy number and no file matches, say so and escalate. A short apology is cheaper than a confident mistake.

Updating is now a one-file change in the web UI. When we moved the refund window, I edited one line and the next Telegram reply used it. No redeploy, no prompt rewrite.

Result

Escalations dropped from roughly half of all conversations to about 15 percent in three weeks. What is left is the work we want a human on: custom contracts, outage calls, anything with money attached.

Nobody reviews every reply anymore. We sample a few a day and check the knowledge files once a week. The refund question has been answered the same way every time since the change.

The lesson I keep coming back to: the hard part of a working agent is not the model or the deployment. It is having your knowledge written down in small, current, reachable pieces. That work pays off whether or not an agent ever reads it.

AgentForge

AI agents for real work. If you want to see how this works in practice, visit our product site and blog.

Share
Serhii

Serhii