From Chatbot Demos to Self-improving Systems

The launch of ChatGPT spurned off a great race to build with AI and chatbots were the first real application that everyone was tinkering with. In 2023, we were among the earliest teams building a chatbot for customer support at our previous employer.

GPT-3.5 was the best model at that point. RAG was all the rage and no one’s even heard about agents yet. We shipped the chatbot quickly. It was able to answer questions in English and helped users do basic things like reset passwords.

It showed lots of promise but we were not quite impressed with what we built. RAG was a good technique to build chatbots but the chatbot performance is only as good as the knowledge base it has access to. We had to painstakingly prune and update the entire knowledge base to make sure content is up to date.

And once we got past the low hanging informational queries, we spent a lot of time understanding all the Standard Operating Procedures our support team had, translating them into actionable AI instructions and working with different Eng teams to build the right APIs to make the SOPs work.

Evaluations of the chat responses happened offline, in a spreadsheet. We downloaded the generated chat logs and had product, legal, content and trust and safety teams review them. Based on the review results, we had to find the right spot to tweak the prompt to get around the error, and hope it doesn’t break something else. It was extremely slow and painful!

In reality we spent more time dealing with cross-functional coordination than “building in AI”. If you think this is ancient history, this is still how many teams are building these chatbots today, either with their own engineering teams or Forward Deployed Engineers.

This way of building with AI was just too inefficient for us. A few key components were probably missing:

• Some type of feedback loop that feels as natural as speaking to humans and automatically improves the AI

• An intelligence layer that helps you turn SOPs into AI agents, refines conversation style, and builds a memory layer

• An integration suite where internal systems, external data and policies come together

This is what we are building and shipping now at Champ AI - the first agentic support system that has these things built-in. And we are solving each of them in the following ways:

• Turn feedback into instant improvements — Our AI will take your feedback in natural language and updates integrations, prompts, and memory accordingly. Unlike feedback given to humans, one feedback cycle updates your entire fleet of agents instantly. 

• AI Manager, Analyst, and Tester — A suite of background agents that automatically keep your system data in sync, helps you identify gaps based on conversations, and generates simulated testing suite to pressure test your configuration. 

• Customer tickets, data and policy integrated in one platform — We are starting with customer tickets, leveraging AI to find data requirements and fine tune policies over time. On data access, we are able to complete entire integration without any engineering input by tapping into read access to existing databases and browser automation.

At Champ AI, we’re not just building yet another chatbot. We’re building the first agentic support system -where feedback, policies, data, and conversations all connect into one living, self-improving loop. That means no more prompt surgery, no more endless spreadsheet reviews, and no more duct-taping APIs just to keep up.

Support finally feels like a system: one that learns, adapts, and earns your trust with every interaction. This is the shift we wished we had back then, and it’s the one we’re shipping now.