Where We Started, Why We Build

We were among the earliest teams building an AI support chatbot. This was early 2023 and my last project at Instacart.

We used a state-of-the-art model called GPT-3.5. RAG felt like magic and no one’s heard about agents yet. We shipped the chatbot quickly and it was able to answer questions in English (and in Spanish too) and help users do things like reset passwords.

It felt like magic but I was not impressed with what we built. RAG is great but we quickly realized the answers are only as good as the knowledge base we put in. We had to painstakingly prune the entire knowledge base to make sure content is up to date.

Help users take actions via APIs felt like magic but each API call has to be built and hooked up by hand. In reality we spent more time dealing with cross-team and cross-functional coordination than “building in AI”.

Evaluations and reviews of the chat responses happen offline, in a spreadsheet. We download the generated chat logs and had product, legal, content and trust and safety teams review them. Extremely slow and painful process. And even as we built out an UI for this flow, the way to improve chat behaviors is still cumbersome. Based on the review results, someone needs to find the right spot to tweak agent prompt to get around the error, and hope it doesn’t break something else.

If you think this is ancient history, this is still how many teams are building these chatbots today, either with their own engineering teams or Forward Deployed Engineers.

Something feels off. A few key components were probably missing:

• Some type of feedback loop that feels as natural as speaking to humans and giving feedback and automatically improving bot responses

• An intelligence layer that helps you build support bots, refine conversation style, and set up integrations

• An integration place where inquiries, data, and policy comes together

This is what we are building and shipping now at Champ AI - the first agentic support system that has these things built-in. And we are solving each of them in the following ways:

• Feedback loop that automatically improves your automation — Just like providing feedback to humans, you just need to speak naturally and our AI will take the feedback and updates integration, prompts, and data accordingly. Unlike feedback given to humans, one feedback cycle updates your entire fleet of agents instantly. 

• Built-in AI Automation Manager, Analyst, and Tester — We built a suite of background agents that automatically keep you system data in sync, helps you identify gaps based on conversations, and generates simulated testing suite to pressure tests your configuration. 

• Customer tickets, data and policy integrated in one platform — We are starting with customer tickets, leveraging AI to find data requirements and fine tune policies over time. On data access, we are able to complete entire integration without any engineering input by tapping into read access to existing databases and browser automation.

In the end, we didn’t make a smarter chatbot; we closed the loop. Every conversation updates the facts, policies become executable, and agents reconfigure themselves with a versioned trail you can trust. Support stops feeling like prompt surgery and starts behaving like a system.