Using Conversation Tags to Measure Carbon bot’s Success Rate

At Rasa, we have a couple of assistants we run in production to test new features and dogfood the product-that is, gather feedback by using the product ourselves. This lets us stress-test updates in a production environment and gain experience in building and maintaining an assistant the same way Rasa users do.

One assistant we maintain is Sara, which is embedded in the Rasa docs to help developers get started and learn about Rasa. Another I've been working on is Carbon bot, which encourages people to offset the carbon emitted by flying. If you're curious, you can chat with Carbon bot on Messenger and see the source code on GitHub.

I use Carbon bot to try out new algorithms, like the TED policy, and also to alpha test new features in Rasa Open Source and Rasa X. One of the themes we've been thinking about is conversation success. That is, can you tell and keep track of how many conversations are successful?

I'm increasingly convinced that you need external information to help decide which conversations were successful. Just looking at the messages is never quite enough information. Carbon bot will send users a link where they can buy offsets, so I decided to start by tracking how many people actually click on that link. It's not perfect, but a decent proxy for measuring conversion.

I was excited to see Apple publish a paper last year on leveraging what they term user engagement signals-events like tapping on content or listening to a song in full-to understand the user's true intent. Interestingly, they go one step further by feeding these signals into a framework for labeling training data. To extend the concept to a common Rasa use case, customer support, you might track whether the user initiated another customer service contact within a 24 hour window. If the customer doesn't come back right away, it might indicate that they were satisfied with the help they received from the bot.

In all of these cases, the information about success is coming from the outside world, so we added an API endpoint to Rasa X that lets you programatically add tags to conversations. These are my notes on how it works in Carbon bot. I'll also give a preview of what's next for conversation tags in the upcoming 0.26.0 release of Rasa X.

Measuring success

It won't be surprising to anyone who's built an AI assistant that measuring success can be subjective. Success can mean a lot of different things:

• Did the assistant correctly classify a message's intent and extract the right entities?
• Was the right response selected?
• Was the user's sentiment positive?
• Did the assistant help the user achieve their goal?

It's the last metric that we're tracking with conversation tags. Carbon bot's purpose is to educate travelers and persuade them to purchase carbon offsets. If the user clicks the link to purchase offsets, that's a pretty good approximation of helping the user achieve their goal. We can't necessarily track whether the user completed their purchase, but we can assume the user was interested enough to check it out.

With conversation tags, you can apply labels to conversations in Rasa X and use the label as a filter. Currently, tags can be created using the Rasa X REST API, allowing you to apply tags to conversations programmatically. This is especially useful for tracking user behavior that happens in your application, including behavior that isn't visible from the conversations themselves. Any event that can be tracked in your code, like a click or follow-up contact, can be used as a trigger for tagging a conversation. Once conversations have been tagged, you can write a script to filter and measure tagged conversations to extract useful data about how often the target event occurs, and you can filter conversations by tag in the Rasa X UI to review the transcripts.

How it works: Tagging conversations in Carbon bot

Enough talk-let's look at the code!

Our goal is to tag all of the conversations in Carbon bot where the user clicked the link to purchase carbon offsets. Here's the basic process:

The link users click to visit the carbon offset site actually routes to a static HTML page along the way. The HTML page runs a script to send a request to the Rasa X API, tagging the conversation, and then the user is routed on to the carbon offset site. This process happens without any disruption for the user.

First, we use a custom action to build the link destination URL. There are four important parameters: the URL of the static HTML page, the ID of the current conversation, and the URL for the carbon offset site. We also add a parameter with the label the conversation should be tagged with, e.g. link-1-clicked.

url = f"https://rasa.com/carbon/index.html?" \
f"&rasaxhost=https://carbon.rasa.com" \
f"&conversationId={tracker.sender_id}" \
f"&destination=https://offset.earth%2F%3Fr%3D5de3ac5d7e813f00184649ea"

link_1_url = url + f"&label=link-1-clicked"

The final destination URL looks something like this:

https://rasa.com/carbon/index.html?&rasaxhost=https://carbon.rasa.com&conversationId=72058939-a83c-4a31-87fd-f27de0&destination=https://offset.earth%2F%3Fr%3D5de3ac5d7e813f00184649ea&label=link-1-clicked

The static HTML page is hosted on the Rasa website at https://rasa.com/carbon/index.html. When the user hits that page, a simple script extracts the conversation ID and tag label from the URL and uses it to send a request to the conversation tags endpoint. In the example below, we're tagging the conversation with the label link-1-clicked.

<script>

// extract Rasa X host, conversation ID, and tag label from URL parameters
    const urlParams = new URLSearchParams(window.location.search);
    const rasaxhost = urlParams.get('rasaxhost');
    const destination = urlParams.get('destination');
    const conversationId = urlParams.get('conversationId');
    const label = urlParams.get('label');
    
// build request body and request URL for API request
    const payload = JSON.stringify([{"value": label, "color": "aaaaaa"}]);
    const url = `${rasaxhost}/api/conversations/${conversationId}/tags`;
    
// send API request
    try {
        const Http = new XMLHttpRequest();
        Http.open("POST", url, false);
        Http.send(payload);
    }
    catch(error) {
    console.error(error);
    }
    
// redirect user to their final destination (carbon offset site)
    window.location.replace(destination);
</script>

Tags are created using the /conversations/{conversation_id}/tags endpoint, with the value of the tag label and the color hex code specified in the request body. You can filter by tags in Rasa X as of version 0.25.2. Check out the API documentation for full details.

Tracking conversions

Now that the conversations that "convert" are automatically tagged, I can use a script to pull the data from the API and calculate the conversion.

I could also do fun things like build a model to try and predict which conversations are likely to convert, but that's for another day.

What's Next

So far, we've released full API support for conversation tags and we've added the ability to filter tags in the UI. In an upcoming release, we'll be adding full support for conversation tags in the Rasa X UI, which will include creating new tags and editing existing ones.

What will creating tags in the UI allow you to do? It'll open up new use cases beyond what we've discussed here. While the API is great for tagging conversations programmatically, tagging in the UI means team members will be able to label conversations as they review them in Rasa X. This means you'll be able to categorize conversations by criteria that may require a bit of human judgement.

Stay tuned for more announcements on conversation tags in the coming weeks!

Conclusion

Our work with Carbon bot helps us test new features and ideas. We built Rasa X on the belief that real conversations are the best way to improve AI assistants, and conversation tags are a useful tool for organizing and prioritizing conversations. Long term, we'll be investing in more features that help product teams review conversations, make improvements, and understand the success rates of their assistants.

Keep up with news about conversation tags and other new features in Rasa X by subscribing to the newsletter, checking the change log, and following us on twitter.