Skip to content
/Tutorials

August 2nd, 2021

Prioritise Labelling with the UnexpecTEDIntentPolicy

  • portrait of Vincent Warmerdam

    Vincent Warmerdam

Let's say you've got an assistant running in production. It turned out that it's a popular interface so lots of users are interacting with it. You'd like to learn from all these interactions so you set yourself up to start labelling.

This can be intimidating though. Especially when your assistant is popular, you're bound to have many conversation logs to go through. Even with a large team, going through every conversation can be a daunting task.

This is what it might feel like.

Instead of going through every conversation, it would be better to have a system that can help you prioritize which conversations to label first. There's nothing wrong with labelling conversations that give a good user experience but they will likely only confirm the situations that your assistant can handle. It'd be much better if we could find examples where the assistant falls short because these examples might shed light on areas where the assistant could improve.

With that in mind, we're interested in finding conversations where the user did something unexpected. The idea here is that these conversations are more likely to teach you how to improve the assistant than a conversation that went as expected.

It'd be great if we knew what conversations to focus on.

To help Rasa users who face this problem we've recently released a new policy that does exactly this. It's called the UnexpecTEDIntentPolicy and it's available as of Rasa 2.8.

How Does it Work?

The UnexpecTEDIntentPolicy uses a model architecture which is a slight variation of TED (hence the name!) to detect conversations where an unexpected intent occurred. The model looks at the story path in the conversation so far and as well as the currently observed intent to figure out how likely it is to see the intent in the current conversation according to the training data which the assistant was trained on.

If this similarity is too low and exceeds a threshold, the policy will emit an action_unlikely_intent-action. This action is logged and can therefore be used in hindsight in Rasa X to help you label conversations.

How would you use it?

If you want to use the UnexpecTEDIntentPolicy you'll need to add it to your config.yml file first.

policies:
- name: UnexpecTEDIntentPolicy
epochs: 200

When you now train a new model it will contain the UnexpecTEDIntentPolicy that will emit the action_unlikely_intent-action. This action will be logged just like any other action in the conversation which means that you can use it for filtering in Rasa X. You can find the action in the "Actions" submenu when you filter.

You can find the action_unlikely_intent in the filter tab.

Once the filter is active you'll be able to focus on the conversations that have an action_unlikely_intent in them. Since something unexpected happens in these conversations these conversations would be good candidates to prioritize.

When the action_unlikely_intent appears it's good to check what went wrong.

Learn More

You can learn more about the UnexpecTEDIntentPolicy on our documentation or on the Algorithm Whiteboard youtube channel. We've made three videos that explain all the details of the new system.

We hope you'll be able to use this system to find conversations that improve your assistant. If you have any feedback on the algorithm, feel free to let us know on our forum.