Fin unresolved conversation analysis and optimization in bulk

  • 13 April 2024
  • 3 replies
  • 98 views
L

Hello everyone,

I was wondering how Intercom users and the Intercom Team themselves are analyzing and working on optimizations to obtain higher Fin resolution rates and CSAT.

Some of these actions might include:

  • The Optimize tab in AI chatbot -> Overview -> Optimize.
  • The available Reports (Fin answer resolution rate, Fin unresolved conversation topics, Fin CSAT score, etc.)
  • Analyzing and updating Help Center content and snippets 

What are your experiences and best practices to doing so in bulk? Are there any recommended features to use for such purposes in the product itself? Can Fin be leveraged (if not now, in the future) to analyze its own unresolved conversations and provide optimization recommendations?

For now, I have only encountered external AI-based data analysis tools that could perform such analysis in bulk (let’s say I want to analyze all bot-unresolved conversations in the last quarter, there could be thousands of them). Another way to go is also manual analysis, but that can take a lot of human capacity. 

Specifically, let’s say you want to raise Fin’s resolution rate from an already better than average result to something higher. 

What are Intercom’s top recommended actions for doing so that could produce the best results, apart from the mentioned in this article?

I would like to point out that we have already reviewed all available articles in the relevant categories on Intercom and read through existing documentation and posts. 

Love to see where this conversation could go. 

Cheers, 



 

Fred Walton 13 days ago

Hi @Luka Dujmovic 

 

Fred here from the Automated and Proactive Support Team at Intercom.

 

Thank you for your question on conversation analysis.

 

It seems like you have explored many of the options available within Intercom and would like a bulk view of well organized and structured insights - Almost like a pivot table that can help with analyzing improvements. The reality is that this technology is so new within the Customer Support space, we’re still figuring it out ourselves!

 

Although there are ways to automate and focus on specific areas that can be improved (AI helps with that), the best analysis still requires some human input to review and assess Fin performance. Here are some of the effective strategies we employ at Intercom:

 

  • The Optimize tab in Fin AI Agent - This feature allows you to see unresolved questions which are grouped by AI into topics and can be organized by volume, abandonment rate, or routed to team rate. These topics are a great starting point for deeper analysis. For example, you could focus on a topic area that was driving the most volume, see the questions customers are asking, and then improve the sources Fin is using to answer those questions.
  • Leveraging custom reports:
    • Use the “Unresolved conversations” metric.
    • Segment by conversation topics. If you already use conversation topics you can enhance your reporting by segmenting your report by ‘topics’. This approach helps to organize the unanswered customer questions into relevant groups/product areas. If you haven't set up topics yet, consider creating them based on FAQs or specific product areas that your company offers end users. This will help categorize conversations and facilitate easier management. 
    • Tip: chart drill-in and exporting chart data can be used for different use cases to assess the conversation transcripts.
  • Reviewing Fin AI Agent CSAT performance and filtering by poor ratings to see specific conversations where the customer experience could be improved either through workflow setup or AI content. 
  • Examining resolution rates for content by checking the ‘Resolved’ column found in Fin AI Agent > Content. For example, you can search and filter for new content you’ve created and assess how each individual source is performing. If its ‘Resolved’ rate is over 50% you’re off to a good start! If it’s less than 50% click on the source to optimize it for AI. If it’s an article, you can also see conversations customers started after viewing this source to find out where they still needed help.
  • External in-house script built to assess, summarize and group Fin AI Agent performance based on different Product Areas.

 

We hope these practices will assist you in efficiently managing and enhancing AI interactions with your customers. In the meantime, I have sent this thread over to the relevant product teams who are continuously looking at new ways to leverage AI within the product for analysis and optimization recommendations, stay tuned for exciting additions in the future!

 

If you have any further questions or need additional assistance, please feel free to reach out.

View original

3 replies

Fred Walton

Hi @Luka Dujmovic 

 

Fred here from the Automated and Proactive Support Team at Intercom.

 

Thank you for your question on conversation analysis.

 

It seems like you have explored many of the options available within Intercom and would like a bulk view of well organized and structured insights - Almost like a pivot table that can help with analyzing improvements. The reality is that this technology is so new within the Customer Support space, we’re still figuring it out ourselves!

 

Although there are ways to automate and focus on specific areas that can be improved (AI helps with that), the best analysis still requires some human input to review and assess Fin performance. Here are some of the effective strategies we employ at Intercom:

 

  • The Optimize tab in Fin AI Agent - This feature allows you to see unresolved questions which are grouped by AI into topics and can be organized by volume, abandonment rate, or routed to team rate. These topics are a great starting point for deeper analysis. For example, you could focus on a topic area that was driving the most volume, see the questions customers are asking, and then improve the sources Fin is using to answer those questions.
  • Leveraging custom reports:
    • Use the “Unresolved conversations” metric.
    • Segment by conversation topics. If you already use conversation topics you can enhance your reporting by segmenting your report by ‘topics’. This approach helps to organize the unanswered customer questions into relevant groups/product areas. If you haven't set up topics yet, consider creating them based on FAQs or specific product areas that your company offers end users. This will help categorize conversations and facilitate easier management. 
    • Tip: chart drill-in and exporting chart data can be used for different use cases to assess the conversation transcripts.
  • Reviewing Fin AI Agent CSAT performance and filtering by poor ratings to see specific conversations where the customer experience could be improved either through workflow setup or AI content. 
  • Examining resolution rates for content by checking the ‘Resolved’ column found in Fin AI Agent > Content. For example, you can search and filter for new content you’ve created and assess how each individual source is performing. If its ‘Resolved’ rate is over 50% you’re off to a good start! If it’s less than 50% click on the source to optimize it for AI. If it’s an article, you can also see conversations customers started after viewing this source to find out where they still needed help.
  • External in-house script built to assess, summarize and group Fin AI Agent performance based on different Product Areas.

 

We hope these practices will assist you in efficiently managing and enhancing AI interactions with your customers. In the meantime, I have sent this thread over to the relevant product teams who are continuously looking at new ways to leverage AI within the product for analysis and optimization recommendations, stay tuned for exciting additions in the future!

 

If you have any further questions or need additional assistance, please feel free to reach out.

L

Thank you for the exhaustive answer @Fred Walton! It’s really helpful, and it shows me that we are going in the right direction (as I have been using most of these reports already for doing just that, but it’s quite a manual process for now). As you said, it’s new ground, and we are all still finding efficient ways to use extract, process, and use the data.

As a product feature suggestion, I would highly recommend you consider implementing GPT for such types of data analysis within the product itself. You already have the data, you have Fin, and you would just need to leverage its power for internal users who are looking for these kinds of answers. Another user made the same suggestion already here. 

 

Fred Walton

Hi @Luka Dujmovic, thought I’d follow up here.

While we are using GPT internally to carry out specific analysis to improve Fin AI Agent’s performance - this is an area where a lot of iterative updates are happening. The team are focussed on a new pillar of our AI first platform, ‘AI Analyst’. This release aims to resolve the issues that you are seeing.

AI Analyst was mentioned in our recent product broadcast, you can find out more by viewing this timestamped announcement video.

Reply


Terms & ConditionsCookie settings