Fin's flawed assumed resolved & pricing design

I’ve noticed the same behavior with Fin taking credit for interactions when someone intervenes and also when a timeout happens, and the customer ends up being passed to a teammate. I’ve brought this up to Intercom’s product team before. While I can understand why it’s set up this way, my argument is: can Fin recognize when the interaction happens after the timeout and not take credit for it? For example, similar to how CSAT surveys are only sent after 250 characters, maybe Fin could exclude interactions where a timeout occurs, and the agent handles most of the resolution after a certain point or a certain amount of characters occur between a teammate and customer. 

​@bosbeest According to Intercom, they DO NOT count assumed resolutions, and do not bill you for those if the same user reaches out and talks to a teammate within 24 hours, on the same ticket or another ticket. I was told this on a call with them months ago, but I haven’t done too much looking and investigating with that to confirm. But if this is the case, you probably aren’t being billed for those like you’re assuming? 
I too wish we had some additional options to help with those drop-offs though. Especially when I’m not super happy with Fin’s answer. 

I’ve been putting a lot of effort into fixing all of Fin’s answers with more robust knowledge content, and overall it’s been a great experience for us!

Not a quick fix by any means! but in my company we do robust chat audit of 100’s of replies from Fin which were not great. We go through the chats and pick up on trends and topics that Fin repeatedly fails at. We then add or update the content. This is not going to immediately help with your problem of Fin thinking its answered something correctly, however, over time it definitely reduces the number of chats answered incorrectly. I do find that most issues my Fin bot has encountered can be somewhat fixed by really doubling down on the content. 

We experience this as well and I can tell you what we’ve done. 

1. The hard truth is, yes – your customers are going to get frustrated at first until you train Fin better. We segmented our customer base to who we were willing to take that risk with and filtered out our VIPs. It was hard for our team to watch at first, but we asked that they wait until the customer click “talk to a human” before they intervened. Then, we made sure to “Add a snippet” to correct the misinformation once the issue was resolved. We’re now at close to a 30% Confirmed Resolution rate and have filtered out customers we know only write in with requests that Fin can’t handle. 
2. I agree that “Assumed Resolution” is a misnomer. We do still have agents stepping in before the customer asks to speak to a person because the guidance provided was wrong. I don’t think you are be billed for those resolutions, but the category name should change to better describe the behavior we’re seeing. 
3. If you haven’t yet, you *need* to revise your knowledge base. Fin can only be as good as the information you provide it, so making sure your knowledge base is up to date and easy to understand will greatly improve your performance. 

Here is a post on Reddit may provide you some options: https://www.reddit.com/r/SaaS/comments/1ibeigd/ai_automation_options_on_intercom/

Something that drastically improved our Confirmed Resolutions and overall accuracy was turning on “Content From Conversations”. It scanned all of our previous conversations and generated it’s own snippets from there. 

I think that took us from 15% to almost 30% Confirmed Resolutions in a very short timeframe. 

Hi Trevor, if I understand you correctly - in your case a customer reaches out again. In my case there is no second interaction with the same client. We simply answer the customer's question before they can click “Talk to Human” because we see it's wrong. This is counted as an assumed resolution and this is most definitely being billed. 

Wow I could have sworn this was already taken into account by default! I thought this was one of Fin's greatest selling points and I thought I'd seen it use older conversations before 🤷‍♂️ I've enabled it now and it will be learning from the conversations of the past 30 days. Wish that could be the past year though!

It's clear that more, strong content is the way to go for now. Still don't like the way they count nonsense resolutions though. 

This weekend brought a new situation to light where Intercom's “assuming” of AI resolutions is badly designed and unfair (for us, not for them $$$). 

We had an outage in certain region of the world, which caused multiple customers to send messages stating things weren’t working. 

According to Intercom, I should have ALL of these customers try nonsensical suggestions from Fin first, even though we know it's a server side issue. So I can't send messages to all these people until they've tried all the completely useless solutions - unless I'm fine with paying $1 per reply. 

Does Intercom have anything to say about this?

​@bosbeest curious what your ideal solution would look like in these scenarios? Did you put a banner in the chat tool during the outage or was that not practical due to it being out of hours or over weekend? Would the new beta Fin guidance ability, allowing you to tell Fin what to do with outages of unknown origin, help for specifying flow and Fin’s responses ahead of the next time this happens? https://www.intercom.com/help/en/articles/10210126-fin-guidance-beta  Or did you have some other ideal solution in mind? Curious what you envisage the best fix to be, product or commercial or other? 

Hi Julian,

I'll try to state it in the shortest possible way:

My ideal scenario is one where Intercom comes up with an alternative to their current way of billing or assumption of resolution. They've set it up in the current way because from the very first stage of design, they mistrust their customers and assume we will all try to circumvent their billing structure. That's why interrupting Fin will count as an “assumed resolution”. These are Intercom's words, not mine. 

To answer your other questions: 

I have not used a banner as I wasn't aware of the functionality. However, this would not mitigate the issue at hand. 

I have coincidentally played around with the guidance ability and it seems very powerful. However, you can't set up rules for an AI to understand that there is something wrong with for example a certificate for a specific region of the world. Nor for any other back-end malfunction for that matter. So unfortunately, this would also not be able to mitigate the issue at hand. 

Thanks for the insights. I was hoping FIN guidance would help with some of these issues. The product and commercial model is rapidly evolving so hopefully you get a model that works for your full needs soon. J 

Unfortunately not I'm afraid. 

Anyone from Intercom care to pitch in on this? Kinda what I was hoping for in a forum called “Connect with the Intercom team” 😅

​@bosbeest It’s Mat from the Support Engineering Team 😀

I am sorry that Fin guidance haven't fulfilled your expectations.

We announced and launched Fin Tasks which I think might be better fit for your needs.

You can read more about it here.

Dear ​@mateusz.leszkiewicz, I'm not sure what you think you are pitching in on, but this thread isn't about Fin Guidance nor can my problem be solved with Fin Tasks. Would be super awesome if you'd at least read the initial message 🙏

I must say it's quite a weird practice to not read the topic, answer with a random marketing message and then also mark your own answer as best answer 😅

​@bosbeest I am sorry that you perceived my message as a marketing stunt but I am not trying to upsell you anything - you have Fin guidance present in your workspace already.

I though about your case (yes I’ve read the thread) and I suggested looking into the fin Guidance since it is literally a way to adjust how Fin interacts with your customers.

You can define in details what resolution mean for you and adjust how Fin reacts to your customers queries. 

We have great articles that describes how you can achieve what you want with Fin (already linked in my previous post) and adjust the Fin resolution definition to better fit your business case.

I hope now my message makes more sense.

Alright let's discuss.

I suggested looking into the fin Guidance since it is literally a way to adjust how Fin interacts with your customers.

Yes I understand that is the goal of this functionality. How can this change “stepping in because Fin's answer is wrong” being billed to me? Unless you mean that Fin will escalate to agents faster, which I've now set up. Which however negates the entire usecase of using an AI 😅

You can define in details what resolution mean for you and adjust how Fin reacts to your customers queries. 

I would honestly love for you to show me where I can change what a resolution means. Because I want “answering a customer before they press speak to human - because FIN's answer is nonsense” to not count as a resolution. If you can show me this, then you have exactly solved my issue. 

that describes how you can achieve what you want with Fin

I want to not be billed, when I step in because Fin's answer is nonsense. Please show me the exact article you're referencing and I couldn't be happier 😺

Regarding marking your post as best answer, is that an automation or are you honestly doing this with your own post? 😅

​@bosbeest 

Fin answers are based on your database - the better your knowledge base the better your Fin answers are.
 

This is how Fin resolution works:

The resolution process in Fin involves two types of resolutions:
1. Hard resolution: When a customer confirms satisfaction by:
• Clicking the "That helped 👍" quick reply button
• Entering an affirmative response like "Yes" or "Sure"
2. Soft resolution: When a customer exits the conversation without requesting further assistance within 24 hours of Fin's last answer.
A resolution is counted when either of these occurs following Fin's last answer in a conversation.

If you feel like you are being billed for the resolutions that are not satisfying please reach out to us so we can dive deep into the specifics of your setup and help you tweak the process.

When I wrote that you can adjust how Fin define the resolution by using Guidance - you can tell Fin how to approach specific topics or your customers queries. I don't know the specifics of your business so I can't tell you how this might work in details.
nevertheless you can identify the conversation where Fin fails to deliver proper answer and tweak your knowledge base or how Fin reacts.
It is not much different to how you would approach teaching your admins how to reply to your customers - except once you put your time once - Fin will remember it forever and the answers will be reproducible.

Here are the best practices you can use when setting up Fin Guidances.

If it comes to Best Answer I marked my answers as Best indeed - I don't like to do that - but it is a part of our internal automation process. I am only doing it when I feel that my answer is actually answering the given question.
I am only human though and I am making the mistakes - apologies if I haven't answer yours correctly.

Please reach out to us via Messenger in your Workspace if you need any help setting up Fin - you shouldn't be charged for the unresolved conversations. 

Hi Mateusz, 

Thanks for your elaborate reply. However, I still have the feeling we're not on the same page regarding what my initial problem is, and why I created this thread. Therefore, I'll try to rephrase it. 

This is an unfair practice, to both your customers (me) and my customers - because you are incentivising me to let the customer try the wrong answer, then click “Speak to Human” and only then I can reply, to avoid paying $1.

Your reply on this seems to be that this is my own fault, as my documentation and guidance is not set up correctly.

But regardless of how good or bad my documentation is, Fin will give wrong answers - you can't document every single incident a customer might encounter. 

The reason of Fin's wrong answer in these cases, is not missing documentation and not missing guidance. I'll give a great example of last weekend:

• A server was overloaded and a group of customers was impacted.
• They all contacted us through chat as always.
• We started instantly replying that we knew they had issues and that we were working on this.
• Each of these replies cost us $1, as we did not want to wait for Fin to give an incorrect answer, stressing people out and wasting everyone's time. 

Do you see how your resolution assumption is flawed?

​@bosbeest I see what you mean now but I still believe that it might be a matter of optimisation.

Have you reached out to us with specific examples of this wrongful resolution so we can recommend a plan of action?

 

​@mateusz.leszkiewicz I honestly don't know how to get past your insistence that this is a problem that can be solved from my side. If a unique outage happens, or basically anything that cannot be anticipated or documented, I cannot be prepared and neither can Fin. But as you insist:

• A server crashes and burns
• All customers are impacted
• Every customer experiences this in a different part of their infrastructure they bought from us, so:
  • The SaaS platform
  • The Order Station screen
  • The Printing Station screen
  • The Capturing camera station
  • Basically, all is down but different people see it at differents parts of our ecosystem
• All these customers contact us stating different things are broken
• Fin tries to help them but obviously cannot know what is wrong
• We don't yet know what's wrong
• We send all incoming messages a reply before/during/right after Fin's reply
• Every message costs us $1

Please let me know how this can be mitigated by optimisation. Please give me a specific way, not a link to a general page of optimisation.