What happens when Fin gives a customer the wrong answer but nobody notices? | Community
Skip to main content

What happens when Fin gives a customer the wrong answer but nobody notices?

  • September 14, 2026
  • 4 replies
  • 61 views
Forum|alt.badge.img+2

We've all spent time improving Fin's knowledge, testing answers and looking at resolution rates.

But I'm curious about what happens after Fin actually sends the answer.

Say Fin tells a customer:

“Yes, you can get a full refund.”

…but your actual policy doesn't allow it.

Or it gives an answer that sounds completely normal, but contains something your team shouldn't be promising.

How are you catching those cases today?

Manual QA? Monitors? Sampling conversations? Customer complaints? Or do they mostly get missed?

I'm particularly interested in teams running Fin at meaningful volume. How are you handling this in practice?

    4 replies

    Forum|alt.badge.img+2
    • Jon Vaughan
    • Connector
    • September 17, 2026

    IMO resolution rate is the wrong metric for this because it just means the customer didn't ask for a human, not that the answer was right. A wrong answer that doesn't trigger an escalation looks identical to a correct one.

    Would recommend doing sampling; pull a random slice of conversations Fin marked resolved and read them against policy. Its a bit tedious but you can e.g. search within categories of problem that are higher risk or look for text in responses that wouldn’t be permissable.

    Another signal worth watching is repeat contact after an AI resolution, coming back on the same or similar topic within a few days after Fin resolution. Comparing CSAT  on AI  vs human might also surface a gap, though response rates are not high
     

    as a bit of a disclaimrer i work for a company that produces a tool for doing this kind of checking, isara.ai, which is why i saw this, but lmk if i can help further

    https://www.isara.ai || https://www.intercom.com/app-store/?app_package_code=tappa-insights&search=isara
    Forum|alt.badge.img+2
    • Parvesh
    • Author
    • Connector
    • September 17, 2026

    Can you text chatshield and give me honest feedback?

    Chatshield.cloud

    Forum|alt.badge.img+2
    • Parvesh
    • Author
    • Connector
    • September 17, 2026

    ​@Jon Vaughan  Hey , I would like to connect you on linkdin so we have a talk, I will literally appreciate. Please take my concern 

    Tonje Ness Meinhardt
    Forum|alt.badge.img+3

    Hi,

    When we first launched Fin, we reviewed conversations daily. This helped us identify incorrect answers, misunderstandings, and areas where guidance needed to be improved. Most of the major issues were caught and corrected during that phase, often by adding more specific guidance where content could be interpreted differently across products or services.
     

    Today, we still review conversations regularly, although typically a few times per week rather than daily.
     

    We have also added chatbot terms and conditions, linked directly in Messenger, informing customers that AI-generated responses may occasionally contain errors.
     

    We also tested Monitors while they were available at no additional cost. We found them quite useful and had a monitor configured to detect potential inaccuracies or policy-related issues. That said, their effectiveness depended largely on either the customer indicating that something was wrong or an agent identifying the issue after escalation.

      Terms & ConditionsAccessibility statement