Building a reliable Fin AI agent and Outcomes means testing more than the happy path. We need to know whether Fin follows the right procedure, asks for missing information, handles exceptions, and hands off when it should. We also need to know whether a change that fixes one workflow breaks another.
Today, Preview is useful for debugging a single conversation, but running tests one by one takes too much time. Evals help with batch testing, but their scope and run limits make it hard to test several workflows together, give each workflow its own success criteria, and rerun the full set after every change.
I’d like Intercom to add a built-in sandbox enviornment where teams can:
- Test safely. Run draft configurations against simulated conversations without contacting customers or triggering live actions. Simulate tool responses, failures, and handoffs.
- Batch test across workflows. Run refunds, onboarding, and escalations in the same batch, with separate success criteria for each scenario. For example, a refund test should pass only if Fin verifies eligibility before attempting the refund.
- Run QA automatically and without limits. Generate test cases from Procedures, save them as reusable suites, and rerun them on demand, on a schedule, or whenever the configuration changes. Sandbox runs should be free and unlimited, with no monthly quota on simulations or evaluations.
- See what passed, what failed, and why. Show success rates by workflow and criterion, compare results with the published configuration, and link each failure to the conversation and the step where Fin went off track.
The goal is to make QA part of building rather than another feature that has a financial barrier. Before publishing a change, I want to run the full regression suite and see whether Fin follows our protocols and achieves the intended outcomes. We should be able to do that after every meaningful change, as often as we need without limits.