Summary:
Manual reviewer notes added to conversations within a Monitor should be exportable, either via the Dataset Export (CSV) or surfaced in the Reporting API, alongside scorecard scores.
The problem:
When manually reviewing conversations in a Monitor, reviewers can add qualitative notes to explain their scoring decisions. These notes are currently only visible within the Monitor UI, one conversation at a time, and can't be exported via CSV, the Reporting API, or S3 export.
This creates a noticeable gap: the qualitative insight that gives context to scores, the why behind a pass or fail, is trapped behind individual conversation views. To extract patterns, a reviewer has to manually open and copy notes from each conversation, which isn't realistic at scale.
What I'd expect instead:
Reviewer notes should be included in the Dataset Export (scorecard_evaluation or scorecard_attribute_evaluation datasets) as an additional column, and ideally also surfaced via the Reporting API so teams can build dashboards or send data to external tools.
Even a basic bulk export, such as a CSV of conversation ID, attribute, score, and note, would make a meaningful difference for teams using Monitors for structured QA workflows.
Impact:
Without exportable notes, qualitative QA analysis at scale just isn't practical. It limits Monitors to surface-level pass/fail tracking, rather than the richer insight tool they're designed to be.