We've started using simulations more as a way to regression test our procedures. Every time we make edits (adding guidance, removing edge-case handling, tightening up logic, etc.), we go back and re-run the simulations that are relevant to that procedure to make sure nothing broke in the process.
The problem is there's currently no way to organize or categorize simulations by what they're actually testing. As we build out more and more of them, it's getting really hard to know at a glance what's already covered for a given procedure, which ones we should be re-running after a specific edit, or whether a simulation even exists for the scenario we're trying to test. Right now we're stuck scrolling through one long undifferentiated list, and more than once we've ended up recreating a simulation that already existed simply because we couldn't find it.
The ask: let us group simulations into folders within the sidebar where the current suite of simulations lives (the Simulations tab in the procedure's testing panel). Ideally these folders would be tied to a specific procedure or issue. This would let us:
- See at a glance what's covered/tested for a given procedure without having to hunt through the full list
- Quickly pull up and re-run the right batch of simulations whenever we edit that procedure, or just run the full suite for that group in one go
- Stop recreating duplicate simulations because we couldn't locate the one we already had
A couple of things that would make this even more useful:
- Auto-suggested folder placement based on which procedure a simulation was created under, since that context already exists at creation time, rather than having to manually file every simulation ourselves
- Last-run status and timestamp visible at the folder level, so we can tell at a glance which simulations are already verified against the latest edit vs. stale, without opening each one individually
This would also help us stay within our daily usage limits. Right now, not being able to find and reuse existing simulations means we end up spinning up new ones more often than we need to, which burns through our daily allotment faster than it should. Being able to see status at a glance would cut down on that further, since we wouldn't need to re-run something just to check if it's already been verified.
We've been finding simulations more and more valuable for this kind of ongoing regression testing as our procedures get more complex, so anything that makes them easier to manage and reuse would go a long way. Bulk actions (archiving or deleting superseded simulations) would also be a nice addition down the line, since one-off tests tend to pile up as procedures evolve.