vinitium · case study

Trusting a model with your test suite

Role
Product designer, build optimization exploration
Duration
In flight
Status
in flight
Trusting a model with your test suite — opening motif Many faint strands fan toward a gate; a chosen few continue on, drawn brighter — prediction as selection.

Teams with hour-long test suites don't need faster machines — they need to run fewer, smarter tests. I'm designing BrowserStack's exploration of predictive test selection and orchestration: the design problem isn't the ML, it's persuading an engineer to let a confidence curve decide what doesn't run before their release ships.

Problem

Test selection asks users to accept a trade no dashboard has made legible: skip most of the suite, accept a quantified risk, get your feedback in minutes instead of hours. The prediction model needs a learning period before its confidence means anything, competitors bury the risk half of the trade, and one silently missed regression destroys more trust than a thousand fast builds earn.

Where it stands

In flight. The research foundation is set — user interviews, a vendor teardown, and the framing decision that the confidence curve is a design object, not an implementation detail. The exploration's current bet: time saved means nothing unless risk taken is equally visible.

The trade nobody has made legible

Predictive test selection is a known category with a known promise: given a code change, run only the tests that could plausibly break, and cut suite time dramatically. The demand is real — teams we interviewed (an enterprise device lab lead, a developer-experience owner at a consumer platform, an internal QA lead) all described the same pain from different angles: full suites re-run for one-file changes, senior engineers hand-picking test subsets from memory, failed tests re-run wholesale after fixes. The market opportunity is withheld: market sizing ; the design opportunity is that everyone selling this today leads with the saving and whispers the risk.

What the vendor teardown taught

I tore down the public-facing behavior of the products in this space — the ML-selection specialists, the impact-analysis platforms, the test framework with built-in orchestration strategies:

What the market shows users, compared Vendors compared across four behaviors: showing time saved, exposing selection logic, foregrounding miss risk, and offering simple orchestration toggles. Most show savings prominently; almost none foreground risk. time saved selection logic risk visibility simple toggles ML-selection vendors impact-analysis platforms framework-native the gap worth designing ● yes ◐ partial ○ no

The teardown’s shape: savings dashboards are table stakes; legible risk is the unclaimed column.

Time-savings pages are everywhere — sessions run, minutes saved, the before/after table. What’s missing everywhere is the other half of the ledger: what was skipped, what confidence that skip carried, and what the near-misses looked like. That absence is rational (it’s a harder sell) and it’s exactly why an engineer’s default posture toward these tools is suspicion.

The confidence curve is a design object

The model that powers selection has a property the UI cannot hide from: it starts ignorant. It needs a learning period — withheld: training-volume specifics of real builds — before its confidence curve stabilizes. Most products treat this as a footnote. I’m treating it as the product’s first chapter:

Trust follows the confidence curve Stages from observing mode through partial selection to full selection, widening as model confidence and user-verified accuracy accumulate. observe only model watches, suite runs whole shadow picks selection shown, not enforced assisted runs user approves the skip full selection

The staged-trust concept the exploration is converging on: the model earns autonomy the same way a new team member does — visibly.

The design bet: let the user watch the model be right before asking them to act on it. A shadow mode that shows what would have been skipped — and that every skipped test would have passed — converts the learning period from an apology into the trust-building phase.

Where it stands

This is in-flight work, and this study will grow its outcome the honest way — after it exists. The current state: research and teardown complete, the staged-trust framing adopted as the exploration’s spine, orchestration controls (run-failed-first, auto-cancel, flaky-test quarantine) sketched as the simpler sibling that ships value while the model learns. The open threads live in the margins of this page, and the full walkthrough happens in conversation.

The full walkthrough — screens, numbers, names — happens in conversation. Start one →