Skip to content
New here? Rent a GPU by the second
Kracht
Sign inSign up

Studio Evaluation

Score a change
before you ship it.

Scored items against a threshold, compared with the last run, so the signal is what moved rather than what the number is today.

Open the console
Studio
Score
per item
Threshold
gates a release
Crossed
since the last run
0 ≠ none
nothing measured

You cannot run one yet

example data

Running a dataset needs a workflow that can be run on demand, and Studio saves a graph but does not execute one. The console page says exactly that, and this page says it here rather than at the bottom.

Datasets

A set of items, scored

A dataset is the cases you care about getting right. Each item carries a score, and the set carries a threshold, which is the number a release has to clear.

  • Score per item, never one number standing in for the set
  • Set a threshold that gates a release rather than decorating a chart
  • Follow a dataset to every run that used it
Open the console
Five scored items against a threshold of 0.80. Refund wording scores 0.94, address parse 0.88 and date in prose 0.81, all clearing it. Currency scores 0.72 and negation 0.41, both below.

Movement

What crossed since the previous run

An absolute score is not actionable and a trend is. The view is built around what changed: what crossed the threshold, and what crossed back, which is the number that stops a release.

  • Compare against the last run, always
  • Watch what crossed the threshold and what crossed back
  • Act on the crossing rather than on the score
What changed since the previous run. Date in prose rose from 0.74 to 0.81 and crossed the threshold. Address parse rose from 0.83 to 0.88, already above it. Currency fell from 0.86 to 0.72 and crossed back below, which is the one that stops a release.

Honesty

It will not give you a percentage it cannot stand behind

Two items scoring perfectly is not a hundred percent pass rate. When there are too few scored items for a percentage to mean anything, the console says that instead of showing one.

  • Refuse a number that looks like an answer but is not one
  • Show ‘nothing measured’ rather than a zero, because a zero is a measurement
  • Mark where dataset items will live as the open question it still is

That last point is genuinely undecided. It is on the console page as an open question rather than described as though it were settled.

An empty state reading: nothing measured. Too few scored items for a percentage to mean anything.

Resources

Where to go next

Documentation

How it works, endpoint by endpoint

Evaluation

Datasets, thresholds and what moved

Observability

See what a run actually did

Frequently asked questions

No. Running a dataset needs a workflow that can be run on demand, and Studio saves a graph but does not execute one. The console page says the same thing in the same words.

Want to know whether a change made it better?

Open the console
Two sides, one marketplace

Spin up a GPU, or earn from your own

Rent on-demand compute by the second, or list idle hardware and let it pay for itself. Same marketplace, both directions.

Start renting
List your hardware

No card required

85%
revenue to providers
Per-second
billing, live
EUR
metered to the microeuro