Score a prompt change before you ship it
Run the same set against the old and the new prompt on identical hardware, then read the delta rather than trusting a hunch.
Everything below describes what this is being built to do. It needs Studio, which you can look around today but cannot yet use for the whole of this job. The problems are real now; the answers are the part still being built.
Everyone can feel that a prompt change helped. Almost nobody can show it.
Why it is hard
A before/after is usually a vibe
One run on the old prompt and one on the new, read side by side, is a comparison of two anecdotes, not two samples.
Hardware drifts between runs
Comparing runs from different sessions on different cards adds a second variable to the one actually being tested.
A single score hides where it changed
An aggregate number can rise while it gets worse on the cases that matter most, and nothing below it says which.
What answers it
Two arms, one set, identical hardware
The same input set runs against both prompts on the same card class, so the prompt is the only thing that changed.
One scoreboard, not two logs
Both runs land in one place built to be read against each other, rather than two exports somebody diffs by hand.
The delta is the answer
The comparison reads as a delta per case, not one aggregate number, which is the part not built yet.
What you use from Kracht
Related jobs
- StudioplannedRun a multi-step agent pipeline on GPUs you controlEach step gets the card it needs rather than the one the whole pipeline was sized for. The shape of the work and the cost of the work become the same picture.Read the job
- StudioplannedBring a workflow you already wroteLangGraph, LangChain and CrewAI graphs are import sources rather than competitors. The intent is that an existing graph arrives as nodes you can see, not as a rewrite.Read the job
- StudioplannedPut a person in the middle of a pipelineThe run stops at the approval and waits. That is the difference between something you can put in front of a customer and a script somebody watches.Read the job