PromptLab
A prompt workbench with git-style versioning, rubric scoring, A/B comparison, and a regression gate before anything ships.
The gap
Prompt engineering still lives in Slack threads and Notion dumps. Nobody can answer: what changed last Tuesday, did quality go up or down, and will this regress the cases we already fixed? Without version control and evaluation, teams ship vibes — then discover regressions in production.
What I built
PromptLab versions prompts like code (diffs, authors, timestamps), runs them against fixed eval sets with rubric scorers, and blocks promotion when the regression suite fails. A/B views compare two versions side-by-side with score distributions. Workers support OpenAI, Anthropic, and local endpoints.
How it fits together
Technical choices
- ●Treat prompts as versioned artifacts — every change is diffable and attributable.
- ●Rubric scores over vibes — human-written criteria, machine-applied at scale.
- ●Gate deploys on regression suites — same instinct as CI for application code.
System features
- Git-style prompt history
Diffs, authors, and rollback — prompts stop living in chat scrolls.
- Rubric evaluation
Async scorers grade outputs against explicit criteria, not gut feel.
- A/B comparison matrix
See per-case wins/losses before you promote a version.
- Regression gate
A version cannot ship if critical fixtures regress vs baseline.