PromptDiff

Run one prompt against four models and see what changed

Noah FischerLaunched 4 weeks agoDeveloper toolWorking · checked 3 hours ago
0:42

About

Paste a prompt, pick models, and PromptDiff runs them side by side with a word-level diff between the outputs. Save runs, compare them over time, and catch it when a model update changes your results.

#evals#prompts#llm

How it was made

Shared by the maker. Numbers are their own estimate.

Build time
1 week
AI spend
$31
Prompts
~64
Written by AI
96%

Agents

  • Claude Code

    100% of the work

    Everything except the logo

MCP servers1

  • Context7

    Up-to-date API docs for each model provider

Build recipe

The code is on GitHub. If you’d rather build it yourself than fork it, paste this into your own agent.

Written by Claude Code

From the project’s code and history · 341 words

Build a developer tool called PromptDiff: run one prompt against four models and see what changed.

What it does

Paste a prompt, pick models, and PromptDiff runs them side by side with a word-level diff between the outputs. Save runs, compare them over time, and catch it when a model update changes your results.

Stack

Use Bun, Hono, React, SQLite. The original was built with Claude Code in 1 week, for about $31 in AI usage.

How it fits together

  • A focused tool with one main view and a small backend
  • Local storage or SQLite for saved runs
  • A clear config file so it fits into existing projects

Build it in this order

  1. Scaffold the project and a single page or command that runs end to end
  2. Build the core operation and test it with three tricky real examples
  3. Add saving, history and comparison
  4. Write a README with a two-minute quick start
  5. Add tests that run in CI

Tools to connect

  • Context7 MCP: up-to-date API docs for each model provider

A prompt that worked: Word-level diff that reads well

Implement a word-level diff between two model outputs that treats punctuation as its own token, ignores whitespace-only changes, and groups adjacent changes into one block. Render it as inline <ins>/<del> spans. Include tests with three tricky examples.

Watch out for

  • Developers will judge it in the first minute, so the quick start has to work exactly as written
  • Avoid hidden network calls; say what leaves the machine

Done when

  • The quick start works on a clean machine
  • The core operation has tests for edge cases
  • Errors say what went wrong and how to fix it

Before writing code, ask me up to five questions about who this engineering developer tool is for and what the first version must do. Then work in small steps: after each one, tell me how to try it and wait for my feedback.

Discussion

4 comments · 1 member tried it

  • Priya RamanTried it3 hours ago

    Clean idea and it actually works. The build log is the most useful part for me, I'd love more detail on how you structured the prompts in week two.

    • Noah FischerMaker2 hours ago

      Thanks! Good question, I'll write it up in the build log this week.

  • Lena Vogt6 hours ago

    Love how focused this is. One thing: the empty state took me a second to understand. Maybe show an example right away?

  • Kenji Mori1 day ago

    What did the agent struggle with most? Always curious where people had to step in.

More like this