Parsimony
Open-source benchmark measuring AI code bloat
Developer ToolsOpen sourceOpen source (repo not verified)#ai-coding#benchmark#open-source
Listed by the ProductPerch team from Hacker News · not launched here yet
What is Parsimony?
Parsimony is an open-source benchmark that measures how much code AI coding agents add to existing projects. Unlike traditional benchmarks that focus only on solve rates or cost, Parsimony tracks the net size of changes introduced by AI models. It counts specific syntax elements like statements, function calls, comparisons, names, and literals rather than raw lines of code.
The tool calculates net coding units by subtracting removed code from added code. It evaluates performance using published results from SWE-bench Verified, focusing on Python tasks. Models are ranked by their mean net units added on solved tasks, with lower scores indicating more concise changes. The platform provides interactive scatter plots and bar charts to compare metrics across different AI agents.
Users can upload their own benchmark results using a provided JSON format to generate comparable charts directly in the browser. The evaluation uses the mini-SWE-agent framework and parses both original and patched files without executing tests. Currently, the benchmark covers Python only, though the methodology allows for expansion.
Parsimony features
- Tracks net coding units added by AI agents
- Counts syntax elements instead of raw lines
- Ranks models by conciseness on solved tasks
- Generates interactive scatter plots and bar charts
- Accepts custom benchmark results via JSON
- Evaluates Python patches using SWE-bench Verified
Who uses Parsimony and for what
- Compare AI coding agents based on code conciseness
- Evaluate how much bloat AI patches introduce to a codebase
- Track model efficiency improvements across versions
- Visualize trade-offs between solve rate and code size
- Import personal benchmark data for internal model testing
Parsimony FAQ
Is Parsimony free?
Yes, Parsimony is an open-source project available at no cost. You can access the benchmark tools and charts directly through the website. The website does not mention paid tiers.
What programming languages does Parsimony support?
The current benchmark measures Python tasks only. The evaluation parses Python files and applies patches to calculate net coding units. The website does not say if other languages will be supported.
How does Parsimony measure code changes?
Parsimony counts syntax elements like statements, calls, comparisons, names, and literals. It calculates net units by subtracting removed code from added code. Lines of code are not counted. The website does not say if future updates will change this method.
Can I use my own benchmark data with Parsimony?
Yes, you can download an example JSON file, fill in your results, and open it in the browser to generate the same charts and tables. The processed file stays locally in your browser. The website does not say if server-side uploads are supported.
Does Parsimony run tests to verify AI patches?
No, the benchmark relies on published pass/fail results from SWE-bench Verified. Tests are not rerun during evaluation. Only the patch files are parsed to measure size changes. The website does not say if test execution will be added later.
Comments (0)
Comments and upvotes open when this product launches on ProductPerch.
More Developer Tools launches
Monitor Laravel scheduled tasks and queues
Free product launch platform for indie makers
PloyzOpen source
The open-source cloud on servers you rent
