AgentsBenchmark

Benchmark configuration

Create a Github repository

Containing the initial codebase for the benchmark
Define benchmark steps and gold patches

Create github issues and commit a gold patch PR for each step
Publish progressive tests
Including a CI Github action to pass progressive tests for each step, based on the PR step number, following this format:
```
#[current_step] [PR_name]
```
All progressive tests anterior to this step will run

Benchmark execution per agent (for each step)

Set the previous gold patch as a base

By checking out the proper Git branch
Create a space for the code to be generated
By creating a Git branch following this format :
```
#[step_number]_[agent_name]_[agent_model]_[run_number]
```
Pass the issue contents as a prompt to the agent

Either manually (chat based agents) or automatically (i.e. SWE-bench)
Create a GitHub Pull Request with the generated code

By manually committing (if required) the generated code and creating a GitHub PR from the defined branch

Execution environment

Leaderboard	Overall score	Approx. Cost
Amazon Q Pro IntelliJ plugin	72.5%	$38 $2.24 / solved
SWE-Agent v0.6.0 (docker)(CLI)	25%	$20 $3.33 / solved
Aider v0.37.0 GPT-4o	Non-functional
ByteDance MarsCode Agent (In IDE)	No file insertion feature
OpenDevin v0.6.2 GPT-4o	Ruby not supported
OpenCSG StarShip	Access requested

Run Result	Icon	Points awarded
Passed		100
Passed but unwanted or wrong code		75
Failed		0
Errored		0
No run required		-

Benchmark #1 - Rails Blog