For companies: explore licensing operational data for AI. Check eligibility

SWE-Bench Task Auditor

$70–$90/hr

Apply on Mercor →

New to Mercor? before applying.

Get new roles by emailFollow on X

Jobs › Finance › Mercor

Pay
$70 to $90 per hour
Pay type
Hourly
Platform
Mercor
Field
Finance
Where
Remote, for applicants in United States
First seen
Posted
History
Half of Mercor roles listed since Oct 2 were gone within 28 days (of 329). How long roles stay open
Feed read

This listing comes from Mercor's own public job feed. The posting on Mercor is the source of truth.

About the role

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications

  • 3+ years professional software engineering
  • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
  • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
  • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications

  • Familiarity with SWE-Bench (Verified) or similar repository benchmarks
  • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
  • Prior code-review or task-grading experience

More: Every role for finance professionals · Roles open in United States · How Mercor pays, in its own words · What AI training roles pay · Free resume help