Software Engineering Evaluator (contract, Remote)

United States·Los Angelesmid

Software EngineerSoftware Engineering

1 views0 saves0 applied

Apply Now

Quick Summary

Overview

About the roleAI Labs is seeking a Software Engineering Evaluator for a remote, hourly contract supporting the assessment of AI systems that write, debug, and review production code.

Technical Tools

Software EngineerSoftware Engineering

About the Role

~1 min read

AI Labs is seeking a Software Engineering Evaluator for a remote, hourly contract supporting the assessment of AI systems that write, debug, and review production code. The role requires 2–3 years of professional software engineering experience, strong command of at least one listed programming language, sound software engineering fundamentals, code review experience, excellent written English, and the ability to evaluate complex code with independent, evidence-based judgment.

Responsibilities

~1 min read

→Design task-specific grading criteria for feature implementations, bug fixes, refactors, tests, and code reviews.
→Evaluate AI-generated code for correctness, including failures that pass the code's own tests.
→Assess edge-case handling, security, performance, and readability in coding deliverables.
→Write clear, evidence-based justifications for every evaluation score.
→Apply consistent judgment across software engineering evaluation tasks.
→Review code that may appear correct and run without failing while identifying subtle defects.
→Reason through ambiguity and distinguish clearly between known facts and inferences during evaluations.
→Iterate on evaluation work based on feedback from senior reviewers.

Requirements

~1 min read

2–3 years of professional software engineering experience shipping production code.
Strong command of at least one of Python, JavaScript/TypeScript, Java, Go, C++, C#, or a comparable language.
Solid grounding in data structures, algorithms, testing, debugging, and version control.
Experience reviewing other engineers' code.
Excellent written English.
Ability to follow complex instructions, reason through ambiguity, and spot subtle errors in code that looks correct and runs without failing.
Ability to clearly separate what you know from what you are inferring.
Ability to work independently, learn new tools quickly, and take feedback well.
Based in and working from the US, Canada, or UK.
Available for 10+ hours per week for the full 12 weeks.

What We Offer

~1 min read

✓$50–$125 per hour, paid weekly via Stripe or Wise.

✓Remote work on your own schedule within the 10+ hour weekly engagement.

✓Initial 12-week contract with the possibility that projects may be extended based on needs and performance.

✓Opportunity to evaluate AI systems for leading AI research organizations.

✓Reasonable accommodations available upon request.

Python
JavaScript/TypeScript
Java
Go
C++
C#
Data Structures
Algorithms
Testing
Debugging
Version Control
Code Review
Production Code
Grading Criteria Design
Evaluating AI-Generated Code
Correctness Evaluation
Edge-Case Handling
Security
Performance
Readability
Backend Systems
Distributed Systems
Infrastructure
Open-Source Contributions
Competitive Programming
Evaluating AI Outputs
Grading Technical Work
Stripe
Wise

Location & Eligibility

Where is the job

Los Angeles, United States

On-site at the office

Who can apply

About this job

Pay per year: ≈ $104,000–$260,000 a year; 5% below the $191,682 median for Software Engineer jobs on Jobera (6,303 salaries).
Who can apply: United States
Posted: Today
Employer: has 396 open jobs on Jobera.