Role Description & Scope
Executive Role Brief & Overview
Join a high-impact reinforcement learning and benchmark evaluation program designed for experienced practitioners. Verified remuneration is structured at $65–$125/hr, rewarding high-precision rubric adherence and analytical depth. Core deliverables for the SWE-bench Real-World Issue Resolution Specialist role center on RLHF & LLM Rubric Evaluation, multi-step reasoning verification, and factual edge-case auditing. Enjoy frictionless remote execution with asynchronous rubric queues, zero commute, and flexible part-time or full-time bandwidth. To maximize acceptance speed into the active evaluation cohort, launch your application directly via our verified partner gateway.
- Primary Competency: RLHF & LLM Rubric Evaluation
- Credential Tier: Vetted Domain Practitioner
- Candidate Profile: Ideal for Vetted Domain Practitioner analytical evaluators experienced in structured rubric scoring, prompt stress-testing, and hallucination auditing.
- Verified Compensation: $65 - $125 / hr
What is the role overview and compensation benchmark for this position?
- Role Scope & Domain: Verified remote contract for SWE-bench Real-World Issue Resolution Specialist within the AI Training track on verified partner platforms.
- Direct Compensation: Rates range from $65 to $125 per hour (category benchmark average: $73/hr) with direct escrow disbursements.
- Work Structure: Fully remote, flexible scheduling with independent task execution.
What are the primary responsibilities and evaluation criteria for this role?
- Core Evaluation Tasks: Review and benchmark generative AI completions for factual accuracy, reasoning integrity, and compliance with project rubrics.
- Domain Focus: Apply specialized knowledge in analytical evaluation and rubric design to identify hallucinations, logic gaps, and edge-case errors.
- Grading Standards: Calibrate outputs against structured rubric criteria and maintain high inter-annotator agreement on partner evaluation batches.
Detailed Role Specifications & Project Scope
Evaluate model performance on realistic GitHub repo issue tickets and multi-file patches.\nResponsibilities:\n- Reproduce reported repository bugs, isolate root causes, and evaluate proposed diffs.\n- Ensure proposed patches do not introduce regressions in existing test suites.\n- Score model explanations for repo context comprehension and file locating precision.\nRequirements:\n- 3+ years contributing to open-source or large multi-package codebases.\n- Expertise in debugging unfamiliar codebases rapidly.