Role Description & Scope
Executive Role Brief & Overview
Join a high-impact reinforcement learning and benchmark evaluation program designed for experienced practitioners. Approved contributors receive direct contractor compensation of $55–$100/hr, disbursed on a reliable weekly escrow schedule. Core deliverables for the Flaky Test Detection & Reproduction Benchmark Engineer role center on RLHF & LLM Rubric Evaluation, multi-step reasoning verification, and factual edge-case auditing. Work entirely on your own schedule with 24/7 cloud benchmark access and flexible weekly hour commitments. Immediate start dates are available for vetted practitioners; use our verified referral link to enter the priority review pipeline.
- Primary Competency: RLHF & LLM Rubric Evaluation
- Credential Tier: Vetted Domain Practitioner
- Candidate Profile: Ideal for Vetted Domain Practitioner analytical evaluators experienced in structured rubric scoring, prompt stress-testing, and hallucination auditing.
- Verified Compensation: $55 - $100 / hr
What is the role overview and compensation benchmark for this position?
- Role Scope & Domain: Verified remote contract for Flaky Test Detection & Reproduction Benchmark Engineer within the AI Training track on verified partner platforms.
- Direct Compensation: Rates range from $55 to $100 per hour (category benchmark average: $73/hr) with direct escrow disbursements.
- Work Structure: Fully remote, flexible scheduling with independent task execution.
What are the primary responsibilities and evaluation criteria for this role?
- Core Evaluation Tasks: Review and benchmark generative AI completions for factual accuracy, reasoning integrity, and compliance with project rubrics.
- Domain Focus: Apply specialized knowledge in analytical evaluation and rubric design to identify hallucinations, logic gaps, and edge-case errors.
- Grading Standards: Calibrate outputs against structured rubric criteria and maintain high inter-annotator agreement on partner evaluation batches.
Detailed Role Specifications & Project Scope
Train AI models to diagnose, isolate, and fix non-deterministic (flaky) test suites.\nResponsibilities:\n- Evaluate test diagnostics for race conditions, timezone dependencies, and unmocked network calls.\n- Verify test determinism by running stress test loops across varied virtual environments.\n- Create rubrics for identifying root causes of async synchronization failures.\nRequirements:\n- 3+ years in Software QA, Test Automation, or CI/CD Engineering.\n- Strong analytical and debugging mindset.