Role Description & Scope
Executive Role Brief & Overview
This verified remote AI evaluation contract offers an immediate opportunity for specialized domain experts. Contractors earn $60–$105/hr with guaranteed weekly payouts and priority access to bonus calibration batches. Daily workflows for the Senior QA Lead - Inter-Annotator Agreement (IAA) contract emphasize RLHF & LLM Rubric Evaluation, comparative model ranking, and rubric-driven error taxonomy. Flexible weekly allocations and asynchronous review pipelines make this remote contract ideal for high-performing specialists. Rolling intake is currently live—complete your brief technical screening via the verified apply route to lock in your batch allocation.
- Primary Competency: RLHF & LLM Rubric Evaluation
- Credential Tier: Senior
- Candidate Profile: Ideal for Senior analytical evaluators experienced in structured rubric scoring, prompt stress-testing, and hallucination auditing.
- Verified Compensation: $60 - $105 / hr
What is the role overview and compensation benchmark for this position?
- Role Scope & Domain: Verified remote contract for Senior QA Lead - Inter-Annotator Agreement (IAA) within the Quality Assurance & Testing track on verified partner platforms.
- Direct Compensation: Rates range from $60 to $105 per hour (category benchmark average: $27/hr) with direct escrow disbursements.
- Work Structure: Fully remote, flexible scheduling with independent task execution.
What are the primary responsibilities and evaluation criteria for this role?
- Core Evaluation Tasks: Review and benchmark generative AI completions for factual accuracy, reasoning integrity, and compliance with project rubrics.
- Domain Focus: Apply specialized knowledge in analytical evaluation and rubric design to identify hallucinations, logic gaps, and edge-case errors.
- Grading Standards: Calibrate outputs against structured rubric criteria and maintain high inter-annotator agreement on partner evaluation batches.
Detailed Role Specifications & Project Scope
Oversee data quality, statistical agreement metrics, and rubric governance across annotation cohorts.\nResponsibilities:\n- Compute and monitor Cohen's Kappa, Fleiss' Kappa, and Krippendorff's Alpha across evaluators.\n- Run calibration syncs to resolve edge-case disagreements and eliminate rubric ambiguity.\n- Implement automated spot-checking pipelines to catch annotator drift.\nRequirements:\n- 3+ years in Data Quality Assurance, Statistical Quality Control, or Research Operations.\n- Strong statistical proficiency and team leadership abilities.