Role Description & Scope
Executive Role Brief & Overview
Our verified partner network has opened rolling allocations for this remote model calibration and rubric role. Verified remuneration is structured at $60–$85/hr, rewarding high-precision rubric adherence and analytical depth. Contributors in the AI Evaluation Rubric Auditor role leverage RLHF & LLM Rubric Evaluation to benchmark model performance across high-difficulty domain scenarios. Work entirely on your own schedule with 24/7 cloud benchmark access and flexible weekly hour commitments. Because cohort seats fill on a rolling basis, submit your profile early via our verified partner routing link to secure priority review.
- Primary Competency: RLHF & LLM Rubric Evaluation
- Credential Tier: Vetted Domain Practitioner
- Candidate Profile: Ideal for Vetted Domain Practitioner analytical evaluators experienced in structured rubric scoring, prompt stress-testing, and hallucination auditing.
- Verified Compensation: $60 - $85 / hr
What is the role overview and compensation benchmark for this position?
- Role Scope & Domain: Verified remote contract for AI Evaluation Rubric Auditor within the Finance & Accounting track on verified partner platforms.
- Direct Compensation: Rates range from $60 to $85 per hour (category benchmark average: $104/hr) with direct escrow disbursements.
- Work Structure: Fully remote, flexible scheduling with independent task execution.
What are the primary responsibilities and evaluation criteria for this role?
- Core Evaluation Tasks: Review and benchmark generative AI completions for factual accuracy, reasoning integrity, and compliance with project rubrics.
- Domain Focus: Apply specialized knowledge in LLM to identify hallucinations, logic gaps, and edge-case errors.
- Grading Standards: Calibrate outputs against structured rubric criteria and maintain high inter-annotator agreement on partner evaluation batches.
Detailed Role Specifications & Project Scope
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
\- Was the information available?
\- Does the rubric accurately grade what the prompt asked for?
\- Would a valid answer have been marked wrong?
\- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
\- Experience providing feedback to writers/reviewers
\- Consistent availability
\- Can read a long agent transcript and find where it went wrong.
\- Precise writing, and you check the data instead of trusting the reference answer.