Home > Jobs > Finance & Accounting > AI Evaluation Rubric Auditor
✓ Verified Finance & Accounting

AI Evaluation Rubric Auditor

Compensation $60 - $85 / hr
Location Remote
TIER 3/5 Standard AI Vetting
Active Matching Pool
Interview Format 20-min AI Case & Resume Walkthrough
Insider Preparation Tip Align your resume keywords tightly with the role description before launching the automated video interview.

Key Skills & Domain Expertise

LLM
SPONSORED (728x90) Accelerate your AI benchmarks: Explore verified developer tooling & APIs

Role Description & Scope

Executive Role Brief & Overview

Our verified partner network has opened rolling allocations for this remote model calibration and rubric role. Verified remuneration is structured at $60–$85/hr, rewarding high-precision rubric adherence and analytical depth. Contributors in the AI Evaluation Rubric Auditor role leverage RLHF & LLM Rubric Evaluation to benchmark model performance across high-difficulty domain scenarios. Work entirely on your own schedule with 24/7 cloud benchmark access and flexible weekly hour commitments. Because cohort seats fill on a rolling basis, submit your profile early via our verified partner routing link to secure priority review.

  • Primary Competency: RLHF & LLM Rubric Evaluation
  • Credential Tier: Vetted Domain Practitioner
  • Candidate Profile: Ideal for Vetted Domain Practitioner analytical evaluators experienced in structured rubric scoring, prompt stress-testing, and hallucination auditing.
  • Verified Compensation: $60 - $85 / hr

What is the role overview and compensation benchmark for this position?

  • Role Scope & Domain: Verified remote contract for AI Evaluation Rubric Auditor within the Finance & Accounting track on verified partner platforms.
  • Direct Compensation: Rates range from $60 to $85 per hour (category benchmark average: $104/hr) with direct escrow disbursements.
  • Work Structure: Fully remote, flexible scheduling with independent task execution.

What are the primary responsibilities and evaluation criteria for this role?

  • Core Evaluation Tasks: Review and benchmark generative AI completions for factual accuracy, reasoning integrity, and compliance with project rubrics.
  • Domain Focus: Apply specialized knowledge in LLM to identify hallucinations, logic gaps, and edge-case errors.
  • Grading Standards: Calibrate outputs against structured rubric criteria and maintain high inter-annotator agreement on partner evaluation batches.

Detailed Role Specifications & Project Scope

The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

\- Was the information available?

\- Does the rubric accurately grade what the prompt asked for?

\- Would a valid answer have been marked wrong?

\- Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves

\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

\- Experience providing feedback to writers/reviewers

\- Consistent availability

\- Can read a long agent transcript and find where it went wrong.

\- Precise writing, and you check the data instead of trusting the reference answer.

ROLE REALITY CHECK Domain Expert Precision & Citation Verification • High-Stakes Eval

Field Inspection & Rubric Realities

1. Technical Task Profile

Daily specialist queues for AI Evaluation Rubric Auditor require deep verification across LLM. Contributors audit multi-step domain reasoning, verify primary-source citations or quantitative derivations, and rewrite flawed model completions to gold-standard accuracy. Each task batch undergoes strict rubric calibration against a seeded golden set to sustain inter-annotator agreement and Cohen's kappa, including S2S audio latency and barge-in interruption checks when auditing voice-enabled domain assistants.

2. Screening Pitfalls

Automated intake for Tier 3/5 (Standard AI Vetting — 20-min AI Case & Resume Walkthrough) enforces deterministic halting criteria. During Micro1 Zara automated technical interviews, candidates are flagged if spoken or coded explanations stall beyond the S2S audio latency window, or if barge-in interruption recovery fails during live follow-up probes. On Mercor and partner asynchronous video screens, recording constraints require concise, evidence-backed walkthroughs without reading off external scripts. Across written rubric calibration gates, the #1 disqualification trigger is missing a P0 factual or spec error on a seeded golden set item, which immediately drops your Cohen's kappa and inter-annotator agreement score below cohort onboarding thresholds.

3. Benchmark Verification

Cross-referencing 96 verified active Finance & Accounting contracts in our local index shows hourly compensation spanning $8/hr to $400/hr. At $60 - $85 / hr, this AI Evaluation Rubric Auditor contract is positioned as a high-volume entry allocation relative to the $100/hr category median. It carries a Vetting Tier 3/5 (Standard AI Vetting) classification, meaning applicants who maintain consistent inter-annotator agreement on the initial golden set and respect all halting criteria during rubric calibration qualify for weekly escrow payouts and priority batch allocation.

FINANCE & ACCOUNTING • PRE-SCREEN CHECK 0 / 3 Answered

Pre-Screen Readiness Check

Quick 3-step pre-screen check: Test your instincts against real partner grading rules for this specific role before launching the official application.

SKILL CHECK: LLM True / False • Inter-Annotator Rubric Gate

1. Rubric Calibration & Cohen's Kappa Drift

True or False: When two annotators disagree on a qualitative response tone, an acceptable method to maintain high Cohen's kappa (κ ≥ 0.80) is for one annotator to adjust their score to match their peer without documenting an explicit rubric justification.

SKILL CHECK: FINANCE & ACCOUNTING True / False • GAAP / IFRS Revenue Recognition Gate

2. ASC 606 Performance Obligations & Milestone Timing

True or False: Under ASC 606 / IFRS 15, upfront non-refundable implementation fees received by a SaaS provider must always be recognized as immediate revenue upon contract signing rather than amortized over the estimated customer lifecycle.

SKILL CHECK: MERCOR VETTING Single-Choice • Mercor Case Walkthrough

3. Mercor Timed Domain & Video Case Walkthrough

When completing the timed asynchronous video and domain assessment for this partner role, what do reviewers prioritize most?