AI Benchmark Quality Reviewer

Responsibilities:
• Validate task quality: Check that instructions, source materials, reference solutions, and evaluation criteria are consistent, with no hidden requirements or missing information.
• Review agent performance: Inspect execution traces, tool calls, and generated deliverables to determine whether successes and failures are justified.
• Audit grading logic: Identify brittle checks, incorrect expected answers, unsupported rubric criteria, and cases where valid alternative solutions are unfairly penalized.
• Investigate discrepancies: Distinguish genuine model limitations from task defects, grader errors, and environment or tool failures. Assess automated QC findings independently rather than accepting them at face value.
• Document decisions: Provide concise, evidence-backed findings and actionable feedback, flag uncertainty, and verify that revisions resolve identified issues.

What we're looking for:




• Technical fluency: Comfortable reading Python, SQL, shell scripts, structured data, and execution logs to understand task setup and grading behavior.
• Analytical judgment: Able to independently check calculations, reconcile conflicting evidence, and assess the correctness and completeness of professional deliverables.
• Clear communication: Strong written English, attention to detail, and experience providing specific, reproducible feedback.
• Relevant experience preferred: AI evaluation, technical QA, data analysis, or benchmark development; familiarity with Harbor task setup.

Engagement details
• Engagement type: Contractor, hourly pay
• Commitment: 40 hours a week 8 hrs PST overlap mandatory
• Duration: 10 weeks, starting immediately

📌 Freelancing | AI Benchmark Quality Reviewer | Remote | Immediate Joiner (Modena)
🏢 PGC Digital (America) Inc: CMMI Level 3
📍 Modena