Generalist - English
Location: Geography restricted to US, UK, Canada Type: Full-time or Part-time Contract Work Fluent Language Skills Required: English
Why This Role Exists
Mercor partners with leading AI teams to improve the quality, usefulness, and reliability of general-purpose conversational AI systems. These systems are used across a wide range of everyday and professional scenarios, and their effectiveness depends on how clearly, accurately, and helpfully they respond to real user questions.
This project focuses on evaluating and improving general chat behavior in large language models (LLMs). You will assess model-generated responses across diverse topics, provide high-quality human feedback, and help ensure AI systems communicate in ways that are accurate, well-reasoned, and aligned with human expectations.
What You’ll Do
Evaluate LLM-generated responses on their ability to effectively answer user queries
Conduct fact-checking using trusted public sources and external tools
Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies
Assess reasoning quality, clarity, tone, and completeness of responses
Ensure model responses align with expected conversational behavior and system guidelines
Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines
Who You Are
You hold a Bachelor’s degree
You have significant experience using large language models (LLMs) and understand how and why people use them
You have excellent writing skills and can clearly articulate nuanced feedback
You have strong attention to detail and consistently notice subtle issues others may overlook
You are adaptable and comfortable moving across topics, domains, and customer requirements
You have a background or experience in domains requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering)
You have excellent college-level mathematics skills
Nice-to-Have Specialties
Prior experience with RLHF, model evaluation, or data annotation work
Experience writing or editing high-quality written content
Experience comparing multiple outputs and making fine-grained qualitative judgments
Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
What Success Looks Like
You identify factual inaccuracies, reasoning errors, and communication gaps in model responses
You produce clear, consistent, and reproducible evaluation artifacts
Your feedback leads to measurable improvements in response quality and user experience
Mercor customers trust the quality of their AI systems because your evaluations surface issues before public release
About Mercor Mercor connects elite creative and technical talent with leading AI research labs, headquartered in San Francisco, CA. Our distinguished investors include Benchmark, General Catalyst, Peter Thiel, Adam D’Angelo, Larry Summers, and Jack Dorsey. Apply today and redefine digital creativity alongside groundbreaking AI technologies! We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.