LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to help improve the performance of large language models. You will assess AI‑generated text for factual accuracy, coherence, safety, bias, and alignment with our guidelines, providing actionable feedback to the research and engineering teams.

Key responsibilities

• Evaluate and rank model outputs using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple responses to the same prompt and justify the preferred output.
• Craft adversarial prompts to expose biased or harmful behavior and report vulnerabilities.
• Provide concise feedback on recurring failure patterns to modeling and training teams.
• Participate in cross‑checking sessions to calibrate scoring standards and ensure inter‑rater reliability.
• Analyze under‑performing outputs, hypothesize root causes, and flag novel model behaviors.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in computer science or a related field.
• Proven experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
• Ability to interpret model outputs and explain strengths or weaknesses based on logic, tone, and factuality.

Required skills

• Prompt engineering and adversarial testing.
• Reinforcement Learning from Human Feedback (RLHF) methodology.
• A/B testing applied to AI models.
• Data annotation, cleaning, and dataset preparation for fine‑tuning.
• Analysis of IAA scores and calibration of evaluator judgments.