About this role
The team is seeking an LLM / Agentic Evaluation Rig Engineer to develop and maintain the evaluation system that determines the quality of AI outputs before they are delivered to customers. This role is critical as it involves ensuring that the AI's commentary is reliable and accurately reflects externally reported financials. You will take ownership of the evaluation infrastructure, including datasets, scoring mechanisms, harnesses, and continuous integration gates that ensure the AI meets strict quality standards.
Key Responsibilities:
- Design and implement evaluation frameworks for AI outputs.
- Develop and manage datasets used for training and evaluation.
- Create scoring systems to assess the quality of AI-generated content.
- Establish continuous integration gates to ensure compliance with quality metrics.
- Collaborate with cross-functional teams to integrate evaluation processes into the AI development lifecycle.
- Monitor and report on the performance of AI outputs, making adjustments as necessary.
Required Skills & Qualifications:
- Strong experience in AI evaluation methodologies and metrics.
- Proficiency in programming languages such as Python or similar.
- Familiarity with machine learning frameworks and tools.
- Experience in data management and analysis.
- Excellent problem-solving skills and attention to detail.
- Ability to work collaboratively in a fast-paced environment.
Experience:
- 5-8 years of relevant experience in AI evaluation or related fields.
What we offer:
- Opportunity to work on cutting-edge AI technologies.
- Collaborative and innovative work environment.
- Professional development and growth opportunities.
Applications are read by our talent team, usually within two working days.
If you look like a fit we will call you, and you will hear from us either way.