About the Role

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make better-informed decisions about production use.

What You'll Do

Evaluate newly released AI models across a range of use cases.

Build benchmarks from the ground up, including dataset development and labeler coordination.

Develop scalable and accurate automated evaluation methods for generated text.

Work with engineering teams to implement and scale evaluation systems.

Collaborate with AI labs and enterprise teams to translate evaluation needs into research projects.

What We're Looking For

At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation, or language models.

A track record of published research, ideally in NLP or benchmarking, and hands-on experience creating benchmarks or evaluation methods.

Strong Python skills and experience building clean, maintainable software; Django, Flask, or similar Python HTTP server experience is useful.

Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant deep learning techniques.

Strong computer science fundamentals and the ability to turn research findings into practical systems.

Experience collaborating through team development workflows, Git, code review, and feedback.

A degree in Computer Science, Machine Learning, or a related field; a Master's or PhD is preferred. Candidates with a Bachelor's degree and 1 to 3 years of applied experience may also be considered.

Compensation & Benefits

Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California.