About the Role

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams understand model capabilities and make informed decisions before deploying models in production.

What You'll Do

Evaluate newly released AI models across a range of capabilities and use cases.

Build benchmarks from the ground up, including dataset development, labeler coordination, and research publication.

Develop scalable and accurate methods for automatically evaluating generated text.

Work with engineering teams to implement evaluation methods in production systems.

Collaborate with AI labs and enterprise customers to translate evaluation needs into research initiatives.

What We're Looking For

At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation methods, NLP, or language models; applied experience is valued.

A record of published research, ideally related to NLP or benchmarking.

Strong Python skills and experience building clean, maintainable software; familiarity with Django, Flask, or similar Python HTTP frameworks.

Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant deep learning techniques.

Experience designing benchmarks and translating research findings into practical, scalable systems.

A computer science, machine learning, or related academic background; a master's or PhD is preferred, and a bachelor's degree with relevant applied experience may be considered.

Experience collaborating on engineering teams, including Git workflows, code review, and incorporating feedback.

Compensation & Benefits

Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.