About Black Forest Labs

We’re the team building the foundational layer for visual intelligence: multimodal models trained jointly on images, video, audio, and action to see, generate, and understand the world.

Our founding team pioneered latent diffusion and Stable Diffusion, the breakthroughs that made generative AI accessible to millions. With FLUX, we're pushing into generative media at scale, machines that act in the physical world, and emerging applications in computer use.

Our FLUX models top public benchmarks and run in production in some of the world's biggest platforms. We're a small, fast growing research lab competing with the largest companies in the world. From our labs in Freiburg to San Francisco, we're building what comes next.

Why This Role

Frontier image and video models are now good enough that the hard question is no longer whether they work, but whether a new one is actually better, and how you'd know. Aesthetics, prompt following, editing fidelity, text rendering, character consistency: none of these collapse into a single number, and public benchmarks have fallen well behind the models they're meant to measure.

You'll work on the system BFL uses to answer that question. Every FLUX checkpoint, every finetuning and RL run, and every competitor release goes through our evaluation stack, and you'll help build and sharpen it. Product Research relies on what it shows to decide what to fix next, and to confirm the fix worked.

It gets harder from here. FLUX 3 is our first multimodal model, spanning image, video, audio and action, and each new modality needs its own way of being measured.

What You'll Work On

What you focus on will depend on where you're strongest and what the team needs most when you join.

In scope:

Building and maintaining the evaluation stack that every model checkpoint has to pass through and prove itself on, from automated benchmarking and LLM-as-judge pipelines, to human evaluation through panels (Prolific and otherwise).

Designing the breadth of evaluation across the taxonomy, from prompt adherence, editing fidelity, aesthetics, text rendering, subject and character consistency, to safety.

Evaluating the evaluation stack itself: checking that human and automated evals agree, that results stay stable over time, and improving them when they don't.

Benchmarking FLUX against the frontier models continuously, while obtaining a meaningful perspective on where we are better or worse than competitors and what such differences really mean.

Finding the ways in which the quality of the model's output falls short that are not captured by aggregate numbers.

Closing the loop, ideally, with the RL and finetuning researchers, by defining target metrics for them to aim for, and verifying that a new checkpoint improved the weakness without introducing new ones.

Designing how best to exploit the signals we already capture through the API and MCP, and what new signals we should capture, with help from research, data engineering and legal.

Maintaining the usefulness of existing metrics as their natural lifecycle brings up concerns of gaming and overfitting, and anticipating this for new metrics.

What We're Looking For

You have made meaningful contributions to the evaluation of generative models already. You understand that a benchmark figure without careful consideration of sample size and judge agreement with humans is at best uninformed optimism and at worst dangerously misleading, and you have been the person that found that inconvenient regression that nobody wanted to talk about.

You likely have:

Advanced degree (MS/PhD) in a relevant field or demonstrated equivalent practical experience

Practical experience with evaluating generative models, in a hands-on way. You may have worked with diffusion or flow-matching based image or video models already, but the capability to evaluate them in earnest transfers to other modalities too

Very strong Python and software engineering skills, with pipelines that run reliably and that other people can read if needed