DataBricks Engineer

You must have

At least 2 years of commercial Databricks experience (current)

Strong AWS Skills

Be able to work 2 days a week onsite in Central London

Ideally have worked on a large scale DataBricks migration project previously

Strong understanding of Medallion Architecture

DataBricks certification would be welcome but we prefer hands on experience over theoretical

About Red Badger

We’re a tech and product consultancy, building enterprise software products for blue chip companies, multinationals and PLC’s. The best thing about Red Badger is; of course; the team. We’ve been around for 15+ years now and we are over 120 strong. We are really proud of our people; we support and learn a lot from each other; we work really hard but have fun doing it. We are a diverse group made up of 22 different nationalities, speaking 17 different languages. Headquartered in London we also have offices in Leeds and Cape Town.

The Role

Red Badger is looking for a DataBricks Engineer to join our Data & AI practice. This is a delivery-focused role: you'll work directly with clients and Red Badger consultants, from all the cross functional disciplines, to shape complex technical solutions as an output from discovery phase, and then lead a team of data engineers to build the solution through to production. You'll work closely with the Director, Data & AI, and partner on pre-sales and proposals to add technical credibility and deepen our client partnerships.

You love solving complex problems, and customer value is your north star. You are comfortable consulting with technology stakeholders outside of your team. You are a technology polyglot, and an evangelist for agile ways of working.

Day to day, you will spend roughly half of your time hands on, pairing with data engineers on the team, and helping them pick the right design to solve the problem. In the other half, you’ll be supporting the other functions (delivery, product, software engineering) and our clients with decisions impacted by the technical strategy of the project and providing support and line management to data engineers on your team.

Key Responsibilities

Discipline Expertise and Leadership

Technology breadth: Deep, production-grade experience across AWS &/or Azure native services, with expert-level Databricks and/or Snowflake and an Agile delivery mindset.
Storage & Lakehouse: RDS, DynamoDB, S3, SageMaker Lakehouse,
Processing & ETL: Glue, EMR, Athena, CDC/DMS, DataBrew, Kinesis, dbt
Data management & metadata: Lake Formation, Glue Data Catalog, SageMaker Catalog / DataZone
AI: Bedrock (including AgentCore) and SageMaker AI
Storage & lakehouse: SQL, PostgreSQL, Cosmos, Blob, Data Lake, OneLake, Fabric Warehouse
Processing & ETL: Fabric Data Factory, Data Factory CDC, Fabric/Synapse Spark, ADF, Synapse Pipelines, Fabric, dbt
Data management & metadata: Purview
AI: Foundry, (including Agent Service), ML
On AWS, you're fluent across the core services, for example:
On Azure, you’re fluent across the core services, for example:
Data Product mentality: You build Data Products where every pipeline and dataset is discoverable, addressable and secure by design.
Data capabilities: You know the full data lifecycle, from discovery, storage, pipelines, ETL and streaming to data quality, governance, metadata management and cataloguing, through to advanced analytics and decision-making.
AI: You understand how AI is built and deployed in modern data platforms (Databricks, Snowflake) and cloud-native services like Amazon Bedrock, with enough depth to guide teams and advise clients, spanning LLMs and SLMs, agentic automation, Human in the Loop and MLOps.
Data Architecture Design: You know when to use RAG (Retrieval-Augmented Generation), Serverless, DWH models, Data Mesh, Data Fabric, Medallion and Data Products.
Hands on mindset: Fluent in the language of data, whether SQL, Python, PySpark, R or Scala. Open formats including Parquet, Iceberg and JSON.

Agile Ways of Working

Run and contribute to agile ceremonies, helping to shape and improve the agile process