Staff Data Engineer (Databricks specialist)
- Market p25–p75
- £99k–£112k
- Opened · 30d
- +3,920
- Closed · 30d
- −2,668
- Opened · 7d
- +388% of open
- All data engineering
- 3,878open
Accountabilities:
- Own and drive the technical architecture of complex, cross-functional data initiatives, covering ingestion, transformation, storage, and serving layers.
- Design, develop, and maintain scalable, high-performance data pipelines and distributed data platforms using Databricks and cloud-native technologies across Azure, AWS, or GCP.
- Lead enterprise Master Data Management (MDM) initiatives, including golden records, entity resolution, match-and-merge processes, data domains, reference data, hierarchies, and data stewardship.
- Architect and integrate agentic AI and LLM-powered solutions, including autonomous agents, RAG pipelines, AI copilots, and orchestration frameworks, into enterprise data platforms.
- Build and maintain data foundations for machine learning and AI applications, including feature stores, vector databases, embeddings, and ML/LLMOps pipelines.
- Design and deliver backend data services and APIs using REST or GraphQL, ensuring curated datasets are accessible to applications, analytics, and business intelligence platforms.
- Establish and promote engineering standards covering data modeling, quality, testing, observability, CI/CD, deployment, and responsible AI-assisted development.
- Implement enterprise-wide data governance, lineage, cataloging, and quality frameworks to ensure data consistency, reliability, and transparency.
- Lead technical design reviews, provide architectural guidance, and influence engineering decisions across multiple teams.
- Collaborate with product, analytics, and engineering leadership to define scalable data roadmaps aligned with business objectives and AI-driven opportunities.
- Identify and resolve systemic performance, reliability, and scalability challenges across the data ecosystem.
- Mentor and coach senior and mid-level engineers, promoting technical excellence and best practices in data engineering, MDM, and AI.
- Evaluate emerging technologies, frameworks, and industry trends to continuously improve platform capabilities, engineering efficiency, and resilience.
Requirements
- Bachelor's degree in Computer Science, Engineering, Management Information Systems, or a related discipline is preferred.
- 12+ years of professional experience in data engineering or software engineering, with a proven track record of architecting enterprise-scale data platforms and pipelines.
- Expert-level SQL skills and strong proficiency in Python. Experience with Scala or Java is an advantage.
- Extensive hands-on experience with Databricks and modern cloud data platforms such as Snowflake, Azure Synapse, BigQuery, or Amazon Redshift.
- Strong understanding of cloud-native architecture, distributed systems, data modeling methodologies (dimensional modeling, Data Vault, Lakehouse), and ETL/ELT architectures.
- Proven experience designing and implementing enterprise MDM solutions, including entity resolution, match-and-merge, golden records, reference data, and hierarchy management. Familiarity with Informatica, Reltio, Profisee, or similar platforms is highly valued.
- Practical experience building or integrating agentic AI systems, LLM-powered applications, RAG pipelines, and AI agent orchestration frameworks such as LangChain, AutoGen, Semantic Kernel, or MCP.
- Experience developing backend data services and REST/GraphQL APIs, with the ability to contribute across the technology stack.
- Strong knowledge of relational and NoSQL databases, as well as modern data lake and lakehouse formats such as Delta Lake, Apache Iceberg, and Parquet.
- Deep understanding of CI/CD pipelines, Infrastructure as Code (IaC), DevOps, and DataOps practices.
- Demonstrated ability to lead large-scale technical initiatives involving multiple teams and stakeholders.
- Excellent communication, collaboration, and problem-solving skills, with the ability to influence technical decisions without direct managerial authority.
- Proven mentoring and coaching capabilities, with a passion for developing engineering talent and promoting knowledge sharing.
Nice to have:
- Experience with data governance, lineage, and cataloging solutions such as Unity Catalog, Microsoft Purview, Collibra, or Alation.
How Jobgether works
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1