Accountabilities:

Enterprise Architecture and Solution Design: Lead the architecture and design of scalable, production-ready Lakehouse solutions using the Databricks Data Intelligence Platform, including Medallion Architecture (Bronze, Silver, and Gold layers), Delta Lake, and Databricks SQL.
Data Governance and Security: Design and implement enterprise-wide governance and security frameworks using Unity Catalog, row- and column-level access controls, data masking, attribute-based access control (ABAC), and Delta Sharing.
Platform Modernization and Migration: Define and lead technical strategies for migrating legacy enterprise data warehouses and Hadoop environments, including Teradata, Netezza, Snowflake, and Cloudera, to Databricks-based architectures.
Data Ingestion and Processing: Establish scalable patterns for batch and real-time data processing using Structured Streaming, Delta Live Tables (DLT), Auto Loader, and Apache Spark optimization techniques.
Technical Leadership and Quality Assurance: Guide engineering teams through complex delivery initiatives, establish reference architectures, conduct architectural reviews, review code contributions, and implement standardized CI/CD practices using GitHub Actions, GitLab, Azure DevOps, Terraform, and Databricks Asset Bundles (DABs).
Performance Optimization: Identify and resolve complex Spark and Delta Lake performance challenges by optimizing query execution, partitioning, Z-Ordering, caching, and Photon configurations.
Cloud Cost Management: Establish FinOps practices and compute governance standards, including cluster policies, autoscaling, auto-termination, serverless configurations, and cost allocation models to optimize Databricks consumption.
AI and Advanced Analytics Integration: Collaborate with machine learning and AI specialists to integrate MLflow, Feature Store, Model Serving, Vector Search, and Retrieval-Augmented Generation (RAG) pipelines into enterprise data platforms.
Client Advisory and Stakeholder Engagement: Serve as a trusted technical advisor to senior client stakeholders, including Chief Architects, Directors of Engineering, and Chief Data Officers. Lead technical workshops, architecture design sessions, and proof-of-concept evaluations.
Technology Partner Collaboration: Work closely with Databricks Partner Solutions Architects and field engineering teams to assess emerging capabilities, evaluate preview features, and ensure alignment with platform best practices.
Architectural Innovation and Knowledge Sharing: Develop reusable architectural blueprints, migration accelerators, reference architectures, and standardized configurations to strengthen internal technical capabilities and delivery practices.

Requirements

Professional Experience: At least 8 years of experience in data architecture, data engineering, or enterprise software development, including a minimum of 3 years designing and architecting enterprise solutions based on Apache Spark and the Databricks Data Intelligence Platform.
Databricks Certification: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional certification.
Cloud Architecture Expertise: Strong architectural proficiency in at least one major cloud platform (AWS, Azure, or GCP), including VPC/VNet networking, private endpoints, cloud identity and access management, Entra ID integration, and cloud object storage solutions such as S3, ADLS Gen2, or GCS.
Data Modeling and Governance: Advanced knowledge of modern data modeling methodologies, including dimensional modeling, Data Vault, and Medallion Architecture, with hands-on experience implementing Unity Catalog across multiple workspaces in production environments.
Programming and Automation: Advanced proficiency in Python, PySpark, and SQL, combined with practical experience using Infrastructure as Code tools such as Terraform and orchestration platforms such as Databricks Workflows or Airflow.
Technical Delivery Leadership: Demonstrated experience leading technical delivery teams, translating complex business and functional requirements into architectural designs, technical specifications, and user stories, and resolving critical implementation challenges.

How Jobgether works

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1