Accountabilities:

Streaming Data Pipeline Development: Design, build, and maintain robust Apache Spark Structured Streaming applications that process Change Data Capture (CDC) events into reliable, near-real-time data models.
Stream Processing Reliability: Implement and optimize stateful processing, slowly changing dimensions (SCD), checkpointing, state stores, late-arriving and partial records, and safe recovery mechanisms to ensure accurate and consistent data processing.
AWS Infrastructure Management: Own and scale AWS EMR clusters, managing instance sizing, resource allocation, managed scaling, multi-cluster deployments, driver and executor memory optimization, and workload consolidation to improve efficiency and reduce infrastructure costs.
Lakehouse Management: Maintain and optimize Apache Iceberg tables at scale by designing partitioning strategies and implementing compaction, snapshot expiration, and delete-file cleanup, including specialized maintenance for high-churn tables.
Data Source Integration: Onboard new business domains onto the streaming platform by configuring CDC sources and destinations through Estuary Flow, setting up Kafka-compatible topics, validating backfills in Snowflake, and coordinating with source database owners.
Production Troubleshooting: Investigate and resolve complex production incidents across the data platform, addressing issues such as connection failures, offset regressions, token expiration, schema inconsistencies, and write conflicts while focusing on root-cause resolution.
Monitoring and Observability: Develop monitoring capabilities and diagnostic tools to improve incident detection and resolution, including integrating Spark driver and executor logs with Grafana Loki, implementing watchdog mechanisms, and creating AI-assisted operational runbooks.
CI/CD and Developer Experience: Improve deployment workflows, configuration management, automated pipelines, and operational safeguards to enable engineering teams to independently and safely deploy new data models.
Cross-Team Collaboration: Support partner engineering teams with ad hoc data ingestion, backfill requests, technical escalations, and platform-related guidance.

Requirements

At least 7 years of professional experience in data engineering, including a minimum of 3 years operating production-grade streaming data pipelines.
Apache Spark Expertise: Strong hands-on experience with Apache Spark, particularly Structured Streaming, state stores such as RocksDB, checkpointing, stateful processing, and performance optimization.
Lakehouse Technologies: Production experience with modern lakehouse table formats, ideally Apache Iceberg, including partitioning strategies, table maintenance, compaction, and snapshot management. Experience with Delta Lake or Apache Hudi is also relevant.
Cloud Infrastructure: Experience operating Spark workloads on AWS EMR, Amazon EKS, or comparable platforms, with a strong understanding of cluster management, capacity planning, autoscaling, resource allocation, and infrastructure cost optimization.
Streaming and CDC: Solid understanding of Change Data Capture principles and Kafka semantics, including offsets, consumer configurations, message delivery guarantees, and event processing reliability.
Programming Skills: Strong proficiency in Python and SQL. Experience with Scala or Java is considered an advantage.
Orchestration and Automation: Experience with Apache Airflow or similar workflow orchestration tools, alongside CI/CD pipelines and infrastructure-as-code practices.
Troubleshooting and Analytical Skills: A systematic approach to debugging, with the ability to interpret logs, investigate complex issues, formulate hypotheses, and validate solutions using data.
Communication and Collaboration: Ability to work independently while collaborating effectively with distributed engineering teams and cross-functional stakeholders.
Nice to Have: Experience with Snowflake, particularly querying external or Iceberg tables, as well as monitoring and observability tools such as Grafana, Loki, or Alloy.

How Jobgether works

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1