Lead Data Engineer

Public sector agencies, utilities and large health systems licence our asset management platform under multi-year contracts. We run one of the larger SaaS platforms for commercial insurance carriers, processing millions of policy transactions a year. Releases go out Tuesday mornings, never on Fridays.

Job summary

Stonehill Data is hiring a Lead Data Engineer to work remotely from India. This hire will help replace a set of fragile nightly scripts with tested, monitored pipelines. Fully remote. You can work from anywhere in India. We do not have many meetings. Most updates happen in writing. A stable internet connection and a backup power option are required.

Key responsibilities

• Build batch and streaming pipelines with Airflow, Dagster or Prefect.
• Document datasets and lineage in a catalogue such as DataHub or Atlan.
• Ingest data from APIs, SaaS tools and databases with Fivetran, Airbyte or custom Python.
• Process large datasets with Spark on Databricks or EMR.
• Respond to pipeline failures and backfill data when needed.
• Design table layouts and partitioning for Parquet, Delta Lake or Iceberg.
• Tune slow queries and expensive jobs.
• Maintain CI for data code and review pull requests.

What you need

• Working knowledge of AWS, GCP or Azure data services.
• Familiarity with Git, CI/CD and testing practices for data code.
• Hands-on with an orchestration tool such as Airflow.
• Understanding of dimensional modelling and slowly changing dimensions.
• Patience with messy source data and the habit of documenting assumptions.
• Strong Python for data processing.

Preferred

• Dbt Certified Developer or Databricks certification.
• Terraform for data infrastructure.
• Reverse ETL tools like Hightouch or Census.
• Kafka, Flink or Spark Structured Streaming.

What you get

• Flexible hours.

Candidates must be eligible to work in India.

Please submit your CV in English.

Duration: 9 months. Start: Immediately.

Demo job created on 11jobs for testing and demonstrations. It is not an actual opening at Stonehill Data Pvt Ltd.