Data Engineer

Annual releases gave way to two-week sprints years ago, and the platform now handles over 200 million API calls a day. We run one of the larger SaaS platforms for commercial insurance carriers, processing millions of policy transactions a year. On-call rotates weekly and pays a flat stipend whether or not the pager goes off.

Overview:

This is a new role. The team has doubled in size over the past year. Data engineering role supporting analysts and data scientists. Most of the work is in dbt, Airflow and the cloud warehouse. Remote, with occasional travel to Abu Dhabi (a few times a year).

What you'll be doing:

• Respond to pipeline failures and backfill data when needed
• Process large datasets with Spark on Databricks or EMR
• Document datasets and lineage in a catalogue such as DataHub or Atlan
• Maintain CI for data code and review pull requests
• Work with analysts to define metrics and fix discrepancies between reports
• Stream events through Kafka or Kinesis into the lake and warehouse
• Ingest data from APIs, SaaS tools and databases with Fivetran, Airbyte or custom Python

Required qualifications:

• 3+ years of data engineering, or backend work with data at volume
• Understanding of dimensional modelling and slowly changing dimensions
• Familiarity with Git, CI/CD and testing practices for data code
• Patience with messy source data and the habit of documenting assumptions

Preferred qualifications:

• Kafka, Flink or Spark Structured Streaming
• Dbt Certified Developer or Databricks certification

AED 29,000-AED 36,000 monthly

Why work with us:

• Medical insurance
• 30 days annual leave
• Annual flight ticket
• Housing allowance
• Education allowance for children

Please submit your CV in English.

Start: Flexible.

Demo job created on 11jobs for testing and demonstrations. It is not an actual opening at Plowshare Software.