Data Engineer II - Spark

Product, engineering and support teams work out of offices in Toronto, Ontario and two nearshore delivery centers. We run one of the larger SaaS platforms for commercial insurance carriers, processing millions of policy transactions a year. On-call rotates weekly and pays a flat stipend whether or not the pager goes off.

Job summary

The previous person in this role moved to another team. This hire will help replace a set of fragile nightly scripts with tested, monitored pipelines. This is a full-time remote role. We use an employer of record in the US, so you get a proper employment contract, not a freelance agreement. Expect to work a schedule that overlaps with US Eastern business hours until about noon ET.

Key responsibilities

• Model data in dbt using staging, intermediate and mart layers
• Build batch and streaming pipelines with Airflow, Dagster or Prefect
• Ingest data from APIs, SaaS tools and databases with Fivetran, Airbyte or custom Python
• Handle PII according to retention, masking and access policies
• Work with analysts to define metrics and fix discrepancies between reports
• Process large datasets with Spark on Databricks or EMR

Minimum qualifications

• 2+ years of data engineering, or backend work with data at volume
• Hands-on with an orchestration tool such as Airflow
• Strong Python for data processing

Compensation and benefits

Compensation: $129,000 - $166,000 a year

• Wellness allowance
• Public holidays observed per your country
• Co-working space stipend
• Paid parental leave

Thistlebeam is an Equal Opportunity Employer, including disability and protected veteran status.

Interview process: a phone call, a meeting with the hiring manager, and a final conversation with the team.

Start: Within 30 days.

Demo job created on 11jobs for testing and demonstrations. It is not an actual opening at Thistlebeam.