Data Engineer

PWK Software employs 330 people across engineering hubs in Sydney and three other locations, building ERP software for mid-market manufacturers.

Job summary

We're looking for another Data Engineer to build and maintain the pipelines that feed our warehouse, dashboards and models. We work hybrid out of Sydney, usually three days in, two days remote.

What you'll be doing

• Write data quality tests with dbt or Great Expectations and alert on failures.
• Process large datasets with Spark on Databricks or EMR.
• Work with analysts to define metrics and fix discrepancies between reports.
• Model data in dbt using staging, intermediate and mart layers.
• Manage warehouse objects, roles and spend in Snowflake, BigQuery or Redshift.
• Tune slow queries and expensive jobs.
• Stream events through Kafka or Kinesis into the lake and warehouse.
• Document datasets and lineage in a catalogue such as DataHub or Atlan.

Qualifications

• 2+ years of data engineering, or backend work with data at volume.
• Familiarity with Git, CI/CD and testing practices for data code.
• Hands-on with an orchestration tool such as Airflow.
• Working knowledge of AWS, GCP or Azure data services.
• Expert SQL, including window functions and query tuning.
• Experience with a cloud data warehouse (Snowflake, BigQuery, Redshift or Synapse)

Nice to have

• Kafka, Flink or Spark Structured Streaming.
• Dbt Certified Developer or Databricks certification.

Compensation: $155,000 - $190,000 per year

Benefits

• Superannuation on top of salary.
• Paid parental leave.
• Hybrid working.
• Salary packaging.
• 17.5% leave loading.
• Novated leasing.

Start: Within 30 days.

Demo job created on 11jobs for testing and demonstrations. It is not an actual opening at PWK Software.