Lead Data Engineer

Description:

This government body is looking for a collaborative Data Engineer to join the team to scope to scope modernisation of the data pipeline. Specifically, the purpose of this role is to identify how an analytics-engineering-led approach could be implemented technically to ensure there is a robust and maintainable pipeline delivered to high QA standards, how that approach could be congruent with the existing practices of their analytics engineering teams and how that approach could serve the three main customer groups.

Skills Required

Essential

• Excellent python, SQL and jinja skills
• Understanding and awareness of R
• Applied automated testing, for example with pytest
• Familiar with version control tools, such as git.
• Experience with cloud platforms, ideally AWS
• Experience of orchestrations tools, ideally airflow
• Experience of building CI/CD workflows, ideally with GitHub Actions
• Experience of reviewing, designing and developing data pipelines
• Experience of using dbt, including implementing dbt best practices
• Applied data modelling (conceptual, logical, physical) experience

Desirable

• Experience of infrastructure as code, ideally Terraform
• Data Lake Table Formats, such as Iceberg
• Kubernetes and container-based workflows
• Orchestration systems, such as Airflow
• Experience of guiding others to apply best practices and ensuring practices are up to date with latest developments in data engineering
• Working in or leading an Agile team