As a Principal GCP Data Engineer, you'll be a true subject matter expert in using the data processing and management capabilities of Google Cloud to develop data-driven solutions for our clients.
You will typically lead a team or the solution delivery effort, demonstrating technical excellence through leading by example. You could be providing technical support, leading an engineering team or working across multiple teams as a subject matter expert who is critical to the success of a large programme of work.
Your team members will look to you as a trusted expert and will expect you to define the end-to-end software development lifecycle in line with modern best practices. As part of your responsibilities, you will be expected to:
- Develop robust data processing jobs using tools such as Google Cloud Dataflow, Dataproc and BigQuery
- Design and deliver automated data pipelines that use orchestration tools such as Cloud Composer
- Design end-to-end solutions and contribute to architecture discussions beyond data processing
- Own the development process for your team, building strong principles and putting robust methods and patterns in place across architecture, scope, code quality and deployments.
- Shape team behaviour for writing specifications and acceptance criteria, estimating stories, sprint planning and documentation.
- Actively define and evolve PA's data engineering standards and practices, ensuring we maintain a shared, modern and robust approach.
- Lead and influence technical discussions with client stakeholders to achieve the collective buy in required to be successful
- Coach and mentor team members, regardless of seniority, and work with them to build their expertise and understanding.
Qualifications
To be successful in this role, you will need to have:
- Experience delivering and deploying production ready data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc
- Experience developing end to end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam.
- Expert understanding of when to use a range of data storage technologies including relational/non relational, document, row based/columnar data stores, data warehousing and data lakes.
- Expert understanding of data pipeline patterns and approaches such as event driven architectures, ETL/ELT, stream processing and data visualisation.
- Experience working with business owners to translate business requirements into technical specifications and solution designs that satisfies the data requirements of the business.
- Experience working with metadata management products such as Cloud Data Catalog and Collibra and Data Governance tools like Dataplex
- Experience in developing solutions on GCP using cloud native principles and patterns.
- Experience building data quality alerting and data quarantine solutions to ensure downstream datasets can be trusted.
- Experience implementing CI/CD pipelines using techniques including as git code control/branching, automated tests and automated deployments.
- Comfortable working in an Agile team using Scrum or Kanban methodologies.
In addition to the above, we would be thrilled if you also had:
- Experience of working on migrations of enterprise scale data platforms including Hadoop and traditional data warehouses
- An understanding of machine learning model development lifecycle, feature engineering, training and testing
- Good understanding or hands on experience of Kafka
- Experience as a DBA or developer on RDBMS such as PostgreSQL, MySQL, Oracle or SQL Server
- Experience designing data applications to meet non functional requirements such as performance and availability
- You are pragmatic and already understand that writing code is only part of what a data engineer does.
- You can clearly communicate with both clients and peers, describing technical issues and solutions in both written and meeting/workshop contexts.
- You are able to clearly explain technical concepts to non technical audiences at all levels of an organisation.
- You are able to influence and persuade senior and specialist client stakeholders, potentially across multiple organisational boundaries without direct authority.
- You are a confident problem solver and troubleshooter.
- You are confident and generous in sharing your specialist knowledge, ideas and solutions.
- You are constantly learning and able to make others better by consciously teaching and unconsciously inspiring.
Additional information
Benefits package at PA:
- Private medical insurance
- Interest free season ticket loan
- 25 days annual leave with the opportunity to buy 5 additional days
- Company pension scheme
- Annual performance based bonus
- Life and Income protection insurance
- Tax efficient benefits (cycle to work, give as you earn, childcare benefits)
- Voluntary benefits (Dental, critical illness, spouse/partner life assurance)