Senior Databricks Engineer

Role Title: Sr.Databricks Engineer
Location: Glasgow
Days on site: 2-3
Pay Rate: £405 per day

MUST BE PAYE THROUGH UMBRELLA

Role Description:

We are currently migrating our data pipelines from AWS to Databricks, and are seeking a Senior Databricks Engineer to lead and contribute to this transformation. This is a hands-on engineering role focused on designing, building, and optimizing scalable data solutions using the Databricks platform.

Key Responsibilities:
. Lead the migration of existing AWS-based data pipelines to Databricks.
. Design and implement scalable data engineering solutions using Apache Spark on Databricks.
. Collaborate with cross-functional teams to understand data requirements and translate them into efficient pipelines.
. Optimize performance and cost-efficiency of Databricks workloads.
. Develop and maintain CI/CD workflows for Databricks using GitLab or similar tools.
. Ensure data quality and reliability through robust unit testing and validation frameworks.
. Implement best practices for data governance, security, and access control within Databricks.
. Provide technical mentorship and guidance to junior engineers.

Must-Have Skills:
. Strong hands-on experience with Databricks and Apache Spark (preferably PySpark).
. Proven track record of building and optimizing data pipelines in cloud environments.
. Experience with AWS services such as S3, Glue, Lambda, Step Functions, Athena, IAM, and VPC.
. Proficiency in Python for data engineering tasks.
. Familiarity with GitLab for version control and CI/CD.
. Strong understanding of unit testing and data validation techniques.

Preferred Qualifications:
. Experience with Databricks Delta Lake, Unity Catalog, and MLflow.
. Knowledge of CloudFormation or other infrastructure-as-code tools.
. AWS or Databricks certifications.
. Experience in large-scale data migration projects.
. Background in Finance Industry.

Carbon60, Lorien & SRG - The Impellam Group STEM Portfolio are acting as an Employment Business in relation to this vacancy.