Lead Databricks Engineer - Single Customer View

Lead Databricks Engineer - Single Customer View

Location: Bournemouth (Hybrid)Contract: 12-month FTCSalary: £85,000 + Benefits

Lead Databricks Engineer: About the Role

Join our Single Customer View (SCV) Programme, a strategic initiative within Financial Services aimed at delivering a unified, trusted view of customer data.

We're seeking a highly skilled Lead Databricks Engineer to design and implement scalable data pipelines that form the backbone of our Lakehouse platform, enabling accurate analytics, reporting, and regulatory compliance.

You'll work with cutting-edge technologies including Databricks, PySpark, and Azure Data Factory, applying best practices in data engineering and governance to support this critical programme.

Lead Databricks Engineer: Key Responsibilities

• Build and maintain Databricks pipelines (batch and incremental) using PySpark and SQL.
• Orchestrate end-to-end workflows with Azure Data Factory.
• Develop and optimise Delta Lake tables (partitioning, schema evolution, vacuuming).
• Implement Medallion Architecture (Bronze, Silver, Gold) for transforming raw data into business-ready datasets.
• Apply robust monitoring, logging, and error-handling frameworks.
• Integrate pipelines with downstream systems such as Power BI.
• Collaborate with analysts, business teams, and engineers to deliver consistent, well-documented datasets.
• Support deployments and automation via Azure DevOps CI/CD.
• Gather and refine requirements from business stakeholders.

Lead Databricks Engineer: About You

• Strong PySpark development skills for large-scale data engineering.
• Proven experience with Databricks pipelines and workflow management.
• Expertise in Azure Data Factory orchestration.
• Solid knowledge of Delta Lake and Lakehouse principles.
• Hands-on experience with SQL for data transformation.
• Familiarity with Azure services (ADLS/Blob, Key Vault, SQL).
• Knowledge of ETL/ELT frameworks and monitoring.

Desirable:

• Understanding of Microsoft Fabric.
• Experience with DevOps practices (CI/CD).
• Awareness of data governance and lineage tools.
• Experience in Financial Services.

Rewards & Benefits

• 30 days' holiday + BH
• Competitive pension scheme
• Annual bonus scheme
• Private medical insurance

+ Many more

About Us

We're committed to building an inclusive culture and value diversity in all forms. Flexible working options are available. Please note: we cannot offer Skilled Worker Visa Sponsorship for this role.

Reasonable Adjustments:

Respect and equality are core values to us. We are proud of the diverse and inclusive community we have built, and we welcome applications from people of all backgrounds and perspectives. Our success is driven by our people, united by the spirit of partnership to deliver the best resourcing solutions for our clients.

If you need any help or adjustments during the recruitment process for any reason, please let us know when you apply or talk to the recruiters directly so we can support you.