Data Engineer

  • Harnham - Data & Analytics Recruitment
  • 16/11/2026
Full time Information Technology Telecommunications

Job Description

Lead Databricks Engineer

£600 - £800 per day

London Contract

An opportunity for an experienced Lead Databricks Engineer to take ownership of the future-state data platform for a large financial services organisation.

The business is looking to move away from traditional, slow batch processing towards a more modern, scalable and streaming-led data architecture. You'll play a key role in defining the data strategy, shaping the technical roadmap and leading delivery across a complex data environment.

The Company

A well-established organisation within the UK financial services and pensions sector, operating at significant scale with substantial volumes of legacy and live data.

The business is undertaking a major data transformation, focused on improving how data is ingested, managed, governed and made available for analytics. You'll work alongside internal teams and offshore delivery partners, helping to establish technical direction and drive progress across the wider programme.

The Role & Responsibilities

  • Define and lead the end-to-end data strategy, assessing the existing architecture and establishing a clear target-state vision and delivery roadmap.

  • Architect and develop a modern Databricks data platform, implementing medallion architecture principles across Bronze, Silver and Gold layers.

  • Establish a robust Gold layer to support reliable, scalable analytics and business reporting.

  • Drive the transition from monthly batch processing towards more frequent, near-real-time and streaming-based data ingestion where appropriate.

  • Take ownership of technical decisions, helping prioritise data use cases and balance architectural requirements with business objectives.

  • Work closely with internal engineering teams and offshore delivery partners to resolve blockers, establish clear ownership and maintain delivery momentum.

  • Engage with senior stakeholders, providing clear updates on technical progress, priorities, risks and recommended next steps.

Skills & Experience

  • Proven experience operating as a Lead or Principal Data Engineer, ideally within financial services, pensions or another regulated environment.

  • Strong hands-on experience designing and delivering enterprise-scale data platforms using Databricks.

  • Deep understanding of medallion architecture, including the design and optimisation of Bronze, Silver and Gold layers.

  • Experience modernising legacy data environments and integrating large, complex datasets into modern cloud-based data platforms.

  • Exposure to streaming technologies and real-time data ingestion, with experience moving away from traditional batch-led architectures.

  • Ability to define technical strategy, shape data architecture and translate business requirements into practical engineering solutions.

  • Experience leading delivery across distributed onshore and offshore engineering teams.

  • Strong stakeholder management and communication skills, with the ability to influence senior decision-makers and take ownership of technical direction.