Lead Databricks Engineer - Hybrid - Permanent

Lead Databricks Engineer - Hybrid - Permanent

Overview
We are seeking an experienced Lead Azure Databricks Engineer to design, build and optimise our enterprise data platforms. The role requires deep technical expertise, strong delivery capability and experience working in highly regulated environments within the Lloyd's of London market.

You will play a key role in shaping our cloud data platform, enabling advanced analytics and ensuring secure, scalable and reliable solutions across the organisation. This is a fully hands-on engineering role.

Key responsibilities

• Design and build enterprise data platforms using Microsoft Azure services including Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure Functions and Azure Databricks

• Develop scalable pipelines using Delta Lake, PySpark and Unity Catalog

• Work with underwriting, actuarial, delegated authority, bordereaux, exposure management, reinsurance, finance and risk teams to deliver solutions aligned to Lloyd's market requirements and regulations including Solvency II

• Translate business requirements into scalable cloud data solutions with architects, SMEs and product owners

• Lead the resolution of complex technical challenges and drive delivery

• Promote best practices in data engineering, cloud architecture, CI/CD and data life cycle management

• Provide technical leadership and mentoring to engineers

• Improve platform performance, resilience, cost efficiency and observability

• Deliver reliable, high quality data pipelines across the platform

Required experience and skills

• Strong engineering experience with Microsoft Azure data services and Azure Databricks in large scale environments

• Deep knowledge of Delta Lake, medallion architecture, distributed compute and lakehouse data platforms

• Strong development skills in Python, PySpark and Spark SQL

• Experience implementing CI/CD for data platforms using Azure DevOps

• Knowledge of data governance, lineage, access control and secure cloud engineering

• Excellent communication skills and ability to work with senior stakeholders

• Proven delivery mindset with strong ownership and accountability in fast paced environments

To apply for this role please submit your CV or contact Dillon Blackburn (see below)

Tenth Revolution Group are the go-to recruiter for Data & AI roles in the UK offering more opportunities across the country than any other recruitment agency. We're the proud sponsor and supporter of SQLBits, Power Platform World Tour, and the London Fabric User Group. We are the global leaders in Data & AI recruitment.