Databricks Technical Lead

Databricks Technical Lead l Contract | 12 Months | Outside IR35 l Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)

The role
Method Resourcing have been asked by one of our consulting partners to find a hands-on Databricks Technical Lead to design and deliver a new Azure Databricks platform for a major life sciences organisation replacing its enterprise data and analytics stack.

The programme uses AI and LLM tooling to reverse-engineer existing pipelines, business logic and reports into documented, tested code. You'll mentor three senior data engineers, own the technical architecture and stay hands-on in development.

What you'll do

• Architect and build the platform on Azure Databricks (Delta Lake, Unity Catalog, medallion architecture)
• Use AI agents and LLMs to reverse-engineer Legacy reports, queries and transformations, and build repeatable AI-assisted processes for code generation, testing and migration
• Build automated testing and reconciliation to validate migrated data against existing reporting
• Integrate SAP, Azure Data Lake, Salesforce and other systems, and design gold-layer models for Power BI
• Set engineering standards (Workflows, CI/CD, IaC, governance) and mentor the team through to UAT and cutover

What you'll need

• Extensive hands-on experience delivering and technically leading enterprise solutions on Azure Databricks
• Demonstrable use of AI, LLMs or agents to reverse-engineer data pipelines or reporting logic
• Strong Python, PySpark and SQL
• Deep knowledge of Delta Lake, Unity Catalog, Workflows and medallion architecture
• Automated validation, reconciliation and testing experience
• Azure data engineering (ADLS Gen2, ADF)
• Databricks CI/CD (Terraform, Asset Bundles or Azure DevOps)
• SAP ECC or S/4HANA integration experience
• Dimensional modelling and Power BI semantic model familiarity
• Proven team leadership while staying hands-on

Desirable

• Databricks certification (Data Engineer Professional or Solutions Architect)
• Platform migration, carve-out or separation experience
• Reusable AI agent frameworks for data engineering
• Strong governance and data quality experience
• Consulting background

Contract

• outside IR35
• £900-£1,000 per day
• Remote with some travel required
• 12 month contract

If this is you, send your CV to (see below) and if suitable, i'll be in touch. As you can appreciate, this role will receive a high volume of candidates so I will do my best to respond if possible.

Databricks Technical Lead l Contract | 12 Months | Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)

RSG Plc is acting as an Employment Business in relation to this vacancy.