Senior Data Engineer
Department: Data
Employment Type: Permanent - Full Time
Location: London
Reporting To: Head of Data
Description
Role Overview: This is a broad, high-impact engineering role at Defaqto. You will be central to the structural transformation of how we manage and use market data - leading the technical build of a new core data model - while also owning the engineering that powers our research team's day-to-day operations and the wider data strategy.
What you'll do
We are at a pivotal moment in Defaqto's data journey. Our market data estate spans thousands of financial products, maintained by a team of 30+ researchers, and is being redesigned from the ground up into a unified market data model, replacing a flat, siloed structure with a scalable, enrichment-ready architecture.
This role sits within the Data Team but works across the organisation, partnering and collaborating closely with Research T& Insights, Product, and IT.
The successful candidate will help shape and build our next-generation data platform, contributing to architectural and tooling decisions alongside the CTO, Head of Data, and external data architect. Google BigQuery is the current lakehouse platform, but key tooling choices are still being defined.
We are looking for an experienced engineer with strong technical judgement who can turn strategy into scalable, production-ready solutions. Alongside platform delivery, this person will lead the development and evolution of our research automation capability and play a key role in Defaqto's wider data and automation strategy.
The initial focus will be implementing the agreed data model and technology approach while progressing automation initiatives in parallel.
Data Model - Technical Build
- Implement the physical data model based on the agreed logical design and technology recommendations agreed during the architecture phase
- Build and maintain the transformation layer in the agreed tooling: staging models, mart models, tests, and documentation
- Design and implement compatibility views that allow existing research tooling to continue operating during transition
- Own the data quality framework - tests, monitoring, and alerting across the data model and downstream consumers
- Support the phased migration of the research data entry platform to write natively to the new schema
- Contribute engineering perspective to ongoing architecture decisions as the data model evolves beyond the initial scope
Research Automation & Engineering
- Assess automation opportunities across research and data workflows and recommend the appropriate technical approach - whether rules-based, deterministic, or AI-assisted - based on the specific problem, data characteristics, and accuracy requirements
- Build and maintain automation pipelines that reduce manual research effort, selecting from the appropriate toolset including structured transformation, rules engines, and machine learning or LLM-based approaches where warranted
- Apply LLM and AI tooling to problems where unstructured data, variable formats, or language ambiguity make deterministic approaches insufficient - for example, extracting structured product attributes from provider documents or websites
- Ensure automation outputs - regardless of the method used - flow into the data model with appropriate quality controls and human-review checkpoints where confidence thresholds require it
- Be the primary technical partner for the Research & Insights team's data engineering needs
- Stay current with developments across the automation and AI engineering space and apply them with judgement, not by default
- Work with Research & Insights and Product stakeholders to identify high-value automation opportunities and maintain a prioritised delivery roadmap
Data Strategy Implementation
- Contribute to and implement Defaqto's broader data strategy across the data estate
- Support the integration of acquired datasets (e.g. pricing and claims data) into the data model framework
- Build and maintain the data catalogue and lineage documentation
- Work toward an enrichment-ready data architecture that combines product attributes, performance, behaviour, and experience data.
- Participate in evaluating and adopting new data tooling as the stack evolves
The opportunity
- A genuine opportunity to shape how a data-rich business transforms its core data infrastructure
- Broad scope - you will work across engineering, strategy, and automation rather than a narrow specialist track
- Collaborative team environment with a clear data strategy and leadership backing
- Hybrid working with flexibility
What you'll need to succeed Essential requirements
- Strong hands-on experience with SQL and data modelling for analytical workloads (OLAP)
- Proficiency with dbt and familiarity with the broader SQL transformation layer ecosystem (SQLMesh, Dataform, or equivalent) - the specific tooling will be confirmed during the architecture phase and we are looking for adaptability across this space rather than allegiance to one tool
- Experience with Google BigQuery or an equivalent cloud data warehouse - BigQuery is our existing lakehouse platform
- Proven ability to design and implement data pipelines and ETL/ELT processes
- Experience working with version control (Git) and treating data infrastructure as code
- Proficiency with Python for data engineering and automation work
- Experience assessing and implementing automation solutions - able to evaluate whether a problem calls for rules-based logic, a deterministic pipeline, or an AI/ML approach, and build accordingly
- Practical experience working with LLMs or AI APIs in an engineering context - understanding of when they are and are not the right tool
- Ability to translate business rules and logic into reliable, testable data models
- Strong communication skills - able to work with non-technical stakeholders including researchers and product teams
Desirable requirements
- Experience in financial services, insurance, or fintech data environments
- Familiarity with data quality frameworks such as Great Expectations or Soda
- Experience with Google Cloud Platform services beyond BigQuery (e.g. Dataplex, Cloud Composer, Pub/Sub)
- Exposure to .NET or relational SQL Server environments - relevant to legacy platform transition work
- Understanding of data cataloguing, lineage, and metadata management
- Experience applying ML or LLM-based approaches to document extraction, classification, or data enrichment problems
- Familiarity with LLM orchestration frameworks such as LangChain or LlamaIndex, or experience with RAG architectures - valued where directly applicable, not a requirement in isolation
- Familiarity with semantic or metrics layers (e.g. dbt Semantic Layer, Cube.dev)
Your approach to work: Engineering Standards & Ways of Working
- Champion engineering best practices across the data team: version control, code review, testing, documentation
- Ensure all data models and pipelines are maintainable, observable, and well-documented
- Contribute to defining and upholding data engineering standards across the organisatio
Important to know Location
This is a hybrid role where you'll work from the London office 3 times each week.
Right to Work
Applicants must already hold a legal right to work in the UK without time restrictions and without the need for future sponsorship. We are unable to provide Skilled Worker visa sponsorship.