SC Cleared Python Data Engineer - Azure Cloud

Job Title: SC Cleared Python Data Engineer - Azure Cloud

Contract: 12 months

Rate: Up to £410/day (Inside IR35)

Location: UK Based - Remote

Start: January 2026

Clearance: Active SC Clearance required

Role Overview

We are seeking an SC Cleared Python Data Engineer with strong hands-on experience in PySpark, Delta Lake, and Azure cloud services. The role focuses on designing and delivering scalable, well-tested data pipelines, with particular importance placed on the ability to understand, explain, and design PySpark architectures and demonstrate deep, production-grade Python expertise.

You will work in a containerised, cloud-native environment, delivering maintainable, configurable, and test-driven solutions as part of a multi-disciplinary engineering team.

Key Responsibilities

• Design, develop, and maintain data ingestion and transformation pipelines using Python and PySpark.
• Clearly articulate PySpark architecture, execution models, and performance considerations to both technical and non-technical stakeholders.
• Implement unit and BDD testing (Behave or similar), including effective mocking and dependency management.
• Design and optimise Delta Lake tables to support ACID transactions, schema evolution, and incremental processing.
• Build and manage Docker-based environments for development, testing, and deployment.
• Develop configuration-driven, reusable codebases suitable for multiple environments.
• Integrate Azure services including Azure Functions, Key Vault, and Blob/Data Lake Storage.
• Optimise Spark jobs for performance, scalability, and reliability in production.
• Collaborate with Cloud, DevOps, and Data teams to support CI/CD pipelines and environment consistency.
• Produce clear technical documentation and follow cloud security and data governance best practices.

Required Skills & Experience

• Strong Python expertise, with a demonstrable depth of experience in designing modular, testable, production-quality code.
• Proven experience explaining and designing PySpark architectures, including distributed processing and performance tuning.
• Hands-on experience with Behave or similar BDD frameworks, including mocking and patching techniques.
• Solid understanding of Delta Lake concepts, transactional guarantees, and optimisation strategies.
• Experience using Docker across development and deployment workflows.
• Practical experience with Azure services (Functions, Key Vault, Blob Storage, ADLS Gen2).
• Experience building configuration-driven applications.
• Strong problem-solving skills and ability to work independently in agile environments.

Desirable Experience

• Databricks or Synapse with Delta Lake.
• CI/CD pipelines (Azure DevOps or similar) and infrastructure-as-code.
• Knowledge of Azure data security and governance best practices.
• Experience working in distributed or multi-team environments.