Senior Data Engineer: Data Platform DevOps & Architecture

Edgefolio
London

About the Role

We’re seeking a Senior Data Engineer to transform our analytics infrastructure from monolithic SQL/PostgreSQL systems into a modern Azure-based data platform with DevOps at its core. This permanent position focuses on building an automated, self-service architecture designed for continuous evolution.

You’ll bring 5+ years of data engineering experience with proven expertise in implementing enterprise-scale data platforms and DevOps methodologies. You’ll architect solutions using modern patterns, establish automated data pipelines with CI/CD practices, implement infrastructure-as-code, and mentor our team while building systems that adapt to emerging technologies.

We need someone who prioritises maintainable, scalable, and adaptable systems. We’re committed to Azure but flexible on specific products – we value pragmatic solutions based on actual needs.

Structure & Expectations

  • Reports to: Product organisation – you’ll work closely with product teams to align data capabilities with business objectives
  • Location: London office 3 days per week, flexible remote work the rest of the time
  • Team: You’ll be the senior technical lead and engineer for data platform initiatives, working with business analysts, engineers, and product stakeholders
  • Timeline: Initial focus on migration (6 months), then platform evolution, DevOps maturity, and capability expansion

Core Technical Requirements

Azure Data Platform Architecture

  • Microsoft Azure data stack expertise: ADF, Delta Lake, ADLS Gen2, Synapse, Databricks (with flexibility to integrate non-Microsoft solutions)
  • Experience with PostgreSQL and SQL Server, including data replication/integration patterns for migration
  • Dimensional and relational data modeling expertise
  • Medallion architecture implementation (bronze→silver→gold)

Data Engineering & Pipelines

  • Proficiency in Python and SQL for ELT/ETL pipelines
  • Batch, streaming, and micro-batching architectures
  • CDC patterns and incremental load strategies at scale
  • Experience with Delta Change Data Feed or equivalent real-time sync mechanisms
  • Experience building ingestion pipelines from diverse external APIs (CRM, analytics, finance, observability tools) into a unified lakehouse

DevOps & Infrastructure

  • Infrastructure as Code (Terraform preferred)
  • CI/CD with Azure DevOps or GitHub Actions
  • Test automation for data pipelines (unit, integration, regression)
  • Azure networking and security best practices
  • Container orchestration (Kubernetes/AKS) experience beneficial
  • Event-driven architectures and API integration patterns
  • Monitoring and alerting with Azure Monitor and Log Analytics
  • Performance tuning and cost optimisation
  • Git version control proficiency

Governance, Compliance & Security

  • Strong understanding of information security requirements
  • Experience with Microsoft Purview or similar governance frameworks
  • GDPR/data protection implementation, including the right-to-be-forgotten
  • RBAC, encryption strategies, and audit logging
  • Data lineage and quality enforcement

Non-Technical Requirements

Communication & Documentation

  • Strong technical writing – please provide examples
  • Clear presentation skills for technical and non-technical audiences
  • Experience creating architecture diagrams, data flows, and runbooks

Leadership

  • Track record of training and upskilling teams
  • Experience with structured project planning: from high-level initiatives through quarterly planning to executable tasks
  • Comfort working with product managers and translating between technical and business contexts

What You’ll Do

  • Architect a modern data platform with DevOps principles for long-term business evolution
  • Establish automated engineering best practices for data, governance, and operations
  • Build robust batch and real-time processing pipelines (integrating web application data with analytics sources and external APIs)
  • Create comprehensive documentation and training materials
  • Translate business requirements into maintainable technical solutions
  • Mentor team members and drive continuous improvement

Why This Matters

You'll establish the foundation for our data platform's next decade. This platform will power our next generation of internal and customer products while empowering our business with real-time insights for faster, smarter decisions.

Application Requirements

Please provide:

  1. Technical documentation samples
  2. A description of a challenging data platform project you’ve delivered
  3. Your approach to implementing GDPR compliance in a lakehouse environment
Posted 2025-09-30

Recommended Jobs

Principle Public Protection Officer

Barnet Council
Barnet, Greater London

Job Category : Planning Location : Barnet Council Hours Per Week : 36.00 Start Date: Immediate Start Start Time : 09:00 End Time : 17:15 Salary: £17.98 Existing technical knowledge o…

View Details
Posted 2026-06-27

Chemistry Teacher - Outstanding School - Merton...

Marchant Recruitment
Merton, Greater London

Chemistry Teacher - Outstanding School - Merton (September Start) Are you a hard-working, motivated and driven teacher who wants good support and career progression? • Chemistry Teacher Role • T…

View Details
Posted 2026-06-27

Business Development Representative

AiDASH, Inc.
London

About AiDASH AiDASH is leading the PreventionFirst™movement for electric utilities and transforming grid resilience through its pioneering platform that unifies vegetation, asset, storm, and wildf…

View Details
Posted 2026-06-27

Computer Science Specialist Tutor | Remote & Flexible...

FindTutors
London

FindTutors is one of the leading platforms for private tutoring and personalised education in the United Kingdom. We connect students with qualified tutors across academic subjects, making learning …

View Details
Posted 2026-04-30

English Teacher - Outstanding Comprehensive School -...

Marchant Recruitment
London

English Teacher – Dynamic and well-resourced comprehensive school – Tower Hamlets A well-regarded and vibrant comprehensive school in Tower Hamlets, East London, is seeking an enthusiastic and inno…

View Details
Posted 2025-10-07

Project Architect, Global Services

Verkada
London

Who We Are Verkada is transforming how organizations protect their people and places with an integrated, privacy-sensitive AI-powered platform that includes solutions for video security, access co…

View Details
Posted 2026-06-27

Fullstack Engineer (Frontend focus)

CoreWeave Europe
London

CoreWeave is The Essential Cloud for AI™. Built for pioneers by pioneers, CoreWeave delivers a platform of technology, tools, and teams that enables innovators to build and scale AI with confidence. …

View Details
Posted 2026-06-27

Mandarin speaking Job - Sales Executive (Property Developer) - rj

People First Recruitment
London

Please follow us on WeChat to see all our Cantonese and Mandarin jobs, interview tips and London news: Your New Job Title: Mandarin speaking Sales Executive (Property Developer) The Skills You…

View Details
Posted 2026-05-12

Year 6 Teacher | Outstanding Primary School in Harrow |...

Marchant Recruitment
Harrow, Greater London

Are you a passionate and committed qualified Primary Teacher looking for an exciting Year 6 teaching role from January 2026? Do you want to join a forward-thinking school that nurtures confident, re…

View Details
Posted 2025-10-01

PE Teacher

Ethos Education
Isleworth, Greater London

PE Teacher - Hounslow - ASAP Start Are you a qualified PE Teacher looking for flexible teaching work in a supportive school environment? A welcoming school in Hounslow is seeking an enthusiastic…

View Details
Posted 2026-06-27