Senior Data Engineer with Databricks and PySpark, Exposure Platform

Project overviewThe Exposure Platform is a long-running enterprise data transformation initiative focused on replacing legacy SQL-based logic with modern, scalable data processing solutions. The platform handles large data volumes and supports multiple business domains through a shared and business-critical codebase. The work emphasises maintainability, performance, and strong software engineering practices across the organisation.TeamYou will work in a small cross-functional data engineering team consisting of senior data engineers and software engineers. The team collaborates closely through code reviews, shared design discussions, and agreed engineering standards while contributing to a common codebase. The team is currently expanding and values structured collaboration and technical ownership.Position overviewWe are looking for a Senior Data Engineer to support a large-scale transformation from SQL Server-based systems to a Databricks and Delta Lake platform. You will focus on enterprise-grade data engineering and software development, building maintainable and scalable data processing solutions used by multiple teams. This role is not focused on analytics or reporting, but on core data transformation and platform development.Technology stackDatabricks, Delta Lake, Python, PySpark, SQL Server, Azure Data Factory, Azure DevOps, Git, CI CD pipelinesResponsibilitiesRead, understand, and reason about complex SQL stored procedures and embedded business logicRedesign and implement existing SQL logic as clean and maintainable Python and PySpark code in DatabricksDevelop production-grade transformation code using reusable packages, modules, and componentsApply software engineering best practices, including clean code, object-oriented design, modularisation, and refactoringDesign and evolve data models across Bronze, Silver, and Gold layersWork with very large data volumes and highly parallel event-driven data transformationsParticipate actively in code reviews and technical design discussionsContribute to the stability, scalability, and long-term maintainability of the shared data engineering codebaseRequirementsStrong experience with Python and PySpark in production data engineering environmentsHands-on experience working with Databricks and Delta LakeStrong SQL skills with the ability to read, analyse, and translate complex stored proceduresExperience working in large shared codebases beyond notebook-based developmentSolid understanding of object-oriented programming and software engineering principlesExperience applying clean code practices, refactoring, and maintainable designStrong background in data modelling, including transactional and analytical modelsExperience working with layered data architectures such as Bronze, Silver, and GoldAbility to analyse existing code line by line and explain technical and business logic clearlyNice to haveExperience with Power BI for data consumption or validationExposure to enterprise-scale data platforms in complex organisational environments