Data Engineer – Databricks (Performance & Security)

About the RoleWe are looking for a Data Engineer focused on Databricks platform excellence, mastering features, security, performance optimization, and operational best practices. You will play a key role in improving how we leverage Databricks across our data platform, ensuring scalable, secure, and efficient workloads. This role suits someone passionate about infrastructure, governance, and performance — and who’s eager to expand knowledge into streaming technologies.Key Responsibilities Optimize Databricks performance across jobs, clusters, and data workloads. Implement and maintain governance, security, and permissions via Unity Catalog and Terraform. Contribute to Databricks feature enablement — Delta Live Tables, Databricks Asset Bundles (DABs), and cost efficiency improvements. Monitor cluster and job performance using Grafana and internal observability tools. Support data engineers in troubleshooting and fine-tuning jobs and SQL queries. Explore and progressively implement streaming features (Structured Streaming, Autoloader, Kafka integration). Manage Databricks environments integrated with Airflow on AKS, Azure ADLS Gen2, and Terraform. Collaborate with DataOps to ensure high availability, logging, and monitoring coverage. Implement performance and security benchmarks for production workloads. Key Requirements Deep knowledge of Databricks configuration, optimization, and administration. Experience with PySpark, SQL, and data performance tuning. Familiarity with Unity Catalog, DABs, and Terraform integration. Strong understanding of data governance, cost optimization, and cluster policies. Basic to intermediate experience with Kafka or interest in learning streaming processing. Nice to Have Experience with modern data orchestration tools. Knowledge of additional data lakes or storage solutions. Understanding of cloud architecture and services. Familiarity with version control systems, particularly Git. Other DetailsThis role provides flexibility with remote working options, offering a chance to engage in cutting-edge technologies while ensuring data excellence. Candidates should be prepared for ongoing learning and adaptation to new tools and methodologies, particularly within the Databricks ecosystem.