Senior / Lead Data Engineer - DataBricks

**Overview** Owns the design and optimisation of scalable data pipelines and lakehouse solutions on Databricks, enabling analytics and data\-driven decision\-making across the organisation. Acts as a technical leader — setting standards, mentoring engineers, and shaping the platform's evolution. This is a fixed term contract initially with a view to permanency. **Core Competencies** * Technical leadership and architectural decision\-making on Databricks * Delivery of business value through scalable lakehouse solutions * Strong stakeholder communication and cross\-team collaboration * Continuous improvement and engineering excellence **Key Responsibilities** * *Pipeline Engineering* — Design and own scalable ETL/ELT pipelines (PySpark, Spark SQL, Delta Live Tables) following medallion architecture, with strong reliability, monitoring, and cost/performance optimisation. * *Data Modelling \& Lakehouse* — Govern Delta Lake models and modelling standards across Bronze/Silver/Gold layers, ensuring data quality, integrity, and lifecycle management. * *Platform \& Architecture* — Own Databricks platform architecture and tooling decisions (Delta Lake, Unity Catalog, Databricks SQL, structured streaming), aligned with enterprise security and governance. * *Reliability \& Observability* — Define SLAs and implement observability (lineage, monitoring, metrics); resolve performance bottlenecks and manage cost. * *Leadership \& Collaboration* — Act as technical lead for the data engineering function; mentor engineers; lead design reviews and architecture discussions. * *Governance \& Auditability* — Ensure traceable, well\-governed data processes via Unity Catalog; maintain documentation and support compliance. * *Partner Management* — Oversee external delivery partners, ensuring quality, standards adherence, and knowledge transfer. **Experience \& Skills** * **5\+ years** in data engineering, with **3\+ years** hands\-on on Databricks * Strong SQL and PySpark/Spark SQL expertise * Proven experience with Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL * Solid grasp of medallion architecture, ETL/ELT design, and Spark performance tuning * CI/CD experience (Databricks Asset Bundles, Git/Repos) * Familiarity with structured streaming, Auto Loader, and batch/near\-real\-time processing * Databricks certification (Data Engineer Associate/Professional) desirable * Must be based within a commutable distance to Dublin * Must have the relevant work status to work in Ireland. **Education** Degree in Computer Science, Data Engineering, or related field