Principal Data Engineer – Databricks

**Principal Data Engineer – Databricks** We're looking for an experienced Principal Data Engineer to lead the Databricks side of a major technology and data transformation program. It's an initial 12\-month contract. You'll set the technical direction for the data platform, own the engineering standards and still stay hands\-on, while building up the capability of an internal data engineering team. **WHAT'S ON OFFER** * Initial 12\-month contract * $1,200 per day inclusive of super * Perth\-based opportunity * Principal\-level role in a large enterprise data environment * Own the Databricks platform architecture, engineering standards and delivery approach * Lead, mentor and grow a team of data engineers **THE ROLE:** You'll report into senior technology leadership and work closely with enterprise architects, technical leads and business stakeholders. You'll be the senior technical authority on the Databricks Lakehouse platform, setting the design and engineering direction and staying hands\-on in the most complex delivery work. You'll also build up the engineering practice by setting the standards, patterns and ways of working the team uses. **Key Responsibilities** * Own the end\-to\-end architecture and technical direction of the enterprise Databricks Lakehouse platform * Define and put in place engineering standards, reusable frameworks and design patterns, including Medallion architecture, Delta Lake and Delta Live Tables * Lead solution design for data migration, replication, transformation and integration across complex source systems * Design and oversee ingestion from SQL databases, APIs and streaming platforms such as Kafka, including batch and real\-time patterns * Stay hands\-on with complex pipelines and performance tuning, including Spark optimisation, handling complex data structures such as JSON, and moving data into efficient formats * Lead governance and security for the platform using Unity Catalog, covering access controls, lineage, metadata and catalogue management * Set up CI/CD, infrastructure\-as\-code and DevOps practices so the platform can be delivered reliably and repeatably * Drive platform cost management, observability and operational excellence * Act as technical lead and mentor to the data engineering team, uplifting capability through code reviews, pairing, design forums and knowledge sharing * Advise senior stakeholders on data engineering strategy, platform roadmap and technology decisions, and explain complex concepts in business terms **THE COMPANY:** An established Australian organisation running a major data and technology transformation program and investing heavily in modern data platforms and engineering capability. They want a Principal Data Engineer who can lead the technical direction of their Databricks platform and help build a high\-performing internal team. **SKILLS \& EXPERIENCE:** * Significant experience in senior or principal data engineering roles, with deep hands\-on expertise in Databricks and Apache Spark * Proven track record of designing and delivering enterprise\-scale Lakehouse platforms, ideally on Azure or AWS * Expert SQL and Python (PySpark), including building complex transformation and ingestion frameworks * Strong experience with Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows * Experience with orchestration and transformation tools such as Airflow and dbt * Proven experience integrating data from APIs, relational databases and streaming platforms such as Kafka * Strong knowledge of data migration, replication and modernisation strategies, including moves from legacy platforms * Experience setting up CI/CD, infrastructure\-as\-code (e.g. Terraform) and DevOps practices in data environments * Strong understanding of data governance, security, metadata management and data quality frameworks * Demonstrated technical leadership, including setting engineering standards and mentoring and developing data engineers * Excellent communication and stakeholder engagement skills, with confidence advising at senior levels * Databricks certifications highly regarded Please note: You must have Full Working Rights in Australia and be based in Perth. To apply, click the link and upload your current resume in Microsoft Word format (.doc or .docx), or send your CV to dash.mann@fullcirclepartners.com.au for a confidential discussion. Please note: Only suitable candidates will be contacted about this vacancy.