Data Lead

**Role:** Data Lead **Type:** Full time – Perm / Contract **Location:** Sydney, AU **Status:** Valid visa with full working rights **Must Have:** Pyspark, Azure Synapse. **Key Responsibilities** * **Framework Development:** Design and build metadata\-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources. * **Architecture \& Design:** Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake. * **Technical Leadership:** Lead technical design discussions, mentor junior/mid\-level engineers, and conduct rigorous code reviews. * **Pipeline Optimization:** Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform. * **CI/CD \& DevOps:** Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices. * **Governance \& Security:** Integrate ingestion pipelines with Unity Catalog for enterprise\-grade data governance and lineage tracking. **Required Qualifications** * **Experience:** 8\+ years in Data Engineering, with at least 3–4 years of hands\-on technical leadership experience leading engineering pods. * **Core Databricks \& Spark:** Advanced proficiency in PySpark and Databricks SQL. * **Ingestion Expertise:** Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data. * **Cloud Platforms:** Strong experience working with AWS, Azure, or GCP cloud\-native services (e.g., S3, ADLS). * **Engineering Standards:** Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context. * **Soft Skills:** Excellent stakeholder management and the ability to translate complex technical architectures into business value. **Preferred Qualifications** * Experience with Kafka, dbt, or Delta Live Tables (DLT). * Databricks Certified Data Engineer Associate/Professional credentials **Interested?** Please apply, thanks for your interest and time. To read about our privacy policy https://www.frconsultancy.com.au/privacy\-policy