Lead GCP Data Engineer

**About Us:** Renaissance Info Systems is a technology and digital recruitment agency, connecting contract and permanent professionals with clients across Asia\-Pacific. We aim to differentiate ourselves through our level of responsiveness, and our understanding that comes from being an IT recruitment agency from the IT Industry. Our recruiters balance sophisticated and simple inter\-personal techniques to assure a strong candidate network. **Know More:** http://www.reninfo.com.au **Role \- Lead Data Engineer** **Description \& Requirements:** * Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch * Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development * Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team * Perform pair programming with customer’s data engineers * Project manage sprint\-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery * Collaborate with cross\-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program **Minimum Qualifications** **Expertise Required** In\-depth knowledge and experience of GCP data and analytics technologies * Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP * Define solution architecture and detailed design, and perform hands\-on implementation for – data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on\-prem files, on\-prem databases, APIs, etc. real\-time data ingestion and event\-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery * Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL * BQ query performance optimisation * CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform * Orchestration setup and Cloud Composer DAG development for data pipeline workflows * Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) * Experience applying Generative AI technologies and integrations in enterprise environments. **Tools and languages experience Required** **Must have** * GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform * Must have: programming knowledge and willingness to be hands\-on \- Python, Java **Good to Have** * Experience with dbt or Dataform, Terraform in context of data pipelines * Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE * Understanding and experience with NoSQL databases such as HBASE, MongoDB * Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations **Certification** * Good to have GCP Data engineering **Regards,** **Reshu Seth** **Recruitment Consultant** **Renaissance InfoSystem** **M: \+61 478 487 026** **E: reshu@reninfo.com.au** **W: http://www.reninfo.com.au**