Senior Data Engineer

Role Description \& Requirements: ● Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch ● Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development ● Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team ● Perform pair programming with customer’s data engineers ● Project manage sprint\-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery ● Collaborate with cross\-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program Minimum Qualifications ● Expertise Required ○ In\-depth knowledge and experience of GCP data and analytics technologies ○ Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP ○ Define solution architecture and detailed design, and perform hands\-on implementation for ■ data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on\-prem files, on\-prem databases, APIs, etc. ■ real\-time data ingestion and event\-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery ○ Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL ○ BQ query performance optimisation ○ CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform ○ Orchestration setup and Cloud Composer DAG development for data pipeline workflows ○ Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) ○ Experience applying Generative AI technologies and integrations in enterprise environments. Tools and languages experience Required Must have ● GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform ● Must have: programming knowledge and willingness to be hands\-on \- Python, Java