Senior Data Infrastructure Engineer (Spatial Data Platform)

Our client is looking for a hands-on **Senior Data Infrastructure Engineer** to build, scale, and operate the petabyte-scale data lakehouse behind our spatial data products. Every field and lab capture—robotics, egocentric headsets, multi-camera video, depth, IMU—lands in this platform, gets versioned, and powers downstream 3D reconstructions (Gaussian Splats, SLAM) and ML training clusters. This is a true engineering role. We need someone who has **personally built and run a PB-scale platform in production** , not just used one. **Key Responsibilities** * **PB-Scale Lakehouse:** Design object storage layouts (S3), partitioning strategies, catalog architectures, and open table formats (Iceberg/Delta). * **Data Versioning & Ingest:** Build reproducible dataset lineage (LakeFS) and high-throughput ingest pipelines (validation, checksumming, deduplication, bronze $\rightarrow$ gold promotion). * **Distributed CPU/GPU Compute:** Operate large-scale processing (Spark, Ray, Flink) for frame extraction, sync/calibration, SLAM/3DGS batch jobs, and auto-labeling. * **ML Training Integration:** Keep GPU clusters fed using sharded training formats (WebDataset, Mosaic Streaming), caching, and optimized cross-region transfers. * **Curation & Querying:** Build indexing, search, and high-performance SQL query layers (Trino, DuckDB, Spark SQL) so researchers can slice data without copying it. * **Reliability & Infrastructure:** Maintain IaC (Terraform) on AWS, handle pipeline backfills/recovery, optimize storage/egress costs, and participate in on-call rotation. **Qualifications** * **Experience:** 6+ years in data infrastructure or distributed systems engineering. * **Scale:** Proven track record setting up and operating a live **petabyte-scale** data platform (able to speak concretely to throughput, object limits, and failure modes). * **Tech Stack:** * **Languages:** Python + at least one of Go, Rust, C++, Java, or Scala. * **Table Formats & Versioning:** Iceberg/Delta Lake, Catalogs (Glue/Nessie), LakeFS or content-addressed storage. * **Processing & Orchestration:** Spark, Ray, or Flink; Airflow, Argo, or Prefect. * **Cloud/IaC:** Strong AWS (S3, EKS/ECS, IAM) and Terraform experience. **Nice to Have** * Experience with unstructured/multimodal spatial data (video, LiDAR, point clouds, ROS bags/MCAP). * GPU-accelerated pipelines (NVIDIA DALI, NVDEC, RAPIDS) or ML data loaders (WebDataset, Petastorm). * Query engines (Trino, DuckDB, ClickHouse) and embedding/vector search.