Lead Data Engineer – Azure Databricks

Lead Data Engineer – Azure Databricks **Job Type:** Full-Time **Location:** Cary, NC – On-site / Hybrid **Experience:** 12–18 Years **Salary:** $140,000 – $145,000 per year + Benefits Job Summary We are seeking a highly experienced and hands-on **Lead Data Engineer** to join an enterprise-scale data and AI platform initiative. The successful candidate will be responsible for the architecture, development, optimization, and delivery of modern data engineering solutions using **Azure Databricks, PySpark, Scala, Python, Delta Lake, and Azure data services**. This is a hands-on technical leadership role. The ideal candidate must have recent experience writing, reviewing, and troubleshooting production code and should be comfortable working across both **data engineering and AI/LLM solutions**. Key Responsibilities * Design and implement scalable **Lakehouse / Medallion Architecture** using Bronze, Silver, and Gold layers. * Develop production-grade data pipelines using **Python, Scala, PySpark, Spark Structured Streaming, and Delta Lake**. * Build metadata-driven and parameterized ingestion frameworks for batch, database extracts, CDC, and streaming data. * Work with **Azure Databricks, ADLS Gen2, Azure Data Factory, Azure Event Hubs, and Kafka**. * Design Delta Lake tables, data contracts, partitioning strategies, schema evolution, and data retention processes. * Troubleshoot and optimize Spark workloads, clusters, pipelines, and overall platform performance. * Establish coding standards, automated testing practices, and conduct code/PR reviews. * Implement CI/CD pipelines using **Azure DevOps, Terraform, and Databricks Asset Bundles**. * Develop AI-assisted solutions for data ingestion, canonical mapping, data quality, and reconciliation. * Design and implement production-grade **LLM, RAG, and agentic AI workflows**. * Work with embeddings, vector search, hybrid retrieval, prompt engineering, and tool-calling architectures. * Develop semantic-layer and **text-to-SQL** capabilities for enterprise data access. * Implement data governance, lineage, access control, and PII protection using **Unity Catalog**. * Monitor data quality, schema drift, data anomalies, reconciliation issues, and pipeline health. * Mentor data engineers and provide technical guidance on architecture, development, and best practices. * Present and defend technical designs to engineering teams, architects, and business stakeholders. Required Qualifications * **12–18 years** of professional experience in Data Engineering, Data Platforms, or related fields. * Expert-level proficiency in **Python, Scala, and PySpark**. * Strong SQL and data modeling experience, including dimensional and normalized data models. * Extensive hands-on experience with **Azure Databricks and Delta Lake**. * Strong knowledge of **Databricks Jobs & Workflows, Unity Catalog, cluster management, and performance tuning**. * Experience with **Azure Data Lake Storage Gen2 (ADLS Gen2)**. * Strong experience with **Azure Data Factory** and metadata-driven pipeline frameworks. * Experience with **Azure Event Hubs, Kafka, or Spark Structured Streaming**. * Proven experience designing and delivering enterprise-scale **Lakehouse / Medallion architectures**. * At least **3 years of hands-on experience designing and deploying production LLM/AI systems**. * Strong understanding of **RAG, embeddings, vector/hybrid retrieval, prompt engineering, and agentic workflows**. * Experience with **LangChain, LlamaIndex, or LangGraph**. * Experience with at least one AI platform such as **Azure OpenAI, OpenAI, or Databricks Model Serving**. * Experience implementing AI evaluation frameworks, golden datasets, regression testing, and human-in-the-loop validation. * Strong understanding of Azure security, including **Entra ID, RBAC, Managed Identities, Key Vault, ACLs, and Private Endpoints**. * Experience with **Azure DevOps, Terraform, CI/CD, and automated testing**. * Excellent communication, documentation, problem-solving, and stakeholder management skills. Preferred Qualifications * Experience with **Knowledge Graphs and Ontologies**. * Knowledge of **RDF, SPARQL, Neo4j, or graph-based data modeling**. * Experience developing enterprise-scale **Text-to-SQL or semantic-layer solutions**. * Experience with ML-based anomaly detection for transactional or time-series data. * Previous experience in **Financial Services or Insurance**. * Experience with **LLMOps / MLOps**, model and prompt versioning, cost management, and observability. * Experience with **dbt, Great Expectations**, or similar data quality frameworks. * Databricks Data Engineer Professional certification. * Azure **DP-203, DP-700, or AZ-305** certification. What We Are Looking For We are looking for a senior engineer who combines **strong architecture skills with hands-on coding ability**. Candidates should be actively involved in production engineering and comfortable taking ownership from technical design through implementation, testing, deployment, and production support. Candidates who have not written or reviewed production code recently may not be suitable for this position. Application Requirements Please submit the following information along with your updated resume: * Full Name * Current Location * Contact Number * Email Address * Work Authorization – **US Citizen** * LinkedIn Profile * Availability to Start * Interview Availability for the Next 3 Days * Updated Resume **Equal Opportunity Employer:** We are committed to providing equal employment opportunities to all qualified applicants. Pay: $140,000.00 - $145,000.00 per year Work Location: In person