[Remote] Project Delivery Specialist - Lead Databricks Engineer - Data Lakes - Remote
Note: The job is a remote job and is open to candidates in USA. Deloitte is a leading consulting firm seeking a Lead Databricks Engineer specializing in Data Lakes. The role involves leading the end-to-end delivery of data lake workloads on Databricks, implementing pipelines, and ensuring data quality while mentoring other engineers.
Responsibilities
- Lead end-to-end delivery of data lake/lakehouse workloads on Databricks: ingestion, processing, curation, and consumption layers
- Architect and implement pipelines using Spark (PySpark/Scala), Delta Lake, and orchestration tooling (e.g., Databricks Workflows, ADF/Airflow-as applicable)
- Define data modeling patterns (bronze/silver/gold, dimensional/denormalized, CDC handling) and optimize for analytics use cases
- Establish engineering standards: repo structure, coding conventions, branching strategy, documentation, and reusable frameworks
- Implement data quality and validation controls (reconciliation, anomaly detection, schema enforcement, expectations)
- Drive performance tuning: partitioning, file sizing, Z-ORDER, caching, cluster sizing, job parallelism, and query optimization
- Partner with governance/security to implement access controls (e.g., Unity Catalog, RBAC/ABAC), PII handling, encryption, and audit logging. Build/maintain CI/CD (continuous integration/continuous delivery) for notebooks/jobs (Databricks Repos, Git integration; IaC where applicable)
- Monitor and improve operational reliability: observability dashboards, alerting, runbooks, incident response support
- Manage dependencies across teams (source system owners, platform/infrastructure, BI, data science) and provide delivery estimates and plans
- Mentor engineers, conduct code reviews, and lead technical design sessions
- Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and / or technical teams, including escalating any matters that require additional attention and consideration from engagement management
- Independently and collaboratively lead client engagement workstreams focused on improvement, optimization, and transformation of processes including implementing leading practice workflows, addressing deficits in quality, and driving operational outcomes Skills
- 7+ years experience with Apache Spark, Databricks SQL, Python, ETL, Lakeflow, Cloud Orchestration
- 7+ years in data engineering with strong hands-on Databricks experience in production environments
- Advanced skills in Spark and PySpark/Scala, with strong SQL capabilities
- Deep experience with Delta Lake and lakehouse/data lake design patterns
- Experience building batch and/or streaming pipelines; familiarity with CDC concepts
- Proven technical leadership (leading squads, setting standards, design ownership)
- Experience with cloud storage and security fundamentals (object storage, networking basics, IAM)
- Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
- Limited immigration sponsorship may be available
- Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve Company Overview
- Deloitte is a business consulting company that offers audit, consulting, financial advisory, and tax services. It was founded in 1845, and is headquartered in London, England, GBR, with a workforce of 10001+ employees. Its website is Company H1B Sponsorship
- Deloitte has a track record of offering H1B sponsorships, with 6871 in 2025, 4911 in 2024, 5604 in 2023, 8090 in 2022, 5993 in 2021, 10388 in 2020. Please note that this does not guarantee sponsorship for this specific role. Apply tot his job
Apply tot his job Apply To this Job