[Remote] Data Engineering Lead
Note: The job is a remote job and is open to candidates in USA. MediaRadar, Inc. is a company that provides marketing, sales, and analytics leaders with critical intelligence through its AI-enabled platform. The Data Engineering Lead will be responsible for designing scalable systems, integrating Machine Learning models into ETL pipelines, and leading a team towards agile practices while maintaining high performance and cost-efficiency.
Responsibilities
- Architect and implement complex, end-to-end data pipelines using Azure Databricks and PySpark
- Optimize Apache Spark jobs, tune Databricks units, and define cluster policies to minimize compute costs
- Develop a proactive monitoring and alerts framework to ensure 99.9% reliability and mitigate system issues before they impact end-users
- Architect and design high-performance schemas in PostgreSQL, managing indexing, partitioning, and optimizing complex analytical queries
- Lead a lean team toward cross-trained agility, moving away from 'siloed specialists'
- Anticipate future data needs and design High-Velocity Architecture that is highly scalable and manageable to handle sudden volume increases
- Collaborate with ML teams to integrate automated model orchestration into robust ETL pipelines
- Collaborate with the offshore team lead to facilitate seamless knowledge transfer and operational continuity across time zones
Skills
- Core: Python, PostgreSQL + pgvector
- Big Data: Azure Databricks, PySpark, Delta Lake
- DevOps: Docker, Git, Azure DevOps, CI/CD
- 10+ years of experience in Data or Software Engineering with deep codebase involvement
- 3+ years as a Technical Lead managing agile teams
- Proven ability to lead lean, high-impact teams while maintaining high individual output
- Experience with cross-training advocacy and scaling data processing through automation
- Workflow Orchestration: Experience with Apache Airflow
- Containerization: Familiarity with Azure Kubernetes Service (AKS)
Company Overview
Company H1B Sponsorship