Experienced Remote Data Engineer – Big Data Processing, Cloud Migration, and Distributed Systems Expertise
Introduction to PulsePoint and Our Mission Imagine being part of a leading healthcare ad technology company that leverages real-world data in real-time to optimize campaign performance and revolutionize health decision-making. At PulsePoint, we're committed to delivering unparalleled results to our clients by targeting healthcare professionals and patients with an unprecedented level of accuracy. As a part of Internet Brands, a KKR portfolio company and owner of WebMD Health Corp, we're poised for exponential growth and seeking talented individuals to join our team. Our Data Engineering team plays a vital role in driving this growth, and we're excited to invite a skilled Data Engineer to join our ranks. About Our Data Engineering Team Our Data Engineering team is responsible for designing, building, and maintaining reliable and scalable enterprise-level distributed transactional data processing systems. We process over 80 billion impressions a day, generating over 20 TB of data (200 TB uncompressed), which is used to drive our optimization engines, update budgets, and provide real-time statistics and reports. We're a team that thrives on adopting proven open-source technologies to make our data infrastructure more nimble, scalable, and robust. Some of the cutting-edge technologies we've recently implemented include Kafka, Spark Streaming, Presto, Airflow, and Kubernetes. Job Overview As a Data Engineer at PulsePoint, you will be responsible for designing, building, and maintaining large-scale data systems, optimizing data processing workflows, and ensuring data quality across our systems. You will collaborate with a small team of diverse technology backgrounds, providing mentorship and guidance to junior team members. Your expertise will be crucial in driving our business growth, supporting new initiatives, and enhancing our data infrastructure.
Key Responsibilities
- Design, build, and maintain reliable and scalable enterprise-level distributed transactional data processing systems for scaling existing business and supporting new business initiatives
- Optimize jobs to utilize Kafka, Hadoop, Presto, Spark, and Kubernetes resources in the most efficient way
- Monitor and provide transparency into data quality across systems (accuracy, consistency, completeness, etc.)
- Increase accessibility and effectiveness of data (work with analysts, data scientists, and developers to build/deploy tools and datasets that fit their use cases)
- Collaborate within a small team with diverse technology backgrounds
- Provide mentorship and guidance to junior team members
Team Responsibilities
- Ingest, validate, and process internal and third-party data
- Create, maintain, and monitor data flows in Python, Spark, Hive, SQL, and Presto for consistency, accuracy, and lag time
- Maintain and enhance framework for jobs (primarily aggregate jobs in Spark and Hive)
- Create different consumers for data in Kafka using Spark Streaming for near-time aggregation
- Tools evaluation
- Backups/Retention/High Availability/Capacity Planning
- Review/Approval - DDL for database, Hive Framework jobs, and Spark Streaming to ensure they meet our standards
Technologies We Use We're proud to work with a range of cutting-edge technologies, including:
- Python - primary repo language
- Airflow/Luigi - for job scheduling
- Docker - packaged container image with all dependencies
- Graphite - for monitoring data flows
- Hive - SQL data warehouse layer for data in HDFS
- Kafka - distributed commit log storage
- Kubernetes - Distributed cluster resource manager
- Presto/Trino - fast parallel data warehouse and data federation layer
- Spark Streaming - Near-time aggregation
- SQL Server - Reliable OLTP RDBMS
- GCP - BigQuery for performance, Looker for dashboards
Requirements
To be successful in this role, you'll need:
- 5+ years of data engineering experience
- Fluency in Python and SQL
- Experience in Scala/Java is a plus (Polyglot programmer preferred!)
- Hive/Presto experience
- Proficiency in Linux
- Strong understanding of RDBMS and query optimization
- Passion for engineering and computer science around data
- Ability to work during East Coast U.S. hours (9am-6pm EST preferred, but we can be flexible as long as you can work until 1 or 2pm EST)
- Knowledge and exposure to distributed production systems (e.g., Hadoop)
- Knowledge and exposure to Cloud migration (AWS/GCP/Azure) is a plus
Selection Process Our selection process typically involves the following steps:
- Recruiter Screen (30 minutes)
- Hiring Manager Interview (45 minutes)
Apply tot his job Apply To this Job