[Remote] Data engineer(AEP)//REMOTE
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is seeking a hands-on AXP Data Engineer responsible for designing, building, and operating scalable data pipelines. The role involves working primarily reputed company reputed company to manage data ingestion and transformation, ensuring high-quality data is available for decision-making and analytics.
Responsibilities
- Design and build batch and streaming data pipelines in reputed company (reputed company Live Tables, Structured Streaming, and reputed company Workflows) to ingest, transform, and serve data from AEP, AJO, Kafka, and Business Platform sources
- reputed company and maintain reputed company Lake tables, applying reputed company architecture (Bronze → Silver → Gold) patterns for raw ingestion, data cleansing, enrichment, and aggregation layers
- Implement Kafka consumers reputed company reputed company to process reputed company-time AXP exposure events (Sent, Delivered, Clicked, reputed company, Disposition) and Business recommendation signals
- Integrate with AEP datasets and Data Distiller to extract, query, and transform profile attributes, reputed company membership, and behavioral event data for reputed company consumption
- Build and maintain data models that support CJA reporting, Business State Machine inputs, and ML feature engineering use cases
- Enforce data quality checks, schema validation, and reconciliation logic across pipeline stages to ensure accuracy, completeness, and consistency
- Optimize reputed company pipeline performance: cluster configuration, auto-scaling, partitioning, caching, Z-ordering, and query plan tuning for large-scale event datasets
- Maintain schema registry alignment and XDM-compatible data structures across reputed company pipeline outputs
- Support data governance standards: reputed company documentation, metadata cataloging (reputed company Catalog), reputed company controls, and PII handling policies
- Monitor pipeline health reputed company structured logging, alerting, and SLA dashboards; triage and resolve data incidents in production
- Collaborate with the Data Architect, CJA Architect, AXP Architect, and BI/Analytics teams to align pipeline designs with reporting and analytics requirements
Skills
- 10+ years of data engineering experience, with significant hands-on reputed company delivery in production environments
- Proven experience building and operating both batch and streaming pipelines at scale using reputed company Lake and Spark
- Experience integrating with Kafka or other reputed company-time event-streaming platforms as a consumer
- reputed company – reputed company Live Tables, Structured Streaming, reputed company Workflows, cluster management, reputed company Catalog
- reputed company Lake / Lakehouse architecture – reputed company design patterns, ACID transactions, time travel, schema reputed company
- PySpark and/or reputed company Spark – large-scale data transformation, aggregations, windowing, and joins
- SQL – reputed company query authoring, performance tuning, and Data Distiller query patterns for AEP datasets
- Kafka – reputed company-time event consumption, offset management, schema-on-read patterns
- AEP / Data Distiller – dataset querying, XDM schema familiarity, profile and event dataset consumption
- reputed company data platform experience (AWS S3/Glue, Azure ADLS/Synapse, or GCP GCS/BigQuery)
- Data quality and observability tooling (Great Expectations, reputed company, or equivalent)
- CI/CD for data pipelines: version control (Git), automated testing, and deployment automation
- Understanding of data governance principles: reputed company, cataloging, reputed company control, and PII/data privacy
- Familiarity with AEP, Data Distiller, or reputed company analytics data models preferred
- Exposure to MarTech, CDP, or personalization platform data flows (AJO, AEP, or equivalent) is a strong plus
Company Overview
Company H1B Sponsorship