Data Engineer - People Force Consulting Inc

DUE TO THE NATURE OF PROJECT CITIZENS ARE ENCOURAGED TO APPLY

Data Engineer

Long Term! 6 12 months

Remote / EST hours

Proficient in AWS, Databricks, and Azure DevOps, with a focus on strong analytical skills in PySpark, Delta Live Tables, Change Data Capture (CDC), and on-premises to AWS data migration.

Technical Skills

AWS (Amazon Web Services)

1. Core Services: Proficiency with core AWS services like EC2, S3, RDS, Lambda, and VPC.
2. Data Services: Experience with AWS data services Glue and EMR.

1. AWS DMS: Knowledge of AWS Database Migration Service (DMS) for migrating databases to AWS.

1. CDC: Understanding of Change Data Capture (CDC) techniques to capture and replicate changes from source databases to target databases.

1. Security: Understanding of AWS security best practices, IAM, and encryption.

Databricks

1. PySpark & Spark SQL: Strong analytical skills in PySpark & Spark SQL for big data processing and analysis.

1. Delta Live Tables: Expertise in using Delta Live Tables for building reliable and scalable data pipelines.

1. Notebooks: Strong utilization of Databricks Notebooks for data analysis.
2. Workflows : Setting up and monitoring Databricks Workflows.

1. Data Integration: Experience integrating Databricks with AWS services.

DevOps Principles

1. CI/CD Pipelines: CI/CD pipelines using Azure Pipelines.

1. Version Control: Proficiency with Azure Repos and Git for version control.

1. Automation: Scripting and automation using PowerShell, Bash, or Python. Automating the build, test, and deployment processes

Infrastructure as Code (IaC)

1. Terraform: Experience with Terraform for managing AWS and Azure infrastructure.

On Prem integration with AWS

1. Integrating on prem data with AWS and Databricks.
2. Thoroughly test and validate the data to ensure it has been transferred correctly and is fully functional.

Optimization and Monitoring

1. Optimize AWS services and Databricks for performance and cost-efficiency.

1. Proficiency in setting up monitoring and logging using tools like AWS CloudWatch to track the performance and health of the complete data flow.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.