AWS Data Engineer

Sr Data Engineer:

 

Roles Responsibilities

1. Requirement Analysis Solution Design

 Collaborate with business and data stakeholders to understand data requirements, transformation rules, and use cases

 Translate functional requirements into scalable data engineering and ETL design solutions

Define approach for data ingestion, transformation, orchestration, and access control

2. Data Engineering Development testing

Design and develop robust ETL pipelines using AWS Glue PySpark

Implement complex data transformations including joins, aggregations, conditional logic, and business rule processing

Build reusable, metadata-driven ETL frameworks where applicable

 Optimize data pipelines for performance, scalability, and reliability

 Perform unit testing and end-to-end validation of data pipelines

 Implement data quality checks including integrity, completeness, and reconciliation

 Validate transformation logic and ensure accuracy of aggregated and processed data

3. Orchestration Integration

Develop and manage workflow orchestration using AWS Step Functions and EventBridge

Implement event-driven and schedule-based pipeline execution strategies

Ensure seamless integration between data pipelines, control frameworks, and downstream systems

4. Infrastructure Deployment IaC

Provision and manage cloud infrastructure using Terraform Infrastructure as Code

Deploy and configure AWS services including Glue, Lambda, DynamoDB, and orchestration components

Ensure consistent, repeatable, and scalable deployments aligned with DevOps practices

5. Security Governance

 Implement secure data access controls using IAM and Lake Formation

 Ensure compliance with data governance policies and role-based access requirements

 Manage data security, encryption, and access auditing

7. Monitoring, Support Continuous Improvement

Set up monitoring, logging, and alerting mechanisms for data pipelines e.g., SNS, audit logs

Troubleshoot data and pipeline issues, ensuring timely resolution

Continuously enhance pipeline performance, reliability, and maintainability

Collaborate in an agile setup to drive continuous delivery and improvements

 

Must-Have Skills

     7-10 years of AWS Data Engineering experience.

     Strong experience in AWS Glue PySpark and data processing on S3

     Proven expertise in ETL development and complex data transformations

     Hands-on experience with AWS Step Functions, EventBridge, and orchestration patterns

     Proficiency in Terraform Infrastructure as Code

     Strong SQL and data modeling skills

     Experience with data validation, reconciliation, and quality frameworks

     Solid understanding of IAM, security, and cloud best practices

Nice-to-Have Skills

     Familiarity with Lake Formation and data governance frameworks

     Experience with DynamoDB or metadata-driven ETL architectures

     Exposure to event-driven architectures

     Knowledge of CICD tools

Salary: 12000 - 15000 MYR Per month