Title: AWS Cloud Data Engineer
Bangalore, KA, IN
Job Description
Must Have
AWS Cloud Data Engineer
Job Description
Role Overview, Responsibilities, Skills, and Qualifications
Position Summary
An AWS Cloud Data Engineer is a specialist who designs, develops, and maintains scalable data architectures and solutions using Amazon Web Services (AWS). This role sits at the intersection of data engineering and cloud computing, requiring deep expertise in both traditional data pipelines and the AWS ecosystem. The ideal candidate will be passionate about building robust data systems, automating workflows, and leveraging cloud technologies to solve complex business problems.
Key Responsibilities
• Data Pipeline Development: Design, build, and manage scalable, reliable, and secure data pipelines on AWS, enabling seamless data ingestion, processing, and storage across various data sources.
• Cloud Architecture Design: Architect and implement data solutions using AWS services such as S3, Redshift, Glue, Lambda, EC2, EMR, Kinesis, DynamoDB, RDS, Aurora, and others.
• ETL (Extract, Transform, Load): Develop, optimize, and monitor ETL processes to transform raw data into meaningful insights. Leverage services like AWS Glue, Data Pipeline, and third-party ETL tools.
• Data Modeling: Design efficient and scalable data models for both structured and unstructured data. Implement best practices for data modeling on cloud-native platforms.
• Automation and Orchestration: Automate data workflows using AWS Step Functions, Lambda, CloudWatch Events, and other orchestration tools.
• Data Quality & Governance: Ensure data quality, integrity, and security. Implement and enforce data governance and compliance standards (e.g., GDPR, HIPAA).
• Performance Optimization: Monitor and tune data solutions for optimal cost, speed, and reliability.
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to gather requirements, understand data needs, and deliver solutions.
• Troubleshooting & Support: Provide ongoing support for data infrastructure, identifying and resolving issues proactively.
• Documentation: Maintain comprehensive documentation of data pipelines, architecture decisions, workflows, and operational procedures.
• Research & Innovation: Stay up-to-date with the latest trends in AWS cloud and data engineering, recommending new tools and approaches to enhance the organization’s data capabilities.
Good to have
Required Skills and Qualifications
• Educational Background: Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, Mathematics, or a related field.
• Cloud Expertise: Deep hands-on experience with AWS services, particularly those related to data storage, processing, and analytics (S3, Redshift, Glue, EMR, Lambda, DynamoDB, etc.).
• Programming Skills: Proficiency in at least one programming language commonly used in data engineering, such as Python, Java, or Scala.
• Database Technologies: Experience with relational databases (PostgreSQL, MySQL, Microsoft SQL Server, Aurora) and NoSQL databases (DynamoDB, MongoDB).
• ETL Tools: Familiarity with ETL frameworks and orchestration tools such as AWS Glue, Apache Airflow, and others.
• Data Warehousing: Expertise in designing, implementing, and maintaining data warehouses, with a strong understanding of Redshift, Snowflake, or similar platforms.
• Deep understanding of VPC, subnets, routing tables, NAT gateways and VPNs
• Expertise in AWS Athena, Glue and EMR for big data processing
• Experience with QuickSight for data visualization
• Integration of Generative AI models, Snowflake, Foundry and 3rd party AWS
• Proficiency in SageMaker for building and deploying machine learning models
• Big Data Technologies: Experience working with large-scale data processing frameworks such as Apache Spark, Hadoop, or Flink.
• DevOps & CI/CD: Understanding of DevOps practices and tools for deploying and managing data pipelines (e.g., CloudFormation, Terraform, CodePipeline, Jenkins).
• Security & Compliance: Knowledge of AWS security best practices, IAM policies, encryption techniques, and compliance frameworks.
Preferred Qualifications and Certifications
• AWS Certified Data Analytics – Specialty
• AWS Certified Solutions Architect – Associate or Professional
• Experience with containerization technologies (Docker, Kubernetes, AWS ECS/EKS)
• Background in machine learning workflows and integration with data pipelines
• Experience with data visualization tools (Tableau, QuickSight, Power BI)
• Knowledge of data streaming solutions (Kinesis, Kafka, AWS MSK)
• Experience working in an Agile/Scrum environmen
• Communication: Excellent verbal and written communication skills; ability to convey complex technical concepts to both technical and non-technical audiences.
• Problem-Solving: Strong analytical and troubleshooting skills, with a knack for identifying and resolving data pipeline bottlenecks and inefficiencies.
• Adaptability: Comfort adapting to new technologies and navigating rapid changes in cloud and data engineering landscapes.
EQUAL OPPORTUNITY