Title: Data Engineer
KA, IN
Job Description
Must Have
Role: Analyst - Data Engineering
Description:
Responsibilities:
• Lead functional teams or projects, managing multiple data engineering initiatives simultaneously to ensure timely delivery.
• Design, develop, and maintain scalable ETL pipelines leveraging Databricks, AWS Glue, and S3 for processing and transforming large-scale datasets.
• Expertise in data modeling and architecture with strong proficiency in SQL, Databricks, Python, PySpark, and Azure Fabric.
• Query and analyze complex datasets using SQL and Python to support business intelligence and analytics initiatives.
• Develop and document data models, schemas, and standards to ensure data consistency, usability, and governance.
• Monitor, troubleshoot, and optimize data pipeline performance and reliability, minimizing downtime and maximizing efficiency.
• Implement robust data validation and testing techniques to ensure high data quality and integrity.
• Communicate potential risks, mitigations, and business impacts clearly and promptly to stakeholders.
• Define technical requirements and design solutions leveraging existing technologies and industry best practices.
• Solve complex data-related problems by analyzing multiple sources of information and applying innovative approaches.
• Ensure all data, processes, and pipelines adhere to security protocols and compliance standards.
Must Have
• 6+ years of experience in data engineering, with a focus on big data technologies and cloud platforms.
• Proficiency in PySpark, AWS Glue, S3, SQL, and Python.
• Demonstrated experience leading data engineering teams or projects.
• Strong analytical and problem-solving skills, with the ability to handle complex data challenges.
• Excellent communication and interpersonal skills, enabling collaboration with technical and non-technical stakeholders.
• Deep understanding of data modeling, schema design, and database management.
• Knowledge of data quality assurance techniques and compliance requirements.
• Bachelor’s degree in computer science, Information Systems, or a related field; advanced degree preferred.
• Pharmaceutical data knowledge is Plus
• Certifications in relevant technologies (e.g., AWS Certified Data Analytics, Databricks Certified Data Engineer) is REQUIRED.
Good to have
EQUAL OPPORTUNITY