Title: Senior Data Engineer
KA, IN
Job Description
Must Have
Role: Senior Data Engineer
Description: The Senior Data Engineer plays a pivotal role within our Data Engineering team, leading the design, development, and maintenance of robust data pipelines and infrastructure. This position involves managing multiple projects, collaborating with cross-functional teams, and delivering high-quality data solutions that align with business objectives. The ideal candidate brings extensive experience in data engineering, with expertise in technologies such as PySpark, AWS Glue, and S3, and a proven ability to lead complex data initiatives.
To excel in this role, you must be a proactive problem-solver with excellent communication skills, capable of translating complex technical concepts into actionable insights for business stakeholders. You should be adept at leading teams, managing project timelines, and ensuring that all deliverables meet the highest standards of quality and compliance.
Responsibilities:
• Lead functional teams or projects, managing multiple data engineering initiatives simultaneously to ensure timely delivery.
• Design, implement, and maintain scalable ETL pipelines using Databricks, PySpark, AWS Glue, and S3 to process and transform large datasets.
• Query and analyze complex datasets using SQL and Python to support business intelligence and analytics initiatives.
• Develop and document data models, schemas, and standards to ensure data consistency, usability, and governance.
• Monitor, troubleshoot, and optimize data pipeline performance and reliability, minimizing downtime and maximizing efficiency.
• Implement robust data validation and testing techniques to ensure high data quality and integrity.
• Communicate potential risks, mitigations, and business impacts clearly and promptly to stakeholders.
• Define technical requirements and design solutions leveraging existing technologies and industry best practices.
• Solve complex data-related problems by analyzing multiple sources of information and applying innovative approaches.
• Ensure all data, processes, and pipelines adhere to security protocols and compliance standards.
Desired Profile
• 4+ years of experience in data engineering, with a focus on big data technologies and cloud platforms.
• Proficiency in PySpark, AWS Glue, S3, SQL, and Python.
• Demonstrated experience leading data engineering teams or projects.
• Strong analytical and problem-solving skills, with the ability to handle complex data challenges.
• Excellent communication and interpersonal skills, enabling collaboration with technical and non-technical stakeholders.
• Deep understanding of data modeling, schema design, and database management.
• Knowledge of data quality assurance techniques and compliance requirements.
• Bachelor’s degree in computer science, Information Systems, or a related field; advanced degree preferred.
• Pharmaceutical data knowledge is Plus
• Certifications in relevant technologies (e.g., AWS Certified Data Analytics, Databricks Certified Data Engineer) is REQUIRED.
Good to have
EQUAL OPPORTUNITY