Lead Data Engineer
Job ID: Lea-ETP-Pun-869
Location: Pune
Position: Lead Data Engineer
Experience: 6-9 years
Location: Remote
Employment Type: Full-Time
Overview: We are looking for an experienced Lead Data Engineer to join our team and take ownership of end-to-end data engineering projects. The ideal candidate will have strong technical expertise in Spark, Databricks, SQL, DBT, and Python, and will be capable of leading a team of data engineers in building scalable and efficient data systems. You will drive architectural decisions, mentor junior team members, and ensure the successful delivery of data solutions that meet business needs.
Responsibilities:
- Lead the design and implementation of large-scale data pipelines and architectures using Spark, Databricks, and DBT.
- Drive the technical direction of data projects, ensuring best practices are followed for code quality, performance, and scalability.
- Mentor and guide junior and mid-level data engineers, providing technical leadership and ensuring high-quality code and design.
- Collaborate with data scientists, analysts, and business stakeholders to understand requirements and translate them into technical specifications.
- Develop and maintain complex SQL queries and data models to support various analytical use cases.
- Write Python scripts to automate data workflows and enhance pipeline efficiency.
- Ensure data governance, security, and compliance within data engineering practices.
- Troubleshoot and resolve performance bottlenecks and data quality issues.
- Contribute to strategic initiatives, such as data platform optimization and cloud migration efforts.
Minimum Qualifications:
- 6-8 years of experience in data engineering or related roles, with a proven track record of leading data engineering projects.
- Expert-level proficiency in Spark and Databricks for building scalable data pipelines.
- Advanced SQL skills for data manipulation and performance tuning.
- Strong experience with DBT for data modeling and transformation.
- Proficiency in Python for data engineering tasks and automation.
- Demonstrated experience with large-scale data systems and architecture design.
- Strong leadership skills with experience mentoring and managing data engineering teams.
- Excellent communication skills, with the ability to interact with both technical and non-technical stakeholders.
Good to Have:
- Experience with AWS cloud services, such as S3, Lambda, IAM, and Kinesis.