Job Title: Lead Data Engineer
Experience Required: 6 to 10 Years
Positions Available: 2
Location: Indore
Slug: lead-data-engineer-indore
Roles and Responsibilities:
Data Architecture & Design:
Lead the design, development, and optimization of scalable data pipelines and architectures, ensuring high-performance data solutions tailored to business needs.
Data Integration & ETL Processes:
Oversee the integration of data from multiple sources into centralized data warehouses, ensuring the accuracy, consistency, and timeliness of data via automated ETL (Extract, Transform, Load) processes.
Big Data Technologies:
Implement and manage big data technologies, including Hadoop, Spark, and Kafka, to handle complex, high-volume data sets efficiently.
Data Modeling & Warehousing:
Lead the design and implementation of data models that support reporting, analytics, and business intelligence tools, working closely with data analysts and business teams.
Collaboration with Cross-Functional Teams:
Work alongside data scientists, software engineers, and business stakeholders to understand requirements and deliver solutions that meet organizational goals.
Data Quality & Governance:
Ensure that data quality and data governance standards are followed, implementing automated testing and validation procedures to ensure data integrity.
Performance Optimization:
Continuously monitor and optimize the performance of data pipelines and databases to ensure low-latency processing and minimal downtime.
Cloud Infrastructure Management:
Manage cloud-based data infrastructures on platforms like AWS, Azure, or GCP to ensure scalable, cost-effective data storage and processing.
Mentorship & Team Leadership:
Mentor and guide junior team members, providing technical leadership, best practices, and career development opportunities.
Documentation & Reporting:
Maintain comprehensive documentation of data architecture, pipeline designs, and processes, and provide regular updates to management regarding project status and KPIs.
Troubleshooting & Issue Resolution:
Lead the resolution of complex data-related issues, providing timely fixes for data quality and processing issues to ensure uninterrupted data flow.
Stay Updated with Industry Trends:
Keep up to date with the latest trends and best practices in data engineering, and propose innovative solutions to improve the efficiency and scalability of data systems.
Skills and Qualifications:
Strong proficiency in SQL, Python, Java, or Scala.
Experience with cloud platforms (AWS, Azure, GCP).
Expertise in big data technologies (Hadoop, Spark, Kafka).
Hands-on experience with data modeling, ETL processes, and data warehousing.
Solid understanding of data governance, security, and compliance.
Strong communication and collaboration skills to work across cross-functional teams.
Ability to manage multiple projects simultaneously while delivering high-quality results.
