Lead Data Engineer – Transfer Solutions
Job ID: R-233628
Location: Pune, Maharashtra, India – 411006
Experience Required: 10+ Years
Education: Bachelor’s / Master’s Degree in Computer Science or related field
Service Line: Software Engineering
Role Overview
Mastercard is looking for a highly experienced Lead Data Engineer to design and manage large-scale data migration pipelines, transforming OLTP systems into modern big data platforms like Cloudera and Databricks. This is a key role in the Transfer Solutions division responsible for enabling seamless, secure, and high-performance data flows supporting Mastercard Move, a next-gen payment initiative.
Key Responsibilities
Design and implement scalable data migration pipelines from Oracle-based OLTP systems to platforms such as Cloudera CDP/CDH and Databricks.
Build and optimize ETL workflows using Python, Scala, Spark, and Apache NiFi.
Develop Change Data Capture (CDC) pipelines with exception handling, reconciliation, and data integrity checks.
Enforce strong data security policies including encryption, access controls, and retention.
Create and maintain data models, taxonomy, and cataloging systems.
Establish data lineage, provenance, and governance compliance mechanisms.
Collaborate with cross-functional teams to gather migration requirements and ensure solution quality.
Monitor, troubleshoot, and continuously enhance pipeline performance and reliability.
Stay updated with latest data engineering tools, trends, and best practices.
Required Skills
10+ years of professional experience in data engineering.
Minimum 2 years in a technical leadership or lead engineer role.
Hands-on with OLTP systems (Oracle), data egress strategies, and data modeling.
Proficient in programming with Python, Scala, and Java.
Expertise in Apache Spark, Flink, Kafka, and Apache NiFi.
Experience with Cloudera Data Platform (CDP/CDH), Apache Ozone, AWS, Databricks, S3, and Glue.
Familiarity with Medallion architecture, data lakes, data warehouses, and large-scale ETL systems.
Strong understanding of data security practices including masking, encryption, and role-based access.
Excellent analytical, communication, and collaboration skills.
Preferred Skills
Awareness of regulatory compliance in data privacy and handling (e.g., GDPR, PCI-DSS).
Prior experience in financial or payment domain is a plus.
Strong leadership qualities and ability to navigate technical ambiguity.
