Senior Data Engineer (Azure Databricks / PySpark) (RARR Job 5620)

 We are looking for a hands-on Data Engineer to design, develop, and optimize data pipelines on Azure Databricks, converting Informatica and Kalido logic into scalable PySpark notebooks.
Key Responsibilities:

• Convert source-to-target mappings into PySpark Databricks notebooks.
• Build end-to-end data pipelines using ADF + Databricks Jobs with proper logging and monitoring.
• Optimize code for performance and cost-efficiency (cluster sizing, caching).
• Implement unit tests and data validation to ensure like-to-like logic migration.
• Collaborate with DevOps to enable CI/CD pipelines and version control.
• Support SIT/UAT defect resolution and production cutover readiness.

Required Skills:

• 5+ years data engineering experience (2+ years on Azure Databricks).
• Expert in PySpark, Delta Lake, and SQL performance tuning.
• Strong knowledge of ADF, ADLS Gen2, and data orchestration patterns.
• Experience with Git, code reviews, and Agile development.
• Familiarity with ETL conversion projects (Informatica to cloud preferred).