SR Data Engineer (NCS/Job/ 3239)

Sr Data Engineer

Location: Hyderabad, India

Reporting Manager: Head of Analytics

Work Pattern: Full Time, 5 days in the office

Minimum Experience as a data engineer: 5 years

The senior data engineer is a data professional who uses her/his expertise in data engineering and programming to build systems that collect, manage and convert raw data into usable information for data analysts, data scientists and business analysts. Construct datasets that are easy to analyse and support company requirements. Combine raw information from different sources to create consistent and machine-readable formats. Develop and test architectures that enable data extraction and transformation for predictive or prescriptive modelling.

Key Responsibilities

Lakehouse & Medallion Architecture

· Design and implement Medallion architecture (Bronze, Silver, Gold layers) for scalable data processing with focus on schema enforcement, audit trails, versioning, time travel and incremental processing strategies.

· Design and maintain optimized data models for Lakehouse and warehouse consumption

 

Spark Engineering & Performance Optimization

· Develop high-performance distributed data pipelines using Apache Spark

· Optimize Spark workloads through partitioning, caching, broadcast joins, and query tuning, execution plans analysis and memory optimization

· Implement efficient incremental data processing and change data capture strategies

· Monitor and troubleshoot Spark job failures, latency, and resource bottlenecks

 

CI/CD & Data Pipeline Automation

· Design and implement CI/CD pipelines for data engineering workflows. Automate build, test, and deployment of data pipelines across environments

· Enable infrastructure-as-code and reproducible deployments

 

Data Pipeline Engineering

· Build scalable batch and streaming pipelines, Implement incremental data loading and watermark-based processing

· Design resilient ETL/ELT workflows with monitoring and retry mechanisms

· Develop reusable data transformation frameworks

· Implement pipeline observability, logging, and alerting

 

 

Technical Skills (Revised)

Core Data Engineering

· 6+ years building distributed data pipelines

· Strong Python and SQL expertise

· Data modeling and data lifecycle management

 

Spark & Distributed Processing (Mandatory)

· Strong experience with Apache Spark (PySpark/Scala Spark)

· Spark performance tuning and optimization

· Partitioning strategies and file optimization

· Execution plan analysis and query optimization

· Distributed computing concepts

 

Lakehouse & Medallion Architecture

· Experience implementing Bronze, Silver, Gold data layers

· Delta Lake / Lakehouse architectures

· Incremental processing and time travel

· Data versioning and schema evolution

 

CI/CD & DevOps for Data

· Git-based version control workflows

· CI/CD tools (Azure DevOps / GitHub Actions / Jenkins or similar)

· Deployment automation and environment management

· Infrastructure-as-code concepts

 

Data Platform & Cloud

· Experience with Azure/AWS/GCP data platforms

· Data lakes and analytical warehouses

· Workflow orchestration tools (Airflow or similar)