Data Engineer (NCS/Job/ 2745)

Key Responsibilities

Data Pipeline Development:

Design, implement, and optimize ETL/ELT pipelines for structured and unstructured data.

Integrate data from multiple internal and external sources into centralized systems.

Build scalable batch and real-time data processing workflows.

Data Infrastructure & Architecture:

Develop and maintain data lake, data warehouse, or data mesh architectures.

Ensure high availability, performance, and scalability of data systems.

Implement data modeling best practices to support analytics and ML use cases.

Software Engineering:

Write clean, efficient, and maintainable code in Python, Java, or Scala.

Implement unit tests, integration tests, and CI/CD pipelines for data workflows.

Apply software engineering principles to build reusable data services and APIs.

Data Quality & Governance:

Implement validation, monitoring, and observability to ensure data accuracy and reliability.

Enforce data security, compliance, and privacy standards (e.g., GDPR).

Collaborate with stakeholders to establish data definitions, lineage, and documentation.

 

Requirements :

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

2–4 years of experience in data engineering, software engineering, or backend systems.

Strong programming skills in Python, Java, or Scala.

Experience with distributed data processing frameworks (Apache Spark, Flink, Beam, or similar).

Proficiency with SQL and relational databases; familiarity with NoSQL systems (Cassandra, MongoDB, DynamoDB).

Hands-on experience with cloud platforms (AWS, GCP, Azure) and data services (e.g., Redshift, BigQuery, Snowflake, Databricks).

Familiarity with workflow orchestration tools (Airflow, Luigi, Dagster) and CI/CD pipelines.