Data Engineer Spark & MLOps (NCS/Job/ 3828)

Key Responsibilities

• Design, develop, and optimize large-scale data pipelines using Apache Spark (PySpark/Scala Spark).
• Build and maintain ETL/ELT workflows for structured and unstructured data.
• Collaborate with Data Scientists and ML Engineers to operationalize machine learning models.
• Implement MLOps best practices, including model deployment, monitoring, versioning, and retraining pipelines.
• Develop and maintain CI/CD pipelines for data and ML workloads.
• Ensure data quality, governance, security, and compliance across data platforms.
• Monitor and optimize Spark jobs for performance, scalability, and cost efficiency.
• Work with cloud-native services and containerized environments.

Required Skills

• Strong experience with Apache Spark (PySpark preferred).
• Proficiency in Python and SQL.
• Experience with data orchestration tools such as Airflow or similar.
• Hands-on experience with MLOps tools such as MLflow, Kubeflow, or Azure ML.
• Experience with cloud platforms (AWS, Azure, or GCP).
• Knowledge of Docker, Kubernetes, and containerized deployments.
• Familiarity with CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
• Understanding of data warehousing, data lakes, and distributed computing concepts.

Preferred Skills

• Experience with Databricks.
• Knowledge of Kafka or other streaming technologies.
• Exposure to GenAI/LLM operationalization and model monitoring.
• Experience with Delta Lake, Snowflake, or similar modern data platforms.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.
• 4+ years of experience in Data Engineering, Big Data, or MLOps.
• Strong problem-solving and communication skil