Data Engineer(NCS/Job/ 1277)

1. Experience Level - 3 to 4 years
2. Budget - Upto 1L per month
3. Tenure of the engagement - 6 to 12 months
4. Work Location - Bangalore
5. Project start date (Tentative) - Immediate

Data Engineer

Position Overview:

We are seeking a skilled Python Data Engineer (some overlap with Developer skills) to join our dynamic team. In this role, you will be responsible for developing and maintaining preprocessing scripts for streaming data from video OTT platform. You will work closely with our data engineering team to ensure seamless data integration and processing. You will also be responsible for optimizing SQL querying for optimized performance.

Key Responsibilities:

- Develop and maintain preprocessing scripts to handle streaming data from a video OTT platform.

- Retrieve data from data lakes or data warehouses, process it according to business needs, and load it back into the data warehouse.

- Utilize GitHub or similar repositories to manage code versions and collaborate with team members.

- Access servers directly to test and debug scripts, ensuring optimal performance.

- Analyse data within data warehouses using SQL to identify anomalies and prepare data for processing.

- Optimize SQL queries for faster execution with optimized memory consumption.

- Implement and manage Redis queues or sets to control the flow of data processing tasks.

Required Knowledge and Skills:

- Proficiency in Python, with a strong understanding of object-oriented programming and familiar with packages such as Pandas, NumPy, OS, SYS, datetime, JSON, and traceback.

- Expertise in exception handling, logging, file handling, and multiprocessing/multithreading.

- Strong knowledge of SQL for data analysis and processing.

- Strong knowledge of Spark with hands-on PySpark experience.

- Experience with Redis, Git and Linux commands.