Pyspark Developer(RARR Job 5231)

We are looking for experienced Pyspark Developers with strong expertise in big data technologies to join our growing team. The ideal candidate will have hands-on experience in Pyspark, Spark, SQL, Hive, and Unix Shell Scripting, and be capable of developing scalable and robust data pipelines.

Key Responsibilities:

• Develop and maintain robust data processing pipelines using Pyspark and Apache Spark

• Write complex SQL queries for data extraction, transformation, and analysis

• Design and execute Unix Shell Scripts for automation and batch processing

• Work with Hive for data warehousing and querying large datasets

• Collaborate with data engineers, analysts, and business stakeholders to deliver end-to-end data solutions

• Optimize existing jobs for performance and scalability

• Troubleshoot and resolve data pipeline issues efficiently

• Ensure timely delivery and adherence to coding standards

Required Skills:

• Strong hands-on experience in Pyspark or a combination of Python + Apache Spark

• Proficient in Advanced SQL – writing complex joins, aggregations, and optimization

• Experience in Hive for querying and managing big data

• Solid knowledge of Unix Shell Scripting for job scheduling and automation

• Good understanding of data engineering principles and ETL processes

• Excellent debugging and troubleshooting skills

• Effective communication and team collaboration skills