Data Engineer (NCS/Job/ 2115)

Data Engineer:

Required qualifications to be successful in this role

Must have Skills:

• Development experience with Spark (PySpark), Python and SQL

• Extensive knowledge building data pipelines

• Hands on experience with Databricks Devlopment

• Strong experience with

• Strong experience developing on Linux OS

• Experience with scheduling and orchestration (e.g. Databricks Workflows, airflow, prefect, control-m)

Good to have skills:

• Solid understanding of distributed systems, data structures, design principles

• Agile Development Methodologies (e.g. SAFe, Kanban, Scrum)

• Comfortable communicating with teams via showcases/demos

• Play key role in establishing and implementing migration patterns for the Data Lake Modernization project

• Actively migrate use cases from our on premises Data Lake to Databricks on GCP

• Collaborate with Product Management and business partners to understand use case requirements and reporting

• Adhere to internal development best practices/lifecycle (e.g. Testing, Code Reviews, CI/CD, Documentation)

• Document and showcase feature designs/workflows

• Participate in team meetings and discussions around product development

• Stay up to date on industry latest industry trends and design patterns

• 3+ years experience with GIT

• 3+ years experience with CI/CD (e.g. Azure Pipelines)

• Experience with streaming technologies, such as Kafka, Spark

• Experience building applications on Docker and Kubernetes

• Cloud experience (e.g. Azure, Google)