Azure Data Engineer (NCS/Job/ 3866)

Job Summary

We are seeking a skilled Azure Data Engineer with 4–6 years of experience in designing, developing, and maintaining scalable data solutions on Microsoft Azure. The ideal candidate should have strong expertise in data integration, ETL/ELT development, cloud-based data platforms, and data warehousing solutions. The role involves working closely with business stakeholders, data architects, and analytics teams to build robust and efficient data pipelines that support enterprise reporting and advanced analytics initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure services.
• Build and optimize data ingestion frameworks from various structured and unstructured data sources.
• Develop and manage data solutions using Azure Data Factory (ADF), Azure Databricks, and Azure Synapse Analytics.
• Create and maintain data models, data marts, and enterprise data warehouse solutions.
• Implement data transformation, cleansing, and validation processes to ensure data quality and integrity.
• Develop and optimize SQL queries, stored procedures, and database objects for high-performance data processing.
• Collaborate with business analysts, data scientists, and reporting teams to deliver reliable data solutions.
• Monitor, troubleshoot, and optimize data pipelines and workflows.
• Implement security, governance, and compliance standards for cloud data platforms.
• Participate in code reviews, performance tuning, and deployment activities.
• Support CI/CD implementation and automation for data engineering projects.

Required Skills

Azure Data Engineering

• Strong experience with:
  • Azure Data Factory (ADF)
  • Azure Databricks
  • Azure Synapse Analytics
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure SQL Database
  • Azure Blob Storage

Data Engineering & ETL

• Experience in designing and developing ETL/ELT pipelines.
• Strong understanding of data warehousing concepts and dimensional modeling.
• Experience handling large-scale datasets and distributed data processing.

Programming & Querying

• Strong proficiency in SQL.
• Hands-on experience with Python and/or PySpark.
• Experience in performance tuning and query optimization.

Cloud & DevOps

• Knowledge of CI/CD pipelines using Azure DevOps.
• Experience with Git-based source control.
• Understanding of cloud security, monitoring, and governance best practices.

Preferred Skills

• Experience with Delta Lake and Lakehouse architecture.
• Knowledge of real-time data processing and streaming technologies.
• Exposure to Power BI integration and reporting solutions.
• Experience with Agile/Scrum methodologies.
• Microsoft Azure certifications are preferred.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 4–6 years of experience in Data Engineering, with significant hands-on experience in Azure cloud technologies.

Good to Have

• Azure Data Engineer Associate certification.
• Experience with Spark optimization and distributed computing.
• Exposure to data governance and metadata management tools.
• Knowledge of modern data architecture patterns and best practices.

Key Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently and collaboratively in a hybrid work environment.
• Strong attention to detail and commitment to delivering high-quality solutions.