Data Engineer(N-170524501739)

• Develop new data pipelines using Databricks notebooks and Azure Data Factory to ingest and process data efficiently, ensuring reliability and scalability.
• Utilize Databricks and Delta tables to optimize the performance of both new and existing data processing jobs, aiming to reduce operational costs and improve efficiency.
• Maintain the data platform focusing on process monitoring, troubleshooting, and data readiness, ensuring high-quality data for regular reporting and system optimization.
• Work with other data engineers to design and implement enhancements to the overall data platform, improving functionality and performance.
• Effectively collaborate with operations, product management, and other departments to gather requirements, troubleshoot issues, and design system enhancements within an Agile SCRUM framework.
• Participate in on-call support, addressing and resolving production issues as they arise, and coordinate with stakeholders to ensure continuous system operation.
• Ensure a smooth transition of developed data pipelines to the L2 Support team for post-production maintenance, reducing the need for escalations.
• Lead the end-to-end implementation of multiple competing data projects, prioritizing and managing them simultaneously from development through to deployment within the Agile Scrum framework, while demonstrating autonomy, strategic project management skills, and initiative.
• Use DataFrame or PySpark operations to extract data from Azure Delta Lake, creating reports that support business decisions and meet client needs.
• Actively engage in release activities, coordinating with cloud engineering teams for necessary infrastructure requirements.
• Efficiently onboard new team members and guide users of the data cloud platform, troubleshooting and resolving any access or permission issues to ensure full utilization of the platform for their projects.
• Strategically manage and integrate third-party data sources to complement and enhance our proprietary POS data, maximizing data value and insights.
• Actively explore and evaluate new technologies or features through proof of concept (POC) and proof of value (POV) projects, driving innovation and technological advancement.
• Build or improve data pipelines focusing on compliance, ensuring adherence to GDPR, CCPA, and other relevant regulations, and safeguarding data privacy and security.
• Provide mentorship and guidance to junior data engineers, fostering their professional growth and development in data engineering practices while encouraging innovation and collaboration within the team.

Job Responsibilities:

• Advance cloud security by enhancing protect, detect, and respond capabilities, focusing on control implementation, architectural design improvements, and secure software development processes.
• Strategically design, implement, and maintain robust cloud security solutions across Microsoft Azure, AWS, and Oracle Cloud, ensuring comprehensive coverage and protection.
• Proactively collaborate with application developers and other cross-functional teams to embed security best practices into Azure architecture and development lifecycle.
• Develop and implement security policies, standards, and guidelines for Azure-based environments in compliance with industry regulations and best practices.
• Keep abreast of industry trends and emerging technologies in Azure cloud security and application development, recommending strategic improvements for enhanced security posture.
• Effectively remediate exploitable vulnerabilities in cloud environments by collaborating with cloud and product engineers and building robust processes to address security concerns as required.
• Perform thorough security assessments and vulnerability testing on Azure-based systems and applications, proactively identifying, and mitigating potential security risks.
• Create and implement strategic plans for timely remediation of identified vulnerabilities in collaboration with relevant technical teams.
• Provide expert guidance as a Cloud Security Subject Matter Expert (SME) in managing and resolving security incidents, leveraging in-depth knowledge to support and enhance the organizations incident response capabilities.
• Actively engage in responding to and resolving security incidents, particularly focusing on Azure-based breaches, contributing expertise to effectively manage and mitigate incidents.
• Vigilantly monitor Azure-specific security logs and alerts, conduct in-depth analysis to identify potential threats, and implement timely actions to reduce security risks.
• Offer guidance and mentorship to junior security team members and play a pivotal role in knowledge sharing, training initiatives, and fostering a culture of continuous learning in cloud security.