Job Description

Key Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines using Azure Databricks and its native services, including Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Cosmos DB, and Azure Data Lake Storage.
• Collaborate with cross-functional teams to understand data requirements, develop technical specifications, and implement solutions that meet business needs.
• Perform data profiling, cleansing, and transformation to ensure data quality, consistency, and accuracy throughout the data lifecycle.
• Optimize data pipelines for performance, reliability, and cost-effectiveness, leveraging cloud-native technologies and best practices.
• Implement security and compliance measures to protect sensitive data and ensure regulatory compliance, including data encryption, access controls, and auditing.
• Monitor and troubleshoot data pipeline issues, proactively identifying and resolving issues to minimize downtime and ensure data availability.
• Stay current with emerging technologies and trends in data engineering and cloud computing and evaluate opportunities for innovation and optimization.
• Mentor junior team members, providing technical guidance, training, and support to foster their professional development and growth.

Qualifications:

• Bachelor's or master’s degree in computer science, Engineering, or a related field.
• 6 to 8 years of experience working as a Data Engineer or a similar role, with a focus on cloud-based data platforms and services, particularly Azure Databricks and its native services.
• Strong proficiency in SQL, Python, and Spark, with hands-on experience developing ETL processes and data pipelines.
• Experience working with Azure cloud services such as Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Cosmos DB, and Azure Data Lake Storage.
• Knowledge of data modelling techniques, dimensional modelling, and data warehousing concepts.
• Excellent analytical and problem-solving skills, with the ability to troubleshoot complex data issues and implement effective solutions.
• Strong communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.

Required Knowledge, Skills, and Abilities

• Python
• Azure databricks (adb)
• ETL
• SQL