Job Description
This profile must have at least 6 years' experience and the prerequisites, in order of priority, are as follows:
- Python & software engineering
- Azure Cloud & DevOps
- Data Engineering with databricks
Senior Platform Data Engineer
You will work within a product team to design, develop, maintain, and evolve an internal developer platform (IDP). This platform provides tools to accelerate and standardize the delivery of digital, data, and AI products within the Azure ecosystem.
Main Activities
- Platform Development:
- Implement reusable modules in Python.
- Develop command-line features for deploying and managing code in Azure.
- Maintain and improve code generation templates.
- Release Process Maintenance:
- Set up CI pipelines to automate testing and documentation.
- Set up CD pipelines to automate delivery with release notes.
- Implement testing strategies to ensure adequate coverage.
- User Support:
- Create documentation and knowledge bases.
- Participate in demonstrations of new features.
- Assist and troubleshoot user issues.
- Collaboration with Data Experts:
- Assist Data Scientists and MLEs with model experiments and deployments.
- Help Data Engineers with Databricks tasks.
- Support Data Analysts in industrializing their DBT workflows.
- Security and Compliance:
- Adhere to industry standards and privacy regulations.
- Integrate automated security and compliance checks.
- Continuous Improvement:
- Stay up-to-date on technological advancements.
- Propose and implement optimizations.
- Proactively identify areas for improvement and automation.
Required Skills
- Python and Software Engineering:
- Proficiency in Python and related tools (poetry, venv, sphinx, mkdocs).
- Software engineering practices: design patterns, testing, clean code.
- DevOps:
- Experience with platforms like Azure DevOps.
- Implementation of CI/CD pipelines.
- Knowledge of containerization (Docker).
- Cloud Platform:
- Expertise in Azure Cloud (AKS, Functions, App Services).
- Advanced usage of Databricks.
- Data Processing:
- Knowledge of PySpark.
- Familiarity with data storage formats like Parquet and Delta.
- Knowledge of DBT is a plus.
- Security and Compliance:
- Understanding of role-based access controls.
- Knowledge of Azure service accounts.
- Monitoring and Troubleshooting:
- Familiarity with monitoring tools like Azure Monitor.
- Ability to debug applications and data pipelines.
Priority Technologies
- Languages and Tools: Python, CLI, Cookiecutter, Databricks, DBT.
- DevOps and CI/CD: Azure DevOps, Docker.
- Cloud: Azure, Azure Databricks, Azure Machine Learning Services
