Purpose: responsible for designing, developing, and maintaining data pipelines and infrastructure to support our data analytics and business intelligence initiatives.

Specifications but not limited to:

• Design, build, and maintain data pipelines and ETL processes.
• Collaborate with data scientists and analysts to understand data requirements and ensure data availability.
• Optimize data processing and transformation for performance and scalability.
• Troubleshoot and resolve data-related issues and errors.
• Migrate data pipelines from legacy systems.
• Implement data quality checks and monitoring to ensure data integrity.
• Work with cloud-based data storage systems and data warehouses in Azure and Databricks.
• Develop and maintain documentation for data engineering processes and data pipelines.
• Define and implement data models and data architecture that support business needs.
• Stay updated on industry best practices and emerging technologies in data engineering, data modeling, and data architecture

Qualifications and Experience:

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• 5+ years Proven experience as a Data Engineer or a similar role.
• Strong proficiency in Python and Apache Spark.
• Experience working with cloud platforms such as Azure and Databricks.
• Proficiency in SQL for data manipulation and transformation.
• Familiarity with data warehousing, data architecture, and best practices.
• Knowledge of containerization technologies (e.g., Docker, Kubernetes).
• Experience with Git version control systems.
• Experience with DBT or Delta Live tables.
• Understanding of Azure data security and compliance best practices.
• Knowledge of data streaming and real-time processing technologies.
• Proficiency and experience in a programming language