As a Data Engineer, you will play a key role in developing and maintaining robust data infrastructure on Microsoft Azure. You'll be responsible for creating efficient data workflows that power analytics, reporting, and advanced data science applications.
Key Responsibilities
- Design and implement data integration pipelines using Azure Data Factory, Databricks, and Synapse Analytics to support enterprise-wide data needs.
- Develop and maintain ETL and ELT processes that move and transform data from diverse sources into cloud data platforms like Azure Data Lake and data warehouses.
- Ensure data processing jobs are optimized for speed, reliability, and resource efficiency across distributed environments.
- Work closely with data architects, analysts, and business teams to define requirements and deliver accurate, timely data solutions.
- Establish data validation processes, monitoring systems, and alerting mechanisms to maintain data integrity.
- Support automated deployment through CI/CD pipelines, using Git and infrastructure-as-code practices for consistent, repeatable releases.
Qualifications
- Minimum of five years in data engineering or a similar role, with proven experience in cloud data platforms and large-scale data systems.
- Direct experience with core Azure services including Databricks, Data Factory, Data Lake Storage, and Synapse Analytics.
- Strong programming skills in Python and SQL, with expertise in data transformation and pipeline development.
- Solid grasp of distributed computing frameworks like Apache Spark and techniques for performance tuning.
- Experience with DevOps workflows, version control (Git), and CI/CD in cloud environments.
- Background in data modeling and the ability to diagnose and improve complex data workflows.
