Responsibilities
- Design and manage robust, scalable data pipelines using Python, SQL, and cloud-based Azure technologies.
- Create and refine ETL and ELT processes to handle enterprise-level data volumes efficiently.
- Utilize core Azure tools including Data Factory, Databricks, Data Lake, and Synapse Analytics for data solutions.
- Develop flexible frameworks to ingest data from APIs, relational databases, flat files, and cloud platforms.
- Apply PySpark and Pandas for data transformation, cleaning, validation, and performance tuning.
- Architect and implement data lake and lakehouse environments using Medallion patterns.
- Write efficient SQL queries, stored procedures, and logic for data integrity checks.
- Monitor data workflows, resolve performance issues, and ensure high data quality standards.
- Partner with data analysts, BI developers, architects, and business teams to align technical deliverables with business goals.
- Establish CI/CD pipelines and automate deployments following version control best practices.
- Document technical designs, operational procedures, and data processing standards.
- Assist in transitioning on-premise systems to the cloud and modernizing legacy data infrastructure.
Compensation
Competitive salary and benefits package
Work Arrangement
Hybrid or remote options available
Team
Collaborative engineering team focused on data innovation
Responsibilities
- Design and manage robust, scalable data pipelines using Python, SQL, and cloud-based Azure technologies.
- Create and refine ETL and ELT processes to handle enterprise-level data volumes efficiently.
- Utilize core Azure tools including Data Factory, Databricks, Data Lake, and Synapse Analytics for data solutions.
- Develop flexible frameworks to ingest data from APIs, relational databases, flat files, and cloud platforms.
- Apply PySpark and Pandas for data transformation, cleaning, validation, and performance tuning.
- Architect and implement data lake and lakehouse environments using Medallion patterns.
- Write efficient SQL queries, stored procedures, and logic for data integrity checks.
- Monitor data workflows, resolve performance issues, and ensure high data quality standards.
- Partner with data analysts, BI developers, architects, and business teams to align technical deliverables with business goals.
- Establish CI/CD pipelines and automate deployments following version control best practices.
- Document technical designs, operational procedures, and data processing standards.
- Assist in transitioning on-premise systems to the cloud and modernizing legacy data infrastructure.
Available for qualified candidates
