Responsibilities
- Develop real time data pipelines and solve big data problems utilizing modern technologies like Spark, Python, Java/Scala, SQL, and AWS cloud environment.
- Write reliable, maintainable, well-documented code that will scale to support millions of respondents.
- Create and maintain optimal data pipeline architecture.
- Execute the full software development life cycle as part of an Agile team.
- Collaborate with Product Managers to refine and modify requirements.
- Follow established design paradigms and design patterns.
- Utilize abilities in unit testing and integration testing.
- Participate in the on-call rotation to monitor shipped features for success and issues.
- Support the business teams and product managers in data extracts and data analysis.
- Demonstrate proficiency in developing software for the user interface, business logic, data modeling and systems, and component integration.
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
- Perform pull requests reviews and CI/CD pipeline integration.
- Develop and manage end-to-end ETL pipelines for both real-time and batch processing, ensuring the secure and efficient migration of large-scale datasets from MSSQL environments to cloud-based data warehouses.
- Engineer robust streaming architecture using SNS, Kinesis Data Streams, and Kinesis Data Analytics to enable real-time insights and schema-validated ingestion, ensuring high-quality data delivery.
- Utilize the following tools and technologies: Github, SQL, REST APIs, and Jira.
Requirements
- Proficiency in Spark, Python, Java/Scala, SQL, and AWS cloud environment.
- Experience writing reliable, maintainable, well-documented code that scales.
- Experience creating and maintaining data pipeline architecture.
- Experience executing the full software development life cycle in an Agile environment.
- Ability to collaborate with Product Managers to refine and modify requirements.
- Knowledge of established design paradigms and design patterns.
- Experience with unit testing and integration testing.
- Willingness to participate in on-call rotation.
- Ability to support business teams and product managers in data extracts and data analysis.
- Proficiency in developing software across user interface, business logic, data modeling, systems, and component integration.
- Experience building analytics tools that provide actionable insights into key business performance metrics.
- Experience with pull request reviews and CI/CD pipeline integration.
- Experience developing and managing end-to-end ETL pipelines for real-time and batch processing.
- Experience migrating large-scale datasets from MSSQL to cloud-based data warehouses securely and efficiently.
- Experience engineering streaming architecture using SNS, Kinesis Data Streams, and Kinesis Data Analytics.
- Familiarity with Github, SQL, REST APIs, and Jira.
Work Arrangement
Remote (Worldwide) — United States
Additional Information
- Position allows work from home anywhere in the United States.
- Headquarters located at 1235 Saint Thomas Street, Suite 201, New Orleans, LA 70130.
