AWS Data Engineer
Description
One of our client is adding a fully remote Data Engineer (AWS) to an existing team of Engineers supporting a data lake (RDL - Renewables Data Lake). They are supporting an AWS stack as listed below which is the platform that houses all the data and intelligence collected from all their sites in North America. This is a 9-month contract with the possibility to extend.
Scope of Work
The Renewable Data Lake (RDL) is focused on building renewable asset optimization solutions around the world and transforming the data into meaningful metrics for our customers (internal stakeholders).
Our team ingests massive volumes of data and generates a series of KPIs that help us and our customers understand how our sites are performing, where and why they are under-performing, what the most common problems are, etc.
The number of sites and the volume of data is growing fast. We'll need your help building smart, reliable, and performant solutions to make the most of the data.
Our current stack includes Python Redshift, RDS, Glue, PySpark, Lambdas, S3, the AWS developer tools, and other AWS technologies.
If you have good ideas for alternative solutions, we want to hear about it!
Responsibilities
• 60%: Design, develop, and maintain data applications and solutions to enable efficient data processing, analysis, and visualization
• 5%: Collaborate with data scientists, analysts, and business stakeholders to understand requirements and translate them into scalable and efficient data solutions
• 5%: Optimize and tune data applications for performance and scalability
• 5%: Implement data quality checks and ensure data integrity and consistency
• 5%: Lead projects and initiatives, providing technical guidance and ensuring successful delivery
• 5%: Manage project timelines, resources, and deliverables to meet business objectives
• 5%: Mentor and coach junior team members, fostering their growth and development
• 10% - Other Duties as Assigned
Qualifications
Education/Experience
• 10+ years of experience working in software engineering
• 6+ years working with Big Data in a cloud environment, AWS strongly preferred
• Bachelor's degree in Computer Science or other related technical field or relevant experience
• Experience developing multi-platform software utilizing various programming environments and tools
• Knowledge of agile methodologies
• Knowledge of QA processes and test-driven development
Skills/Knowledge/Abilities
• Strong experience with database design, specifically geared towards Big Data
• Familiar with serverless approach, micro-service-oriented architecture, event-driven architecture
• Strong experience working in a AWS cloud technology environment: Lambda functions, Step functions, SQS, DynamoDB, Spark with Glue, along with EMR, EMR serverless, Spark streaming
• Experience building and maintaining critical, reliable ETL pipelines
• Experience writing high-quality, production-ready code, preferably in Python
• Comfortable with Git (AWS CodeCommit) and Unix command lines
• DevOps experience: strive for scalability and automation via CI/CD and Infrastructure as Code. A plus if knowledge of CDK
• Proactive communicator who can translate between technical and non-technical stakeholders
• Self-starter and autonomous, strong bias for action in high ambiguity environment
• Explore and experiment with the limits of cloud technologies
• Help mentor and educate your teammates (and vice versa) to expand everyone's skillsets
• Displays personal and corporate integrity
Description
One of our client is adding a fully remote Data Engineer (AWS) to an existing team of Engineers supporting a data lake (RDL - Renewables Data Lake). They are supporting an AWS stack as listed below which is the platform that houses all the data and intelligence collected from all their sites in North America. This is a 9-month contract with the possibility to extend.
Scope of Work
The Renewable Data Lake (RDL) is focused on building renewable asset optimization solutions around the world and transforming the data into meaningful metrics for our customers (internal stakeholders).
Our team ingests massive volumes of data and generates a series of KPIs that help us and our customers understand how our sites are performing, where and why they are under-performing, what the most common problems are, etc.
The number of sites and the volume of data is growing fast. We'll need your help building smart, reliable, and performant solutions to make the most of the data.
Our current stack includes Python Redshift, RDS, Glue, PySpark, Lambdas, S3, the AWS developer tools, and other AWS technologies.
If you have good ideas for alternative solutions, we want to hear about it!
Responsibilities
• 60%: Design, develop, and maintain data applications and solutions to enable efficient data processing, analysis, and visualization
• 5%: Collaborate with data scientists, analysts, and business stakeholders to understand requirements and translate them into scalable and efficient data solutions
• 5%: Optimize and tune data applications for performance and scalability
• 5%: Implement data quality checks and ensure data integrity and consistency
• 5%: Lead projects and initiatives, providing technical guidance and ensuring successful delivery
• 5%: Manage project timelines, resources, and deliverables to meet business objectives
• 5%: Mentor and coach junior team members, fostering their growth and development
• 10% - Other Duties as Assigned
Qualifications
Education/Experience
• 10+ years of experience working in software engineering
• 6+ years working with Big Data in a cloud environment, AWS strongly preferred
• Bachelor's degree in Computer Science or other related technical field or relevant experience
• Experience developing multi-platform software utilizing various programming environments and tools
• Knowledge of agile methodologies
• Knowledge of QA processes and test-driven development
Skills/Knowledge/Abilities
• Strong experience with database design, specifically geared towards Big Data
• Familiar with serverless approach, micro-service-oriented architecture, event-driven architecture
• Strong experience working in a AWS cloud technology environment: Lambda functions, Step functions, SQS, DynamoDB, Spark with Glue, along with EMR, EMR serverless, Spark streaming
• Experience building and maintaining critical, reliable ETL pipelines
• Experience writing high-quality, production-ready code, preferably in Python
• Comfortable with Git (AWS CodeCommit) and Unix command lines
• DevOps experience: strive for scalability and automation via CI/CD and Infrastructure as Code. A plus if knowledge of CDK
• Proactive communicator who can translate between technical and non-technical stakeholders
• Self-starter and autonomous, strong bias for action in high ambiguity environment
• Explore and experiment with the limits of cloud technologies
• Help mentor and educate your teammates (and vice versa) to expand everyone's skillsets
• Displays personal and corporate integrity
