Job Title: Data Engineer

Location: Remote

Contract: £500 Per Day

Role Overview

Our client is seeking a Data Engineer with a solid foundation in both data engineering and business intelligence, capable of understanding and addressing business needs. This role involves managing and optimizing AWS-based data processes, orchestrating data flows, and supporting business intelligence capabilities. The ideal candidate will not only bring technical expertise but also a strong business acumen to facilitate effective collaboration with non-technical stakeholders.

Key Responsibilities

• Data Pipeline Management:

• Database Integration: Maintain MySQL databases on AWS (approx. 25TB) as primary data sources feeding the data pipeline.

• Real-Time Data Ingestion: Utilize Apache Kafka to manage data flows from MySQL into AWS S3 buckets, enabling real-time streaming.

• Data Processing: Implement ETL processes using AWS Glue and PySpark to build data lakes within S3 for further analysis.

• Business Intelligence (BI):

• Facilitate data availability for BI purposes, supporting daily updates and aiming to move from overnight batch jobs to near-real-time 15-minute refreshes.

• Collaborate on BI dashboard and reporting creation to provide actionable insights.

• Complex Data Reconciliation:

• Reconcile transaction records across multiple merchants, currencies, and network operators, addressing both straightforward and complex data relationships.

Technical Requirements

• Data Engineering Expertise:

• Proficiency with MySQL, Apache Kafka, AWS Glue, PySpark, and AWS S3 for managing data pipelines and ETL processes.

• Business Intelligence Skills:

• Experienced in designing and maintaining BI reports and dashboards with frequent data refresh capabilities.

• Data Reconciliation Experience:

• Skilled in reconciling transactional data across diverse systems, operators, and currencies to ensure data accuracy.

Business & Communication Skills

• Business Acumen:

• Ability to translate business requirements into data solutions and work collaboratively with non-technical stakeholders to develop reports and dashboards tailored to business needs.

• Communication:

• Strong communication skills to bridge technical and business needs, working closely with stakeholders to ensure data solutions support strategic objectives.