Data Engineer
We are seeking a Data Engineer to help design and develop a fully configurable, Kubernetes-hosted Spark Data Ingestion application. This solution will ingest, validate, and transform data from a variety of Kafka streams before pushing it to the client’s Delta Lake. Additionally, the platform will support the AI Operations team’s NLP workflows. The ideal candidate will be comfortable collaborating with cross-functional teams and will ensure thorough documentation and knowledge transfer across the organisation.
Responsibilities
• Participate in daily stand-ups to keep the project team informed about progress, blockers, and next steps.
• Prepare and present work completed during each sprint ceremony, showcasing application features and improvements.
• Collaborate closely with the Research team to understand their data validation requirements and integrate these checks into the ingestion workflow.
• Work closely with permanent members of the Data Engineering team to co-build the application, ensuring that knowledge transfer is continuous and well-documented.
• Maintain up-to-date documentation and runbooks detailing system architecture, configuration options, deployment steps, and operational support procedures.
• Conduct weekly consultations with the Head of Data Engineering to discuss and refine system architecture, ensuring alignment with best practices and business objectives.
We are seeking a Data Engineer to help design and develop a fully configurable, Kubernetes-hosted Spark Data Ingestion application. This solution will ingest, validate, and transform data from a variety of Kafka streams before pushing it to the client’s Delta Lake. Additionally, the platform will support the AI Operations team’s NLP workflows. The ideal candidate will be comfortable collaborating with cross-functional teams and will ensure thorough documentation and knowledge transfer across the organisation.
Responsibilities
• Participate in daily stand-ups to keep the project team informed about progress, blockers, and next steps.
• Prepare and present work completed during each sprint ceremony, showcasing application features and improvements.
• Collaborate closely with the Research team to understand their data validation requirements and integrate these checks into the ingestion workflow.
• Work closely with permanent members of the Data Engineering team to co-build the application, ensuring that knowledge transfer is continuous and well-documented.
• Maintain up-to-date documentation and runbooks detailing system architecture, configuration options, deployment steps, and operational support procedures.
• Conduct weekly consultations with the Head of Data Engineering to discuss and refine system architecture, ensuring alignment with best practices and business objectives.
