Spark Data Engineer (6 Month Contract)

Job Title: Spark Data Engineer

Location: Virtual/Hybrid, occasional travel to Oxford

Reporting to: Head of Data Engineering

Contract Type: 6-month contract

Salary: Daily Rate Outside IR35

Who we are

Akrivia Health are global leaders in the application of real-world data & evidence for mental health

and dementias, providing valuable insights for research. With the largest and richest repository of

real-world data in the world, we enable our clients and collaborators to accelerate clinical trials and

to identify, develop and deliver effective new drugs, devices and services to patients and caregivers.

We provide our research support and data curation services to the NHS for free, in order to support

mental health provision, service improvement and improved patient outcomes across our network.

Our Precision Neuroscience Initiative – GlobalMinds – is creating the UK’s largest biobank of patients

with mental health conditions to transform research and alleviate disease burden in this area of

critical unmet medical needs.

Duties & Responsibilities

We are seeking an experienced Spark Data Engineer to work closely with the Head of Data

Engineering in building a fully configurable, Kubernetes hosted Spark Data Ingestion application. The

application will be responsible for consuming data from various Kafka streams, validating and

transforming the data before being pushed to our delta lake. It will also be used by our AI Operations

team to run NLP workflows.

Key Responsibilities
• Attend daily stand-ups with Project team keeping members apprised of progress
• Sprint ceremony demonstrations of work completed
• Work closely with Research team to understand how they want to validate their data
• Work closely with the permanent members of the Data Engineering team collaborating on application build out & ensuring knowledge transfer sessions are frequent
• Keeping documentation & runbooks up to date
• Weekly consultation with Head of Data Engineering on architecture & design

Essential
• Delta Tables: Knowledge of Delta tables and their use in conjunction with Spark
• Configuration: Use of Jinja templating JSON to configure downstream tasks
• Orchestration: Experience using task orchestration programs like Airflow
• Sql & NoSql: Understanding of NoSql environments such as MongoDB & Redis
• Coding Standards: Good understanding of modular programming & abstraction
• Testing: Strong commitment to setting up good battery of unit tests for validating transforms applied to data
• Reporting & Auditing: An understanding of how to report on data lineage and surfacing data transform actions back to end consumer
• Containerization & Orchestration: Advanced knowledge of Docker and Kubernetes for managing and scaling applications and usage of docker-compose for local testing of services (including running Airflow, Spark + Kafka locally)
• Problem-solving: Excellent troubleshooting skills and a proactive problem-solving mindset

Desirable
• Understanding of avro & parquet file formats
• Understanding of test-driven development or business driven development
• Knowledge of machine learning operations (MLOps) for supporting AI/ML workflows
• Basic understanding of spark-nlp library
• Experience working cross-functionally with AI and/or research teams

Our Culture

This is an exciting opportunity to join a dynamic and friendly team who are passionate about making

positive changes in people’s lives. At Akrivia Health, our culture is one of integrity, respect,

collaboration and trust.

Our commitment to equality, diversity and inclusion

At Akrivia Health we understand that a diversity of perspectives not only fosters innovation,

creativity and learning, but is also crucial for understanding and addressing the challenges in mental

health and dementia. We are a committed equal opportunities employer and encourage applications

from all individuals, regardless of their race, gender, disability or background.

Right to work:

We have a legal responsibility to ensure that you have the right to work in the UK before you can

start working for us. If you do not have the right to work in the UK already, any offer of employment

we make to you will be conditional upon you gaining it.

Please submit your CV and cover letter to recruitment@akriviahealth.com by 28th February 2025 Due

to the high volume of applications, we are only able to respond to those selected for interview. If

you require any reasonable adjustments during the interview process, please do let us know so we

can make suitable arrangements for you.

To find out more about us please visit: https://akriviahealth.com/