Data Engineer - Python/Pyspark - Contract - Outside IR35
Please Read Full Summary Before Applying
Methods are looking for 3 - 4 Data Engineers with experience using Pyspark in Python for a new contract engagement. You will need to be able to commit to 4 or 5 days on a client site in the West Midlands. This role will require SC clearance or to be willing to go through the SC process.
This role also requires Sole UK Nationals.
We are seeking a seasoned Senior Data Engineer (Infrastructure) to join our team. This role is essential for designing, building, and maintaining sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The position involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, crucial for our organisational growth and innovation.
Requirements
• Develop and Manage Data Pipelines: You will design, construct, and maintain efficient and reliable data pipelines using Python, capable of supporting both streaming and batch data processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.
• Hybrid Cloud and Data Storage Solutions: Implement and manage data storage solutions leveraging both on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.
• Containerisation and Orchestration: Utilise Docker for containerisation and Kubernetes for orchestration, ensuring scalable and efficient deployment of applications across both cloud-based and on-premises environments.
• Workflow Automation: Employ tools such as Apache NiFi and Apache Airflow to automate data flows and manpyage complex workflows within hybrid environments.
This role will require you to have or be willing to go through Security Clearance. As part of the onboarding process candidates will be asked to complete a Baseline Personnel Security Standard; details of the evidence required to apply may be found on the government website Gov.UK. If you are unable to meet this and any associated criteria, then your employment may be delayed, or rejected.
Details of this will be discussed with you at interview
