Data Engineer (Python, Spark, Pandas, SQL, GX Open Source Module)
Job Title: Data Engineer (Python, Spark, Pandas, SQL, GX Open Source Module)
Job Summary:
We are looking for a skilled Data Engineer with expertise in Python, Spark, Pandas, SQL, and Great Expectations (GX) Open Source Module to join our dynamic team. The ideal candidate will play a key role in building, optimising, and maintaining data pipelines, ensuring data quality, and supporting business intelligence and analytics needs.
Key Responsibilities:
• Design, develop, and optimize scalable ETL/ELT pipelines using Python and Apache Spark.
• Work with Pandas to process, clean, and transform large datasets efficiently.
• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Implement data validation and quality checks using the Great Expectations (GX) Open Source Module.
• Monitor and troubleshoot data pipeline performance issues, ensuring high availability and reliability.
Required Skills & Experience:
• Strong proficiency in Python, with experience in writing efficient and scalable code for data processing.
• Hands-on experience with Apache Spark for large-scale data processing.
• Expertise in Pandas for data manipulation and transformation.
• Solid understanding of SQL and relational database concepts, with experience in query optimization.
• Experience working with Great Expectations (GX) Open Source Module for data validation and quality assurance.
• Familiarity with cloud-based data platforms (AWS, Azure, GCP) is a plus.
• Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment.
Preferred Qualifications:
• Experience with big data technologies such as Hadoop, Delta Lake, or Snowflake.
• Understanding of CI/CD for data pipelines and version control (Git).
• Exposure to containerization tools like Docker and orchestration frameworks like Airflow.
If you are passionate about data engineering, quality assurance, and scalable data solutions, we’d love to hear from you! Apply now and be part of an exciting journey in data-driven innovation.
Job Title: Data Engineer (Python, Spark, Pandas, SQL, GX Open Source Module)
Job Summary:
We are looking for a skilled Data Engineer with expertise in Python, Spark, Pandas, SQL, and Great Expectations (GX) Open Source Module to join our dynamic team. The ideal candidate will play a key role in building, optimising, and maintaining data pipelines, ensuring data quality, and supporting business intelligence and analytics needs.
Key Responsibilities:
• Design, develop, and optimize scalable ETL/ELT pipelines using Python and Apache Spark.
• Work with Pandas to process, clean, and transform large datasets efficiently.
• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Implement data validation and quality checks using the Great Expectations (GX) Open Source Module.
• Monitor and troubleshoot data pipeline performance issues, ensuring high availability and reliability.
Required Skills & Experience:
• Strong proficiency in Python, with experience in writing efficient and scalable code for data processing.
• Hands-on experience with Apache Spark for large-scale data processing.
• Expertise in Pandas for data manipulation and transformation.
• Solid understanding of SQL and relational database concepts, with experience in query optimization.
• Experience working with Great Expectations (GX) Open Source Module for data validation and quality assurance.
• Familiarity with cloud-based data platforms (AWS, Azure, GCP) is a plus.
• Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment.
Preferred Qualifications:
• Experience with big data technologies such as Hadoop, Delta Lake, or Snowflake.
• Understanding of CI/CD for data pipelines and version control (Git).
• Exposure to containerization tools like Docker and orchestration frameworks like Airflow.
If you are passionate about data engineering, quality assurance, and scalable data solutions, we’d love to hear from you! Apply now and be part of an exciting journey in data-driven innovation.
