Python Data Engineer

Job Title: Python Data Engineer

Job Description:

We are seeking a highly skilled Python Data Engineer to join our team and contribute to the development of scalable data pipelines, APIs, and containerized applications. The ideal candidate will have a strong background in Python development, data processing, and cloud storage solutions.

Responsibilities:
• Implement and maintain unit tests using Python unit test frameworks (e.g., pytest, unittest).
• Develop and optimize data processing workflows using Pandas and NumPy for large-scale data manipulation.
• Build and manage Apache Airflow pipelines for automated data ingestion and transformation.
• Containerize applications using Docker/Kubernetes for scalable deployment.
• Design and implement RESTful APIs using Flask or similar frameworks.
• Develop and manage caching solutions with Redis for high-performance data retrieval.
• Apply composition patterns over inheritance for maintainable and scalable software architecture.
• Write optimized SQL queries and integrate them into Python applications using appropriate modules (e.g., SQLAlchemy, psycopg2).
• Work with MinIO S3 storage, handling Parquet files efficiently using Pandas and the MinIO Python SDK.

Requirements:
• Strong proficiency in Python with experience in data engineering and backend development.
• Hands-on experience with Pandas, NumPy, and other Python data processing libraries.
• Experience developing and orchestrating workflows using Apache Airflow.
• Proficiency in Docker and containerized application development.
• Strong understanding of REST API development using Flask or equivalent frameworks.
• Experience working with Redis as a caching mechanism.
• Familiarity with unit testing frameworks such as pytest or unittest.
• Understanding of software design patterns, particularly composition vs inheritance.
• Ability to write and optimize SQL queries and integrate them into Python applications.
• Experience working with MinIO S3 storage and handling Parquet files in Pandas.

Submit resume to jobs@OSIengineering.com

Location: Remote

Job Type: Contract

Pay Rate: $60-$75/h) (DOE)

Tony Do

408.550.2800 x115