Big Data Consultant

Big Data Engineer

Rockville, Maryland (Hybrid)

Long term contract

NO C2C PLEASE

ConsultNet is seeking a Big Data Engineer to support an ongoing effort in Rockville, Maryland. This role has onsite requirements with a minimum of two days per week onsite in the office.

Responsibilities:
• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
• Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
• Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.
• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
• Work with data scientists and analysts to support data-driven decision-making across the organization.
• Ability to write and maintain automated unit, integration, and end-to-end tests
• Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.

Preferred:
• Experience with big data tools such as Hadoop, Spark, and Presto/Trino
• Experience with production data pipelines/ETL systems
• Comfortable in AWS console and using AWS cli tools
• AWS certifications